Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[6.3] Fix rocprof-sys-sample cannot get GPU related tracing #87

Open
wants to merge 1 commit into
base: release/rocm-rel-6.3
Choose a base branch
from

Conversation

adjordje-amd
Copy link

@adjordje-amd adjordje-amd commented Jan 20, 2025

After unloading the RCCL library, the library destructor triggers undefined behavior by invoking OnLoad function.
Disable the RCCL backend to avoid issues with the library destructor and undefined behavior.

@adjordje-amd adjordje-amd force-pushed the fix/smi-stats-missing branch 6 times, most recently from e0cfaf7 to e084087 Compare January 20, 2025 15:36
@dgaliffiAMD
Copy link
Collaborator

Thanks, Aleks.
This does fix the original issue for the transpose example; rocprof-sys-sample --PTDH -I all -- ./transpose now includes "rocm-smi" samples. However, it seems to have introduced a new type failure for certain workloads. For example, when I run rocprof-sys-sample -PTDH -I all -v 2 -- ls -al, the trace fails to exit. We get stuck while tearing down:

[rocprof-sys][597260] Removing temporary file '/tmp/rocprofsys-ls-output/281783/sampling-597260.dat'...
[rocprof-sys][597260][0][rocprofsys_finalize] Finalized: 0.045335 sec wall_clock,    2.304 MB peak_rss,    2.359 MB page_rss, 0.010000 sec cpu_clock,   22.0 % cpu_util

@dgaliffiAMD dgaliffiAMD added the release promotion Tagged pull request is a candidate for cherry picking to a release branch label Jan 23, 2025
@dgaliffiAMD dgaliffiAMD changed the title Fix rocprof-sys-sample cannot get GPU related tracing [6.3] Fix rocprof-sys-sample cannot get GPU related tracing Jan 23, 2025
The RCCL library destructor triggers undefined behaviour by invoking OnLoad function.
Disabling the RCCL backend to prevent this issue.
@adjordje-amd adjordje-amd force-pushed the fix/smi-stats-missing branch from e084087 to 880b6f8 Compare January 24, 2025 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release promotion Tagged pull request is a candidate for cherry picking to a release branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants