🐛 Describe the bug
hta/analyzers/straggler.py identifies communication kernels by symbol name with:
comm_op_ids = [sym_map[s] for s in sym_table if s.startswith("ncclKernel")]
Modern NCCL renamed its device kernels from ncclKernel_* to ncclDevKernel_* (e.g. ncclDevKernel_AllReduce_Sum_f32_RING_LL, ncclDevKernel_Generic), so on traces collected with any recent NCCL build this matches zero symbols and comm_op_ids comes back empty
Steps to reproduce
- Collect a PyTorch profiler trace from a multi-rank distributed job using latest NCCL (any recent PyTorch/CUDA container qualifies). GPU communication kernels in the trace are named
ncclDevKernel_*.
- Run straggler analysis:
from hta.trace_analysis import TraceAnalysis
analyzer = TraceAnalysis(trace_dir="<path_to_traces>")
analyzer.get_potential_stragglers()
- No communication kernels are found because the symbol filter only matches the legacy
ncclKernel prefix.
Expected behavior
Straggler analysis should recognize NCCL communication kernels regardless of the NCCL version's naming scheme, i.e. match both ncclKernel_* and ncclDevKernel_* , and return straggler candidates for traces from modern NCCL builds.
Environment
OS (Linux, Mac, Windows): Linux
OS version: Ubuntu (kernel 6.17)
Python version: 3.12.3
HTA version: 0.6.1 (main)
How did you installed HTA (pip, source): source
(The bug is environment-independent)
Additional Info
No response
🐛 Describe the bug
hta/analyzers/straggler.pyidentifies communication kernels by symbol name with:Modern NCCL renamed its device kernels from
ncclKernel_*toncclDevKernel_*(e.g.ncclDevKernel_AllReduce_Sum_f32_RING_LL,ncclDevKernel_Generic), so on traces collected with any recent NCCL build this matches zero symbols andcomm_op_idscomes back emptySteps to reproduce
ncclDevKernel_*.ncclKernelprefix.Expected behavior
Straggler analysis should recognize NCCL communication kernels regardless of the NCCL version's naming scheme, i.e. match both
ncclKernel_*andncclDevKernel_*, and return straggler candidates for traces from modern NCCL builds.Environment
OS (Linux, Mac, Windows): Linux
OS version: Ubuntu (kernel 6.17)
Python version: 3.12.3
HTA version: 0.6.1 (main)
How did you installed HTA (pip, source): source
(The bug is environment-independent)
Additional Info
No response