Skip to content

Straggler analysis silently finding zero communication kernels on traces from modern NCCL builds #360

Description

@benettia

🐛 Describe the bug

hta/analyzers/straggler.py identifies communication kernels by symbol name with:

comm_op_ids = [sym_map[s] for s in sym_table if s.startswith("ncclKernel")]

Modern NCCL renamed its device kernels from ncclKernel_* to ncclDevKernel_* (e.g. ncclDevKernel_AllReduce_Sum_f32_RING_LL, ncclDevKernel_Generic), so on traces collected with any recent NCCL build this matches zero symbols and comm_op_ids comes back empty

Steps to reproduce

  1. Collect a PyTorch profiler trace from a multi-rank distributed job using latest NCCL (any recent PyTorch/CUDA container qualifies). GPU communication kernels in the trace are named ncclDevKernel_*.
  2. Run straggler analysis:
from hta.trace_analysis import TraceAnalysis

analyzer = TraceAnalysis(trace_dir="<path_to_traces>")
analyzer.get_potential_stragglers()
  1. No communication kernels are found because the symbol filter only matches the legacy ncclKernel prefix.

Expected behavior

Straggler analysis should recognize NCCL communication kernels regardless of the NCCL version's naming scheme, i.e. match both ncclKernel_* and ncclDevKernel_* , and return straggler candidates for traces from modern NCCL builds.

Environment

OS (Linux, Mac, Windows): Linux
OS version: Ubuntu (kernel 6.17)
Python version: 3.12.3
HTA version: 0.6.1 (main)
How did you installed HTA (pip, source): source

(The bug is environment-independent)

Additional Info

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions