What is your question?
I'm using HTA to analyze my tracem which is got from the pytorch profiler. However I got NaN for the communication time, and I dive into the code and I found in the communication_analysis.py, the communication kernel in my trace is 0 (which is not the case since I'm doing multi-gpu training). Later I found HTA is parsing the communication kernel using: NCCL_KERNEL_RE = re.compile(r"^nccl.*Kernel") in utils.py. However, I found my trace only have nccl::Allreduce as the name after I parse it. Is there a specific parameter I should set to let the pytorch profiler profile the kernel-level detail, or I made some mistake in writing the training script, so the nccl kernel is never being called?
Code
/home/yu/.local/lib/python3.8/site-packages/hta/analyzers/communication_analysis.py:73: RuntimeWarning: invalid value encountered in scalar divide
return (shifted_overlap["time_y"] - shifted_overlap["time_x"]).sum() / (
What have you tried?
No response
Environment
Python: 3.10.16
Pytorch: 2.7
nccl/2.19.3-1.awsplugin
cuda/11.8.0
What is your question?
I'm using HTA to analyze my tracem which is got from the pytorch profiler. However I got NaN for the communication time, and I dive into the code and I found in the communication_analysis.py, the communication kernel in my trace is 0 (which is not the case since I'm doing multi-gpu training). Later I found HTA is parsing the communication kernel using: NCCL_KERNEL_RE = re.compile(r"^nccl.*Kernel") in utils.py. However, I found my trace only have nccl::Allreduce as the name after I parse it. Is there a specific parameter I should set to let the pytorch profiler profile the kernel-level detail, or I made some mistake in writing the training script, so the nccl kernel is never being called?
Code
/home/yu/.local/lib/python3.8/site-packages/hta/analyzers/communication_analysis.py:73: RuntimeWarning: invalid value encountered in scalar divide
return (shifted_overlap["time_y"] - shifted_overlap["time_x"]).sum() / (
What have you tried?
No response
Environment
Python: 3.10.16
Pytorch: 2.7
nccl/2.19.3-1.awsplugin
cuda/11.8.0