how can I solve the problem：Test NCCL failure common.cu:1012 'internal error - please report this issue to the NCCL developers / ' .. dell03 pid 543317: Test failure common.cu:891

[mstJuly]

No description provided.

[sjeaugey]

If you are running with NCCL 2.18, you can ignore that message, and simply run again setting NCCL_DEBUG=WARN to figure out why NCCL failed.

If you still see that with recent NCCL versions, please provide the log with NCCL_DEBUG=INFO.

[mstJuly]

If you are running with NCCL 2.18, you can ignore that message, and simply run again setting NCCL_DEBUG=WARN to figure out why NCCL failed.如果您运行的是 NCCL 2.18，则可以忽略该消息，只需设置 NCCL_DEBUG=WARN 再次运行即可找出 NCCL 失败的原因。

If you still see that with recent NCCL versions, please provide the log with NCCL_DEBUG=INFO.如果您在最新的 NCCL 版本中仍然看到该问题，请提供包含 NCCL_DEBUG=INFO 的日志。

Thanks, I solved the problem, I changed the version to 2.19.3 and I was able to test successfully!