Where Did My NCCL Calls Go? A Profiler Comparison

Abstract

Deep learning has become an increasingly important technology, with a profound impact on science and society, where modern models need to use several devices for training and inference. Communication among devices plays a crucial role in the performance of distributed deep learning applications, where NCCL is the most frequently used library for GPU-GPU communications. However, performance analysis tools for NCCL still lack maturity, with no clearly established alternative.In this paper, we compare several profiling tools for NCCL based on the accuracy and completeness of the collected data and the overhead introduced. We also introduce a NCCL profiler tool called ncclsee that overcomes the limitations of existing tools. Our experiments with micro-benchmarking and a real-world application show that existing profilers often miss a significant number of communication events, leading to inaccurate results. In contrast, ncclsee records all NCCL calls in every rank, offering more accurate measurements with negligible overhead.