NVIDIA's NCCL: a detailed introduction to NCCL, installation, and usage
Table of contents
1. Use CUDA to compile PyTorch to get built-in NCCL support
T1. Automatically install and configure NCCL
T2. Manually install and configure NCCL
Configure environment variables
(1), Integrate NCCL into the deep learning framework
(2), initialize the NCCL environment
(3), use NCCL communication operation
Introduction to NCCL
NCCL (NVIDIA Collective Communications Library) is a high-performance multi-GPU communication library developed by NVIDIA for fast data transmission and collaborative computing between multiple NVIDIA GPUs. It can provide support for distributed training and data parallel acceleration in the fields of deep learning and high-performance computing.
NCCL implements multi-GPU and multi-node communication primitives optimized for NVIDIA GPUs and networking . NCCL provides routines such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point sending and receiving. The routines are optimized to achieve high bandwidth and low latency on the PCIe and NVLink high-speed interconnects within the nodes and on the NVIDIA Mellanox network between the nodes.
Leading deep learning frameworks such as Caffe2, Chainer, MxNet, PyTorch, and TensorFlow have integrated NCCL into multi-GPU multi-node systems to accelerate deep learning training.
1. Use CUDA to compile PyTorch to get built-in NCCL support
PyTorch needs to be compiled with CUDA for built-in NCCL support. By compiling PyTorch with CUDA, you can get the version with embedded NCCL to support distributed training.
NCCL installation
Currently NCCL does not support installation and use on Windows. NCCL is primarily developed and optimized for the Linux operating system and integrated with deep learning frameworks on Linux. Therefore, if you are doing deep learning development on a Windows system, you may not be able to install and use NCCL directly. However, it is still possible to use GPU and CUDA for deep learning training on a Windows system, it just cannot use the specific optimizations and functions provided by NCCL.
T1. Automatically install and configure NCCL
In some cases, NCCL may have been automatically installed with the NVIDIA GPU driver and integrated with CUDA and PyTorch. Therefore, before installing manually, make sure you do not have an installed version of NCCL on your system.
T2. Manually install and configure NCCL
Note that the steps below are only for manual installation and configuration of NCCL.
Download NCCL
NCCL download address: NVIDIA Collective Communications Library (NCCL) | NVIDIA Developer
Download the NCCL installer package for your operating system and GPU. Make sure to choose the version that is compatible with your system and CUDA version .
NCCL is available for download as part of the NVIDIA HPC SDK and as a standalone package for Ubuntu and Red Hat.
Install NCCL
Unzip the downloaded NCCL installation package, and execute the installation according to the installation guide in it. Typically, an installation step consists of running a specific installation script or executing a predefined installation command.
Configure environment variables
After the installation is complete, the path to NCCL needs to be added to the environment variables of the system so that other applications can find it. The exact steps vary by operating system, but usually involve adding NCCL's library path to the LD_LIBRARY_PATH (Linux) or PATH (Windows) environment variable.
verify installation
After the installation is complete, you can use the following code snippet to verify the installation of NCCL:
python -c "import torch; print(torch.cuda.nccl.version())"or
import torch
print(torch.cuda.nccl.is_available())If the output is True, NCCL has been successfully installed and integrated with PyTorch.
How to use NCCL
1. Basic Usage
(1), Integrate NCCL into the deep learning framework
NCCL is integrated with mainstream deep learning frameworks (such as PyTorch, TensorFlow) to accelerate multi-GPU multi-node training. You need to ensure that the deep learning framework you use has integrated NCCL, and configure it according to the documentation and examples of the corresponding framework.
(2), initialize the NCCL environment
In your deep learning code, you need to initialize the NCCL environment for multi-GPU communication. This usually involves creating NCCL communication groups, setting device identifiers, etc. For the specific initialization process, please refer to the official documentation and sample code of NCCL.
(3), use NCCL communication operation
Once the NCCL environment is initialized, you can use the communication operations provided by NCCL to perform parallel communication tasks such as all-gather, all-reduce, and broadcast. These communication operations will take full advantage of the high bandwidth and low latency between GPUs and networks to improve deep learning training performance on multi-GPU multi-node systems.