View a markdown version of this page

Get started with EFA and NCCL for ML workloads on Amazon EC2 - Amazon Elastic Compute Cloud
Step 1: Prepare an EFA-enabled security groupStep 2: Launch a temporary instanceStep 3: Install NVIDIA GPU drivers, NVIDIA CUDA Toolkit, and cuDNNStep 4: Install GDRCopyStep 5: Install the EFA softwareStep 6: Install NCCLStep 7: Install the NCCL testsStep 8: Test your EFA and NCCL configurationStep 9: Install your machine learning applicationsStep 10: Create an EFA and NCCL-enabled AMIStep 11: Terminate the temporary instanceStep 12: Launch EFA and NCCL-enabled instances into a cluster placement groupStep 13: Enable passwordless SSH

Get started with EFA and NCCL for ML workloads on Amazon EC2

The NVIDIA Collective Communications Library (NCCL) is a library of standard collective communication routines for multiple GPUs across a single node or multiple nodes. You can use NCCL together with EFA, Libfabric, and MPI to support various machine learning workloads. For more information, see the NCCL website.

Requirements

  • Supported instance types include EFA-supported P series and G series instance types. For more information, see Amazon EC2 accelerated computing instances.

  • Supported base AMIs: Amazon Linux 2023, Ubuntu 26.04, Ubuntu 24.04, Ubuntu 22.04, Debian 12 and RHEL 10.

  • EFA supports only NCCL 2.4.2 and later.

For more information about running machine learning workloads with EFA and NCCL using an AWS Deep Learning AMIs, see Using EFA on the DLAMI in the AWS Deep Learning AMIs Developer Guide.

Step 1: Prepare an EFA-enabled security group

An EFA requires a security group that allows all inbound and outbound traffic to and from the security group itself. The following procedure creates a security group that allows all inbound and outbound traffic to and from itself, and that allows inbound SSH traffic from any IPv4 address for SSH connectivity.

Important

  • This security group is intended for testing purposes only. For your production environments, we recommend that you create an inbound SSH rule that allows traffic only from the IP address from which you are connecting, such as the IP address of your computer, or a range of IP addresses in your local network.

  • The self-referencing inbound and outbound rules (allowing all traffic to and from the security group itself) are mandatory for EFA to function. Without these rules, EFA traffic between instances will be blocked and NCCL communication will fail.

For other scenarios, see Security group rules for different use cases.

To create an EFA-enabled security group

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Security Groups and then choose Create security group.

  3. In the Create security group window, do the following:

    1. For Security group name, enter a descriptive name for the security group, such as EFA-enabled security group.

    2. (Optional) For Description, enter a brief description of the security group.

    3. For VPC, select the VPC into which you intend to launch your EFA-enabled instances.

    4. Choose Create security group.

  4. Select the security group that you created, and on the Details tab, copy the Security group ID.

  5. With the security group still selected, choose Actions, Edit inbound rules, and then do the following:

    1. Choose Add rule.

    2. For Type, choose All traffic.

    3. For Source type, choose Custom and paste the security group ID that you copied into the field.

    4. Choose Add rule.

    5. For Type, choose SSH.

    6. For Source type, choose Anywhere-IPv4.

    7. Choose Save rules.

  6. With the security group still selected, choose Actions, Edit outbound rules, and then do the following:

    1. Choose Add rule.

    2. For Type, choose All traffic.

    3. For Destination type, choose Custom and paste the security group ID that you copied into the field.

    4. Choose Save rules.

Step 2: Launch a temporary instance

Launch a temporary instance that you can use to install and configure the EFA software components. You use this instance to create an EFA-enabled AMI from which you can launch your EFA-enabled instances.

To launch a temporary instance

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances, and then choose Launch Instances to open the new launch instance wizard.

  3. (Optional) In the Name and tags section, provide a name for the instance, such as EFA-instance. The name is assigned to the instance as a resource tag (Name=EFA-instance).

  4. In the Application and OS Images section, select an AMI for one of the supported operating systems.

  5. In the Instance type section, select a supported instance type.

  6. In the Key pair section, select the key pair to use for the instance.

  7. In the Network settings section, choose Edit, and then do the following:

    1. For Subnet, choose the subnet in which to launch the instance.

      Important

      You must select a subnet. If you do not select a subnet, you can't enable the instance for EFA.

    2. For Firewall (security groups), choose Select existing security group, and then select the security group that you created in the previous step.

    3. Expand the Advanced network configuration section.

      For Network interface 1, select Network card index = 0, Device index = 0, and Interface type = EFA with ENA.

      (Optional) If you are using a multi-card instance type, for each additional network interface required, choose Add network interface, for Network card index select the next unused index, and then select Device index = 1 and Interface type = EFA with ENA or EFA-only.

  8. In the Storage section, configure the volumes as needed.

    Note

    You must provision an additional 10 to 20 GiB of storage for the NVIDIA CUDA Toolkit. If you do not provision enough storage, you will receive an insufficient disk space error when you attempt to install the NVIDIA drivers and CUDA toolkit.

  9. In the Summary panel on the right, choose Launch instance.

Step 3: Install NVIDIA GPU drivers, NVIDIA CUDA Toolkit, and cuDNN

Amazon Linux 2023
To install the NVIDIA GPU drivers, NVIDIA CUDA Toolkit, and cuDNN

  1. To ensure that all of your software packages are up to date, perform a quick software update on your instance.

    $ sudo dnf upgrade -y && sudo reboot

    After the instance has rebooted, reconnect to it.

  2. Install the utilities that are needed to install the NVIDIA GPU drivers and the NVIDIA CUDA Toolkit.

    $ sudo dnf groupinstall 'Development Tools' -y && sudo dnf install -y dkms kernel-devel-$(uname -r) kernel-headers-$(uname -r)

  3. Disable the nouveau open source drivers.

    1. Install the required utilities and the kernel headers package for the version of the kernel that you are currently running.

      $ sudo yum install -y wget kernel-devel-$(uname -r) kernel-headers-$(uname -r)

    2. Add nouveau to the /etc/modprobe.d/blacklist.conf deny list file.

      $ cat << EOF | sudo tee --append /etc/modprobe.d/blacklist.conf
blacklist vga16fb
blacklist nouveau
blacklist rivafb
blacklist nvidiafb
blacklist rivatv
EOF

    3. Append GRUB_CMDLINE_LINUX="rdblacklist=nouveau" to the grub file and rebuild the GRUB configuration.

      $ echo 'GRUB_CMDLINE_LINUX="rdblacklist=nouveau"' | sudo tee -a /etc/default/grub \
&& sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  4. Reboot the instance and reconnect to it.

  5. Add the CUDA network repository.

    $ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

  6. Download and install the NVIDIA GPU driver.

    $ wget https://us.download.nvidia.com/tesla/580.167.08/NVIDIA-Linux-x86_64-580.167.08.run \
&& sudo sh NVIDIA-Linux-x86_64-580.167.08.run -m kernel-open --no-drm --disable-nouveau --dkms --silent

  7. Install the NVIDIA CUDA Toolkit and cuDNN.

    $ sudo dnf install -y cuda-toolkit-13-0 libcudnn9-cuda-13 libcudnn9-devel-cuda-13

  8. Reboot the instance and reconnect to it.

  9. (Instances with NVSwitch, such as P-series multi-GPU instances) Install and start the NVIDIA Fabric Manager. G-series instances do not use NVSwitch and do not require Fabric Manager.

    $ sudo dnf install -y https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/nvidia-fabricmanager-580.167.08-1.el8.x86_64.rpm \
&& sudo systemctl enable nvidia-fabricmanager && sudo systemctl start nvidia-fabricmanager

  10. Ensure that the CUDA paths are set each time that the instance starts.

    • For bash shells, add the following statements to /home/username/.bashrc and /home/username/.bash_profile. 

      export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

    • For tcsh shells, add the following statements to /home/username/.cshrc.

      setenv PATH=/usr/local/cuda/bin:$PATH
setenv LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

  11. To confirm that the NVIDIA GPU drivers are functional, run the following command.

    $ nvidia-smi -q | head

    The command should return information about the NVIDIA GPUs, NVIDIA GPU drivers, and NVIDIA CUDA Toolkit.

Ubuntu 26.04, Ubuntu 24.04, and Ubuntu 22.04
To install the NVIDIA GPU drivers, NVIDIA CUDA Toolkit, and cuDNN

  1. To ensure that all of your software packages are up to date, perform a quick software update on your instance.

    $ sudo apt-get update && sudo apt-get upgrade -y

  2. Install the utilities that are needed to install the NVIDIA GPU drivers and the NVIDIA CUDA Toolkit.

    $ sudo apt-get update && sudo apt-get install build-essential -y

  3. To use the NVIDIA GPU driver, you must first disable the nouveau open source drivers.

    1. Install the required utilities and the kernel headers package for the version of the kernel that you are currently running.

      $ sudo apt-get install -y gcc make linux-headers-$(uname -r)

    2. Add nouveau to the /etc/modprobe.d/blacklist.conf deny list file.

      $ cat << EOF | sudo tee --append /etc/modprobe.d/blacklist.conf
blacklist vga16fb
blacklist nouveau
blacklist rivafb
blacklist nvidiafb
blacklist rivatv
EOF

    3. Open /etc/default/grub using your preferred text editor and add the following. 

      GRUB_CMDLINE_LINUX="rdblacklist=nouveau"

    4. Rebuild the GRUB configuration.

      $ sudo update-grub

  4. Reboot the instance and reconnect to it.

  5. Add the CUDA repository and install the NVIDIA GPU drivers, NVIDIA CUDA Toolkit, and cuDNN.

    • Ubuntu 26.04