How to Set Up NVIDIA CUDA and Container Toolkits on RHEL 10

/ Christopher Paquin

Introduction

In this step-by-steps guide we will replace the out of the box nouveau drivers on RHEL 10.1 with NVIDIA Drivers. We will also install the the NVIDIA CUDA Toolkit and the Nvidia Container Toolkit.

What’s New

For RHEL 10, Red Hat has simplified the driver installation process through the Extensions channel. You might not need nvidia-detect if you use this simplified method, as the RHEL 10 can now handle the detection and installation for you. 

NVIDIA Driver install via built-in RHEL drivers Command

Enable the required repos

# sudo subscription-manager repos --enable=rhel-10-for-x86_64-appstream-rpms
# sudo subscription-manager repos --enable=rhel-10-for-x86_64-baseos-rpms
# sudo subscription-manager repos --enable=codeready-builder-for-rhel-10-x86_64-rpms

Install the rhel-drivers package:
This package provides the simplified installation utility:

# sudo dnf install rhel-drivers

Install NVIDIA drivers:
Use the rhel-drivers command to automatically install the correct NVIDIA kernel and user-mode drivers:

# sudo rhel-drivers install nvidia

Reboot

Now verify drivers with nvidia-smi command. Below we can see that the driver has loaded properly and my GPU is visible. 

nvidia-smi
Wed Mar 11 14:39:26 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:04:00.0 Off |                    0 |
| N/A   34C    P8              9W /   70W |       0MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Note that in the output above you can see a mention of CUDA version. This does not mean that CUDA is installed, rather it shows you the maximum CUDA version supported by the currently installed driver. 

NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0

Installing the NVIDIA Cuda Toolkit

In order to install the CUDA toolkit you must first add the appropriate NVIDIA repo for RHEL 10

dnf config-manager --add-repo   https://developer.download.nvidia.com/compute/cuda/repos/rhel10/x86_64/cuda-rhel10.repo

Now we can install the toolkit

sudo dnf install -y cuda-toolkit

This installs:

  • nvcc
  • cuBLAS
  • cuDNN libraries
  • NCCL
  • profiling tools
  • headers and dev libraries

Installing the NVIDIA Container Toolkit

Again we need to add the appropriate repo.

sudo dnf config-manager --add-repo \
https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo

Then install the toolkit.

sudo dnf install -y nvidia-container-toolkit

Now we need to configure the nvidia container toolkit to use podman, since that is default in RHEL.

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

This will create “/etc/cdi/nvidia.yaml”

We can now verify that the toolkit is properly configured to use our local GPU.

nvidia-ctk cdi list
INFO[0000] Found 3 CDI devices                          
nvidia.com/gpu=0
nvidia.com/gpu=GPU-836394e6-a996-65fe-346c-dff40777b64b
nvidia.com/gpu=all

Now we want to verify our entire stack (Driver, CUDA, Container Toolkit, and Podman) and run nvidia-smi in a container

podman run --rm --device nvidia.com/gpu=all \
nvidia/cuda:12.4.1-base-ubi9 nvidia-smi

We should see podman pull the container, launch the container, and run the nvidia-smi command via the container, the output should reflect the presence of out GPU.

podman run --rm --device nvidia.com/gpu=all \
nvidia/cuda:12.4.1-base-ubi9 nvidia-smi
✔ docker.io/nvidia/cuda:12.4.1-base-ubi9
Trying to pull docker.io/nvidia/cuda:12.4.1-base-ubi9...
Getting image source signatures
Copying blob 9d63f91420d1 done   | 
Copying blob 1153e061da4e done   | 
Copying blob 179428f5acc5 done   | 
Copying blob 0c5b8a057cc7 done   | 
Copying blob 39150e63d9d9 done   | 
Copying blob 8f6600363965 done   | 
Copying config 3c259000df done   | 
Writing manifest to image destination
Wed Mar 11 18:49:36 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:04:00.0 Off |                    0 |
| N/A   34C    P8              9W /   70W |       0MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Additionally we can verify cuda was installed correctly via the command below.

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Nov__7_07:23:37_PM_PST_2025
Cuda compilation tools, release 13.1, V13.1.80
Build cuda_13.1.r13.1/compiler.36836380_0

Note if the command above fails, you may need to setup $PATH for your user and/or root. So run the commands below to update $PATH for all users.

echo 'export PATH=/usr/local/cuda/bin:$PATH' | sudo tee /etc/profile.d/cuda.sh
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' | sudo tee -a /etc/profile.d/cuda.sh

Then source the profile.

source /etc/profile.d/cuda.sh

Leave a Reply