Category: RHEL9

  • Fixing the OpenSSH Post-Quantum Warning on RHEL 9

    Fixing the OpenSSH Post-Quantum Warning on RHEL 9

    Fired up my RHEL9 server today, something that I have not done in a while due to the savage summer heat in the southern US. All my Dell servers have had the summer off for this reason.

    Upon login via ssh I saw a warning message, which I do not think I have seen before on this machine – all my other Dell servers are RHEL 10, and this one was built with RHEL 9 due to deprecated support for older processors in RHEL 10. Specifically the Intel Xeon E5-2697 (Ivy Bridge) v2.

    Anyway, below is the warning message.

    ** WARNING: connection is not using a post-quantum key exchange algorithm.
    ** This session may be vulnerable to "store now, decrypt later" attacks.
    ** The server may need to be upgraded. See https://openssh.com/pq.html

    This warning means that the SSH client supports post-quantum key exchange, but the RHEL 9 SSH server did not offer one. It does not indicate that the current connection is broken; it warns about possible future decryption of captured traffic.

    Hence the “store now, decrypt later message above.


    SNDL/HNDL

    “Store Now, Decrypt Later” (also called Harvest Now, Decrypt Later) is when an attacker intercepts encrypted traffic, not with the idea of decrypting it now, but rather once they have access to a more powerful quantum computer running advanced algorithms. Today’s standard SSH connections rely on classical asymmetric cryptography (like Elliptic Curve Diffie-Hellman) which quantum computers can theoretically break.

    Meaning, you may not be vulnerable now, but you could be later.

    Most newer OpenSSH clients warn when a connection does not negotiate a post-quantum key-exchange algorithm because long-lived or sensitive data captured today could remain valuable in the future.

    Honestly this is not a big concern in my lab, but the warning will drive me up a wall.


    The RHEL 9 Fix

    The good news is that RHEL 9.1 and later support the following hybrid post-quantum SSH key-exchange method. So we can remediate.

    sntrup761x25519-sha512@openssh.com

    This combines the post-quantum sntrup761 algorithm with the traditional X25519 elliptic-curve key exchange. Because it is hybrid, the connection remains protected as long as either component remains secure.

    Red Hat added support for this algorithm to the RHEL 9 system-wide cryptographic-policy framework, but it must be explicitly enabled through a custom subpolicy.


    Confirm the RHEL Version

    First, lets confirm that the server is running RHEL 9.1 or later:

    cat /etc/redhat-release

    You can also check the installed OpenSSH and crypto-policy package versions:

    rpm -q openssh-server crypto-policies

    Update OpenSSH and the Crypto-Policy Packages

    sudo dnf update openssh-server crypto-policies

    This ensures that the server has the RHEL packages that include support for the SNTRUP hybrid key-exchange algorithm.

    In my case, this step initially revealed a separate Red Hat subscription problem:

    Status code: 403 for https://cdn.redhat.com/...

    A 403 from the Red Hat CDN generally means that the server can reach the repository, but the system is not currently authorized to download its content. This may be caused by an expired developer subscription, stale registration data, an invalid entitlement, or a system identity left over from an earlier installation.

    Useful commands for checking the registration state include:

    sudo subscription-manager identity
    sudo subscription-manager status
    sudo subscription-manager repos --list-enabled

    As I said this machine has been powered off for a while, so lets refresh registration and repository metadata.

    sudo subscription-manager refresh
    sudo dnf clean all
    sudo rm -rf /var/cache/dnf
    sudo dnf makecache

    In my case, the system registration is stale, so I need to unregister and re-register.

    sudo subscription-manager unregister
    sudo subscription-manager clean
    sudo subscription-manager register

    After registration, we ensure that the standard RHEL 9 repositories are enabled:

    sudo subscription-manager repos \
    --enable=rhel-9-for-x86_64-baseos-rpms \
    --enable=rhel-9-for-x86_64-appstream-rpms

    Then rebuild the DNF cache:

    sudo dnf clean all
    sudo rm -rf /var/cache/dnf
    sudo dnf makecache

    Create a Custom Crypto-Policy Module

    Next we create a new policy module named SNTRUP:

    sudo tee /etc/crypto-policies/policies/modules/SNTRUP.pmod >/dev/null <<'EOF'
    key_exchange = +SNTRUP
    EOF

    The module name must use uppercase characters because crypto-policy module filenames are conventionally uppercase and end with .pmod.

    The policy adds the SNTRUP algorithm to the key-exchange methods permitted by the existing system-wide policy.


    Apply the Policy

    Apply the custom module on top of the normal RHEL DEFAULT policy:

    sudo update-crypto-policies --set DEFAULT:SNTRUP

    Confirm that the new policy is active:

    update-crypto-policies --show

    You are looking for expected output as shown below.

    DEFAULT:SNTRUP

    RHEL uses system-wide cryptographic policies to configure applications including OpenSSH, TLS libraries, IPsec, DNSSEC, and Kerberos. Using a crypto-policy module is therefore preferable to manually changing generated OpenSSH backend files.


    Restart SSH

    Next we restart sshd so that it reads the updated cryptographic policy:

    sudo systemctl restart sshd

    And we confirm that the service restarted without issue

    sudo systemctl status sshd --no-pager

    Verify the Server Configuration

    Run the command below to check the current SSH server key-exchange configuration:

    sudo sshd -T | grep -i '^kexalgorithms'

    The output should include:

    sntrup761x25519-sha512@openssh.com

    We can also inspect the generated OpenSSH server crypto-policy backend:

    grep -i kexalgorithms \
    /etc/crypto-policies/back-ends/opensshserver.config

    Files beneath /etc/crypto-policies/back-ends/ are generated by the crypto-policy framework and should not normally be edited directly. Manual changes may be overwritten the next time the system policy is updated.

    Now it’s time to test.

    7. Test a New SSH Connection

    Open a new terminal window, leaving the existing window open just in case we broke something. In the second window, connect with verbose logging:

    ssh -vv user@server

    Look for a line similar to:

    debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com

    The original post-quantum warning should no longer appear.

    A more focused test can be performed by explicitly requesting the algorithm:

    ssh \
    -o KexAlgorithms=sntrup761x25519-sha512@openssh.com \
    user@server

    If this succeeds, both the SSH client and server support the hybrid key exchange.

    We can check whether the local SSH client recognizes the algorithm with:

    ssh -Q kex | grep -i sntrup

    Expected output:

    sntrup761x25519-sha512@openssh.com

    Depending on the client version, another SNTRUP variant may also be listed.


    Troubleshooting

    If the algorithm does not appear in the output from sshd -T, first confirm that the custom policy is active:

    update-crypto-policies --show

    Then confirm that the policy module contains the correct entry:

    cat /etc/crypto-policies/policies/modules/SNTRUP.pmod

    It should contain:

    key_exchange = +SNTRUP

    Reapply the policy and restart SSH:

    sudo update-crypto-policies --set DEFAULT:SNTRUP
    sudo systemctl restart sshd

    Check the SSH service logs for errors:

    sudo journalctl -u sshd -b --no-pager

    You can also validate the SSH configuration before restarting the service:

    sudo sshd -t

    No output means that the configuration passed validation.


    Important FIPS Consideration

    Do not replace an existing FIPS policy with DEFAULT:SNTRUP on a system required to operate in FIPS mode.

    Check the current policy first:

    update-crypto-policies --show

    Also check whether the kernel is operating in FIPS mode:

    cat /proc/sys/crypto/fips_enabled

    A value of 1 means that FIPS mode is active.

    Algorithms permitted by the normal RHEL DEFAULT policy are not automatically permitted under the RHEL FIPS policy. Red Hat documents that certain Curve25519-based SSH key exchanges are not allowed in FIPS mode, so the SNTRUP hybrid method should not be enabled without validating the applicable compliance requirements.


    Rolling Back the Change

    To remove the SNTRUP subpolicy and return to the standard RHEL default policy:

    sudo update-crypto-policies --set DEFAULT
    sudo systemctl restart sshd

    Confirm the result:

    update-crypto-policies --show

    Expected output:

    DEFAULT

    The custom policy file can then be removed if it is no longer needed:

    sudo rm -f /etc/crypto-policies/policies/modules/SNTRUP.pmod

    Final Result

    After enabling the RHEL crypto-policy module, restarting sshd, and creating a new connection, the server and client should negotiate:

    sntrup761x25519-sha512@openssh.com

    The SSH session still uses familiar symmetric encryption and authentication mechanisms. The portion being changed is the initial key exchange used to establish the session keys.

    This does not make the entire SSH protocol “post-quantum.” It adds a hybrid post-quantum key-exchange mechanism designed to protect the session-establishment process against both traditional attacks and the future possibility of cryptographically capable quantum computers.

    This server that has a few years of useful service (specifically for my Infiniband testing), but cannot move to RHEL 10. So enabling the supported RHEL 9 hybrid key exchange is “good enough”.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 2: Infiniband Setup

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 2: Infiniband Setup
    Black silhouette of a cat with an arched back.

    Part 1: Of this Project Log can be found here

    Now that the 3x Mellanox MCX455A-ECAT ConnectX-4 Adapters have arrived, its time to install them into their respective servers (columbia.lab, prometheus.lab, and viper.lab)


    Verify Mellanox CX4s are Detected

    Once installed, log into the IDRAC of each host and verify that the CX-4 appears in system inventory. Sample output below from one of the hosts.

    Note that you may need to boot the system for the CX-4 to appear in the IDRAC inventory (as Collect System Inventory on Restart” (CSIOR) will run when starting up)

    InfiniBand.Slot.1-1 - PCI Device
    BusNumber 129
    DataBusWidth 16x or x16
    Description ConnectX-4 VPI IB EDR/100 GbE Single Port QSFP28 Adapter
    Device Type PCIDevice
    DeviceDescription InfiniBand.Slot.1-1
    DeviceNumber 0
    FQDD InfiniBand.Slot.1-1
    FunctionNumber 0
    InstanceID InfiniBand.Slot.1-1
    LastSystemInventoryTime 2026-03-14T22:33:28
    LastUpdateTime 2026-03-15T03:33:07
    Manufacturer Mellanox Technologies
    PCIDeviceID 1013
    PCISubDeviceID 0033
    PCISubVendorID 15B3
    PCIVendorID 15B3
    SlotLength Long Length
    SlotType PCI Express Gen 3

    Once each machine has booted to the running OS, you can confirm that the RHEL properly detects the CX-4 with lspci

    lspci | grep -i mel
    44:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

    Verify Numa Topology Via nvidia-smi

    Ideally, for best performance, your GPUS and InfiniBand adapters will be NUMA local to each other. If you were deploying a similar setup in a production environment, NUMA alignment would be critical.

    Our lab setup is less than ideal due to the limited number of PCI slots. In many Dell servers, PCIe risers for GPUs have only one PCIe slot. Stick two of these risers in a single server, and you end up with only 3 slots free on riser 1 (half lenght) which is where we had to install our CX-4s.

    All this being said, we “should” be fine for functional testing. Lets review each of our 3 nodes below. Since NVIDIA Drivers are already installed on all three systems, we can run nvidia-smi and confirm that the CX-4 is in the output and review the topology


    Nvidia-smi output on host viper.lab

    This server viper.lab is a Dell R720 running RHEL 9, and has 2x Nvidia Telsa P4 GPUs installed along with the CX-4.

    nvidia-smi topo -m
    GPU0 GPU1 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
    GPU0 X PHB SYS 0,2,4,6,8,10 0 N/A
    GPU1 PHB X SYS 0,2,4,6,8,10 0 N/A
    NIC0 SYS SYS X

    In the output above. Both GPUs are local to NUMA node 0 and connected to each other through a PCIe host bridge (PHB), while the ConnectX NIC (mlx5_0) is topologically remote from both GPUs (SYS), making the setup workable but not ideal for GPUDirect RDMA performance. This should be ok for our lab as we are performing functional tests, and performance is secondary. Time will tell.

    Nvidia SMI output on host columbia.lab

    Columbia.lab is a Dell R730, with 1x Nvidia Telsa T4 installed along with one CX-4 (possible to move to another slot, if we had the full length bracket for the CX-4, or had half-length brackets for our NICs on Riser 2.

    nvidia-smi topo -m
    GPU0 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
    GPU0 X SYS 0,2,4,6,8,10 0 N/A
    NIC0 SYS X

    In the output above, we can see that this system has one GPU (T4) on NUMA node 0 and and CX-4, but the GPU-to-NIC path is SYS, indicating a topologically distant connection that is usable but sub-optimal for GPUDirect RDMA performance. Again, may be fine for functional testing.

    Nvidia SMI output on host prometheus.lab

    This Dell R730 has 2x NVIDIA Tesla T4s installed, as well as the recently installed CX-4

    nvidia-smi topo -m
    GPU0 GPU1 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
    GPU0 X PHB SYS 0,2,4,6,8,10 0 N/A
    GPU1 PHB X SYS 0,2,4,6,8,10 0 N/A
    NIC0 SYS SYS X

    This output shows two GPUs on the same NUMA node with a moderate GPU-to-GPU path (PHB) and relatively distant NIC connectivity (SYS), which is acceptable for many workloads but not ideal for high-performance GPU-to-NIC or GPUDirect-style traffic.


    Verify Drivers Loaded Properly

    In the first step we saw the CX-4 in the IDRAC, and in the output of lspci. We will now check that the driver has loaded properly. Rinse and repeat on each host.

    lsmod | egrep 'mlx5_core|mlx5_ib|ib_core'
    mlx5_ib 561152 0
    macsec 73728 1 mlx5_ib
    mlx5_core 3153920 2 mlx5_fwctl,mlx5_ib
    mlxfw 49152 1 mlx5_core
    psample 20480 1 mlx5_core
    tls 159744 2 bonding,mlx5_core
    pci_hyperv_intf 12288 1 mlx5_core
    ib_uverbs 217088 2 rdma_ucm,mlx5_ib
    ib_core 573440 12 rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm

    Lets review the imporant/relevant output below

    mlx5_ib 561152 0
    mlx5_core 3153920 2 mlx5_fwctl,mlx5_ib
    ib_uverbs 217088 2 rdma_ucm,mlx5_ib
    ib_core 573440 12 ...
    • mlx5_core – main low-level kernel driver for Mellanox/NVIDIA ConnectX-4/5-class adapters.
      • the kernel sees the adapter family and has loaded the base driver
      • this is required for the card to function at all
      • other Mellanox modules are depending on it
    • mlx5_ib – This is the InfiniBand/RDMA driver layer for mlx5 devices.
      • the adapter is not just using the generic Ethernet driver path
      • the system has the RDMA / InfiniBand-capable driver loaded
      • the kernel is prepared to expose the card as an IB/RDMA device
    • ib_core – the core InfiniBand subsystem in the kernel.
      • the Linux IB stack is loaded
      • multiple RDMA/IB-related modules are attached to it
      • the host is set up for InfiniBand/RDMA functionality, not just plain NIC support
    • ib_uverbs – This is the userspace verbs interface.
      • userspace RDMA tools and libraries should be able to talk to the device
      • commands like ibv_devinfo, ibstat, and RDMA applications have the proper kernel interface available

    Show Devices and Port State

    First we need to install some prerequesits

    sudo dnf install rdma-core infiniband-diags libibverbs-utils -y

    ibv_devices

    We can show InfiniBand devices with ibv_devices, which shows local RDMA/InfiniBand devices that the OS can see on that host. It does not enumerate remote hosts, switches, or the rest of the IB fabric.

    We will run this command on each host and capture the output.

    On columbia.lab

    [root@columbia ~]# ibv_devices
    device node GUID
    ------ ----------------
    mlx5_0 248a070300ac5414

    On viper.lab

    root@viper:~# ibv_devices
    device node GUID
    ------ ----------------
    mlx5_0 248a070300ac5f6c

    On prometheus.lab

    [root@prometheus ~]$ ibv_devices
    device node GUID
    ------ ----------------
    mlx5_0 248a070300ac5610
    [root@prometheus ~]$

    ibstat

    Now that we have confirmed all devices are present and accounted for lets check for links. In the output below you can see that we have link “Physical state: LinkUp“, but since we have not configured subnet manager on any of our nodes, the logical fabric is “State: Initializing“.

    ibstat
    CA 'mlx5_0'
    CA type: MT4115
    Number of ports: 1
    Firmware version: 12.28.4512
    Hardware version: 0
    Node GUID: 0x248a070300ac5610
    System image GUID: 0x248a070300ac5610
    Port 1:
    State: Initializing
    Physical state: LinkUp
    Rate: 40
    Base lid: 65535
    LMC: 0
    SM lid: 0
    Capability mask: 0x2659e848
    Port GUID: 0x248a070300ac5610
    Link layer: InfiniBand

    Run ibstat on any of your remaining nodes. Ensure that you see “Physical state: LinkUp”. You may also want to make notes of “Firmware version: 12.28.4512“. We have the same firmware on all three CX-4s.


    ibv_definfo

    We can also run ibv_devinfo, which gives a detailed view of the local RDMA / InfiniBand device and its ports. It is more detailed than ibv_devices and overlaps somewhat with ibstat, but from the verbs / RDMA stack perspective.

    Example output below:

     ibv_devinfo
    hca_id:	mlx5_0
    	transport:			InfiniBand (0)
    	fw_ver:				12.28.4512
    	node_guid:			248a:0703:00ac:5414
    	sys_image_guid:			248a:0703:00ac:5414
    	vendor_id:			0x02c9
    	vendor_part_id:			4115
    	hw_ver:				0x0
    	board_id:			DEL2180110032
    	phys_port_cnt:			1
    		port:	1
    			state:			PORT_INIT (2)
    			max_mtu:		4096 (5)
    			active_mtu:		4096 (5)
    			sm_lid:			0
    			port_lid:		65535
    			port_lmc:		0x00
    			link_layer:		InfiniBand
    
    

    This output shows us the following…

    • Local RDMA devices
      • Example: mlx5_0, mlx5_1
    • Port state
      • Example: PORT_ACTIVE, PORT_DOWN, PORT_INIT
    • Physical link state
      • Example: LINK_UP, POLLING, DISABLED
    • Negotiated link details
      • Speed and link width
    • Fabric info
      • Local LID and SM LID
    • Device identifiers
      • Node GUID, port GUID, system image GUID
    • Transport / firmware details
      • Transport type and device-specific details
    • RDMA capabilities
      • Limits such as QPs, CQs, MR size, atomic support, GID table size

    rdma link

    The current output of “rdma link” shows use that our InfiniBand ports are connected but as we know the fabric is not initialized.

    rdma link
    link mlx5_0/1 subnet_prefix fe80:0000:0000:0000 lid 65535 sm_lid 0 lmc 0 state INIT physical_state LINK_UP

    Specifically the output shows us the following.

    • mlx5_0/1
      • Device mlx5_0, port 1
    • subnet_prefix fe80:0000:0000:0000
      • Normal default InfiniBand subnet prefix
    • lid 65535
      • The port does not have a valid assigned LID yet
    • sm_lid 0
      • No subnet manager is detected
    • lmc 0
      • LID mask control is 0; not important here
    • state INIT
      • The port is not fully active yet
    • physical_state LINK_UP
      • The physical link is up and the cable/port side is working

    Setup Subnet Manager on one Host

    For our lab, we are going to only setup subnet manager on one host. Pick your always-on host. Multiple instances of subnet manager can be used, but again, not needed for our current objective.

    On the selected host run the following to install required packages.

     dnf install -y rdma-core opensm infiniband-diags

    Next, start and enable the service

    sudo systemctl enable --now opensm

    Then check to ensure that the service started without error.

    sudo systemctl status opensm --no-pager
    journalctl -u opensm -b --no-pager

    Now we can re-check the fabric on each host. Now we see the Fabric status is “State: Active”

     ibstat
    CA 'mlx5_0'
    	CA type: MT4115
    	Number of ports: 1
    	Firmware version: 12.28.4512
    	Hardware version: 0
    	Node GUID: 0x248a070300ac5f6c
    	System image GUID: 0x248a070300ac5f6c
    	Port 1:
    		State: Active
    		Physical state: LinkUp
    		Rate: 40
    		Base lid: 4
    		LMC: 0
    		SM lid: 1
    		Capability mask: 0x2659e848
    		Port GUID: 0x248a070300ac5f6c
    		Link layer: InfiniBand
    root@viper:~# 
    
    

    rdma link shows similar output.

    rdma link
    link mlx5_0/1 subnet_prefix fe80:0000:0000:0000 lid 4 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP

    Confirm Infiniband Fabric Topology

    You can run the following commands to confirm that your fabric is up and running

    Run the “ibnetdiscover” to see host adapters, links, GUIDs, port relationships, and switches. Example output below

    ibnetdiscover
    #
    # Topology file: generated on Sat Mar 14 21:15:43 2026
    #
    # Initiated from node 248a070300ac5f6c port 248a070300ac5f6c
    vendid=0x2c9
    devid=0xbd36
    sysimgguid=0x2c902004cf11b
    switchguid=0x2c902004cf118(2c902004cf118)
    Switch 8 "S-0002c902004cf118" # "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0
    [1] "H-248a070300ac5f6c"[1](248a070300ac5f6c) # "viper mlx5_0" lid 4 4xQDR
    [2] "H-248a070300ac5414"[1](248a070300ac5414) # "columbia mlx5_0" lid 1 4xQDR
    [3] "H-248a070300ac5610"[1](248a070300ac5610) # "prometheus mlx5_0" lid 2 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5414
    caguid=0x248a070300ac5414
    Ca 1 "H-248a070300ac5414" # "columbia mlx5_0"
    [1](248a070300ac5414) "S-0002c902004cf118"[2] # lid 1 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5610
    caguid=0x248a070300ac5610
    Ca 1 "H-248a070300ac5610" # "prometheus mlx5_0"
    [1](248a070300ac5610) "S-0002c902004cf118"[3] # lid 2 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5f6c
    caguid=0x248a070300ac5f6c
    Ca 1 "H-248a070300ac5f6c" # "viper mlx5_0"
    [1](248a070300ac5f6c) "S-0002c902004cf118"[1] # lid 4 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR

    In the output above we can see the following

    • One Mellanox InfiniScale-IV switch is present in the fabric
      • Switch GUID: 0x2c902004cf118
      • Switch LID: 3
      • Model family shown as Infiniscale-IV Mellanox Technologies
      • It is an 8-port switch
    • Three hosts are connected to the switch
      • columbia on switch port 2, LID 1
      • prometheus on switch port 3, LID 2
      • viper on switch port 1, LID 4
    • All three hosts are being seen as CA / HCA nodes
      • columbia mlx5_0
      • prometheus mlx5_0
      • viper mlx5_0
    • All discovered links are running at:
      • 4xQDR
      • That means a 4-lane QDR InfiniBand link, which aligns with a 40 Gb/s class IB link

    This output confirms that OpenSM is working properly, that the switch is visible, and all three nodes are connected to the fabric.

    Run “ibnodes” which provides a similar output to ibdiscover, albeit a bit less verbose.

    ibnodes
    Ca : 0x248a070300ac5610 ports 1 "prometheus mlx5_0"
    Ca : 0x248a070300ac5414 ports 1 "columbia mlx5_0"
    Ca : 0x248a070300ac5f6c ports 1 "viper mlx5_0"
    Switch : 0x0002c902004cf118 ports 8 "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0

    Run “ibswitches” to see switches only.

     ibswitches
    Switch	: 0x0002c902004cf118 ports 8 "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0
    
    

    iblinkinfo will show you InfiniBand topology info

     iblinkinfo
    CA: viper mlx5_0:
          0x248a070300ac5f6c      4    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    1[  ] "Infiniscale-IV Mellanox Technologies" ( )
    CA: columbia mlx5_0:
          0x248a070300ac5414      1    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    2[  ] "Infiniscale-IV Mellanox Technologies" ( )
    Switch: 0x0002c902004cf118 Infiniscale-IV Mellanox Technologies:
               3    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       4    1[  ] "viper mlx5_0" ( )
               3    2[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       1    1[  ] "columbia mlx5_0" ( )
               3    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    1[  ] "prometheus mlx5_0" ( )
               3    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
    CA: prometheus mlx5_0:
          0x248a070300ac5610      2    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    3[  ] "Infiniscale-IV Mellanox Technologies" ( )
    
    

    In the output above, we see …

    • 4x 10Gbps per lane
    • InfiniBand speed class is QDR
    • aggregate raw signaling rate is about 40 Gb/s
    • Down/Polling – unused/not-connected switch ports

    And finally, “sminfo” will show you info on subnet manager.

    Specifically (below) we see that subnet manager is reachable on LID1, with a GUID of 0x248a070300ac5414 (which belongs to columbia mlx5_0). We also see “activity count 446” which shows that subnet manager has processed fabric-management activity 466 times since startup (basically a liveness/activty counter).

    Additionally the output below show us the priority of the subnet manager instance (0 in this case), while state 3 SMINFO_MASTER shows us that this instance of subnet manager is in the master state and is the active controller in our IB fabric (assigning LIDS, managing paths/routing)

    sminfo
    sminfo: sm lid 1 sm guid 0x248a070300ac5414, activity count 446 priority 0 state 3 SMINFO_MASTER

    Configuring IP over InfiniBand

    IP over InfiniBand, or IPoIB, allows an InfiniBand fabric to carry normal IP traffic between hosts. That means systems connected by InfiniBand can use familiar network tools and services such as ping, ssh, scp, NFS, and other TCP/IP-based applications over the IB link instead of only using native RDMA-aware software.

    IPoIB is not required for RDMA itself, and it is also not inherently required for technologies like GPUDirect RDMA. RDMA and GPUDirect RDMA operate through the RDMA/verbs stack and the InfiniBand fabric, not through the IP emulation layer that IPoIB provides. NVIDIA’s current networking/operator docs describe RDMA and GPUDirect RDMA enablement separately from IPoIB, and they also document IPoIB as an optional deployment pattern rather than a prerequisite.

    We use IPoIB when we want the simplicity of standard IP networking on top of the higher-speed, low-latency InfiniBand fabric. In a small lab or cluster, this is useful for private host-to-host traffic, storage traffic, migration traffic, testing, or other east-west communication, while leaving the normal Ethernet interfaces in place for management access, internet access, and general connectivity.

    IPoIB Addresses for our lab

    Our lab uses 10.1.x.x for its existing IP scheme, so to avoid any confusion, we will use 172.16.x.x addresses for our small private subnet on the IB network. Note that we do not need a gateway.

    HOSTIPoIB AddressINTERFACE
    prometheus.lab172.16.50.11/24ibp129s0
    columbia.lab172.16.50.12/24ibp129s0
    viper.lab172.16.50.13/24ibp68s0

    As part of our initial temporary test, we will apply the IPoIB addresses to the indicated interfaces on each host (all as outlined above. Example temporary config will be for one host. However we will run the command (modified) for each host in our cluster.

    sudo ip link set ibp129s0 up
    sudo ip addr add 172.16.50.11/24 dev ibp129s0

    As you go host to host, verify that the address was assigned correctly.

    9: ibp129s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 1000
    link/infiniband 00:00:03:f2:fe:80:00:00:00:00:00:00:24:8a:07:03:00:ac:56:10 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 172.16.50.11/24 scope global ibp129s0
    valid_lft forever preferred_lft forever

    Also verify your routing table.

     netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.10.1       0.0.0.0         UG        0 0          0 bridge0
    10.1.10.0       0.0.0.0         255.255.255.0   U         0 0          0 bridge0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 idrac
    172.16.50.0     0.0.0.0         255.255.255.0   U         0 0          0 ibp129s0
    
    

    Now perform ping tests from each host and ensure that they can hit the remaining hosts in your cluster. For example.

    ping -I ibp129s0 -c 2 172.16.50.12
    ping -I ibp129s0 -c 2 172.16.50.13

    Once you have tested all three hosts, we can move forward with configuring persistent network configs.


    Persistent RHEL 10 / NetworkManager setup

    prometheus.lab

    sudo nmcli connection add type infiniband ifname ibp129s0 con-name ib-ibp129s0
    sudo nmcli connection modify ib-ibp129s0 ipv4.method manual ipv4.addresses 172.16.50.11/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp129s0
    
    
    
    
    

    columbia.lab

    sudo nmcli connection add type infiniband ifname ibp129s0 con-name ib-ibp129s0
    sudo nmcli connection modify ib-ibp129s0 ipv4.method manual ipv4.addresses 172.16.50.12/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp129s0

    viper.lab

    sudo nmcli connection add type infiniband ifname ibp68s0 con-name ib-ibp68s0
    sudo nmcli connection modify ib-ibp68s0 ipv4.method manual ipv4.addresses 172.16.50.13/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp68s0

    Confirm Routing

    Use “ip route” to ensure that we have the proper route in place for our IPoIB network

    ip route
    default via 10.1.10.1 dev bridge0 proto static metric 425
    10.1.10.0/24 dev bridge0 proto kernel scope link src 10.1.10.25 metric 425
    169.254.0.0/16 dev idrac proto kernel scope link src 169.254.0.2 metric 100
    172.16.50.0/24 dev ibp68s0 proto kernel scope link src 172.16.50.13 metric 150

    Also confirm that NetworkManger sees our IB devices correctly (as Infiniband)

    nmcli device status
    DEVICE TYPE STATE CONNECTION
    bridge0 bridge connected bridge0
    idrac ethernet connected idrac
    ibp68s0 infiniband connected ib-ibp68s0
    bond0 bond connected bond0
    enp65s0f0 ethernet connected bond0-port0
    enp65s0f1 ethernet connected bond0-port1
    
    
    
    
    

    IP, IPoIB, and RDMA Usage Matrix

    We can use this simple decision matrix to ensure that we understand when to use traditional IP for host to host communication, vs when to use IPoIB, and when native RDMA/IB

    Use caseEthernetIPoIBNative RDMA / IB
    Host management, SSH, web UI, package installsBest choicePossible, but usually unnecessaryNo
    Internet access / default routeBest choiceNoNo
    General admin traffic between hostsBest choiceGood for isolated lab trafficNo
    Simple host-to-host testing with ping, ssh, scp, rsync over IB fabricNoBest choiceNo
    NFS/SMB using normal IP networking over the IB fabricNoBest choiceNo
    Fast private storage or migration traffic using standard TCP/IP appsNoBest choiceNo
    RDMA-aware apps using verbs/libibverbsNoNoBest choice
    MPI or cluster workloads built for native IB/RDMANoSometimes, if app specifically uses IPBest choice
    GPUDirect RDMA / high-performance GPU-to-network workflowsNoNoBest choice
    Lowest latency / highest efficiency IB data pathNoNoBest choice
    Easiest troubleshooting and least risk of routing mistakesBest choiceGood if kept isolatedMore specialized

    Next Steps.

    So in Part 2 of our project, we focused on getting InfiniBand up and running, as well as IPoIB. We validated connectivity and setup subnet manager and made sure that our fabric was initialized. We leared a number of IB related command and learned how to read their output. Good Stuff.

    In our next post we will start working with the various NVIDIA tools and projects, many of which will rely on our IB network. Additionally we may try to update firmware on our CX-4s and our IB Switch, however I may skip this step or circle back to it later.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 1: Project Overview

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 1: Project Overview

    Introduction

    InfiniBand is a mature interconnect technology known for high bandwidth and low latency. It has long been used in supercomputing and HPC environments, and has also been deployed in certain storage and clustered infrastructure designs as an alternative to Fibre Channel.

    More recently, InfiniBand has seen strong continued adoption in large-scale AI and GPU clusters, where its high bandwidth, ultra-low latency, and support for technologies such as RDMA, GPUDirect RDMA, and NCCL make it well suited for distributed training and other GPU-to-GPU communication workloads.

    This project involves the architecture, deployment, and optimization of a high-speed InfiniBand (IB) fabric to facilitate low-latency, high-throughput communication between 3x dual-homed, GPU enabled, RHEL 9/10.1 servers.

    By integrating Mellanox ConnectX-4 adapters with an InfiniScale IV switch, we will establish a dedicated Remote Direct Memory Access (RDMA) backend separate from the standard management LAN.

    Additionally, we will become more familiar with the setup, configuration, and troubleshooting of InfiniBand networks and adapters, while also exploring the broad set of NVIDIA tools and technologies currently available to support multi-GPU clusters.

    This project also encompasses the installation and configuration of NVIDIA drivers, CUDA, the NVIDIA Container Toolkit, and other supported elements of the NVIDIA software stack needed to support HCP/AI Clusters and environments.

    Primary Objectives Summary

    • Fabric orchestration: Deploy and manage a QDR InfiniBand fabric using OpenSM.
    • RDMA enablement: Configure IPoIB in Connected Mode with a 65,520 MTU and validate RDMA functionality across the fabric.
    • GPU acceleration: Enable and test GPUDirect RDMA with nvidia-peermem for NVIDIA Tesla T4 and P4 GPUs.
    • Platform enablement: Install and configure NVIDIA drivers, CUDA, the NVIDIA Container Toolkit, and other supported NVIDIA software stack components required for GPU-enabled workloads.
    • Operations and telemetry: Develop hands-on familiarity with InfiniBand diagnostics, troubleshooting, and NVIDIA GPU monitoring tools.

    Bill of Materials (BOM)

    Below is the BOM for this project.

    ComponentQtyPARTDESCRIPTION
    Switch1xMellanox InfiniScale IV Is5022 Switch8-port Non-blocking Unmanaged 40Gb/s InfiniBand Switch System
    NICs3xMCX455A-ECAT MELLANOX CONNECTX-4 1 PORT EDR 100GB IB QSFP28 Infiniband/Ethernet Adapter
    GPU (New)2xNVIDIA Tesla P4 8GB GDDR5 (with active cooling mods) 
    GPU (Existing)3xNVIDIA Tesla T4 16GB GDDR5 (with active cooling mods) 
    Cabling3xFS 40Gbps QSFP+ 2M Passive DAC (QSFP-PC02) 3x Host to 1x Switch
    Server1xDell R720 (RHEL 9)2 x Intel Xeon E5-2697 (Ivy Bridge) v2 Twelve-Core Processor 2.7GHz 8.0GT/s 30MB LGA 2011, 128GB
    Server1xDell R730 (RHEL 10.1)2x Intel(R) Xeon(R) CPU E5-2690 (Broadwell-EP) v4 @ 2.40GHz, 256GB
    Server 1xDell R730 (RHEL 10.1)2x Intel® Xeon® Processor E5-2699 v4 (Broadwell-EP) @2.20GHz, 768GB

    Notes On the Bill of Materials (BOM)

    There as been a bit of flux on the exact BOM for this project (and my lab). A Dell R720 was recently added to my lab, replacing a mammoth T620. And while I like the tower form factor due to its ample number of PCI slots and spare SATA power cables, it has to sit on a rack mount shelf, takes up 5RU and is really heavy amd hard to move.

    So the CPUs/Memory from the T620 were migrated to the R720. However RHEL 10 deprecated support for Intel v2 processors, so I had to deploy RHEL 9 instead of RHEL10. RHEL 9 supported the CX-3 with in-band drivers, however support for the CX-3 was dropped from RHEL 10. So I had to switch to CX-4s, which were more costly. However further research found that RDMA was not supported on the CX-3, so I needed to move to the CX4 (or newer) anyways.

    IB Switch – I was able to pick up the unmanaged Mellanox InfiniScale switch on eBay quite cheaply. Being unmanaged it does not run subnet manager, which I will need to run on my primary host. Limited to 40GBe per port.

    IB Adapters – Initially, as stated above, I intended to use Mellanox CX-3 adapters as they were incredibly inexpensive (like $12 USD), however due to CX-3 driver being dropped in RHEL 10, I switched to CX-4s, which were supported out of the box on both RHEL 9 and RHEL 10. These adapters were not as cheap, but still affordable. Additionally, I needed low profile brackets in order to fit into the existing open PCIe slots in my servers. I did not want to use OFED drivers, as I was looking to have the adapters supported out of the box, and was not interested in fiddling with drivers. Additionally, support docs for RDMA mentioned CX-4 or newer were required.

    GPUS – Workstation class GPUs (like my 1x 3070 and my 2x 3060s 12Gb are not supported. I needed Datacenter Class NVIDIA CPUs installed in all 3 Dell Servers. I already owned 3x NVIDIA Tesla T4 (installed in R730s). Picked up 2x additional GPUs (NVIDIA Tesla M4), which are for functional validation only – not purchased for their performance or VRAM. Additional requirement was to stick with low-power GPUs that ran on PCIe power alone and did not require additional power connections.


    Nvidia T4/P4 Comparison and Feature Support

    Our LUT (Lab Under Test) consists of 5 GPUs, deployed across 3 Servers. Details below

    FeatureNVIDIA T4NVIDIA Tesla P4
    ArchitectureTuringPascal
    Release dateSeptember 12, 2018September 12, 2016
    CUDA cores2,5602,560
    Tensor cores320None
    vRAM16 GB GDDR68 GB GDDR5
    Memory bandwidth300–320+ GB/s192 GB/s
    PCIe interfacePCIe Gen3 x16PCIe Gen3
    Form factorLow-profile, single-slot, passiveLow-profile, single-slot, passive
    Max power70 W75 W
    ECC memory supportYesYes
    NVENC / NVDECYesYes
    NVIDIA vGPU supportYesYes
    GPUDirect RDMAYes, conditionally supportedYes, conditionally supported
    RDMA NIC requirementConnectX-4 or laterConnectX-4 or later
    GPUDirect RDMA topology requirementGPU and NIC should share the same upstream PCIe root complex for best support/performanceGPU and NIC should share the same upstream PCIe root complex for best support/performance
    NVLinkNoNo
    MIGNoNo

    Additional GPUs Available

    In addition to the 5x GPUs currently installed in my existing lab servers, I have few additional GPUs that are not currently deployed. I will include them below for reference.

    Generally speaking, these are either “older“, “hungrier” or “hotter” than the GPUs that I already have in service.

    FeatureNVIDIA Tesla K20NVIDIA Tesla P100
    ArchitectureKeplerPascal
    Release dateNovember 2012April 5, 2016
    CUDA cores2,4963,584
    Tensor coresNoneNone
    vRAM5 GB GDDR516 GB HBM2
    Memory bandwidth208 GB/s732 GB/s
    PCIe interfacePCIe Gen2 x16PCIe Gen3 x16
    Form factorFull-height, dual-slotFull-height, dual-slot
    Max power225 W250 W
    ECC memory supportYesYes
    NVENC / NVDECNoNo
    NVIDIA vGPU supportNoYes
    GPUDirect RDMAYes, conditionally supportedYes, conditionally supported
    RDMA NIC requirementConnectX-4 or laterConnectX-4 or later
    GPUDirect RDMA topology requirementGPU and NIC should share the same upstream PCIe root complex for best support/performanceGPU and NIC should share the same upstream PCIe root complex for best support/performance
    NVLinkNoDepends on model; PCIe P100: No, SXM2 P100: Yes
    MIGNoNo

    NVIDIA Tool Test Matrix

    Below is a Matrix of technologies, their supportability in my soon to be deployed stack, and a brief description of each technology.

    Technology / ProductSupportedDescriptionPriorityNOTES
    RDMA (InfiniBand / verbs)SupportedRemote Direct Memory Access. It is a networking technology that lets one computer access or transfer data directly to the memory of another computer without involving the remote CPU or operating system in the data path.
    HighDirect memory access over the network using your ConnectX-4 InfiniBand cards. This is the base networking capability you will use for low-latency, high-throughput node-to-node transfers.

    Native fit for CX-4 + IB switch
    GPUDirect RDMASupported GPUDirect RDMA is an NVIDIA technology that lets a third-party PCIe device directly read from or write to GPU memory without first copying the data through system RAM.HighLets a supported NIC perform RDMA directly to/from GPU memory, bypassing extra CPU copies. NVIDIA documents GPUDirect RDMA for Tesla/Quadro GPUs and requires ConnectX-4 or later NICs, with best results when GPU and NIC share the same upstream PCIe root complex. (NVIDIA Docs)

    One of the most relevant GPU+IB features for this setup
    GPUDirect StorageUnclearDesigned for direct data movement between storage and GPU memory. It is primarily positioned around storage stacks rather than IB switching alone, so whether your exact lab can validate it depends on OS, filesystem, NVMe/storage path, and supported software stack rather than just T4/P4 + CX-4. (NVIDIA Docs)Low NVIDIA technology that allows data to move directly between storage and GPU memory using DMA, instead of first bouncing through CPU memory.

    Best with NVME storage, have only SSDs and HDDs, no NVME support in Dell models under test.
    MIG (Multi-Instance GPU)Not supportedMIG starts with newer architectures and is documented in NVIDIA’s MIG guide as an Ampere-era feature. NVIDIA’s cloud-native docs explicitly note that Tesla T4 does not support MIG. P4 also predates MIG. (NVIDIA Docs)NoneNot supported.

    Multi-Instance GPU. It is an NVIDIA technology that lets a single supported GPU be partitioned into multiple smaller, isolated GPU instances
    Time-slicing / shared GPU schedulingSupportedGood for shared-lab/VM experiments
    Allows multiple workloads share one physical GPU by giving each workload a small turn on the GPU scheduler
    MediumSince T4 does not support MIG, NVIDIA documents time-slicing as a way to share T4 across multiple smaller jobs. This is useful for Kubernetes/OpenShift experiments or general shared-lab validation. P4 can also be shared through virtualization/software scheduling rather than MIG. (NVIDIA Docs)
    NVIDIA vGPUSupportedNVIDIA’s virtual GPU stack allows partitioning/sharing GPUs across VMs for compute, VDI, or graphics use cases. Both T4 and P4 are in NVIDIA’s supported vGPU product documentation. (NVIDIA Docs)HighNVIDIA’s docs describe it as enabling multiple VMs to have simultaneous, direct access to a single physical GPU using NVIDIA drivers inside the guest OS.

    It lets multiple virtual machines share one physical NVIDIA GPU.
    DCGM (Data Center GPU Manager)SupportedNVIDIA’s primary datacenter GPU management and telemetry framework for health, diagnostics, topology, clocks, thermals, ECC, profiling, and integration with cluster tooling. It is explicitly built for Tesla/datacenter GPUs. (NVIDIA Docs)HighDCGM is NVIDIA’s datacenter GPU management and monitoring framework. NVIDIA describes it as a lightweight user-space library/agent for administering NVIDIA datacenter GPUs in clusters and datacenters.
    DCGM ExporterSupportedPrometheus exporter built on top of DCGM that exposes GPU metrics over HTTP for scraping. Good fit for validating telemetry, dashboards, and alerting with your servers. (NVIDIA Docs)HighNVIDIA’s Prometheus exporter for GPU metrics, will utilize existing Grafana instance

    Easy to validate and useful operationally
    NVIDIA MerlinSupportedMerlin is NVIDIA’s recommender-system framework stack for training and especially inference pipelines. T4 is a strong fit; P4 may work for smaller or older inference experiments, but T4 is the more relevant target. Support is practical rather than “card-specific” in docs, since Merlin rides on the CUDA/framework/container stack. (NVIDIA Developer)MediumLowNVIDIA framework for building recommender systems.

    A recommender system is the kind of ML system used for things like:
    product recommendations
    “people also watched”
    next-best content
    ranking search or feed results
    ad / click-through prediction
    TensorRTSupportedTensorRT is NVIDIA’s SDK/runtime for optimizing trained neural-network models for inference on NVIDIA GPUs. It takes a model from frameworks such as TensorFlow, PyTorch, or ONNX and builds an optimized inference engine that can use precision modes such as FP32, FP16, and INT8 where supported.HighNVIDIA’s inference optimizer/runtime. Very relevant for T4 and still usable on P4. T4 benefits significantly from Tensor Cores, so it is the better platform for validation. (NVIDIA Developer)
    CUDASupportedCore GPU programming/runtime stack. Required for most of the technologies you listed and the base layer for custom validation, benchmarks, peer access tests, and GPU-aware applications. (NVIDIA Docs)HighFoundation for most modern NVIDIA workflows
    NVIDIA Container ToolkitSupportedEnables Docker/Podman/Kubernetes containers to access NVIDIA GPUs cleanly. Useful for validating Merlin, TensorRT, PyTorch, RAPIDS, and exporter containers. (NVIDIA Docs)HighFoundation for most modern NVIDIA workflows, integrate with Podman on RHEL
    NVIDIA GPU OperatorSupportedKubernetes/OpenShift operator that automates driver, toolkit, DCGM, exporter, and related GPU software deployment. Best fit if you want to turn a lab into a small cluster validation environment. (NVIDIA Docs)LowGood if you want Kubernetes/OpenShift validation, however no plans to run OCP in near future.
    NVIDIA Fabric ManagerNot supported / not applicableNVIDIA Fabric Manager is software for managing NVSwitch / NVLink GPU fabrics inside supported multi-GPU servers. NVIDIA says it configures the NVSwitch memory fabric to form a single memory fabric among participating GPUs and monitors the NVLinks that support that fabric.NoneFabric Manager is for NVSwitch-based systems, not T4/P4 PCIe accelerator setups. Installed cards do not use NVSwitch. (NVIDIA Docs)

    Not supported here
    NVLinkNot supportedNVLink is NVIDIA’s high-speed direct interconnect for GPUs. It provides a much faster path for GPU-to-GPU communication than ordinary PCIe alone, and in some platforms it is also used for CPU/GPU or switch-based interconnect designs. NVIDIA describes it as a direct GPU-to-GPU interconnect used to scale multi-GPU I/O within a server.NoneNot Supported. Neither T4 nor P4 provides NVLink. Multi-node connectivity would be via InfiniBand/RDMA, not GPU-to-GPU NVLink. (NVIDIA)
    NVIDIA NIM / inference microservicesUnclear / limitedNVIDIA NIM is NVIDIA’s set of prebuilt, optimized, containerized inference microservices for running AI models on NVIDIA GPUs. NVIDIA describes NIM as portable microservices that simplify deployment of AI models across cloud, datacenter, workstation, and edge environments, typically exposing standard APIs for integration into applicationsLowPossible in some cases, but modern NIM profiles often assume newer GPUs and larger memory footprints than P4, and sometimes more than T4 depending on model size. It is worth testing selectively with small models, but I would not assume broad support on P4/T4 without checking the specific NIM/model requirements. (NVIDIA Docs)
    Mixed precision inference / trainingT4: Supported / P4: LimitedMixed precision means using a mix of higher-precision and lower-precision numeric formats in AI workloads so you get better speed and lower memory use without giving up model quality where precision still matters.LowT4 supports Tensor Cores and is much better for FP16/INT8 inference acceleration. P4 lacks Tensor Cores, so mixed-precision benefits are more limited and workload-dependent. (NVIDIA)
    NCCL (multi-GPU collectives)Supported, but topology-dependentNVIDIA Collective Communications Library. It is NVIDIA’s library for fast GPU-to-GPU communication, including both multi-GPU within a server and multi-node across servers. NVIDIA describes it as a topology-aware library of collective communication primitives optimized for NVIDIA GPUs and networking.MediumUseful for experimenting with multi-GPU and possibly multi-node communication patterns. It can work over PCIe and network paths, but the quality of results depends heavily on topology and software stack. This is a practical support judgment rather than a clean per-card matrix in the cited pages. (NVIDIA Docs)

    Is Supported on by both T4/P4
    Single-host NCCL tests across multiple GPUs in one server

    Multi-node NCCL tests over InfiniBand

    GPUDirect RDMA-assisted NCCL when the stack and PCIe topology cooperate


    GPUDirect P2P / peer-to-peerUnclear / topology-dependentGPU peer-to-peer memory access: one NVIDIA GPU can directly access or copy data to another NVIDIA GPU’s memory without staging the transfer through host RAMPeer-to-peer GPU memory access can work in some PCIe topologies, but support and performance vary a lot by motherboard, root complex, ACS/IOMMU behavior, and driver stack. Possibly worth validating experimentally in homelab. (NVIDIA Docs)

    Project Status

    Currently I am in the “waiting for hardware” to arrive stage of the project (mainly due to the switch from CX-3s to CX-4s). So lets take stock of where we are in the project and outline our next steps.

    Current State

    • IB Switch Racked
    • GPUs physically installed in all systems
    • Rough list of supported technologies to install and test
    • Basic installation steps and IB troubleshooting documented

    Next Steps

    • Install all CX-4, ensure drivers are installed/loaded properly, and possibly update CX-4 firmware
    • Install NVIDIA drivers, CUDA, and Container Toolkit on all GPU enabled hardware
    • Install IB cables, and power up the Mellanox IB Switch (and hope its not too loud)
    • Install 1x instance of active Subnet Manager (OpenSM) on one of target hosts
    • Configure IB IPs.
    • Work through listed/supported technologies in the NVIDIA matrix above.

    My goal is to document my progress in future posts.

  • Configuring LACP on TP-Link SX3008F for RHEL 9/10

    Configuring LACP on TP-Link SX3008F for RHEL 9/10

    Goal here was to create 3 LACP Port-Channels on a tp-link SX3008F 10gbe switch connecting to RHEL 9/10 hosts also configured to use LACP.

    Specific Requirements

    • Jumbo Frames (mtu 9216)
    • LACP (802.3ad)
    • RHEL side = “xmit_hash_policy=layer3+4”
    • Switch side = src-dst-ip

    End result should be 20Gbe connectivity between hosts for fast NFS backups of Virtual Machines (SSD and HDD NFS) shares). Additional tuning was performed on the NFS host for optimum throughput


    Switch Side Config

    The tplink is Cisco like, but not exactly so commands were a bit of a challenge to nail down specific syntax.

    Jumbo Frames

    On this switch either jumbo frames in on or off for the switch, there is no per port config

    jumbo-size 9216

    Switch Port Config

    1st interface. Both ports added to channel-group 1 which we will define in a later step

    interface ten-gigabitEthernet 1/0/6
    description "columbia bond0 member"
    switchport general allowed vlan 10 untagged
    switchport pvid 10
    no switchport general allowed vlan 1
    channel-group 1 mode active
    exit

    2nd interface

    interface ten-gigabitEthernet 1/0/7
    description "columbia bond0 member"
    switchport general allowed vlan 10 untagged
    switchport pvid 10
    no switchport general allowed vlan 1
    channel-group 1 mode active
    exit

    Define Port-Channel (channel-group)

    interface port-channel 1
    description "columbia bond0"
    switchport general allowed vlan 10 untagged
    switchport pvid 10
    no switchport general allowed vlan 1
    exit

    Load Balance Mode

    The port-channel load-balance src-dst-ip command configures a network switch to distribute traffic across aggregated links (EtherChannel) based on a mathematical hash of both the source and destination IP addresses.

    This method ensures that traffic between the same two IP addresses consistently uses the same link while providing more even distribution compared to relying on MAC addresses alone.

    We will match this config on the server side via “xmit_hash_policy=layer3+4”

    port-channel load-balance src-dst-ip

    Don’t forget to save your work

    
    write memory

    Health Check

    # show etherchannel summary

    Here we can see both ports 6 and 7 are in PO1

    A command line interface output displaying network port flags, statuses, groups, port-channel details, and protocol information.

    # show lacp neighbor

    Command line output displaying LACP neighbor status, including flags, port details, and device information.

    # show vlan id 10

    Here we can see both ports 6 and 7 are in the correct vlan

    Command line output displaying VLAN configuration with ID 10, showing its name, status as active, and associated ports.

    # show interface status ten-gigabitEthernet 1/0/6

    Here we can see that we have 10G link and both ports have correct description

    Command line output displaying the status of two ten-gigabit Ethernet interfaces, including port number, status, speed, duplex, flow control, active medium, and description.

    # show lacp internal

    Terminal output showing LACP (Link Aggregation Control Protocol) internal status, including device flags, channel group information, and details for two ports with their respective states and settings.

    Configuring the RHEL Side

    Here we need to have the following

    • 2 interfaces in a bond
    • 1 bond
    • 1 bridge
      • IP is on bridge
      • bond is connected to bridge

    The Logical Order of Operations

    1. The Bridge (bridge0): The “Top Level” virtual switch that holds the IP address.
    2. The Bond (bond0): The logical aggregation of physical NICS. It is a “Port” of the bridge.
    3. The Bond Ports (enp...): The physical wires. These are “Ports” of the bond.

    Step 1: Create the Bridge (The Anchor)

    You create the bridge first because the bond needs a “controller (aka bond)” to point to.

    nmcli connection add type bridge con-name bridge0 ifname bridge0 \
    ipv4.method manual ipv4.addresses 10.1.10.21/24 ipv4.gateway 10.1.10.1 \
    ipv4.dns 10.1.10.74 ipv6.method disabled \
    802-3-ethernet.mtu 9216

    Step 2: Create the Bond (The Controller)

    Note that the controller is the bridge we just made. We specify LACP (802.3ad) and the hashing policy here.

    nmcli connection add type bond con-name bond0 ifname bond0 \
    connection.controller bridge0 connection.port-type bridge \
    bond.options "mode=802.3ad,xmit_hash_policy=layer3+4" \
    802-3-ethernet.mtu 9216

    Step 3: Attach the Physical Interfaces (The Slaves)

    Now we tell the physical hardware to report to the bond. Crucial: The MTU must be set at this level so the hardware buffers are sized correctly for jumbo frames.

    # First Port
    nmcli connection add type ethernet con-name bond0-port1 ifname enp130s0f0 \
    connection.controller bond0 connection.port-type bond \
    802-3-ethernet.mtu 9216
    # Second Port
    nmcli connection add type ethernet con-name bond0-port2 ifname enp130s0f1 \
    connection.controller bond0 connection.port-type bond \
    802-3-ethernet.mtu 9216

    Step 4: Verification Checklist

    Once created, we bring the stack up from the top. NetworkManager will automatically trigger the underlying ports.

    1. Bring it up: nmcli connection up bridge0
    2. Verify MTU Consistency: Every device in the chain must match.
      • ip link show | grep 9216
    3. Verify LACP Sync: The switch must see the server.
      • cat /proc/net/bonding/bond0 (Look for “Partner Mac Address”)
    4. Verify Bridge Membership:
      • bridge link show (The bond should be listed as a member of the bridge).

    Why this order matters

    • If you define the physical ports first without a controller, they might try to get a DHCP address on their own.
    • By setting MTU 9216 at every single stage of the nmcli command, you prevent the kernel from defaulting any segment to 1500, which causes the exact “packet loss” issue you experienced during the jumbo ping tests.

  • Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator (OMSA) is Dell’s on-host hardware management and monitoring framework for PowerEdge servers. 

    It runs inside the operating system and provides direct visibility into system hardware such as RAID controllers, physical and virtual disks, power supplies, fans, temperatures, memory, processors, and chassis health. 

    OMSA communicates with the server’s iDRAC and hardware controllers to retrieve real-time status and exposes this information through command-line tools like omreport, optional web interfaces, and SNMP for monitoring systems.

     It is primarily used for hardware diagnostics, RAID and storage monitoring, fault detection, and health reporting, allowing administrators to verify system integrity and troubleshoot hardware issues without leaving the operating system.

    So now that we have an idea of what it is, lets install it.

    Note: Our system under test is a Dell R720. I probably should have lead with that.


    Installation

    First you will need to download the bootstrap script as shown below.

    curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi

    Now we will make it executable.

    chmod +x bootstrap.cgi

    And then we run it.

    sudo ./bootstrap.cgi

    Now we have completed the following.

    • Added Dell’s official repo
    • Installed Dell signing keys
    • Configured OMSA package sources

    Now Install OpenManage (omreport)

    sudo dnf install srvadmin*

    This installs:

    • omreport
    • omconfig
    • Storage monitoring
    • RAID tools
    • CIM providers

    Starting and Enabling Services

    Apparently, service names can vary from OS to OS, so run the command below to verify the correct service names

    systemctl list-unit-files | grep -Ei 'dsm|omsa|srvadmin'
    dsm_om_connsvc.service enabled disabled
    dsm_om_shrsvc.service disabled disabled
    dsm_sa_datamgrd.service enabled disabled
    dsm_sa_eventmgrd.service enabled disabled
    dsm_sa_snmpd.service enabled disabled

    Now that we have the names of the services we can start and enable them as shown below

    # sudo systemctl enable --now dsm_om_connsvc dsm_sa_datamgrd dsm_sa_eventmgrd dsm_sa_snmpd dsm_om_shrsvc

    OpenManage Server Administrator (OMSA) Services
    Service NameFull NamePurposeRequired?Notes
    dsm_om_connsvcOMSA Connection ServiceProvides the web-based interface (HTTPS/1311) used to access OMSA remotely or locally✅ Yes (if using OMSA web UI)This is what allows access via https://<host&gt;:1311
    dsm_om_shrsvcOMSA Shared ServicesProvides shared libraries and backend support for other OMSA components⚠️ UsuallyRequired by other OMSA components; often left disabled unless needed
    dsm_sa_datamgrdSystems Management Data ManagerCollects and maintains hardware inventory and system data✅ YesRequired for hardware monitoring (disks, temps, fans, etc.)
    dsm_sa_eventmgrdSystems Management Event ManagerHandles hardware events, alerts, and logs✅ YesRequired for alerts and log reporting
    dsm_sa_snmpdOMSA SNMP AgentProvides SNMP interface for monitoring tools (LibreNMS, Zabbix, etc.)⚠️ OptionalNeeded only if using SNMP monitoring

    Service Review

    Now let’s review each service in detail, and see how we can leverage each one.


    dsm_om_connsvc

    First we need to ensure our Firewall allows this traffic. So run the commands below (RHEL), ubuntu may use ufw.

    # firewall-cmd --permanent --add-port=1311/tcp
    # firewall-cmd --reload

    Now in your web browser, navigate to https://viper.lab:1311/ (Note https, and change host name accordingly).

    Similar in appearance to the Idrac – you can find system information here, some of which does not appear in the Idrac itself. Everything is read-only.

    Screenshot of Dell OpenManage Server Administrator showing Fan Probes Information, including fan redundancy status and probe list with readings, thresholds, and status indicators.

    dsm_om_shrsvc

    (OMSA Shared Services) is a supporting service, not a management interface or monitoring agent.

    It provides:

    • Shared libraries
    • Inter-process communication
    • Authentication helpers
    • Common backend utilities

    These are required by other OMSA components to function correctly. So nothing really to see here.


    dsm_sa_datamgrd.service

    This is the service that collects all the data that is displayed in the WebUI (port 1311)


    dsm_sa_eventmgrd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    This service relies entirely on the existing system SNMP configuration, so no need to modify anything if you already have snmp configured and running with your specfic community string.


    dsm_sa_snmpd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol

    It acts as the interface between the data collected internally by OMSA and third-party monitoring platforms such as LibreNMS, Zabbix, or Nagios. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    dsm_sa_snmpd.service acts as the Dell OMSA integration layer between the server’s hardware telemetry and the system’s SNMP stack, allowing Dell-specific health, storage, and sensor data to be exposed through standard SNMP queries. 

    It does not replace or conflict with existing monitoring agents such as Zabbix or LibreNMS (which I am using) Instead, it complements them by extending the SNMP data they can read

    dsm_sa_snmpd provides access to Dell-specific MIBs that Zabbix Agent and LibreNMS Agent (or any other agent you may use) can query via the system’s SNMP daemon. 

    Basically, dsm_sa_snmpd is the bridge that exposes OMSA’s hardware awareness to your existing monitoring stack.
    Note that if you are using Zabbix you will need to install additional templates. For example, this one, note that I have not tried it yet.

    Ompreport

    Omreport is the command-line interface (CLI) frontend for OMSA. It queries the OMSA backend services and formats the results for humans or scripts

    Note we have to use the absolute path to run this as the working dir is not in our $PATH.

    /opt/dell/srvadmin/sbin/omreport system summary

    If you want to add to your $PATH, run the command below

    echo 'export PATH=$PATH:/opt/dell/srvadmin/sbin' | sudo tee /etc/profile.d/dell-omsa.sh
    source /etc/profile

    Example usage below.

    omreport system summary
    omreport chassis temps
    omreport chassis fans
    omreport chassis power

    Here is an example I found useful. For some reason my raid controller keeps throwing these errors which do not seem to go away despite the fact that both SAS cables are connected (tried switching them) and all disks are healthy.

    # omreport system esmlog
    Severity : Critical
    Date and Time : Sat Jan 31 21:15:18 2026
    Description : The storage BP1 SAS B cable is not connected, or is improperly connected.
    Severity : Ok
    Date and Time : Sat Jan 31 21:15:20 2026
    Description : The chassis is closed while the power is off.
    Severity : Critical
    Date and Time : Sat Jan 31 21:15:21 2026
    Description : The storage BP1 SAS A cable is not connected, or is improperly connected.

    Enter omconfig.


    omconfig

    omconfig is the configuration and control interface for Dell OpenManage Server Administrator (OMSA). 

    Unlike omreport, which is strictly read-only, omconfig is used to change system and OMSA behavior—including alerting, storage actions, and certain hardware-related settings exposed by OMSA. 

    Omconfig It operates by issuing commands to the OMSA backend services (primarily dsm_sa_datamgrd and dsm_sa_eventmgrd) and applying those changes at the software management layer, not directly to hardware firmware. 

    Common uses include enabling or disabling alert actions, configuring storage-related behavior, managing notification settings, and triggering maintenance operations such as consistency checks. 
    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    So back to my two SAS cable alerts. Lets leverage omreport and omconfig to diagnose and hopefully ether determine that there is an actual issue, or suppress the alerts.


    OMCONFIG and OMREPORT – Real world Usage

    First lets take a sneaky-peak at my raid controller.

    omreport storage controller
    Controller PERC H730 Adapter(Slot 4)
    Controller
    ID : 0
    Status : Ok
    Name : PERC H730 Adapter
    Slot ID : PCIe Slot 4
    State : Ready
    Firmware Version : 25.5.9.0001
    -----truncated---

    According to chatgpt I am getting these errors because the system expects that I have an external enclosure connected. I do not think that is the case, as that seems dumb. More than likely this alert is due to the fact that I am using an H730 (a 13G controller) (and its cables) in an R720 (a 12G server) in Slot 4, which is a PCIe Adapter and not the integrated “mini” version.

    So let’s look at the disks and make sure all is well

    Here we can see that the PERC H730 adapter does not “See” the connector states as connected, rather just ready

    omreport storage connector controller=0
    List of Connector(s) on Controller PERC H730 Adapter (Slot 4)
    ID : 0
    Status : Ok
    Name : Connector 0
    State : Ready
    Connector Type : SAS Port RAID Mode
    Termination : Not Applicable
    SCSI Rate : Not Applicable
    ID : 1
    Status : Ok
    Name : Connector 1
    State : Ready
    Connector Type : SAS Port RAID Mode
    Termination : Not Applicable
    SCSI Rate : Not Applicable

    In order to use this 13th gen raid controller in a 12th gen Dell Server I could not use the original SAS cables. Rather I used SFF-8087 to SFF-8643 cables Which can cause the sideband signals (SGPIO / SES) which tell the iDRAC everything is okay which is why we are seeing the errors.

    As stated above this is not actually a problem as I can see my disks. Lets do that below.

    omreport storage pdisk controller=0 | awk '
    /^ID[[:space:]]*:/ {id=$3; state=""; status=""}
    /^State[[:space:]]*:/ {state=$3}
    /^Status[[:space:]]*:/ {status=$3}
    (state!="" && status!="" && id!="") {print id, state, status; id=""; state=""; status=""}
    '
    0:0 Online Non-Critical
    0:1 Online Non-Critical
    0:2 Online Non-Critical
    0:3 Online Non-Critical
    0:4 Online Non-Critical
    0:5 Online Non-Critical
    0:6 Online Non-Critical
    0:7 Online Non-Critical

    Here we can see we have 8 disks, no critical errors

    First lets clear the esmlog with omconfig – this will clear the error till we reboot

    omconfig system esmlog action=clear

    Sadly there is not a permanent fix. I would need to get a different raid controller that uses the original cables.


  • Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Introduction

    In this step-by-steps guide we will replace the out of the box nouveau drivers on RHEL9 with Nvidia Drivers. We will also install the the Nvidia CUDA Toolkit and the Nvidia Container Toolkit.


    GPU and Driver Inspection

    First we need to make sure that our Nvdia GPU is recognized by Red Hat Enterprise Linux 9 (RHEL9).

    lspci -nn | grep -i nvidia
    b6:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [A40] [10de:2235] (rev a1)
    

    Using the command below we can see that we are currently using the non-propietary nouveau driver.

    # lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nouveau
    	Kernel modules: nouveau
    

    Configuring Repositories for the Nvidia Driver Install

    First we need to enable the RHEL9 CodeReady Builder repo. Note we are running these commands as root.

    # subscription-manager repos --enable codeready-builder-for-rhel-9-$(uname -i)-rpms
    

    Next we will need to install and configure the EPEL repo.

    # dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
    

    Now we install the ELRepo project repo – this will provide nvidia-detect which we can utilize later

    # dnf -y  install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm

    Prerequisites for Nvidia Driver Install

    Now we need to install dependencies and build tools.

    # dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc make dkms acpid libglvnd-glx libglvnd-opengl libglvnd-devel pkgconfig
    

    Install Nvidia Drivers

    Install nvidia-detect from the ELRepo project repo.

    # dnf -y install nvidia-detect

    Now install the Nvidia Drivers.

    # dnf -y install $(nvidia-detect)

    Now reboot.


    Confirming Nvidia Driver Installation

    Now lets run the command below one more time.

    [root@gpu ~]# lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [250] Latency Tolerance Reporting
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nvidia
    	Kernel modules: nouveau, nvidia_drm, nvidia
    

    As you can see in the output below, the kernel is loading the Nvidia driver. We can still see nouveau kernel modules listed, but that is fine, as they are not loaded. We can confirm this with the command below.

    # lsmod | grep nouveau

    The above command should not output anything, while the opposite should be true for the command below.

    # lsmod | grep nvidia

    Configure Nvidia Persistenced

    Start and enable nvidia-persistenced.service. This will enable persistence-mode which will keep the nvidia device state from going “stale”

    # systemctl enable nvidia-persistenced.service
    # systemctl start nvidia-persistenced.service
    

    Installing the Nvidia CUDA Toolkit

    We will now follow the official guide and install the Nvidia CUDA toolkit. Per that guide, we need to enable a few repos, however two of those repos should be enabled by default, and the other one we enabled above, however I will list them here for the sake of documentation.

    # subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
    # subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
    # subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

    Now we install the Nvidia repo for the CUDA toolkit.

    dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
    

    Now install the CUDA toolkit as shown below

    # sudo dnf -y install cuda-toolkit

    Confirm that the toolkit is installed and note the version.

    # rpm -qa cuda-toolkit
    cuda-toolkit-12.8.1-1.x86_64
    

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well. Note that the cuda version should match the one that you installed above.

    export PATH=/usr/local/cuda-12.8/bin:$PATH

    Now test nvcc as shown below.

    # nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Feb_21_20:23:50_PST_2025
    Cuda compilation tools, release 12.8, V12.8.93
    Build cuda_12.8.r12.8/compiler.35583870_0
    

    Installing the Nvidia Container Toolkit

    Next we will install the Nvidia Container Toolkit, which allows users to run GPU-accelerated containerized applications.

    A bit about Container Management in RHEL 9

    The default container packages in RHEL 9 are as follows.

    1. Podman – daemonless container image
    2. Buildah – tool for building OCI (Open Container Initiative) container images
    3. Skopeo – tool for managing container images and repos
    4. CRIU – tool to create and save running container checkpoints to disk
    5. Udica – tool for managing SELinux policies for containers

    Installation of the toolkit

    We will follow the instructions as documented here.

    First we configure the repo

    # curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

    Then install via dnf

    # dnf install -y nvidia-container-toolkit

    Configuring the Container Toolkit for Podman

    Generate the CDI specification file using the command below.

    # nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Now lets check the names of the generated device(s).

    # nvidia-ctk cdi list
    INFO[0000] Found 3 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=GPU-7e880be2-891c-72e3-9515-0fd51240e7f4
    nvidia.com/gpu=all
    

    References

    1. https://medium.com/@blackhorseya/step-by-step-guide-to-installing-nvidia-drivers-on-rhel-9-1107e0cd641d
    2. https://access.redhat.com/discussions/227d2101-b4e3-490a-aa1c-601c407ec038
    3. https://darryldias.me/2022/install-nvidia-drivers-on-rhel-9/
    4. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    5. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-rhel-rocky
    6. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html