Tag: gpu

  • The AMD R9700 – Ubuntu 24.04 Quickstart Guide

    The AMD R9700 – Ubuntu 24.04 Quickstart Guide

    ….and I bought an AMD GPU.

    It’s the first time I’ve gone non-Nvidia since before 2010 as I’ve been a Nvidia loyalist since the GeForce 256 back in 1999 (was 3dfx prior). But Nvidia GPU pricing on anything with decent VRAM has gotten absurd, even on the used market. I kept hearing that ROCm (Radeon Open Compute) support for inference has finally matured. So I decided to find out for myself.

    I picked up the AMD Radeon AI PRO R9700, 32GB. Not cheap, but brand new its cheaper than an RTX 3090 24gb from 2020.

    Since this is my first AMD in very long while, going to have to figure out drivers. So let’s get started. First lets gather some info by running the commands below.

    lspci -nn | grep -Ei 'VGA|Display|AMD|ATI'
    lsmod | grep amdgpu
    dmesg | grep -i amdgpu | tail -50

    Verify GPU Detection

    Prior to card swap, purged my Ubuntu 24.04 system of all Nvidia drivers and tools.

    Lets check to see if the card is detected properly

    lspci -nn | grep -Ei 'VGA|Display|AMD|ATI'

    This command searches the PCI device list for anything matching VGA, Display, AMD, or ATI. In the output below we see the following

    • 03:00.0 is the PCI id of the R9700 GPU
    • Vendor ID 1002 confirms AMD Device
    • 7551 is how the kernel identifies this GPU generation/device
    • The two PCI bridge entries show the GPU exposes an internal PCIe switch/bridge topology
    01:00.0 PCI bridge: AMD Navi 10 XL Upstream Port of PCI Express Switch
    02:00.0 PCI bridge: AMD Navi 10 XL Downstream Port of PCI Express Switch
    03:00.0 VGA compatible controller: AMD Device [1002:7551]
    03:00.1 Audio device: AMD Device [1002:ab40]

    The important takeaway is that Ubuntu already recognizes the R9700 correctly and the in-kernel AMDGPU driver is successfully initializing it.

    lsmod | grep amdgpu

    This command above shows whether the AMDGPU kernel module is loaded and what other kernel modules depend on it.

    amdgpu 21188608 26

    This tells us:

    • amdgpu is loaded
    • the module occupies roughly 21 MB of kernel module memory
    • 26 references/users currently depend on it
    dmesg | grep -i amdgpu | tail -50

    The command above shows the last 50 kernel messages associated with AMDGPU initialization.

    The most important things we see are:

    MEM ECC is active.

    The GPU’s VRAM ECC capability is active.

    VRAM: 32624M

    The kernel detects approximately 32 GB of VRAM.

    Detected VRAM RAM=32624M, BAR=32768M

    Preparing for ROCm Install

    First lets make sure that the kernel has created the ROCm compute device

     ls -l /dev/kfd
    crw-rw---- 1 root render 235, 0 Aug  9 16:20 /dev/kfd
    
    
    

    /dev/kfd is the Kernel Fusion Driver device that ROCm/HIP uses to submit compute workloads to the AMD GPU. AMD’s ROCm installation process relies on access through the render/video device groups.

    In the output above, you can see that the Kernel Fusion Driver is owned by root with group membership of render. We need to add my user to the render group. Note that I had to reboot to pick up the changes.

    sudo usermod -aG render cpaquin

    Now we can add the ROCm 7.2.2 package repo, which is two steps. This one below…

    sudo mkdir --parents --mode=0755 /etc/apt/keyrings
    wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

    And this next one to actually add the repo.

    sudo tee /etc/apt/sources.list.d/rocm.list <<'EOF'
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2.2 noble main
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/7.2.1/ubuntu noble main
    EOF

    Now let’s be smart and ensure that the repo was added properly.

    cat /etc/apt/sources.list.d/rocm.list
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2.2 noble main
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/7.2.1/ubuntu noble main

    Now before we install anything, per AMD recommendation, we need to pin the above repo priority. Pinning means settings which repository Ubuntu prefers when the same package is available from multiple repositories, which means prefer the AMD repos over Ubuntu repos for AMD bits and bobbles.

    sudo tee /etc/apt/preferences.d/rocm-pin-600 <<'EOF'
    Package: *
    Pin: release o=repo.radeon.com
    Pin-Priority: 600
    EOF

    Now we update metadata.

    sudo apt update

    We can see the repo preference via pinning below.

     apt policy rocm
    rocm:
      Installed: (none)
      Candidate: 7.2.2.70202-86~24.04
      Version table:
         7.2.2.70202-86~24.04 600
            600 https://repo.radeon.com/rocm/apt/7.2.2 noble/main amd64 Packages
    
    
    

    ROCm Install

    First, lets make sure we have the following installed.

    sudo apt install python3-setuptools python3-wheel

    Now we can install ROCm. It takes a bit so be patient, or don’t.

    sudo apt install rocm

    Now lets verify that ROCm can see the GPU

    rocminfo

    Truncated output below

    rocminfo
    ROCk module is loaded
    =====================
    HSA System Attributes
    =====================
    Runtime Version: 1.18
    Runtime Ext Version: 1.15
    System Timestamp Freq.: 1000.000000MHz
    Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
    Machine Model: LARGE
    System Endianness: LITTLE
    Mwaitx: DISABLED
    XNACK enabled: NO
    DMAbuf Support: YES
    VMM Support: YES
    *******
    Agent 2
    *******
    Name: gfx1201
    Uuid: GPU-407fcdae2bc6b11d
    Marketing Name: AMD Radeon AI PRO R9700
    Vendor Name: AMD
    Feature: KERNEL_DISPATCH
    Profile: BASE_PROFILE
    Float Round Mode: NEAR
    Max Queue Number: 128(0x80)
    Queue Min Size: 64(0x40)
    Queue Max Size: 131072(0x20000)
    Queue Type: MULTI
    Node: 1
    Device Type: GPU

    And now run amd-smi.

    amd-smi
    +------------------------------------------------------------------------------+
    | AMD-SMI 26.2.2+671d39a71e amdgpu version: 7.0.0-28 ROCm version: 7.2.2 |
    | VBIOS version: 00158746 |
    | Platform: Linux Baremetal |
    |-------------------------------------+----------------------------------------|
    | BDF GPU-Name | Mem-Uti Temp UEC Power-Usage |
    | GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage |
    |=====================================+========================================|
    | 0000:03:00.0 ...Radeon AI PRO R9700 | 1 % 31 °C 0 9/300 W |
    | 0 0 N/A N/A | 3 % 20.0 % 661/32624 MB |
    +-------------------------------------+----------------------------------------+
    +------------------------------------------------------------------------------+
    | Processes: |
    | GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % |
    |==============================================================================|
    | No running processes found |

    What we are looking at:

    • ROCm 7.2.2 is active.
    • The Radeon AI PRO R9700 is correctly identified.
    • Temperature is 31 °C.
    • Power draw is only 9 W / 300 W at idle.
    • Fan is at 20%.
    • VRAM usage is 661 MB / 32,624 MB, which is normal for the display/driver stack.
    • No uncorrectable ECC errors: UEC 0.
    • No compute processes are currently using the GPU.

    Test ROCm inside Docker

    Can a Docker container access the R9700 through ROCm? Lets find out.

    docker run --rm \
    --device=/dev/kfd \
    --device=/dev/dri \
    --group-add 992 \
    --group-add 44 \
    rocm/rocm-terminal:latest \
    rocminfo

    What this does:

    • starts a temporary AMD ROCm container
    • passes /dev/kfd into it for ROCm compute
    • passes /dev/dri into it for the GPU device nodes
    • gives the container the needed video (44) and render (992) group access
    • runs rocminfo inside the container
    • deletes the container afterward because of --rm

    If it works, we should see the same R9700 identification inside Docker that we saw on the host, including something like:


    Pytorch Test

    Next we’ll use AMD’s validated PyTorch container for ROCm 7.2 / Ubuntu 24.04 / Python 3.12 / PyTorch 2.9.1. AMD recommends Docker for PyTorch because the image already contains a tested PyTorch+ROCm combination.

    docker run --rm \
    --device=/dev/kfd \
    --device=/dev/dri \
    --group-add 992 \
    --group-add 44 \
    --security-opt seccomp=unconfined \
    --ipc=host \
    --shm-size 8G \
    rocm/pytorch:rocm7.2_ubuntu24.04_py3.12_pytorch_release_2.9.1 \
    python3 -c "import torch; print('PyTorch:', torch.__version__); print('ROCm:', torch.version.hip); print('GPU available:', torch.cuda.is_available()); print('GPU:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NONE')"

    This will likely download a fairly large container image the first time.

    What we want to see is essentially:

    PyTorch: 2.9.1+rocm7.2.0.git7e1940d4
    ROCm: 7.2.26015-fc0010cf6a
    GPU available: True
    GPU: AMD Radeon AI PRO R9700

    One potentially confusing point: PyTorch keeps the API name torch.cuda even when using AMD ROCm.

    torch.cuda.is_available() returning True means PyTorch has a usable GPU backend; on this container that backend is HIP/ROCm, not NVIDIA CUDA. AMD uses this same test in its ROCm PyTorch verification instructions.

    So the R9700 is now validated for actual ROCm/PyTorch compute inside Docker. Next Step VLLM and test some models.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 3: RDMA Performance Testing

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 3: RDMA Performance Testing

    Before moving on to Part 3 of this project, lets review what we have accomplished thus far.

    In Part 1 and Part 2 we have…

    • Did a bit of planning and scoping
    • Built a 3-node GPU cluster (viper, columbia, prometheus)
    • Interconnected with InfiniBand Installed and validated ConnectX-4 NICs and RDMA stack (mlx5, ib_core, etc.)
    • Brought up the InfiniBand fabric using OpenSM (links active, LIDs assigned)
    • Verified topology and connectivity (ibstat, ibnetdiscover)
    • Configured IP over InfiniBand for basic networking between nodes Identified PCIe/NUMA limitations affecting optimal GPU↔NIC performance

    We are now ready to do some performance testing of our Infiniband network.


    Pre-Test Setup

    Before we can get started on our perf testing we have bit of work to do. We are going to install a few packages, and configure some tunables.

    Diagnostic Tools

    First lets make sure that we have a couple tools installed, so lets install some rpms.

    sudo dnf install infiniband-diags libibverbs-utils librdmacm-utils -y

    Kernel Modules

    InfiniBand and GPUDirect require specific modules to load at boot. So lets create hpc.conf in /etc/modules-load.d/. This creates (or overwrites) /etc/modules-load.d/hpc.conf. This file ensures each module loads automatically at boot via systemd-modules-load. Run this on each host.

    sudo tee /etc/modules-load.d/hpc.conf >/dev/null <<'EOF'
    ib_ipoib
    ib_umad
    ib_uverbs
    nvidia-peermem
    EOF

    Then force load the modules.

    sudo modprobe ib_ipoib ib_umad ib_uverbs nvidia-peermem

    Below is a short breakdown/description for each module.

    ModuleHow it’s used
    ib_ipoibProvides IP networking over InfiniBand (e.g., ib0) for SSH, NFS, TCP/IP
    ib_umadEnables userspace IB management tools (e.g., ibstat, fabric queries)
    ib_uverbsCore RDMA interface used by applications (MPI, NCCL, libibverbs)
    nvidia-peermemEnables GPUDirect RDMA for direct GPU ↔ NIC memory transfers (no CPU copy)

    Locked Memory Limits

    RDMA works by “pinning” memory so the OS cannot swap it to disk. So we need to create /etc/security/limits.d/99-hpc.conf as shown below.

    sudo tee /etc/security/limits.d/99-hpc.conf >/dev/null <<'EOF'
    * soft memlock unlimited
    * hard memlock unlimited
    EOF

    Performance & RDMA Benchmarking

    Health Check

    First lets run the following commands on any host under test, just to make sure the InfiniBand network is healthy before we start any testing. Run each line individually and make note of the output.

    hostname
    ibstat
    ibv_devinfo | egrep 'hca_id|transport|fw_ver|port:|link_layer|active_mtu|sm_lid|port_lid'

    You are specifically interesting in the following

    • Device Present (mlx5)
    • State: Active
    • Physical state: LinkUp
    • Link layer: InfiniBand

    Confirm HCA Name and Port Number

    Run on any device under test – we will need this for our test on our receiver and sender side.

    ibv_devices

    Output from columbia.lab.

     device          	   node GUID
     ------          	----------------
     mlx5_0          	248a070300ac5414
    
    

    Output from prometheus.lab

     device          	   node GUID
     ------          	----------------
     mlx5_0          	248a070300ac5610
    
    

    Run the RDMA Latency Test

    For our ib_send_lat (latency test) our device IP addresses are as follows.

    • columbia.lab – 172.16.50.12
    • prometheus.lab -172.16.50.11

    On our first device, columbia.lab, we run the following and leave it running.

    ib_send_lat -d mlx5_0 -i 1

    Now over on prometheus, run the command below. Insert the IP from columbia captured above. You will see a good bit of output in your terminal window.

    ib_send_lat -d mlx5_0 -i 1 <columbia_ip>

    Key configuration details

    So assuming the test did not fail, you are going to see some data spit out. Lets make sense of some of it.

    ParameterValueMeaning
    Devicemlx5_0ConnectX-4 (mlx5 driver)
    TransportIB (RC)Reliable Connection (standard RDMA mode)
    MTU4096Optimal for IB performance
    Queue Pairs1Single stream test
    Inline data236BSmall messages optimized
    Link typeInfiniBandCorrect mode

    What this test is actually doing

    ib_send_lat:

    • Registers memory with the NIC
    • Creates RDMA queue pairs
    • Sends messages using:
      • ibv_post_send()
    • Measures completion latency via completion queues (CQs)

    This is direct RDMA messaging, not IP networking.

    Our Overall results

    • Average latency: ~1.15 µs
    • Typical latency: ~1.14 µs
    • Minimum latency: 1.06 µs
    • Outliers: up to 11.41 µs
    • Conclusion: Healthy RDMA performance

    While InfiniBand ≠ RDMA test by default, our test ib_send_lat specifically uses RDMA verbs, so a successful result proves RDMA is working.

    In the output above, our average latency confirms that RDMA is functioning, as is kernel bypass. Note, that while we are using TCP/IP to setup the test, the actual data transfer is NIC to NIC and memory to memory. The queue pair exchange confirms RDMA session, as QPs were created on both nodes and transitioned through the required queue pair states shown below.

    StateNamePurposeAnalogy
    INITInitializeLocal QP setupPhone powered on
    RTRReady to ReceiveCan receive remote dataYou know the other person’s number
    RTSReady to SendFully operational (send + receive)Call connected and talking

    Run the RDMA Bandwidth Test

    For this test we will run ib_send_bw. This test measures the following.

    • Throughput (bandwidth) of RDMA send operations
    • NIC-to-NIC data transfer rate
    • Memory → NIC → fabric → NIC → memory

    Again this test uses IP to establish the initial connection between nodes, but make no mistake we are using RDMA verbs and are testing IB traffic (not IP traffic).

    So over on our first node (columbia.lab) we run the following.

    ib_send_bw -d mlx5_0 -i 1 -a

    Why these flags

    FlagPurpose
    -d mlx5_0Select your ConnectX-4 device
    -i 1Use IB port 1
    -aSweep all message sizes

    And on our second node we run the command shown below.

    ib_send_bw -d mlx5_0 -i 1 -a <columbia_ip>

    Assuming that this command does not fail, you will see a bunch of output that we need to interpret. This output is truncated, but I wanted to give you an idea of what to expect in the output.

    ib_send_bw -d mlx5_0 -i 1 -a 172.16.50.12
    ---------------------------------------------------------------------------------------
    Send BW Test
    Dual-port : OFF Device : mlx5_0
    Number of qps : 1 Transport type : IB
    Connection type : RC Using SRQ : OFF
    PCIe relax order: ON Lock-free : OFF
    WARNING: CPU is not PCIe relaxed ordering compliant.
    WARNING: You should disable PCIe RO with `--disable_pcie_relaxed` for both server and client.
    ibv_wr* API : ON Using DDP : OFF
    TX depth : 128
    CQ Moderation : 100
    CQE Poll Batch : 16
    Mtu : 4096[B]
    Link type : IB
    Max inline data : 0[B]
    rdma_cm QPs : OFF
    Data ex. method : Ethernet
    ---------------------------------------------------------------------------------------
    local address: LID 0x02 QPN 0x0107 PSN 0xed831e
    remote address: LID 0x01 QPN 0x0107 PSN 0xa2c511
    ---------------------------------------------------------------------------------------
    #bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
    Conflicting CPU frequency values detected: 1200.000000 != 1300.046000. CPU Frequency is not max.
    2 1000 7.79 7.40 3.879797

    Keep in mind that our IB bottleneck is our 40Gbe IB Switch. Here is what we can interpret from our test data.

    • Plateau was about: 3776.9 MiB/s which is about 31.7 Gbit/s
    • That is a normal practical result for a nominal 40 Gb InfiniBand-class link
    • Our plateau is consistent and stable, which is good

    Our InfiniBand link is healthy enough to sustain near-expected throughput, there are no obvious severe bottleneck or broken configuration. We are seeing some GPU frequency warnings, and some “PCIe relaxed ordering” warnings so lets fix those any try the test again.

    What is PCIe Relaxed Ordering? PCIe Relaxed Ordering is a performance feature where The CPU/NIC is allowed to reorder memory transactions. This can improve throughput by reducing stalls and increasing parallelism

    On both hosts, run the command below.

    cpupower frequency-set -g performance


    Now back on the first host, kick off the listen side of the test.

    ib_send_bw -d mlx5_0 -i 1 -a -q 4 --disable_pcie_relaxed

    And on the other server we kick off the test itself.

    ib_send_bw -d mlx5_0 -i 1 -a -q 4 --disable_pcie_relaxed <columbia_ip>

    Why these flags

    FlagPurpose
    -d mlx5_0Select your ConnectX-4 device
    -i 1Use IB port 1
    -aSweep all message sizes
    -q 4Use multiple queue pairs (better utilization)
    --disable_pcie_relaxedMatch your CPU capabilities and remove warning

    So lets summarize our output.

    • Almost identical throughput as initial test
    • Slight improvement in consistency
    • Cleaner test conditions (set cpu-frequency to performance)
    • Multiple QPs established (we see 4 QPNs)
    • We still see CPU Frequency is not max, however this is non issue as we have already saturated our links.


    Wrap Up

    In our previous post, we stood up our IB network, and performed some basic fabric tests. Today was all about performance testing and testing with the actual RDMA verb stack. We found that our fabric was pretty much performing as expected out of the box with minimal tuning, as we are hitting near-theoretical limits for our 40Gb hardware.

    We have a stable, low latency, high bandwidth IB Fabric.

    I was hoping to get to GPU direct testing today, however that looks like it might a bit of a beast and I think I will call it a day and do a bit more research on the topic.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 1: Project Overview

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 1: Project Overview

    Introduction

    InfiniBand is a mature interconnect technology known for high bandwidth and low latency. It has long been used in supercomputing and HPC environments, and has also been deployed in certain storage and clustered infrastructure designs as an alternative to Fibre Channel.

    More recently, InfiniBand has seen strong continued adoption in large-scale AI and GPU clusters, where its high bandwidth, ultra-low latency, and support for technologies such as RDMA, GPUDirect RDMA, and NCCL make it well suited for distributed training and other GPU-to-GPU communication workloads.

    This project involves the architecture, deployment, and optimization of a high-speed InfiniBand (IB) fabric to facilitate low-latency, high-throughput communication between 3x dual-homed, GPU enabled, RHEL 9/10.1 servers.

    By integrating Mellanox ConnectX-4 adapters with an InfiniScale IV switch, we will establish a dedicated Remote Direct Memory Access (RDMA) backend separate from the standard management LAN.

    Additionally, we will become more familiar with the setup, configuration, and troubleshooting of InfiniBand networks and adapters, while also exploring the broad set of NVIDIA tools and technologies currently available to support multi-GPU clusters.

    This project also encompasses the installation and configuration of NVIDIA drivers, CUDA, the NVIDIA Container Toolkit, and other supported elements of the NVIDIA software stack needed to support HCP/AI Clusters and environments.

    Primary Objectives Summary

    • Fabric orchestration: Deploy and manage a QDR InfiniBand fabric using OpenSM.
    • RDMA enablement: Configure IPoIB in Connected Mode with a 65,520 MTU and validate RDMA functionality across the fabric.
    • GPU acceleration: Enable and test GPUDirect RDMA with nvidia-peermem for NVIDIA Tesla T4 and P4 GPUs.
    • Platform enablement: Install and configure NVIDIA drivers, CUDA, the NVIDIA Container Toolkit, and other supported NVIDIA software stack components required for GPU-enabled workloads.
    • Operations and telemetry: Develop hands-on familiarity with InfiniBand diagnostics, troubleshooting, and NVIDIA GPU monitoring tools.

    Bill of Materials (BOM)

    Below is the BOM for this project.

    ComponentQtyPARTDESCRIPTION
    Switch1xMellanox InfiniScale IV Is5022 Switch8-port Non-blocking Unmanaged 40Gb/s InfiniBand Switch System
    NICs3xMCX455A-ECAT MELLANOX CONNECTX-4 1 PORT EDR 100GB IB QSFP28 Infiniband/Ethernet Adapter
    GPU (New)2xNVIDIA Tesla P4 8GB GDDR5 (with active cooling mods) 
    GPU (Existing)3xNVIDIA Tesla T4 16GB GDDR5 (with active cooling mods) 
    Cabling3xFS 40Gbps QSFP+ 2M Passive DAC (QSFP-PC02) 3x Host to 1x Switch
    Server1xDell R720 (RHEL 9)2 x Intel Xeon E5-2697 (Ivy Bridge) v2 Twelve-Core Processor 2.7GHz 8.0GT/s 30MB LGA 2011, 128GB
    Server1xDell R730 (RHEL 10.1)2x Intel(R) Xeon(R) CPU E5-2690 (Broadwell-EP) v4 @ 2.40GHz, 256GB
    Server 1xDell R730 (RHEL 10.1)2x Intel® Xeon® Processor E5-2699 v4 (Broadwell-EP) @2.20GHz, 768GB

    Notes On the Bill of Materials (BOM)

    There as been a bit of flux on the exact BOM for this project (and my lab). A Dell R720 was recently added to my lab, replacing a mammoth T620. And while I like the tower form factor due to its ample number of PCI slots and spare SATA power cables, it has to sit on a rack mount shelf, takes up 5RU and is really heavy amd hard to move.

    So the CPUs/Memory from the T620 were migrated to the R720. However RHEL 10 deprecated support for Intel v2 processors, so I had to deploy RHEL 9 instead of RHEL10. RHEL 9 supported the CX-3 with in-band drivers, however support for the CX-3 was dropped from RHEL 10. So I had to switch to CX-4s, which were more costly. However further research found that RDMA was not supported on the CX-3, so I needed to move to the CX4 (or newer) anyways.

    IB Switch – I was able to pick up the unmanaged Mellanox InfiniScale switch on eBay quite cheaply. Being unmanaged it does not run subnet manager, which I will need to run on my primary host. Limited to 40GBe per port.

    IB Adapters – Initially, as stated above, I intended to use Mellanox CX-3 adapters as they were incredibly inexpensive (like $12 USD), however due to CX-3 driver being dropped in RHEL 10, I switched to CX-4s, which were supported out of the box on both RHEL 9 and RHEL 10. These adapters were not as cheap, but still affordable. Additionally, I needed low profile brackets in order to fit into the existing open PCIe slots in my servers. I did not want to use OFED drivers, as I was looking to have the adapters supported out of the box, and was not interested in fiddling with drivers. Additionally, support docs for RDMA mentioned CX-4 or newer were required.

    GPUS – Workstation class GPUs (like my 1x 3070 and my 2x 3060s 12Gb are not supported. I needed Datacenter Class NVIDIA CPUs installed in all 3 Dell Servers. I already owned 3x NVIDIA Tesla T4 (installed in R730s). Picked up 2x additional GPUs (NVIDIA Tesla M4), which are for functional validation only – not purchased for their performance or VRAM. Additional requirement was to stick with low-power GPUs that ran on PCIe power alone and did not require additional power connections.


    Nvidia T4/P4 Comparison and Feature Support

    Our LUT (Lab Under Test) consists of 5 GPUs, deployed across 3 Servers. Details below

    FeatureNVIDIA T4NVIDIA Tesla P4
    ArchitectureTuringPascal
    Release dateSeptember 12, 2018September 12, 2016
    CUDA cores2,5602,560
    Tensor cores320None
    vRAM16 GB GDDR68 GB GDDR5
    Memory bandwidth300–320+ GB/s192 GB/s
    PCIe interfacePCIe Gen3 x16PCIe Gen3
    Form factorLow-profile, single-slot, passiveLow-profile, single-slot, passive
    Max power70 W75 W
    ECC memory supportYesYes
    NVENC / NVDECYesYes
    NVIDIA vGPU supportYesYes
    GPUDirect RDMAYes, conditionally supportedYes, conditionally supported
    RDMA NIC requirementConnectX-4 or laterConnectX-4 or later
    GPUDirect RDMA topology requirementGPU and NIC should share the same upstream PCIe root complex for best support/performanceGPU and NIC should share the same upstream PCIe root complex for best support/performance
    NVLinkNoNo
    MIGNoNo

    Additional GPUs Available

    In addition to the 5x GPUs currently installed in my existing lab servers, I have few additional GPUs that are not currently deployed. I will include them below for reference.

    Generally speaking, these are either “older“, “hungrier” or “hotter” than the GPUs that I already have in service.

    FeatureNVIDIA Tesla K20NVIDIA Tesla P100
    ArchitectureKeplerPascal
    Release dateNovember 2012April 5, 2016
    CUDA cores2,4963,584
    Tensor coresNoneNone
    vRAM5 GB GDDR516 GB HBM2
    Memory bandwidth208 GB/s732 GB/s
    PCIe interfacePCIe Gen2 x16PCIe Gen3 x16
    Form factorFull-height, dual-slotFull-height, dual-slot
    Max power225 W250 W
    ECC memory supportYesYes
    NVENC / NVDECNoNo
    NVIDIA vGPU supportNoYes
    GPUDirect RDMAYes, conditionally supportedYes, conditionally supported
    RDMA NIC requirementConnectX-4 or laterConnectX-4 or later
    GPUDirect RDMA topology requirementGPU and NIC should share the same upstream PCIe root complex for best support/performanceGPU and NIC should share the same upstream PCIe root complex for best support/performance
    NVLinkNoDepends on model; PCIe P100: No, SXM2 P100: Yes
    MIGNoNo

    NVIDIA Tool Test Matrix

    Below is a Matrix of technologies, their supportability in my soon to be deployed stack, and a brief description of each technology.

    Technology / ProductSupportedDescriptionPriorityNOTES
    RDMA (InfiniBand / verbs)SupportedRemote Direct Memory Access. It is a networking technology that lets one computer access or transfer data directly to the memory of another computer without involving the remote CPU or operating system in the data path.
    HighDirect memory access over the network using your ConnectX-4 InfiniBand cards. This is the base networking capability you will use for low-latency, high-throughput node-to-node transfers.

    Native fit for CX-4 + IB switch
    GPUDirect RDMASupported GPUDirect RDMA is an NVIDIA technology that lets a third-party PCIe device directly read from or write to GPU memory without first copying the data through system RAM.HighLets a supported NIC perform RDMA directly to/from GPU memory, bypassing extra CPU copies. NVIDIA documents GPUDirect RDMA for Tesla/Quadro GPUs and requires ConnectX-4 or later NICs, with best results when GPU and NIC share the same upstream PCIe root complex. (NVIDIA Docs)

    One of the most relevant GPU+IB features for this setup
    GPUDirect StorageUnclearDesigned for direct data movement between storage and GPU memory. It is primarily positioned around storage stacks rather than IB switching alone, so whether your exact lab can validate it depends on OS, filesystem, NVMe/storage path, and supported software stack rather than just T4/P4 + CX-4. (NVIDIA Docs)Low NVIDIA technology that allows data to move directly between storage and GPU memory using DMA, instead of first bouncing through CPU memory.

    Best with NVME storage, have only SSDs and HDDs, no NVME support in Dell models under test.
    MIG (Multi-Instance GPU)Not supportedMIG starts with newer architectures and is documented in NVIDIA’s MIG guide as an Ampere-era feature. NVIDIA’s cloud-native docs explicitly note that Tesla T4 does not support MIG. P4 also predates MIG. (NVIDIA Docs)NoneNot supported.

    Multi-Instance GPU. It is an NVIDIA technology that lets a single supported GPU be partitioned into multiple smaller, isolated GPU instances
    Time-slicing / shared GPU schedulingSupportedGood for shared-lab/VM experiments
    Allows multiple workloads share one physical GPU by giving each workload a small turn on the GPU scheduler
    MediumSince T4 does not support MIG, NVIDIA documents time-slicing as a way to share T4 across multiple smaller jobs. This is useful for Kubernetes/OpenShift experiments or general shared-lab validation. P4 can also be shared through virtualization/software scheduling rather than MIG. (NVIDIA Docs)
    NVIDIA vGPUSupportedNVIDIA’s virtual GPU stack allows partitioning/sharing GPUs across VMs for compute, VDI, or graphics use cases. Both T4 and P4 are in NVIDIA’s supported vGPU product documentation. (NVIDIA Docs)HighNVIDIA’s docs describe it as enabling multiple VMs to have simultaneous, direct access to a single physical GPU using NVIDIA drivers inside the guest OS.

    It lets multiple virtual machines share one physical NVIDIA GPU.
    DCGM (Data Center GPU Manager)SupportedNVIDIA’s primary datacenter GPU management and telemetry framework for health, diagnostics, topology, clocks, thermals, ECC, profiling, and integration with cluster tooling. It is explicitly built for Tesla/datacenter GPUs. (NVIDIA Docs)HighDCGM is NVIDIA’s datacenter GPU management and monitoring framework. NVIDIA describes it as a lightweight user-space library/agent for administering NVIDIA datacenter GPUs in clusters and datacenters.
    DCGM ExporterSupportedPrometheus exporter built on top of DCGM that exposes GPU metrics over HTTP for scraping. Good fit for validating telemetry, dashboards, and alerting with your servers. (NVIDIA Docs)HighNVIDIA’s Prometheus exporter for GPU metrics, will utilize existing Grafana instance

    Easy to validate and useful operationally
    NVIDIA MerlinSupportedMerlin is NVIDIA’s recommender-system framework stack for training and especially inference pipelines. T4 is a strong fit; P4 may work for smaller or older inference experiments, but T4 is the more relevant target. Support is practical rather than “card-specific” in docs, since Merlin rides on the CUDA/framework/container stack. (NVIDIA Developer)Medium–LowNVIDIA framework for building recommender systems.

    A recommender system is the kind of ML system used for things like:
    product recommendations
    “people also watched”
    next-best content
    ranking search or feed results
    ad / click-through prediction
    TensorRTSupportedTensorRT is NVIDIA’s SDK/runtime for optimizing trained neural-network models for inference on NVIDIA GPUs. It takes a model from frameworks such as TensorFlow, PyTorch, or ONNX and builds an optimized inference engine that can use precision modes such as FP32, FP16, and INT8 where supported.HighNVIDIA’s inference optimizer/runtime. Very relevant for T4 and still usable on P4. T4 benefits significantly from Tensor Cores, so it is the better platform for validation. (NVIDIA Developer)
    CUDASupportedCore GPU programming/runtime stack. Required for most of the technologies you listed and the base layer for custom validation, benchmarks, peer access tests, and GPU-aware applications. (NVIDIA Docs)HighFoundation for most modern NVIDIA workflows
    NVIDIA Container ToolkitSupportedEnables Docker/Podman/Kubernetes containers to access NVIDIA GPUs cleanly. Useful for validating Merlin, TensorRT, PyTorch, RAPIDS, and exporter containers. (NVIDIA Docs)HighFoundation for most modern NVIDIA workflows, integrate with Podman on RHEL
    NVIDIA GPU OperatorSupportedKubernetes/OpenShift operator that automates driver, toolkit, DCGM, exporter, and related GPU software deployment. Best fit if you want to turn a lab into a small cluster validation environment. (NVIDIA Docs)LowGood if you want Kubernetes/OpenShift validation, however no plans to run OCP in near future.
    NVIDIA Fabric ManagerNot supported / not applicableNVIDIA Fabric Manager is software for managing NVSwitch / NVLink GPU fabrics inside supported multi-GPU servers. NVIDIA says it configures the NVSwitch memory fabric to form a single memory fabric among participating GPUs and monitors the NVLinks that support that fabric.NoneFabric Manager is for NVSwitch-based systems, not T4/P4 PCIe accelerator setups. Installed cards do not use NVSwitch. (NVIDIA Docs)

    Not supported here
    NVLinkNot supportedNVLink is NVIDIA’s high-speed direct interconnect for GPUs. It provides a much faster path for GPU-to-GPU communication than ordinary PCIe alone, and in some platforms it is also used for CPU/GPU or switch-based interconnect designs. NVIDIA describes it as a direct GPU-to-GPU interconnect used to scale multi-GPU I/O within a server.NoneNot Supported. Neither T4 nor P4 provides NVLink. Multi-node connectivity would be via InfiniBand/RDMA, not GPU-to-GPU NVLink. (NVIDIA)
    NVIDIA NIM / inference microservicesUnclear / limitedNVIDIA NIM is NVIDIA’s set of prebuilt, optimized, containerized inference microservices for running AI models on NVIDIA GPUs. NVIDIA describes NIM as portable microservices that simplify deployment of AI models across cloud, datacenter, workstation, and edge environments, typically exposing standard APIs for integration into applicationsLowPossible in some cases, but modern NIM profiles often assume newer GPUs and larger memory footprints than P4, and sometimes more than T4 depending on model size. It is worth testing selectively with small models, but I would not assume broad support on P4/T4 without checking the specific NIM/model requirements. (NVIDIA Docs)
    Mixed precision inference / trainingT4: Supported / P4: LimitedMixed precision means using a mix of higher-precision and lower-precision numeric formats in AI workloads so you get better speed and lower memory use without giving up model quality where precision still matters.LowT4 supports Tensor Cores and is much better for FP16/INT8 inference acceleration. P4 lacks Tensor Cores, so mixed-precision benefits are more limited and workload-dependent. (NVIDIA)
    NCCL (multi-GPU collectives)Supported, but topology-dependentNVIDIA Collective Communications Library. It is NVIDIA’s library for fast GPU-to-GPU communication, including both multi-GPU within a server and multi-node across servers. NVIDIA describes it as a topology-aware library of collective communication primitives optimized for NVIDIA GPUs and networking.MediumUseful for experimenting with multi-GPU and possibly multi-node communication patterns. It can work over PCIe and network paths, but the quality of results depends heavily on topology and software stack. This is a practical support judgment rather than a clean per-card matrix in the cited pages. (NVIDIA Docs)

    Is Supported on by both T4/P4
    Single-host NCCL tests across multiple GPUs in one server

    Multi-node NCCL tests over InfiniBand

    GPUDirect RDMA-assisted NCCL when the stack and PCIe topology cooperate


    GPUDirect P2P / peer-to-peerUnclear / topology-dependentGPU peer-to-peer memory access: one NVIDIA GPU can directly access or copy data to another NVIDIA GPU’s memory without staging the transfer through host RAMPeer-to-peer GPU memory access can work in some PCIe topologies, but support and performance vary a lot by motherboard, root complex, ACS/IOMMU behavior, and driver stack. Possibly worth validating experimentally in homelab. (NVIDIA Docs)

    Project Status

    Currently I am in the “waiting for hardware” to arrive stage of the project (mainly due to the switch from CX-3s to CX-4s). So lets take stock of where we are in the project and outline our next steps.

    Current State

    • IB Switch Racked
    • GPUs physically installed in all systems
    • Rough list of supported technologies to install and test
    • Basic installation steps and IB troubleshooting documented

    Next Steps

    • Install all CX-4, ensure drivers are installed/loaded properly, and possibly update CX-4 firmware
    • Install NVIDIA drivers, CUDA, and Container Toolkit on all GPU enabled hardware
    • Install IB cables, and power up the Mellanox IB Switch (and hope its not too loud)
    • Install 1x instance of active Subnet Manager (OpenSM) on one of target hosts
    • Configure IB IPs.
    • Work through listed/supported technologies in the NVIDIA matrix above.

    My goal is to document my progress in future posts.

  • Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Introduction

    In this step-by-steps guide we will replace the out of the box nouveau drivers on RHEL9 with Nvidia Drivers. We will also install the the Nvidia CUDA Toolkit and the Nvidia Container Toolkit.


    GPU and Driver Inspection

    First we need to make sure that our Nvdia GPU is recognized by Red Hat Enterprise Linux 9 (RHEL9).

    lspci -nn | grep -i nvidia
    b6:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [A40] [10de:2235] (rev a1)
    

    Using the command below we can see that we are currently using the non-propietary nouveau driver.

    # lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nouveau
    	Kernel modules: nouveau
    

    Configuring Repositories for the Nvidia Driver Install

    First we need to enable the RHEL9 CodeReady Builder repo. Note we are running these commands as root.

    # subscription-manager repos --enable codeready-builder-for-rhel-9-$(uname -i)-rpms
    

    Next we will need to install and configure the EPEL repo.

    # dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
    

    Now we install the ELRepo project repo – this will provide nvidia-detect which we can utilize later

    # dnf -y  install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm

    Prerequisites for Nvidia Driver Install

    Now we need to install dependencies and build tools.

    # dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc make dkms acpid libglvnd-glx libglvnd-opengl libglvnd-devel pkgconfig
    

    Install Nvidia Drivers

    Install nvidia-detect from the ELRepo project repo.

    # dnf -y install nvidia-detect

    Now install the Nvidia Drivers.

    # dnf -y install $(nvidia-detect)

    Now reboot.


    Confirming Nvidia Driver Installation

    Now lets run the command below one more time.

    [root@gpu ~]# lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [250] Latency Tolerance Reporting
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nvidia
    	Kernel modules: nouveau, nvidia_drm, nvidia
    

    As you can see in the output below, the kernel is loading the Nvidia driver. We can still see nouveau kernel modules listed, but that is fine, as they are not loaded. We can confirm this with the command below.

    # lsmod | grep nouveau

    The above command should not output anything, while the opposite should be true for the command below.

    # lsmod | grep nvidia

    Configure Nvidia Persistenced

    Start and enable nvidia-persistenced.service. This will enable persistence-mode which will keep the nvidia device state from going “stale”

    # systemctl enable nvidia-persistenced.service
    # systemctl start nvidia-persistenced.service
    

    Installing the Nvidia CUDA Toolkit

    We will now follow the official guide and install the Nvidia CUDA toolkit. Per that guide, we need to enable a few repos, however two of those repos should be enabled by default, and the other one we enabled above, however I will list them here for the sake of documentation.

    # subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
    # subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
    # subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

    Now we install the Nvidia repo for the CUDA toolkit.

    dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
    

    Now install the CUDA toolkit as shown below

    # sudo dnf -y install cuda-toolkit

    Confirm that the toolkit is installed and note the version.

    # rpm -qa cuda-toolkit
    cuda-toolkit-12.8.1-1.x86_64
    

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well. Note that the cuda version should match the one that you installed above.

    export PATH=/usr/local/cuda-12.8/bin:$PATH

    Now test nvcc as shown below.

    # nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Feb_21_20:23:50_PST_2025
    Cuda compilation tools, release 12.8, V12.8.93
    Build cuda_12.8.r12.8/compiler.35583870_0
    

    Installing the Nvidia Container Toolkit

    Next we will install the Nvidia Container Toolkit, which allows users to run GPU-accelerated containerized applications.

    A bit about Container Management in RHEL 9

    The default container packages in RHEL 9 are as follows.

    1. Podman – daemonless container image
    2. Buildah – tool for building OCI (Open Container Initiative) container images
    3. Skopeo – tool for managing container images and repos
    4. CRIU – tool to create and save running container checkpoints to disk
    5. Udica – tool for managing SELinux policies for containers

    Installation of the toolkit

    We will follow the instructions as documented here.

    First we configure the repo

    # curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

    Then install via dnf

    # dnf install -y nvidia-container-toolkit

    Configuring the Container Toolkit for Podman

    Generate the CDI specification file using the command below.

    # nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Now lets check the names of the generated device(s).

    # nvidia-ctk cdi list
    INFO[0000] Found 3 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=GPU-7e880be2-891c-72e3-9515-0fd51240e7f4
    nvidia.com/gpu=all
    

    References

    1. https://medium.com/@blackhorseya/step-by-step-guide-to-installing-nvidia-drivers-on-rhel-9-1107e0cd641d
    2. https://access.redhat.com/discussions/227d2101-b4e3-490a-aa1c-601c407ec038
    3. https://darryldias.me/2022/install-nvidia-drivers-on-rhel-9/
    4. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    5. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-rhel-rocky
    6. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html
  • How to Install Instructlab + VLLM with Nvidia Cuda Support on Ubuntu 22.04

    How to Install Instructlab + VLLM with Nvidia Cuda Support on Ubuntu 22.04

    Installing Prerequisites

    We are installing on a very fresh install of Ubuntu 22.04 with an Nvidia 3070ti GPU. Nvidia Drivers were installed at build time. However lets check to see what version of the Nvidia driver we are currently running.

    $ nvidia-detector
    nvidia-driver-570

    While we are at it, let’s make sure that we have nvidia-smi installed which is included in the package shown below

    $ sudo apt install nvidia-utils-570
    

    We will need gcc, which the meta-package below will include (along with make and other dev tools).

    # sudo apt install build-essential -y

    Lets check what version of python we have installed.

    $ python3 --version
    Python 3.10.12
    

    Now install python-venv which is the python module that supports creating virtual environments. Reference the python version shown in the output in the step above

    # sudo apt install python3.10-venv -y

    Installing the Nvidia Cuda Toolkit

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that version is probably too old. Uninstall with “sudo apt purge nvidia-cuda* -y“

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Nvidia states that you need to add the following env vars to your .bashrc, however you will also need to add them to your python virtual env — in venv/bin/activate. These variables will be needed during the install process below.

     export CUDA_HOME=/usr/local/cuda                           
     export PATH=${CUDA_HOME}/bin:${PATH}                       
     export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
    

    Installing Instructlab

    First we make a directory were we want our python virtual env to live.

    $ mkdir instructlab
    $ cd instructlab/

    Now we create the python virtual environment.

    
    $ python3 -m venv --upgrade-deps venv
    $ source venv/bin/activate

    Note that every time you want to use instructlab cli (ilab) post-install you must source the file above. venv/bin/activate

    The install guide found here has you run this command to clear out the pip package cache (for this one package. Not required for first install, but may be on subsequent install runs.

    $ pip cache remove llama_cpp_python

    We then run the 3 commands shown below. Apparently there is a known bug and we will not pick up these required packages as needed, so we need to install them manually, and in the order shown below.

    $ pip install --upgrade pip wheel setuptools
    $ pip install torch
    $ pip install flash-attn

    Now we are ready to install instructlab.

    $ pip install 'instructlab[cuda]' /
       -C cmake.args="-DLLAMA_CUDA=on" /
       -C cmake.args="-DLLAMA_NATIVE=off"

    Lets verify the install

    $ ilab --version
    ilab, version 0.24.0
    

    Installing vLLM

    The instructions have you wrap up the of Instructlab with a pip install of vllm, however this install will fail due as it cannot find nvcc without setting the env variables. So set the following env vars as shown below.

    $ export CUDA_HOME=/usr/local/cuda                           
    $ export PATH=${CUDA_HOME}/bin:${PATH}                       
    $ export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

    The instructions for installing instructlab with Nvidia Cuda support as documented on instructlab.ai currently have you pip install a very specific version of vllm (with cuda support), however I have found that this command will fail (at least on Fedora 40, Ubuntu 22.04, and Ubuntu 24.04) that command is shown below.

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    So this is a method that I have found to work around the issue.

    First run pip install on vllm

    $ pip install vllm

    Confirm vllm installed correctly.

    $ pip show vllm
    Name: vllm
    Version: 0.7.3
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email: 
    License: Apache License
    
    Author-email: 
    License: Apache 2.0
    Location: /home/cpaquin/Workspace/instructlab/venv/lib/python3.10/site-packages
    Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, ninja, numpy, nvidia-ml-py, openai, outlines, pillow, prometheus-fastapi-instrumentator, prometheus_client, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, vllm-flash-attn, xformers
    

    Now remove vllm

    $ pip uninstall vllm

    Now run the original pip install command as documented

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    Now check vllm version, which we can see has cuda support

    $ pip show vllm
    Name: vllm
    Version: 0.5.2.3+cu128
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email: 
    License: Apache 2.0
    Location: /home/cpaquin/Workspace/instructlab/venv/lib/python3.10/site-packages
    Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, ninja, numpy, nvidia-ml-py, openai, outlines, pillow, prometheus-fastapi-instrumentator, prometheus_client, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, vllm-flash-attn, xformers
    Required-by: 
    

    Testing VLLM

    Below is the command that I ran to test that vllm is working and is in fact, leveraging the GPU

    $ vllm serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --dtype float16 --gpu-memory-utilization 0.8

    One up and running you will see output similar to what is shown below.

    INFO:     Started server process [1252947]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
    

    Testing Pytorch Module

    You can create and run the python script below to test if your GPU is identified correctly. This step is not required, but its handy in identifying potential issues with gpu/cuda.

    import torch
    print("CUDA available:", torch.cuda.is_available())
    print("CUDA version:", torch.version.cuda)
    print("Torch version:", torch.__version__)
    print("GPU count:", torch.cuda.device_count())
    print("GPU name:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "None")

    Output below for reference.

    $ python3 ./test.py 
    CUDA available: True
    CUDA version: 12.4
    Torch version: 2.5.1+cu124
    GPU count: 1
    GPU name: NVIDIA GeForce RTX 3070 Ti

    Instructlab Init and Setup

    Now that we have ilab installed we can initialize our environment with the command below. This command will download the Taxonomy repository to our local machine along with a config file. Apparently ilab init has a few default GPUs available to choose from, but since my current GPU is

    Our config file is located in our home directory, if you want to take a look. Note that our GPU, being a Nvidia RTX 3070 TI, will have differing config values than what we selected above (A100), so in a later step we will need to edit our config

    /home/cpaquin/.config/instructlab/config.yaml

    To see the info regarding your workstation, you can run the command below. We will use some of the data below to update our config

     ilab system info
    Platform:
      sys.version: 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0]
      sys.platform: linux
      os.name: posix
      platform.release: 6.8.0-52-generic
      platform.machine: x86_64
      platform.node: scar-Z97-HD3
      platform.python_version: 3.10.12
      os-release.ID: ubuntu
      os-release.VERSION_ID: 22.04
      os-release.PRETTY_NAME: Ubuntu 22.04.5 LTS
      memory.total: 31.30 GB
      memory.available: 27.11 GB
      memory.used: 3.60 GB
    
    InstructLab:
      instructlab.version: 0.24.0
      instructlab-dolomite.version: 0.2.0
      instructlab-eval.version: 0.5.1
      instructlab-quantize.version: 0.1.0
      instructlab-schema.version: 0.4.2
      instructlab-sdg.version: 0.7.0
      instructlab-training.version: 0.7.0
    
    Torch:
      torch.version: 2.3.1+cu121
      torch.backends.cpu.capability: AVX2
      torch.version.cuda: 12.1
      torch.version.hip: None
      torch.cuda.available: True
      torch.backends.cuda.is_built: True
      torch.backends.mps.is_built: False
      torch.backends.mps.is_available: False
      torch.cuda.bf16: True
      torch.cuda.current.device: 0
      torch.cuda.0.name: NVIDIA GeForce RTX 3070 Ti
      torch.cuda.0.free: 7.1 GB
      torch.cuda.0.total: 7.7 GB
      torch.cuda.0.capability: 8.6 (see https://developer.nvidia.com/cuda-gpus#compute)
    
    llama_cpp_python:
      llama_cpp_python.version: 0.3.6
      llama_cpp_python.supports_gpu_offload: False
    

    Editing Generated Config

    Use the command below to edit the generated config.

    $ ilab config edit

    Here we go through the file and ensure that we are setting the number of GPUs correctly. We also modify some additonal settings that have differed from the default (as ilab thinks we now have a $40k GPU). Below is some of what we have configured.

    # Metadata pertaining to the specifics of the system which the Configuration is
    # meant to be applied to.
    metadata:
      # Manufacturer, Family, and SKU of the system CPU, ex: Apple M3 Max
      # Default: None
      cpu_info:
      # Amount of GPUs on the system, ex: 8
      # Default: None
      gpu_count: 1
      # Family of the system GPU, ex: H100
      # Default: None
      gpu_family: RTX 3070 TI
      # Manufacturer of the system GPU, ex: Nvidia
      # Default: None
      gpu_manufacturer: Nvidia
      # Specific SKU related information about the given GPU, ex: PCIe, NVL
      # Default: None
      gpu_sku:
    
    


    Working with Models

    Before we start working with a model, lets take a look at the ilab command parameters. We can see that we have several options such as download, list, test, train, chat, serve, etc.

    Lets download a model with the command below. The command below will download the default model sets in the config file

    $ ilab model download

    output below

    View your downloaded models via the command below

    $ ilab model list

    Now we serve the default model with the command below

    $ ilab model serve

    Open another terminal window and source the env file from the instructlab directory

    $ source venv/bin/activate
    

    Then run the command below to begin an interactive chat session

    $ ilab model chat

    References

    1. https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local
    2. https://github.com/instructlab/instructlab/issues/2422
    3. https://github.com/instructlab/instructlab
    4. https://developers.redhat.com/blog/2024/06/12/getting-started-instructlab-generative-ai-model-tuning#getting_started_with_instructlab
    5. https://www.techtarget.com/searchcio/definition/synthetic-data#:~:text=Synthetic%20data%20is%20increasingly%20used,fresh%20domain%20knowledge%20and%20explainability.
    6. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#meta-packages
    7. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.2/html/building_your_rhel_ai_environment/initializing_instructlab#initialize_ilab

  • Finding and Mapping Jetson OS and JetPack Versions on the Nvidia Jetson

    Finding and Mapping Jetson OS and JetPack Versions on the Nvidia Jetson

    Updated – 7/1/2026

    Below are all the methods that I have found to either find your Jetson OS version, or your Jetpack version (which includes Jetson OS version, Ubuntu version, CUDA Version, NVIDIA drivers, and firmware).

    First let’s review the matrix and see how JetsonOS Maps to JetPack version (along with Ubuntu version, CUDA version, and release date)

    Note:

    • Nano: supported through JetPack 4.6.x
    • Xavier: supported through JetPack 5.1.x
    • Orin: required for JetPack 6.x and newer
    L4TJetPackUbuntuCUDARelease DateRelease TypeDevicesSupport StateProd Ready?
    39.27.224.0413.2.1Jun 2026ProdOrin, Thor/T5000/T4000ActiveOrin: Preferred / Thor: Preferred
    38.47.124.0413.xJan 2026ProdThor/T5000/T4000ActiveThor: Yes / Orin: No
    38.2 / 38.2.1724.0413.xAug 2025ProdThor/T5000ActiveThor: Yes / Orin: No
    36.5.06.2.222.0412.62025ProdOrinMaintenanceOrin: Stable fallback
    36.4.46.2.122.0412.6Jun 2025ProdOrinMaintenanceOrin: Yes
    36.4.36.222.0412.6Early 2025ProdOrinMaintenanceOrin: Yes
    36.4.06.122.0412.6Sep 2024ProdOrinMaintenanceOrin: Acceptable
    36.3.0622.0412.2May 2024ProdOrinSustainingNo
    36.2.06.0 DP22.0412.2Dec 2023DPOrinDPNo
    35.6.45.1.620.0411.8Feb 2026ProdXavier, OrinSustainingXavier: Preferred / Orin: Legacy fallback
    35.6.25.1.520.0411.8Mar 2024ProdXavier, OrinSustainingXavier: Yes
    35.6.15.1.520.0411.8Jan 2024ProdXavier, OrinSustainingXavier: Yes
    35.6.05.1.420.0411.4Dec 2023ProdXavier, OrinSustainingXavier: Acceptable
    35.5.05.1.320.0411.4Sep 2023ProdXavier, OrinSustainingNo
    35.4.15.1.220.0411.4May 2023ProdXavier, OrinSustainingNo
    35.3.15.1.120.0411.4Feb 2023ProdXavier, OrinSustainingNo
    35.2.15.120.0411.4Aug 2022ProdXavier, OrinSustainingNo
    35.1.05.0.220.0411.4Jun 2022ProdXavierSustainingNo
    34.1.15.0.1 DP20.0411.4Apr 2022DPXavierDPNo
    34.1.05.0 DP20.0411.4Mar 2022DPXavierDPNo
    32.7.64.6.618.0410.2Nov 2024ProdNano, TX1, TX2, XavierEOLNano: Final supported
    32.7.54.6.518.0410.2Jun 2024ProdNano, TX1, TX2, XavierEOLNo
    32.7.44.6.418.0410.22024ProdNano, TX1, TX2, XavierEOLNo
    32.7.34.6.318.0410.2Dec 2022ProdNano, TX1, TX2, XavierEOLNo
    32.7.24.6.218.0410.2Aug 2022ProdNano, TX1, TX2, XavierEOLNo
    32.7.14.6.118.0410.2May 2022ProdNano, TX1, TX2, XavierEOLNo
    32.6.14.618.0410.2Nov 2021ProdNano, TX1, TX2, XavierEOLNo
    32.5.24.5.118.0410.2Jul 2021ProdNano, TX1, TX2, XavierEOLNo
    32.5.14.5.118.0410.2Jun 2021ProdNano, TX1, TX2, XavierEOLNo
    32.5.04.518.0410.2Jan 2021ProdNano, TX1, TX2, XavierEOLNo
    32.4.44.4.118.0410.2Oct 2020ProdNano, TX2, XavierEOLNo
    32.4.34.418.0410.2Jul 2020ProdNano, TX2, XavierEOLNo
    32.4.24.4 DP18.0410.2May 2020DPNano, TX2, XavierDPNo
    32.3.14.318.0410Dec 2019ProdNano, TX2, XavierEOLNo
    32.2.34.2.318.0410Sep 2019ProdNano, TX2, XavierEOLNo
    32.2.14.2.218.0410Jul 2019ProdNano, TX2, XavierEOLNo
    32.2.04.2.118.0410Jun 2019ProdNano, TX2, XavierEOLNo
    32.1.04.218.0410Mar 2019ProdNano, TX2, XavierEOLNo
    31.1.04.1.118.0410Nov 2018ProdTX1, TX2EOLNo
    31.0.24.118.0410Oct 2018ProdTX1, TX2EOLNo
    31.0.1418.049Sep 2018ProdTX1, TX2EOLNo
    28.4.03.3.316.049May 2019ProdTX1, TX2EOLNo
    28.2.13.3 / 3.2.116.049Feb 2018ProdTX1, TX2EOLNo
    28.2.03.216.049Dec 2017ProdTX1, TX2EOLNo
    28.1.03.116.048Oct 2017ProdTX1EOLNo
    27.1.0316.048Mar 2017ProdTX1EOLNo
    24.2.13.0 / 2.3.116.047Dec 2016ProdTK1, TX1EOLNo
    24.2.02.316.047Sep 2016ProdTK1, TX1EOLNo
    24.1.02.2 / 2.2.116.047Jun 2016ProdTK1, TX1EOLNo
    23.2.02.116.046.5Mar 2016ProdTK1EOLNo
    23.1.0216.046.5Jan 2016ProdTK1EOLNo
    21.5.02.3 / 2.3.114.046.5Oct 2015ProdTK1EOLNo
    21.4.02.2 / 2.1 / 2.0 / 1.2 DP14.046.5Jun 2015MixedTK1EOLNo
    21.3.01.1 DP14.046Apr 2015DPTK1DPNo
    21.2.01.0 DP14.046Feb 2015DPTK1DPNo

    Jetson Linux (L4T) / JetPack Version Matrix (Unified)

    With Support Status and Recommended Baselines



    Commands to Find your Jetpack/Jetson OS Version

    This is not an extensive list, but its what I have used in the past


    jetsonInfo.py

    Git clone the repo below.

    https://github.com/jetsonhacks/jetsonUtilities

    and then run jetsonInfo.py.

    # ./jetsonInfo.py
    NVIDIA NVIDIA Jetson Orin NX Engineering Reference Developer Kit
    L4T 36.4.3 [ JetPack UNKNOWN ]
    Ubuntu 22.04.5 LTS
    Kernel Version: 5.15.148-tegra
    CUDA NOT_INSTALLED
    CUDA Architecture: 8.7
    OpenCV version: 4.5.4
    OpenCV Cuda: NO
    CUDNN: ii libcudnn9
    TensorRT: NOT_INSTALLED
    Vision Works: NOT_INSTALLED
    VPI: NOT_INSTALLED
    Vulcan: 1.3.204

    L4T 36.4.3 is part of Jetpack 6.2, looks like the repo has not been updated since 2021 which is why Jetpack 6.2 is not recognized (also missing installed TensorRT). See section below for Jetson OS to Jetpack Version mapping

    Per the git-repo

    1. The hardware designator is derived from the file: ‘/proc/cpuinfo’
    2. The L4T version is derived from the file: ‘/etc/nv_tegra_release’
    3. The Ubuntu version is derived from the file: ‘/etc/os-release’
    4. The Linux kernel version is derived from the file: ‘/proc/version’

    Via /etc/nv_tegra_release

    Below in bold you can see the Jetson OS release and revision.

    # cat /etc/nv_tegra_release
    # R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025
    # KERNEL_VARIANT: oot
    TARGET_USERSPACE_LIB_DIR=nvidia
    TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

    Via Apt

    # sudo apt-cache show nvidia-jetpack
    Package: nvidia-jetpack
    Source: nvidia-jetpack (6.2)
    Version: 6.2+b77
    Architecture: arm64
    Maintainer: NVIDIA Corporation
    Installed-Size: 194
    Depends: nvidia-jetpack-runtime (= 6.2+b77), nvidia-jetpack-dev (= 6.2+b77)
    Homepage: http://developer.nvidia.com/jetson
    Priority: standard
    Section: metapackages
    Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.2+b77_arm64.deb
    Size: 29298
    SHA256: 70553d4b5a802057f9436677ef8ce255db386fd3b5d24ff2c0a8ec0e485c59cd
    SHA1: 9deab64d12eef0e788471e05856c84bf2a0cf6e6
    MD5sum: 4db65dc36434fe1f84176843384aee23
    Description: NVIDIA Jetpack Meta Package
    Description-md5: ad1462289bdbc54909ae109d1d32c0a8

    Via nvidia-lt4-core pkg

    # dpkg-query --show nvidia-l4t-core
    nvidia-l4t-core 36.4.3-20250107174145

  • Selecting a GPU for a Dell T620

    Selecting a GPU for a Dell T620

    I just recently posted about installing a GPU Power Supply Expansion Board into the Dell T620. I suggest giving that post a read before you continue, as this will be post builds upon what was discussed previously. This post will be a bit of a deep dive into what you should consider when choosing a GPU for your T620.

    Ok, so first things first, you need to choose a GPU that is compatible with your T620, as not everything is going to work. Choosing a GPU that is a good fit for your workload, well that comes second.

    Let’s start by looking at what Dell tested and certified.

    Officially Supported GPUs for the T620

    ManufacturerModelHeatsinkPower Usage
    NvidiaTesla C2075Active215W
    NvidiaQuadro K4000Active80W
    NvidiaQuadro Q6000Active204W
    NvidiaTesla K20AActive225W
    ATI™ FirePro™V7800Active150W
    AMD FireProW7000Active127.7W

    According to the documentation, only the above GPUs are “supported” on the Dell T620. Note that this really means that these are the only cards that Dell certified/tested. It does not mean that there are not any other compatible cards out there. These GPUs are honestly quite old. The only still relevant use case for any of them in this day and age is video transcoding on the quadro card. However, you might be able to do some lightweight AI/ML work on the Tesla K20A (2688 Cuda Cores, Kepler Architecture)

    Dell documentation further expands on what is supported. Specifically…

    1. Up to four 300W, full-length, single- or double-wide GPU cards
    2. GPUs with up to 6GB of dedicated DDR5 memory
    3. GPUs that are actively cooled.
    4. No more than 2 power connectors per card
    5. Power connectors located on the back of the card
    6. GPU enablement kit (Power & Cooling Required)

    Additionally Dell documentation states the following…

    1. All cards are the same make/model
    2. 1000W power supplies are required (technically for systems with more than 2 cards)
    3. You need not to cram your T620 too full with GPUs and Raid Controllers to avoid overheating

    What About other GPUs?

    So this is where we start to go off the rails, and venture into the unknown. Let’s start with what we know, which is what Dell officially supported and what their documentation says about choosing a GPU

    Is 6GB the VRAM Limit?

    At the time, Dell only were able to test and certify a handful of GPUs, and none of those GPUs had more than 6GB of VRAM. So it’s very possible that this is why Dell documentation tops out at 6GB as supported. I doubt this is an actual real-world limit.

    The T620 was released in 2012 and probably went end-of-life in 2017 – 2019. None of the officially “supported” cards were released after 2013. Available GPUs with more than 6GB VRAM were probably not widely available, and if they were, Dell probably had finished their initial round of GPU certification and moved on to better things. Any GPUs with more than 6GB would probably run hotter and consumed more power, which could be an issue. Additionally there could be conflicts at the firmware/bios level for mapping/managing more than 6GB of VRAM. Who knows? Generically, they probably drew the line at 6GB because “that is what they knew worked”.

    What about Power and Cooling?

    Adding GPUs to a system will increase cooling and power needs. Replacing existing PSUs, with 1000w PSUs (as recommended by Dell) is very simple and actually pretty cheap. I think I picked up mine for less than $30 USD a pair on ebay. Dual 1000w PSUs was the only supported configuration if you wanted to order a T620 with GPU. However, you still needed to get that power to the GPUs themselves.

    This is where the GPU Enablement Kit comes into play. The enablement kit consists of…

    1. Power expansion board
    2. GPU Power Cables
    3. A Fan Gantry (shown below)
    T620 Fan Gantry

    The fan gantry is not easily found, and they are expensive. You will probably spend more on one of these that you did for your T620. They run about $300+ USD on ebay.

    I have found 3 part numbers on line

    1. 0VDY5 – fan
    2. 8G79K – gantry
    3. 2R4DV – Fan + Gantry?

    The fan gantry from a T630 will also work as well, albeit with a couple of slight modifications. As you need to cut a bit of metal, took me about 2 minutes to modify mine with some tin snips.

    The TOP square I modified a slight bit with tin snips. You will need to do the same for a 4 squares

    Note that you will need T620 Fans, as the T630 fan plugs do not line up with the power receiver on the motherboard (hence why you need to modify the gantry)

    T620 Fan on the left. T630 Fan on the right. Notice the offset of the power plug is different between the two server models.

    Note that the Fan Gantry for the T630 is not any cheaper, it’s just a bit easier to find . The part number for the T620 fan is 0TW71C, which is the same fan that are at the rear of the baffle.

    Adding non-Dell proprietary fans to the chassis of the T620 is either much harder or much more expensive, as most of the unused power connectors in the chassis are proprietary. There are also no standard mounting brackets for common fan sizes.

    However, there is power to be had. There is connector coming off the drive backplane that provides SATA power to the CDROM. And in theory you could do this…

    Male Sata–Female Molex == Male Molex–Female Molex == Fan Controller with 2pin Molex

    Now you can add a fan controller and can add some more fans, but you will have to get creative. A few pci slot blower fans? PCI slot Graphics Card Cooler possibly?


    Similar Alternative GPUs with Active Cooling

    Assuming we need to keep to similar power usage and VRAM, let’s look at a few older, yet not as anemic GPUs.

    1. NVIDIA GeForce GTX 1660 Super

    • VRAM: 6GB GDDR6
    • TDP: 125W
    • Cooling: Actively cooled
    • Performance: Great for basic AI/ML tasks, especially if you are focused on inference or lighter training workloads.

    2. NVIDIA RTX A2000 (Professional Card)

    • VRAM: 6GB GDDR6
    • TDP: 70W
    • Cooling: Actively cooled, compact design
    • Performance: A professional GPU tailored for AI/ML workloads and well-suited for power-constrained environments.

    3. NVIDIA T1000

    • VRAM: 4GB GDDR6
    • TDP: 50W
    • Cooling: Actively cooled, compact design
    • Performance: Suitable for lightweight AI/ML applications and inference tasks, ideal if energy efficiency is a priority.

    What about GPUs without External Power

    No power supply expansion board?

    Let’s look at GPUs that do not require external power. Most PCIe 3.0 slots provide only 25 watts of power, however 75W is the maximum power delivery from a single PCIe 3.0 slot. The Dell T620 has 4 such X16 slots. This alone is appealing and is one of the reasons the Dell T-Series are popular with home labbers. By contract, my newer R630 only provides max of 25W per PCIe slot.

    T620 Power Provided Per PCIE Slot

    SLOTPOWERCPU CONNECTION
    Slot 1 (x8 lanes)up to 25WCPU1
    Slot 2 (x16 lanes)up to 75WCPU1
    Slot 3 (x4 lanes)up to 25WPlatform Controller Hub
    Slot 4 (x16 lanes)up to 75WCPU1
    Slot 5 (x16 lanes)up to 75WCPU2
    Slot 6 (x8 lanes)up to 25WCPU2
    Slot 7 (x16 lanes)up to 75WCPU2

    GPUs that run on 75W or less

    Below is a list of possible GPUs that you can run without additional power requirements, other than what select PCIe slots in the T620 will provide.

    NVIDIA GPUs

    1. NVIDIA Tesla T4

    • Performance: Excellent for AI inference and light training.
    • Power: 70W (fits within PCIe 3.0 constraints).
    • VRAM: 16GB GDDR6.
    • Features: Tensor Cores, CUDA support, and FP16/FP32/INT8 capabilities for AI/ML.
    • Notes: Designed for data centers and supports efficient AI workloads.
    • Cooling: Passive (gets super hot)

    2. NVIDIA Quadro T1000 / T2000

    • Performance: Entry-level professional GPUs for AI tasks.
    • Power: 50-75W.
    • VRAM: 4GB-6GB GDDR6.
    • Features: CUDA cores and optimized drivers for compute tasks.
    • Notes: Focused on balanced performance and efficiency.
    • Cooling: Active

    3. NVIDIA GeForce GTX 1650 (Low Profile/Standard)

    • Performance: Suitable for light AI/ML workloads.
    • Power: 75W.
    • VRAM: 4GB GDDR5/GDDR6.
    • Features: CUDA support for training and inference.
    • Notes: Ensure you get a variant without external power connectors.
    • Cooling: Active

    4. NVIDIA GeForce GTX 1050 Ti

    • Performance: Basic AI/ML tasks and experiments.
    • Power: 75W.
    • VRAM: 4GB GDDR5.
    • Features: CUDA cores and basic AI capabilities.
    • Notes: An older but efficient card for small-scale projects.
    • Cooling: Active

    AMD GPUs

    1. AMD Radeon RX 6400

    • Performance: Entry-level GPU for basic AI workloads.
    • Power: ~53W.
    • VRAM: 4GB GDDR6.
    • Features: RDNA 2 architecture with good efficiency.
    • Notes: Limited AI-focused features but sufficient for lightweight tasks.
    • Cooling: Active

    2. AMD Radeon Pro WX 3200

    • Performance: Professional GPU for light compute tasks.
    • Power: 50W.
    • VRAM: 4GB GDDR5.
    • Features: Optimized drivers for professional workloads.
    • Notes: Reliable for basic AI workloads and professional use.
    • Cooling: Active

    Intel GPUs

    1. Intel Arc A310

    • Performance: Basic GPU for lightweight AI and inference.
    • Power: ~75W.
    • VRAM: 4GB GDDR6.
    • Features: Support for AI frameworks like TensorFlow and PyTorch.
    • Notes: Newer entry from Intel with decent AI potential.
    • Cooling: Active

    What about Modern GPUs?

    Let me start out by saying that it’s possible that many present day GPUs might actually operate without issue in the T620. That being said, what you risk is the bios/lifecycle controller throwing an error as it does not like what it sees happening in its slot.

    A bus fatal error was detected on a component at slot 7

    Above is the error that I encountered attempting to run a Telsa T4 card. I attempted with several different 16x slots, and the error moved with each slot that I tried. This error usually resulted in an OS kernel panic. In my case, its possible that the Nvidia T4, being passively cooled, could have been part of the problem, the card was too hot to touch. Although temperature could have had nothing to do with it and the card may just not “work” in the T620 (at least not yet, as I continue to troubleshoot)

    Regarding newer cards, through my research I have found that a number of folks have had good luck with a few different CPUs (See Reference Section Below). Bottom line, you may get lucky.

    Myself, well I am not giving up on getting the Nvidia T4 to run in my T620. I still have a some experimentation to do. Mainly some bios config tweaking and a bit of add on cooling. I also have some ideas regarding the lifecycle controler, but I have yet to test anything.

    Troubleshooting/Stability

    Below are a few ideas that I have regarding troubleshooting, good luck. I will create a new post at some point in the near future regarding my sucess with the Nvidia T4 GPU.

    1. Slot Disablement  – disable unused PCIE slots. PCIE power stability is especially important
      • “Boot Driver Disabled”. Will investigate this setting. This allow the slot to be visible to the OS, but will not be available as a boot device.
    2. BIOS upgrade – make sure you are running the latest bios (2.9.0)
    3. PCI Slot Selection – choose 16x slot closest to a CPU.
      • I have seen that many have success in slot 5
    4. Cooling – Make sure your GPU is not getting too hot.
      • Pull hard disks to improve air flow on the right side of chassis
      • PCI slot cooler
      • PCI slot blower
    5. Disable Dell Lifecycle Controller?
    6. Disable Collect System Inventory On Restart (CSIOR)?

    Conclusion

    Bottom line, these machines were not exactly designed to run with GPUs, and only a handful of GPUs were actually tested and supported on the Dell T620. The list of supported cards is short, and these GPUs are quite old. However, just because its not supported, does not mean that it will not work. It just means it was not tested. Your mileage may vary. I’ve seen a number of post from individuals who are running more modern GPUs the T620 and they do so without issue.

    Dell Documentation

    1. Dell PowerEdge T620 Technical Guide
    2. Dell T620 GPU Card Installation Guidelines and Requirements

    Reference

    1. Is X7C1K compatible with T620
    2. T620’s device manager recognizes GTX780Ti, but programs do not “see” the card?!
    3. T620 gpu issue
    4. PowerEdge T620: 6-pin PCIe power connector not working (GTX 1060 3GB & GTX 680)
    5. T620’s device manager recognizes GTX780Ti, but programs do not “see” the card?!
    6. GTX 1080 not working in T620
    7. Adding GPU for display – Dell T620 and T630 (change from UEFI to BIOS?)
    8. Dell T620 GPU > use kit from T630? (Radeon HD5770. Confirmed T620 kit works in T620)
    9. Adding GPU to Dell T620 (1660 ti and a 750 ti.)
    10. Dell EMC PowerEdge T640: Remove/Install GPU PIB (Pretty much the same as the T620)