• How to Install Instructlab + VLLM with Nvidia Cuda Support on Ubuntu 22.04

    How to Install Instructlab + VLLM with Nvidia Cuda Support on Ubuntu 22.04

    Installing Prerequisites

    We are installing on a very fresh install of Ubuntu 22.04 with an Nvidia 3070ti GPU. Nvidia Drivers were installed at build time. However lets check to see what version of the Nvidia driver we are currently running.

    $ nvidia-detector
    nvidia-driver-570

    While we are at it, let’s make sure that we have nvidia-smi installed which is included in the package shown below

    $ sudo apt install nvidia-utils-570
    

    We will need gcc, which the meta-package below will include (along with make and other dev tools).

    # sudo apt install build-essential -y

    Lets check what version of python we have installed.

    $ python3 --version
    Python 3.10.12
    

    Now install python-venv which is the python module that supports creating virtual environments. Reference the python version shown in the output in the step above

    # sudo apt install python3.10-venv -y

    Installing the Nvidia Cuda Toolkit

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that version is probably too old. Uninstall with “sudo apt purge nvidia-cuda* -y“

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Nvidia states that you need to add the following env vars to your .bashrc, however you will also need to add them to your python virtual env — in venv/bin/activate. These variables will be needed during the install process below.

     export CUDA_HOME=/usr/local/cuda                           
     export PATH=${CUDA_HOME}/bin:${PATH}                       
     export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
    

    Installing Instructlab

    First we make a directory were we want our python virtual env to live.

    $ mkdir instructlab
    $ cd instructlab/

    Now we create the python virtual environment.

    
    $ python3 -m venv --upgrade-deps venv
    $ source venv/bin/activate

    Note that every time you want to use instructlab cli (ilab) post-install you must source the file above. venv/bin/activate

    The install guide found here has you run this command to clear out the pip package cache (for this one package. Not required for first install, but may be on subsequent install runs.

    $ pip cache remove llama_cpp_python

    We then run the 3 commands shown below. Apparently there is a known bug and we will not pick up these required packages as needed, so we need to install them manually, and in the order shown below.

    $ pip install --upgrade pip wheel setuptools
    $ pip install torch
    $ pip install flash-attn

    Now we are ready to install instructlab.

    $ pip install 'instructlab[cuda]' /
       -C cmake.args="-DLLAMA_CUDA=on" /
       -C cmake.args="-DLLAMA_NATIVE=off"

    Lets verify the install

    $ ilab --version
    ilab, version 0.24.0
    

    Installing vLLM

    The instructions have you wrap up the of Instructlab with a pip install of vllm, however this install will fail due as it cannot find nvcc without setting the env variables. So set the following env vars as shown below.

    $ export CUDA_HOME=/usr/local/cuda                           
    $ export PATH=${CUDA_HOME}/bin:${PATH}                       
    $ export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

    The instructions for installing instructlab with Nvidia Cuda support as documented on instructlab.ai currently have you pip install a very specific version of vllm (with cuda support), however I have found that this command will fail (at least on Fedora 40, Ubuntu 22.04, and Ubuntu 24.04) that command is shown below.

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    So this is a method that I have found to work around the issue.

    First run pip install on vllm

    $ pip install vllm

    Confirm vllm installed correctly.

    $ pip show vllm
    Name: vllm
    Version: 0.7.3
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email: 
    License: Apache License
    
    Author-email: 
    License: Apache 2.0
    Location: /home/cpaquin/Workspace/instructlab/venv/lib/python3.10/site-packages
    Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, ninja, numpy, nvidia-ml-py, openai, outlines, pillow, prometheus-fastapi-instrumentator, prometheus_client, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, vllm-flash-attn, xformers
    

    Now remove vllm

    $ pip uninstall vllm

    Now run the original pip install command as documented

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    Now check vllm version, which we can see has cuda support

    $ pip show vllm
    Name: vllm
    Version: 0.5.2.3+cu128
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email: 
    License: Apache 2.0
    Location: /home/cpaquin/Workspace/instructlab/venv/lib/python3.10/site-packages
    Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, ninja, numpy, nvidia-ml-py, openai, outlines, pillow, prometheus-fastapi-instrumentator, prometheus_client, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, vllm-flash-attn, xformers
    Required-by: 
    

    Testing VLLM

    Below is the command that I ran to test that vllm is working and is in fact, leveraging the GPU

    $ vllm serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --dtype float16 --gpu-memory-utilization 0.8

    One up and running you will see output similar to what is shown below.

    INFO:     Started server process [1252947]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
    

    Testing Pytorch Module

    You can create and run the python script below to test if your GPU is identified correctly. This step is not required, but its handy in identifying potential issues with gpu/cuda.

    import torch
    print("CUDA available:", torch.cuda.is_available())
    print("CUDA version:", torch.version.cuda)
    print("Torch version:", torch.__version__)
    print("GPU count:", torch.cuda.device_count())
    print("GPU name:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "None")

    Output below for reference.

    $ python3 ./test.py 
    CUDA available: True
    CUDA version: 12.4
    Torch version: 2.5.1+cu124
    GPU count: 1
    GPU name: NVIDIA GeForce RTX 3070 Ti

    Instructlab Init and Setup

    Now that we have ilab installed we can initialize our environment with the command below. This command will download the Taxonomy repository to our local machine along with a config file. Apparently ilab init has a few default GPUs available to choose from, but since my current GPU is

    Our config file is located in our home directory, if you want to take a look. Note that our GPU, being a Nvidia RTX 3070 TI, will have differing config values than what we selected above (A100), so in a later step we will need to edit our config

    /home/cpaquin/.config/instructlab/config.yaml

    To see the info regarding your workstation, you can run the command below. We will use some of the data below to update our config

     ilab system info
    Platform:
      sys.version: 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0]
      sys.platform: linux
      os.name: posix
      platform.release: 6.8.0-52-generic
      platform.machine: x86_64
      platform.node: scar-Z97-HD3
      platform.python_version: 3.10.12
      os-release.ID: ubuntu
      os-release.VERSION_ID: 22.04
      os-release.PRETTY_NAME: Ubuntu 22.04.5 LTS
      memory.total: 31.30 GB
      memory.available: 27.11 GB
      memory.used: 3.60 GB
    
    InstructLab:
      instructlab.version: 0.24.0
      instructlab-dolomite.version: 0.2.0
      instructlab-eval.version: 0.5.1
      instructlab-quantize.version: 0.1.0
      instructlab-schema.version: 0.4.2
      instructlab-sdg.version: 0.7.0
      instructlab-training.version: 0.7.0
    
    Torch:
      torch.version: 2.3.1+cu121
      torch.backends.cpu.capability: AVX2
      torch.version.cuda: 12.1
      torch.version.hip: None
      torch.cuda.available: True
      torch.backends.cuda.is_built: True
      torch.backends.mps.is_built: False
      torch.backends.mps.is_available: False
      torch.cuda.bf16: True
      torch.cuda.current.device: 0
      torch.cuda.0.name: NVIDIA GeForce RTX 3070 Ti
      torch.cuda.0.free: 7.1 GB
      torch.cuda.0.total: 7.7 GB
      torch.cuda.0.capability: 8.6 (see https://developer.nvidia.com/cuda-gpus#compute)
    
    llama_cpp_python:
      llama_cpp_python.version: 0.3.6
      llama_cpp_python.supports_gpu_offload: False
    

    Editing Generated Config

    Use the command below to edit the generated config.

    $ ilab config edit

    Here we go through the file and ensure that we are setting the number of GPUs correctly. We also modify some additonal settings that have differed from the default (as ilab thinks we now have a $40k GPU). Below is some of what we have configured.

    # Metadata pertaining to the specifics of the system which the Configuration is
    # meant to be applied to.
    metadata:
      # Manufacturer, Family, and SKU of the system CPU, ex: Apple M3 Max
      # Default: None
      cpu_info:
      # Amount of GPUs on the system, ex: 8
      # Default: None
      gpu_count: 1
      # Family of the system GPU, ex: H100
      # Default: None
      gpu_family: RTX 3070 TI
      # Manufacturer of the system GPU, ex: Nvidia
      # Default: None
      gpu_manufacturer: Nvidia
      # Specific SKU related information about the given GPU, ex: PCIe, NVL
      # Default: None
      gpu_sku:
    
    


    Working with Models

    Before we start working with a model, lets take a look at the ilab command parameters. We can see that we have several options such as download, list, test, train, chat, serve, etc.

    Lets download a model with the command below. The command below will download the default model sets in the config file

    $ ilab model download

    output below

    View your downloaded models via the command below

    $ ilab model list

    Now we serve the default model with the command below

    $ ilab model serve

    Open another terminal window and source the env file from the instructlab directory

    $ source venv/bin/activate
    

    Then run the command below to begin an interactive chat session

    $ ilab model chat

    References

    1. https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local
    2. https://github.com/instructlab/instructlab/issues/2422
    3. https://github.com/instructlab/instructlab
    4. https://developers.redhat.com/blog/2024/06/12/getting-started-instructlab-generative-ai-model-tuning#getting_started_with_instructlab
    5. https://www.techtarget.com/searchcio/definition/synthetic-data#:~:text=Synthetic%20data%20is%20increasingly%20used,fresh%20domain%20knowledge%20and%20explainability.
    6. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#meta-packages
    7. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.2/html/building_your_rhel_ai_environment/initializing_instructlab#initialize_ilab

  • Getting Started with Instructlab: Installing on Fedora 40

    Getting Started with Instructlab: Installing on Fedora 40

    Prerequisite – Install python 3.11. The default on Fedora 40 is python 3.12

    $ sudo dnf -y install python3.11.x86_64 python3.11-devel.x86_64  -y

    Install automake and gcc (GNU Compiler Collection)

    $ sudo dnf install make automake gcc gcc-c++
    

    Add the following env vars to your .bashrc, and source

    export CUDA_HOME=/usr/local/cuda
    export PATH=${CUDA_HOME}/bin:${PATH}
    export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
    

    Create your virtual env and activate

    $ python3.11 -m venv --upgrade-deps venv
    $ source venv/bin/activate
    

    Note that every time you want to use instructlab cli (ilab) post-install you must source the file above. venv/bin/activate

    We then run the 3 commands shown below. Apparently there is a known bug and we will not pick up these required packages as needed, so we need to install them manually.

    $ pip install --upgrade pip wheel setuptools
    $ pip install torch
    $ pip install flash-attn

    The install guide found here has you run this command to clear out the pip package cache (for this one package. Not required for first install, but may be on subsequent runs.

    $ pip cache remove llama_cpp_python

    Now install instructlab

    $ pip install 'instructlab[cuda]' -C cmake.args="-DLLAMA_CUDA=on" -C cmake.args="-DLLAMA_NATIVE=off" 

    Install vllm.

    NOTE: The vllm install is currently failing with this error – ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory. I am looking to find a resolution or work-around

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    Resources

    1. https://docs.instructlab.ai/getting-started/linux_nvidia/
    2. https://github.com/vllm-project/vllm
    3. https://www.redhat.com/en/topics/ai/what-is-vllm
    4. https://github.com/opendatahub-io

  • Drivers and CUDA Toolkit Setup for Nvidia on Fedora 40

    Drivers and CUDA Toolkit Setup for Nvidia on Fedora 40

    First determine what driver is installed. Below we can see that we are running the “nouveau” driver and not the Nvidia proprietary driver.

    $ sudo lspci -n -n -k | grep -A 2 -e VGA -e 3D
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
    	Subsystem: PNY Device [196e:138f]
    	Kernel driver in use: nouveau
    

    Add rpm fusion repos.

    $ sudo dnf install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

    Dnf update.

    $ sudo dnf update -y

    Install kernel-devel which provides kernel headers and makefiles required to build kernel modules.

    $ sudo dnf -y install kernel-devel

    Install akmod-nividia, which checks if there are any missing kernel mods at boot time and will if so, will build them.

    $ sudo dnf -y install akmod-nvidia 

    Now install the CUDA driver for xorg-x11-drv-nvidia.

    $ sudo dnf -y install xorg-x11-drv-nvidia-cuda

    Enable/start Nvidia nvidia-persistenced.service.

    sudo systemctl enable --now nvidia-persistenced.service

    Reboot.

    Now we check to make sure that the driver is loaded.

    $ sudo lspci -n -n -k | grep -A 2 -e VGA -e 3D
    [sudo] password for cpaquin: 
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
    	Subsystem: PNY Device [196e:138f]
    	Kernel driver in use: nvidia
    

    Run “nvidia-smi” to show the exact driver revision. Below we have 565.77. And this driver is compatible with CUDA Version: 12.7.


    Nvidia Cuda Toolkit Install

    We now need to install the nvidia-cuda-toolkit. We will get this directly from nvidia. According to the driver/toolkit version matrix found here, we should install Cuda Toolkit 12.6.

    First install the following pre-req. Nvcc will not compile on gcc14 which is default for Fedora 40.

    # sudo dnf install gcc13-c++ -y

    Pull down the installer

    wget https://developer.download.nvidia.com/compute/cuda/12.6.2/local_installers/cuda_12.6.2_560.35.03_linux.run
    

    Make the file executable

    $ sudo chmod +x cuda_12.6.2_560.35.03_linux.run

    Now install. Since we already have drivers installed, make sure that you uncheck all install options except for the toolkit itself.

    sudo ./cuda_12.6.2_560.35.03_linux.run

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well.

    export PATH=/usr/local/cuda-12.6/bin:$PATH

    Next add “/usr/local/cuda-12.6/lib64” to “/etc/ld.so.conf” and run “ldconfig” as root. See example below

    # cat /etc/ld.so.conf
    include ld.so.conf.d/*.conf
    /usr/local/cuda-12.6/lib64

    Now run ldconfig

    # ldconfig

    Now test nvcc as shown below.

    $ nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2024 NVIDIA Corporation
    Built on Thu_Sep_12_02:18:05_PDT_2024
    Cuda compilation tools, release 12.6, V12.6.77
    Build cuda_12.6.r12.6/compiler.34841621_0
    

    Resources

    1. https://www.reddit.com/r/Fedora/comments/1gpp5a4/how_to_install_nvcc_on_fedora_41/
    2. https://rpmfusion.org/Howto/NVIDIA#Akmods
    3. https://packages.fedoraproject.org/pkgs/kernel/kernel-devel/
    4. https://rpmfusion.org/Configuration
    5. https://forums.developer.nvidia.com/t/cuda-toolkit-support-for-fedora-40/308193/2
    6. https://forums.developer.nvidia.com/t/cuda-toolkit-installation-problems-on-fedora-41-forty-one/317186/2
  • Homelab: Shushing a loud Dell Server.

    Homelab: Shushing a loud Dell Server.

    Rack mount servers are notoriously loud. They are designed to run in data centers which and not next to your head. In a data center no one really notices or cares if a machine’s fans are spinning faster than they need to. 2RU Servers are bad, but 1U are even worse as the smaller the fan the faster they need to spin to move air. Start plugging certified PCI cards into slots and things only get worse. This is exactly my experience with 2 nvidia Tesla T4s in my “used-to-be” quiet Dell T620 (Idrac 7)

    Note: Passing fan overrides to the idrac does not survive a reboot/restart and fans won’t spint up if temps rise. See https://github.com/christopherpaquin/Dell-Server-Fan-Shusher for a better solution (GPU aware).

    So lets learn to control the noise a bit.

    First off you need to enable IPMI over Idrac. In Idrac7 (and Idrac8), log into the web interface, and navigate to IDRAC Settings > Network. Scroll down to IPMI Settings, and make sure that “Enable IPMI Over LAN” is checked. This will allow you to ssh to the idrac and run IPMI commands.

    Installing IPMI tool

    You will first need to install ipmitool on your workstation.

    $ apt install ipmitool -y

    Manual Fan Control

    Run the command below to turn on manual control over the fans. You need to do this before you can change their speeds.

    ipmitool -I lanplus -H $IP -U $USER -P $PASS raw 0x30 0x30 0x01 0x00
    

    If you want to turn off manual control, and return to auto control, you can run the command below

    ipmitool -I lanplus -H $IP -U $USER -P $PASS raw 0x30 0x30 0x01 0x01
    

    Finding your Fans

    Run the command below to find our how may system fans you have as well as their numbering

    $ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin sdr list | grep Fan
    Fan1             | 1440 RPM          | ok
    Fan2             | 1200 RPM          | ok
    Fan3             | disabled          | ns
    Fan4             | disabled          | ns
    Fan5             | disabled          | ns
    Fan6             | disabled          | ns
    Fan Redundancy   | Not Readable      | ns
    

    Run the command below to set all fans at 40% speed.

    1. 0xff is all fans,
    2. 0x26 is approx 40% (scale is from 0 to 64)
    $ ipmitool -I lanplus -H $IP -U $USER -P $PASS raw 0x30 0x30 0x02 0xff 0x26

    Check the results.

    $ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin sdr list | egrep -i fan[1,2]
    Fan1             | 2400 RPM          | ok
    Fan2             | 2040 RPM          | ok
    

    You can also target individual fans. In the example below we will increase fan1 to 100%. Where…

    1. 0x02 is fan1
    2. 0x64 is 100%
    $  ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin raw 0x30 0x30 0x02 0x01 0x64

    Lets do the same for fan1 (0x00 is fan1)

    $ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin raw 0x30 0x30 0x02 0x00 0x64

    Now lets bring both fans back to about 40%

    The one-liner below will show you temps as you want to make sure nothing gets too hot.

    $ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin sdr list | grep Temp
    Inlet Temp       | 20 degrees C      | ok
    Temp             | 30 degrees C      | ok
    Temp             | 25 degrees C      | ok
    

    Additional Tools

    Here are a few tools that I have found in my travels

    1. Dell IDRAC Fan Controller Docker – is a docker container that allows you to control fan speeds. Have not tried it but looks pretty simple to use.
    2. R710 Fan Control
    3. Fan Control and Monitoring Script
    4. IDRAC Fan Control
    5. Server Manager

  • Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 24.04

    Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 24.04

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that we do not want conflicts. Uninstall with “sudo apt purge nvidia-cuda* -y“. More info on ubuntu repo install vs nvidia repo install later in post.

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2404-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2404-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2404-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Update $Path

    Add the the following to your path

    /usr/local/cuda-12.8/bin

    Installing the Nvidia Cuda toolkit from Ubuntu Vs Nvidia

    The key differences between installing the NVIDIA CUDA Toolkit from the Ubuntu repositories vs. NVIDIA’s official website come down to version availability, ease of installation, and compatibility. Here’s are a few differences:

    1. Installing from Ubuntu Repositories (apt install nvidia-cuda-toolkit)

    Pros:

    • Easier installation – The package is well-integrated into Ubuntu’s package management (apt), meaning dependency resolution is generally smoother.
    • More stable – The versions in the repositories are tested against the Ubuntu ecosystem to ensure compatibility.
    • Automatic updates – Managed through apt, making it easier to keep up with system updates.

    Cons:

    • Often outdated – Ubuntu’s repositories may not have the latest CUDA version, which can be an issue if you need newer features or performance improvements.
    • Limited driver version control – The version of CUDA available in the Ubuntu repos may not always match the latest NVIDIA drivers, which can lead to compatibility issues.

    2. Installing from NVIDIA’s Official Website

    Pros:

    • Latest version – You get the newest CUDA Toolkit, which can include performance improvements, bug fixes, and new features.
    • Better control over versions – You can install specific versions to match requirements for AI/ML frameworks like TensorFlow or PyTorch.
    • More driver compatibility – NVIDIA ensures that the toolkit and drivers are aligned, reducing the risk of version mismatches.

    Cons:

    • More manual setup – You have to install dependencies manually, such as the correct NVIDIA driver, and configure paths properly.
    • Updates are not automatic – Unlike apt, updates need to be managed manually.
  • Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 22.04

    Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 22.04

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that version is probably too old. Uninstall with “sudo apt purge nvidia-cuda* -y“

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Nvidia states that you need to add the following env vars to your .bashrc, however you will also need to add them to your python virtual env — in venv/bin/activate. These variables will be needed during the install process below.

     export CUDA_HOME=/usr/local/cuda                           
    export PATH=${CUDA_HOME}/bin:${PATH}
    export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

    Update $PATH

    The the following to your path

    /usr/local/cuda-12.8/bin

    Reference

    1. https://developer.nvidia.com/cuda-downloads
  • Getting Started with RamaLama with Nvidia Cuda Support On Ubuntu 24.04

    Getting Started with RamaLama with Nvidia Cuda Support On Ubuntu 24.04

    Introduction to RamaLama

    Streamlining AI Deployment with OCI Containers

    RamaLama is an open-source project developed to simplify AI model deployment and management using OCI (Open Container Initiative) containers. Ramalama enables seamless execution of AI workloads across different hardware configurations, supporting both GPU-accelerated and CPU-based environments.

    By leveraging container engines like Podman and Docker, RamaLama includes all necessary dependencies, eliminating complex installation and dependency nightmares.

    Ramalama integrates with AI model registries such as Hugging Face and Ollama, providing flexibility in model selection. Key features include automatic GPU detection, CPU fallback, and optional direct execution on the host system.

    Prerequisites

    Updating Ubuntu

    First let’s confirm our Ubuntu version

    $ sudo lsb_release -a
    No LSB modules are available.
    Distributor ID:	Ubuntu
    Description:	Ubuntu 22.04.5 LTS
    Release:	22.04
    Codename:	jammy
    

    Run the two commands below to update your package cache and install any updates. Reboot if required.

    $ sudo apt-get update
    $ sudo apt-get upgrade -y

    Installing podman

    $ sudo apt -y install podman
    # sudo podman --version
    podman version 5.0.3

    Installing Nvidia Drivers

    Assuming you added third-party repos at build time, we should be able to check the suggested NVIDIA driver version. Do so with the command below

    $ nvidia-detector
    nvidia-driver-545

    The command below confirms that we do not have an NVIDIA driver loaded.

    $ cat /proc/driver/nvidia/version
    cat: /proc/driver/nvidia/version: No such file or directory

    Now let’s install the NVIDIA driver.

    $ sudo ubuntu-drivers --gpgpu install

    And we need to install the nvidia-utils package. Make sure the package version matches your installed driver.

    $ sudo apt install nvidia-utils-535-server

    Now reboot.

    Once your system is back up. Run the command below to verify that the drivers installed correctly. At the top of the output you should see your driver version and CUDA API version

    $ sudo nvidia-smi

    You must configure the persistence daemon (nvidia-persistenced) to start at boot and run continuously. Otherwise, the driver may unload, causing the Tesla GPUs to deinitialize, requiring a full reinitialization when nvidia-smi is executed. Additionally, failing to keep nvidia-persistenced running could lead to more severe issues, such as GPU crashes, depending on the workload.

    Enable and start the service below

    $ sudo systemctl start nvidia-persistenced
    $ sudo systemctl status nvidia-persistenced
    

    Installing Nvidia Cuda Toolkit

    Run the command below to install the nvidia-cuda-toolkit from the default Ubuntu repos.

    $ sudo apt install nvidia-cuda-toolkit -y

    Now test to ensure proper install and that new binary files are in your path.

    ~# nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2021 NVIDIA Corporation
    Built on Thu_Nov_18_09:45:30_PST_2021
    Cuda compilation tools, release 11.5, V11.5.119
    Build cuda_11.5.r11.5/compiler.30672275_0
    

    Configuring Podman with Nvidia Cuda Support

    Following along with this document from the RamaLama github page, we first need to install the nvidia-container-toolkit.

    First configure the repo. Note that this is one command. See here for more info.

    $ sudo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
        sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

    Refresh the repos

    # sudo apt-get update

    Then install the toolkit

    $ sudo apt install nvidia-container-toolkit -y

    Then run the command below to create the CDI spec file

    $ sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Now lets check the detected devices

    $ nvidia-ctk cdi list

    My two Telsa T4s have been detected

    INFO[0000] Found 5 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=1
    nvidia.com/gpu=GPU-1d877ac8-5df1-34b0-4f86-59945e37d2ba
    nvidia.com/gpu=GPU-9491a3e6-ea29-ba4e-4403-083244d5575c
    nvidia.com/gpu=all
    

    Test the install/config

    $ sudo podman run --rm --device=nvidia.com/gpu=all fedora nvidia-smi

    Installing Python pip

    Lets first make sure that python 3 is installed

    $ python3 --version
    Python 3.12.7

    Now we need to install the virtual-env module for python

    $ sudo apt install python3.10-venv

    Installing RamaLama via Pip in a Python Virt Env

    Create a directory for ramalama, and cd to that directory

    $ mkdir ramalama && cd ramalama

    Create virtual env and source to activate

    $ python3 -m venv --upgrade-deps venv
    $ source venv/bin/activate

    Now pip install

    $ pip install ramalama

    Now run a model as a test

    $ ramalama run instructlab/merlinite-7b-lab

    In another window, run the commands shown below to view the ramalama container running in your python virtual env. Note that the output of podman ps will be empty unless your “activate” your virtual env.

    $ source venv/bin/activate
    $ podman ps

    Screenshot below


    Example CLI Commands

    Pull a model.

    $ ramalama pull ollama://mistral

    List downloaded models.

    $ ramalama list
    NAME                             MODIFIED       SIZE   
    ollama://mistral:latest          42 seconds ago 3.83 GB
    ollama://merlinite-7b-lab:latest 10 hours ago   4.07 GB
    

    Run a model

    $ ramalama run mistral:latest

    Info on ramalama itself. Output will tell you detected GPUs and what driver is being used.

    $ ramalama info

    –dryrun flag provides the podman command used to serve/run a model

    $  ramalama --dryrun run instructlab/merlinite-7b-lab 
    podman run --rm -i --label ai.ramalama --name ramalama_6buqEjuCUm --env=HOME=/tmp --init --security-opt=label=disable --cap-drop=all --security-opt=no-new-privileges --label ai.ramalama.model=instructlab/merlinite-7b-lab --label ai.ramalama.engine=podman --label ai.ramalama.runtime=llama.cpp --label ai.ramalama.command=run --pull=newer -t --device /dev/dri --device nvidia.com/gpu=all -e CUDA_VISIBLE_DEVICES=0 --network none --mount=type=bind,src=/home/cpaquin/.local/share/ramalama/models/ollama/merlinite-7b-lab:latest,destination=/mnt/models/model.file,ro quay.io/ramalama/cuda:latest llama-run -c 2048 --temp 0.8 --ngl 999 /mnt/models/model.file 
    
    

    There are a whole load of other topics that I will eventually get into with ramalama

    1. GPU Support/Enablement
    2. RAG
    3. Whisper

    More to come at a later date. In the meantime, take a look at the “Resources” section below.

    Additional Video

    Resources

    1. https://github.com/containers/ramalama
    2. https://github.com/containers/ramalama/blob/main/docs/ramalama.1.md
    3. https://developers.redhat.com/articles/2024/11/22/how-ramalama-makes-working-ai-models-boring
    4. https://developers.redhat.com/blog/2024/12/17/simplifying-ai-ramalama-and-llama-run
    5. https://www.linkedin.com/pulse/ollama-much-try-ramalama-surya-rekha-tw5kf/