Tag: nvidia

  • How to Install Instructlab + VLLM with Nvidia Cuda Support on Ubuntu 22.04

    How to Install Instructlab + VLLM with Nvidia Cuda Support on Ubuntu 22.04

    Installing Prerequisites

    We are installing on a very fresh install of Ubuntu 22.04 with an Nvidia 3070ti GPU. Nvidia Drivers were installed at build time. However lets check to see what version of the Nvidia driver we are currently running.

    $ nvidia-detector
    nvidia-driver-570

    While we are at it, let’s make sure that we have nvidia-smi installed which is included in the package shown below

    $ sudo apt install nvidia-utils-570
    

    We will need gcc, which the meta-package below will include (along with make and other dev tools).

    # sudo apt install build-essential -y

    Lets check what version of python we have installed.

    $ python3 --version
    Python 3.10.12
    

    Now install python-venv which is the python module that supports creating virtual environments. Reference the python version shown in the output in the step above

    # sudo apt install python3.10-venv -y

    Installing the Nvidia Cuda Toolkit

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that version is probably too old. Uninstall with “sudo apt purge nvidia-cuda* -y“

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Nvidia states that you need to add the following env vars to your .bashrc, however you will also need to add them to your python virtual env — in venv/bin/activate. These variables will be needed during the install process below.

     export CUDA_HOME=/usr/local/cuda                           
     export PATH=${CUDA_HOME}/bin:${PATH}                       
     export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
    

    Installing Instructlab

    First we make a directory were we want our python virtual env to live.

    $ mkdir instructlab
    $ cd instructlab/

    Now we create the python virtual environment.

    
    $ python3 -m venv --upgrade-deps venv
    $ source venv/bin/activate

    Note that every time you want to use instructlab cli (ilab) post-install you must source the file above. venv/bin/activate

    The install guide found here has you run this command to clear out the pip package cache (for this one package. Not required for first install, but may be on subsequent install runs.

    $ pip cache remove llama_cpp_python

    We then run the 3 commands shown below. Apparently there is a known bug and we will not pick up these required packages as needed, so we need to install them manually, and in the order shown below.

    $ pip install --upgrade pip wheel setuptools
    $ pip install torch
    $ pip install flash-attn

    Now we are ready to install instructlab.

    $ pip install 'instructlab[cuda]' /
       -C cmake.args="-DLLAMA_CUDA=on" /
       -C cmake.args="-DLLAMA_NATIVE=off"

    Lets verify the install

    $ ilab --version
    ilab, version 0.24.0
    

    Installing vLLM

    The instructions have you wrap up the of Instructlab with a pip install of vllm, however this install will fail due as it cannot find nvcc without setting the env variables. So set the following env vars as shown below.

    $ export CUDA_HOME=/usr/local/cuda                           
    $ export PATH=${CUDA_HOME}/bin:${PATH}                       
    $ export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

    The instructions for installing instructlab with Nvidia Cuda support as documented on instructlab.ai currently have you pip install a very specific version of vllm (with cuda support), however I have found that this command will fail (at least on Fedora 40, Ubuntu 22.04, and Ubuntu 24.04) that command is shown below.

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    So this is a method that I have found to work around the issue.

    First run pip install on vllm

    $ pip install vllm

    Confirm vllm installed correctly.

    $ pip show vllm
    Name: vllm
    Version: 0.7.3
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email: 
    License: Apache License
    
    Author-email: 
    License: Apache 2.0
    Location: /home/cpaquin/Workspace/instructlab/venv/lib/python3.10/site-packages
    Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, ninja, numpy, nvidia-ml-py, openai, outlines, pillow, prometheus-fastapi-instrumentator, prometheus_client, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, vllm-flash-attn, xformers
    

    Now remove vllm

    $ pip uninstall vllm

    Now run the original pip install command as documented

    $ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

    Now check vllm version, which we can see has cuda support

    $ pip show vllm
    Name: vllm
    Version: 0.5.2.3+cu128
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email: 
    License: Apache 2.0
    Location: /home/cpaquin/Workspace/instructlab/venv/lib/python3.10/site-packages
    Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, ninja, numpy, nvidia-ml-py, openai, outlines, pillow, prometheus-fastapi-instrumentator, prometheus_client, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch, torchvision, tqdm, transformers, typing_extensions, uvicorn, vllm-flash-attn, xformers
    Required-by: 
    

    Testing VLLM

    Below is the command that I ran to test that vllm is working and is in fact, leveraging the GPU

    $ vllm serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --dtype float16 --gpu-memory-utilization 0.8

    One up and running you will see output similar to what is shown below.

    INFO:     Started server process [1252947]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
    

    Testing Pytorch Module

    You can create and run the python script below to test if your GPU is identified correctly. This step is not required, but its handy in identifying potential issues with gpu/cuda.

    import torch
    print("CUDA available:", torch.cuda.is_available())
    print("CUDA version:", torch.version.cuda)
    print("Torch version:", torch.__version__)
    print("GPU count:", torch.cuda.device_count())
    print("GPU name:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "None")

    Output below for reference.

    $ python3 ./test.py 
    CUDA available: True
    CUDA version: 12.4
    Torch version: 2.5.1+cu124
    GPU count: 1
    GPU name: NVIDIA GeForce RTX 3070 Ti

    Instructlab Init and Setup

    Now that we have ilab installed we can initialize our environment with the command below. This command will download the Taxonomy repository to our local machine along with a config file. Apparently ilab init has a few default GPUs available to choose from, but since my current GPU is

    Our config file is located in our home directory, if you want to take a look. Note that our GPU, being a Nvidia RTX 3070 TI, will have differing config values than what we selected above (A100), so in a later step we will need to edit our config

    /home/cpaquin/.config/instructlab/config.yaml

    To see the info regarding your workstation, you can run the command below. We will use some of the data below to update our config

     ilab system info
    Platform:
      sys.version: 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0]
      sys.platform: linux
      os.name: posix
      platform.release: 6.8.0-52-generic
      platform.machine: x86_64
      platform.node: scar-Z97-HD3
      platform.python_version: 3.10.12
      os-release.ID: ubuntu
      os-release.VERSION_ID: 22.04
      os-release.PRETTY_NAME: Ubuntu 22.04.5 LTS
      memory.total: 31.30 GB
      memory.available: 27.11 GB
      memory.used: 3.60 GB
    
    InstructLab:
      instructlab.version: 0.24.0
      instructlab-dolomite.version: 0.2.0
      instructlab-eval.version: 0.5.1
      instructlab-quantize.version: 0.1.0
      instructlab-schema.version: 0.4.2
      instructlab-sdg.version: 0.7.0
      instructlab-training.version: 0.7.0
    
    Torch:
      torch.version: 2.3.1+cu121
      torch.backends.cpu.capability: AVX2
      torch.version.cuda: 12.1
      torch.version.hip: None
      torch.cuda.available: True
      torch.backends.cuda.is_built: True
      torch.backends.mps.is_built: False
      torch.backends.mps.is_available: False
      torch.cuda.bf16: True
      torch.cuda.current.device: 0
      torch.cuda.0.name: NVIDIA GeForce RTX 3070 Ti
      torch.cuda.0.free: 7.1 GB
      torch.cuda.0.total: 7.7 GB
      torch.cuda.0.capability: 8.6 (see https://developer.nvidia.com/cuda-gpus#compute)
    
    llama_cpp_python:
      llama_cpp_python.version: 0.3.6
      llama_cpp_python.supports_gpu_offload: False
    

    Editing Generated Config

    Use the command below to edit the generated config.

    $ ilab config edit

    Here we go through the file and ensure that we are setting the number of GPUs correctly. We also modify some additonal settings that have differed from the default (as ilab thinks we now have a $40k GPU). Below is some of what we have configured.

    # Metadata pertaining to the specifics of the system which the Configuration is
    # meant to be applied to.
    metadata:
      # Manufacturer, Family, and SKU of the system CPU, ex: Apple M3 Max
      # Default: None
      cpu_info:
      # Amount of GPUs on the system, ex: 8
      # Default: None
      gpu_count: 1
      # Family of the system GPU, ex: H100
      # Default: None
      gpu_family: RTX 3070 TI
      # Manufacturer of the system GPU, ex: Nvidia
      # Default: None
      gpu_manufacturer: Nvidia
      # Specific SKU related information about the given GPU, ex: PCIe, NVL
      # Default: None
      gpu_sku:
    
    


    Working with Models

    Before we start working with a model, lets take a look at the ilab command parameters. We can see that we have several options such as download, list, test, train, chat, serve, etc.

    Lets download a model with the command below. The command below will download the default model sets in the config file

    $ ilab model download

    output below

    View your downloaded models via the command below

    $ ilab model list

    Now we serve the default model with the command below

    $ ilab model serve

    Open another terminal window and source the env file from the instructlab directory

    $ source venv/bin/activate
    

    Then run the command below to begin an interactive chat session

    $ ilab model chat

    References

    1. https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local
    2. https://github.com/instructlab/instructlab/issues/2422
    3. https://github.com/instructlab/instructlab
    4. https://developers.redhat.com/blog/2024/06/12/getting-started-instructlab-generative-ai-model-tuning#getting_started_with_instructlab
    5. https://www.techtarget.com/searchcio/definition/synthetic-data#:~:text=Synthetic%20data%20is%20increasingly%20used,fresh%20domain%20knowledge%20and%20explainability.
    6. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#meta-packages
    7. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.2/html/building_your_rhel_ai_environment/initializing_instructlab#initialize_ilab

  • Drivers and CUDA Toolkit Setup for Nvidia on Fedora 40

    Drivers and CUDA Toolkit Setup for Nvidia on Fedora 40

    First determine what driver is installed. Below we can see that we are running the “nouveau” driver and not the Nvidia proprietary driver.

    $ sudo lspci -n -n -k | grep -A 2 -e VGA -e 3D
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
    	Subsystem: PNY Device [196e:138f]
    	Kernel driver in use: nouveau
    

    Add rpm fusion repos.

    $ sudo dnf install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

    Dnf update.

    $ sudo dnf update -y

    Install kernel-devel which provides kernel headers and makefiles required to build kernel modules.

    $ sudo dnf -y install kernel-devel

    Install akmod-nividia, which checks if there are any missing kernel mods at boot time and will if so, will build them.

    $ sudo dnf -y install akmod-nvidia 

    Now install the CUDA driver for xorg-x11-drv-nvidia.

    $ sudo dnf -y install xorg-x11-drv-nvidia-cuda

    Enable/start Nvidia nvidia-persistenced.service.

    sudo systemctl enable --now nvidia-persistenced.service

    Reboot.

    Now we check to make sure that the driver is loaded.

    $ sudo lspci -n -n -k | grep -A 2 -e VGA -e 3D
    [sudo] password for cpaquin: 
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1)
    	Subsystem: PNY Device [196e:138f]
    	Kernel driver in use: nvidia
    

    Run “nvidia-smi” to show the exact driver revision. Below we have 565.77. And this driver is compatible with CUDA Version: 12.7.


    Nvidia Cuda Toolkit Install

    We now need to install the nvidia-cuda-toolkit. We will get this directly from nvidia. According to the driver/toolkit version matrix found here, we should install Cuda Toolkit 12.6.

    First install the following pre-req. Nvcc will not compile on gcc14 which is default for Fedora 40.

    # sudo dnf install gcc13-c++ -y

    Pull down the installer

    wget https://developer.download.nvidia.com/compute/cuda/12.6.2/local_installers/cuda_12.6.2_560.35.03_linux.run
    

    Make the file executable

    $ sudo chmod +x cuda_12.6.2_560.35.03_linux.run

    Now install. Since we already have drivers installed, make sure that you uncheck all install options except for the toolkit itself.

    sudo ./cuda_12.6.2_560.35.03_linux.run

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well.

    export PATH=/usr/local/cuda-12.6/bin:$PATH

    Next add “/usr/local/cuda-12.6/lib64” to “/etc/ld.so.conf” and run “ldconfig” as root. See example below

    # cat /etc/ld.so.conf
    include ld.so.conf.d/*.conf
    /usr/local/cuda-12.6/lib64

    Now run ldconfig

    # ldconfig

    Now test nvcc as shown below.

    $ nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2024 NVIDIA Corporation
    Built on Thu_Sep_12_02:18:05_PDT_2024
    Cuda compilation tools, release 12.6, V12.6.77
    Build cuda_12.6.r12.6/compiler.34841621_0
    

    Resources

    1. https://www.reddit.com/r/Fedora/comments/1gpp5a4/how_to_install_nvcc_on_fedora_41/
    2. https://rpmfusion.org/Howto/NVIDIA#Akmods
    3. https://packages.fedoraproject.org/pkgs/kernel/kernel-devel/
    4. https://rpmfusion.org/Configuration
    5. https://forums.developer.nvidia.com/t/cuda-toolkit-support-for-fedora-40/308193/2
    6. https://forums.developer.nvidia.com/t/cuda-toolkit-installation-problems-on-fedora-41-forty-one/317186/2
  • Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 24.04

    Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 24.04

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that we do not want conflicts. Uninstall with “sudo apt purge nvidia-cuda* -y“. More info on ubuntu repo install vs nvidia repo install later in post.

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2404-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2404-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2404-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Update $Path

    Add the the following to your path

    /usr/local/cuda-12.8/bin

    Installing the Nvidia Cuda toolkit from Ubuntu Vs Nvidia

    The key differences between installing the NVIDIA CUDA Toolkit from the Ubuntu repositories vs. NVIDIA’s official website come down to version availability, ease of installation, and compatibility. Here’s are a few differences:

    1. Installing from Ubuntu Repositories (apt install nvidia-cuda-toolkit)

    Pros:

    • Easier installation – The package is well-integrated into Ubuntu’s package management (apt), meaning dependency resolution is generally smoother.
    • More stable – The versions in the repositories are tested against the Ubuntu ecosystem to ensure compatibility.
    • Automatic updates – Managed through apt, making it easier to keep up with system updates.

    Cons:

    • Often outdated – Ubuntu’s repositories may not have the latest CUDA version, which can be an issue if you need newer features or performance improvements.
    • Limited driver version control – The version of CUDA available in the Ubuntu repos may not always match the latest NVIDIA drivers, which can lead to compatibility issues.

    2. Installing from NVIDIA’s Official Website

    Pros:

    • Latest version – You get the newest CUDA Toolkit, which can include performance improvements, bug fixes, and new features.
    • Better control over versions – You can install specific versions to match requirements for AI/ML frameworks like TensorFlow or PyTorch.
    • More driver compatibility – NVIDIA ensures that the toolkit and drivers are aligned, reducing the risk of version mismatches.

    Cons:

    • More manual setup – You have to install dependencies manually, such as the correct NVIDIA driver, and configure paths properly.
    • Updates are not automatic – Unlike apt, updates need to be managed manually.
  • Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 22.04

    Step-by-Step Nvidia CUDA Toolkit Installation Guide: Ubuntu 22.04

    “The NVIDIA CUDA Toolkit is a software development kit that helps users create GPU-accelerated applications. It includes libraries, compilers, debuggers, and optimization tools”. Since we have an Nvidia GPU we will install it as shown below.

    Note: If you installed the nvidia-cuda-toolkit from default ubuntu noble repo, uninstall it first as that version is probably too old. Uninstall with “sudo apt purge nvidia-cuda* -y“

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
    

    First we wget the Nvidia Cuda Repository pin as shown above. Then we move it to /etc/apt/preferences.d/ (shown below).

    This is a configuration file that is used to prioritize packages from the NVIDIA CUDA repository when installing CUDA on a Linux system. Basically it tells apt where to get the Cuda Toolkit.

    $ sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

    Now we wget the repo. Note that this is a pretty hefty .deb and may take a few minutes.

    $ wget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb

    And install it.

    $ sudo dpkg -i cuda-repo-ubuntu2204-12-8-local_12.8.0-570.86.10-1_amd64.deb
    $ sudo cp /var/cuda-repo-ubuntu2204-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/

    Finally we install the toolkit.

    $ sudo apt-get update
    $ sudo apt-get -y install cuda-toolkit-12-8
    $ sudo apt install cuda-runtime-12-8
    

    Nvidia states that you need to add the following env vars to your .bashrc, however you will also need to add them to your python virtual env — in venv/bin/activate. These variables will be needed during the install process below.

     export CUDA_HOME=/usr/local/cuda                           
    export PATH=${CUDA_HOME}/bin:${PATH}
    export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

    Update $PATH

    The the following to your path

    /usr/local/cuda-12.8/bin

    Reference

    1. https://developer.nvidia.com/cuda-downloads
  • Essential Commands to Monitor Nvidia GPUs in Linux

    Identify Your GPU Via the Linux CLI

    Identify that your card is recognized by the OS via the CLI command below, hwinfo

    # hwinfo --gfxcard --short
    graphics card:                                                  
                           nVidia TU104GL [Tesla T4]
                           nVidia TU104GL [Tesla T4]
                           Matrox G200eR2
    
    Primary display adapter: #58
    

    Or you can see similar output with lshw

    # lshw -C display
      
      *-display
           description: 3D controller
           product: TU104GL [Tesla T4]
           vendor: NVIDIA Corporation
           physical id: 0
           bus info: pci@0000:43:00.0
           logical name: /dev/fb0
           version: a1
           width: 64 bits
           clock: 33MHz
           capabilities: pm bus_master cap_list fb
           configuration: depth=32 driver=nvidia latency=0 mode=1280x1024 visual=truecolor xres=1280 yres=1024
           resources: iomemory:3800-37ff iomemory:3810-380f irq:106 memory:d0000000-d0ffffff memory:38000000000-3800fffffff memory:38110000000-38111ffffff memory:d1000000-d13fffff memory:38010000000-3810fffffff memory:38112000000-38131ffffff
    

    Nvidia-smi

    Nvidia System Management interface (nvidia-smi) is a cli command which facilitates management and monitoring of Nvidia GPUs (mainly Tesla, GRID, Quatro, and Titan products). It ships with Nvidia GPU drivers on Linux. It is an extension buildt on the Nvidia Management Library. Official documentation for nvidia-smi can be found here.

    The output comprises two tables. The first table provides comprehensive details about all detected GPUs (e.g., one GPU in the provided example), while the second table enumerates the processes actively utilizing the GPUs. Below are detailed explanations of each parameter:

    • Temp (Temperature): Indicates the GPU core temperature in Celsius. Typically, temperature regulation is managed by data center infrastructure or external cooling solutions. Values like “44°C” are normal operating conditions, but sustained temperatures exceeding 90°C should trigger immediate action to prevent hardware degradation.
    • Perf (Performance State): Represents the current performance state of the GPU, ranging from P0 (highest performance) to P12 (lowest performance).
    • Persistence-M (Persistence Mode): Specifies whether the NVIDIA driver remains loaded in memory even in the absence of active processes like nvidia-smi. When “On,” this mode reduces driver load latency for GPU-dependent applications such as CUDA workloads.
    • Pwr: Usage/Cap (Power Usage/Capacity): Displays the current power draw of the GPU relative to its total power capacity, measured in Watts.
    • Bus-Id: Represents the PCI bus address of the GPU in the format domain:bus:device.function (hexadecimal). This identifier is critical for targeting specific GPUs in systems with multiple devices.
    • Disp.A (Display Active): Denotes whether memory on the GPU is allocated for display purposes. An “Off” value signifies no display context is associated with the GPU, making it dedicated to compute tasks.
    • Memory-Usage: Indicates memory utilization on the GPU, expressed as the amount of memory in use versus total available memory. Machine learning frameworks like TensorFlow may preallocate the full GPU memory capacity upon initialization, irrespective of immediate requirements.
    • Volatile Uncorr. ECC (Volatile Uncorrected ECC): Tracks uncorrected memory errors since the last driver initialization. Error Correction Code (ECC) is designed to detect and correct memory errors, ensuring data integrity during GPU operations.
    • GPU-Util (GPU Utilization): Reports the percentage of time over the sample interval during which one or more kernels actively used the GPU.
    • Compute M. (Compute Mode): Specifies the GPU’s compute mode. In “Default” mode, multiple processes can access the GPU concurrently. Other modes may restrict access to a single process or prohibit access entirely.
    • GPU (Index): Enumerates the GPUs detected in the system. The index corresponds to the NVML (NVIDIA Management Library) device index, enabling precise identification in multi-GPU environments.
    • PID (Process ID): Lists the process identifier of applications utilizing GPU resources.
    • Type: Describes the context of GPU usage—“C” for Compute tasks, “G” for Graphics tasks, and “C+G” for combined Compute and Graphics contexts.
    • Process Name: Identifies the executable or application utilizing GPU resources.
    • GPU Memory Usage: Reports the GPU memory utilized by each individual process.

    Use the -a switch for more detail

    Sample Nvidia-SMI Commands

    Query memory, free memory, and used memory

    # nvidia-smi --query-gpu=index,name,uuid,memory.total,memory.free,memory.used --format=csv
    index, name, uuid, memory.total [MiB], memory.free [MiB], memory.used [MiB]
    0, Tesla T4, GPU-9491a3e6-ea29-ba4e-4403-083244d5575c, 15360 MiB, 14928 MiB, 2 MiB
    1, Tesla T4, GPU-1d877ac8-5df1-34b0-4f86-59945e37d2ba, 15360 MiB, 14928 MiB, 2 MiB
    

    Query Temperatures

    # nvidia-smi --query-gpu=name,temperature.gpu --format=csv
    name, temperature.gpu
    Tesla T4, 34
    Tesla T4, 30
    

    Query PCI Slot

    # nvidia-smi --query-gpu=index,name,pci.bus_id --format=csv
    index, name, pci.bus_id
    0, Tesla T4, 00000000:02:00.0
    1, Tesla T4, 00000000:43:00.0
    

    Show Numa Affinity

    Non-Uniform Memory Access (numa) is a term used on systems with more than one bus/CPU. In the example below my GPU is installed in the numa node local to CPU0. use the flag “topo -m”


    Show Running Stats with dmon

    The dmon flag is unsed to show running statistics for one or more CPUs at 1s intervals. Dmon accepts a slew of options which are explained here.

    Below are the available base metrics and associated metric letter.

    SWITCHDESCRIPTION
    pPower Usage and Temperature
    uUtilization
    cProc and Mem Clocks
    vPower and Thermal Violations
    mFB, Bar1 and CC Protected Memory
    eECC Errors and PCIe Replay errors
    tPCIe Rx and Tx Throughput

    The nvidia-smi dmon command is also able to query available GPM (GPU Performance Monitor) metrics as shown in the example below

    #nvidia-smi dmon --gpm-metrics <gpmMetric1, gpmMetric2, ... ,gpmMetricN>
    <gpmMetricX> R

    The table below shows some of the available metrics and associated metric number. A complete list of metrics can be found here.

    METRICVARDESCRIPTION
    Graphics Activity = 1
    SM Activity  = 2
    SM Occupancy = 3
    Integer Activity=4
    Tensor Activity = 5
    DFMA Tensor Activity= 6
    HMMA Tensor Activity = 7
    IMMA Tensor Activity= 9
    DRAM Activity = 10
    FP64 Activity= 11
    FP32 Activity= 12
    FP16 Activity= 13
    PCIe TX= 20
    PCIe RX= 21
    NVDEC 0-7 Activity= 30-37
    NVOFA 0 Activity= 50
    NVJPG 0-7 Activity   = 40-47
    NVLink Total RX= 60
    NVLink Total TX= 61
    NVLink L0-17 RX= 62, 64, 66, …, 96
    NVLink L0-17 TX = 63, 65, 67, …, 97

    Enable persistence mode to reduce initialization overhead and keep the GPU active and running.

    # sudo nvidia-smi -pm 1
    Enabled persistence mode for GPU 00000000:43:00.0.
    All done.

    NVtop

    More info on nvtop can be found here

    $ sudo apt install nvtop

    Output example below


    GPUstat

    A wrapper of sorts for nvidia-smi. More info here.

    # apt install gpustat -y

    Output below

    galactica    Thu Feb  6 22:43:59 2025  535.183.01
    [0] Tesla T4 | 36°C,   0 % |     2 / 15360 MB |
    [1] Tesla T4 | 30°C,   0 % |     2 / 15360 MB |
    

    Installing the Nvidia Container Toolkit on Ubuntu 22.04

    The NVIDIA Container Toolkit is a set of tools that enables the use of NVIDIA GPUs within Docker and other container runtimes. It allows GPU-accelerated applications to run inside containers by providing the necessary drivers, libraries, and runtime components. Instructions for installing the Nvidia Container Toolkit are below. The official Nvidia doc can be found here, where you can also find guides for installing via DNF/Yum or Zypper

    ~# curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
        sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /
    #deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/experimental/deb/$(ARCH) /
    

    Now refresh packages list

    ~# sudo apt-get update

    Now install the Container Toolkit

    # sudo apt-get install -y nvidia-container-toolkit

    Configure the NVIDIA-Container Toolkit with Containerd

    For Ubuntu, the default runtime is containerd. In the example below we configure integration with containerd. Which modifies /etc/containerd/config.toml

    #sudo nvidia-ctk runtime configure --runtime=containerd
    #sudo systemctl restart containerd

    or use docker as shown below

    Configure the NVIDIA-Container Toolkit Configuring with Docker

    The nvidia-ctk command modifies the /etc/docker/daemon.json file on the host. The file is updated so that Docker can use the NVIDIA Container Runtime.

    # sudo nvidia-ctk runtime configure --runtime=docker

    I have also seen that it may be necessary to add default-runtime parameter directly to the file “/etc/docker/daemon.json“.

    Run the command below to see what runtimes Docker is using

    docker info | grep "Runtime"
     Runtimes: io.containerd.runc.v2 nvidia runc
     Default Runtime: runc

    Make a backup copy of /etc/docker/daemon.json

    # cp /etc/docker/daemon.json /etc/docker/daemon.json.ORIG

    Modify the file as shown below.

    # cat daemon.json
    {
        "runtimes": {
            "nvidia": {
                "path": "/usr/bin/nvidia-container-runtime",
                "runtimeArgs": []
             } 
        },
        "default-runtime": "nvidia" 
    }

    Restart Docker

    # systemctl restart docker

    Check the output of docker info and ensure that the nvidia is the default runtime

    # docker info | grep "Runtime"
     Runtimes: io.containerd.runc.v2 nvidia runc
     Default Runtime: nvidia
    

    Find your Nvidia Devices in /dev

    # sudo ls -la /dev | grep nvidia
    crw-rw-rw- 1 root root 195, 0 Feb 4 03:54 nvidia0
    crw-rw-rw- 1 root root 195, 1 Feb 4 03:54 nvidia1

    NGC CLI

    NVIDIA NGC (Nvidia GPU CLoud) CLI is a command-line interface tool for managing Docker containers in the NVIDIA NGC Registry. Download the CLI here.

    Once downloaded, unzip the Zip file and make the binary executable

    chmod u+x ngc-cli/ngc && chmod u+x ngc-cli/ngc

    Add the binary path to your path

    echo "export PATH=\"\$PATH:$(pwd)/ngc-cli\"" >> ~/.bash_profile && source ~/.bash_profile

    You will need an Nvidia Cloud account and and API key, follow the setup guide here to get started.

    Then docker login as shown below using your API key as your password

    # docker login nvcr.io
    Username: $oauthtoken
    Password: 
    WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
    Configure a credential helper to remove this warning. See
    https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    
    Login Succeeded
    

    nvidia-ctk

    A Container Device Interface (CDI) device is a standard way to manage container hardware access. More specifically it is used to assign GPU to containers through the Nvidia Container Toolkit

    Run the command below to generate the CDI specification file

    #  sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Then run the list command below to see what GPUs were detected.

    # nvidia-ctk cdi list
    INFO[0000] Found 5 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=1
    nvidia.com/gpu=GPU-1d877ac8-5df1-34b0-4f86-59945e37d2ba
    nvidia.com/gpu=GPU-9491a3e6-ea29-ba4e-4403-083244d5575c
    nvidia.com/gpu=all

    Running a Sample Docker Workload

    Run the command below to test to make sure docker is working properly

    # sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

    The command above should run nvidia-smi once and then exit.

    GPU Burn With Docker

    Pop a second terminal window, run nvtop and run the command below. You should see load on your GPUs. In the example below, 60 represents the number of seconds to run the test.

    # sudo docker run --gpus all --rm oguzpastirmaci/gpu-burn 60

    See below. GPUs running at 100% load.


    More Commands

    Reference

    1. https://org.ngc.nvidia.com/setup/installers/cli
    2. https://docs.nvidia.com/deploy/nvidia-smi/index.html
    3. https://taozhi.medium.com/monitor-nvidia-gpu-by-nvidia-smi-cli-56198fbf8e62
    4. https://www.gpu-mart.com/blog/monitor-gpu-utilization-with-nvidia-smi
    5. https://programmersought.com/article/84455484104/
    6. https://docs.nvidia.com/deploy/driver-persistence/index.html
    7. https://www.incredibuild.com/integrations/cuda#:~:text=Compute%20Unified%20Architecture%20(CUDA)%20is,tasks%20on%20GPU%20using%20CUDA.
    8. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    9. https://docs.nvidia.com/deploy/nvml-api/group__nvmlGpmEnums.html
    10. https://docs.nvidia.com/deploy/pdf/NVML_API_Reference_Guide.pdf
    11. https://docs.nvidia.com/deploy/nvidia-smi/index.html
    12. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    13. https://hub.docker.com/r/oguzpastirmaci/gpu-burn

  • Ollama CLI Quick Start Guide and Tutorial for Beginners – Part 1

    Ollama CLI Quick Start Guide and Tutorial for Beginners – Part 1

    This 2 part guide is written specifically for those who are just getting started with Ollama. Note that I originally wrote this post with the Nvidia Jetson Orin Nano in mind, as that is where I was initially running Olama… in part 2 I switch to something more powerful.

    That being said, the information below regarding installing Ollama and pulling a model is not specific to the Jetson and should work for anyone who wants to get started quickly with Ollama. Also worth noting that in this quick start guide we are installing ollama as a service, not as a container, as you would do if using jetson-containers [1]

    Installing ollama

    Use the command below to install ollama.

    $ sudo curl -fsSL https://ollama.com/install.sh | sh

    The install script downloads ollama, required Jetpack 6 components (on Jetson Devices), creates the ollama user, creates an api endpoint, and enables & starts the ollama service.

    $ sudo curl -fsSL https://ollama.com/install.sh | sh 
    >>> Installing ollama to /usr/local
    >>> Downloading Linux arm64 bundle
    ######################################################################## 100.0%
    >>> Downloading JetPack 6 components
    ######################################################################## 100.0%
    >>> Creating ollama user...
    >>> Adding ollama user to render group...
    >>> Adding ollama user to video group...
    >>> Adding current user to ollama group...
    >>> Creating ollama systemd service...
    >>> Enabling and starting ollama service...
    Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service.
    >>> NVIDIA JetPack ready.
    >>> The Ollama API is now available at 127.0.0.1:11434.
    >>> Install complete. Run "ollama" from the command line.
    

    Run the command below to test functionality and verify installed version of ollama.

    $ ollama --version
    ollama version is 0.5.7

    Configure ollama to Listen on all Interfaces

    By default ollama listens on 127.0.0.1. If you want to configure it to listen on all interfaces so that you can interact with it remotely, you will need to modify the service configuration as shown below.

    $ sudo vi /etc/systemd/system/ollama.service

    Add the following line to the file.

    Environment=”OLLAMA_HOST=0.0.0.0″

    Below is the service file post edit.

    $ cat ollama.service
    [Unit]
    Description=Ollama Service
    After=network-online.target
    
    [Service]
    ExecStart=/usr/local/bin/ollama serve
    User=ollama
    Group=ollama
    Restart=always
    RestartSec=3
    Environment="PATH=/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
    Environment="OLLAMA_HOST=0.0.0.0"
    

    Reload systemctl daemons so the change will take effect.

    $ sudo systemctl daemon-reload

    Restart the Service

    $ sudo systemctl restart ollama.service

    Check to ensure ollama is listening on all interfaces/IPs.

    $ netstat -a | grep 11434
    tcp6       0      0 [::]:11434              [::]:*                  LISTEN     

    Test connectivity from a remote host using telnet.

    $ telnet 10.1.10.11 11434
    

    Configure Alternative Download Directory For Ollama Models

    Obviously this step is optional, but you can set the download directory if wanted with the parameter below. Apparently the default location is “~/.ollama/models”

    Environment="OLLAMA_MODELS=/home/cpaquin/Download/ollama/models"

    You will need to restart ollama service


    Additional Packages (for non-jetson users)

    For those not running a jetson and installing jetpack, you may need a few additional packages

    Installing a Model

    Pull a Llama Model (Optimized for Jetson)

    For the Jetson Orin Nano (4GB), Iit is recommend using a small or quantized model such as:

    • TinyLlama (1B) → Good for Jetson Nano.
    • Llama 2 7B (Q4_0 or Q8) → Use GGUF quantization for lower RAM usage (Did not do this, so ran into issues) .
    • Mistral 7B (Q4_K) → More efficient than Llama 2.

    Running the command below shows us that we have yet to install a model.

    $ ollama list
    NAME    ID    SIZE    MODIFIED 

    Run the command below to download a model. In the example below we are downloading/installing Llama 2 7B (3.8GB).

    $ ollama pull llama2:7b

    PRO TIP:
    You can browse available models at https://ollama.com/

    Let’s config successful download and install with “ollama list“

    $ ollama list
    NAME         ID              SIZE      MODIFIED      
    llama2:7b    78e26419b446    3.8 GB    3 minutes ago

    Interacting with Ollama via the CLI

    You can interact with Ollama (and the loaded model) via the CLI in one of two ways.

    1. Predefined Prompt
    2. Interactive Mode

    Via Predefined Prompt

    In this mode, you call the model and pass the prompt in one step

    ~$ ollama run llama2:7b "Tell me about the Nvidia Jetson Orin Nano"

    Interactive Mode

    ollama run llama2:7b

    A Wild Error Appears

    While attempting to run an interactive session, we see that the ollma runner process was terminated

    ~$ ollama run llama2:7b 
    Error: llama runner process has terminated: signal: killed

    Let’s watch the output of journalctl and watch for errors as we try again

     sudo journalctl -f -u ollama.service

    While watching journalctl we see the following

    Feb 02 17:21:26 jetson.lab ollama[2306]: time=2025-02-02T17:21:25.650-05:00 level=WARN source=server.go:562 msg="client connection closed before server finished loading, aborting load"
    Feb 02 17:21:26 jetson.lab ollama[2306]: time=2025-02-02T17:21:25.668-05:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start: context canceled"
    Feb 02 17:21:26 jetson.lab ollama[2306]: [GIN] 2025/02/02 - 17:21:25 | 499 |  4.679059726s |       127.0.0.1 | POST     "/api/generate"
    Feb 02 17:21:26 jetson.lab systemd[1]: ollama.service: Failed with result 'oom-kill'.
    

    Apparently we are getting “oom-killed“.

    Lets watch free memory while we execute “ollama run” again. We will “watch” free -m

     watch -d -n 1 free -m

    We can see free memory drop to about 500mb, which may or may not be enough to run the rest of the system. When ollama is not running we have about 2485MB of free memory (shown below)

    $ free -m
                   total        used        free      shared  buff/cache   available
    Mem:            3601         940        2485           0         176        2483
    Swap:           1800         338        1462
    

    So we have a few options

    1. Attempt to reduce the amount of free memory available to Ollama (lets try)
    2. Tune ollama to attempt to use less memory (might be possible)
    3. Modify oom-killer behavior (probably a bad idea)
    4. Change to a lighter weight model (best idea)

    Freeing up System Memory

    Lets disable the Desktop GUI in ubuntu and see what that buys us in free memory. Note that you can probably just skip this section as I eventually move to a smaller model, but there is some good troubleshooting information here for those new to Linux.

    # sudo systemctl set-default multi-user.target

    And now immediately move to cli mode without reboot

    $ sudo init 3

    Output from ‘free -m” has not changed.

    $ sudo free -m
                   total        used        free      shared  buff/cache   available
    Mem:            3601         859        2402           4         339        2554
    Swap:           1800         164        1636
    
    

    We will reboot just in case…. ok that is a tiny bit better

    $ sudo free -m
                   total        used        free      shared  buff/cache   available
    Mem:            3601         458        2709          18         433        2938
    Swap:           1800           0        1800
    

    Now lets see if we can find a few services that we do not need, and stop and disable them.

    $ sudo systemctl disable --now bluetooth
    $ sudo systemctl disable --now avahi-daemon.service avahi-daemon.socket

    Then reboot. Once back up and running a cursory check of “free -m” shows that our efforts were mostly in vain. Lets try another model

    So this time we are going with a small model to test basic functionality and see if we are still running into oom errors.

    ollama pull tinyllama

    Ok much better…

    ~$ ollama run tinyllama
    >>> Send a message (/? for help)
    

    Lets check memory, and see how much headroom we have… Its not a lot

    ~$ free -m
                   total        used        free      shared  buff/cache   available
    Mem:            3601        1835         924           0         841        1575
    Swap:           1800         147        1653
    

    Inspecting System Utilization

    For this step we are going to use a couple of tools, most of which are custom to the Jetson.

    First we launch jtop [2]. And n another terminal windows, we load up the tinylama model and enter our prompt. While ollama is working, we watch observe jtop.

    So our prompt goes…

    can you tell me the history of the company Digital and their line of PDP computers
    Certainly! The company Digital was founded in England in 1963 by two college students, John Cocking and Michael Kearns. They were inspired to start a computer company after 
    witnessing the emergence of the personal digital assistant (PDA) market, which had been dominated by smaller, less advanced companies such as Acorn Computers and Marmalade.....trunc...
    

    While this is running we are watching jtop. Below we can see that our GPU is at 80% load and at times approached 100%. Memory usage is high, and CPU utilization is low.


    Measuring Tokens Per Second

    This seems to be the measurement that many use to determine how fast their machine is, so lets give it a try. To output tokens per second we add the “–verbose” flag.

    ~$ ollama run tinyllama --verbose "Can you tell me as much as you know about the Dell T620 Server"

    I get a 6 bullet list that is almost completely incorrect. What exactly is a DelT620? And no the T620 was intel xeon powered, not powered by AMD’s EPYC 7551 processor. But that does not matter, what comes next is what we are looking for.

    total duration:       25.885975737s
    load duration:        1.246583429s
    prompt eval count:    52 token(s)
    prompt eval duration: 283ms
    prompt eval rate:     183.75 tokens/s
    eval count:           464 token(s)
    eval duration:        24.354s
    eval rate:            19.05 tokens/s

    Let’s break down the output above line by line.

    TERMVALUEWHAT IT MEANS
    total duration25.885975737stotal duration: refers to the total time taken for the entire process of generating a response. This includes all stages such as:
    – Model Loading: If model isn’t already loaded into memory, this ime accounts for loading it.
    – Tokenization: Converting input text into tokens that the model can process.
    – Inference Time: The time spent by the model generating the response token by token.
    – Post-processing: Any steps taken after generation, such as formatting the output.
    – Communication Overhead: Time spent handling requests and responses, especially if running in a client-server setup.
    load duration1.246583429sload duration: refers to the amount of time spent loading the model into memory before it can start processing input. This step includes:
    – Model Retrieval: If the model is not already cached in memory, Ollama retrieves it from disk or another source.
    – Model Initialization: Preparing the model, including loading weights into VRAM (if using a GPU) or RAM (if running on a CPU).
    – Graph Compilation (if applicable): Some backends may optimize or compile the model for execution.
    – Memory Allocation: Ensuring that enough memory is available for inference.
    prompt eval count52 token(s)prompt eval count: refers to the number of tokens that were processed (evaluated) from the initial prompt before the model starts generating a response.
    Breakdown:
    – Prompt Tokens: Before the model generates any output, it first processes (evaluates) the input text (prompt).
    – Evaluation: The model tokenizes the input text and processes these tokens through its neural network.
    – Count Meaning: This number represents how many tokens were in the prompt that the model had to evaluate before responding.
    prompt eval duration283ms“prompt eval duration” refers to the total time taken to process (evaluate) the input prompt before the model begins generating a response.
    Breakdown:
    – Tokenization: The input text is broken down into tokens that the model can process.
    – Model Inference on Prompt Tokens: The model evaluates these tokens, passing them through its neural network layers to set up the internal context.
    – Context Initialization: The model updates its internal state based on the prompt before starting to generate output.
    prompt eval rate183.75 tokens/sprompt eval rate” refers to the speed at which the model processes the tokens from the input prompt. Measured in tokens per second (tokens/sec) where prompt-eval-rate = prompt eval count/prompt eval duration
    eval count464 tokens/s“eval count” refers to the total number of tokens that were processed during the generation phase—i.e., the number of new tokens the model generated as output.
    eval duration24.354s“eval duration” refers to the total time taken for the model to generate its response, measured from the moment it starts producing tokens until it finishes.
    eval rate19.05 tokens/s“eval rate” refers to the speed at which the model generates tokens during the response phase. It is typically measured in tokens per second (tokens/sec). – eval_count / eval_duration = eval rate

    Customizing Model Behavior

    You can customize how the model responds with the set command. For example, lets say I want to tell the model to respond to me as if it were a child

    $ ollama run tinyllama
    >>> /set system For any question asked can you respond as if you were a child, using only small words, short sentences, and lacking any specific technical details or jargon
    Set system message.
    >>> /save kid2
    Created new model 'kid2'
    >>> /bye
    
    

    Then we load the replicated model.

    ~$ ollama run kid2
    >> what was the US civil war about 
    The United States Civil War was fought between the Union and Confederate states over whether the union should remain intact as a single nation or become a separate nation known 
    as the Confederate States of America. The war began in April 1861 after President Abraham Lincoln announced that he would veto any bill to abolish the constitutional right to 
    secession unless Congress passed a similar law.
    
    As the war progressed, each side unleashed their forces on the other and suffered significant losses. The north won major victories in the first year of the war, but southern 
    states began withdrawing from the union due to economic hardship, making it necessary for Lincoln to appeal to them with a proposal to call off fighting and establish a 
    government in the south. In response, the Confederate army launched its own attack on April 12, 1865, which led to Lincoln's assassination a few months later. The US civil war 
    was one of the most devastating conflicts in world history, causing significant destruction to both sides and leaving millions dead or wounded.
    

    Ok, so that was not exactly child-like, however this is probably the fault of the model that I am using. Lets see if I can coax it to answer the question differently.

    >> /set system For any question asked can you respond as if you were a scholar, using a large vocabulary of diverse words. Be very specific in your responses and provide colorful deta
    ... ils
    Set system message.
    >>> what was the US civil war about
    The United States Civil War was fought between the Union and Confederate states over whether the union should remain intact as a single nation or become a separate nation known 
    as the Confederate States of America. The war began in April 1861 after President Abraham Lincoln announced that he would veto any bill to abolish the constitutional right to 
    secession unless Congress passed a similar law.
    
    The main cause of the war was the belief by some members of the southern states that they had been wronged and mistreated by the federal government, as well as a desire for 
    self-determination and statehood. The north won major victories in the first year of the war, but southern states began withdrawing from the union due to economic hardship, 
    making it necessary for Lincoln to appeal to them with a proposal to call off fighting and establish a government in the south. In response, the Confederate army launched its own 
    attack on April 12, 1865, which led to Lincoln's assassination a few months later. The US civil war was one of the most devastating conflicts in world history, causing 
    significant destruction to both sides and leaving millions dead or wounded.
    

    Ok so not great, but again probably the model. Anyway you get the point. I’ll try this again with a different model in the near future of one of my more powerful servers/


    Removing a Model

    Remove a model with the “rm” switch. See below

    ~$ ollama list
    NAME                ID              SIZE      MODIFIED    
    kid2:latest         ca8452f00cd5    637 MB    2 hours ago    
    kid:latest          ce8e59f0d306    637 MB    2 hours ago    
    tinyllama:latest    2644915ede35    637 MB    3 hours ago    
    llama2:7b           78e26419b446    3.8 GB    5 hours ago    
    cpaquin@jetson:~$ ollama rm kid:latest kid2:latest llama2:7b 
    deleted 'kid:latest'
    deleted 'kid2:latest'
    deleted 'llama2:7b'
    

    Training a Model

    Most available models operate pretty well when you are asking them about information that they have been trained on. Above you can see that the tinyllama model has not been trained properly to even remotely respond as if it was a child. In order to do that, the model would need to be trained.

    So lets train tinyllama. Keep in mind that these models have short-term memory constraints, meaning they only retain training data during an active conversation. Once you close the session and start a new one, the model will not remember any information from the previous interaction. To overcome the short-term memory limitation, one would need a backend database or some form of persistent storage, which would you to save and retrieve relevant information across sessions. Depending on your needs, you could use:

    • Relational Databases (SQL) – MySQL, PostgreSQL, or SQLite for structured data.
    • NoSQL Databases – MongoDB, Redis, or Firebase for more flexible storage.
    • Vector Databases – Pinecone, FAISS, or ChromaDB for storing embeddings in AI applications.
    • File Storage – JSON, CSV, or other formats for lightweight persistence.

    The backend system could then integrate with the AI model, fetching and updating information as needed, effectively giving it “memory” beyond a single session. More about this later. For now let’s “teach” tiny llama about Star Trek. Specifically, I prompted the model with this question, and its response was incomplete.

    >>> do you know about any of the star trek tv shows
    Certainly! Here are some popular Star Trek TV shows:
    
    1. Star Trek: The Original Series (1966-1968)
    2. Star Trek: The Next Generation (1987-1994)
    3. Star Trek: Deep Space Nine (1993-1999)
    4. Star Trek: Voyager (1995-2001)
    5. Star Trek: Enterprise (2001-2005)
    6. Star Trek: Discovery (2017-)
    

    I then made sure that was all the data it had on Star Trek TV shows.

    >>> is that all the star trek shows
    Yes, that's all the Star Trek TV series I could find.

    And we are off to the races! Here is what I “taught” the model

     Let me tell you about some other Start trek TV shows that you are unaware of. Star Trek: The Animated Series premiered in 1972 and ran for 2 seasons, it featured the same character
    ... s that appeared Star Trek: The Original Series. Star Trek: Lower Decks premiered in 2020 and ran for 5 seasons. Sadly it was recently cancelled. It was an animated show which was  
    ... comedic in nature, often making fun of the tropes features throughout the other Star Trek shows and movies. Star Trek: Prodigy premiered in 2021 and ran for 2 seasons, it was an an
    ... animated show that originally aired on the kids station Nickelodeon. It was geared towards children, but was still fun for adults to watch as well. Especially with their children. 
    ... Star Trek: Strange New Worlds premiered in 2022 and has been running for 2 seasons, a third season is on its way. This show takes place on the starship Enterprise, but before Capta
    ... in Kirk was in command. The Captain, is Captain Pike, who was a character originally introduced in the 2 part pilot episode of the Original Series. Would you like to know more about star trek?
    

    Here we are witnessing what is called a hallucination which is when an artificial intelligence (AI) model generates an incorrect or misleading output.

    We do see that the model does now know about Star Trek: Strange New Worlds, but it basically made up the plot. There is no Captain Alex Danforth anywhere on google, and I have never heard of the U.S.S Explorer. Ava Acres is a real actor, but has nothing to do with Star Trek. James Frain played Sarek in Star Trek Discovery, so at least he is associated with the series. The model did remember that I mentioned a Star Trek show that aired on Nickelodeon, and got the series dates correct, but associated it with a completely made up show.

    Fascinating…

    >>> Tell me about Star Trek: Strange New Worlds 
    Yes, I do have more information on Star Trek: Stragrg New Worlds. This is a science fiction TV show that originally aired on Nickelodeon between 2021 and 2023. It's set in the 
    same universe as Star Trek: Lower Decks, but takes place in the 25th century.
    
    The show follows the crew of the U.S.S. Explorer, a new ship that was built specifically for this mission. The crew is made up of Captain Alex Danforth (played by James Frain), 
    First Officer Renee "Ren" Dahla (played by Ava Acres), and Engineer Jesse Johnson (played by David Hewison).
    
    The Explorer has been sent to explore a new planet that's rich in resources. The crew encounters various alien species, including a group of intelligent, hostile beings who have 
    been observing their activities for some time.
    
    The show is filled with action and adventure as the Explorer faces numerous challenges on its mission. It also explores themes such as colonialism, power dynamics, and 
    responsibility to others. The show's premise may seem familiar to Star Trek fans, but it features new characters and a fresh take on the classic series universe.
    
    If you enjoy Nickelodeon-style science fiction and have enjoyed Star Trek: Lower Decks, then I think you will enjoy Star Trek: Stragrg New Worlds.
    

    Part 2…. coming soon.

    Resources

    1. https://www.hostinger.com/tutorials/ollama-cli-tutorial
    2. https://www.jetson-ai-lab.com/tutorial_ollama.html
    3. https://www.kdnuggets.com/ollama-tutorial-running-llms-locally-made-super-simple
    4. https://sarinsuriyakoon.medium.com/run-ollama-in-5-minutes-free-open-source-llm-on-your-local-bonus-section-e520e452de86

    Reference

    [1] https://github.com/dusty-nv/jetson-containers

    [2] https://github.com/rbonghi/jetson_stats

  • Finding and Mapping Jetson OS and JetPack Versions on the Nvidia Jetson

    Finding and Mapping Jetson OS and JetPack Versions on the Nvidia Jetson

    Updated – 7/1/2026

    Below are all the methods that I have found to either find your Jetson OS version, or your Jetpack version (which includes Jetson OS version, Ubuntu version, CUDA Version, NVIDIA drivers, and firmware).

    First let’s review the matrix and see how JetsonOS Maps to JetPack version (along with Ubuntu version, CUDA version, and release date)

    Note:

    • Nano: supported through JetPack 4.6.x
    • Xavier: supported through JetPack 5.1.x
    • Orin: required for JetPack 6.x and newer
    L4TJetPackUbuntuCUDARelease DateRelease TypeDevicesSupport StateProd Ready?
    39.27.224.0413.2.1Jun 2026ProdOrin, Thor/T5000/T4000ActiveOrin: Preferred / Thor: Preferred
    38.47.124.0413.xJan 2026ProdThor/T5000/T4000ActiveThor: Yes / Orin: No
    38.2 / 38.2.1724.0413.xAug 2025ProdThor/T5000ActiveThor: Yes / Orin: No
    36.5.06.2.222.0412.62025ProdOrinMaintenanceOrin: Stable fallback
    36.4.46.2.122.0412.6Jun 2025ProdOrinMaintenanceOrin: Yes
    36.4.36.222.0412.6Early 2025ProdOrinMaintenanceOrin: Yes
    36.4.06.122.0412.6Sep 2024ProdOrinMaintenanceOrin: Acceptable
    36.3.0622.0412.2May 2024ProdOrinSustainingNo
    36.2.06.0 DP22.0412.2Dec 2023DPOrinDPNo
    35.6.45.1.620.0411.8Feb 2026ProdXavier, OrinSustainingXavier: Preferred / Orin: Legacy fallback
    35.6.25.1.520.0411.8Mar 2024ProdXavier, OrinSustainingXavier: Yes
    35.6.15.1.520.0411.8Jan 2024ProdXavier, OrinSustainingXavier: Yes
    35.6.05.1.420.0411.4Dec 2023ProdXavier, OrinSustainingXavier: Acceptable
    35.5.05.1.320.0411.4Sep 2023ProdXavier, OrinSustainingNo
    35.4.15.1.220.0411.4May 2023ProdXavier, OrinSustainingNo
    35.3.15.1.120.0411.4Feb 2023ProdXavier, OrinSustainingNo
    35.2.15.120.0411.4Aug 2022ProdXavier, OrinSustainingNo
    35.1.05.0.220.0411.4Jun 2022ProdXavierSustainingNo
    34.1.15.0.1 DP20.0411.4Apr 2022DPXavierDPNo
    34.1.05.0 DP20.0411.4Mar 2022DPXavierDPNo
    32.7.64.6.618.0410.2Nov 2024ProdNano, TX1, TX2, XavierEOLNano: Final supported
    32.7.54.6.518.0410.2Jun 2024ProdNano, TX1, TX2, XavierEOLNo
    32.7.44.6.418.0410.22024ProdNano, TX1, TX2, XavierEOLNo
    32.7.34.6.318.0410.2Dec 2022ProdNano, TX1, TX2, XavierEOLNo
    32.7.24.6.218.0410.2Aug 2022ProdNano, TX1, TX2, XavierEOLNo
    32.7.14.6.118.0410.2May 2022ProdNano, TX1, TX2, XavierEOLNo
    32.6.14.618.0410.2Nov 2021ProdNano, TX1, TX2, XavierEOLNo
    32.5.24.5.118.0410.2Jul 2021ProdNano, TX1, TX2, XavierEOLNo
    32.5.14.5.118.0410.2Jun 2021ProdNano, TX1, TX2, XavierEOLNo
    32.5.04.518.0410.2Jan 2021ProdNano, TX1, TX2, XavierEOLNo
    32.4.44.4.118.0410.2Oct 2020ProdNano, TX2, XavierEOLNo
    32.4.34.418.0410.2Jul 2020ProdNano, TX2, XavierEOLNo
    32.4.24.4 DP18.0410.2May 2020DPNano, TX2, XavierDPNo
    32.3.14.318.0410Dec 2019ProdNano, TX2, XavierEOLNo
    32.2.34.2.318.0410Sep 2019ProdNano, TX2, XavierEOLNo
    32.2.14.2.218.0410Jul 2019ProdNano, TX2, XavierEOLNo
    32.2.04.2.118.0410Jun 2019ProdNano, TX2, XavierEOLNo
    32.1.04.218.0410Mar 2019ProdNano, TX2, XavierEOLNo
    31.1.04.1.118.0410Nov 2018ProdTX1, TX2EOLNo
    31.0.24.118.0410Oct 2018ProdTX1, TX2EOLNo
    31.0.1418.049Sep 2018ProdTX1, TX2EOLNo
    28.4.03.3.316.049May 2019ProdTX1, TX2EOLNo
    28.2.13.3 / 3.2.116.049Feb 2018ProdTX1, TX2EOLNo
    28.2.03.216.049Dec 2017ProdTX1, TX2EOLNo
    28.1.03.116.048Oct 2017ProdTX1EOLNo
    27.1.0316.048Mar 2017ProdTX1EOLNo
    24.2.13.0 / 2.3.116.047Dec 2016ProdTK1, TX1EOLNo
    24.2.02.316.047Sep 2016ProdTK1, TX1EOLNo
    24.1.02.2 / 2.2.116.047Jun 2016ProdTK1, TX1EOLNo
    23.2.02.116.046.5Mar 2016ProdTK1EOLNo
    23.1.0216.046.5Jan 2016ProdTK1EOLNo
    21.5.02.3 / 2.3.114.046.5Oct 2015ProdTK1EOLNo
    21.4.02.2 / 2.1 / 2.0 / 1.2 DP14.046.5Jun 2015MixedTK1EOLNo
    21.3.01.1 DP14.046Apr 2015DPTK1DPNo
    21.2.01.0 DP14.046Feb 2015DPTK1DPNo

    Jetson Linux (L4T) / JetPack Version Matrix (Unified)

    With Support Status and Recommended Baselines



    Commands to Find your Jetpack/Jetson OS Version

    This is not an extensive list, but its what I have used in the past


    jetsonInfo.py

    Git clone the repo below.

    https://github.com/jetsonhacks/jetsonUtilities

    and then run jetsonInfo.py.

    # ./jetsonInfo.py
    NVIDIA NVIDIA Jetson Orin NX Engineering Reference Developer Kit
    L4T 36.4.3 [ JetPack UNKNOWN ]
    Ubuntu 22.04.5 LTS
    Kernel Version: 5.15.148-tegra
    CUDA NOT_INSTALLED
    CUDA Architecture: 8.7
    OpenCV version: 4.5.4
    OpenCV Cuda: NO
    CUDNN: ii libcudnn9
    TensorRT: NOT_INSTALLED
    Vision Works: NOT_INSTALLED
    VPI: NOT_INSTALLED
    Vulcan: 1.3.204

    L4T 36.4.3 is part of Jetpack 6.2, looks like the repo has not been updated since 2021 which is why Jetpack 6.2 is not recognized (also missing installed TensorRT). See section below for Jetson OS to Jetpack Version mapping

    Per the git-repo

    1. The hardware designator is derived from the file: ‘/proc/cpuinfo’
    2. The L4T version is derived from the file: ‘/etc/nv_tegra_release’
    3. The Ubuntu version is derived from the file: ‘/etc/os-release’
    4. The Linux kernel version is derived from the file: ‘/proc/version’

    Via /etc/nv_tegra_release

    Below in bold you can see the Jetson OS release and revision.

    # cat /etc/nv_tegra_release
    # R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025
    # KERNEL_VARIANT: oot
    TARGET_USERSPACE_LIB_DIR=nvidia
    TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

    Via Apt

    # sudo apt-cache show nvidia-jetpack
    Package: nvidia-jetpack
    Source: nvidia-jetpack (6.2)
    Version: 6.2+b77
    Architecture: arm64
    Maintainer: NVIDIA Corporation
    Installed-Size: 194
    Depends: nvidia-jetpack-runtime (= 6.2+b77), nvidia-jetpack-dev (= 6.2+b77)
    Homepage: http://developer.nvidia.com/jetson
    Priority: standard
    Section: metapackages
    Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.2+b77_arm64.deb
    Size: 29298
    SHA256: 70553d4b5a802057f9436677ef8ce255db386fd3b5d24ff2c0a8ec0e485c59cd
    SHA1: 9deab64d12eef0e788471e05856c84bf2a0cf6e6
    MD5sum: 4db65dc36434fe1f84176843384aee23
    Description: NVIDIA Jetpack Meta Package
    Description-md5: ad1462289bdbc54909ae109d1d32c0a8

    Via nvidia-lt4-core pkg

    # dpkg-query --show nvidia-l4t-core
    nvidia-l4t-core 36.4.3-20250107174145