• Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Introduction

    In this step-by-steps guide we will replace the out of the box nouveau drivers on RHEL9 with Nvidia Drivers. We will also install the the Nvidia CUDA Toolkit and the Nvidia Container Toolkit.


    GPU and Driver Inspection

    First we need to make sure that our Nvdia GPU is recognized by Red Hat Enterprise Linux 9 (RHEL9).

    lspci -nn | grep -i nvidia
    b6:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [A40] [10de:2235] (rev a1)
    

    Using the command below we can see that we are currently using the non-propietary nouveau driver.

    # lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nouveau
    	Kernel modules: nouveau
    

    Configuring Repositories for the Nvidia Driver Install

    First we need to enable the RHEL9 CodeReady Builder repo. Note we are running these commands as root.

    # subscription-manager repos --enable codeready-builder-for-rhel-9-$(uname -i)-rpms
    

    Next we will need to install and configure the EPEL repo.

    # dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
    

    Now we install the ELRepo project repo – this will provide nvidia-detect which we can utilize later

    # dnf -y  install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm

    Prerequisites for Nvidia Driver Install

    Now we need to install dependencies and build tools.

    # dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc make dkms acpid libglvnd-glx libglvnd-opengl libglvnd-devel pkgconfig
    

    Install Nvidia Drivers

    Install nvidia-detect from the ELRepo project repo.

    # dnf -y install nvidia-detect

    Now install the Nvidia Drivers.

    # dnf -y install $(nvidia-detect)

    Now reboot.


    Confirming Nvidia Driver Installation

    Now lets run the command below one more time.

    [root@gpu ~]# lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [250] Latency Tolerance Reporting
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nvidia
    	Kernel modules: nouveau, nvidia_drm, nvidia
    

    As you can see in the output below, the kernel is loading the Nvidia driver. We can still see nouveau kernel modules listed, but that is fine, as they are not loaded. We can confirm this with the command below.

    # lsmod | grep nouveau

    The above command should not output anything, while the opposite should be true for the command below.

    # lsmod | grep nvidia

    Configure Nvidia Persistenced

    Start and enable nvidia-persistenced.service. This will enable persistence-mode which will keep the nvidia device state from going “stale”

    # systemctl enable nvidia-persistenced.service
    # systemctl start nvidia-persistenced.service
    

    Installing the Nvidia CUDA Toolkit

    We will now follow the official guide and install the Nvidia CUDA toolkit. Per that guide, we need to enable a few repos, however two of those repos should be enabled by default, and the other one we enabled above, however I will list them here for the sake of documentation.

    # subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
    # subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
    # subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

    Now we install the Nvidia repo for the CUDA toolkit.

    dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
    

    Now install the CUDA toolkit as shown below

    # sudo dnf -y install cuda-toolkit

    Confirm that the toolkit is installed and note the version.

    # rpm -qa cuda-toolkit
    cuda-toolkit-12.8.1-1.x86_64
    

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well. Note that the cuda version should match the one that you installed above.

    export PATH=/usr/local/cuda-12.8/bin:$PATH

    Now test nvcc as shown below.

    # nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Feb_21_20:23:50_PST_2025
    Cuda compilation tools, release 12.8, V12.8.93
    Build cuda_12.8.r12.8/compiler.35583870_0
    

    Installing the Nvidia Container Toolkit

    Next we will install the Nvidia Container Toolkit, which allows users to run GPU-accelerated containerized applications.

    A bit about Container Management in RHEL 9

    The default container packages in RHEL 9 are as follows.

    1. Podman – daemonless container image
    2. Buildah – tool for building OCI (Open Container Initiative) container images
    3. Skopeo – tool for managing container images and repos
    4. CRIU – tool to create and save running container checkpoints to disk
    5. Udica – tool for managing SELinux policies for containers

    Installation of the toolkit

    We will follow the instructions as documented here.

    First we configure the repo

    # curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

    Then install via dnf

    # dnf install -y nvidia-container-toolkit

    Configuring the Container Toolkit for Podman

    Generate the CDI specification file using the command below.

    # nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Now lets check the names of the generated device(s).

    # nvidia-ctk cdi list
    INFO[0000] Found 3 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=GPU-7e880be2-891c-72e3-9515-0fd51240e7f4
    nvidia.com/gpu=all
    

    References

    1. https://medium.com/@blackhorseya/step-by-step-guide-to-installing-nvidia-drivers-on-rhel-9-1107e0cd641d
    2. https://access.redhat.com/discussions/227d2101-b4e3-490a-aa1c-601c407ec038
    3. https://darryldias.me/2022/install-nvidia-drivers-on-rhel-9/
    4. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    5. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-rhel-rocky
    6. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html
  • Upgrade Cisco Catalyst WS-C4500X IOS: Step-by-Step Guide

    Upgrade Cisco Catalyst WS-C4500X IOS: Step-by-Step Guide

    I recently picked up a Cisco Catalyst WS-C4500X for my homelab as I am in the process of moving to all 10GBe. Before I put the new switch into service I want to make sure that it is running the latest IOS image.

    I will be using a USB drive that has already been formatted as FAT32 (FAT16 is also acceptable). I have copied the .bin file to the usb drive.

    Plug the USB drive into the front of the switch. Enter enable mode and run dir on the usb drive as shown below.

    Switch>en
    Switch#dir usb0:
    Directory of usb0:/
    
       37  drwx       16384  Mar 22 2025 23:28:06 +00:00  .Trash-1000
       38  -rwx   195351416  Mar 18 2025 01:16:52 +00:00  cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin
    
    31440846848 bytes total (31220072448 bytes free)
    

    Now we need to copy the image from the USB drive to the bootflash on the switch.

    Switch#copy usb0:cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin bootflash:

    The file will take a bit to copy. Once the copy is complete you will be returned to the prompt.

    We now want to confirm that the file is in bootflash.

    Switch#show bootflash:

    Output below, cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin is our new file. As you can see by the dates this switch has not been upgraded in a long while.

    Now we need to change the config-register

    Switch>en
    Password: 
    Switch#conf t
    Enter configuration commands, one per line.  End with CNTL/Z.
    Switch(config)#config-register 0x2102
    

    Now we clear out the contents of the bootflash

    Switch(config)# no boot system

    We must now save the running config in order to see the updated boot vars

    Switch#show boot
    BOOT variable =
    CONFIG_FILE variable does not exist
    BOOTLDR variable does not exist
    Configuration register is 0x2101 (will be 0x2102 at next reload)

    Now we configure the switch to boot the new IOS image.

    Switch(config)# boot system bootflash:cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin

    Now exit config mode and save changes

    Switch(config)#exit
    Switch#copy running-config startup-config
    Destination filename [startup-config]? 
    Building configuration...
    Compressed configuration from 1491 bytes to 878 bytes[OK]

    Now we confirm our boot config

    Switch#show boot
    BOOT variable = bootflash:cat4500e-universalk9.SPA.03.11.11.E.152-7.E11.bin,12;
    CONFIG_FILE variable does not exist
    BOOTLDR variable does not exist
    Configuration register is 0x2101 (will be 0x2102 at next reload)
    

    Configuration register 0x2101 will boot the first boot image shown in the output of dir bootflash:

    Configuration register 0x2102 overrides this behavior and boots the image specified in “BOOT variable” output.


    Reference

    1. https://youtu.be/vaiSC8SzHEc?si=kHM5ak1W_89zHx_j
    2. https://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/50421-config-register-use.html
  • How to Reset Cisco Catalyst WS-C4500X to Factory Defaults

    How to Reset Cisco Catalyst WS-C4500X to Factory Defaults

    Need to reset your Cisco Catalyst 4500X  back to the factory default settings? Have you forgotten your password? Follow the steps below.

    Note: that I am assuming that you have already established a console connection to the switch using a Cisco serial cable (rollover cable).

    Interrupt the boot sequence with Ctrl-C within the first 5 seconds after power on. This will allow you to enter the “ROM” aka ROM Monitor.

    Configure the switch to boot-up without reading the configuration memory (NVRAM).

    You will then see the following options below. Select as shown.

    Clear the config as shown below

    Now boot the switch with the boot command

    Reference

    1. https://www.cisco.com/c/en/us/support/docs/switches/catalyst-4500-series-switches/116436-technote-product-00.html
    2. https://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/21229-pswdrec-cat4000-supiii-21229.html
  • How to Install and Configure Dnsmasq with Web Frontend on RHEL 9

    How to Install and Configure Dnsmasq with Web Frontend on RHEL 9

    In this post we are going to install and configure dnsmasq with a simple webUI acting as a front end for our less technical users. Our goal is to simplify dns in our lab sandboxes, and keep users from directly modifying our dnsmasq config files.

    Installation

    First we need to install dnsmasq.

    # dnf -y install dnsmasq

    Now lets enable and start the service. We will also check the status of the service to ensure that we do not have any issues with the default config.

    # systemctl start dnsmasq
    # systemctl enable dnsmasq
    # systemctl status dnsmasq

    Configuration

    Next let’s make a backup of the default configuration file before we start making modification.

    cp /etc/dnsmasq.conf /etc/dnsmasq.conf.orig

    I made the following modifications.

    1. listen-address is the loopback and our routable ip address
    2. expand-hosts, we uncomment this in order to allow dnsmasq to automatically expand the hostnames to fully qualified domain names
    3. domain – this is the local domain that we will serve via dnsmasq
    4. dhcp-range – this is the range of IP addresses that dnsmasq is allowed to hand out
    interface=enp1s0
    listen-address=127.0.0.1,192.168.65.7
    expand-hosts
    domain=sandbox3.localdomain
    dhcp-range=192.168.65.20,192.168.65.40,255.255.255.128,12h
    

    Next we check our config file for any errors

    # dnsmasq --test
    dnsmasq: syntax check OK.

    Now lets restart dnsmasq

    # systemctl restart dnsmasq

    Firewall Config

    Now we need to modify firewalld

    [root@dns ~]#  firewall-cmd --add-service=dns --permanent
    success
    [root@dns ~]# firewall-cmd --add-service=dhcp --permanent
    success
    [root@dns ~]# firewall-cmd --reload
    success

    Testing

    I prefer to use nslookup for testing, so lets install it

    # dnf -y install bind-utils

    Now let’s make sure we are able to resolve addresses, using our local instance of dnsmasq.

    # nslookup
    > server localhost
    Default server: localhost
    Address: ::1#53
    Default server: localhost
    Address: 127.0.0.1#53
    > google.com
    Server:		localhost
    Address:	::1#53
    
    Non-authoritative answer:
    Name:	google.com
    Address: 142.251.40.174
    Name:	google.com
    Address: 2607:f8b0:4006:821::200e
    

    Install and Configure the Webui

    For the webui – we are going to use a simple one that I found via the link below. https://github.com/akivajp/dnsmasq-webconf

    First, we need to install git and pip

    # dnf -y install git
    # dnf -y install pip

    Then we follow the directions which I will repeat here.

    # mkdir -p ~/git && cd ~/git
    # git clone https://github.com/akivajp/dnsmasq-webconf.git

    Then we use pip to install jinja2

    # pip install --user bottle jinja2

    Now change directory

    # cd dnsmasq-webconf/

    We now need to poke a hole for http traffic in our local firewall

    # firewall-cmd --add-service=http --permanent
    # firewall-cmd --reload

    Now start the front end

    #  python ~/git/dnsmasq-webconf/app/index.py 80 --leases /var/lib/dnsmasq/dnsmasq.leases --hosts /etc/hosts --config /etc/dnsmasq.conf

    Creating a Service

    So far we have dnsmasq configured and running, and we have installed a web front end and have been able to start it on the command line. Now we need to configure the front end to start as a service when the OS boots. So we now need to turn the webUI into a systemd service.

    First we create the following service file

    # vi /etc/systemd/system/dnsmasq-webconf.service

    The contents of which are below. Note that we have modified relative paths to be absolute paths.

    [Unit]
    Description=DNSMasq WebConf
    After=network.target
    
    [Service]
    ExecStart=/usr/bin/python3 /root/git/dnsmasq-webconf/app/index.py 80 --leases /var/lib/dnsmasq/dnsmasq.leases --hosts /etc/hosts --config /etc/dnsmasq.conf
    Restart=always
    User=root
    WorkingDirectory=/root/git/dnsmasq-webconf/app
    StandardOutput=journal
    StandardError=journal
    
    [Install]
    WantedBy=multi-user.target

    Now we need to reload systemd

    # systemctl daemon-reload

    Now we can start the service

    # systemctl start dnsmasq-webconf.service

  • Resetting a Lost BMC Password with ipmitool

    Resetting a Lost BMC Password with ipmitool

    I recently got my hands on a couple of gigabyte servers. These machines came preinstalled with Ubuntu 20.04. Credentials for a OS local user account were on a sticker on the machines. However there was no indication of what the BMC credentials were. According to this document, there should be default credentials that we can use on the motherboard, however, they were not working.

    These machines came with Ubuntu 20.04 on them, and did not have ipmitool installed.

    $ sudo apt install ipmitool --fix-missing

    Once installed we need to determine what BMC users exist.

    $ sudo ipmitool user list 1

    In the output below you can see that there is one user – “admin” – identified as ID #2

    Now lets reset the password for “admin”.

    $ sudo ipmitool user set password 2

    We are then prompted to create a new password.

    We should now be able to load the BMC web interface and login with our new credentials

  • How to Connect to a Cisco Serial Port via Minicom

    How to Connect to a Cisco Serial Port via Minicom

    Prerequisites

    You will need the following. A RJ45 to RS232 (serial) roll-over cable. Shown below

      And a USB to RS232(DB9) adapter (below)

        Connect the RJ45 to the serial port on your Cisco switch. Connect the two Db9s and plug the usb port into your workstation.

        Your should see the adapter recognized as ttyUSB0. You can check this with the command below

        $ sudo dmesg | grep tty
        [692949.868426] usb 3-12: FTDI USB Serial Device converter now attached to ttyUSB0
        

        Install and Configure Minicom

        Next you will need need to install minicom. Since I am running Ubuntu, I will use apt. If using Centos/Fedora you will use dnf.

        $ sudo apt install minicom -y

        Then we need to configure minicom. Start by running the command below.

        $ sudo minicom -s

        Select “Serial Port Setup”

        Change the Following

        1. Serial Device : /dev/ttyUSB0
        2. Bps/Par/Bits : 9600 8N1
        Correct minicom config for Cisco

        Save your config and relaunch minicom.

      1. Simple RAG with Ollama, OpenWebUI, and VectorDB on Ubuntu 22.04

        Simple RAG with Ollama, OpenWebUI, and VectorDB on Ubuntu 22.04

        Prerequisites

        I have already installed Nvidia proprietary drivers and the Nvidia Cuda Toolkit. I documented the install of the Cuda toolkit in an older post which can be found here.

        Since I have Nvidia GPUs in my host system, and I intend to run some services in containers, I want to make sure that I install the nvidia-container-toolkit. Instructions on how to setup the repo on Ubuntu 22.04 can be found here. Once you have setup the repo, you can follow the steps below, or just follow the instructions found in the link above.

        $ sudo apt-get install -y nvidia-container-toolkit

        I will be using Docker as my container runtime. So I need to configure the container runtime to use docker.

        $ sudo nvidia-ctk runtime configure --runtime=docker

        Now restart Docker.

        $ sudo systemctl restart docker

        Install Ollama

        Run the command below to install ollama as a service. You can also choose to run containerized ollama, however those steps are not documented here.

        curl -fsSL https://ollama.com/install.sh | sh

        Once complete, run the command below to confirm installation, and check version.

        $ ollama --version
        ollama version is 0.5.12

        Add the environment variable below to the ollama service file in order to listen on all interfaces. This way we can access remotely if needed.

        Environment=”OLLAMA_HOST=0.0.0.0″

        Edit the ollama service files and add the line above to the bottom of the file.

        $ sudo vi /etc/systemd/system/ollama.service

        The file should appear as shown below.

        [Unit]
        Description=Ollama Service
        After=network-online.target
        
        [Service]
        ExecStart=/usr/local/bin/ollama serve
        User=ollama
        Group=ollama
        Restart=always
        RestartSec=3
        Environment="PATH=/usr/local/cuda/bin:/home/cpaquin/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
        Environment="OLLAMA_HOST=0.0.0.0"
        
        [Install]
        WantedBy=default.target
        

        Now restart ollama and reload systemd

        $ sudo systemctl restart ollama.service
        $ sudo systemctl daemon-reload

        Confirm ollama is listening on all interfaces.

        $ netstat -a | grep 11434
        tcp6 0 0 [::]:11434 [::]:* LISTEN

        Test Connectivity from a remote host. If you are unable to reach the ip/port you may need to modify firewall on the remote system.

        $ telnet 10.1.10.14 11434
        Trying 10.1.10.14…
        Connected to 10.1.10.14.
        Escape character is '^]'.

        Install OpenWeb UI

        I am using the open-webui:cuda container as I am running dual Nvidia GPUs. I want OpenWeb-UI to bind to the primary interface on the host so that I can access it from my workstation.

        # docker run -d -p 10.1.10.14:3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

        The -v option in the command below will create a volume for OpenWeb-UI. You can verify with the command below

        $ docker volume ls
        DRIVER    VOLUME NAME
        local     open-webui
        

        Confirm that the container has started.

        $ docker ps

        If it has failed to start use “docker logs <container-id>” to troubleshoot.

        Now, in a web browser, navigate to the ip/port combo you entered in the docker run command shown above. You should be greeted with the OpenWeb-UI getting started page.


        Install ChromaDB

        Use the command below to instanciate the ChromaDB container. 10.1.10.14 is the ip of my host, adjust as needed to fit your needs

        $ docker run -d \
          --name chromadb \
          -p 10.1.10.14:8000:8000 \
          -v chroma_data:/chroma_db \
          --restart unless-stopped \
          chromadb/chroma

        Confirm you can curl api.

        curl http://10.1.10.14:8000/api/v1
        

        Reinstantiate OpenWebUI with ChromaDB Connectivity

        Stop current OpenWebUI container.

        $ docker stop open-webui

        Delete the container

        $ docker remove open-webui

        Recreate the container but add the following

        1. -e VECTORDB_PROVIDER=chroma
        2. -e CHROMADB_SERVER_HOST=”http://10.1.10.14:8000&#8243; (modify IP to fit your env)
        $ docker run -d -p 10.1.10.14:3000:8080 --gpus all \
        --add-host=host.docker.internal:host-gateway \
        -v open-webui:/app/backend/data \
        -e VECTORDB_PROVIDER=chroma \
        -e CHROMADB_SERVER_HOST="http://10.1.10.14:8000" \
        --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

        Check to ensure that env variables were set correctly

        $ docker exec -it open-webui env | grep -i chroma
        VECTORDB_PROVIDER=chroma
        CHROMADB_SERVER_HOST=http://10.1.10.14:8000
        
        

        Setup RAG in OpenWebUI

        In order to setup RAG you will need to do the following.

        1. Create a Knowledge Base
        2. Upload files
        3. Create the Model that will use the Knowledge Base

        First we will create a knowledge base. Navigate to Workspace > Knowledge > + Create a Knowledge Base. Choose a name for your Knowledge base and add a description.

        We are going to name ours, Gordon Lightfoot

        Now select Create Knowledge

        Look real hard for the text “Drag and drop a file to upload or select a file to view”. This is where you w drag and drop your documents. We have two documents to add to our collection.

        You can also upload entire directories or sync with a directory

        Now navigate to Workspace > Models > + Add New Model. Imput a name for your custom model and choose a base model. I am choosing tinyllama for this test.

        Scroll down and select Save & Create

        Now let’s chat with our new model. Select Workspace and then select your new model (My Gordon Lightfoot Model)

        You are now ready to chat with your model

        Let’s ask it a question

        So there you go. A working RAG implementation with Ollama, OpenWebUI, and VectorDB. I am sure that there are a lot more features to explore here and I am sure I have a lot more tuning to do. But for now I am off to a good start.


        References

        1. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
        2. https://docs.openwebui.com/tutorials/tips/rag-tutorial/