Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Category: AMD

  • Local vLLM and llama.cpp Dashboard

    Local vLLM and llama.cpp Dashboard

    Introduction

    Over the last several months I have been doing a bit of experimentation locally hosted LLMs, primarily with vLLM, llama.cpp, different quantization methods, GPU configurations, and models. My overall goal is a smart enough local agent that can code and troubleshoot, and ask smarter frontier models for help/review.

    Part of this journey has been learning about all the tuning knobs. As knowing the api is responding is not really enough.

    I also want to know things like:

    • What model is actually loaded?
    • How much VRAM is it consuming?
    • How much VRAM do I still have available?
    • Is the GPU actually doing anything?
    • What context size am I running?
    • How much KV cache is being consumed?
    • What parameters did I start the inference engine with?
    • How fast is the model actually responding?

    Most of this information is available somewhere, but I wanted to not have to hunt and peck for data.

    Normally, I would check Docker, query an API endpoint, run amd-smi (or nvidia-smi), look through vLLM startup logs, inspect environment variables, and then run another script just to benchmark the model. Not hard, but requires a few steps or maybe a script.

    Instead, I built a dashboard.

    The project is called the Inference Engine Dashboard, and the goal is pretty simple: give me one place to see what my local inference server is doing.

    This is not a proxy, or a chat engine. There are already plenty of those.

    This is an operations and monitoring dashboard for vLLM and llama.cpp.

    What I Wanted From the Dashboard

    My primary use case is a workstation running a local LLM while I am also using the machine normally.

    That makes VRAM particularly important, as I need to keep enough VRAM available if I want to fire up the desktop (DE).

    For example, I may intentionally configure vLLM to use less than the maximum available GPU memory because I still need enough VRAM left for my desktop, browser, IDE, and other applications.

    In that environment, I want to be able to open a webpage and immediately see something like:

    • GPU: AMD Radeon AI PRO R9700
    • VRAM used: 23 GB / 32 GB
    • Inference process: 21 GB
    • Other GPU applications: 2 GB
    • Remaining headroom: 9 GB
    • Model: Qwen
    • Context: 32K
    • KV cache utilization
    • GPU utilization
    • GPU temperature
    • Current token throughput

    Basically, the things I normally end up checking manually.

    Local-First Architecture

    I deliberately designed the dashboard to run on the same physical machine as the inference engine as some information is easy to get remotely from the vLLM or llama.cpp API, but detailed hardware and process information is not.

    To accurately determine what is happening on the host, the dashboard may need access to:

    • GPU devices
    • GPU process information
    • Docker container metadata
    • Inference process IDs
    • ROCm libraries
    • Model cache directories
    • Host CPU and memory information

    The application itself runs using Docker Compose.

    docker compose ps

    I also did not want to simply give the application unrestricted access to /var/run/docker.sock, so Docker access is provided through a restricted Docker socket proxy.

    Model caches and other host resources are mounted read-only wherever possible.

    That was an important design decision because eventually I may allow the dashboard to manage inference engines, but I don’t want to jump directly from “monitoring dashboard” to “application that can arbitrarily manipulate containers.”

    Current Dashboard Information

    The main dashboard provides a quick overview of the host and inference environment.

    It includes information such as:

    Host

    • Hostname
    • IP address
    • Operating system
    • Kernel
    • CPU
    • Total system memory

    GPU

    • GPU model
    • GPU utilization
    • Temperature
    • Power consumption
    • VRAM used
    • VRAM available
    • Inference-process VRAM
    • Other GPU process usage

    Inference Engine

    • Active model
    • Backend
    • API health
    • Metrics health
    • Maximum context
    • KV-cache utilization
    • Current throughput
    • Runtime configuration

    The dashboard also displays locally cached models so I can see what is already available on the host.

    If the inference engine or GPU tooling doesn’t expose a particular value, the dashboard displays it as unavailable rather than trying to estimate it.

    Monitoring vLLM

    vLLM exposes quite a bit of useful information, but it isn’t all available from one location.

    The dashboard therefore combines information from several sources, including:

    /health
    /v1/models
    /metrics

    along with:

    • Docker inspection
    • Container environment variables
    • vLLM startup logs
    • GPU process telemetry

    From those sources the dashboard can display information including:

    • Active model
    • Served model name
    • vLLM version
    • Container image
    • Maximum model length
    • GPU memory utilization target
    • Precision
    • Quantization
    • Maximum sequences
    • Maximum batched tokens
    • Prefix caching
    • KV-cache type
    • KV-cache allocation
    • KV-cache utilization
    • KV-cache token capacity
    • CPU offload
    • Swap configuration
    • Maximum concurrency
    • Prompt throughput
    • Output throughput
    • Model-weight VRAM
    • Inference-process VRAM
    • Other GPU VRAM usage
    • Remaining VRAM headroom

    Some of the more detailed memory information is only available in the vLLM startup logs.

    This is another reason the dashboard runs locally.

    If vLLM doesn’t provide enough information to reliably calculate a value, the dashboard simply reports it as unavailable.

    Monitoring llama.cpp

    One of the more interesting parts of the project was adding llama.cpp support as llama.cpp and vLLM expose very different information.

    My first thought was to simply use the same dashboard fields and populate whatever llama.cpp could provide.

    That quickly resulted in a screen containing a whole bunch of:

    N/A
    N/A
    N/A
    N/A

    which isn’t particularly useful.

    So the dashboard now uses provider-specific runtime views.

    For llama.cpp it queries endpoints including:

    /health
    /v1/models
    /props
    /metrics
    /v1/completions

    Depending on what the server exposes, the llama.cpp runtime panel can display:

    • Active GGUF model
    • Configured context window
    • Native model context window
    • GGUF quantization
    • Model parameter count
    • Model file size
    • Parallel slots
    • GPU layers
    • Batch size
    • Microbatch size
    • Temperature
    • Top-K
    • Top-P
    • Min-P
    • Repeat penalty
    • llama.cpp build version
    • Container image
    • Token throughput
    • Inference-process GPU memory

    This makes the llama.cpp view much more useful than trying to force llama.cpp into a vLLM-shaped dashboard.

    GGUF Model Discovery

    The dashboard can also scan a configured directory containing GGUF models.

    For example:

    /models

    or wherever your llama.cpp model library happens to live.

    The directory is mounted read-only into the dashboard container and any discovered GGUF files are displayed in the model selector.

    At the moment this is informational.

    The dashboard can see the models, but it cannot switch to them yet.

    That is intentional.

    AMD GPU Monitoring

    My current workstation uses an AMD GPU, so ROCm support was a requirement from the beginning.

    On AMD systems the dashboard primarily uses:

    amd-smi

    with ROCm SMI available as a fallback.

    In addition to total GPU memory consumption, I wanted to know who was using the VRAM.

    That allows the dashboard to distinguish between:

    Inference Process
    Other GPU Processes
    Unattributed GPU Memory
    Free VRAM

    This is particularly useful on a workstation.

    If my inference engine is configured to consume around 23 GB of a 32 GB GPU, I want to know whether the remaining GPU memory is actually available or whether Chrome, my desktop environment, an IDE, or something else has already consumed part of it.

    The Interactive Benchmark

    I also wanted a quick way to test a model after changing its configuration. Wasn’t looking for a massive benchmark suite.

    Just something that answers:

    “Does this model actually feel fast when I interact with it?“

    The dashboard includes a:

    Run TTFT + token rate test button.

    The benchmark sends a deterministic streaming request through the OpenAI-compatible endpoint.

    /v1/completions

    The current test uses:

    Temperature: 0
    Seed: 1
    Maximum output: 128 tokens
    Concurrency: 1
    Warm-up requests: 0

    The test records three values that I care about for interactive use.

    Time to First Token

    How long did I wait before the model actually started responding?

    Output Tokens Per Second

    Once generation started, how quickly did the model generate tokens?

    End-to-End Latency

    How long did the complete request take?

    The benchmark results are stored in SQLite along with information including:

    • Model
    • Prompt tokens
    • Completion tokens
    • Seed
    • Time to first token
    • Generation time
    • End-to-end latency
    • Output tokens/sec
    • Raw streaming evidence

    That means I can start building some history as I experiment with different models and runtime settings.

    What This Benchmark Is Not

    The built-in benchmark is intended as a quick interactive performance test.It is not intended to replace a proper inference benchmarking suite.

    Currently it does not test:

    • Multiple simultaneous users
    • P50/P95/P99 latency
    • Long-context performance
    • Sustained GPU utilization
    • Warm versus cold requests
    • Maximum batch throughput
    • Extended resource sampling

    Those are all things I may eventually add.

    For now I mainly want to answer:

    I changed something. Did the model get faster or slower?

    For that, the test works pretty well.

    Deployment

    Deployment is intentionally simple.

    Clone the repository, configure the environment, and run:

    ./scripts/deploy.sh

    Host-specific settings are stored in a gitignored:

    .env

    file.

    The same dashboard can be configured for either inference engine by setting:

    INFERENCE_BACKEND=vllm

    or:

    INFERENCE_BACKEND=llama_cpp

    and providing the appropriate API endpoint and model-cache location.

    The containers use Docker’s:

    unless-stopped

    restart policy, so the dashboard will come back automatically after the host reboots and Docker starts.

    Why Model Switching Is Disabled

    You may notice that the dashboard can discover models but currently doesn’t allow you to activate one.

    This is intentional.

    It would actually be fairly easy to add a button that stops an inference container and starts another model.

    The problem is everything that can happen after I push that button.

    For example:

    1. Stop the running inference engine.
    2. Start the new model.
    3. Model doesn’t fit in VRAM.
    4. Container crashes.
    5. API never becomes healthy.
    6. Previous configuration is now gone.
    7. My inference server is offline.

    Not particularly useful.

    Before enabling model switching I want the application to support something closer to a transaction:

    Save current configuration
            ↓
    Stop current model
            ↓
    Start requested model
            ↓
    Validate container
            ↓
    Validate API
            ↓
    Validate model
            ↓
    SUCCESS

    If something fails:

    FAIL
     ↓
    Restore previous configuration
     ↓
    Restart previous model
     ↓
    Validate health

    Once that logic exists, enabling lifecycle controls makes much more sense.

    Security

    Authentication is also not implemented yet.

    So at this point the dashboard should only be exposed on a trusted network.

    That is less concerning while the application is primarily read-only, but it becomes much more important once lifecycle controls are introduced.

    Before enabling model switching or container management I plan to add authentication and additional authorization controls.

    Why I Built It

    This really came out of repeatedly doing the same troubleshooting process.

    I would SSH into the workstation and run something like:

    docker ps

    Then:

    amd-smi

    Then:

    curl http://localhost:8000/v1/models

    Then:

    curl http://localhost:8000/metrics

    Then look through the container logs:

    docker logs <container>

    Then maybe run another script to see how fast the model was responding.

    None of those things are particularly difficult.

    It is just a lot of little pieces of information spread across several different tools.

    Now I can open one page and immediately see:

    • What model is running
    • Whether the inference server is healthy
    • How much VRAM it is consuming
    • What else is using the GPU
    • How much VRAM I have left
    • What context size is configured
    • How much KV cache is being used
    • How the inference engine was configured
    • How quickly the model starts responding
    • How many tokens/sec it is generating

    Which is basically what I wanted.

    What’s Next

    There are several things I would eventually like to add.

    The big one is obviously safe model switching.

    Beyond that I would like to explore:

    • Model configuration profiles
    • Transactional model switching
    • Automatic rollback
    • Authentication
    • Historical GPU metrics
    • Historical VRAM consumption
    • Benchmark history and comparison charts
    • Longer-context benchmarks
    • Concurrent request testing
    • NVIDIA GPU support
    • Additional inference engines

    There is also an interesting possibility of having the dashboard recommend runtime settings based on available GPU memory and the selected model.

    But I don’t want this to become another giant AI management platform.

    The original goal is still the most important one:

    Open one page and understand what my local inference server is doing.

    So far, it has made experimenting with vLLM and llama.cpp considerably easier.

    Resources

    1. Project Repository
      https://github.com/christopherpaquin/vLLM-Llama.ccp-Dashboard
    2. vLLM
      https://github.com/vllm-project/vllm
    3. llama.cpp
      https://github.com/ggml-org/llama.cpp

  • The AMD R9700 – Ubuntu 24.04 Quickstart Guide

    The AMD R9700 – Ubuntu 24.04 Quickstart Guide

    ….and I bought an AMD GPU.

    It’s the first time I’ve gone non-Nvidia since before 2010 as I’ve been a Nvidia loyalist since the GeForce 256 back in 1999 (was 3dfx prior). But Nvidia GPU pricing on anything with decent VRAM has gotten absurd, even on the used market. I kept hearing that ROCm (Radeon Open Compute) support for inference has finally matured. So I decided to find out for myself.

    I picked up the AMD Radeon AI PRO R9700, 32GB. Not cheap, but brand new its cheaper than an RTX 3090 24gb from 2020.

    Since this is my first AMD in very long while, going to have to figure out drivers. So let’s get started. First lets gather some info by running the commands below. 

    lspci -nn | grep -Ei 'VGA|Display|AMD|ATI'
    lsmod | grep amdgpu
    dmesg | grep -i amdgpu | tail -50

    Verify GPU Detection

    Prior to card swap, purged my Ubuntu 24.04 system of all Nvidia drivers and tools.

    Lets check to see if the card is detected properly

    lspci -nn | grep -Ei 'VGA|Display|AMD|ATI'

    This command searches the PCI device list for anything matching VGA, Display, AMD, or ATI. In the output below we see the following

    • 03:00.0 is the PCI id of the R9700 GPU
    • Vendor ID 1002 confirms AMD Device
    • 7551 is how the kernel identifies this GPU generation/device
    • The two PCI bridge entries show the GPU exposes an internal PCIe switch/bridge topology
    01:00.0 PCI bridge: AMD Navi 10 XL Upstream Port of PCI Express Switch
    02:00.0 PCI bridge: AMD Navi 10 XL Downstream Port of PCI Express Switch
    03:00.0 VGA compatible controller: AMD Device [1002:7551]
    03:00.1 Audio device: AMD Device [1002:ab40]

    The important takeaway is that Ubuntu already recognizes the R9700 correctly and the in-kernel AMDGPU driver is successfully initializing it.

    lsmod | grep amdgpu

    This command above shows whether the AMDGPU kernel module is loaded and what other kernel modules depend on it.

    amdgpu 21188608 26

    This tells us:

    • amdgpu is loaded
    • the module occupies roughly 21 MB of kernel module memory
    • 26 references/users currently depend on it
    dmesg | grep -i amdgpu | tail -50

    The command above shows the last 50 kernel messages associated with AMDGPU initialization.

    The most important things we see are:

    MEM ECC is active.

    The GPU’s VRAM ECC capability is active.

    VRAM: 32624M

    The kernel detects approximately 32 GB of VRAM.

    Detected VRAM RAM=32624M, BAR=32768M

    Preparing for ROCm Install

    First lets make sure that the kernel has created the ROCm compute device

     ls -l /dev/kfd
crw-rw---- 1 root render 235, 0 Aug  9 16:20 /dev/kfd

    /dev/kfd is the Kernel Fusion Driver device that ROCm/HIP uses to submit compute workloads to the AMD GPU. AMD’s ROCm installation process relies on access through the render/video device groups.

    In the output above, you can see that the Kernel Fusion Driver is owned by root with group membership of render. We need to add my user to the render group. Note that I had to reboot to pick up the changes.

    sudo usermod -aG render cpaquin

    Now we can add the ROCm 7.2.2 package repo, which is two steps. This one below…

    sudo mkdir --parents --mode=0755 /etc/apt/keyrings
    wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

    And this next one to actually add the repo. 

    sudo tee /etc/apt/sources.list.d/rocm.list <<'EOF'
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2.2 noble main
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/7.2.1/ubuntu noble main
    EOF

    Now let’s be smart and ensure that the repo was added properly.

    cat /etc/apt/sources.list.d/rocm.list
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2.2 noble main
    deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/7.2.1/ubuntu noble main

    Now before we install anything, per AMD recommendation, we need to pin the above repo priority. Pinning means settings which repository Ubuntu prefers when the same package is available from multiple repositories, which means prefer the AMD repos over Ubuntu repos for AMD bits and bobbles.

    sudo tee /etc/apt/preferences.d/rocm-pin-600 <<'EOF'
    Package: *
    Pin: release o=repo.radeon.com
    Pin-Priority: 600
    EOF

    Now we update metadata.

    sudo apt update

    We can see the repo preference via pinning below.

     apt policy rocm
rocm:
  Installed: (none)
  Candidate: 7.2.2.70202-86~24.04
  Version table:
     7.2.2.70202-86~24.04 600
        600 https://repo.radeon.com/rocm/apt/7.2.2 noble/main amd64 Packages

    ROCm Install

    First, lets make sure we have the following installed. 

    sudo apt install python3-setuptools python3-wheel

    Now we can install ROCm. It takes a bit so be patient, or don’t.

    sudo apt install rocm

    Now lets verify that ROCm can see the GPU

    rocminfo

    Truncated output below

    rocminfo
    ROCk module is loaded
    =====================    
    HSA System Attributes    
    =====================    
    Runtime Version:         1.18
    Runtime Ext Version:     1.15
    System Timestamp Freq.:  1000.000000MHz
    Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
    Machine Model:           LARGE                              
    System Endianness:       LITTLE                             
    Mwaitx:                  DISABLED
    XNACK enabled:           NO
    DMAbuf Support:          YES
    VMM Support:             YES
    *******                  
    Agent 2                  
    *******                  
      Name:                    gfx1201                            
      Uuid:                    GPU-407fcdae2bc6b11d               
      Marketing Name:          AMD Radeon AI PRO R9700            
      Vendor Name:             AMD                                
      Feature:                 KERNEL_DISPATCH                    
      Profile:                 BASE_PROFILE                       
      Float Round Mode:        NEAR                               
      Max Queue Number:        128(0x80)                          
      Queue Min Size:          64(0x40)                           
      Queue Max Size:          131072(0x20000)                    
      Queue Type:              MULTI                              
      Node:                    1                                  
      Device Type:             GPU

    And now run amd-smi.

    amd-smi
    +------------------------------------------------------------------------------+
    | AMD-SMI 26.2.2+671d39a71e    amdgpu version: 7.0.0-28 ROCm version: 7.2.2    |
    | VBIOS version: 00158746                                                      |
    | Platform: Linux Baremetal                                                    |
    |-------------------------------------+----------------------------------------|
    | BDF                        GPU-Name | Mem-Uti   Temp   UEC       Power-Usage |
    | GPU  HIP-ID  OAM-ID  Partition-Mode | GFX-Uti    Fan               Mem-Usage |
    |=====================================+========================================|
    | 0000:03:00.0 ...Radeon AI PRO R9700 | 1 %      31 °C   0             9/300 W |
    |   0       0     N/A             N/A | 3 %     20.0 %            661/32624 MB |
    +-------------------------------------+----------------------------------------+
    +------------------------------------------------------------------------------+
    | Processes:                                                                   |
    |  GPU        PID  Process Name          GTT_MEM  VRAM_MEM  MEM_USAGE     CU % |
    |==============================================================================|
    |  No running processes found                                                  |

    What we are looking at:

    • ROCm 7.2.2 is active.
    • The Radeon AI PRO R9700 is correctly identified.
    • Temperature is 31 °C.
    • Power draw is only 9 W / 300 W at idle.
    • Fan is at 20%.
    • VRAM usage is 661 MB / 32,624 MB, which is normal for the display/driver stack.
    • No uncorrectable ECC errors: UEC 0.
    • No compute processes are currently using the GPU.

    Test ROCm inside Docker

    Can a Docker container access the R9700 through ROCm? Lets find out.

    docker run --rm \
      --device=/dev/kfd \
      --device=/dev/dri \
      --group-add 992 \
      --group-add 44 \
      rocm/rocm-terminal:latest \
      rocminfo

    What this does:

    • starts a temporary AMD ROCm container
    • passes /dev/kfd into it for ROCm compute
    • passes /dev/dri into it for the GPU device nodes
    • gives the container the needed video (44) and render (992) group access
    • runs rocminfo inside the container
    • deletes the container afterward because of --rm

    If it works, we should see the same R9700 identification inside Docker that we saw on the host, including something like:

    Pytorch Test

    Next we’ll use AMD’s validated PyTorch container for ROCm 7.2 / Ubuntu 24.04 / Python 3.12 / PyTorch 2.9.1. AMD recommends Docker for PyTorch because the image already contains a tested PyTorch+ROCm combination.

    docker run --rm \
      --device=/dev/kfd \
      --device=/dev/dri \
      --group-add 992 \
      --group-add 44 \
      --security-opt seccomp=unconfined \
      --ipc=host \
      --shm-size 8G \
      rocm/pytorch:rocm7.2_ubuntu24.04_py3.12_pytorch_release_2.9.1 \
      python3 -c "import torch; print('PyTorch:', torch.__version__); print('ROCm:', torch.version.hip); print('GPU available:', torch.cuda.is_available()); print('GPU:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NONE')"

    This will likely download a fairly large container image the first time.

    What we want to see is essentially:

    PyTorch: 2.9.1+rocm7.2.0.git7e1940d4
    ROCm: 7.2.26015-fc0010cf6a
    GPU available: True
    GPU: AMD Radeon AI PRO R9700

    One potentially confusing point: PyTorch keeps the API name torch.cuda even when using AMD ROCm.

    torch.cuda.is_available() returning True means PyTorch has a usable GPU backend; on this container that backend is HIP/ROCm, not NVIDIA CUDA. AMD uses this same test in its ROCm PyTorch verification instructions.

    So the R9700 is now validated for actual ROCm/PyTorch compute inside Docker. Next Step VLLM and test some models.