• RHEL 10 – Enable Health Monitoring for NVIDIA GPUs Using DCGM Exporter

    RHEL 10 – Enable Health Monitoring for NVIDIA GPUs Using DCGM Exporter

    Nvidia Datacenter GPU Manager (DCGM) is a suite of tools that provides health monitoring, performance telemetry, and proactive diagnostics for NVIDIA GPUs deployed on bare-metal servers. It is mainly installed for managing a fleet of GPUs across a large number of GPU enabled nodes.

    DCGM does not have a WebUI, and is often seen “exporting” metrics into Grafana or other enterprise observability tools (like Zabbix) via Prometheus Scrape. I have both Grafana and Zabbix in my lab environment so I have a couple of options to where I want to visualize the data.

    In this post, we are going to …

    • Validate GPU Host Setup
    • Install Nvidia DCGM
    • Enable Health Watches via DCGM
    • Setup dcgm-exporter for exporting of GPU metrics

    Nvidia Drivers Install on RHEL 10.1

    A lot has changed in RHEL 10.1 regarding the installation of NVIDIA drivers on RHEL 10. You can now install them via rhel-drivers.

    The post below is pretty throughout on the topic
    https://www.redhat.com/en/blog/introducing-new-and-simplified-ai-accelerator-driver-experience-rhel

    Since I already have the NVIDIA drivers installed on my two GPU enabled nodes, I will just confirm they are both running the same driver version (for consistency) and will ensure the driver is loading properly with nvidia-smi. I will also install CUDA and the nvidia-container-toolkit.


    Host Validation

    In my lab I have 2x Dell R730s with Nvidia Tesla T4s installed (Turing-class GPUs). They are not the most modern GPUs but they are Data center class, can be purchased pretty cheaply used, and only draw ~70w of power and therefore do not require any additional power connections from a riser card or the system board.

    So on each host we will first start off by insuring we have the proper NVIDIA drivers installed.

    [root@columbia ~]# rhel-drivers list
    Available drivers:
    amdgpu:latest
    > nvidia:590.48.01

    Now we run “nvidia-smi” to confirm the driver is loaded and the GPU is recognized.

    nvidia-smi
    Tue Feb 24 10:49:29 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 580.105.08 Driver Version: 580.105.08 CUDA Version: 13.0 |
    +-----------------------------------------+------------------------+----------------------+
    | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
    | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
    | | | MIG M. |
    |=========================================+========================+======================|
    | 0 Tesla T4 Off | 00000000:04:00.0 Off | 0 |
    | N/A 34C P8 13W / 70W | 0MiB / 15360MiB | 0% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+

    Moving on 2 our second host and verify the installed driver.

    [root@prometheus ~]$ rhel-drivers list
    Available drivers:
    amdgpu:latest
    *> nvidia:590.48.01

    And nvidia-smi output, we can see that there are two Telsa T4s in this machine. Nice!

    [root@prometheus ~]$ nvidia-smi
    Tue Feb 24 10:52:55 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 590.48.01 Driver Version: 590.48.01 CUDA Version: 13.1 |
    +-----------------------------------------+------------------------+----------------------+
    | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
    | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
    | | | MIG M. |
    |=========================================+========================+======================|
    | 0 Tesla T4 Off | 00000000:04:00.0 Off | 0 |
    | N/A 30C P8 9W / 70W | 0MiB / 15360MiB | 0% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+
    | 1 Tesla T4 Off | 00000000:06:00.0 Off | 0 |
    | N/A 32C P8 9W / 70W | 0MiB / 15360MiB | 0% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+

    Lets also make sure that we have the same CUDA version installed across hosts.

    [root@prometheus ~]$ rpm -qa | grep cuda-toolkit
    cuda-toolkit-config-common-13.1.80-1.noarch
    cuda-toolkit-13-config-common-13.1.80-1.noarch
    cuda-toolkit-13-1-config-common-13.1.80-1.noarch
    cuda-toolkit-13-1-13.1.1-1.x86_64
    cuda-toolkit-13.1.1-1.x86_64

    We also need to update $PATH for nvcc if we have not done so already.

    cat << 'EOF' | sudo tee /etc/profile.d/cuda.sh
    export PATH=/usr/local/cuda/bin:$PATH
    export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
    EOF

    Then source the path.

    # source /etc/profile.d/cuda.sh

    And now run nvcc.

    nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Tue_Dec_16_07:23:41_PM_PST_2025
    Cuda compilation tools, release 13.1, V13.1.115
    Build cuda_13.1.r13.1/compiler.37061995_0

    We also will need to install the Nvidia Container Toolkit on both GPU hosts, using the RHEL 9 version below.

    sudo dnf config-manager \
    --add-repo=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo

    Once again we are going to skip the gpgcheck. Again this is a homelab.

    sudo dnf install -y --nogpgcheck nvidia-container-toolkit

    And now we validate install on both hosts.

    # nvidia-ctk --version
    NVIDIA Container Toolkit CLI version 1.18.2
    commit: 9e88ed39710fd94c7e49fbb26d96492c45e574fb

    Now we need to generate the CDI specification, as Podman does not use Docker-style run-times, and instead using Container Device Interface (CDI).

    sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    The command above creates /etc/cdi/nvidia.yaml which describes installed GPUs and NVML libraries.
    We now need to restart podman and confirm podman sees our CDI devices.

    cat <<'EOF' | sudo tee /etc/containers/containers.conf.d/99-cdi.conf
    [engine]
    cdi_enabled=true
    cdi_spec_dirs=["/etc/cdi","/var/run/cdi"]
    EOF

    Now we need to verify podman can utilize our GPUs. Again we are running these commands on both our GPU enabled servers.

        podman run --rm --device nvidia.com/gpu=all docker.io/nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi -L
    GPU 0: Tesla T4 (UUID: GPU-9491a3e6-ea29-ba4e-4403-083244d5575c)
    GPU 1: Tesla T4 (UUID: GPU-1d877ac8-5df1-34b0-4f86-59945e37d2ba)
    
    

    DCGM install on RHEL 10

    The install is pretty straight forward, for this post I am performing these steps on both my GPU enabled hosts.

    Add Nvidia Repo

    Note that I could not find DCGM in the RHEL10 repos, so going with RHEL9 repos. Seems to work without issue thus far.

    # dnf config-manager --add-repo \
    https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

    Now install as shown below.

    Note we are skipping the gpgcheck due to RHEL10 newer OpenGPG verification stack. Since this is a lab, and not production, this is acceptable for testing.

    # dnf install -y --nogpgcheck datacenter-gpu-manager-4-core datacenter-gpu-manager-4-proprietary

    Now we can start the service.

    # systemctl enable --now nvidia-dcgm

    And we check that the service is running.

    systemctl status nvidia-dcgm
    ● nvidia-dcgm.service - NVIDIA DCGM service
    Loaded: loaded (/usr/lib/systemd/system/nvidia-dcgm.service; enabled; preset: disabled)
    Active: active (running) since Tue 2026-02-24 10:37:35 EST; 36min ago
    Invocation: da9cd3a2c8a5463a95a3605b68adf253
    Main PID: 2263 (nv-hostengine)
    Tasks: 17 (limit: 1646190)
    Memory: 91.7M (peak: 93.1M)
    CPU: 23.253s
    CGroup: /system.slice/nvidia-dcgm.service
    └─2263 /usr/bin/nv-hostengine -n --service-account nvidia-dcgm
    Feb 24 10:37:35 prometheus.lab systemd[1]: Started nvidia-dcgm.service - NVIDIA DCGM service.
    Feb 24 10:37:37 prometheus.lab nv-hostengine[2263]: DCGM initialized
    Feb 24 10:37:37 prometheus.lab nv-hostengine[2263]: Started host engine version 4.5.2 using port number:>

    dcgmi discovery -l

    Now that dcgmi is installed lets confirm it can see our GPUs.

    This command lists all discovered GPUs. Shows GPU id, PCI BUS ID and Model Name. Should be comparable to the output of nvidia-smi. This command confirms that dcgmi can talk to the host engine.

    dcgmi discovery -l
    2 GPUs found (Active).
    +--------+----------------------------------------------------------------------+
    | GPU ID | Device Information |
    +--------+----------------------------------------------------------------------+
    | 0 | Name: Tesla T4 |
    | | PCI Bus ID: 00000000:04:00.0 |
    | | Device UUID: GPU-9491a3e6-ea29-ba4e-4403-083244d5575c |
    +--------+----------------------------------------------------------------------+
    | 1 | Name: Tesla T4 |
    | | PCI Bus ID: 00000000:06:00.0 |
    | | Device UUID: GPU-1d877ac8-5df1-34b0-4f86-59945e37d2ba |
    +--------+----------------------------------------------------------------------+

    Enable Health Watches

    Health Watches are background checks that DCGM performs on the GPU subsystems. While they are not required for metrics export they do provide additional metrics related to GPU health, so lets enable them.

    dcgmi health -s a
    Health monitor systems set successfully.

    dcgm-exporter

    NVIDIA DCGM Exporter is an open-source tool (container) that collects real-time telemetry data from NVIDIA GPUs—such as utilization, memory usage, temperature, and power consumption—and exposes them in a Prometheus-compatible format.

    We are going to run it via Quadlet ( /etc/containers/systemd/dcgm-exporter.container)

    Now lets start the container

     sudo systemctl daemon-reload
    [root@columbia ~]# sudo systemctl start dcgm-exporter.service
    [root@columbia ~]# podman ps
    CONTAINER ID  IMAGE                                  COMMAND     CREATED         STATUS         PORTS       NAMES
    8804d788f7f5  docker.io/nvidia/dcgm-exporter:latest              10 seconds ago  Up 10 seconds              systemd-dcgm-exporter
    
    

    Lets verify it is listening on port 9400.

    ss -tnlp | grep 9400
    LISTEN 0 4096 *:9400 *:* users:(("dcgm-exporter",pid=135704,fd=24))

    Now lets poke a hole in our firewall to allow the traffic.

    firewall-cmd --add-port=9400/tcp --permanent
    success
    [root@prometheus ~]$ sudo firewall-cmd --reload

    And test locally with curl.

    curl -s -o /dev/null -w "%{http_code}" http://10.1.10.23:9400/metrics

    We can also view the metrics in our browser

    Screenshot of a command-line output displaying NVIDIA GPU statistics including clock frequency, temperature, power usage, and memory utilization.

    Next steps will be to scrape and import.
    Which I will do after lunch.

    Resources

    Learn more from NVIDIA’s official resources:

  • Fix GPG Check Failed Error on RHEL 10.1

    Overview

    On some RHEL 10.1 installs users are running into this error, post-install, when attempting to install packages via dnf.

    Unsure if the issue is isolated to users attempting to install RHEL via the full DVD ISO, from the minimal boot ISO, and users deploying RHEL 10.1 via kickstart.

     GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
    The GPG keys listed for the "Red Hat Enterprise Linux 10 for x86_64 - AppStream (RPMs)" repository are already installed but they are not correct for this package.
    Check that the correct key URLs are configured for this repository.. Failing package is: gnupg2-smime-2.4.5-3.el10_1.x86_64
     GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
    The downloaded packages were saved in cache until the next successful transaction.
    You can remove cached packages by executing 'yum clean packages'.
    Error: GPG check FAILED

    More Details

    The issue is caused by GPG keys installed in /etc/pki/rpm-gpg, specifically RPM-GPG-KEY-redhat-release

    This issue is being tracked via BUG RHEL-144980


    Workaround

    I am currently aware of two workaround, one manual, one much more simple. Details below


    Workaround #1 (Simple)

    Use dnf to install new GPG keys used via dnf

    $ sudo dnf update redhat-release

    After doing so you should run the following

    $ sudo dnf clean all

    You should now be able to update RHEL and install additional RPMs.


    Workaround #2 (manual)

    You can delete the current key in “/etc/pki/rpm-gpg”

    Delete the following

    • RPM-GPG-KEY-redhat-release
    • RPM-GPG-KEY-redhat-beta (if applicable)

    Then SCP known working copies of the keys that you just deleted from a RHEL 10.1 host that is not experiencing this issue. I would suspect an earlier build of RHEL 10.1 or possibly a later build or RHEL 10.1 once available will not experience this issue.

    Once you replace the keys, clean up dnf cache

    $ sudo dnf clean-all
  • Configuring LACP on TP-Link SX3008F for RHEL 9/10

    Configuring LACP on TP-Link SX3008F for RHEL 9/10

    Goal here was to create 3 LACP Port-Channels on a tp-link SX3008F 10gbe switch connecting to RHEL 9/10 hosts also configured to use LACP.

    Specific Requirements

    • Jumbo Frames (mtu 9216)
    • LACP (802.3ad)
    • RHEL side = “xmit_hash_policy=layer3+4”
    • Switch side = src-dst-ip

    End result should be 20Gbe connectivity between hosts for fast NFS backups of Virtual Machines (SSD and HDD NFS) shares). Additional tuning was performed on the NFS host for optimum throughput


    Switch Side Config

    The tplink is Cisco like, but not exactly so commands were a bit of a challenge to nail down specific syntax.

    Jumbo Frames

    On this switch either jumbo frames in on or off for the switch, there is no per port config

    jumbo-size 9216

    Switch Port Config

    1st interface. Both ports added to channel-group 1 which we will define in a later step

    interface ten-gigabitEthernet 1/0/6
    description "columbia bond0 member"
    switchport general allowed vlan 10 untagged
    switchport pvid 10
    no switchport general allowed vlan 1
    channel-group 1 mode active
    exit

    2nd interface

    interface ten-gigabitEthernet 1/0/7
    description "columbia bond0 member"
    switchport general allowed vlan 10 untagged
    switchport pvid 10
    no switchport general allowed vlan 1
    channel-group 1 mode active
    exit

    Define Port-Channel (channel-group)

    interface port-channel 1
    description "columbia bond0"
    switchport general allowed vlan 10 untagged
    switchport pvid 10
    no switchport general allowed vlan 1
    exit

    Load Balance Mode

    The port-channel load-balance src-dst-ip command configures a network switch to distribute traffic across aggregated links (EtherChannel) based on a mathematical hash of both the source and destination IP addresses.

    This method ensures that traffic between the same two IP addresses consistently uses the same link while providing more even distribution compared to relying on MAC addresses alone.

    We will match this config on the server side via “xmit_hash_policy=layer3+4”

    port-channel load-balance src-dst-ip

    Don’t forget to save your work

    
    write memory

    Health Check

    # show etherchannel summary

    Here we can see both ports 6 and 7 are in PO1

    A command line interface output displaying network port flags, statuses, groups, port-channel details, and protocol information.

    # show lacp neighbor

    Command line output displaying LACP neighbor status, including flags, port details, and device information.

    # show vlan id 10

    Here we can see both ports 6 and 7 are in the correct vlan

    Command line output displaying VLAN configuration with ID 10, showing its name, status as active, and associated ports.

    # show interface status ten-gigabitEthernet 1/0/6

    Here we can see that we have 10G link and both ports have correct description

    Command line output displaying the status of two ten-gigabit Ethernet interfaces, including port number, status, speed, duplex, flow control, active medium, and description.

    # show lacp internal

    Terminal output showing LACP (Link Aggregation Control Protocol) internal status, including device flags, channel group information, and details for two ports with their respective states and settings.

    Configuring the RHEL Side

    Here we need to have the following

    • 2 interfaces in a bond
    • 1 bond
    • 1 bridge
      • IP is on bridge
      • bond is connected to bridge

    The Logical Order of Operations

    1. The Bridge (bridge0): The “Top Level” virtual switch that holds the IP address.
    2. The Bond (bond0): The logical aggregation of physical NICS. It is a “Port” of the bridge.
    3. The Bond Ports (enp...): The physical wires. These are “Ports” of the bond.

    Step 1: Create the Bridge (The Anchor)

    You create the bridge first because the bond needs a “controller (aka bond)” to point to.

    nmcli connection add type bridge con-name bridge0 ifname bridge0 \
    ipv4.method manual ipv4.addresses 10.1.10.21/24 ipv4.gateway 10.1.10.1 \
    ipv4.dns 10.1.10.74 ipv6.method disabled \
    802-3-ethernet.mtu 9216

    Step 2: Create the Bond (The Controller)

    Note that the controller is the bridge we just made. We specify LACP (802.3ad) and the hashing policy here.

    nmcli connection add type bond con-name bond0 ifname bond0 \
    connection.controller bridge0 connection.port-type bridge \
    bond.options "mode=802.3ad,xmit_hash_policy=layer3+4" \
    802-3-ethernet.mtu 9216

    Step 3: Attach the Physical Interfaces (The Slaves)

    Now we tell the physical hardware to report to the bond. Crucial: The MTU must be set at this level so the hardware buffers are sized correctly for jumbo frames.

    # First Port
    nmcli connection add type ethernet con-name bond0-port1 ifname enp130s0f0 \
    connection.controller bond0 connection.port-type bond \
    802-3-ethernet.mtu 9216
    # Second Port
    nmcli connection add type ethernet con-name bond0-port2 ifname enp130s0f1 \
    connection.controller bond0 connection.port-type bond \
    802-3-ethernet.mtu 9216

    Step 4: Verification Checklist

    Once created, we bring the stack up from the top. NetworkManager will automatically trigger the underlying ports.

    1. Bring it up: nmcli connection up bridge0
    2. Verify MTU Consistency: Every device in the chain must match.
      • ip link show | grep 9216
    3. Verify LACP Sync: The switch must see the server.
      • cat /proc/net/bonding/bond0 (Look for “Partner Mac Address”)
    4. Verify Bridge Membership:
      • bridge link show (The bond should be listed as a member of the bridge).

    Why this order matters

    • If you define the physical ports first without a controller, they might try to get a DHCP address on their own.
    • By setting MTU 9216 at every single stage of the nmcli command, you prevent the kernel from defaulting any segment to 1500, which causes the exact “packet loss” issue you experienced during the jumbo ping tests.

  • Moving Beyond OMSA: A Guide to Dell iSM Installation on RHEL 10 and PowerEdge R730

    Moving Beyond OMSA: A Guide to Dell iSM Installation on RHEL 10 and PowerEdge R730

    The shift from srvadmin (OMSA) to iSM (iDRAC Service Module) marks the end of bloated, “in-band” server management. This occurred between Dell 12th gen and Dell 13th gen server.

    If you have a 12th Gen Dell Server, you can still leverage Dell srvadmin (Idrac 7). I wrote a post on it here.

    While OMSA ran as a heavy suite of services directly on the host, the modern dcism acts as a lightweight bridge that offloads the heavy lifting to the dedicated iDRAC hardware.

    For RHEL 10 users, this transition is no longer optional but necessary, trading the aging omreport ecosystem for a more secure, stable, and kernel-friendly management experience.

    So in this post we are going to go through the install of DISM (Dell IDRAC Service Module), on RHEL 10, running on a Dell R730.


    Set Up the Dell Repository

    First we need to setup the official Dell System Update (DSU) utility. Install Dell GPG keys when prompted.

    curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi
    bash bootstrap.cgi

    Install the ISM Package

    dnf install dcism

    Also install racadm

    sudo dnf install srvadmin-idracadm7

    Start and Enable the Service

    sudo systemctl enable --now dcismeng.service
    systemctl status dcismeng.service

    Example Commands

    This first command will make sure that DISM is healthy

    /opt/dell/srvadmin/iSM/bin/dcismcfg --getismstatus

    And ensure that racadm is functioning properly

    /opt/dell/srvadmin/sbin/racadm getsysinfo

    You can also reset the idrac from the OS

    /opt/dell/srvadmin/iSM/bin/Invoke-iDRACHardReset

    You can generate a full hardware diagnostic bundle directly from the (TSR).

    Note that the used Dell R730 that I just purchased on Ebay is actually under hardware support till October of 2026 – so if I need on-site hardware support a Dell technician will be dispatched to my house. I will provide snacks and a static strap.

    /opt/dell/srvadmin/iSM/bin/Invoke-SupportAssistCollection

    Note: The SupportAssist Collection feature is not supported on systems earlier than 14th generation. 🙁

    You can check your disks for SMART errors

    /opt/dell/srvadmin/iSM/bin/ismsmartlog --dev=/dev/sdb

    Local RAID & Storage Management

    Since omreport storage is gone, use racadm. With the iSM bridge active, Local RACADM is now your primary interface for your raid controller

    racadm storage get pdisks
    racadm storage get vdisks
    racadm storage get controllers

    The “Full Power Cycle” (The “Cold Boot” Tool) – Flea Power Drain

    On an R730, if you have a hardware component acting up (like a hung NIC), a standard OS reboot often isn’t enough because the motherboard stays powered.

    • What it does: It tells the iDRAC to perform a “virtual” AC power pull. The server will shut down, the flea power will be drained, and it will cold-boot. This is a powerful feature to have in a remote RHEL 10 environment.
     /opt/dell/srvadmin/iSM/bin/Invoke-FullPowerCycle

    Real-Time “Tech Support Report” (TSR)

    If you ever need to open a support ticket with Dell, they will ask for a TSR. Previously, you had to log into the iDRAC web UI and wait 10 minutes.

    • What it does: It triggers the iDRAC to gather every log, firmware version, and hardware event into a .zip file and saves it directly to the iDRAC (or can be pulled to the OS).
    /opt/dell/srvadmin/iSM/bin/Invoke-SupportAssistCollection

    Viewing Active Hardware Alerts

    The racadm utility (which uses the iSM bridge) is the best way to see why your server has an amber light or a “System Health” warning.

    • View the System Event Log (SEL): This is the “Hardware Event Viewer.” It shows power supply failures, memory errors, and thermal trips.
    /opt/dell/srvadmin/sbin/racadm getsel

    Installing perccli

    PERCCLI (PowerEdge RAID Controller Command Line Interface) is the specialized successor to the older MegaCLI tool. While RACADM is great for general server health, PERCCLI is the “surgical tool” specifically for your RAID controller (like the H730 in your R730).

    Install as shown below

    dnf -y install perccli.noarch

    Now you can show all controllers

     /opt/MegaRAID/perccli/perccli64 show

    Show events.

     /opt/MegaRAID/perccli/perccli64 /c0 show eventloginfo

    Show all disks

    /opt/MegaRAID/perccli/perccli64 /c0/eall/sall show

    Epilogue

    So in a nutshell. Dell srvadmin was replaced with DISM starting on Dell 13th gen servers. You have a couple powerful commands at your disposal with a ton of various ways to run both of them.

    Also available is perccli, which is also a nice tool on its own.

  • Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator (OMSA) is Dell’s on-host hardware management and monitoring framework for PowerEdge servers. 

    It runs inside the operating system and provides direct visibility into system hardware such as RAID controllers, physical and virtual disks, power supplies, fans, temperatures, memory, processors, and chassis health. 

    OMSA communicates with the server’s iDRAC and hardware controllers to retrieve real-time status and exposes this information through command-line tools like omreport, optional web interfaces, and SNMP for monitoring systems.

     It is primarily used for hardware diagnostics, RAID and storage monitoring, fault detection, and health reporting, allowing administrators to verify system integrity and troubleshoot hardware issues without leaving the operating system.

    So now that we have an idea of what it is, lets install it.

    Note: Our system under test is a Dell R720. I probably should have lead with that.


    Installation

    First you will need to download the bootstrap script as shown below.

    curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi

    Now we will make it executable.

    chmod +x bootstrap.cgi

    And then we run it.

    sudo ./bootstrap.cgi

    Now we have completed the following.

    • Added Dell’s official repo
    • Installed Dell signing keys
    • Configured OMSA package sources

    Now Install OpenManage (omreport)

    sudo dnf install srvadmin*

    This installs:

    • omreport
    • omconfig
    • Storage monitoring
    • RAID tools
    • CIM providers

    Starting and Enabling Services

    Apparently, service names can vary from OS to OS, so run the command below to verify the correct service names

    systemctl list-unit-files | grep -Ei 'dsm|omsa|srvadmin'
    dsm_om_connsvc.service enabled disabled
    dsm_om_shrsvc.service disabled disabled
    dsm_sa_datamgrd.service enabled disabled
    dsm_sa_eventmgrd.service enabled disabled
    dsm_sa_snmpd.service enabled disabled

    Now that we have the names of the services we can start and enable them as shown below

    # sudo systemctl enable --now dsm_om_connsvc dsm_sa_datamgrd dsm_sa_eventmgrd dsm_sa_snmpd dsm_om_shrsvc

    OpenManage Server Administrator (OMSA) Services
    Service NameFull NamePurposeRequired?Notes
    dsm_om_connsvcOMSA Connection ServiceProvides the web-based interface (HTTPS/1311) used to access OMSA remotely or locally✅ Yes (if using OMSA web UI)This is what allows access via https://<host&gt;:1311
    dsm_om_shrsvcOMSA Shared ServicesProvides shared libraries and backend support for other OMSA components⚠️ UsuallyRequired by other OMSA components; often left disabled unless needed
    dsm_sa_datamgrdSystems Management Data ManagerCollects and maintains hardware inventory and system data✅ YesRequired for hardware monitoring (disks, temps, fans, etc.)
    dsm_sa_eventmgrdSystems Management Event ManagerHandles hardware events, alerts, and logs✅ YesRequired for alerts and log reporting
    dsm_sa_snmpdOMSA SNMP AgentProvides SNMP interface for monitoring tools (LibreNMS, Zabbix, etc.)⚠️ OptionalNeeded only if using SNMP monitoring

    Service Review

    Now let’s review each service in detail, and see how we can leverage each one.


    dsm_om_connsvc

    First we need to ensure our Firewall allows this traffic. So run the commands below (RHEL), ubuntu may use ufw.

    # firewall-cmd --permanent --add-port=1311/tcp
    # firewall-cmd --reload

    Now in your web browser, navigate to https://viper.lab:1311/ (Note https, and change host name accordingly).

    Similar in appearance to the Idrac – you can find system information here, some of which does not appear in the Idrac itself. Everything is read-only.

    Screenshot of Dell OpenManage Server Administrator showing Fan Probes Information, including fan redundancy status and probe list with readings, thresholds, and status indicators.

    dsm_om_shrsvc

    (OMSA Shared Services) is a supporting service, not a management interface or monitoring agent.

    It provides:

    • Shared libraries
    • Inter-process communication
    • Authentication helpers
    • Common backend utilities

    These are required by other OMSA components to function correctly. So nothing really to see here.


    dsm_sa_datamgrd.service

    This is the service that collects all the data that is displayed in the WebUI (port 1311)


    dsm_sa_eventmgrd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    This service relies entirely on the existing system SNMP configuration, so no need to modify anything if you already have snmp configured and running with your specfic community string.


    dsm_sa_snmpd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol. 

    It acts as the interface between the data collected internally by OMSA and third-party monitoring platforms such as LibreNMS, Zabbix, or Nagios. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    dsm_sa_snmpd.service acts as the Dell OMSA integration layer between the server’s hardware telemetry and the system’s SNMP stack, allowing Dell-specific health, storage, and sensor data to be exposed through standard SNMP queries. 

    It does not replace or conflict with existing monitoring agents such as Zabbix or LibreNMS (which I am using) Instead, it complements them by extending the SNMP data they can read

    dsm_sa_snmpd provides access to Dell-specific MIBs that Zabbix Agent and LibreNMS Agent (or any other agent you may use) can query via the system’s SNMP daemon. 

    Basically, dsm_sa_snmpd is the bridge that exposes OMSA’s hardware awareness to your existing monitoring stack.
    Note that if you are using Zabbix you will need to install additional templates. For example, this one, note that I have not tried it yet.

    Ompreport

    Omreport is the command-line interface (CLI) frontend for OMSA. It queries the OMSA backend services and formats the results for humans or scripts

    Note we have to use the absolute path to run this as the working dir is not in our $PATH.

    /opt/dell/srvadmin/sbin/omreport system summary

    If you want to add to your $PATH, run the command below

    echo 'export PATH=$PATH:/opt/dell/srvadmin/sbin' | sudo tee /etc/profile.d/dell-omsa.sh
    source /etc/profile

    Example usage below.

    omreport system summary
    omreport chassis temps
    omreport chassis fans
    omreport chassis power

    Here is an example I found useful. For some reason my raid controller keeps throwing these errors which do not seem to go away despite the fact that both SAS cables are connected (tried switching them) and all disks are healthy.

    # omreport system esmlog
    Severity : Critical
    Date and Time : Sat Jan 31 21:15:18 2026
    Description : The storage BP1 SAS B cable is not connected, or is improperly connected.
    Severity : Ok
    Date and Time : Sat Jan 31 21:15:20 2026
    Description : The chassis is closed while the power is off.
    Severity : Critical
    Date and Time : Sat Jan 31 21:15:21 2026
    Description : The storage BP1 SAS A cable is not connected, or is improperly connected.

    Enter omconfig.


    omconfig

    omconfig is the configuration and control interface for Dell OpenManage Server Administrator (OMSA). 

    Unlike omreport, which is strictly read-only, omconfig is used to change system and OMSA behavior—including alerting, storage actions, and certain hardware-related settings exposed by OMSA. 

    Omconfig It operates by issuing commands to the OMSA backend services (primarily dsm_sa_datamgrd and dsm_sa_eventmgrd) and applying those changes at the software management layer, not directly to hardware firmware. 

    Common uses include enabling or disabling alert actions, configuring storage-related behavior, managing notification settings, and triggering maintenance operations such as consistency checks. 
    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    So back to my two SAS cable alerts. Lets leverage omreport and omconfig to diagnose and hopefully ether determine that there is an actual issue, or suppress the alerts.


    OMCONFIG and OMREPORT – Real world Usage

    First lets take a sneaky-peak at my raid controller.

    omreport storage controller
    Controller PERC H730 Adapter(Slot 4)
    Controller
    ID : 0
    Status : Ok
    Name : PERC H730 Adapter
    Slot ID : PCIe Slot 4
    State : Ready
    Firmware Version : 25.5.9.0001
    -----truncated---

    According to chatgpt I am getting these errors because the system expects that I have an external enclosure connected. I do not think that is the case, as that seems dumb. More than likely this alert is due to the fact that I am using an H730 (a 13G controller) (and its cables) in an R720 (a 12G server) in Slot 4, which is a PCIe Adapter and not the integrated “mini” version.

    So let’s look at the disks and make sure all is well

    Here we can see that the PERC H730 adapter does not “See” the connector states as connected, rather just ready

    omreport storage connector controller=0
    List of Connector(s) on Controller PERC H730 Adapter (Slot 4)
    ID : 0
    Status : Ok
    Name : Connector 0
    State : Ready
    Connector Type : SAS Port RAID Mode
    Termination : Not Applicable
    SCSI Rate : Not Applicable
    ID : 1
    Status : Ok
    Name : Connector 1
    State : Ready
    Connector Type : SAS Port RAID Mode
    Termination : Not Applicable
    SCSI Rate : Not Applicable

    In order to use this 13th gen raid controller in a 12th gen Dell Server I could not use the original SAS cables. Rather I used SFF-8087 to SFF-8643 cables Which can cause the sideband signals (SGPIO / SES) which tell the iDRAC everything is okay which is why we are seeing the errors.

    As stated above this is not actually a problem as I can see my disks. Lets do that below.

    omreport storage pdisk controller=0 | awk '
    /^ID[[:space:]]*:/ {id=$3; state=""; status=""}
    /^State[[:space:]]*:/ {state=$3}
    /^Status[[:space:]]*:/ {status=$3}
    (state!="" && status!="" && id!="") {print id, state, status; id=""; state=""; status=""}
    '
    0:0 Online Non-Critical
    0:1 Online Non-Critical
    0:2 Online Non-Critical
    0:3 Online Non-Critical
    0:4 Online Non-Critical
    0:5 Online Non-Critical
    0:6 Online Non-Critical
    0:7 Online Non-Critical

    Here we can see we have 8 disks, no critical errors

    First lets clear the esmlog with omconfig – this will clear the error till we reboot

    omconfig system esmlog action=clear

    Sadly there is not a permanent fix. I would need to get a different raid controller that uses the original cables.


  • Install APC Netbotz Advanced View on Ubuntu 24.04

    Install APC Netbotz Advanced View on Ubuntu 24.04

    I recently picked up a Netbotz 450 for almost nothing on Ebay. I have always wanted one of these things since I first deployed one in a SunGard facility in Philly about 12-15 years ago.

    These devices allow you to connect environmental sensors to monitor temperatures, rack intrusion, humidity, noise. You can also connect a cool siren light (not cheap) to them and cameras (also super cheap).

    When I got mine up on my lab network, I was disappointing to find that there is no ssh or telnet available to connect to this device. There is a WebUI, but that interface is only for metrics and alerts. Any real configuration (beyond IP – which can be done on via serial console) has to be done in a Java app, and getting this app to run on a modern Linux is a bit of a pain. Fortunately I have a fix.


    First – The Linux Binary

    While the Linux binary is still available for download in some corners of the internet, I decided to make it available via Google Drive download. You can obtain the file here.

    Second – The Wrapper Script

    This script allowed me to install Netbotz Advanced View and and allowed me to configure the Netbotz on Ubuntu 24.04 in Gnome

    #!/bin/bash
    # Wrapper script to run APC NetBotz install.bin
    # This bypasses the LAX launcher which has encoding issues
    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    # Run the installer to extract files
    echo "Extracting installer files..."
    cd "$SCRIPT_DIR"
    ./install.bin 2>/dev/null
    # Find the most recent install.dir.* directory
    INSTALL_DIR=$(ls -td /tmp/install.dir.* 2>/dev/null | head -1)
    if [ -z "$INSTALL_DIR" ]; then
    echo "Error: Could not find extracted installer directory"
    exit 1
    fi
    echo "Found installer directory: $INSTALL_DIR"
    echo "Launching installer in GUI mode..."
    # Run the installer directly with bundled Java
    cd "$INSTALL_DIR"
    "$INSTALL_DIR/Linux/resource/jre/bin/java" \
    -Xmx50m \
    -Xms16m \
    -cp InstallerData:InstallerData/installer.zip \
    com.zerog.ia.installer.Main \
    -i gui
    echo "Installation complete!"


    I have also made the script available for download here.

    Screenshot of a software interface displaying sensor settings and appliance options, including alerts, configuration, and status readings for various sensors.


    Without ssh or telnet access to these things I am unsure of exactly they could be configured at scale, unless snmp-set can be used to utilized to enable snmp or setup ntp or alert destinations. This, I may or may not investigate.

    Epilogue

    Its also worth noting that snmp is apparently not available until you actually connect a sensor to the device, which makes leads me to believe any non-sensor data will not be available via snmp and there is no API

    nmap 10.1.10.9
    Starting Nmap 7.94SVN ( https://nmap.org ) at 2026-01-31 13:32 EST
    Nmap scan report for 10.1.10.9
    Host is up (0.0054s latency).
    Not shown: 998 closed tcp ports (conn-refused)
    PORT STATE SERVICE
    80/tcp open http
    443/tcp open https

    If you are looking to pull data or automate actions, here are your 2 options:

    EcoStruxure IT / Data Center Expert (REST API)

    If you have a larger environment where your NetBotz 450 is managed by Schneider Electric’s Data Center Expert (DCE) software, you can use the DCE REST API.

    • How it works: You send API calls to the DCE server rather than the NetBotz appliance itself.
    • Capability: This allows you to pull data across all monitored devices (including the 450) through a modern JSON-based interface.

    Modbus/TCP

    For integration with Building Management Systems (BMS) or industrial controllers, the NetBotz 450 supports Modbus/TCP.

    Use case: Useful if you need to feed sensor data into a PLC or SCADA system rather than a custom web app or script.

    Other than these two options, you could possibly use python to “log” into urls and set configurations, but I think this would probably be problematic at best.

    Anyway, I should soon have a sensor arrive via mail, so I will get some more hands on with the device.

  • Fixing Weak SSH Ciphers On a Cisco ASAv

    Fixing Weak SSH Ciphers On a Cisco ASAv

    By default the Cisco ASAv has weak ssh config that many modern OS’s do not support.

    ssh admin@10.1.10.56
    Unable to negotiate with 10.1.10.56 port 22: no matching host key type found. Their offer: ssh-rsa
    

    This error is from the client side ssh client refusing to use any of the host key algorithms that are offered by the ASAv.

    To get around this you can try the following.

    ssh -o HostKeyAlgorithms=+ssh-rsa 
    -o PubkeyAcceptedAlgorithms=+ssh-rsa
    admin@10.1.10.56
    The authenticity of host '10.1.10.56 (10.1.10.56)' can't be established.
    RSA key fingerprint is SHA256:TIbF/exW4uqfmy/yf8JyxddX8DApYduK2NVXNHfRBXI.
    This key is not known by any other names.
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added '10.1.10.56' (RSA) to the list of known hosts.
    ssh_dispatch_run_fatal: Connection to 10.1.10.56 port 22: error in libcrypto

    However as you can see from the above output there is another error. This error is also from the client side OS, as its enforcing its own crypto policy.

    One way around this is to enable legacy crypto on the OS side. Which you can do via the command below.

    sudo update-crypto-policies --set LEGACY
    

    However, this is a bad idea and can screw up a lot of other things on the client side. Rather than work around this we will fix.


    Fix SSH on the Cisco ASAv – Instead of Breaking my Workstation

    Cisco documents that starting in ASA 9.16(1), the ASAv added support for EDDSA and ECDSA SSH host keys, and the ASAv will prefer EDDSA → ECDSA → RSA if those keys exist.

    Given I’m on ASAv 9.20(4), I should be able to generate these keys and then connect with my normal ssh client settings. See commands below.

    enable
    configure terminal
    
    ! Generate an Ed25519 (EdDSA) host key (preferred)
    crypto key generate eddsa edwards-curve ed25519
    
    ! Also generate an ECDSA host key (good fallback)
    crypto key generate ecdsa elliptic-curve 256
    
    write memory
    

    Restarting SSH on the ASAv

    Now we need to restart SSH on the ASAv to switch the host key used by SSH. You will need to run these commands from the console.

    conf t
    no ssh 0.0.0.0 0.0.0.0 management
    ssh 0.0.0.0 0.0.0.0 management
    end

    Now we can verify that keys in use on the ASAv. Specifically we want to look at the EDDSA and ECDSA .

    # show crypto key mypubkey ecdsa
    # show crypto key mypubkey eddsa

    Now we can login normally via SSH from our Workstation

     ssh admin@10.1.10.56
    The authenticity of host '10.1.10.56 (10.1.10.56)' can't be established.
    ECDSA key fingerprint is SHA256:Z5+PHzIl6L9BvvdYl2bBrFGi2rlmefmuv0uc27UTBo4.
    This key is not known by any other names.
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added '10.1.10.56' (ECDSA) to the list of known hosts.
    admin@10.1.10.56's password: 
    User admin logged in to ciscoasa
    Logins over the last 1 days: 1.  
    Failed logins since the last login: 0.  
    Type help or '?' for a list of available commands.
    ciscoasa> 
    

    Further Configuration for SSH on the ASAv

    I did find that after all my changes to SSH on the ASAv, I was still unable to ssh to the asav from several VMs (new RHEL10 build for example).

    ssh admin@10.1.10.56
    kex_exchange_identification: read: Connection reset by peer

    In order to get ssh to work from RHEL 10, I ran the commands below on the ASAv

    conf t
    ssh key-exchange group dh-group14-sha256
    ssh key-exchange group curve25519-sha256

    Details below as to why this was required.

    Command AddedWhy RHEL 10 liked it
    dh-group14-sha256This moves the hashing from SHA-1 to SHA-256. RHEL 10 requires SHA-256 or higher for the handshake to be considered “safe.”
    curve25519-sha256This is the preferred algorithm for modern OpenSSH. It is extremely fast and mathematically robust. Once the ASA offered this, RHEL 10 stopped looking for other options and just connected.