• RHEL 10 – Install and Configure NUT UPS Tools and Monitor Via Zabbix.

    RHEL 10 – Install and Configure NUT UPS Tools and Monitor Via  Zabbix.

    In my personal lab I do not have a network managed UPS. My rack mount UPS can only be monitored via USB from a directly connected host.

    This guide details the installation and configuration of Network UPS Tools (NUT) to monitor a Tripp Lite USB PDU and prepare it for Zabbix integration.


    Prerequisites & Installation

    Install the necessary NUT packages. RHEL 10 uses the 2.8.x branch, which introduces an “instantiated” service model.

    sudo dnf install nut nut-client nut-server
    

    Directory Structure Setup

    RHEL 10 defaults to /etc/ups, we will use /etc/nut to ensure our configuration files are in the correct location and we will create a symbolic link for legacy compatibility.

    # Move config if it exists in the old /etc/ups location
    sudo mkdir -p /etc/nut
    sudo mv /etc/ups/* /etc/nut/ 2>/dev/null
    sudo ln -s /etc/nut /etc/ups
    

    Configuration Files

    The following files in /etc/nut/ must be configured:

    nut.conf

    Set the operation mode:

    MODE=netserver
    

    NUT (Network UPS Tools) Netserver Mode is a configuration where one machine (the server) connects to a UPS, monitors it, and then broadcasts the power status and alerts (like “on battery”) across the local network

    ups.conf

    Define the hardware driver. Note the exact spelling of the section name [tripplite].

    [tripplite]
        driver = usbhid-ups
        port = auto
        desc = "Tripp Lite USB PDU"
    

    upsd.conf

    Control network listening. To allow remote connections (like from a Zabbix server), you must listen on the host’s IP or all interfaces (0.0.0.0).

    LISTEN 127.0.0.1 3493
    LISTEN 10.1.10.21 3493
    

    upsd.users

    Define a user for Zabbix or remote monitoring:

    [zabbix]
        password = your_secure_password
        upsmon master
    

    USB Permissions (udev)

    Create a rule to allow the nut user to access the USB bus for Tripp Lite devices (Vendor ID 09ae).

    Create file: /etc/udev/rules.d/99-nut-ups.rules and add content shown below.

    ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="09ae", GROUP="nut", MODE="0660"

    Then apply

    sudo udevadm control --reload-rules && sudo udevadm trigger

    Firewall

    You need to allow inbound port 3493 since we want to be able to poll the ups remotely from a Zabbix host.

    # Open port 3493 permanently
    sudo firewall-cmd --permanent --add-port=3493/tcp
    sudo firewall-cmd --reload

    Service Management

    RHEL 10 uses a specific sequence. You must trigger the “enumerator” to generate the systemd units for your specific PDU name.

    # Force the system to recognize the [tripplite] section
    sudo /usr/libexec/nut-driver-enumerator.sh
    
    # Enable and start the driver instance
    sudo systemctl enable --now nut-driver@tripplite.service
    
    # Start the data server and monitoring daemon
    sudo systemctl enable --now nut-server.service
    sudo systemctl enable --now nut-monitor.service
    

    Manual Check Commands

    Use these commands to verify health:

    CommandPurpose
    upsc -lLists all configured UPS/PDU devices.
    upsc tripplite@localhostDumps all current PDU metrics (voltage, load, etc.).
    systemctl status nut-driver@trippliteChecks if the USB driver is active and running.
    journalctl -u nut-driver@tripplite -fFollow live logs if the USB connection drops.

    Test Remotely

    From another host, install the nut-client. If this step does not work, but the same commands worked locally, you should double check your Firewall settings.

    $ sudo apt install nut-client

    And test

    $ upsc tripplite@10.1.10.21

    Prepare the Zabbix Agent Permissions

    To allow Zabbix to collect this data automatically. Run the two commands below on the host connected to UPS via USB

    # sudo usermod -a -G nut zabbix
    # sudo systemctl restart zabbix-agent

    Create the Zabbix Agent UserParameters

    sudo vi /etc/zabbix/zabbix_agentd.d/nut_tripplite.conf

    Paste the following lines into the file above

    # Discovery rule to find the UPS name
    UserParameter=nut.ups.discovery,upsc -l | awk '{print "{\"{#UPSNAME}\":\"" $1 "\"}"}' | paste -sd "," | sed 's/^/{"data":[/' | sed 's/$/]}/'
    
    # Fetch specific values (e.g., ups.load, output.voltage)
    UserParameter=nut.ups.value[*],upsc $1@localhost $2 | grep -v "Value:"

    Now restart the zabbix agent

    # sudo systemctl restart zabbix-agent

    Now verify that the Zabbix agent returns UPS data

    # zabbix_agentd -t "nut.ups.value[tripplite,ups.load]"
    nut.ups.value[tripplite,ups.load]             [t|24]
    

    Setting up the Zabbix Web UI

    Lets create a simple template for zabbix. Follow the steps shown below.

    Manual Item Creation

    If you just want the most important metrics, go to your Host in Zabbix and create these Items:

    NameTypeKeyType of Information
    PDU LoadZabbix Agentnut.ups.value[tripplite,ups.load]Numeric (float)
    Input VoltageZabbix Agentnut.ups.value[tripplite,input.voltage]Numeric (float)
    Battery ChargeZabbix Agentnut.ups.value[tripplite,battery.charge]Numeric (float)
    PDU StatusZabbix Agentnut.ups.value[tripplite,ups.status]Character

    Configuring SELinux

    If you check the zabbix webui, you are going to see some failures, as we need to configure selinux.

    Note that we configured Zabbix “items” first so that we would be able to capture the failures in the audit log and generate a policy.

    So now that we have failures, we can tell RHEL to look at all recent security denials involving the zabbix_agent and write a “policy” that allows those specific actions.

    # Search the audit log for zabbix-related denials and create a policy module
    grep "zabbix" /var/log/audit/audit.log | audit2allow -M zabbix_nut_final

    Install the Policy

    The previous command created a file called zabbix_nut_final.pp. Now, load it into the kernel:

     semodule -i zabbix_nut_final.pp

    You should now see data returning in the Zabbix WebUI. You may need to wait a few second or re-run the check.

    Screenshot of a Zabbix web interface displaying the item 'Battery Charge' with details on the last check and last value.

  • RHEL 10.1: Installing the command-line assistant powered by RHEL Lightspeed

    RHEL 10.1: Installing the command-line assistant powered by RHEL Lightspeed

    The RHEL command line assistant, powered by RHEL Lightspeed (available in RHEL 9.6 and 10+), is an optional generative AI tool integrated directly into the command line interface (CLI). It is designed to help users, from beginners to experienced administrators, manage, configure, and troubleshoot Red Hat Enterprise Linux systems using natural language.

    It is primarily offered as a connected service, meaning you need internet connectivity, but there is a developer preview of an offline, self-contained version running locally via Podman containers is also available.

    This is my first look at the command-line-assistant, and have yet to explore its performance or configuration in depth.


    Installation

    #cat /etc/redhat-release 
    Red Hat Enterprise Linux release 10.1 (Coughlan)

    Install the “command-line-assistant” service.

    #sudo dnf install -y command-line-assistant

    Clad Service

    You can stop, start, and restart the service

    # systemctl status clad
    ● clad.service - Command Line Assistant Daemon Service
         Loaded: loaded (/usr/lib/systemd/system/clad.service; static)
         Active: active (running) since Sun 2025-12-21 11:15:29 EST; 6min ago
     Invocation: 6730bac46a434e4eb7a3f4e81a1280dd
           Docs: man:clad(8)
       Main PID: 107284 (clad)
          Tasks: 4 (limit: 3355442)
         Memory: 44M (peak: 46M)
            CPU: 1.134s
         CGroup: /system.slice/clad.service
                 └─107284 /usr/bin/python3 /usr/bin/clad

    The configuration file for the service can be found in “/usr/lib/systemd/system/clad.service”

    The main configuration file is “/etc/xdg/command-line-assistant/config.toml”


    Usage

    Usage is simple.

    # c "how to I create a network bridge via nmcli"

    or

     c "where do I find the configuration files for the clad service"

    Reference

    RESOURCEURL
    Use the RHEL command-line assistant offline with this new developer previewhttps://www.redhat.com/en/blog/use-rhel-command-line-assistant-offline-new-developer-preview
    Installing the command-line assistant powered by RHEL Lightspeedhttps://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/interacting_with_the_command-line_assistant_powered_by_rhel_lightspeed/installing-command-line-assistant-powered-by-rhel-lightspeed

  • Mastering Chrony: Essential Commands and Configurations

    Mastering Chrony: Essential Commands and Configurations

    Chrony is a flexible and modern implementation of the Network Time Protocol (NTP). It is designed to synchronize the system clock with NTP servers, reference clocks, or manual input with high accuracy. It excels in environments where the connection to the time source is intermittent or when the system clock is frequently interrupted (such as virtual machines)

    chrony officially replaced the traditional ntpd (NTP Daemon, provided by the legacy ntp package) as the default time synchronization service starting in RHEL 7.

    FeatureChrony (chronyd)ntpd (Legacy)
    Clock CorrectionSlewing Only (Default): Adjusts the clock gradually and precisely. Can handle frequent clock interruptions without losing accuracy.Slewing & Stepping: Slews for small adjustments but uses disruptive stepping (jumping the time) for large offsets, which can impact applications.
    Start-Up TimeFast: Uses makestep at startup to quickly correct large initial errors, then immediately begins slewing.Slow: Waits for the clock to slowly adjust, often taking longer to achieve synchronization.
    NetworkBetter for Intermittent Connections: Handles situations where the server connection is lost or the clock frequently changes (e.g., suspend/resume).Requires Constant Connectivity: Struggles with intermittent network loss and frequent system time changes.
    Resource UseLow: Very small memory footprint and fewer CPU cycles required.Moderate: Larger memory and CPU consumption.
    SecurityMore Secure: Designed with security in mind, often running with fewer privileges.Older Codebase: Has had more vulnerabilities over the years.

    Chrony Basic Commands

    CommandPurposeNotes
    chronyc sourcestats -vShows long-term source stability. Displays long-term statistics for each source, including clock drift and frequency estimation.Useful for spotting an unstable NTP server or bad hardware clock.
    chronyc clientsShows who is using your server. If you are configured as an NTP server, this command lists the hosts that are currently querying you for time (requires the ntpserver directive in your config).Useful for confirming if your NetApp LIF is actually reaching your RHEL box and making a request.
    chronyc makestepForces an immediate time correction. If your clock is severely off (more than a few minutes), this commands the daemon to immediately jump the clock to the correct time, instead of gradually slewing it.Use with Caution! Sudden time jumps can affect applications like databases.
    chronyc ntpdata $IPThis performs a one-time query to the specified server, simulating how the client would communicate with it.$IP seems to need to exist as source in chronyd.conf
    chronyc add server <IP>Adds a server dynamically. Adds a new NTP server to the running configuration without restarting the service.Changes made this way are not permanent and will be lost if chronyd restarts.

    Service Configuration File

    The main configuration file for chronyd is /etc/chrony.conf

    After any config file change, you need to restart the chronyd service

    # systemctl restart chrony

    Show Chrony Clients

    Running “chronyc -n clients -v” will show you the clients currently using a chrony server as a timesource

    ~# chronyc -n clients -v
    Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
    ===============================================================================
    10.1.10.10                     59      0   9   -   115       0      0   -     -
    10.1.10.21                     29      0   9   -    49       0      0   -     -
    10.1.10.49                     44      0   7   -   117       0      0   -     -
    10.1.10.45                     50      0   6   -    10       0      0   -     -
    10.1.10.13                     36      0   7   -    56       0      0   -     -
    10.1.10.18                     39      0   6   -    20       0      0   -     -
    10.1.10.20                     37      0   7   -    27       0      0   -     -
    10.1.10.15                     35      0   7   -    54       0      0   -     -
    10.1.10.50                     24      0   7   -   160       0      0   -     -
    

    Below is a breakdown of the command output

    Hostname

    • The IP address (or hostname) of each NTP client querying this Chrony server.
    • In this case: multiple clients in the 10.1.10.0/24 subnet.

    NTP

    • Total number of valid NTP packets received from that client.
    • This is cumulative since Chrony started.
    • Example: 10.1.10.10 → 59. Means this client has successfully sent 59 NTP requests.

    What to expect

    • Steadily increasing numbers = healthy client
    • Very low numbers = new client or infrequent polling

    Drop (NTP side)

    • Number of NTP packets dropped from this client.
    • Drops occur if packets are malformed, too frequent, or violate limits.

    Int

    • Current polling interval (log2 seconds).
    • Chrony uses exponential polling.
    IntActual Interval
    664 seconds
    7128 seconds
    8256 seconds
    9512 seconds

    IntL

    • Previous polling interval
    • - means no recent change

    Last (NTP side)

    • Seconds since the last NTP packet was received from that client.

    What to expect

    • Should generally be less than or near the polling interval
    • Larger numbers are fine if the client polls slowly

    Command (Cmd) columns

    These relate to Chrony command/control packets, not time sync traffic.

    Cmd

    • Number of chronyc command requests received from that client
    • 0 means the client is only syncing time, not issuing admin commands

    Drop (Cmd)

    • Dropped command packets
    • 0 is ideal

    Int (Cmd)

    • Poll interval for command packets
    • - means none used

    Last (Cmd)

    • Time since last command packet
    • - means no commands received

    Show Chrony Sources

    # chronyc sources -v
    
      .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
     / .- Source state '*' = current best, '+' = combined, '-' = not combined,
    | /             'x' = may be in error, '~' = too variable, '?' = unusable.
    ||                                                 .- xxxx [ yyyy ] +/- zzzz
    ||      Reachability register (octal) -.           |  xxxx = adjusted offset,
    ||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
    ||                                \     |          |  zzzz = estimated error.
    ||                                 |    |           \
    MS Name/IP address         Stratum Poll Reach LastRx Last sample               
    ===============================================================================
    ^- prod-ntp-3.ntp4.ps5.cano>     2  10   377   625  +1096us[+1096us] +/-   49ms
    ^- prod-ntp-4.ntp4.ps5.cano>     2  10   377   504   +144us[ +144us] +/-   51ms
    ^- alphyn.canonical.com          2  10   377   756   +619us[ +804us] +/-   40ms
    ^- prod-ntp-3.ntp1.ps5.cano>     2  10   377   759  +1352us[+1537us] +/-   51ms
    ^+ chi2.us.ntp.li                2  10   377   552  -1204us[-1204us] +/-   20ms
    ^- ovh.maxhost.io                2  10   377   628  -3688us[-3688us] +/-   58ms
    ^* atl-ntp2-0.mattnordhoffd>     2  10   377   741  +1002us[+1188us] +/- 7918us
    ^- 2a01:7e04::2000:5dff:fe1>     4  10   377   739  -1535us[-1535us] +/-   49ms
    
    

    MS (Mode / State)

    • ^ = NTP server
    • = = peer
    • # = local reference clock

    State indicator:

    • * = current best source (in use)
    • + = combined (used with others)
    • - = selectable but not used
    • x = faulty
    • ~ = too variable
    • ? = unreachable

    Stratum

    • Distance from a reference clock (lower is better)
    • GPS/PPS = stratum 0
    • Typical upstream servers = stratum 1–3

    Poll

    • Polling interval (log2 seconds)
    PollInterval
    664 seconds
    7128 seconds
    8256 seconds

    Reach

    • Octal reachability register (last 8 polls)
    • 377 (octal) = perfect reachability
    • Lower values = packet loss or intermittent connectivity

    LastRx

    • Seconds since last successful response
    • First value = adjusted offset
    • Bracketed value = raw measured offset
    • +/- = estimated error margin

    Units:

    • ns = nanoseconds
    • us = microseconds
    • ms = milliseconds

    Show Source Stats

    # chronyc sourcestats -v
                                 .- Number of sample points in measurement set.
                                /    .- Number of residual runs with same sign.
                               |    /    .- Length of measurement set (time).
                               |   |    /      .- Est. clock freq error (ppm).
                               |   |   |      /           .- Est. error in freq.
                               |   |   |     |           /         .- Est. offset.
                               |   |   |     |          |          |   On the -.
                               |   |   |     |          |          |   samples. \
                               |   |   |     |          |          |             |
    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
    ==============================================================================
    prod-ntp-3.ntp4.ps5.cano>  31  14  106m     +0.120      0.212  +1317us   596us
    prod-ntp-4.ntp4.ps5.cano>  33  19  108m     -0.021      0.542   +316us  1235us
    alphyn.canonical.com       33  14  104m     -0.036      0.234   +594us   635us
    prod-ntp-3.ntp1.ps5.cano>  33  19  104m     +0.096      0.342  +1156us   761us
    chi2.us.ntp.li             33  18  107m     -0.087      0.341  -1596us   873us
    ovh.maxhost.io             31  14   96m     +0.147      0.382  -2735us   765us
    atl-ntp2-0.mattnordhoffd>  33  16  104m     +0.083      0.331   +762us   889us
    2a01:7e04::2000:5dff:fe1>  33  19  104m     -0.284      0.513  -1432us  1236us
    

    This shows:

    • Average offset
    • Standard deviation (jitter)
    • Frequency correction
    • Stability over time

    Show Selected Source

    chronyc tracking
    Reference ID    : EE1C3393 (atl-ntp2-0.mattnordhoffdns.net)
    Stratum         : 3
    Ref time (UTC)  : Fri Dec 26 20:54:37 2025
    System time     : 0.000375564 seconds fast of NTP time
    Last offset     : +0.000185856 seconds
    RMS offset      : 0.000497939 seconds
    Frequency       : 33.487 ppm slow
    Residual freq   : +0.008 ppm
    Skew            : 0.299 ppm
    Root delay      : 0.014126181 seconds
    Root dispersion : 0.002563612 seconds
    Update interval : 1033.8 seconds
    Leap status     : Normal
    

    Reference ID

    • The active NTP source currently disciplining your clock.
    • EE1C3393 is the hexadecimal reference identifier.
    • The hostname confirms DNS resolution of the upstream server.

    ✔ This tells you exactly which server is in use.


    Stratum

    • Indicates the distance from a reference clock.
    • Stratum hierarchy:
      • 0 → GPS / atomic clock (not directly used)
      • 1 → Directly connected to stratum 0
      • 2 → Syncing from stratum 1
      • 3 → Syncing from stratum 2

    ✔ Stratum 3 is normal and acceptable for internet-sourced NTP.


    Ref time (UTC)

    • Timestamp of the last successful synchronization sample.
    • Expressed in UTC.

    ✔ Confirms recent and valid sync.


    System time

    • Your local clock is ahead of true NTP time by ~0.38 ms.
    • Chrony is actively correcting this.

    ✔ Sub-millisecond offset is excellent.


    Last offset

    • Offset measured during the most recent update.
    • Positive = system clock was fast.

    ✔ Very small deviation (~0.19 ms).


    RMS offset

    • Root Mean Square of offsets over recent samples.
    • Represents overall synchronization quality.

    ✔ <1 ms RMS is very good for non-GPS NTP.


    Frequency

    • How far your system clock naturally drifts without correction.
    • ppm = parts per million.
    • “Slow” means your hardware clock loses ~33 µs per second.

    ✔ Completely normal for commodity hardware.


    Residual freq

    • Remaining frequency error after corrections.
    • Shows how well Chrony has tuned the clock.

    ✔ Near-zero = excellent convergence.


    Skew

    0.299 ppm
    
    • Estimated uncertainty in frequency measurement.
    • Lower = higher confidence.

    ✔ <1 ppm is strong stability.


    Root delay

    0.014126181 seconds
    
    • Total round-trip delay to the reference clock.
    • Includes network latency through the NTP chain.

    ✔ ~14 ms is expected for internet NTP.


    Root dispersion

    • Estimated maximum error relative to true time.
    • Accumulates over time since last sync.

    ✔ ~2.5 ms is very good.


    Update interval

    • Time between Chrony updates (~17 minutes).
    • Increases as clock stability improves.

    ✔ Indicates a stable and trusted source.


    Leap status

    • No leap second pending or in progress.

    ✔ Required for accurate timekeeping.

  • Dell iDRAC Service Module on RHEL 10.1

    Dell iDRAC Service Module on RHEL 10.1

    The Dell iDRAC Service Module (ISM) is a tool that can be used for better integration between the Dell iDRAC and a running OS. It can provide additional monitoring and metrics to Idrac by brigding the gap between the OS and the underlying Dell hardware.

    iSM collects data from both the operating system and hardware and merges them into the iDRAC Lifecycle Log for consolidated monitoring.


    Downloading

    I used this link to download for RHEL.

    I then scp’d the gzipped tarball to the target system


    Installation on RHEL

    There are not currently packages for RHEL 10, so I went with the RHEL 9 packages. You will need to install 2 rpms from the tarball. First start with the dcism-ocs rpm.

    # dnf -y install ./dcism-osc-7.0.1.0-123.rpm
    

    Next install the dcism rpm.

    # dnf -y install ./dcism-5.0.1.0-2951.el9.x86_64.rpm
    

    I expect that if I was running a recognized OS, such as RHEL 9, I could have used the script setup.sh to install the rpms in the proper order.

    Start the Service

    Use the command below to start/restart the service

    # systemctl restart dcismeng.service

    Checking the output of “ip a” you will see a new interface that I assume is used to communicate back and forth between the OS and the iDRAC.

    12: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
        link/ether ec:2a:72:01:5b:5b brd ff:ff:ff:ff:ff:ff
        altname enp0s20f0u14u3
        altname enxec2a72015b5b
        inet 169.254.1.2/16 brd 169.254.255.255 scope global idrac
           valid_lft forever preferred_lft forever
    

    Enabling Features

    Now that the iDRAC service module is installed we can now run setup.sh should be able to enable and disable features. Note that I did not have much luck in this area, as I was running into issues with missing files which was probably due to the fact that I installed the rpms manually. Either way, you can see what is enabled by default.

    Available features are shown below.

    OS Information via iDRAC

    Now that the service is running and able to communicate with the iDRAC, I can see the following OS level info after logging into the iDRAC


    Additional Resources

    Being new to the iDRAC service module I will link to some relevant documentation which details features, functionality, and configuration.

    RESOURCEURL
    S.M.A.R.T monitoringhttps://www.dell.com/support/manuals/en-ca/idrac-service-module/ism_4.0.1_user_guide/s.m.a.r.t-monitoring?guid=guid-90e39801-c990-43ea-b4a5-4ec0027c019e&lang=en-us
    Configuring iDRAC Service Module from the iDRAC web interface
    https://www.dell.com/support/manuals/en-ca/idrac-service-module/ism_4.0.1_user_guide/configuring-idrac-service-module-from-the-idrac-web-interface?guid=guid-7ca252a8-5491-4637-90b0-2d4b747185b6&lang=en-us

  • Dell Force10 S4810 Factory Reset, OS Upgrade, and Basic Setup

    Dell Force10 S4810 Factory Reset, OS Upgrade, and Basic Setup

    The Dell Force10 S4810 is a 48 sfp port, 10Gbe Switch. Let’s walk through a factory reset and some basic configuration tasks as well as an OS upgrade from OS8 to OS9.


    Factory Reset

    Follow the steps below for factory reset. After running the reload you will be prompted to save the config, ignore this.

    Force10>en
    Force10>#delete flash://startup-config
    Proceed to delete startup-config.bak [confirm yes/no]:yes 
    Force10>reload
    

    Configure Management IP Address

    In order to be able to connect via ssh/telnet to the S4810 we will need to config a management IP. Below we are configuring an IP address on ManagementEthernet 1/0 , which is the ethernet port next to the console port at the front of the switch.

    interface ManagementEthernet 1/0
     ip address 10.1.10.46/24
     no shutdown
    

    We also need to configure the default route.

    Force10(conf)#management route 0.0.0.0/0 10.1.10.1

    You should now be able to ping the management interface on the switch from your workstation. You should also verify that the switch can ping an ip outside your network, such as 8.8.8.8.


    Enable SSH/Telnet

    First we create an user and grant privileges. Replace <MYUSER> and <MYPASS> with the username and password that you wish to use.

    Force10(conf)#username <MYUSER> admin password <MYPASS> privilege 15 access-class permitall
    

    Now we enable ssh as shown below.

    Force10#configure
    Force10(conf)#ip ssh server enable
    Force10(conf)#crypto key generate rsa 
    Enter key size <1024-2048>. Default<1024>  :2048
    Host key already exists .Overwrite (y/n)?y
    Generating SSHv2 RSA key.
    Force10(conf)#exit
    
    

    Now we should confirm the running config for ssh.

    Force10#show ip ssh
    SSH server                : enabled.
    SSH server version        : v1 and v2.
    Password Authentication   : enabled.
    Hostbased Authentication  : disabled.
    RSA       Authentication  : disabled.
    

    Now we configure telnet.

    Force10#conf t
    Force10(conf)#ip telnet server enable
    Force10(conf)#exit

    Now save your running configuration and attempt a remote connection. Note that you might run into the following error when attempting to ssh to the switch.

    $ ssh cpaquin@10.1.10.49
    Unable to negotiate with 10.1.10.49 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    1

    Note that the message above indicates that the SSH client doesn’t support the algorithms used by the switch as they are legacy and potentially weak. We can work around this as shown below. I expect once we upgrade the switch OS with a more modern ssh implementation, we will no longer need the work around.

    $ ssh -o KexAlgorithms=diffie-hellman-group14-sha1 -c 3des-cbc chris@10.1.10.46
    The authenticity of host '10.1.10.46 (10.1.10.46)' can't be established.
    RSA key fingerprint is SHA256:rXqVL/9Ie74e2ZV3m/BiO2KV7adTBXZYwK73UNPEhCo.
    This key is not known by any other names
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added '10.1.10.46' (RSA) to the list of known hosts.
    chris@10.1.10.46's password: 
    Force10>
    

    Note that once you have upgraded to OS9 you will not need to ssh with the -o or -c options. If you are unable to connect via ssh prior to upgrade you can use telnet for the time being.


    Configure NTP

    First let’s set the timezone and show current time.

    Force10(conf)#clock timezone eastern -5 
    Force10#show clock
    20:46:25.954 eastern Sun Apr 6 2025
    

    Now we will configure the switch to sync time with an upstream ntp server, 10.1.10.10

    #ntp server 10.1.10.10

    Then confirm association

    show ntp associations
       remote       vrf-Id     ref clock     st when poll reach   delay   offset    disp
    ====================================================================================
     10.1.10.10      0      0.0.0.0         16    -   16    0     +0.000   +0.000   +0.000
    

    It may take a few minutes for the output below to show “sycronized”

    Force10#show ntp status  
    Clock is synchronized, stratum 2, reference is 216.239.35.0, vrf-id is 0
    frequency is 6.813 ppm, stability is 0.002 ppm, precision is -18
    reference time  eb9dadd7.8477f9b6  Mon, Apr  7 2025  1:50:15.517 UTC
    clock offset is 0.456662 msec, root delay is 13.220 msec
    root dispersion is 1.261 msec, peer dispersion is 0.608 sec
    peer mode is client
    

    OS/Firmware Upgrade

    Dell Open Networking switches use ONIE (Open Network Install Environment) as the bootloader and OS installer, and it relies on U-Boot (Universal Bootloader) for fundamental features like reading/writing boot flash and environment variables.

    Per Dell, in order to upgrade the switch you need to complete the following steps.

    ● Upgrade the S4810 Dell Networking OS Image and Boot Code
    ● Upgrade the CPLD (not needed in this case)

    So let’s first determine what OS version is running on our switch.

    
    #show ver
    Dell Force10 Real Time Operating System Software
    Dell Force10 Operating System Version: 1.0
    Dell Force10 Application Software Version: 8.3.10.3
    Copyright (c) 1999-2011 by Dell Inc.
    Build Time: Tue May 15 22:31:24 PDT 2012
    Build Path: /sites/sjc/work/build/buildSpaces/build06/E8-3-10/SW/SRC/Cp_src/Tacacs
    Force10 uptime is 3 hour(s), 48 minute(s)
    
    System image file is "system://A"
    
    System Type: S4810 
    Control Processor: Freescale QorIQ P2020 with 2147483648 bytes of memory.
    
    128M bytes of boot flash memory.
    
      1 52-port GE/TE/FG (SE)
     48 Ten GigabitEthernet/IEEE 802.3 interface(s)
      4 Forty GigabitEthernet/IEEE 802.3 interface(s)
    

    You can see in the output above that the switch is currently running 8.3.10.3.

    Before upgrading Dell Networking OS on S4810 from a version prior to 8.3.12.0 to version 9.14(1.14), ensure to increase the partition size by upgrading to version 8.3.12.0.

    According to Dell’s documentation I need to upgrade to 8.3.12.0 before I can upgrade to 9.14.


    Upgrading to 8.3.12.0

    I have configured vsftp on my workstation as I will use ftp as my upgrade method. Note that the step below may take a few minutes to complete.

    Force10#upgrade system ftp: a:
    Address or name of remote host []: 10.1.10.10
    Source file name []: FTOS-SE-8.3.12.2.bin
    User name to login remote host: ftpuser
    Password to login remote host: 
    

    Verify the Dell Networking OS has been upgraded correctly in the upgraded flash partition via the command below.

    Force10>show boot system stack-unit all
    
    Current system image information in the system:
    =============================================
    
    Type          Boot Type     A                        B
    ----------------------------------------------------------------
    Stack-unit 0 is not present.
    Stack-unit 1  FLASH BOOT    8.3.12.2[boot]           8.3.7.0                  
    Stack-unit 2 is not present.
    Stack-unit 3 is not present.
    Stack-unit 4 is not present.
    Stack-unit 5 is not present.
    Stack-unit 6 is not present.
    Stack-unit 7 is not present.
    Stack-unit 8 is not present.
    Stack-unit 9 is not present.
    Stack-unit 10 is not present.
    Stack-unit 11 is not present.
    

    Now save and reload.

    Force10#write mem
    Force10#reload
    Proceed with reload [confirm yes/no]: yes
    

    Upgrade the S4810 Boot Code

    Dell Networking OS version 9.14(1.10) requires S4810 Boot Code version 1.2.0.5. Below we can see that we are currently running 1.2.0.2, so we will need to upgrade the boot code as well prior to upgrading to 9.14(1.10).

    Force10>show system stack-unit 1 | grep "Boot Flash"
    Boot Flash      :  1.2.0.2

    Again we will use ftp as our file transfer/upgrade.

    Force10#upgrade boot ftp:
    Address or name of remote host []: 10.1.10.10
    Source file name []: U-boot.1.2.0.5.bin
    User name to login remote host: ftpuser
    Password to login remote host: 
    !
    Erasing SSeries BootImageUpgrade Table of Contents, please wait
    .!........................................!
    524528 bytes successfully copied
    

    Now save and reload

    Force10#write mem  
    !
    Force10#reload
    

    Upgrade to 9.14

    Once the switch as reloaded we are ready to upgrade to 9.14. We will use ftp once again.

    Force10#upgrade system ftp: a:
    Address or name of remote host []: 10.1.10.10
    Source file name []: FTOS-SE-9.14.1.14.bin
    User name to login remote host: ftpuser
    Password to login remote host: 
    

    Now write to memory and reload the OS.

    Force10#write mem
    !
    
    Force10#reload
    Proceed with reload [confirm yes/no]: yes
    

    Wait for the OS to reload and then use “show ver” to check the running OS image.

    Force10>show ver
    Dell Real Time Operating System Software
    Dell Operating System Version:  2.0
    Dell Application Software Version:  9.14(1.14)
    

    References

    1. https://www.dell.com/community/en/conversations/networking-general/s4810-firmware/647f9ce3f4ccf8a8de1135a6
    2. https://www.dell.com/support/kbdoc/en-us/000182885/ftos-for-s-series-s4810-platform?dgc=SM&cid=304571&lid=spr7038757706&refid=sm_LITHIUM_spr7038757706&linkId=167138678
    3. https://www.reddit.com/r/homelab/comments/c9wdnq/dell_force_10_s60_44t_help_round_2/
    4. https://dl.dell.com/manuals/all-products/esuprt_networking_int/esuprt_networking_operating_systems/dell-emc-os-9_release-notes30_en-us.pdf
    5. https://www.dell.com/support/kbdoc/en-us/000122301/how-to-enable-ssh-and-disable-telnet-on-force10-s-series-switches

  • Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Introduction

    In this step-by-steps guide we will replace the out of the box nouveau drivers on RHEL9 with Nvidia Drivers. We will also install the the Nvidia CUDA Toolkit and the Nvidia Container Toolkit.


    GPU and Driver Inspection

    First we need to make sure that our Nvdia GPU is recognized by Red Hat Enterprise Linux 9 (RHEL9).

    lspci -nn | grep -i nvidia
    b6:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [A40] [10de:2235] (rev a1)
    

    Using the command below we can see that we are currently using the non-propietary nouveau driver.

    # lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nouveau
    	Kernel modules: nouveau
    

    Configuring Repositories for the Nvidia Driver Install

    First we need to enable the RHEL9 CodeReady Builder repo. Note we are running these commands as root.

    # subscription-manager repos --enable codeready-builder-for-rhel-9-$(uname -i)-rpms
    

    Next we will need to install and configure the EPEL repo.

    # dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
    

    Now we install the ELRepo project repo – this will provide nvidia-detect which we can utilize later

    # dnf -y  install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm

    Prerequisites for Nvidia Driver Install

    Now we need to install dependencies and build tools.

    # dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc make dkms acpid libglvnd-glx libglvnd-opengl libglvnd-devel pkgconfig
    

    Install Nvidia Drivers

    Install nvidia-detect from the ELRepo project repo.

    # dnf -y install nvidia-detect

    Now install the Nvidia Drivers.

    # dnf -y install $(nvidia-detect)

    Now reboot.


    Confirming Nvidia Driver Installation

    Now lets run the command below one more time.

    [root@gpu ~]# lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [250] Latency Tolerance Reporting
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nvidia
    	Kernel modules: nouveau, nvidia_drm, nvidia
    

    As you can see in the output below, the kernel is loading the Nvidia driver. We can still see nouveau kernel modules listed, but that is fine, as they are not loaded. We can confirm this with the command below.

    # lsmod | grep nouveau

    The above command should not output anything, while the opposite should be true for the command below.

    # lsmod | grep nvidia

    Configure Nvidia Persistenced

    Start and enable nvidia-persistenced.service. This will enable persistence-mode which will keep the nvidia device state from going “stale”

    # systemctl enable nvidia-persistenced.service
    # systemctl start nvidia-persistenced.service
    

    Installing the Nvidia CUDA Toolkit

    We will now follow the official guide and install the Nvidia CUDA toolkit. Per that guide, we need to enable a few repos, however two of those repos should be enabled by default, and the other one we enabled above, however I will list them here for the sake of documentation.

    # subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
    # subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
    # subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

    Now we install the Nvidia repo for the CUDA toolkit.

    dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
    

    Now install the CUDA toolkit as shown below

    # sudo dnf -y install cuda-toolkit

    Confirm that the toolkit is installed and note the version.

    # rpm -qa cuda-toolkit
    cuda-toolkit-12.8.1-1.x86_64
    

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well. Note that the cuda version should match the one that you installed above.

    export PATH=/usr/local/cuda-12.8/bin:$PATH

    Now test nvcc as shown below.

    # nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Feb_21_20:23:50_PST_2025
    Cuda compilation tools, release 12.8, V12.8.93
    Build cuda_12.8.r12.8/compiler.35583870_0
    

    Installing the Nvidia Container Toolkit

    Next we will install the Nvidia Container Toolkit, which allows users to run GPU-accelerated containerized applications.

    A bit about Container Management in RHEL 9

    The default container packages in RHEL 9 are as follows.

    1. Podman – daemonless container image
    2. Buildah – tool for building OCI (Open Container Initiative) container images
    3. Skopeo – tool for managing container images and repos
    4. CRIU – tool to create and save running container checkpoints to disk
    5. Udica – tool for managing SELinux policies for containers

    Installation of the toolkit

    We will follow the instructions as documented here.

    First we configure the repo

    # curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

    Then install via dnf

    # dnf install -y nvidia-container-toolkit

    Configuring the Container Toolkit for Podman

    Generate the CDI specification file using the command below.

    # nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Now lets check the names of the generated device(s).

    # nvidia-ctk cdi list
    INFO[0000] Found 3 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=GPU-7e880be2-891c-72e3-9515-0fd51240e7f4
    nvidia.com/gpu=all
    

    References

    1. https://medium.com/@blackhorseya/step-by-step-guide-to-installing-nvidia-drivers-on-rhel-9-1107e0cd641d
    2. https://access.redhat.com/discussions/227d2101-b4e3-490a-aa1c-601c407ec038
    3. https://darryldias.me/2022/install-nvidia-drivers-on-rhel-9/
    4. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    5. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-rhel-rocky
    6. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html
  • Upgrade Cisco Catalyst WS-C4500X IOS: Step-by-Step Guide

    Upgrade Cisco Catalyst WS-C4500X IOS: Step-by-Step Guide

    I recently picked up a Cisco Catalyst WS-C4500X for my homelab as I am in the process of moving to all 10GBe. Before I put the new switch into service I want to make sure that it is running the latest IOS image.

    I will be using a USB drive that has already been formatted as FAT32 (FAT16 is also acceptable). I have copied the .bin file to the usb drive.

    Plug the USB drive into the front of the switch. Enter enable mode and run dir on the usb drive as shown below.

    Switch>en
    Switch#dir usb0:
    Directory of usb0:/
    
       37  drwx       16384  Mar 22 2025 23:28:06 +00:00  .Trash-1000
       38  -rwx   195351416  Mar 18 2025 01:16:52 +00:00  cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin
    
    31440846848 bytes total (31220072448 bytes free)
    

    Now we need to copy the image from the USB drive to the bootflash on the switch.

    Switch#copy usb0:cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin bootflash:

    The file will take a bit to copy. Once the copy is complete you will be returned to the prompt.

    We now want to confirm that the file is in bootflash.

    Switch#show bootflash:

    Output below, cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin is our new file. As you can see by the dates this switch has not been upgraded in a long while.

    Now we need to change the config-register

    Switch>en
    Password: 
    Switch#conf t
    Enter configuration commands, one per line.  End with CNTL/Z.
    Switch(config)#config-register 0x2102
    

    Now we clear out the contents of the bootflash

    Switch(config)# no boot system

    We must now save the running config in order to see the updated boot vars

    Switch#show boot
    BOOT variable =
    CONFIG_FILE variable does not exist
    BOOTLDR variable does not exist
    Configuration register is 0x2101 (will be 0x2102 at next reload)

    Now we configure the switch to boot the new IOS image.

    Switch(config)# boot system bootflash:cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin

    Now exit config mode and save changes

    Switch(config)#exit
    Switch#copy running-config startup-config
    Destination filename [startup-config]? 
    Building configuration...
    Compressed configuration from 1491 bytes to 878 bytes[OK]

    Now we confirm our boot config

    Switch#show boot
    BOOT variable = bootflash:cat4500e-universalk9.SPA.03.11.11.E.152-7.E11.bin,12;
    CONFIG_FILE variable does not exist
    BOOTLDR variable does not exist
    Configuration register is 0x2101 (will be 0x2102 at next reload)
    

    Configuration register 0x2101 will boot the first boot image shown in the output of dir bootflash:

    Configuration register 0x2102 overrides this behavior and boots the image specified in “BOOT variable” output.


    Reference

    1. https://youtu.be/vaiSC8SzHEc?si=kHM5ak1W_89zHx_j
    2. https://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/50421-config-register-use.html