Category: Linux

  • Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator (OMSA) is Dell’s on-host hardware management and monitoring framework for PowerEdge servers. 

    It runs inside the operating system and provides direct visibility into system hardware such as RAID controllers, physical and virtual disks, power supplies, fans, temperatures, memory, processors, and chassis health. 

    OMSA communicates with the server’s iDRAC and hardware controllers to retrieve real-time status and exposes this information through command-line tools like omreport, optional web interfaces, and SNMP for monitoring systems.

     It is primarily used for hardware diagnostics, RAID and storage monitoring, fault detection, and health reporting, allowing administrators to verify system integrity and troubleshoot hardware issues without leaving the operating system.

    So now that we have an idea of what it is, lets install it.

    Note: Our system under test is a Dell R720. I probably should have lead with that.


    Installation

    First you will need to download the bootstrap script as shown below.

    curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi

    Now we will make it executable.

    chmod +x bootstrap.cgi

    And then we run it.

    sudo ./bootstrap.cgi

    Now we have completed the following.

    • Added Dell’s official repo
    • Installed Dell signing keys
    • Configured OMSA package sources

    Now Install OpenManage (omreport)

    sudo dnf install srvadmin*

    This installs:

    • omreport
    • omconfig
    • Storage monitoring
    • RAID tools
    • CIM providers

    Starting and Enabling Services

    Apparently, service names can vary from OS to OS, so run the command below to verify the correct service names

    systemctl list-unit-files | grep -Ei 'dsm|omsa|srvadmin'
    dsm_om_connsvc.service enabled disabled
    dsm_om_shrsvc.service disabled disabled
    dsm_sa_datamgrd.service enabled disabled
    dsm_sa_eventmgrd.service enabled disabled
    dsm_sa_snmpd.service enabled disabled

    Now that we have the names of the services we can start and enable them as shown below

    # sudo systemctl enable --now dsm_om_connsvc dsm_sa_datamgrd dsm_sa_eventmgrd dsm_sa_snmpd dsm_om_shrsvc

    OpenManage Server Administrator (OMSA) Services
    Service NameFull NamePurposeRequired?Notes
    dsm_om_connsvcOMSA Connection ServiceProvides the web-based interface (HTTPS/1311) used to access OMSA remotely or locally✅ Yes (if using OMSA web UI)This is what allows access via https://<host&gt;:1311
    dsm_om_shrsvcOMSA Shared ServicesProvides shared libraries and backend support for other OMSA components⚠️ UsuallyRequired by other OMSA components; often left disabled unless needed
    dsm_sa_datamgrdSystems Management Data ManagerCollects and maintains hardware inventory and system data✅ YesRequired for hardware monitoring (disks, temps, fans, etc.)
    dsm_sa_eventmgrdSystems Management Event ManagerHandles hardware events, alerts, and logs✅ YesRequired for alerts and log reporting
    dsm_sa_snmpdOMSA SNMP AgentProvides SNMP interface for monitoring tools (LibreNMS, Zabbix, etc.)⚠️ OptionalNeeded only if using SNMP monitoring

    Service Review

    Now let’s review each service in detail, and see how we can leverage each one.


    dsm_om_connsvc

    First we need to ensure our Firewall allows this traffic. So run the commands below (RHEL), ubuntu may use ufw.

    # firewall-cmd --permanent --add-port=1311/tcp
    # firewall-cmd --reload

    Now in your web browser, navigate to https://viper.lab:1311/ (Note https, and change host name accordingly).

    Similar in appearance to the Idrac – you can find system information here, some of which does not appear in the Idrac itself. Everything is read-only.

    Screenshot of Dell OpenManage Server Administrator showing Fan Probes Information, including fan redundancy status and probe list with readings, thresholds, and status indicators.

    dsm_om_shrsvc

    (OMSA Shared Services) is a supporting service, not a management interface or monitoring agent.

    It provides:

    • Shared libraries
    • Inter-process communication
    • Authentication helpers
    • Common backend utilities

    These are required by other OMSA components to function correctly. So nothing really to see here.


    dsm_sa_datamgrd.service

    This is the service that collects all the data that is displayed in the WebUI (port 1311)


    dsm_sa_eventmgrd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    This service relies entirely on the existing system SNMP configuration, so no need to modify anything if you already have snmp configured and running with your specfic community string.


    dsm_sa_snmpd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol. 

    It acts as the interface between the data collected internally by OMSA and third-party monitoring platforms such as LibreNMS, Zabbix, or Nagios. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    dsm_sa_snmpd.service acts as the Dell OMSA integration layer between the server’s hardware telemetry and the system’s SNMP stack, allowing Dell-specific health, storage, and sensor data to be exposed through standard SNMP queries. 

    It does not replace or conflict with existing monitoring agents such as Zabbix or LibreNMS (which I am using) Instead, it complements them by extending the SNMP data they can read

    dsm_sa_snmpd provides access to Dell-specific MIBs that Zabbix Agent and LibreNMS Agent (or any other agent you may use) can query via the system’s SNMP daemon. 

    Basically, dsm_sa_snmpd is the bridge that exposes OMSA’s hardware awareness to your existing monitoring stack.
    Note that if you are using Zabbix you will need to install additional templates. For example, this one, note that I have not tried it yet.

    Ompreport

    Omreport is the command-line interface (CLI) frontend for OMSA. It queries the OMSA backend services and formats the results for humans or scripts

    Note we have to use the absolute path to run this as the working dir is not in our $PATH.

    /opt/dell/srvadmin/sbin/omreport system summary

    If you want to add to your $PATH, run the command below

    echo 'export PATH=$PATH:/opt/dell/srvadmin/sbin' | sudo tee /etc/profile.d/dell-omsa.sh
    source /etc/profile

    Example usage below.

    omreport system summary
    omreport chassis temps
    omreport chassis fans
    omreport chassis power

    Here is an example I found useful. For some reason my raid controller keeps throwing these errors which do not seem to go away despite the fact that both SAS cables are connected (tried switching them) and all disks are healthy.

    # omreport system esmlog
    Severity : Critical
    Date and Time : Sat Jan 31 21:15:18 2026
    Description : The storage BP1 SAS B cable is not connected, or is improperly connected.
    Severity : Ok
    Date and Time : Sat Jan 31 21:15:20 2026
    Description : The chassis is closed while the power is off.
    Severity : Critical
    Date and Time : Sat Jan 31 21:15:21 2026
    Description : The storage BP1 SAS A cable is not connected, or is improperly connected.

    Enter omconfig.


    omconfig

    omconfig is the configuration and control interface for Dell OpenManage Server Administrator (OMSA). 

    Unlike omreport, which is strictly read-only, omconfig is used to change system and OMSA behavior—including alerting, storage actions, and certain hardware-related settings exposed by OMSA. 

    Omconfig It operates by issuing commands to the OMSA backend services (primarily dsm_sa_datamgrd and dsm_sa_eventmgrd) and applying those changes at the software management layer, not directly to hardware firmware. 

    Common uses include enabling or disabling alert actions, configuring storage-related behavior, managing notification settings, and triggering maintenance operations such as consistency checks. 
    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    So back to my two SAS cable alerts. Lets leverage omreport and omconfig to diagnose and hopefully ether determine that there is an actual issue, or suppress the alerts.


    OMCONFIG and OMREPORT – Real world Usage

    First lets take a sneaky-peak at my raid controller.

    omreport storage controller
    Controller PERC H730 Adapter(Slot 4)
    Controller
    ID : 0
    Status : Ok
    Name : PERC H730 Adapter
    Slot ID : PCIe Slot 4
    State : Ready
    Firmware Version : 25.5.9.0001
    -----truncated---

    According to chatgpt I am getting these errors because the system expects that I have an external enclosure connected. I do not think that is the case, as that seems dumb. More than likely this alert is due to the fact that I am using an H730 (a 13G controller) (and its cables) in an R720 (a 12G server) in Slot 4, which is a PCIe Adapter and not the integrated “mini” version.

    So let’s look at the disks and make sure all is well

    Here we can see that the PERC H730 adapter does not “See” the connector states as connected, rather just ready

    omreport storage connector controller=0
    List of Connector(s) on Controller PERC H730 Adapter (Slot 4)
    ID : 0
    Status : Ok
    Name : Connector 0
    State : Ready
    Connector Type : SAS Port RAID Mode
    Termination : Not Applicable
    SCSI Rate : Not Applicable
    ID : 1
    Status : Ok
    Name : Connector 1
    State : Ready
    Connector Type : SAS Port RAID Mode
    Termination : Not Applicable
    SCSI Rate : Not Applicable

    In order to use this 13th gen raid controller in a 12th gen Dell Server I could not use the original SAS cables. Rather I used SFF-8087 to SFF-8643 cables Which can cause the sideband signals (SGPIO / SES) which tell the iDRAC everything is okay which is why we are seeing the errors.

    As stated above this is not actually a problem as I can see my disks. Lets do that below.

    omreport storage pdisk controller=0 | awk '
    /^ID[[:space:]]*:/ {id=$3; state=""; status=""}
    /^State[[:space:]]*:/ {state=$3}
    /^Status[[:space:]]*:/ {status=$3}
    (state!="" && status!="" && id!="") {print id, state, status; id=""; state=""; status=""}
    '
    0:0 Online Non-Critical
    0:1 Online Non-Critical
    0:2 Online Non-Critical
    0:3 Online Non-Critical
    0:4 Online Non-Critical
    0:5 Online Non-Critical
    0:6 Online Non-Critical
    0:7 Online Non-Critical

    Here we can see we have 8 disks, no critical errors

    First lets clear the esmlog with omconfig – this will clear the error till we reboot

    omconfig system esmlog action=clear

    Sadly there is not a permanent fix. I would need to get a different raid controller that uses the original cables.


  • Mastering Chrony: Essential Commands and Configurations

    Mastering Chrony: Essential Commands and Configurations

    Chrony is a flexible and modern implementation of the Network Time Protocol (NTP). It is designed to synchronize the system clock with NTP servers, reference clocks, or manual input with high accuracy. It excels in environments where the connection to the time source is intermittent or when the system clock is frequently interrupted (such as virtual machines)

    chrony officially replaced the traditional ntpd (NTP Daemon, provided by the legacy ntp package) as the default time synchronization service starting in RHEL 7.

    FeatureChrony (chronyd)ntpd (Legacy)
    Clock CorrectionSlewing Only (Default): Adjusts the clock gradually and precisely. Can handle frequent clock interruptions without losing accuracy.Slewing & Stepping: Slews for small adjustments but uses disruptive stepping (jumping the time) for large offsets, which can impact applications.
    Start-Up TimeFast: Uses makestep at startup to quickly correct large initial errors, then immediately begins slewing.Slow: Waits for the clock to slowly adjust, often taking longer to achieve synchronization.
    NetworkBetter for Intermittent Connections: Handles situations where the server connection is lost or the clock frequently changes (e.g., suspend/resume).Requires Constant Connectivity: Struggles with intermittent network loss and frequent system time changes.
    Resource UseLow: Very small memory footprint and fewer CPU cycles required.Moderate: Larger memory and CPU consumption.
    SecurityMore Secure: Designed with security in mind, often running with fewer privileges.Older Codebase: Has had more vulnerabilities over the years.

    Chrony Basic Commands

    CommandPurposeNotes
    chronyc sourcestats -vShows long-term source stability. Displays long-term statistics for each source, including clock drift and frequency estimation.Useful for spotting an unstable NTP server or bad hardware clock.
    chronyc clientsShows who is using your server. If you are configured as an NTP server, this command lists the hosts that are currently querying you for time (requires the ntpserver directive in your config).Useful for confirming if your NetApp LIF is actually reaching your RHEL box and making a request.
    chronyc makestepForces an immediate time correction. If your clock is severely off (more than a few minutes), this commands the daemon to immediately jump the clock to the correct time, instead of gradually slewing it.Use with Caution! Sudden time jumps can affect applications like databases.
    chronyc ntpdata $IPThis performs a one-time query to the specified server, simulating how the client would communicate with it.$IP seems to need to exist as source in chronyd.conf
    chronyc add server <IP>Adds a server dynamically. Adds a new NTP server to the running configuration without restarting the service.Changes made this way are not permanent and will be lost if chronyd restarts.

    Service Configuration File

    The main configuration file for chronyd is /etc/chrony.conf

    After any config file change, you need to restart the chronyd service

    # systemctl restart chrony

    Show Chrony Clients

    Running “chronyc -n clients -v” will show you the clients currently using a chrony server as a timesource

    ~# chronyc -n clients -v
    Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
    ===============================================================================
    10.1.10.10                     59      0   9   -   115       0      0   -     -
    10.1.10.21                     29      0   9   -    49       0      0   -     -
    10.1.10.49                     44      0   7   -   117       0      0   -     -
    10.1.10.45                     50      0   6   -    10       0      0   -     -
    10.1.10.13                     36      0   7   -    56       0      0   -     -
    10.1.10.18                     39      0   6   -    20       0      0   -     -
    10.1.10.20                     37      0   7   -    27       0      0   -     -
    10.1.10.15                     35      0   7   -    54       0      0   -     -
    10.1.10.50                     24      0   7   -   160       0      0   -     -
    

    Below is a breakdown of the command output

    Hostname

    • The IP address (or hostname) of each NTP client querying this Chrony server.
    • In this case: multiple clients in the 10.1.10.0/24 subnet.

    NTP

    • Total number of valid NTP packets received from that client.
    • This is cumulative since Chrony started.
    • Example: 10.1.10.10 → 59. Means this client has successfully sent 59 NTP requests.

    What to expect

    • Steadily increasing numbers = healthy client
    • Very low numbers = new client or infrequent polling

    Drop (NTP side)

    • Number of NTP packets dropped from this client.
    • Drops occur if packets are malformed, too frequent, or violate limits.

    Int

    • Current polling interval (log2 seconds).
    • Chrony uses exponential polling.
    IntActual Interval
    664 seconds
    7128 seconds
    8256 seconds
    9512 seconds

    IntL

    • Previous polling interval
    • - means no recent change

    Last (NTP side)

    • Seconds since the last NTP packet was received from that client.

    What to expect

    • Should generally be less than or near the polling interval
    • Larger numbers are fine if the client polls slowly

    Command (Cmd) columns

    These relate to Chrony command/control packets, not time sync traffic.

    Cmd

    • Number of chronyc command requests received from that client
    • 0 means the client is only syncing time, not issuing admin commands

    Drop (Cmd)

    • Dropped command packets
    • 0 is ideal

    Int (Cmd)

    • Poll interval for command packets
    • - means none used

    Last (Cmd)

    • Time since last command packet
    • - means no commands received

    Show Chrony Sources

    # chronyc sources -v
    
      .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
     / .- Source state '*' = current best, '+' = combined, '-' = not combined,
    | /             'x' = may be in error, '~' = too variable, '?' = unusable.
    ||                                                 .- xxxx [ yyyy ] +/- zzzz
    ||      Reachability register (octal) -.           |  xxxx = adjusted offset,
    ||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
    ||                                \     |          |  zzzz = estimated error.
    ||                                 |    |           \
    MS Name/IP address         Stratum Poll Reach LastRx Last sample               
    ===============================================================================
    ^- prod-ntp-3.ntp4.ps5.cano>     2  10   377   625  +1096us[+1096us] +/-   49ms
    ^- prod-ntp-4.ntp4.ps5.cano>     2  10   377   504   +144us[ +144us] +/-   51ms
    ^- alphyn.canonical.com          2  10   377   756   +619us[ +804us] +/-   40ms
    ^- prod-ntp-3.ntp1.ps5.cano>     2  10   377   759  +1352us[+1537us] +/-   51ms
    ^+ chi2.us.ntp.li                2  10   377   552  -1204us[-1204us] +/-   20ms
    ^- ovh.maxhost.io                2  10   377   628  -3688us[-3688us] +/-   58ms
    ^* atl-ntp2-0.mattnordhoffd>     2  10   377   741  +1002us[+1188us] +/- 7918us
    ^- 2a01:7e04::2000:5dff:fe1>     4  10   377   739  -1535us[-1535us] +/-   49ms
    
    

    MS (Mode / State)

    • ^ = NTP server
    • = = peer
    • # = local reference clock

    State indicator:

    • * = current best source (in use)
    • + = combined (used with others)
    • - = selectable but not used
    • x = faulty
    • ~ = too variable
    • ? = unreachable

    Stratum

    • Distance from a reference clock (lower is better)
    • GPS/PPS = stratum 0
    • Typical upstream servers = stratum 1–3

    Poll

    • Polling interval (log2 seconds)
    PollInterval
    664 seconds
    7128 seconds
    8256 seconds

    Reach

    • Octal reachability register (last 8 polls)
    • 377 (octal) = perfect reachability
    • Lower values = packet loss or intermittent connectivity

    LastRx

    • Seconds since last successful response
    • First value = adjusted offset
    • Bracketed value = raw measured offset
    • +/- = estimated error margin

    Units:

    • ns = nanoseconds
    • us = microseconds
    • ms = milliseconds

    Show Source Stats

    # chronyc sourcestats -v
                                 .- Number of sample points in measurement set.
                                /    .- Number of residual runs with same sign.
                               |    /    .- Length of measurement set (time).
                               |   |    /      .- Est. clock freq error (ppm).
                               |   |   |      /           .- Est. error in freq.
                               |   |   |     |           /         .- Est. offset.
                               |   |   |     |          |          |   On the -.
                               |   |   |     |          |          |   samples. \
                               |   |   |     |          |          |             |
    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
    ==============================================================================
    prod-ntp-3.ntp4.ps5.cano>  31  14  106m     +0.120      0.212  +1317us   596us
    prod-ntp-4.ntp4.ps5.cano>  33  19  108m     -0.021      0.542   +316us  1235us
    alphyn.canonical.com       33  14  104m     -0.036      0.234   +594us   635us
    prod-ntp-3.ntp1.ps5.cano>  33  19  104m     +0.096      0.342  +1156us   761us
    chi2.us.ntp.li             33  18  107m     -0.087      0.341  -1596us   873us
    ovh.maxhost.io             31  14   96m     +0.147      0.382  -2735us   765us
    atl-ntp2-0.mattnordhoffd>  33  16  104m     +0.083      0.331   +762us   889us
    2a01:7e04::2000:5dff:fe1>  33  19  104m     -0.284      0.513  -1432us  1236us
    

    This shows:

    • Average offset
    • Standard deviation (jitter)
    • Frequency correction
    • Stability over time

    Show Selected Source

    chronyc tracking
    Reference ID    : EE1C3393 (atl-ntp2-0.mattnordhoffdns.net)
    Stratum         : 3
    Ref time (UTC)  : Fri Dec 26 20:54:37 2025
    System time     : 0.000375564 seconds fast of NTP time
    Last offset     : +0.000185856 seconds
    RMS offset      : 0.000497939 seconds
    Frequency       : 33.487 ppm slow
    Residual freq   : +0.008 ppm
    Skew            : 0.299 ppm
    Root delay      : 0.014126181 seconds
    Root dispersion : 0.002563612 seconds
    Update interval : 1033.8 seconds
    Leap status     : Normal
    

    Reference ID

    • The active NTP source currently disciplining your clock.
    • EE1C3393 is the hexadecimal reference identifier.
    • The hostname confirms DNS resolution of the upstream server.

    ✔ This tells you exactly which server is in use.


    Stratum

    • Indicates the distance from a reference clock.
    • Stratum hierarchy:
      • 0 → GPS / atomic clock (not directly used)
      • 1 → Directly connected to stratum 0
      • 2 → Syncing from stratum 1
      • 3 → Syncing from stratum 2

    ✔ Stratum 3 is normal and acceptable for internet-sourced NTP.


    Ref time (UTC)

    • Timestamp of the last successful synchronization sample.
    • Expressed in UTC.

    ✔ Confirms recent and valid sync.


    System time

    • Your local clock is ahead of true NTP time by ~0.38 ms.
    • Chrony is actively correcting this.

    ✔ Sub-millisecond offset is excellent.


    Last offset

    • Offset measured during the most recent update.
    • Positive = system clock was fast.

    ✔ Very small deviation (~0.19 ms).


    RMS offset

    • Root Mean Square of offsets over recent samples.
    • Represents overall synchronization quality.

    ✔ <1 ms RMS is very good for non-GPS NTP.


    Frequency

    • How far your system clock naturally drifts without correction.
    • ppm = parts per million.
    • “Slow” means your hardware clock loses ~33 µs per second.

    ✔ Completely normal for commodity hardware.


    Residual freq

    • Remaining frequency error after corrections.
    • Shows how well Chrony has tuned the clock.

    ✔ Near-zero = excellent convergence.


    Skew

    0.299 ppm
    
    • Estimated uncertainty in frequency measurement.
    • Lower = higher confidence.

    ✔ <1 ppm is strong stability.


    Root delay

    0.014126181 seconds
    
    • Total round-trip delay to the reference clock.
    • Includes network latency through the NTP chain.

    ✔ ~14 ms is expected for internet NTP.


    Root dispersion

    • Estimated maximum error relative to true time.
    • Accumulates over time since last sync.

    ✔ ~2.5 ms is very good.


    Update interval

    • Time between Chrony updates (~17 minutes).
    • Increases as clock stability improves.

    ✔ Indicates a stable and trusted source.


    Leap status

    • No leap second pending or in progress.

    ✔ Required for accurate timekeeping.

  • Essential Python Pip Commands for Linux Administrators

    Essential Python Pip Commands for Linux Administrators

    Pip is the Python package/module installer for Linux and Windows.

    Installing Pip

    On Ubuntu, use the command shown below to install pip via apt. This makes pip available system wide.

    $ sudo apt install python3-pip

    When you are working in a Python virtual environment, pip is automatically installed. More details on Python virtual environments later.


    Installing & Upgrading Packages

    Upgrading pip

    sudo python3 -m pip install --upgrade pip

    Install a package via cli, specifying the package name.

    $ pip install <package name>

    Install a package via cli, use the -r flag and a file listing package names.

    $ pip install -r <filename>

    Install a file from a specific repository, use the -i flag and the repository URL.

    $ pip install <package name> -i https://my-favorite-repo:port

    Upgrade an existing package with the -U flag.

    $ pip install numpy -U

    List all installed packages that have newer versions available.

    $ pip list --outdated

    Uninstalling Dependencies with Autoremove

    NOTE: When you install a package using pip, it automatically installs all required dependencies. However, when you uninstall the package, pip does not remove its dependencies. At least not by default. For this task you can use pip-autoremove.

    Install pip-autoremove as shown below.

    $ pip install pip-autoremove

    Then use pip-autoremove to uninstall.

    $ pip-autoremove <package-name> -y


    Working with Package Versions

    Install a specific version of a package (upgrade or downgrade)

    $ pip install numpy==1.21.0

    List all available versions of a package

    $ pip index versions numpy


    Removing Packages

    Remove an installed package, specifying the package name

    $ pip uninstall numpy

    Remove a list of packages in a file using the -r flag

    $ pip uninstall -r <filename>

    Listing Installed Packages

    Use pip list to show all installed packages and their versions

    $ pip list

    Use pip freeze to list all installed python packages, along with their version. Freeze can be used to create a requirements file in order to re-create and environment with specific package versions.

    $ pip freeze

    Use the show command line argument to find out more information regarding a package, installed or not. Output will show version, summary, author, etc.

    $ pip show <package name>

    Searching for Available Packages

    In order to search for packages for install, you need to search on https://pypi.org/search directly.


    Working with PIP in virtual Environments

    A python virtual env is a self-contained directory that isolates a specific set of python packages and their versions. Used mainly in the context of a project.

    Use the command below to create a virtual environment. In this example we are creating virtual environment foo

    $ python3 -m venv foo

    Then source the newly created environment file to activate

    $ source foo/bin/activate

    To exit the environment without closing your shell, use deactivate

    $ deactivate

    The command below will uninstall packages in a virtual env.

    $ pipenv uninstall <packagename>

    PipX

    Pipx is a tool for installing and running Python applications. It is particularly useful for installing command-line tools written in Python without polluting your system-wide Python environment.. Pipx is built on top of pip.

    Installing pipx on Ubuntu

    $ sudo apt install pipx

    Once installed, run the command below.

    $ pipx ensurepath

    Output below

    When to Use pipx

    • When you need CLI tools written in Python without installing them system-wide.
    • When you want to avoid dependency conflicts between different Python tools.
    • When you frequently install and remove Python CLI applications.

    Pip vs pipx

    Featurepippipx
    PurposeInstalls Python packages globally or in a virtual environment.Installs and runs Python CLI applications in isolated environments.
    Installation LocationInstalls packages system-wide or in a virtual environment (venv).Each package is installed in its own isolated virtual environment.
    Dependency ManagementCan cause dependency conflicts if installed system-wide.Avoids conflicts by keeping each package separate.
    Best Use CaseInstalling Python libraries for development (e.g., numpy, pandas, requests).Installing CLI tools (e.g., black, poetry, httpie).
    Uninstalling PackagesMay leave dependencies behind if installed globally.Removes entire package environments cleanly.
    Running Without InstallingRequires installation before use.Can run commands without installing (pipx run).

    When to Use pipx vs. venv

    1. Use pipx when you need standalone CLI tools that should work globally but remain isolated (e.g., pipx install black).
    2. Use venv when you need an isolated development environment for a specific project (e.g., python -m venv env).

    Resources

    1. https://pip.pypa.io/en/stable/cli/pip_list/
    2. https://bootvar.com/5-useful-python-pip-commands/
    3. https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/
    4. https://daily.dev/blog/pip-essentials-for-python-developers
  • Selecting a GPU for a Dell T620

    Selecting a GPU for a Dell T620

    I just recently posted about installing a GPU Power Supply Expansion Board into the Dell T620. I suggest giving that post a read before you continue, as this will be post builds upon what was discussed previously. This post will be a bit of a deep dive into what you should consider when choosing a GPU for your T620.

    Ok, so first things first, you need to choose a GPU that is compatible with your T620, as not everything is going to work. Choosing a GPU that is a good fit for your workload, well that comes second.

    Let’s start by looking at what Dell tested and certified.

    Officially Supported GPUs for the T620

    ManufacturerModelHeatsinkPower Usage
    NvidiaTesla C2075Active215W
    NvidiaQuadro K4000Active80W
    NvidiaQuadro Q6000Active204W
    NvidiaTesla K20AActive225W
    ATI™ FirePro™V7800Active150W
    AMD FireProW7000Active127.7W

    According to the documentation, only the above GPUs are “supported” on the Dell T620. Note that this really means that these are the only cards that Dell certified/tested. It does not mean that there are not any other compatible cards out there. These GPUs are honestly quite old. The only still relevant use case for any of them in this day and age is video transcoding on the quadro card. However, you might be able to do some lightweight AI/ML work on the Tesla K20A (2688 Cuda Cores, Kepler Architecture)

    Dell documentation further expands on what is supported. Specifically…

    1. Up to four 300W, full-length, single- or double-wide GPU cards
    2. GPUs with up to 6GB of dedicated DDR5 memory
    3. GPUs that are actively cooled.
    4. No more than 2 power connectors per card
    5. Power connectors located on the back of the card
    6. GPU enablement kit (Power & Cooling Required)

    Additionally Dell documentation states the following…

    1. All cards are the same make/model
    2. 1000W power supplies are required (technically for systems with more than 2 cards)
    3. You need not to cram your T620 too full with GPUs and Raid Controllers to avoid overheating

    What About other GPUs?

    So this is where we start to go off the rails, and venture into the unknown. Let’s start with what we know, which is what Dell officially supported and what their documentation says about choosing a GPU

    Is 6GB the VRAM Limit?

    At the time, Dell only were able to test and certify a handful of GPUs, and none of those GPUs had more than 6GB of VRAM. So it’s very possible that this is why Dell documentation tops out at 6GB as supported. I doubt this is an actual real-world limit.

    The T620 was released in 2012 and probably went end-of-life in 2017 – 2019. None of the officially “supported” cards were released after 2013. Available GPUs with more than 6GB VRAM were probably not widely available, and if they were, Dell probably had finished their initial round of GPU certification and moved on to better things. Any GPUs with more than 6GB would probably run hotter and consumed more power, which could be an issue. Additionally there could be conflicts at the firmware/bios level for mapping/managing more than 6GB of VRAM. Who knows? Generically, they probably drew the line at 6GB because “that is what they knew worked”.

    What about Power and Cooling?

    Adding GPUs to a system will increase cooling and power needs. Replacing existing PSUs, with 1000w PSUs (as recommended by Dell) is very simple and actually pretty cheap. I think I picked up mine for less than $30 USD a pair on ebay. Dual 1000w PSUs was the only supported configuration if you wanted to order a T620 with GPU. However, you still needed to get that power to the GPUs themselves.

    This is where the GPU Enablement Kit comes into play. The enablement kit consists of…

    1. Power expansion board
    2. GPU Power Cables
    3. A Fan Gantry (shown below)
    T620 Fan Gantry

    The fan gantry is not easily found, and they are expensive. You will probably spend more on one of these that you did for your T620. They run about $300+ USD on ebay.

    I have found 3 part numbers on line

    1. 0VDY5 – fan
    2. 8G79K – gantry
    3. 2R4DV – Fan + Gantry?

    The fan gantry from a T630 will also work as well, albeit with a couple of slight modifications. As you need to cut a bit of metal, took me about 2 minutes to modify mine with some tin snips.

    The TOP square I modified a slight bit with tin snips. You will need to do the same for a 4 squares

    Note that you will need T620 Fans, as the T630 fan plugs do not line up with the power receiver on the motherboard (hence why you need to modify the gantry)

    T620 Fan on the left. T630 Fan on the right. Notice the offset of the power plug is different between the two server models.

    Note that the Fan Gantry for the T630 is not any cheaper, it’s just a bit easier to find . The part number for the T620 fan is 0TW71C, which is the same fan that are at the rear of the baffle.

    Adding non-Dell proprietary fans to the chassis of the T620 is either much harder or much more expensive, as most of the unused power connectors in the chassis are proprietary. There are also no standard mounting brackets for common fan sizes.

    However, there is power to be had. There is connector coming off the drive backplane that provides SATA power to the CDROM. And in theory you could do this…

    Male Sata–Female Molex == Male Molex–Female Molex == Fan Controller with 2pin Molex

    Now you can add a fan controller and can add some more fans, but you will have to get creative. A few pci slot blower fans? PCI slot Graphics Card Cooler possibly?


    Similar Alternative GPUs with Active Cooling

    Assuming we need to keep to similar power usage and VRAM, let’s look at a few older, yet not as anemic GPUs.

    1. NVIDIA GeForce GTX 1660 Super

    • VRAM: 6GB GDDR6
    • TDP: 125W
    • Cooling: Actively cooled
    • Performance: Great for basic AI/ML tasks, especially if you are focused on inference or lighter training workloads.

    2. NVIDIA RTX A2000 (Professional Card)

    • VRAM: 6GB GDDR6
    • TDP: 70W
    • Cooling: Actively cooled, compact design
    • Performance: A professional GPU tailored for AI/ML workloads and well-suited for power-constrained environments.

    3. NVIDIA T1000

    • VRAM: 4GB GDDR6
    • TDP: 50W
    • Cooling: Actively cooled, compact design
    • Performance: Suitable for lightweight AI/ML applications and inference tasks, ideal if energy efficiency is a priority.

    What about GPUs without External Power

    No power supply expansion board?

    Let’s look at GPUs that do not require external power. Most PCIe 3.0 slots provide only 25 watts of power, however 75W is the maximum power delivery from a single PCIe 3.0 slot. The Dell T620 has 4 such X16 slots. This alone is appealing and is one of the reasons the Dell T-Series are popular with home labbers. By contract, my newer R630 only provides max of 25W per PCIe slot.

    T620 Power Provided Per PCIE Slot

    SLOTPOWERCPU CONNECTION
    Slot 1 (x8 lanes)up to 25WCPU1
    Slot 2 (x16 lanes)up to 75WCPU1
    Slot 3 (x4 lanes)up to 25WPlatform Controller Hub
    Slot 4 (x16 lanes)up to 75WCPU1
    Slot 5 (x16 lanes)up to 75WCPU2
    Slot 6 (x8 lanes)up to 25WCPU2
    Slot 7 (x16 lanes)up to 75WCPU2

    GPUs that run on 75W or less

    Below is a list of possible GPUs that you can run without additional power requirements, other than what select PCIe slots in the T620 will provide.

    NVIDIA GPUs

    1. NVIDIA Tesla T4

    • Performance: Excellent for AI inference and light training.
    • Power: 70W (fits within PCIe 3.0 constraints).
    • VRAM: 16GB GDDR6.
    • Features: Tensor Cores, CUDA support, and FP16/FP32/INT8 capabilities for AI/ML.
    • Notes: Designed for data centers and supports efficient AI workloads.
    • Cooling: Passive (gets super hot)

    2. NVIDIA Quadro T1000 / T2000

    • Performance: Entry-level professional GPUs for AI tasks.
    • Power: 50-75W.
    • VRAM: 4GB-6GB GDDR6.
    • Features: CUDA cores and optimized drivers for compute tasks.
    • Notes: Focused on balanced performance and efficiency.
    • Cooling: Active

    3. NVIDIA GeForce GTX 1650 (Low Profile/Standard)

    • Performance: Suitable for light AI/ML workloads.
    • Power: 75W.
    • VRAM: 4GB GDDR5/GDDR6.
    • Features: CUDA support for training and inference.
    • Notes: Ensure you get a variant without external power connectors.
    • Cooling: Active

    4. NVIDIA GeForce GTX 1050 Ti

    • Performance: Basic AI/ML tasks and experiments.
    • Power: 75W.
    • VRAM: 4GB GDDR5.
    • Features: CUDA cores and basic AI capabilities.
    • Notes: An older but efficient card for small-scale projects.
    • Cooling: Active

    AMD GPUs

    1. AMD Radeon RX 6400

    • Performance: Entry-level GPU for basic AI workloads.
    • Power: ~53W.
    • VRAM: 4GB GDDR6.
    • Features: RDNA 2 architecture with good efficiency.
    • Notes: Limited AI-focused features but sufficient for lightweight tasks.
    • Cooling: Active

    2. AMD Radeon Pro WX 3200

    • Performance: Professional GPU for light compute tasks.
    • Power: 50W.
    • VRAM: 4GB GDDR5.
    • Features: Optimized drivers for professional workloads.
    • Notes: Reliable for basic AI workloads and professional use.
    • Cooling: Active

    Intel GPUs

    1. Intel Arc A310

    • Performance: Basic GPU for lightweight AI and inference.
    • Power: ~75W.
    • VRAM: 4GB GDDR6.
    • Features: Support for AI frameworks like TensorFlow and PyTorch.
    • Notes: Newer entry from Intel with decent AI potential.
    • Cooling: Active

    What about Modern GPUs?

    Let me start out by saying that it’s possible that many present day GPUs might actually operate without issue in the T620. That being said, what you risk is the bios/lifecycle controller throwing an error as it does not like what it sees happening in its slot.

    A bus fatal error was detected on a component at slot 7

    Above is the error that I encountered attempting to run a Telsa T4 card. I attempted with several different 16x slots, and the error moved with each slot that I tried. This error usually resulted in an OS kernel panic. In my case, its possible that the Nvidia T4, being passively cooled, could have been part of the problem, the card was too hot to touch. Although temperature could have had nothing to do with it and the card may just not “work” in the T620 (at least not yet, as I continue to troubleshoot)

    Regarding newer cards, through my research I have found that a number of folks have had good luck with a few different CPUs (See Reference Section Below). Bottom line, you may get lucky.

    Myself, well I am not giving up on getting the Nvidia T4 to run in my T620. I still have a some experimentation to do. Mainly some bios config tweaking and a bit of add on cooling. I also have some ideas regarding the lifecycle controler, but I have yet to test anything.

    Troubleshooting/Stability

    Below are a few ideas that I have regarding troubleshooting, good luck. I will create a new post at some point in the near future regarding my sucess with the Nvidia T4 GPU.

    1. Slot Disablement  – disable unused PCIE slots. PCIE power stability is especially important
      • “Boot Driver Disabled”. Will investigate this setting. This allow the slot to be visible to the OS, but will not be available as a boot device.
    2. BIOS upgrade – make sure you are running the latest bios (2.9.0)
    3. PCI Slot Selection – choose 16x slot closest to a CPU.
      • I have seen that many have success in slot 5
    4. Cooling – Make sure your GPU is not getting too hot.
      • Pull hard disks to improve air flow on the right side of chassis
      • PCI slot cooler
      • PCI slot blower
    5. Disable Dell Lifecycle Controller?
    6. Disable Collect System Inventory On Restart (CSIOR)?

    Conclusion

    Bottom line, these machines were not exactly designed to run with GPUs, and only a handful of GPUs were actually tested and supported on the Dell T620. The list of supported cards is short, and these GPUs are quite old. However, just because its not supported, does not mean that it will not work. It just means it was not tested. Your mileage may vary. I’ve seen a number of post from individuals who are running more modern GPUs the T620 and they do so without issue.

    Dell Documentation

    1. Dell PowerEdge T620 Technical Guide
    2. Dell T620 GPU Card Installation Guidelines and Requirements

    Reference

    1. Is X7C1K compatible with T620
    2. T620’s device manager recognizes GTX780Ti, but programs do not “see” the card?!
    3. T620 gpu issue
    4. PowerEdge T620: 6-pin PCIe power connector not working (GTX 1060 3GB & GTX 680)
    5. T620’s device manager recognizes GTX780Ti, but programs do not “see” the card?!
    6. GTX 1080 not working in T620
    7. Adding GPU for display – Dell T620 and T630 (change from UEFI to BIOS?)
    8. Dell T620 GPU > use kit from T630? (Radeon HD5770. Confirmed T620 kit works in T620)
    9. Adding GPU to Dell T620 (1660 ti and a 750 ti.)
    10. Dell EMC PowerEdge T640: Remove/Install GPU PIB (Pretty much the same as the T620)



  • Red Hat: Identity Management Server Setup and HA on RHEL 7

    cropped-oversaturateddetail.jpg

    Introduction

    Red Hat Identity Management Server provides is a centralized identity management server for Linux, Mac, Windows.

    In this post we are going to setup and configure a HA deployment of Red Hat IDM on two RHEL 7.x servers.

    Red Hat Identity Management Server is based on the upstream project, FreeIPA.

    Prerequisites

    Only a couple of prerequisites for a simple lab setup

    • You need working dns with forward and reverse entries for both IDM servers
    • NSCD needs to be disabled on each IDM server
    • Proper hostname needs to be set on each IDM node (cannot use localhost)
    • IPV6 needs to be enabled, but you do not need to have an IPV6 address on external interfaces
    (more…)
  • OpenSCAP Part 4: Integration with Red Hat Satellite 6.5

    openscap-base

    Introduction

    This is the 4th installment of a series of posts focusing on OpenSCAP.  Previous posts in this series can be found below.

    OpenSCAP Part 1: Introduction and Basic Usage for RHEL 7/CentOS 7/Fedora
    OpenSCAP Part 2: SCAP Content for RHEL 7
    OpenSCAP Part 3: Running Scans from the Command Line in RHEL 7

    This time around we are going to work on integrating OpenSCAP functionality into Red Hat Satellite 6.5

    Installing SCAP Content on Satellite Server

    SSH into your satellite server and run the command below as root.

    # foreman-rake foreman_openscap:bulk_upload:default

    Screenshot from 2019-07-26 17-02-22.png

    By default only a few content files are added.

    Installing Additional SCAP Content

    This can be done via the UI (and possibly via the CLI, however I have not attempted this).

    In this example I have already downloaded RHEL 8 content from here.

    Log into the Satellite UI, and navigate to “Hosts” > “SCAP content”

    Select “Upload New SCAP Content”

    Screenshot from 2019-07-26 17-09-14.png

    And select Choose File to select a file from your local machine.

    Screenshot from 2019-07-26 17-20-54.png

    (more…)

  • OpenSCAP Part 3: Running Scans from the Command Line in RHEL 7

    openscap-base

    Introduction

    In part 1 of this series we were introduced to OpenSCAP and the process of running scans via the SCAP workbench. In part 2, we explored concepts and components that define security/vulnerability scans. In this 3rd post we are going to dive into the command line operation.

    Let’s get started with oscap.

    Installing oscap

    In RHEL 7 oscap can be installed with the following command

    # yum -y install scap-security-guide openscap-scanner

    Content is installed under the following directory. Note that ssg is short for SCAP Security Guide.

    /usr/share/xml/scap/ssg/content

    Lets change directories to the one listed above and view the installed files.

    Screenshot from 2019-07-24 15-58-04

    Using oscap we can view more info on each file shown above. In this example we are going to inspect the ssg-rhel7-ds.xml file.

    # oscap info ssg-rhel7-ds.xml

    (more…)