Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Category: Dell

  • The Homelab Dilemma: Living With Enterprise Servers

    The Homelab Dilemma: Living With Enterprise Servers

    Enterprise Servers are loud and hot, and while a couple of 13th generation Dell servers kept a room nice and toasty in the cold of winter, it a double whammy to your electric bill in the Spring/Summer.

    The noise can be problematic as well, especially since I no longer have a basement, where I relied on my servers to circulate and dehumidify the stuffy air. Now, I actually have to sit in a room with the rack. Which is why I moved it to the bedroom. Strange.

    So why the bedroom, well the white noise (when under control) helps me sleep and I can turn down the heat in the Winter and not freeze at night. And in the day, when I power everything up, I am in my home office (or my second home office – the dinning room), which is already loud enough with my workstations and old Cisco switches. That tiny home office, that is technically a small bedroom for your least favorite child, can get downright uncomfortably warm.

    Why So Much Noise

    As anyone who has worked in a datacenter can tell you. Servers are loud. Let compare fans in an Dell R630 to an R730.

    ServerForm FactorEffective fan sizeFan CountNotes
    Dell R6301U~40 mm blower7 Very high RPM, loud, high static pressure
    Dell R7302U~60 mm blower6Still loud, but more efficient airflow

    This boils down to the 1U server must spin its fans faster that then 2U to move the same amount of air through the server chassis, measured in CFMs (Cubic Feet per Minute).

    Additionally, you install a PCIe card in a server that the server itself does not recognize (like Telsa T4 GPUs) , and you may find your fans spinning at full bore, as the server would rather take flight than overheat.

    Enter the Dell Fan Susher

    Luckily, you can use ipmitools to override a Dell server’s fan speed, and quiet things down a great deal. However, you need to keep your eye on temperatures. For this I created Dell-Server-Fan-Shusher.

    Its a python script that monitors system temps (via sensors), and NVIDIA GPU temps (via nvidia-smi – when present), and sets fan speeds accordingly. It currently has seven threshold levels for temp and fan speeds, all of which can easily be modified.

    Installation is one command and its scheduled either via cron or systemd.

    sudo ./install.sh

    The deployed fan_control.py reads system temps in the following order.

    • Sysfs (/sys/class/hwmon)
    • sensors
    • ipmitool sdr list (last fallback)

    It also gets GPU temps via nvidia-smi and system temps via IPMI (e.g. Inlet, Exhaust, CPU packages, etc.).

    Its Getting Hot in Here

    The susher has been working fine for months, but as I mentioned its no longer Winter and the ambient air in my “server room” has been rising. See output from my Netbotz 450 below. Starting to get hotter than a grandparent’s Florida condo.

    Line graph showing temperature readings over time from a sensor pod. The y-axis represents temperature in Fahrenheit, ranging from 71 to 77 degrees. The x-axis shows time intervals. The maximum temperature recorded is 75.4 degrees, and the minimum is 72.3 degrees.

    So why I am posting all this? So this morning, when I attempted to log into my main hypervisor to start on my 3rd installment of my HPC GPU Cluster journey, I found the host unresponsive and throwing the errors below on the console. Time to reboot.

    Terminal screen showing error messages for a network adapter in Red Hat Enterprise Linux, indicating it has stopped due to overheating and suggesting to restart the computer or replace the adapter.

    Overheating NIC?

    So here is what happened, the ixgbe driver detected that the adapter overheated and disabled it to protect the hardware. When that happens, all ports on that card stop working. In the console, kernel logs are reporting a thermal shutdown of the Intel 10GbE network adapter. Specifically the Intel X540-AT2, which is a dual port 10Gbe copper (RJ45) adapter – with a large heatsink (some variants of this card have a fan.

    lspci -s 0000:82:00.0
    82:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

    A few minutes after a reboot the error returns, and the NIC goes offline again and is accessible only via the IDRAC console. At this point I cannot keep the system online long enough to do any meaningful troubleshooting. So we need to shush the susher. So from the console we disable it and then reboot again.

    # systemctl disable --now dell-r730-fan-control.timer
    Removed '/etc/systemd/system/timers.target.wants/dell-r730-fan-control.timer'.


    From our workstation we connect to the servers IDRAC and set manual fan control and set fan speeds to 100%. We do this while the target system is in the process or rebooting.

    ipmitool -I lanplus -H 10.1.10.20 -U root -P calvin  raw 0x30 0x30 0x01 0x00
    ipmitool -I lanplus -H 10.1.10.20 -U root -P calvin raw 0x30 0x30 0x02 0xff 100

    Hopefully now we can keep the system online long enough to see what is really going on, as the server itself is not reporting any temperature issues on the idrac.

    Troubleshooting

    So first lets check the system logs and see how many times that this has occurred.

    grep -r "over heat\|overheat\|stopped because" /var/log/ 2>/dev/null | tail -50

    We see two events today, and a 3rd event a few days ago (probably patient zero).

    DateTimeEvent
    Mar 22, 202612:04:56Both Intel 10GbE NICs (enp130s0f0, enp130s0f1) stopped due to overheat
    Mar 22, 202613:17:24Same overheat again shortly before reboot at 13:24:59
    Mar 15, 202611:39:31Same overheat on both NICs

    Digging into logs a bit further, we see the following at around 12:04

    Fan speed: 15% (~3800 RPM)

    • GPU temp: 35°C
    • System temps: max 47°C (sensors 24–25 and 39 at 41–47°C)
    • Action: Fan control had just set fans to 15% for the “LOW” threshold.

    Intel X540-AT2 thermal specs

    A quick internet search yields the following. This NIC needs to get pretty hot to experience a thermal shutdown

    SpecificationValueNotes
    Operating temp (ambient)0°C to 55°CMarketing spec for 200 LFM airflow
    Extended ambient0°C to 70°CWith 300 LFM airflow and adequate heatsink
    Tcase max107°CMax case temp at heat spreader (die-level limit)

    NIC temperature visibility

    So the Intel ixgbe driver does not expose temperature to Linux:

    • No hwmon temperature sensor under /sys/...
    • ethtool does not show NIC temperature

    So there is no NIC temperature in system logs. The “over heated” message comes from the NIC’s internal thermal protection. The driver only reports the event; the actual NIC temperature is not logged. At this point we do no have a method to pull the temp from the NIC.

    Additional, there are no log messages like thermal warnings, throttling, or high-temperature notices before the shutdown. So there is really nothing that we can check for in the logs and use as a trigger for our Dell Fan Controller (Shusher).

    EventLast ixgbe message before overheatOverheat
    Mar 22, 12:04:5611:58:02 – NIC Link is Up 10 Gbps12:04:56 – overheat (about 7 minutes later)
    Mar 22, 13:17:2413:11:28 – NIC Link is Up 10 Gbps (after reboot)13:17:24 – overheat (about 6 minutes later)
    Mar 15, 11:39:31No ixgbe messages in preserved logs before this time11:39:31 – overheat

    Additionally, none of our reported temps from lm-sensors show any temperature issues, and we can see that the fan-susher had recently adjusted fans to 15% due to somewhat amenable temps.

    LM-Sensors

    Currently sensors detects the following temperatures on this R730.

    ChipSensorCurrentLimits
    coretemp-isa-0000 (CPU Package 0)Package id 028°Chigh 83°C, crit 93°C
    Core 0–28 (16 cores)22–25°Chigh 83°C, crit 93°C
    coretemp-isa-0001 (CPU Package 1)Package id 129°Chigh 83°C, crit 93°C
    Core 0–28 (16 cores)22–25°Chigh 83°C, crit 93°C

    Lets run sensors-detect and see if we can add any additional sensors that might help us get a better picture of temps across the system

    # sensors-detect

    We answer “Yes” at each prompt. Any new modules/sensors are added to /etc/sysconfig/lm_sensors

    Now we reload all the sensors

     . /etc/sysconfig/lm_sensors; for m in $HWMON_MODULES $BUS_MODULES; do [ -n "$m" ] && sudo modprobe -r $m 2>/dev/null; done; for m in $BUS_MODULES $HWMON_MODULES; do [ -n "$m" ] && sudo modprobe $m; done

    However, no new modules/temps are output by the sensors command, so no additional system temps to feed to the shusher.

    So What Now?

    At this point we have a few options, besides just cranking down the AC.

    We know that our overheat events occurred when system temps were ~47°C with fans at 15%. So, lets start with an overhaul of the susher and increase fan speeds by 10% for each of the 7 temp thresholds as configured in our .env file, and redeploy.

    Updated Fan speed levels (7 tiers)

    LevelTemp thresholdFan speedTrigger
    Very-Low< 20°C20%Below LOW
    Low≥ 20°C25%GPU LOW / System LOW
    Medium-Low≥ 35°C35%MED_LOW
    Medium≥ 50°C45%MED
    Medium-High≥ 60°C60%MED_HIGH
    High≥ 60°C (system) / 70°C (GPU)75%HIGH
    Very-High≥ 75°C90%Auto mode → iDRAC

    Our minimum fan speed is now 20% instead of 10%, which should improve airflow over the NICs and reduce overheat risk.

    Inlet/Exhaust Differential Logging

    For the NIC overheat events (Mar 22 12:04, 13:17 and Mar 15 11:39), inlet/exhaust were not logged, so let’s start logging them on each scheduled run of fan_control.py , so if/when something goes wrong, we can see how hot the air was going in and how much it warmed up inside the chassis.

    Safety Floor

    Additionally lets setup login in fan_control.py if inlet or exhaust gets too high:

    • Inlet ≥ 40°C → minimum fan speed set to 35%
    • Exhaust ≥ 50°C → minimum fan speed set to 35%

    Even if GPU/CPU temps look fine, we still ramp fans to protect things like the NIC when chassis air is hot.

    Next Steps

    For now we are going to let things ride. We have adjusted our fan speeds up, and we setup our safety floor.

    If we continue to see issues, we may need to take additional action. This r730 is running a number of Virtual machines which I use as “lab infrastructure”, so I want it running 24/7. And while I could just keep jacking up fan speeds, I would rather take a more proactive approach.

    1. Remove heatsink and apply fresh thermal paste. This card is long in the tooth. Could be dried up.
    2. Move to an earlier revision of the Intel X540-AT2 which came with active fans. (Cheap)
    3. Move to a card who’s driver can expose temperatures, like the Broadcom NetXtreme-E (e.g. BCM57416 – not as cheap)

    In theory, I like the 3rd option, as it would be nice to be able to pull temperatures from the the NIC, and allow fan_control.py to actually adjust fan speeds intelligently. In practice, however, replacing the NIC is probably overkill unless I run into a dual port actively cooled Intel with both long and short brackets (I like to keep my options open slot-wise). Although I do like getting packages in the mail.

  • Moving Beyond OMSA: A Guide to Dell iSM Installation on RHEL 10 and PowerEdge R730

    Moving Beyond OMSA: A Guide to Dell iSM Installation on RHEL 10 and PowerEdge R730

    The shift from srvadmin (OMSA) to iSM (iDRAC Service Module) marks the end of bloated, “in-band” server management. This occurred between Dell 12th gen and Dell 13th gen server.

    If you have a 12th Gen Dell Server, you can still leverage Dell srvadmin (Idrac 7). I wrote a post on it here.

    While OMSA ran as a heavy suite of services directly on the host, the modern dcism acts as a lightweight bridge that offloads the heavy lifting to the dedicated iDRAC hardware.

    For RHEL 10 users, this transition is no longer optional but necessary, trading the aging omreport ecosystem for a more secure, stable, and kernel-friendly management experience.

    So in this post we are going to go through the install of DISM (Dell IDRAC Service Module), on RHEL 10, running on a Dell R730.

    Set Up the Dell Repository

    First we need to setup the official Dell System Update (DSU) utility. Install Dell GPG keys when prompted. 

    curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi
    bash bootstrap.cgi

    Install the ISM Package

    dnf install dcism

    Also install racadm

    sudo dnf install srvadmin-idracadm7

    Start and Enable the Service

    sudo systemctl enable --now dcismeng.service
    systemctl status dcismeng.service

    Example Commands

    This first command will make sure that DISM is healthy

    /opt/dell/srvadmin/iSM/bin/dcismcfg --getismstatus

    And ensure that racadm is functioning properly

    /opt/dell/srvadmin/sbin/racadm getsysinfo

    You can also reset the idrac from the OS

    /opt/dell/srvadmin/iSM/bin/Invoke-iDRACHardReset

    You can generate a full hardware diagnostic bundle directly from the (TSR).

    Note that the used Dell R730 that I just purchased on Ebay is actually under hardware support till October of 2026 – so if I need on-site hardware support a Dell technician will be dispatched to my house. I will provide snacks and a static strap. 

    /opt/dell/srvadmin/iSM/bin/Invoke-SupportAssistCollection

    Note: The SupportAssist Collection feature is not supported on systems earlier than 14th generation. 🙁

    You can check your disks for SMART errors

    /opt/dell/srvadmin/iSM/bin/ismsmartlog --dev=/dev/sdb

    Local RAID & Storage Management

    Since omreport storage is gone, use racadm. With the iSM bridge active, Local RACADM is now your primary interface for your raid controller

    racadm storage get pdisks
    racadm storage get vdisks
     racadm storage get controllers

    The “Full Power Cycle” (The “Cold Boot” Tool) – Flea Power Drain

    On an R730, if you have a hardware component acting up (like a hung NIC), a standard OS reboot often isn’t enough because the motherboard stays powered.

    • What it does: It tells the iDRAC to perform a “virtual” AC power pull. The server will shut down, the flea power will be drained, and it will cold-boot. This is a powerful feature to have in a remote RHEL 10 environment.
     /opt/dell/srvadmin/iSM/bin/Invoke-FullPowerCycle

    Real-Time “Tech Support Report” (TSR)

    If you ever need to open a support ticket with Dell, they will ask for a TSR. Previously, you had to log into the iDRAC web UI and wait 10 minutes.

    • What it does: It triggers the iDRAC to gather every log, firmware version, and hardware event into a .zip file and saves it directly to the iDRAC (or can be pulled to the OS).
    /opt/dell/srvadmin/iSM/bin/Invoke-SupportAssistCollection

    Viewing Active Hardware Alerts

    The racadm utility (which uses the iSM bridge) is the best way to see why your server has an amber light or a “System Health” warning.

    • View the System Event Log (SEL): This is the “Hardware Event Viewer.” It shows power supply failures, memory errors, and thermal trips.
    /opt/dell/srvadmin/sbin/racadm getsel

    Installing perccli

    PERCCLI (PowerEdge RAID Controller Command Line Interface) is the specialized successor to the older MegaCLI tool. While RACADM is great for general server health, PERCCLI is the “surgical tool” specifically for your RAID controller (like the H730 in your R730).

    Install as shown below

    dnf -y install perccli.noarch

    Now you can show all controllers

     /opt/MegaRAID/perccli/perccli64 show

    Show events.

     /opt/MegaRAID/perccli/perccli64 /c0 show eventloginfo

    Show all disks

    /opt/MegaRAID/perccli/perccli64 /c0/eall/sall show

    Epilogue

    So in a nutshell. Dell srvadmin was replaced with DISM starting on Dell 13th gen servers. You have a couple powerful commands at your disposal with a ton of various ways to run both of them.

    Also available is perccli, which is also a nice tool on its own.

  • Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator: Comprehensive Guide for Hardware Monitoring (RHEL)(Dell 12 Gen)

    Dell OpenManage Server Administrator (OMSA) is Dell’s on-host hardware management and monitoring framework for PowerEdge servers. 

    It runs inside the operating system and provides direct visibility into system hardware such as RAID controllers, physical and virtual disks, power supplies, fans, temperatures, memory, processors, and chassis health. 

    OMSA communicates with the server’s iDRAC and hardware controllers to retrieve real-time status and exposes this information through command-line tools like omreport, optional web interfaces, and SNMP for monitoring systems.

     It is primarily used for hardware diagnostics, RAID and storage monitoring, fault detection, and health reporting, allowing administrators to verify system integrity and troubleshoot hardware issues without leaving the operating system.

    So now that we have an idea of what it is, lets install it.

    Note: Our system under test is a Dell R720. I probably should have lead with that.

    Installation

    First you will need to download the bootstrap script as shown below. 

    curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi

    Now we will make it executable.

    chmod +x bootstrap.cgi

    And then we run it.

    sudo ./bootstrap.cgi

    Now we have completed the following.

    • Added Dell’s official repo
    • Installed Dell signing keys
    • Configured OMSA package sources

    Now Install OpenManage (omreport)

    sudo dnf install srvadmin*

    This installs:

    • omreport
    • omconfig
    • Storage monitoring
    • RAID tools
    • CIM providers

    Starting and Enabling Services

    Apparently, service names can vary from OS to OS, so run the command below to verify the correct service names

    systemctl list-unit-files | grep -Ei 'dsm|omsa|srvadmin'
    dsm_om_connsvc.service                     enabled         disabled
    dsm_om_shrsvc.service                      disabled        disabled
    dsm_sa_datamgrd.service                    enabled         disabled
    dsm_sa_eventmgrd.service                   enabled         disabled
    dsm_sa_snmpd.service                       enabled         disabled

    Now that we have the names of the services we can start and enable them as shown below

    # sudo systemctl enable --now dsm_om_connsvc dsm_sa_datamgrd dsm_sa_eventmgrd dsm_sa_snmpd dsm_om_shrsvc
    OpenManage Server Administrator (OMSA) Services
    Service NameFull NamePurposeRequired?Notes
    dsm_om_connsvcOMSA Connection ServiceProvides the web-based interface (HTTPS/1311) used to access OMSA remotely or locally✅ Yes (if using OMSA web UI)This is what allows access via https://<host&gt;:1311
    dsm_om_shrsvcOMSA Shared ServicesProvides shared libraries and backend support for other OMSA components⚠️ UsuallyRequired by other OMSA components; often left disabled unless needed
    dsm_sa_datamgrdSystems Management Data ManagerCollects and maintains hardware inventory and system data✅ YesRequired for hardware monitoring (disks, temps, fans, etc.)
    dsm_sa_eventmgrdSystems Management Event ManagerHandles hardware events, alerts, and logs✅ YesRequired for alerts and log reporting
    dsm_sa_snmpdOMSA SNMP AgentProvides SNMP interface for monitoring tools (LibreNMS, Zabbix, etc.)⚠️ OptionalNeeded only if using SNMP monitoring

    Service Review

    Now let’s review each service in detail, and see how we can leverage each one.

    dsm_om_connsvc

    First we need to ensure our Firewall allows this traffic. So run the commands below (RHEL), ubuntu may use ufw.

    # firewall-cmd --permanent --add-port=1311/tcp
    # firewall-cmd --reload

    Now in your web browser, navigate to https://viper.lab:1311/ (Note https, and change host name accordingly).

    Similar in appearance to the Idrac – you can find system information here, some of which does not appear in the Idrac itself. Everything is read-only.

    Screenshot of Dell OpenManage Server Administrator showing Fan Probes Information, including fan redundancy status and probe list with readings, thresholds, and status indicators.

    dsm_om_shrsvc

    (OMSA Shared Services) is a supporting service, not a management interface or monitoring agent.

    It provides:

    • Shared libraries
    • Inter-process communication
    • Authentication helpers
    • Common backend utilities

    These are required by other OMSA components to function correctly. So nothing really to see here.

    dsm_sa_datamgrd.service

    This is the service that collects all the data that is displayed in the WebUI (port 1311)

    dsm_sa_eventmgrd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    This service relies entirely on the existing system SNMP configuration, so no need to modify anything if you already have snmp configured and running with your specfic community string.

    dsm_sa_snmpd.service

    dsm_sa_snmpd.service is the OMSA component responsible for exposing Dell hardware monitoring data to external systems using the SNMP protocol. 

    It acts as the interface between the data collected internally by OMSA and third-party monitoring platforms such as LibreNMS, Zabbix, or Nagios. 

    This service does not collect or analyze hardware information itself; instead, it serves as the communication layer that allows remote systems to query or receive notifications about server health. When enabled, it allows OMSA to participate in centralized monitoring environments and enterprise alerting workflows, making it essential in environments where hardware status must be visible outside the server itself.

    dsm_sa_snmpd.service acts as the Dell OMSA integration layer between the server’s hardware telemetry and the system’s SNMP stack, allowing Dell-specific health, storage, and sensor data to be exposed through standard SNMP queries. 

    It does not replace or conflict with existing monitoring agents such as Zabbix or LibreNMS (which I am using) Instead, it complements them by extending the SNMP data they can read

    dsm_sa_snmpd provides access to Dell-specific MIBs that Zabbix Agent and LibreNMS Agent (or any other agent you may use) can query via the system’s SNMP daemon. 

    Basically, dsm_sa_snmpd is the bridge that exposes OMSA’s hardware awareness to your existing monitoring stack.
    Note that if you are using Zabbix you will need to install additional templates. For example, this one, note that I have not tried it yet.

    Ompreport

    Omreport is the command-line interface (CLI) frontend for OMSA. It queries the OMSA backend services and formats the results for humans or scripts

    Note we have to use the absolute path to run this as the working dir is not in our $PATH. 

    /opt/dell/srvadmin/sbin/omreport system summary

    If you want to add to your $PATH, run the command below

    echo 'export PATH=$PATH:/opt/dell/srvadmin/sbin' | sudo tee /etc/profile.d/dell-omsa.sh
    source /etc/profile

    Example usage below.

    omreport system summary
    omreport chassis temps
    omreport chassis fans
    omreport chassis power

    Here is an example I found useful. For some reason my raid controller keeps throwing these errors which do not seem to go away despite the fact that both SAS cables are connected (tried switching them) and all disks are healthy. 

    # omreport system esmlog
    Severity      : Critical
    Date and Time : Sat Jan 31 21:15:18 2026
    Description   : The storage BP1 SAS B cable is not connected, or is improperly connected.
    Severity      : Ok
    Date and Time : Sat Jan 31 21:15:20 2026
    Description   : The chassis is closed while the power is off.
    Severity      : Critical
    Date and Time : Sat Jan 31 21:15:21 2026
    Description   : The storage BP1 SAS A cable is not connected, or is improperly connected.

    Enter omconfig.

    omconfig

    omconfig is the configuration and control interface for Dell OpenManage Server Administrator (OMSA). 

    Unlike omreport, which is strictly read-only, omconfig is used to change system and OMSA behavior—including alerting, storage actions, and certain hardware-related settings exposed by OMSA. 

    Omconfig It operates by issuing commands to the OMSA backend services (primarily dsm_sa_datamgrd and dsm_sa_eventmgrd) and applying those changes at the software management layer, not directly to hardware firmware. 

    Common uses include enabling or disabling alert actions, configuring storage-related behavior, managing notification settings, and triggering maintenance operations such as consistency checks. 
    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    Because omconfig can modify system behavior and suppress alerts, it should be used carefully—especially on production systems—and is best suited for controlled configuration changes rather than routine monitoring.

    So back to my two SAS cable alerts. Lets leverage omreport and omconfig to diagnose and hopefully ether determine that there is an actual issue, or suppress the alerts.

    OMCONFIG and OMREPORT – Real world Usage

    First lets take a sneaky-peak at my raid controller.

    omreport storage controller
     Controller  PERC H730 Adapter(Slot 4)
    Controller
    ID                                            : 0
    Status                                        : Ok
    Name                                          : PERC H730 Adapter
    Slot ID                                       : PCIe Slot 4
    State                                         : Ready
    Firmware Version                              : 25.5.9.0001
    -----truncated---

    According to chatgpt I am getting these errors because the system expects that I have an external enclosure connected. I do not think that is the case, as that seems dumb. More than likely this alert is due to the fact that I am using an H730 (a 13G controller) (and its cables) in an R720 (a 12G server) in Slot 4, which is a PCIe Adapter and not the integrated “mini” version.

    So let’s look at the disks and make sure all is well

    Here we can see that the PERC H730 adapter does not “See” the connector states as connected, rather just ready

    omreport storage connector controller=0
    List of Connector(s) on Controller PERC H730 Adapter (Slot 4)
    ID             : 0
    Status         : Ok
    Name           : Connector 0
    State          : Ready
    Connector Type : SAS Port RAID Mode
    Termination    : Not Applicable
    SCSI Rate      : Not Applicable
    ID             : 1
    Status         : Ok
    Name           : Connector 1
    State          : Ready
    Connector Type : SAS Port RAID Mode
    Termination    : Not Applicable
    SCSI Rate      : Not Applicable

    In order to use this 13th gen raid controller in a 12th gen Dell Server I could not use the original SAS cables. Rather I used SFF-8087 to SFF-8643 cables Which can cause the sideband signals (SGPIO / SES) which tell the iDRAC everything is okay which is why we are seeing the errors.

    As stated above this is not actually a problem as I can see my disks. Lets do that below. 

    omreport storage pdisk controller=0 | awk '
    /^ID[[:space:]]*:/     {id=$3; state=""; status=""}
    /^State[[:space:]]*:/  {state=$3}
    /^Status[[:space:]]*:/ {status=$3}
    (state!="" && status!="" && id!="") {print id, state, status; id=""; state=""; status=""}
    '
    0:0 Online Non-Critical
    0:1 Online Non-Critical
    0:2 Online Non-Critical
    0:3 Online Non-Critical
    0:4 Online Non-Critical
    0:5 Online Non-Critical
    0:6 Online Non-Critical
    0:7 Online Non-Critical

    Here we can see we have 8 disks, no critical errors

    First lets clear the esmlog with omconfig – this will clear the error till we reboot

    omconfig system esmlog action=clear

    Sadly there is not a permanent fix. I would need to get a different raid controller that uses the original cables.

  • Dell Force10 S4810 Factory Reset, OS Upgrade, and Basic Setup

    Dell Force10 S4810 Factory Reset, OS Upgrade, and Basic Setup

    The Dell Force10 S4810 is a 48 sfp port, 10Gbe Switch. Let’s walk through a factory reset and some basic configuration tasks as well as an OS upgrade from OS8 to OS9.

    Factory Reset

    Follow the steps below for factory reset. After running the reload you will be prompted to save the config, ignore this. 

    Force10>en
Force10>#delete flash://startup-config
Proceed to delete startup-config.bak [confirm yes/no]:yes 
Force10>reload

    Configure Management IP Address

    In order to be able to connect via ssh/telnet to the S4810 we will need to config a management IP. Below we are configuring an IP address on ManagementEthernet 1/0 , which is the ethernet port next to the console port at the front of the switch.

    interface ManagementEthernet 1/0
 ip address 10.1.10.46/24
 no shutdown

    We also need to configure the default route.

    Force10(conf)#management route 0.0.0.0/0 10.1.10.1

    You should now be able to ping the management interface on the switch from your workstation. You should also verify that the switch can ping an ip outside your network, such as 8.8.8.8.

    Enable SSH/Telnet

    First we create an user and grant privileges. Replace <MYUSER> and <MYPASS> with the username and password that you wish to use.

    Force10(conf)#username <MYUSER> admin password <MYPASS> privilege 15 access-class permitall

    Now we enable ssh as shown below. 

    Force10#configure
Force10(conf)#ip ssh server enable
Force10(conf)#crypto key generate rsa 
Enter key size <1024-2048>. Default<1024>  :2048
Host key already exists .Overwrite (y/n)?y
Generating SSHv2 RSA key.
Force10(conf)#exit

    Now we should confirm the running config for ssh.

    Force10#show ip ssh
SSH server                : enabled.
SSH server version        : v1 and v2.
Password Authentication   : enabled.
Hostbased Authentication  : disabled.
RSA       Authentication  : disabled.

    Now we configure telnet.

    Force10#conf t
Force10(conf)#ip telnet server enable
Force10(conf)#exit

    Now save your running configuration and attempt a remote connection. Note that you might run into the following error when attempting to ssh to the switch. 

    $ ssh cpaquin@10.1.10.49
Unable to negotiate with 10.1.10.49 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
1

    Note that the message above indicates that the SSH client doesn’t support the algorithms used by the switch as they are legacy and potentially weak. We can work around this as shown below. I expect once we upgrade the switch OS with a more modern ssh implementation, we will no longer need the work around. 

    $ ssh -o KexAlgorithms=diffie-hellman-group14-sha1 -c 3des-cbc chris@10.1.10.46
The authenticity of host '10.1.10.46 (10.1.10.46)' can't be established.
RSA key fingerprint is SHA256:rXqVL/9Ie74e2ZV3m/BiO2KV7adTBXZYwK73UNPEhCo.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.1.10.46' (RSA) to the list of known hosts.
chris@10.1.10.46's password: 
Force10>

    Note that once you have upgraded to OS9 you will not need to ssh with the -o or -c options. If you are unable to connect via ssh prior to upgrade you can use telnet for the time being.

    Configure NTP

    First let’s set the timezone and show current time.

    Force10(conf)#clock timezone eastern -5 
Force10#show clock
20:46:25.954 eastern Sun Apr 6 2025

    Now we will configure the switch to sync time with an upstream ntp server, 10.1.10.10

    #ntp server 10.1.10.10

    Then confirm association

    show ntp associations
   remote       vrf-Id     ref clock     st when poll reach   delay   offset    disp
====================================================================================
 10.1.10.10      0      0.0.0.0         16    -   16    0     +0.000   +0.000   +0.000

    It may take a few minutes for the output below to show “sycronized”

    Force10#show ntp status  
Clock is synchronized, stratum 2, reference is 216.239.35.0, vrf-id is 0
frequency is 6.813 ppm, stability is 0.002 ppm, precision is -18
reference time  eb9dadd7.8477f9b6  Mon, Apr  7 2025  1:50:15.517 UTC
clock offset is 0.456662 msec, root delay is 13.220 msec
root dispersion is 1.261 msec, peer dispersion is 0.608 sec
peer mode is client

    OS/Firmware Upgrade

    Dell Open Networking switches use ONIE (Open Network Install Environment) as the bootloader and OS installer, and it relies on U-Boot (Universal Bootloader) for fundamental features like reading/writing boot flash and environment variables.

    Per Dell, in order to upgrade the switch you need to complete the following steps.

    ● Upgrade the S4810 Dell Networking OS Image and Boot Code
    ● Upgrade the CPLD (not needed in this case)

    So let’s first determine what OS version is running on our switch. 

    
#show ver
Dell Force10 Real Time Operating System Software
Dell Force10 Operating System Version: 1.0
Dell Force10 Application Software Version: 8.3.10.3
Copyright (c) 1999-2011 by Dell Inc.
Build Time: Tue May 15 22:31:24 PDT 2012
Build Path: /sites/sjc/work/build/buildSpaces/build06/E8-3-10/SW/SRC/Cp_src/Tacacs
Force10 uptime is 3 hour(s), 48 minute(s)

System image file is "system://A"

System Type: S4810 
Control Processor: Freescale QorIQ P2020 with 2147483648 bytes of memory.

128M bytes of boot flash memory.

  1 52-port GE/TE/FG (SE)
 48 Ten GigabitEthernet/IEEE 802.3 interface(s)
  4 Forty GigabitEthernet/IEEE 802.3 interface(s)

    You can see in the output above that the switch is currently running 8.3.10.3.

    Before upgrading Dell Networking OS on S4810 from a version prior to 8.3.12.0 to version 9.14(1.14), ensure to increase the partition size by upgrading to version 8.3.12.0.

    According to Dell’s documentation I need to upgrade to 8.3.12.0 before I can upgrade to 9.14.

    Upgrading to 8.3.12.0

    I have configured vsftp on my workstation as I will use ftp as my upgrade method. Note that the step below may take a few minutes to complete. 

    Force10#upgrade system ftp: a:
Address or name of remote host []: 10.1.10.10
Source file name []: FTOS-SE-8.3.12.2.bin
User name to login remote host: ftpuser
Password to login remote host:

    Verify the Dell Networking OS has been upgraded correctly in the upgraded flash partition via the command below. 

    Force10>show boot system stack-unit all

Current system image information in the system:
=============================================

Type          Boot Type     A                        B
----------------------------------------------------------------
Stack-unit 0 is not present.
Stack-unit 1  FLASH BOOT    8.3.12.2[boot]           8.3.7.0                  
Stack-unit 2 is not present.
Stack-unit 3 is not present.
Stack-unit 4 is not present.
Stack-unit 5 is not present.
Stack-unit 6 is not present.
Stack-unit 7 is not present.
Stack-unit 8 is not present.
Stack-unit 9 is not present.
Stack-unit 10 is not present.
Stack-unit 11 is not present.

    Now save and reload.

    Force10#write mem
Force10#reload
Proceed with reload [confirm yes/no]: yes

    Upgrade the S4810 Boot Code

    Dell Networking OS version 9.14(1.10) requires S4810 Boot Code version 1.2.0.5. Below we can see that we are currently running 1.2.0.2, so we will need to upgrade the boot code as well prior to upgrading to 9.14(1.10). 

    Force10>show system stack-unit 1 | grep "Boot Flash"
Boot Flash      :  1.2.0.2

    Again we will use ftp as our file transfer/upgrade.

    Force10#upgrade boot ftp:
Address or name of remote host []: 10.1.10.10
Source file name []: U-boot.1.2.0.5.bin
User name to login remote host: ftpuser
Password to login remote host: 
!
Erasing SSeries BootImageUpgrade Table of Contents, please wait
.!........................................!
524528 bytes successfully copied

    Now save and reload

    Force10#write mem  
!
Force10#reload

    Upgrade to 9.14

    Once the switch as reloaded we are ready to upgrade to 9.14. We will use ftp once again. 

    Force10#upgrade system ftp: a:
Address or name of remote host []: 10.1.10.10
Source file name []: FTOS-SE-9.14.1.14.bin
User name to login remote host: ftpuser
Password to login remote host:

    Now write to memory and reload the OS.

    Force10#write mem
!

Force10#reload
Proceed with reload [confirm yes/no]: yes

    Wait for the OS to reload and then use “show ver” to check the running OS image. 

    Force10>show ver
Dell Real Time Operating System Software
Dell Operating System Version:  2.0
Dell Application Software Version:  9.14(1.14)

    References

    1. https://www.dell.com/community/en/conversations/networking-general/s4810-firmware/647f9ce3f4ccf8a8de1135a6
    2. https://www.dell.com/support/kbdoc/en-us/000182885/ftos-for-s-series-s4810-platform?dgc=SM&cid=304571&lid=spr7038757706&refid=sm_LITHIUM_spr7038757706&linkId=167138678
    3. https://www.reddit.com/r/homelab/comments/c9wdnq/dell_force_10_s60_44t_help_round_2/
    4. https://dl.dell.com/manuals/all-products/esuprt_networking_int/esuprt_networking_operating_systems/dell-emc-os-9_release-notes30_en-us.pdf
    5. https://www.dell.com/support/kbdoc/en-us/000122301/how-to-enable-ssh-and-disable-telnet-on-force10-s-series-switches

  • Selecting a GPU for a Dell T620

    Selecting a GPU for a Dell T620

    I just recently posted about installing a GPU Power Supply Expansion Board into the Dell T620. I suggest giving that post a read before you continue, as this will be post builds upon what was discussed previously. This post will be a bit of a deep dive into what you should consider when choosing a GPU for your T620.

    Ok, so first things first, you need to choose a GPU that is compatible with your T620, as not everything is going to work. Choosing a GPU that is a good fit for your workload, well that comes second.

    Let’s start by looking at what Dell tested and certified.

    Officially Supported GPUs for the T620

    ManufacturerModelHeatsinkPower Usage
    NvidiaTesla C2075Active215W
    NvidiaQuadro K4000Active80W
    NvidiaQuadro Q6000Active204W
    NvidiaTesla K20AActive225W
    ATI™ FirePro™V7800Active150W
    AMD FireProW7000Active127.7W

    According to the documentation, only the above GPUs are “supported” on the Dell T620. Note that this really means that these are the only cards that Dell certified/tested. It does not mean that there are not any other compatible cards out there. These GPUs are honestly quite old. The only still relevant use case for any of them in this day and age is video transcoding on the quadro card. However, you might be able to do some lightweight AI/ML work on the Tesla K20A (2688 Cuda Cores, Kepler Architecture)

    Dell documentation further expands on what is supported. Specifically…

    1. Up to four 300W, full-length, single- or double-wide GPU cards
    2. GPUs with up to 6GB of dedicated DDR5 memory
    3. GPUs that are actively cooled.
    4. No more than 2 power connectors per card
    5. Power connectors located on the back of the card
    6. GPU enablement kit (Power & Cooling Required)

    Additionally Dell documentation states the following…

    1. All cards are the same make/model
    2. 1000W power supplies are required (technically for systems with more than 2 cards)
    3. You need not to cram your T620 too full with GPUs and Raid Controllers to avoid overheating

    What About other GPUs?

    So this is where we start to go off the rails, and venture into the unknown. Let’s start with what we know, which is what Dell officially supported and what their documentation says about choosing a GPU

    Is 6GB the VRAM Limit?

    At the time, Dell only were able to test and certify a handful of GPUs, and none of those GPUs had more than 6GB of VRAM. So it’s very possible that this is why Dell documentation tops out at 6GB as supported. I doubt this is an actual real-world limit.

    The T620 was released in 2012 and probably went end-of-life in 2017 – 2019. None of the officially “supported” cards were released after 2013. Available GPUs with more than 6GB VRAM were probably not widely available, and if they were, Dell probably had finished their initial round of GPU certification and moved on to better things. Any GPUs with more than 6GB would probably run hotter and consumed more power, which could be an issue. Additionally there could be conflicts at the firmware/bios level for mapping/managing more than 6GB of VRAM. Who knows? Generically, they probably drew the line at 6GB because “that is what they knew worked”.

    What about Power and Cooling?

    Adding GPUs to a system will increase cooling and power needs. Replacing existing PSUs, with 1000w PSUs (as recommended by Dell) is very simple and actually pretty cheap. I think I picked up mine for less than $30 USD a pair on ebay. Dual 1000w PSUs was the only supported configuration if you wanted to order a T620 with GPU. However, you still needed to get that power to the GPUs themselves.

    This is where the GPU Enablement Kit comes into play. The enablement kit consists of…

    1. Power expansion board
    2. GPU Power Cables
    3. A Fan Gantry (shown below)
    T620 Fan Gantry

    The fan gantry is not easily found, and they are expensive. You will probably spend more on one of these that you did for your T620. They run about $300+ USD on ebay.

    I have found 3 part numbers on line

    1. 0VDY5 – fan
    2. 8G79K – gantry
    3. 2R4DV – Fan + Gantry?

    The fan gantry from a T630 will also work as well, albeit with a couple of slight modifications. As you need to cut a bit of metal, took me about 2 minutes to modify mine with some tin snips.

    The TOP square I modified a slight bit with tin snips. You will need to do the same for a 4 squares

    Note that you will need T620 Fans, as the T630 fan plugs do not line up with the power receiver on the motherboard (hence why you need to modify the gantry)

    T620 Fan on the left. T630 Fan on the right. Notice the offset of the power plug is different between the two server models.

    Note that the Fan Gantry for the T630 is not any cheaper, it’s just a bit easier to find . The part number for the T620 fan is 0TW71C, which is the same fan that are at the rear of the baffle.

    Adding non-Dell proprietary fans to the chassis of the T620 is either much harder or much more expensive, as most of the unused power connectors in the chassis are proprietary. There are also no standard mounting brackets for common fan sizes.

    However, there is power to be had. There is connector coming off the drive backplane that provides SATA power to the CDROM. And in theory you could do this…

    Male Sata–Female Molex == Male Molex–Female Molex == Fan Controller with 2pin Molex

    Now you can add a fan controller and can add some more fans, but you will have to get creative. A few pci slot blower fans? PCI slot Graphics Card Cooler possibly?

    Similar Alternative GPUs with Active Cooling

    Assuming we need to keep to similar power usage and VRAM, let’s look at a few older, yet not as anemic GPUs.

    1. NVIDIA GeForce GTX 1660 Super

    • VRAM: 6GB GDDR6
    • TDP: 125W
    • Cooling: Actively cooled
    • Performance: Great for basic AI/ML tasks, especially if you are focused on inference or lighter training workloads.

    2. NVIDIA RTX A2000 (Professional Card)

    • VRAM: 6GB GDDR6
    • TDP: 70W
    • Cooling: Actively cooled, compact design
    • Performance: A professional GPU tailored for AI/ML workloads and well-suited for power-constrained environments.

    3. NVIDIA T1000

    • VRAM: 4GB GDDR6
    • TDP: 50W
    • Cooling: Actively cooled, compact design
    • Performance: Suitable for lightweight AI/ML applications and inference tasks, ideal if energy efficiency is a priority.

    What about GPUs without External Power

    No power supply expansion board?

    Let’s look at GPUs that do not require external power. Most PCIe 3.0 slots provide only 25 watts of power, however 75W is the maximum power delivery from a single PCIe 3.0 slot. The Dell T620 has 4 such X16 slots. This alone is appealing and is one of the reasons the Dell T-Series are popular with home labbers. By contract, my newer R630 only provides max of 25W per PCIe slot.

    T620 Power Provided Per PCIE Slot

    SLOTPOWERCPU CONNECTION
    Slot 1 (x8 lanes)up to 25WCPU1
    Slot 2 (x16 lanes)up to 75WCPU1
    Slot 3 (x4 lanes)up to 25WPlatform Controller Hub
    Slot 4 (x16 lanes)up to 75WCPU1
    Slot 5 (x16 lanes)up to 75WCPU2
    Slot 6 (x8 lanes)up to 25WCPU2
    Slot 7 (x16 lanes)up to 75WCPU2

    GPUs that run on 75W or less

    Below is a list of possible GPUs that you can run without additional power requirements, other than what select PCIe slots in the T620 will provide.

    NVIDIA GPUs

    1. NVIDIA Tesla T4

    • Performance: Excellent for AI inference and light training.
    • Power: 70W (fits within PCIe 3.0 constraints).
    • VRAM: 16GB GDDR6.
    • Features: Tensor Cores, CUDA support, and FP16/FP32/INT8 capabilities for AI/ML.
    • Notes: Designed for data centers and supports efficient AI workloads.
    • Cooling: Passive (gets super hot)

    2. NVIDIA Quadro T1000 / T2000

    • Performance: Entry-level professional GPUs for AI tasks.
    • Power: 50-75W.
    • VRAM: 4GB-6GB GDDR6.
    • Features: CUDA cores and optimized drivers for compute tasks.
    • Notes: Focused on balanced performance and efficiency.
    • Cooling: Active

    3. NVIDIA GeForce GTX 1650 (Low Profile/Standard)

    • Performance: Suitable for light AI/ML workloads.
    • Power: 75W.
    • VRAM: 4GB GDDR5/GDDR6.
    • Features: CUDA support for training and inference.
    • Notes: Ensure you get a variant without external power connectors.
    • Cooling: Active

    4. NVIDIA GeForce GTX 1050 Ti

    • Performance: Basic AI/ML tasks and experiments.
    • Power: 75W.
    • VRAM: 4GB GDDR5.
    • Features: CUDA cores and basic AI capabilities.
    • Notes: An older but efficient card for small-scale projects.
    • Cooling: Active

    AMD GPUs

    1. AMD Radeon RX 6400

    • Performance: Entry-level GPU for basic AI workloads.
    • Power: ~53W.
    • VRAM: 4GB GDDR6.
    • Features: RDNA 2 architecture with good efficiency.
    • Notes: Limited AI-focused features but sufficient for lightweight tasks.
    • Cooling: Active

    2. AMD Radeon Pro WX 3200

    • Performance: Professional GPU for light compute tasks.
    • Power: 50W.
    • VRAM: 4GB GDDR5.
    • Features: Optimized drivers for professional workloads.
    • Notes: Reliable for basic AI workloads and professional use.
    • Cooling: Active

    Intel GPUs

    1. Intel Arc A310

    • Performance: Basic GPU for lightweight AI and inference.
    • Power: ~75W.
    • VRAM: 4GB GDDR6.
    • Features: Support for AI frameworks like TensorFlow and PyTorch.
    • Notes: Newer entry from Intel with decent AI potential.
    • Cooling: Active

    What about Modern GPUs?

    Let me start out by saying that it’s possible that many present day GPUs might actually operate without issue in the T620. That being said, what you risk is the bios/lifecycle controller throwing an error as it does not like what it sees happening in its slot. 

    A bus fatal error was detected on a component at slot 7

    Above is the error that I encountered attempting to run a Telsa T4 card. I attempted with several different 16x slots, and the error moved with each slot that I tried. This error usually resulted in an OS kernel panic. In my case, its possible that the Nvidia T4, being passively cooled, could have been part of the problem, the card was too hot to touch. Although temperature could have had nothing to do with it and the card may just not “work” in the T620 (at least not yet, as I continue to troubleshoot)

    Regarding newer cards, through my research I have found that a number of folks have had good luck with a few different CPUs (See Reference Section Below). Bottom line, you may get lucky.

    Myself, well I am not giving up on getting the Nvidia T4 to run in my T620. I still have a some experimentation to do. Mainly some bios config tweaking and a bit of add on cooling. I also have some ideas regarding the lifecycle controler, but I have yet to test anything.

    Troubleshooting/Stability

    Below are a few ideas that I have regarding troubleshooting, good luck. I will create a new post at some point in the near future regarding my sucess with the Nvidia T4 GPU.

    1. Slot Disablement  – disable unused PCIE slots. PCIE power stability is especially important
      • “Boot Driver Disabled”. Will investigate this setting. This allow the slot to be visible to the OS, but will not be available as a boot device.
    2. BIOS upgrade – make sure you are running the latest bios (2.9.0)
    3. PCI Slot Selection – choose 16x slot closest to a CPU.
      • I have seen that many have success in slot 5
    4. Cooling – Make sure your GPU is not getting too hot.
      • Pull hard disks to improve air flow on the right side of chassis
      • PCI slot cooler
      • PCI slot blower
    5. Disable Dell Lifecycle Controller?
    6. Disable Collect System Inventory On Restart (CSIOR)?

    Conclusion

    Bottom line, these machines were not exactly designed to run with GPUs, and only a handful of GPUs were actually tested and supported on the Dell T620. The list of supported cards is short, and these GPUs are quite old. However, just because its not supported, does not mean that it will not work. It just means it was not tested. Your mileage may vary. I’ve seen a number of post from individuals who are running more modern GPUs the T620 and they do so without issue.

    Dell Documentation

    1. Dell PowerEdge T620 Technical Guide
    2. Dell T620 GPU Card Installation Guidelines and Requirements

    Reference

    1. Is X7C1K compatible with T620
    2. T620’s device manager recognizes GTX780Ti, but programs do not “see” the card?!
    3. T620 gpu issue
    4. PowerEdge T620: 6-pin PCIe power connector not working (GTX 1060 3GB & GTX 680)
    5. T620’s device manager recognizes GTX780Ti, but programs do not “see” the card?!
    6. GTX 1080 not working in T620
    7. Adding GPU for display – Dell T620 and T630 (change from UEFI to BIOS?)
    8. Dell T620 GPU > use kit from T630? (Radeon HD5770. Confirmed T620 kit works in T620)
    9. Adding GPU to Dell T620 (1660 ti and a 750 ti.)
    10. Dell EMC PowerEdge T640: Remove/Install GPU PIB (Pretty much the same as the T620)



  • Running RHEL 8 on Dell R710/610 via Raid Controller Retrofit

    Running RHEL 8 on Dell R710/610 via Raid Controller Retrofit

    A while back, I wrote a blog post that outlined a process of injecting deprecated storage controller drivers into RHEL 8 via a Driver Update Disk.

    In a nutshell, this process allows you to install RHEL 8 on the R710/R610 (11th Gen)

    This process worked fine, unless you wanted to yum update your server. Rather than attempt to find a repeatable process of injecting drivers prior to updating, I decided to upgrade my RAID controller to one that was supported.

    My 11th generation, R710, came with a Perc H700. Identified below.

    # lspci -knn | grep 'RAID bus controller'
03:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 05)

    Based on feedback from friends and co-workers. I purchased a PERC H330, which I believe ships with the 13th Gen Dell rackmount servers.

    These cards can be found for pretty cheap on Ebay. Make sure that you order one with the correct bracket based on your needs (full-height/half-height).

    Plus you will need 2 new cables. You are looking for SFF8643 to SFF8087 (mini SAS HD to mini SAS). They are also cheap and can be found on Ebay/Amazon

    Output from one of my Dell R710s. Note I did not remove the original card. 

    # lspci -knn | grep 'RAID bus controller'
03:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 05)
05:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] [1000:005d] (rev 02)

    Afterthoughts

    I’ve only performed this swap on one of my R710s, I have 2 more as well as 2 R610s. Below are are few things to consider if you are looking to perform a similar upgrade

    1. Take a hard look at your existing server. Do you need to purchase a full sized pci card or a mini card? This may likely be the case on the 1u servers and may be a blocker for you
    2. Remove the old card. The server does not quite like having a raid controller installed without cables.
    3. If you buy a full-sized pci card, make sure you get one with the bracket that you need (full height vs. half height vs. no bracket)
    4. You might want to find a cable with 90 degree connectors for the backplane. Mine fit fine on the 710, but I have heard that the 90 degree connector is best for 1U servers.
    5. You may want to consider upgrading your 11th generation server to an 12th generation Dell, R720/R620s can be purchased relatively cheaply and the RAM and disks in your 11th gen will work in a newer 12th generation. (note that the 11th gen Dell CPUs are socket LGA1366, while the 12th gen Dells utilize socket LGA2011)

    Additional Resources

    https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/considerations_in_adopting_rhel_8/hardware-enablement_considerations-in-adopting-rhel-8#removed-adapters_hardware-enablement

  • MegaRaid Cards Via CLI

    Megaman_8bit

    MegaRAID® is LSI’s line of SATA/SAS Storage Controller.

    MegaCLI

    MegaCLI is the Linux console based management utility for LSI SAS controllers. Honestly its a pretty crummy command when compared to HP’s command line tool, but that’s often what you are stuck with when you buy Dell or Supermicro.

    Note that I am running the 64 bit version of MegaCLI which is installed in /opt/MegaRAID/MegaCli and is called MegaCli64. On 32 bit systems its called MegaCLI.

    The command below will dump out a bunch of info, but if you look for the section labeled “Device Present” you can see failed/degraded drives. In this case I have one failed drive out of 4 total drives

    ./MegaCli64 -AdpAllInfo -aALL

     Device Present
    ================
    Virtual Drives    : 2 
      Degraded        : 1 
      Offline         : 0 
    Physical Devices  : 4 
      Disks           : 3 
      Critical Disks  : 0 
      Failed Disks    : 0

    For more specific disk information run the following command.

    ./MegaCli64 -LDPDInfo -aAll

    Using the command above I can see more information on the drive with the failed submirror

    Virtual Disk: 1 (Target Id: 1)
    Name:
    RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
    Size:59.125 GB
    State: Degraded
    Stripe Size: 64 KB
    Number Of Drives:2

    Megactl

    According to Sourceforce, Megactl is.. “is a small collection of programs for examining configuration and status of LSI megaraid adapters, especially Dell PERC RAID adapters, and attached storage devices.”  Get it here.

    In my this case I am running medasasctl which makes it a bit easier to see the failed drive.  In the example below I can see two virtual disks (both raid1), but only 3 physical, which indicates that one of my submirrors has failed.

    megactl-0.4.1]# ./megasasctl
    a0       LSI MegaRAID SAS 9260-8i encl:1 ldrv:2  batt:FAULT, unknown charge state
    a0d0       29GiB RAID 1   1x2  optimal
    a0d1       59GiB RAID 1   1x2  DEGRADED
    a0e252s0    29GiB  a0d0  online  
    a0e252s1    29GiB  a0d0  online  
    a0e252s2    59GiB  a0d1  online