Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Tag: homelab

  • The Homelab Dilemma: Living With Enterprise Servers

    The Homelab Dilemma: Living With Enterprise Servers

    Enterprise Servers are loud and hot, and while a couple of 13th generation Dell servers kept a room nice and toasty in the cold of winter, it a double whammy to your electric bill in the Spring/Summer.

    The noise can be problematic as well, especially since I no longer have a basement, where I relied on my servers to circulate and dehumidify the stuffy air. Now, I actually have to sit in a room with the rack. Which is why I moved it to the bedroom. Strange.

    So why the bedroom, well the white noise (when under control) helps me sleep and I can turn down the heat in the Winter and not freeze at night. And in the day, when I power everything up, I am in my home office (or my second home office – the dinning room), which is already loud enough with my workstations and old Cisco switches. That tiny home office, that is technically a small bedroom for your least favorite child, can get downright uncomfortably warm.

    Why So Much Noise

    As anyone who has worked in a datacenter can tell you. Servers are loud. Let compare fans in an Dell R630 to an R730.

    ServerForm FactorEffective fan sizeFan CountNotes
    Dell R6301U~40 mm blower7 Very high RPM, loud, high static pressure
    Dell R7302U~60 mm blower6Still loud, but more efficient airflow

    This boils down to the 1U server must spin its fans faster that then 2U to move the same amount of air through the server chassis, measured in CFMs (Cubic Feet per Minute).

    Additionally, you install a PCIe card in a server that the server itself does not recognize (like Telsa T4 GPUs) , and you may find your fans spinning at full bore, as the server would rather take flight than overheat.

    Enter the Dell Fan Susher

    Luckily, you can use ipmitools to override a Dell server’s fan speed, and quiet things down a great deal. However, you need to keep your eye on temperatures. For this I created Dell-Server-Fan-Shusher.

    Its a python script that monitors system temps (via sensors), and NVIDIA GPU temps (via nvidia-smi – when present), and sets fan speeds accordingly. It currently has seven threshold levels for temp and fan speeds, all of which can easily be modified.

    Installation is one command and its scheduled either via cron or systemd.

    sudo ./install.sh

    The deployed fan_control.py reads system temps in the following order.

    • Sysfs (/sys/class/hwmon)
    • sensors
    • ipmitool sdr list (last fallback)

    It also gets GPU temps via nvidia-smi and system temps via IPMI (e.g. Inlet, Exhaust, CPU packages, etc.).

    Its Getting Hot in Here

    The susher has been working fine for months, but as I mentioned its no longer Winter and the ambient air in my “server room” has been rising. See output from my Netbotz 450 below. Starting to get hotter than a grandparent’s Florida condo.

    Line graph showing temperature readings over time from a sensor pod. The y-axis represents temperature in Fahrenheit, ranging from 71 to 77 degrees. The x-axis shows time intervals. The maximum temperature recorded is 75.4 degrees, and the minimum is 72.3 degrees.

    So why I am posting all this? So this morning, when I attempted to log into my main hypervisor to start on my 3rd installment of my HPC GPU Cluster journey, I found the host unresponsive and throwing the errors below on the console. Time to reboot.

    Terminal screen showing error messages for a network adapter in Red Hat Enterprise Linux, indicating it has stopped due to overheating and suggesting to restart the computer or replace the adapter.

    Overheating NIC?

    So here is what happened, the ixgbe driver detected that the adapter overheated and disabled it to protect the hardware. When that happens, all ports on that card stop working. In the console, kernel logs are reporting a thermal shutdown of the Intel 10GbE network adapter. Specifically the Intel X540-AT2, which is a dual port 10Gbe copper (RJ45) adapter – with a large heatsink (some variants of this card have a fan.

    lspci -s 0000:82:00.0
    82:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

    A few minutes after a reboot the error returns, and the NIC goes offline again and is accessible only via the IDRAC console. At this point I cannot keep the system online long enough to do any meaningful troubleshooting. So we need to shush the susher. So from the console we disable it and then reboot again.

    # systemctl disable --now dell-r730-fan-control.timer
    Removed '/etc/systemd/system/timers.target.wants/dell-r730-fan-control.timer'.


    From our workstation we connect to the servers IDRAC and set manual fan control and set fan speeds to 100%. We do this while the target system is in the process or rebooting.

    ipmitool -I lanplus -H 10.1.10.20 -U root -P calvin  raw 0x30 0x30 0x01 0x00
    ipmitool -I lanplus -H 10.1.10.20 -U root -P calvin raw 0x30 0x30 0x02 0xff 100

    Hopefully now we can keep the system online long enough to see what is really going on, as the server itself is not reporting any temperature issues on the idrac.

    Troubleshooting

    So first lets check the system logs and see how many times that this has occurred.

    grep -r "over heat\|overheat\|stopped because" /var/log/ 2>/dev/null | tail -50

    We see two events today, and a 3rd event a few days ago (probably patient zero).

    DateTimeEvent
    Mar 22, 202612:04:56Both Intel 10GbE NICs (enp130s0f0, enp130s0f1) stopped due to overheat
    Mar 22, 202613:17:24Same overheat again shortly before reboot at 13:24:59
    Mar 15, 202611:39:31Same overheat on both NICs

    Digging into logs a bit further, we see the following at around 12:04

    Fan speed: 15% (~3800 RPM)

    • GPU temp: 35°C
    • System temps: max 47°C (sensors 24–25 and 39 at 41–47°C)
    • Action: Fan control had just set fans to 15% for the “LOW” threshold.

    Intel X540-AT2 thermal specs

    A quick internet search yields the following. This NIC needs to get pretty hot to experience a thermal shutdown

    SpecificationValueNotes
    Operating temp (ambient)0°C to 55°CMarketing spec for 200 LFM airflow
    Extended ambient0°C to 70°CWith 300 LFM airflow and adequate heatsink
    Tcase max107°CMax case temp at heat spreader (die-level limit)

    NIC temperature visibility

    So the Intel ixgbe driver does not expose temperature to Linux:

    • No hwmon temperature sensor under /sys/...
    • ethtool does not show NIC temperature

    So there is no NIC temperature in system logs. The “over heated” message comes from the NIC’s internal thermal protection. The driver only reports the event; the actual NIC temperature is not logged. At this point we do no have a method to pull the temp from the NIC.

    Additional, there are no log messages like thermal warnings, throttling, or high-temperature notices before the shutdown. So there is really nothing that we can check for in the logs and use as a trigger for our Dell Fan Controller (Shusher).

    EventLast ixgbe message before overheatOverheat
    Mar 22, 12:04:5611:58:02 – NIC Link is Up 10 Gbps12:04:56 – overheat (about 7 minutes later)
    Mar 22, 13:17:2413:11:28 – NIC Link is Up 10 Gbps (after reboot)13:17:24 – overheat (about 6 minutes later)
    Mar 15, 11:39:31No ixgbe messages in preserved logs before this time11:39:31 – overheat

    Additionally, none of our reported temps from lm-sensors show any temperature issues, and we can see that the fan-susher had recently adjusted fans to 15% due to somewhat amenable temps.

    LM-Sensors

    Currently sensors detects the following temperatures on this R730.

    ChipSensorCurrentLimits
    coretemp-isa-0000 (CPU Package 0)Package id 028°Chigh 83°C, crit 93°C
    Core 0–28 (16 cores)22–25°Chigh 83°C, crit 93°C
    coretemp-isa-0001 (CPU Package 1)Package id 129°Chigh 83°C, crit 93°C
    Core 0–28 (16 cores)22–25°Chigh 83°C, crit 93°C

    Lets run sensors-detect and see if we can add any additional sensors that might help us get a better picture of temps across the system

    # sensors-detect

    We answer “Yes” at each prompt. Any new modules/sensors are added to /etc/sysconfig/lm_sensors

    Now we reload all the sensors

     . /etc/sysconfig/lm_sensors; for m in $HWMON_MODULES $BUS_MODULES; do [ -n "$m" ] && sudo modprobe -r $m 2>/dev/null; done; for m in $BUS_MODULES $HWMON_MODULES; do [ -n "$m" ] && sudo modprobe $m; done

    However, no new modules/temps are output by the sensors command, so no additional system temps to feed to the shusher.

    So What Now?

    At this point we have a few options, besides just cranking down the AC.

    We know that our overheat events occurred when system temps were ~47°C with fans at 15%. So, lets start with an overhaul of the susher and increase fan speeds by 10% for each of the 7 temp thresholds as configured in our .env file, and redeploy.

    Updated Fan speed levels (7 tiers)

    LevelTemp thresholdFan speedTrigger
    Very-Low< 20°C20%Below LOW
    Low≥ 20°C25%GPU LOW / System LOW
    Medium-Low≥ 35°C35%MED_LOW
    Medium≥ 50°C45%MED
    Medium-High≥ 60°C60%MED_HIGH
    High≥ 60°C (system) / 70°C (GPU)75%HIGH
    Very-High≥ 75°C90%Auto mode → iDRAC

    Our minimum fan speed is now 20% instead of 10%, which should improve airflow over the NICs and reduce overheat risk.

    Inlet/Exhaust Differential Logging

    For the NIC overheat events (Mar 22 12:04, 13:17 and Mar 15 11:39), inlet/exhaust were not logged, so let’s start logging them on each scheduled run of fan_control.py , so if/when something goes wrong, we can see how hot the air was going in and how much it warmed up inside the chassis.

    Safety Floor

    Additionally lets setup login in fan_control.py if inlet or exhaust gets too high:

    • Inlet ≥ 40°C → minimum fan speed set to 35%
    • Exhaust ≥ 50°C → minimum fan speed set to 35%

    Even if GPU/CPU temps look fine, we still ramp fans to protect things like the NIC when chassis air is hot.

    Next Steps

    For now we are going to let things ride. We have adjusted our fan speeds up, and we setup our safety floor.

    If we continue to see issues, we may need to take additional action. This r730 is running a number of Virtual machines which I use as “lab infrastructure”, so I want it running 24/7. And while I could just keep jacking up fan speeds, I would rather take a more proactive approach.

    1. Remove heatsink and apply fresh thermal paste. This card is long in the tooth. Could be dried up.
    2. Move to an earlier revision of the Intel X540-AT2 which came with active fans. (Cheap)
    3. Move to a card who’s driver can expose temperatures, like the Broadcom NetXtreme-E (e.g. BCM57416 – not as cheap)

    In theory, I like the 3rd option, as it would be nice to be able to pull temperatures from the the NIC, and allow fan_control.py to actually adjust fan speeds intelligently. In practice, however, replacing the NIC is probably overkill unless I run into a dual port actively cooled Intel with both long and short brackets (I like to keep my options open slot-wise). Although I do like getting packages in the mail.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 2: Infiniband Setup

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 2: Infiniband Setup
    Black silhouette of a cat with an arched back.

    Part 1: Of this Project Log can be found here

    Now that the 3x Mellanox MCX455A-ECAT ConnectX-4 Adapters have arrived, its time to install them into their respective servers (columbia.lab, prometheus.lab, and viper.lab)

    Verify Mellanox CX4s are Detected

    Once installed, log into the IDRAC of each host and verify that the CX-4 appears in system inventory. Sample output below from one of the hosts.

    Note that you may need to boot the system for the CX-4 to appear in the IDRAC inventory (as Collect System Inventory on Restart” (CSIOR) will run when starting up)

    InfiniBand.Slot.1-1 - PCI Device
    BusNumber   	129
    DataBusWidth   	16x or x16
    Description   	ConnectX-4 VPI IB EDR/100 GbE Single Port QSFP28 Adapter
    Device Type	PCIDevice
    DeviceDescription   	InfiniBand.Slot.1-1
    DeviceNumber   	0
    FQDD   	InfiniBand.Slot.1-1
    FunctionNumber   	0
    InstanceID   	InfiniBand.Slot.1-1
    LastSystemInventoryTime   	2026-03-14T22:33:28
    LastUpdateTime   	2026-03-15T03:33:07
    Manufacturer   	Mellanox Technologies
    PCIDeviceID   	1013
    PCISubDeviceID   	0033
    PCISubVendorID   	15B3
    PCIVendorID   	15B3
    SlotLength   	Long Length
    SlotType   	PCI Express Gen 3

    Once each machine has booted to the running OS, you can confirm that the RHEL properly detects the CX-4 with lspci

    lspci | grep -i mel
    44:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

    Verify Numa Topology Via nvidia-smi

    Ideally, for best performance, your GPUS and InfiniBand adapters will be NUMA local to each other. If you were deploying a similar setup in a production environment, NUMA alignment would be critical.

    Our lab setup is less than ideal due to the limited number of PCI slots. In many Dell servers, PCIe risers for GPUs have only one PCIe slot. Stick two of these risers in a single server, and you end up with only 3 slots free on riser 1 (half lenght) which is where we had to install our CX-4s.

    All this being said, we “should” be fine for functional testing. Lets review each of our 3 nodes below. Since NVIDIA Drivers are already installed on all three systems, we can run nvidia-smi and confirm that the CX-4 is in the output and review the topology

    Nvidia-smi output on host viper.lab

    This server viper.lab is a Dell R720 running RHEL 9, and has 2x Nvidia Telsa P4 GPUs installed along with the CX-4.

    nvidia-smi topo -m
    	GPU0	GPU1	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID
    GPU0	 X 	PHB	SYS	0,2,4,6,8,10	0		N/A
    GPU1	PHB	 X 	SYS	0,2,4,6,8,10	0		N/A
    NIC0	SYS	SYS	 X

    In the output above. Both GPUs are local to NUMA node 0 and connected to each other through a PCIe host bridge (PHB), while the ConnectX NIC (mlx5_0) is topologically remote from both GPUs (SYS), making the setup workable but not ideal for GPUDirect RDMA performance. This should be ok for our lab as we are performing functional tests, and performance is secondary. Time will tell.

    Nvidia SMI output on host columbia.lab

    Columbia.lab is a Dell R730, with 1x Nvidia Telsa T4 installed along with one CX-4 (possible to move to another slot, if we had the full length bracket for the CX-4, or had half-length brackets for our NICs on Riser 2.

    nvidia-smi topo -m
    	GPU0	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID
    GPU0	 X 	SYS	0,2,4,6,8,10	0		N/A
    NIC0	SYS	 X

    In the output above, we can see that this system has one GPU (T4) on NUMA node 0 and and CX-4, but the GPU-to-NIC path is SYS, indicating a topologically distant connection that is usable but sub-optimal for GPUDirect RDMA performance. Again, may be fine for functional testing.

    Nvidia SMI output on host prometheus.lab

    This Dell R730 has 2x NVIDIA Tesla T4s installed, as well as the recently installed CX-4

    nvidia-smi topo -m
    	GPU0	GPU1	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID
    GPU0	 X 	PHB	SYS	0,2,4,6,8,10	0		N/A
    GPU1	PHB	 X 	SYS	0,2,4,6,8,10	0		N/A
    NIC0	SYS	SYS	 X

    This output shows two GPUs on the same NUMA node with a moderate GPU-to-GPU path (PHB) and relatively distant NIC connectivity (SYS), which is acceptable for many workloads but not ideal for high-performance GPU-to-NIC or GPUDirect-style traffic.

    Verify Drivers Loaded Properly

    In the first step we saw the CX-4 in the IDRAC, and in the output of lspci. We will now check that the driver has loaded properly. Rinse and repeat on each host. 

    lsmod | egrep 'mlx5_core|mlx5_ib|ib_core'
    mlx5_ib               561152  0
    macsec                 73728  1 mlx5_ib
    mlx5_core            3153920  2 mlx5_fwctl,mlx5_ib
    mlxfw                  49152  1 mlx5_core
    psample                20480  1 mlx5_core
    tls                   159744  2 bonding,mlx5_core
    pci_hyperv_intf        12288  1 mlx5_core
    ib_uverbs             217088  2 rdma_ucm,mlx5_ib
    ib_core               573440  12 rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm

    Lets review the imporant/relevant output below

    mlx5_ib               561152  0
    mlx5_core            3153920  2 mlx5_fwctl,mlx5_ib
    ib_uverbs             217088  2 rdma_ucm,mlx5_ib
    ib_core               573440  12 ...
    • mlx5_core – main low-level kernel driver for Mellanox/NVIDIA ConnectX-4/5-class adapters.
      • the kernel sees the adapter family and has loaded the base driver
      • this is required for the card to function at all
      • other Mellanox modules are depending on it
    • mlx5_ib – This is the InfiniBand/RDMA driver layer for mlx5 devices.
      • the adapter is not just using the generic Ethernet driver path
      • the system has the RDMA / InfiniBand-capable driver loaded
      • the kernel is prepared to expose the card as an IB/RDMA device
    • ib_core – the core InfiniBand subsystem in the kernel.
      • the Linux IB stack is loaded
      • multiple RDMA/IB-related modules are attached to it
      • the host is set up for InfiniBand/RDMA functionality, not just plain NIC support
    • ib_uverbs – This is the userspace verbs interface.
      • userspace RDMA tools and libraries should be able to talk to the device
      • commands like ibv_devinfo, ibstat, and RDMA applications have the proper kernel interface available

    Show Devices and Port State

    First we need to install some prerequesits

    sudo dnf install rdma-core infiniband-diags libibverbs-utils -y

    ibv_devices

    We can show InfiniBand devices with ibv_devices, which shows local RDMA/InfiniBand devices that the OS can see on that host. It does not enumerate remote hosts, switches, or the rest of the IB fabric.

    We will run this command on each host and capture the output.

    On columbia.lab

    [root@columbia ~]# ibv_devices
        device          	   node GUID
        ------          	----------------
        mlx5_0          	248a070300ac5414

    On viper.lab

    root@viper:~# ibv_devices
        device          	   node GUID
        ------          	----------------
        mlx5_0          	248a070300ac5f6c

    On prometheus.lab

    [root@prometheus ~]$ ibv_devices
        device          	   node GUID
        ------          	----------------
        mlx5_0          	248a070300ac5610
    [root@prometheus ~]$

    ibstat

    Now that we have confirmed all devices are present and accounted for lets check for links. In the output below you can see that we have link “Physical state: LinkUp“, but since we have not configured subnet manager on any of our nodes, the logical fabric is “State: Initializing“.

    ibstat
    CA 'mlx5_0'
    	CA type: MT4115
    	Number of ports: 1
    	Firmware version: 12.28.4512
    	Hardware version: 0
    	Node GUID: 0x248a070300ac5610
    	System image GUID: 0x248a070300ac5610
    	Port 1:
    		State: Initializing
    		Physical state: LinkUp
    		Rate: 40
    		Base lid: 65535
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x2659e848
    		Port GUID: 0x248a070300ac5610
    		Link layer: InfiniBand

    Run ibstat on any of your remaining nodes. Ensure that you see “Physical state: LinkUp”. You may also want to make notes of “Firmware version: 12.28.4512“. We have the same firmware on all three CX-4s.

    ibv_definfo

    We can also run ibv_devinfo, which gives a detailed view of the local RDMA / InfiniBand device and its ports. It is more detailed than ibv_devices and overlaps somewhat with ibstat, but from the verbs / RDMA stack perspective.

    Example output below:

     ibv_devinfo
hca_id:	mlx5_0
	transport:			InfiniBand (0)
	fw_ver:				12.28.4512
	node_guid:			248a:0703:00ac:5414
	sys_image_guid:			248a:0703:00ac:5414
	vendor_id:			0x02c9
	vendor_part_id:			4115
	hw_ver:				0x0
	board_id:			DEL2180110032
	phys_port_cnt:			1
		port:	1
			state:			PORT_INIT (2)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			0
			port_lid:		65535
			port_lmc:		0x00
			link_layer:		InfiniBand

    This output shows us the following…

    • Local RDMA devices
      • Example: mlx5_0, mlx5_1
    • Port state
      • Example: PORT_ACTIVE, PORT_DOWN, PORT_INIT
    • Physical link state
      • Example: LINK_UP, POLLING, DISABLED
    • Negotiated link details
      • Speed and link width
    • Fabric info
      • Local LID and SM LID
    • Device identifiers
      • Node GUID, port GUID, system image GUID
    • Transport / firmware details
      • Transport type and device-specific details
    • RDMA capabilities
      • Limits such as QPs, CQs, MR size, atomic support, GID table size

    rdma link

    The current output of “rdma link” shows use that our InfiniBand ports are connected but as we know the fabric is not initialized.

    rdma link
    link mlx5_0/1 subnet_prefix fe80:0000:0000:0000 lid 65535 sm_lid 0 lmc 0 state INIT physical_state LINK_UP

    Specifically the output shows us the following.

    • mlx5_0/1
      • Device mlx5_0, port 1
    • subnet_prefix fe80:0000:0000:0000
      • Normal default InfiniBand subnet prefix
    • lid 65535
      • The port does not have a valid assigned LID yet
    • sm_lid 0
      • No subnet manager is detected
    • lmc 0
      • LID mask control is 0; not important here
    • state INIT
      • The port is not fully active yet
    • physical_state LINK_UP
      • The physical link is up and the cable/port side is working

    Setup Subnet Manager on one Host

    For our lab, we are going to only setup subnet manager on one host. Pick your always-on host. Multiple instances of subnet manager can be used, but again, not needed for our current objective.

    On the selected host run the following to install required packages.

     dnf install -y rdma-core opensm infiniband-diags

    Next, start and enable the service

    sudo systemctl enable --now opensm

    Then check to ensure that the service started without error.

    sudo systemctl status opensm --no-pager
    journalctl -u opensm -b --no-pager

    Now we can re-check the fabric on each host. Now we see the Fabric status is “State: Active”

     ibstat
CA 'mlx5_0'
	CA type: MT4115
	Number of ports: 1
	Firmware version: 12.28.4512
	Hardware version: 0
	Node GUID: 0x248a070300ac5f6c
	System image GUID: 0x248a070300ac5f6c
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 4
		LMC: 0
		SM lid: 1
		Capability mask: 0x2659e848
		Port GUID: 0x248a070300ac5f6c
		Link layer: InfiniBand
root@viper:~#

    rdma link shows similar output.

    rdma link
    link mlx5_0/1 subnet_prefix fe80:0000:0000:0000 lid 4 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP

    Confirm Infiniband Fabric Topology

    You can run the following commands to confirm that your fabric is up and running

    Run the “ibnetdiscover” to see host adapters, links, GUIDs, port relationships, and switches. Example output below

    ibnetdiscover
    #
    # Topology file: generated on Sat Mar 14 21:15:43 2026
    #
    # Initiated from node 248a070300ac5f6c port 248a070300ac5f6c
    vendid=0x2c9
    devid=0xbd36
    sysimgguid=0x2c902004cf11b
    switchguid=0x2c902004cf118(2c902004cf118)
    Switch	8 "S-0002c902004cf118"		# "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0
    [1]	"H-248a070300ac5f6c"[1](248a070300ac5f6c) 		# "viper mlx5_0" lid 4 4xQDR
    [2]	"H-248a070300ac5414"[1](248a070300ac5414) 		# "columbia mlx5_0" lid 1 4xQDR
    [3]	"H-248a070300ac5610"[1](248a070300ac5610) 		# "prometheus mlx5_0" lid 2 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5414
    caguid=0x248a070300ac5414
    Ca	1 "H-248a070300ac5414"		# "columbia mlx5_0"
    [1](248a070300ac5414) 	"S-0002c902004cf118"[2]		# lid 1 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5610
    caguid=0x248a070300ac5610
    Ca	1 "H-248a070300ac5610"		# "prometheus mlx5_0"
    [1](248a070300ac5610) 	"S-0002c902004cf118"[3]		# lid 2 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5f6c
    caguid=0x248a070300ac5f6c
    Ca	1 "H-248a070300ac5f6c"		# "viper mlx5_0"
    [1](248a070300ac5f6c) 	"S-0002c902004cf118"[1]		# lid 4 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR

    In the output above we can see the following

    • One Mellanox InfiniScale-IV switch is present in the fabric
      • Switch GUID: 0x2c902004cf118
      • Switch LID: 3
      • Model family shown as Infiniscale-IV Mellanox Technologies
      • It is an 8-port switch
    • Three hosts are connected to the switch
      • columbia on switch port 2, LID 1
      • prometheus on switch port 3, LID 2
      • viper on switch port 1, LID 4
    • All three hosts are being seen as CA / HCA nodes
      • columbia mlx5_0
      • prometheus mlx5_0
      • viper mlx5_0
    • All discovered links are running at:
      • 4xQDR
      • That means a 4-lane QDR InfiniBand link, which aligns with a 40 Gb/s class IB link

    This output confirms that OpenSM is working properly, that the switch is visible, and all three nodes are connected to the fabric.

    Run “ibnodes” which provides a similar output to ibdiscover, albeit a bit less verbose.

    ibnodes
    Ca	: 0x248a070300ac5610 ports 1 "prometheus mlx5_0"
    Ca	: 0x248a070300ac5414 ports 1 "columbia mlx5_0"
    Ca	: 0x248a070300ac5f6c ports 1 "viper mlx5_0"
    Switch	: 0x0002c902004cf118 ports 8 "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0

    Run “ibswitches” to see switches only.

     ibswitches
Switch	: 0x0002c902004cf118 ports 8 "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0

    iblinkinfo will show you InfiniBand topology info

     iblinkinfo
CA: viper mlx5_0:
      0x248a070300ac5f6c      4    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    1[  ] "Infiniscale-IV Mellanox Technologies" ( )
CA: columbia mlx5_0:
      0x248a070300ac5414      1    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    2[  ] "Infiniscale-IV Mellanox Technologies" ( )
Switch: 0x0002c902004cf118 Infiniscale-IV Mellanox Technologies:
           3    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       4    1[  ] "viper mlx5_0" ( )
           3    2[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       1    1[  ] "columbia mlx5_0" ( )
           3    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    1[  ] "prometheus mlx5_0" ( )
           3    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           3    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           3    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           3    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           3    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: prometheus mlx5_0:
      0x248a070300ac5610      2    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    3[  ] "Infiniscale-IV Mellanox Technologies" ( )

    In the output above, we see …

    • 4x 10Gbps per lane
    • InfiniBand speed class is QDR
    • aggregate raw signaling rate is about 40 Gb/s
    • Down/Polling – unused/not-connected switch ports

    And finally, “sminfo” will show you info on subnet manager.

    Specifically (below) we see that subnet manager is reachable on LID1, with a GUID of 0x248a070300ac5414 (which belongs to columbia mlx5_0). We also see “activity count 446” which shows that subnet manager has processed fabric-management activity 466 times since startup (basically a liveness/activty counter).

    Additionally the output below show us the priority of the subnet manager instance (0 in this case), while state 3 SMINFO_MASTER shows us that this instance of subnet manager is in the master state and is the active controller in our IB fabric (assigning LIDS, managing paths/routing)

    sminfo
    sminfo: sm lid 1 sm guid 0x248a070300ac5414, activity count 446 priority 0 state 3 SMINFO_MASTER

    Configuring IP over InfiniBand

    IP over InfiniBand, or IPoIB, allows an InfiniBand fabric to carry normal IP traffic between hosts. That means systems connected by InfiniBand can use familiar network tools and services such as ping, ssh, scp, NFS, and other TCP/IP-based applications over the IB link instead of only using native RDMA-aware software.

    IPoIB is not required for RDMA itself, and it is also not inherently required for technologies like GPUDirect RDMA. RDMA and GPUDirect RDMA operate through the RDMA/verbs stack and the InfiniBand fabric, not through the IP emulation layer that IPoIB provides. NVIDIA’s current networking/operator docs describe RDMA and GPUDirect RDMA enablement separately from IPoIB, and they also document IPoIB as an optional deployment pattern rather than a prerequisite.

    We use IPoIB when we want the simplicity of standard IP networking on top of the higher-speed, low-latency InfiniBand fabric. In a small lab or cluster, this is useful for private host-to-host traffic, storage traffic, migration traffic, testing, or other east-west communication, while leaving the normal Ethernet interfaces in place for management access, internet access, and general connectivity.

    IPoIB Addresses for our lab

    Our lab uses 10.1.x.x for its existing IP scheme, so to avoid any confusion, we will use 172.16.x.x addresses for our small private subnet on the IB network. Note that we do not need a gateway.

    HOSTIPoIB AddressINTERFACE
    prometheus.lab172.16.50.11/24ibp129s0
    columbia.lab172.16.50.12/24ibp129s0
    viper.lab172.16.50.13/24ibp68s0

    As part of our initial temporary test, we will apply the IPoIB addresses to the indicated interfaces on each host (all as outlined above. Example temporary config will be for one host. However we will run the command (modified) for each host in our cluster.

    sudo ip link set ibp129s0 up
    sudo ip addr add 172.16.50.11/24 dev ibp129s0

    As you go host to host, verify that the address was assigned correctly.

    9: ibp129s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 1000
        link/infiniband 00:00:03:f2:fe:80:00:00:00:00:00:00:24:8a:07:03:00:ac:56:10 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
        inet 172.16.50.11/24 scope global ibp129s0
           valid_lft forever preferred_lft forever

    Also verify your routing table.

     netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.1.10.1       0.0.0.0         UG        0 0          0 bridge0
10.1.10.0       0.0.0.0         255.255.255.0   U         0 0          0 bridge0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 idrac
172.16.50.0     0.0.0.0         255.255.255.0   U         0 0          0 ibp129s0

    Now perform ping tests from each host and ensure that they can hit the remaining hosts in your cluster. For example.

    ping -I ibp129s0 -c 2 172.16.50.12
    ping -I ibp129s0 -c 2 172.16.50.13

    Once you have tested all three hosts, we can move forward with configuring persistent network configs.

    Persistent RHEL 10 / NetworkManager setup

    prometheus.lab

    sudo nmcli connection add type infiniband ifname ibp129s0 con-name ib-ibp129s0
    sudo nmcli connection modify ib-ibp129s0 ipv4.method manual ipv4.addresses 172.16.50.11/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp129s0
    




    columbia.lab
    



    
	
    
		
    
			

    sudo nmcli connection add type infiniband ifname ibp129s0 con-name ib-ibp129s0
    sudo nmcli connection modify ib-ibp129s0 ipv4.method manual ipv4.addresses 172.16.50.12/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp129s0
    
		
    
	
    

    



    viper.lab
    



    
	
    
		
    
			

    sudo nmcli connection add type infiniband ifname ibp68s0 con-name ib-ibp68s0
    sudo nmcli connection modify ib-ibp68s0 ipv4.method manual ipv4.addresses 172.16.50.13/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp68s0
    
		
    
	
    

    



    




    Confirm Routing
    




    Use “ip route” to ensure that we have the proper route in place for our IPoIB network
    



    
	
    
		
    
			

    ip route
    default via 10.1.10.1 dev bridge0 proto static metric 425 
    10.1.10.0/24 dev bridge0 proto kernel scope link src 10.1.10.25 metric 425 
    169.254.0.0/16 dev idrac proto kernel scope link src 169.254.0.2 metric 100 
    172.16.50.0/24 dev ibp68s0 proto kernel scope link src 172.16.50.13 metric 150 
    
		
    
	
    

    



    Also confirm that NetworkManger sees our IB devices correctly (as Infiniband)
    



    
	
    
		
    
			

    nmcli device status
    DEVICE     TYPE        STATE                   CONNECTION  
    bridge0    bridge      connected               bridge0     
    idrac      ethernet    connected               idrac       
    ibp68s0    infiniband  connected               ib-ibp68s0  
    bond0      bond        connected               bond0       
    enp65s0f0  ethernet    connected               bond0-port0 
    enp65s0f1  ethernet    connected               bond0-port1 
    
		
    
	
    

    



    




    




    IP, IPoIB, and RDMA Usage Matrix
    




    We can use this simple decision matrix to ensure that we understand when to use traditional IP for host to host communication, vs when to use IPoIB, and when native RDMA/IB
    




    Use caseEthernetIPoIBNative RDMA / IB
    Host management, SSH, web UI, package installsBest choicePossible, but usually unnecessaryNo
    Internet access / default routeBest choiceNoNo
    General admin traffic between hostsBest choiceGood for isolated lab trafficNo
    Simple host-to-host testing with ping, ssh, scp, rsync over IB fabricNoBest choiceNo
    NFS/SMB using normal IP networking over the IB fabricNoBest choiceNo
    Fast private storage or migration traffic using standard TCP/IP appsNoBest choiceNo
    RDMA-aware apps using verbs/libibverbsNoNoBest choice
    MPI or cluster workloads built for native IB/RDMANoSometimes, if app specifically uses IPBest choice
    GPUDirect RDMA / high-performance GPU-to-network workflowsNoNoBest choice
    Lowest latency / highest efficiency IB data pathNoNoBest choice
    Easiest troubleshooting and least risk of routing mistakesBest choiceGood if kept isolatedMore specialized
    




    




    Next Steps.
    




    So in Part 2 of our project, we focused on getting InfiniBand up and running, as well as IPoIB. We validated connectivity and setup subnet manager and made sure that our fabric was initialized. We leared a number of IB related command and learned how to read their output. Good Stuff. 
    




    In our next post we will start working with the various NVIDIA tools and projects, many of which will rely on our IB network. Additionally we may try to update firmware on our CX-4s and our IB Switch, however I may skip this step or circle back to it later.
    




    

    

    


  • 


    

    Install APC Netbotz Advanced View on Ubuntu 24.04
    

     
    

    Install APC Netbotz Advanced View on Ubuntu 24.04
    

    

    

    I recently picked up a Netbotz 450 for almost nothing on Ebay. I have always wanted one of these things since I first deployed one in a SunGard facility in Philly about 12-15 years ago.
    




    These devices allow you to connect environmental sensors to monitor temperatures, rack intrusion, humidity, noise. You can also connect a cool siren light (not cheap)  to them and cameras (also super cheap).
    




    When I got mine up on my lab network, I was disappointing to find that there is no ssh or telnet available to connect to this device. There is a WebUI, but that interface is only for metrics and alerts. Any real configuration (beyond IP – which can be done on via serial console) has to be done in a Java app, and getting this app to run on a modern Linux is a bit of a pain. Fortunately I have a fix.
    




    




    First  –  The Linux Binary
    




    While the Linux binary is still available for download in some corners of the internet, I decided to make it available via Google Drive download. You can obtain the file here. 
    




    Second – The Wrapper Script
    




    This script allowed me to install Netbotz Advanced View and and allowed me to configure the Netbotz on Ubuntu 24.04 in Gnome 
    



    
	
    
		
    
			

    #!/bin/bash
    # Wrapper script to run APC NetBotz install.bin
    # This bypasses the LAX launcher which has encoding issues
    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    # Run the installer to extract files
    echo "Extracting installer files..."
    cd "$SCRIPT_DIR"
    ./install.bin 2>/dev/null
    # Find the most recent install.dir.* directory
    INSTALL_DIR=$(ls -td /tmp/install.dir.* 2>/dev/null | head -1)
    if [ -z "$INSTALL_DIR" ]; then
        echo "Error: Could not find extracted installer directory"
        exit 1
    fi
    echo "Found installer directory: $INSTALL_DIR"
    echo "Launching installer in GUI mode..."
    # Run the installer directly with bundled Java
    cd "$INSTALL_DIR"
    "$INSTALL_DIR/Linux/resource/jre/bin/java" \
        -Xmx50m \
        -Xms16m \
        -cp InstallerData:InstallerData/installer.zip \
        com.zerog.ia.installer.Main \
        -i gui
    echo "Installation complete!"
    
		
    
	
    

    




    I have also made the script available for download here. 
    




    Screenshot of a software interface displaying sensor settings and appliance options, including alerts, configuration, and status readings for various sensors.
    





    Without ssh or telnet access to these things I am unsure of exactly they could be configured at scale, unless snmp-set can be used to utilized to enable snmp or setup ntp or alert destinations. This, I may or may not investigate. 
    




    Epilogue
    




    Its also worth noting that snmp is apparently not available until you actually connect a sensor to the device, which makes leads me to believe any non-sensor data will not be available via snmp and there is no API
    



    
	
    
		
    
			

    nmap 10.1.10.9
    Starting Nmap 7.94SVN ( https://nmap.org ) at 2026-01-31 13:32 EST
    Nmap scan report for 10.1.10.9
    Host is up (0.0054s latency).
    Not shown: 998 closed tcp ports (conn-refused)
    PORT    STATE SERVICE
    80/tcp  open  http
    443/tcp open  https
    
		
    
	
    

    



    




    If you are looking to pull data or automate actions, here are your 2 options:
    




    EcoStruxure IT / Data Center Expert (REST API)
    




    If you have a larger environment where your NetBotz 450 is managed by Schneider Electric’s Data Center Expert (DCE) software, you can use the DCE REST API.
    




      

    • How it works: You send API calls to the DCE server rather than the NetBotz appliance itself.
      • 




    • Capability: This allows you to pull data across all monitored devices (including the 450) through a modern JSON-based interface.
      • 

    




    Modbus/TCP
    




    For integration with Building Management Systems (BMS) or industrial controllers, the NetBotz 450 supports Modbus/TCP.
    




    Use case: Useful if you need to feed sensor data into a PLC or SCADA system rather than a custom web app or script.
    




    Other than these two options, you could possibly use python to “log” into urls and set configurations, but I think this would probably be problematic at best. 

    Anyway, I should soon have a sensor arrive via mail, so I will get some more hands on with the device. 
    

    

    


  • 


    

    Dell Force10 S4810 Factory Reset, OS Upgrade, and Basic Setup
    

      
    

    Dell Force10 S4810 Factory Reset, OS Upgrade, and Basic Setup
    

    

    

    The Dell Force10 S4810 is a 48 sfp port, 10Gbe Switch. Let’s walk through a factory reset and some basic configuration tasks as well as an OS upgrade from OS8 to OS9.  
    




    




    Factory Reset
    




    Follow the steps below for factory reset. After running the reload you will be prompted to save the config, ignore this. 
    




    Force10>en
Force10>#delete flash://startup-config
Proceed to delete startup-config.bak [confirm yes/no]:yes 
Force10>reload

    




    




    Configure Management IP Address
    




    In order to be able to connect via ssh/telnet to the S4810 we will need to config a management IP. Below we are configuring an IP address on ManagementEthernet 1/0 , which is the ethernet port next to the console port at the front of the switch.
    




    interface ManagementEthernet 1/0
 ip address 10.1.10.46/24
 no shutdown

    




    We also need to configure the default route.
    




    Force10(conf)#management route 0.0.0.0/0 10.1.10.1
    




    You should now be able to ping the management interface on the switch from your workstation. You should also verify that the switch can ping an ip outside your network, such as 8.8.8.8.
    




    




    Enable SSH/Telnet 
    




    First we create an user and grant privileges. Replace <MYUSER> and <MYPASS> with the username and password that you wish to use.
    




    Force10(conf)#username <MYUSER> admin password <MYPASS> privilege 15 access-class permitall

    




    Now we enable ssh as shown below. 
    




    Force10#configure
Force10(conf)#ip ssh server enable
Force10(conf)#crypto key generate rsa 
Enter key size <1024-2048>. Default<1024>  :2048
Host key already exists .Overwrite (y/n)?y
Generating SSHv2 RSA key.
Force10(conf)#exit


    




    Now we should confirm the running config for ssh.
    




    Force10#show ip ssh
SSH server                : enabled.
SSH server version        : v1 and v2.
Password Authentication   : enabled.
Hostbased Authentication  : disabled.
RSA       Authentication  : disabled.

    




      Now we configure telnet.
    




    Force10#conf t
Force10(conf)#ip telnet server enable
Force10(conf)#exit
    




    Now save your running configuration and attempt a remote connection. Note that you might run into the following error when attempting to ssh to the switch. 
    




    $ ssh cpaquin@10.1.10.49
Unable to negotiate with 10.1.10.49 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
1
    




    Note that the message above indicates that the SSH client doesn’t support the algorithms used by the switch as they are legacy and potentially weak. We can work around this as shown below. I expect once we upgrade the switch OS with a more modern ssh implementation, we will no longer need the work around. 
    




    $ ssh -o KexAlgorithms=diffie-hellman-group14-sha1 -c 3des-cbc chris@10.1.10.46
The authenticity of host '10.1.10.46 (10.1.10.46)' can't be established.
RSA key fingerprint is SHA256:rXqVL/9Ie74e2ZV3m/BiO2KV7adTBXZYwK73UNPEhCo.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.1.10.46' (RSA) to the list of known hosts.
chris@10.1.10.46's password: 
Force10>

    




    Note that once you have upgraded to OS9 you will not need to ssh with the -o or -c options. If you are unable to connect via ssh prior to upgrade you can use telnet for the time being.
    




    




    Configure NTP
    




    First let’s set the timezone and show current time.
    




    Force10(conf)#clock timezone eastern -5 
Force10#show clock
20:46:25.954 eastern Sun Apr 6 2025

    




    Now we will configure the switch to sync time with an upstream ntp server, 10.1.10.10
    




    #ntp server 10.1.10.10
    




    Then confirm association
    




    show ntp associations
   remote       vrf-Id     ref clock     st when poll reach   delay   offset    disp
====================================================================================
 10.1.10.10      0      0.0.0.0         16    -   16    0     +0.000   +0.000   +0.000

    




    It may take a few minutes for the output below to show “sycronized”
    




    Force10#show ntp status  
Clock is synchronized, stratum 2, reference is 216.239.35.0, vrf-id is 0
frequency is 6.813 ppm, stability is 0.002 ppm, precision is -18
reference time  eb9dadd7.8477f9b6  Mon, Apr  7 2025  1:50:15.517 UTC
clock offset is 0.456662 msec, root delay is 13.220 msec
root dispersion is 1.261 msec, peer dispersion is 0.608 sec
peer mode is client

    




    




    OS/Firmware Upgrade
    




    Dell Open Networking switches use ONIE (Open Network Install Environment) as the bootloader and OS installer, and it relies on U-Boot (Universal Bootloader) for fundamental features like reading/writing boot flash and environment variables.
    




    Per Dell, in order to upgrade the switch you need to complete the following steps. 
    




    ● Upgrade the S4810 Dell Networking OS Image and Boot Code
    ● Upgrade the CPLD (not needed in this case)
    




    So let’s first determine what OS version is running on our switch. 
    




    
#show ver
Dell Force10 Real Time Operating System Software
Dell Force10 Operating System Version: 1.0
Dell Force10 Application Software Version: 8.3.10.3
Copyright (c) 1999-2011 by Dell Inc.
Build Time: Tue May 15 22:31:24 PDT 2012
Build Path: /sites/sjc/work/build/buildSpaces/build06/E8-3-10/SW/SRC/Cp_src/Tacacs
Force10 uptime is 3 hour(s), 48 minute(s)

System image file is "system://A"

System Type: S4810 
Control Processor: Freescale QorIQ P2020 with 2147483648 bytes of memory.

128M bytes of boot flash memory.

  1 52-port GE/TE/FG (SE)
 48 Ten GigabitEthernet/IEEE 802.3 interface(s)
  4 Forty GigabitEthernet/IEEE 802.3 interface(s)

    




    You can see in the output above that the switch is currently running 8.3.10.3. 
    




    Before upgrading Dell Networking OS on S4810 from a version prior to 8.3.12.0 to version 9.14(1.14), ensure to  increase the partition size by upgrading to version 8.3.12.0.
    




    According to Dell’s documentation I need to upgrade to 8.3.12.0 before I can upgrade to 9.14. 
    




    




    Upgrading to 8.3.12.0 
    




    I have configured vsftp on my workstation as I will use ftp as my upgrade method.  Note that the step below may take a few minutes to complete. 
    




    Force10#upgrade system ftp: a:
Address or name of remote host []: 10.1.10.10
Source file name []: FTOS-SE-8.3.12.2.bin
User name to login remote host: ftpuser
Password to login remote host: 

    




    Verify the Dell Networking OS has been upgraded correctly in the upgraded flash partition via the command below. 
    




    Force10>show boot system stack-unit all

Current system image information in the system:
=============================================

Type          Boot Type     A                        B
----------------------------------------------------------------
Stack-unit 0 is not present.
Stack-unit 1  FLASH BOOT    8.3.12.2[boot]           8.3.7.0                  
Stack-unit 2 is not present.
Stack-unit 3 is not present.
Stack-unit 4 is not present.
Stack-unit 5 is not present.
Stack-unit 6 is not present.
Stack-unit 7 is not present.
Stack-unit 8 is not present.
Stack-unit 9 is not present.
Stack-unit 10 is not present.
Stack-unit 11 is not present.

    




    Now save and reload.
    




    Force10#write mem
Force10#reload
Proceed with reload [confirm yes/no]: yes

    




    




    Upgrade the S4810 Boot Code
    




    Dell Networking OS version 9.14(1.10) requires S4810 Boot Code version 1.2.0.5. Below we can see that we are currently running 1.2.0.2, so we will need to upgrade the boot code as well prior to upgrading to 9.14(1.10). 
    




    Force10>show system stack-unit 1 | grep "Boot Flash"
Boot Flash      :  1.2.0.2
    




    Again we will use ftp as our file transfer/upgrade.
    




    Force10#upgrade boot ftp:
Address or name of remote host []: 10.1.10.10
Source file name []: U-boot.1.2.0.5.bin
User name to login remote host: ftpuser
Password to login remote host: 
!
Erasing SSeries BootImageUpgrade Table of Contents, please wait
.!........................................!
524528 bytes successfully copied

    




    Now save and reload
    




    Force10#write mem  
!
Force10#reload

    




    




    Upgrade to 9.14
    




    Once the switch as reloaded we are ready to upgrade to 9.14. We will use ftp once again. 
    




    Force10#upgrade system ftp: a:
Address or name of remote host []: 10.1.10.10
Source file name []: FTOS-SE-9.14.1.14.bin
User name to login remote host: ftpuser
Password to login remote host: 

    




    Now write to memory and reload the OS.
    




    Force10#write mem
!

Force10#reload
Proceed with reload [confirm yes/no]: yes

    




    Wait for the OS to reload and then use “show ver” to check the running OS image. 
    




    Force10>show ver
Dell Real Time Operating System Software
Dell Operating System Version:  2.0
Dell Application Software Version:  9.14(1.14)

    




    




    References
    




      

    1. https://www.dell.com/community/en/conversations/networking-general/s4810-firmware/647f9ce3f4ccf8a8de1135a6
      2. 




    2. https://www.dell.com/support/kbdoc/en-us/000182885/ftos-for-s-series-s4810-platform?dgc=SM&cid=304571&lid=spr7038757706&refid=sm_LITHIUM_spr7038757706&linkId=167138678
      3. 




    3. https://www.reddit.com/r/homelab/comments/c9wdnq/dell_force_10_s60_44t_help_round_2/
      4. 




    4. https://dl.dell.com/manuals/all-products/esuprt_networking_int/esuprt_networking_operating_systems/dell-emc-os-9_release-notes30_en-us.pdf
      5. 




    5. https://www.dell.com/support/kbdoc/en-us/000122301/how-to-enable-ssh-and-disable-telnet-on-force10-s-series-switches
      6. 

    




    

    

    


  • 


    

    Upgrade Cisco Catalyst WS-C4500X IOS: Step-by-Step Guide
    

      
    

    Upgrade Cisco Catalyst WS-C4500X IOS: Step-by-Step Guide
    

    

    

    I recently picked up a Cisco Catalyst WS-C4500X for my homelab as I am in the process of moving to all 10GBe. Before I put the new switch into service I want to make sure that it is running the latest IOS image.
    




    I will be using a USB drive that has already been formatted as FAT32 (FAT16 is also acceptable). I have copied the .bin file to the usb drive.
    




    Plug the USB drive into the front of the switch. Enter enable mode and run dir on the usb drive as shown below. 
    




    Switch>en
Switch#dir usb0:
Directory of usb0:/

   37  drwx       16384  Mar 22 2025 23:28:06 +00:00  .Trash-1000
   38  -rwx   195351416  Mar 18 2025 01:16:52 +00:00  cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin

31440846848 bytes total (31220072448 bytes free)

    




    Now we need to copy the image from the USB drive to the bootflash on the switch.
    




    Switch#copy usb0:cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin bootflash:
    




    The file will take a bit to copy. Once the copy is complete you will be returned to the prompt.
    




    




    We now want to confirm that the file is in bootflash.
    




    Switch#show bootflash:
    




    Output below,  cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin is our new file. As you can see by the dates this switch has not been upgraded in a long while. 
    




    




    Now we need to change the config-register
    




    Switch>en
Password: 
Switch#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#config-register 0x2102

    




    Now we clear out the contents of the bootflash
    




    Switch(config)# no boot system
    




    We must now save the running config in order to see the updated boot vars
    




    Switch#show boot
BOOT variable =
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x2101 (will be 0x2102 at next reload)
    




    Now we configure the switch to boot the new IOS image. 
    




    Switch(config)# boot system bootflash:cat4500e-universal.SPA.03.11.11.E.152-7.E11.bin
    




    Now exit config mode and save changes
    




    Switch(config)#exit
Switch#copy running-config startup-config
Destination filename [startup-config]? 
Building configuration...
Compressed configuration from 1491 bytes to 878 bytes[OK]
    




    Now we confirm our boot config
    




    Switch#show boot
BOOT variable = bootflash:cat4500e-universalk9.SPA.03.11.11.E.152-7.E11.bin,12;
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x2101 (will be 0x2102 at next reload)

    




    Configuration register 0x2101 will boot the first boot image shown in the output of dir bootflash:

    Configuration register 0x2102 overrides this behavior and boots the image specified in “BOOT variable” output.
    




    




    Reference
    




      

    1. https://youtu.be/vaiSC8SzHEc?si=kHM5ak1W_89zHx_j
      2. 




    2. https://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/50421-config-register-use.html
      3. 

    

    

    


  • 


    

    How to Reset Cisco Catalyst WS-C4500X to Factory Defaults
    

     
    

    How to Reset Cisco Catalyst WS-C4500X to Factory Defaults
    

    

    

    Need to reset your Cisco Catalyst 4500X  back to the factory default settings? Have you forgotten your password? Follow the steps below.
    




    Note: that I am assuming that you have already established a console connection to the switch using a Cisco serial cable (rollover cable).
    




    Interrupt the boot sequence with Ctrl-C within the first 5 seconds after power on. This will allow you to enter the “ROM” aka ROM Monitor.
    




    




    Configure the switch to boot-up without reading the configuration memory (NVRAM). 
    




    




    You will then see the following options below. Select as shown.
    




    




    Clear the config as shown below
    




    




    Now boot the switch with the boot command
    




    




    




    Reference
    




      

    1. https://www.cisco.com/c/en/us/support/docs/switches/catalyst-4500-series-switches/116436-technote-product-00.html
      2. 




    2. https://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/21229-pswdrec-cat4000-supiii-21229.html
      3. 

    

    

    


  • 


    

    Installing the GPU Power Supply Expansion Board into the Dell T620
    

    

    Installing the GPU Power Supply Expansion Board into the Dell T620
    

    

    

    




    Introduction
    




    I recently picked up a couple of used Dell T602s for my homelab for AI/ML project work. Dell Tower form factor servers are very attractive to homelabbers due to their availability, their low costs, the fact that they are rather low noise, and due to the fact that they are easily expandable. For example, the DVD rom drive in one of my machines is a standard 5 3/4″ form factor, which I replaced with an DVD burner I had laying around. 
    




    One issue with utilizing a non-ATX powered server for AL/ML is the lack of additional power and cooling options for GPUs. The 16x PCI slots in the T620 provide 75w of power, enough for some older GPUs (like the Tesla T4 – which I will try to install later), however in the case of the T4 it is passively cooled so additional cooling is required. We will deal with additional cooling in another post in the future, for now let’s focus on getting the power we need for a certified GPU (more on this down below). 
    




    What you need is a Dell GPU Power Supply Expansion Module (VDY5T) (and cables PN=3692K). However, it’s almost impossible to obtain one specifically for the T620 (VDY5T), however you can pick up one up for the T630 (X7C1K) on ebay. I have read that others have been successful when doing so.  So I picked up one for myself along with a couple of cables to see if I had the same results…
    




    As a matter of fact, after a cursory glance, and some google-foo, I am not convinced that these parts are not interchangeable. Below you see an ebay item listed as compatible with either machine (not that you should trust ebay sellers). I will continue to do research and once I test my machine I will add more information to this post in the near future. For now, let’s get to the installation steps. 
    




    



    Pages: 1 2