The Dell iDRAC Service Module (ISM) is a tool that can be used for better integration between the Dell iDRAC and a running OS. It can provide additional monitoring and metrics to Idrac by brigding the gap between the OS and the underlying Dell hardware.

iSM collects data from both the operating system and hardware and merges them into the iDRAC Lifecycle Log for consolidated monitoring.

Downloading

I used this link to download for RHEL.

I then scp’d the gzipped tarball to the target system

Installation on RHEL

There are not currently packages for RHEL 10, so I went with the RHEL 9 packages. You will need to install 2 rpms from the tarball. First start with the dcism-ocs rpm.

# dnf -y install ./dcism-osc-7.0.1.0-123.rpm

Next install the dcism rpm.

# dnf -y install ./dcism-5.0.1.0-2951.el9.x86_64.rpm

I expect that if I was running a recognized OS, such as RHEL 9, I could have used the script setup.sh to install the rpms in the proper order.

Start the Service

Use the command below to start/restart the service

# systemctl restart dcismeng.service

Checking the output of “ip a” you will see a new interface that I assume is used to communicate back and forth between the OS and the iDRAC.

12: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether ec:2a:72:01:5b:5b brd ff:ff:ff:ff:ff:ff
    altname enp0s20f0u14u3
    altname enxec2a72015b5b
    inet 169.254.1.2/16 brd 169.254.255.255 scope global idrac
       valid_lft forever preferred_lft forever

Enabling Features

Now that the iDRAC service module is installed we can now run setup.sh should be able to enable and disable features. Note that I did not have much luck in this area, as I was running into issues with missing files which was probably due to the fact that I installed the rpms manually. Either way, you can see what is enabled by default.

Available features are shown below.

OS Information via iDRAC

Now that the service is running and able to communicate with the iDRAC, I can see the following OS level info after logging into the iDRAC

Additional Resources

Being new to the iDRAC service module I will link to some relevant documentation which details features, functionality, and configuration.

Prerequisite – Install python 3.11. The default on Fedora 40 is python 3.12

$ sudo dnf -y install python3.11.x86_64 python3.11-devel.x86_64  -y

Install automake and gcc (GNU Compiler Collection)

$ sudo dnf install make automake gcc gcc-c++

Add the following env vars to your .bashrc, and source

export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

Create your virtual env and activate

$ python3.11 -m venv --upgrade-deps venv
$ source venv/bin/activate

Note that every time you want to use instructlab cli (ilab) post-install you must source the file above. venv/bin/activate

We then run the 3 commands shown below. Apparently there is a known bug and we will not pick up these required packages as needed, so we need to install them manually.

$ pip install --upgrade pip wheel setuptools
$ pip install torch
$ pip install flash-attn

The install guide found here has you run this command to clear out the pip package cache (for this one package. Not required for first install, but may be on subsequent runs.

$ pip cache remove llama_cpp_python

Now install instructlab

$ pip install 'instructlab[cuda]' -C cmake.args="-DLLAMA_CUDA=on" -C cmake.args="-DLLAMA_NATIVE=off" 

Install vllm.

NOTE: The vllm install is currently failing with this error – ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory. I am looking to find a resolution or work-around

$ pip install vllm@git+https://github.com/opendatahub-io/vllm@2024.08.01

Resources

1. https://docs.instructlab.ai/getting-started/linux_nvidia/
2. https://github.com/vllm-project/vllm
3. https://www.redhat.com/en/topics/ai/what-is-vllm
4. https://github.com/opendatahub-io

Rack mount servers are notoriously loud. They are designed to run in data centers which and not next to your head. In a data center no one really notices or cares if a machine’s fans are spinning faster than they need to. 2RU Servers are bad, but 1U are even worse as the smaller the fan the faster they need to spin to move air. Start plugging certified PCI cards into slots and things only get worse. This is exactly my experience with 2 nvidia Tesla T4s in my “used-to-be” quiet Dell T620 (Idrac 7)

Note: Passing fan overrides to the idrac does not survive a reboot/restart and fans won’t spint up if temps rise. See https://github.com/christopherpaquin/Dell-Server-Fan-Shusher for a better solution (GPU aware).

So lets learn to control the noise a bit.

First off you need to enable IPMI over Idrac. In Idrac7 (and Idrac8), log into the web interface, and navigate to IDRAC Settings > Network. Scroll down to IPMI Settings, and make sure that “Enable IPMI Over LAN” is checked. This will allow you to ssh to the idrac and run IPMI commands.

Installing IPMI tool

You will first need to install ipmitool on your workstation.

$ apt install ipmitool -y

Manual Fan Control

Run the command below to turn on manual control over the fans. You need to do this before you can change their speeds.

ipmitool -I lanplus -H $IP -U $USER -P $PASS raw 0x30 0x30 0x01 0x00

If you want to turn off manual control, and return to auto control, you can run the command below

ipmitool -I lanplus -H $IP -U $USER -P $PASS raw 0x30 0x30 0x01 0x01

Finding your Fans

Run the command below to find our how may system fans you have as well as their numbering

$ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin sdr list | grep Fan
Fan1             | 1440 RPM          | ok
Fan2             | 1200 RPM          | ok
Fan3             | disabled          | ns
Fan4             | disabled          | ns
Fan5             | disabled          | ns
Fan6             | disabled          | ns
Fan Redundancy   | Not Readable      | ns

Run the command below to set all fans at 40% speed.

1. 0xff is all fans,
2. 0x26 is approx 40% (scale is from 0 to 64)

$ ipmitool -I lanplus -H $IP -U $USER -P $PASS raw 0x30 0x30 0x02 0xff 0x26

Check the results.

$ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin sdr list | egrep -i fan[1,2]
Fan1             | 2400 RPM          | ok
Fan2             | 2040 RPM          | ok

You can also target individual fans. In the example below we will increase fan1 to 100%. Where…

1. 0x02 is fan1
2. 0x64 is 100%

$  ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin raw 0x30 0x30 0x02 0x01 0x64

Lets do the same for fan1 (0x00 is fan1)

$ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin raw 0x30 0x30 0x02 0x00 0x64

Now lets bring both fans back to about 40%

The one-liner below will show you temps as you want to make sure nothing gets too hot.

$ ipmitool -I lanplus -H 10.1.10.13 -U root -P calvin sdr list | grep Temp
Inlet Temp       | 20 degrees C      | ok
Temp             | 30 degrees C      | ok
Temp             | 25 degrees C      | ok

Additional Tools

Here are a few tools that I have found in my travels

1. Dell IDRAC Fan Controller Docker – is a docker container that allows you to control fan speeds. Have not tried it but looks pretty simple to use.
2. R710 Fan Control
3. Fan Control and Monitoring Script
4. IDRAC Fan Control
5. Server Manager