Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Category: Redhat

  • Mastering Chrony: Essential Commands and Configurations

    Mastering Chrony: Essential Commands and Configurations

    Chrony is a flexible and modern implementation of the Network Time Protocol (NTP). It is designed to synchronize the system clock with NTP servers, reference clocks, or manual input with high accuracy. It excels in environments where the connection to the time source is intermittent or when the system clock is frequently interrupted (such as virtual machines)

    chrony officially replaced the traditional ntpd (NTP Daemon, provided by the legacy ntp package) as the default time synchronization service starting in RHEL 7.

    FeatureChrony (chronyd)ntpd (Legacy)
    Clock CorrectionSlewing Only (Default): Adjusts the clock gradually and precisely. Can handle frequent clock interruptions without losing accuracy.Slewing & Stepping: Slews for small adjustments but uses disruptive stepping (jumping the time) for large offsets, which can impact applications.
    Start-Up TimeFast: Uses makestep at startup to quickly correct large initial errors, then immediately begins slewing.Slow: Waits for the clock to slowly adjust, often taking longer to achieve synchronization.
    NetworkBetter for Intermittent Connections: Handles situations where the server connection is lost or the clock frequently changes (e.g., suspend/resume).Requires Constant Connectivity: Struggles with intermittent network loss and frequent system time changes.
    Resource UseLow: Very small memory footprint and fewer CPU cycles required.Moderate: Larger memory and CPU consumption.
    SecurityMore Secure: Designed with security in mind, often running with fewer privileges.Older Codebase: Has had more vulnerabilities over the years.

    Chrony Basic Commands

    CommandPurposeNotes
    chronyc sourcestats -vShows long-term source stability. Displays long-term statistics for each source, including clock drift and frequency estimation.Useful for spotting an unstable NTP server or bad hardware clock.
    chronyc clientsShows who is using your server. If you are configured as an NTP server, this command lists the hosts that are currently querying you for time (requires the ntpserver directive in your config).Useful for confirming if your NetApp LIF is actually reaching your RHEL box and making a request.
    chronyc makestepForces an immediate time correction. If your clock is severely off (more than a few minutes), this commands the daemon to immediately jump the clock to the correct time, instead of gradually slewing it.Use with Caution! Sudden time jumps can affect applications like databases.
    chronyc ntpdata $IPThis performs a one-time query to the specified server, simulating how the client would communicate with it.$IP seems to need to exist as source in chronyd.conf
    chronyc add server <IP>Adds a server dynamically. Adds a new NTP server to the running configuration without restarting the service.Changes made this way are not permanent and will be lost if chronyd restarts.

    Service Configuration File

    The main configuration file for chronyd is /etc/chrony.conf

    After any config file change, you need to restart the chronyd service

    # systemctl restart chrony

    Show Chrony Clients

    Running “chronyc -n clients -v” will show you the clients currently using a chrony server as a timesource

    ~# chronyc -n clients -v
    Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
    ===============================================================================
    10.1.10.10                     59      0   9   -   115       0      0   -     -
    10.1.10.21                     29      0   9   -    49       0      0   -     -
    10.1.10.49                     44      0   7   -   117       0      0   -     -
    10.1.10.45                     50      0   6   -    10       0      0   -     -
    10.1.10.13                     36      0   7   -    56       0      0   -     -
    10.1.10.18                     39      0   6   -    20       0      0   -     -
    10.1.10.20                     37      0   7   -    27       0      0   -     -
    10.1.10.15                     35      0   7   -    54       0      0   -     -
    10.1.10.50                     24      0   7   -   160       0      0   -     -
    

    Below is a breakdown of the command output

    Hostname

    • The IP address (or hostname) of each NTP client querying this Chrony server.
    • In this case: multiple clients in the 10.1.10.0/24 subnet.

    NTP

    • Total number of valid NTP packets received from that client.
    • This is cumulative since Chrony started.
    • Example: 10.1.10.10 → 59. Means this client has successfully sent 59 NTP requests.

    What to expect

    • Steadily increasing numbers = healthy client
    • Very low numbers = new client or infrequent polling

    Drop (NTP side)

    • Number of NTP packets dropped from this client.
    • Drops occur if packets are malformed, too frequent, or violate limits.

    Int

    • Current polling interval (log2 seconds).
    • Chrony uses exponential polling.
    IntActual Interval
    664 seconds
    7128 seconds
    8256 seconds
    9512 seconds

    IntL

    • Previous polling interval
    • - means no recent change

    Last (NTP side)

    • Seconds since the last NTP packet was received from that client.

    What to expect

    • Should generally be less than or near the polling interval
    • Larger numbers are fine if the client polls slowly

    Command (Cmd) columns

    These relate to Chrony command/control packets, not time sync traffic.

    Cmd

    • Number of chronyc command requests received from that client
    • 0 means the client is only syncing time, not issuing admin commands

    Drop (Cmd)

    • Dropped command packets
    • 0 is ideal

    Int (Cmd)

    • Poll interval for command packets
    • - means none used

    Last (Cmd)

    • Time since last command packet
    • - means no commands received

    Show Chrony Sources

    # chronyc sources -v
    
      .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
     / .- Source state '*' = current best, '+' = combined, '-' = not combined,
    | /             'x' = may be in error, '~' = too variable, '?' = unusable.
    ||                                                 .- xxxx [ yyyy ] +/- zzzz
    ||      Reachability register (octal) -.           |  xxxx = adjusted offset,
    ||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
    ||                                \     |          |  zzzz = estimated error.
    ||                                 |    |           \
    MS Name/IP address         Stratum Poll Reach LastRx Last sample               
    ===============================================================================
    ^- prod-ntp-3.ntp4.ps5.cano>     2  10   377   625  +1096us[+1096us] +/-   49ms
    ^- prod-ntp-4.ntp4.ps5.cano>     2  10   377   504   +144us[ +144us] +/-   51ms
    ^- alphyn.canonical.com          2  10   377   756   +619us[ +804us] +/-   40ms
    ^- prod-ntp-3.ntp1.ps5.cano>     2  10   377   759  +1352us[+1537us] +/-   51ms
    ^+ chi2.us.ntp.li                2  10   377   552  -1204us[-1204us] +/-   20ms
    ^- ovh.maxhost.io                2  10   377   628  -3688us[-3688us] +/-   58ms
    ^* atl-ntp2-0.mattnordhoffd>     2  10   377   741  +1002us[+1188us] +/- 7918us
    ^- 2a01:7e04::2000:5dff:fe1>     4  10   377   739  -1535us[-1535us] +/-   49ms
    
    

    MS (Mode / State)

    • ^ = NTP server
    • = = peer
    • # = local reference clock

    State indicator:

    • * = current best source (in use)
    • + = combined (used with others)
    • - = selectable but not used
    • x = faulty
    • ~ = too variable
    • ? = unreachable

    Stratum

    • Distance from a reference clock (lower is better)
    • GPS/PPS = stratum 0
    • Typical upstream servers = stratum 1–3

    Poll

    • Polling interval (log2 seconds)
    PollInterval
    664 seconds
    7128 seconds
    8256 seconds

    Reach

    • Octal reachability register (last 8 polls)
    • 377 (octal) = perfect reachability
    • Lower values = packet loss or intermittent connectivity

    LastRx

    • Seconds since last successful response
    • First value = adjusted offset
    • Bracketed value = raw measured offset
    • +/- = estimated error margin

    Units:

    • ns = nanoseconds
    • us = microseconds
    • ms = milliseconds

    Show Source Stats

    # chronyc sourcestats -v
                                 .- Number of sample points in measurement set.
                                /    .- Number of residual runs with same sign.
                               |    /    .- Length of measurement set (time).
                               |   |    /      .- Est. clock freq error (ppm).
                               |   |   |      /           .- Est. error in freq.
                               |   |   |     |           /         .- Est. offset.
                               |   |   |     |          |          |   On the -.
                               |   |   |     |          |          |   samples. \
                               |   |   |     |          |          |             |
    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
    ==============================================================================
    prod-ntp-3.ntp4.ps5.cano>  31  14  106m     +0.120      0.212  +1317us   596us
    prod-ntp-4.ntp4.ps5.cano>  33  19  108m     -0.021      0.542   +316us  1235us
    alphyn.canonical.com       33  14  104m     -0.036      0.234   +594us   635us
    prod-ntp-3.ntp1.ps5.cano>  33  19  104m     +0.096      0.342  +1156us   761us
    chi2.us.ntp.li             33  18  107m     -0.087      0.341  -1596us   873us
    ovh.maxhost.io             31  14   96m     +0.147      0.382  -2735us   765us
    atl-ntp2-0.mattnordhoffd>  33  16  104m     +0.083      0.331   +762us   889us
    2a01:7e04::2000:5dff:fe1>  33  19  104m     -0.284      0.513  -1432us  1236us
    

    This shows:

    • Average offset
    • Standard deviation (jitter)
    • Frequency correction
    • Stability over time

    Show Selected Source

    chronyc tracking
    Reference ID    : EE1C3393 (atl-ntp2-0.mattnordhoffdns.net)
    Stratum         : 3
    Ref time (UTC)  : Fri Dec 26 20:54:37 2025
    System time     : 0.000375564 seconds fast of NTP time
    Last offset     : +0.000185856 seconds
    RMS offset      : 0.000497939 seconds
    Frequency       : 33.487 ppm slow
    Residual freq   : +0.008 ppm
    Skew            : 0.299 ppm
    Root delay      : 0.014126181 seconds
    Root dispersion : 0.002563612 seconds
    Update interval : 1033.8 seconds
    Leap status     : Normal
    

    Reference ID

    • The active NTP source currently disciplining your clock.
    • EE1C3393 is the hexadecimal reference identifier.
    • The hostname confirms DNS resolution of the upstream server.

    ✔ This tells you exactly which server is in use.


    Stratum

    • Indicates the distance from a reference clock.
    • Stratum hierarchy:
      • 0 → GPS / atomic clock (not directly used)
      • 1 → Directly connected to stratum 0
      • 2 → Syncing from stratum 1
      • 3 → Syncing from stratum 2

    ✔ Stratum 3 is normal and acceptable for internet-sourced NTP.


    Ref time (UTC)

    • Timestamp of the last successful synchronization sample.
    • Expressed in UTC.

    ✔ Confirms recent and valid sync.


    System time

    • Your local clock is ahead of true NTP time by ~0.38 ms.
    • Chrony is actively correcting this.

    ✔ Sub-millisecond offset is excellent.


    Last offset

    • Offset measured during the most recent update.
    • Positive = system clock was fast.

    ✔ Very small deviation (~0.19 ms).


    RMS offset

    • Root Mean Square of offsets over recent samples.
    • Represents overall synchronization quality.

    ✔ <1 ms RMS is very good for non-GPS NTP.


    Frequency

    • How far your system clock naturally drifts without correction.
    • ppm = parts per million.
    • “Slow” means your hardware clock loses ~33 µs per second.

    ✔ Completely normal for commodity hardware.


    Residual freq

    • Remaining frequency error after corrections.
    • Shows how well Chrony has tuned the clock.

    ✔ Near-zero = excellent convergence.


    Skew

    0.299 ppm
    
    • Estimated uncertainty in frequency measurement.
    • Lower = higher confidence.

    ✔ <1 ppm is strong stability.


    Root delay

    0.014126181 seconds
    
    • Total round-trip delay to the reference clock.
    • Includes network latency through the NTP chain.

    ✔ ~14 ms is expected for internet NTP.


    Root dispersion

    • Estimated maximum error relative to true time.
    • Accumulates over time since last sync.

    ✔ ~2.5 ms is very good.


    Update interval

    • Time between Chrony updates (~17 minutes).
    • Increases as clock stability improves.

    ✔ Indicates a stable and trusted source.


    Leap status

    • No leap second pending or in progress.

    ✔ Required for accurate timekeeping.

  • Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Step-by-Step Nvidia Driver, CUDA Toolkit, & Container Toolkit Install for RHEL9

    Introduction

    In this step-by-steps guide we will replace the out of the box nouveau drivers on RHEL9 with Nvidia Drivers. We will also install the the Nvidia CUDA Toolkit and the Nvidia Container Toolkit.


    GPU and Driver Inspection

    First we need to make sure that our Nvdia GPU is recognized by Red Hat Enterprise Linux 9 (RHEL9).

    lspci -nn | grep -i nvidia
    b6:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [A40] [10de:2235] (rev a1)
    

    Using the command below we can see that we are currently using the non-propietary nouveau driver.

    # lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nouveau
    	Kernel modules: nouveau
    

    Configuring Repositories for the Nvidia Driver Install

    First we need to enable the RHEL9 CodeReady Builder repo. Note we are running these commands as root.

    # subscription-manager repos --enable codeready-builder-for-rhel-9-$(uname -i)-rpms
    

    Next we will need to install and configure the EPEL repo.

    # dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
    

    Now we install the ELRepo project repo – this will provide nvidia-detect which we can utilize later

    # dnf -y  install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm

    Prerequisites for Nvidia Driver Install

    Now we need to install dependencies and build tools.

    # dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc make dkms acpid libglvnd-glx libglvnd-opengl libglvnd-devel pkgconfig
    

    Install Nvidia Drivers

    Install nvidia-detect from the ELRepo project repo.

    # dnf -y install nvidia-detect

    Now install the Nvidia Drivers.

    # dnf -y install $(nvidia-detect)

    Now reboot.


    Confirming Nvidia Driver Installation

    Now lets run the command below one more time.

    [root@gpu ~]# lspci | grep ' NVIDIA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
    b6:00.0 3D controller: NVIDIA Corporation GA102GL [A40] (rev a1)
    	Subsystem: NVIDIA Corporation Device 145a
    	Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0
    	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    	Memory at 38d000000000 (64-bit, prefetchable) [size=64G]
    	Memory at 38f040000000 (64-bit, prefetchable) [size=32M]
    	Capabilities: [60] Power Management version 3
    	Capabilities: [68] Null
    	Capabilities: [78] Express Legacy Endpoint, MSI 00
    	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
    	Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
    	Capabilities: [100] Virtual Channel
    	Capabilities: [250] Latency Tolerance Reporting
    	Capabilities: [258] L1 PM Substates
    	Capabilities: [128] Power Budgeting <?>
    	Capabilities: [420] Advanced Error Reporting
    	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
    	Capabilities: [900] Secondary PCI Express
    	Capabilities: [bb0] Physical Resizable BAR
    	Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
    	Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
    	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
    	Capabilities: [d00] Lane Margining at the Receiver <?>
    	Capabilities: [e00] Data Link Feature <?>
    	Kernel driver in use: nvidia
    	Kernel modules: nouveau, nvidia_drm, nvidia
    

    As you can see in the output below, the kernel is loading the Nvidia driver. We can still see nouveau kernel modules listed, but that is fine, as they are not loaded. We can confirm this with the command below.

    # lsmod | grep nouveau

    The above command should not output anything, while the opposite should be true for the command below.

    # lsmod | grep nvidia

    Configure Nvidia Persistenced

    Start and enable nvidia-persistenced.service. This will enable persistence-mode which will keep the nvidia device state from going “stale”

    # systemctl enable nvidia-persistenced.service
    # systemctl start nvidia-persistenced.service
    

    Installing the Nvidia CUDA Toolkit

    We will now follow the official guide and install the Nvidia CUDA toolkit. Per that guide, we need to enable a few repos, however two of those repos should be enabled by default, and the other one we enabled above, however I will list them here for the sake of documentation.

    # subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
    # subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
    # subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

    Now we install the Nvidia repo for the CUDA toolkit.

    dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
    

    Now install the CUDA toolkit as shown below

    # sudo dnf -y install cuda-toolkit

    Confirm that the toolkit is installed and note the version.

    # rpm -qa cuda-toolkit
    cuda-toolkit-12.8.1-1.x86_64
    

    Add the following to your .bashrc. And if you intend to run/install anything as root, you may want to add it to root’s .bashrc as well. Note that the cuda version should match the one that you installed above.

    export PATH=/usr/local/cuda-12.8/bin:$PATH

    Now test nvcc as shown below.

    # nvcc --version
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Feb_21_20:23:50_PST_2025
    Cuda compilation tools, release 12.8, V12.8.93
    Build cuda_12.8.r12.8/compiler.35583870_0
    

    Installing the Nvidia Container Toolkit

    Next we will install the Nvidia Container Toolkit, which allows users to run GPU-accelerated containerized applications.

    A bit about Container Management in RHEL 9

    The default container packages in RHEL 9 are as follows.

    1. Podman – daemonless container image
    2. Buildah – tool for building OCI (Open Container Initiative) container images
    3. Skopeo – tool for managing container images and repos
    4. CRIU – tool to create and save running container checkpoints to disk
    5. Udica – tool for managing SELinux policies for containers

    Installation of the toolkit

    We will follow the instructions as documented here.

    First we configure the repo

    # curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

    Then install via dnf

    # dnf install -y nvidia-container-toolkit

    Configuring the Container Toolkit for Podman

    Generate the CDI specification file using the command below.

    # nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

    Now lets check the names of the generated device(s).

    # nvidia-ctk cdi list
    INFO[0000] Found 3 CDI devices                          
    nvidia.com/gpu=0
    nvidia.com/gpu=GPU-7e880be2-891c-72e3-9515-0fd51240e7f4
    nvidia.com/gpu=all
    

    References

    1. https://medium.com/@blackhorseya/step-by-step-guide-to-installing-nvidia-drivers-on-rhel-9-1107e0cd641d
    2. https://access.redhat.com/discussions/227d2101-b4e3-490a-aa1c-601c407ec038
    3. https://darryldias.me/2022/install-nvidia-drivers-on-rhel-9/
    4. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
    5. https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-rhel-rocky
    6. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html
  • How to Install and Configure Dnsmasq with Web Frontend on RHEL 9

    How to Install and Configure Dnsmasq with Web Frontend on RHEL 9

    In this post we are going to install and configure dnsmasq with a simple webUI acting as a front end for our less technical users. Our goal is to simplify dns in our lab sandboxes, and keep users from directly modifying our dnsmasq config files.

    Installation

    First we need to install dnsmasq.

    # dnf -y install dnsmasq

    Now lets enable and start the service. We will also check the status of the service to ensure that we do not have any issues with the default config.

    # systemctl start dnsmasq
    # systemctl enable dnsmasq
    # systemctl status dnsmasq

    Configuration

    Next let’s make a backup of the default configuration file before we start making modification.

    cp /etc/dnsmasq.conf /etc/dnsmasq.conf.orig

    I made the following modifications.

    1. listen-address is the loopback and our routable ip address
    2. expand-hosts, we uncomment this in order to allow dnsmasq to automatically expand the hostnames to fully qualified domain names
    3. domain – this is the local domain that we will serve via dnsmasq
    4. dhcp-range – this is the range of IP addresses that dnsmasq is allowed to hand out
    interface=enp1s0
    listen-address=127.0.0.1,192.168.65.7
    expand-hosts
    domain=sandbox3.localdomain
    dhcp-range=192.168.65.20,192.168.65.40,255.255.255.128,12h
    

    Next we check our config file for any errors

    # dnsmasq --test
    dnsmasq: syntax check OK.

    Now lets restart dnsmasq

    # systemctl restart dnsmasq

    Firewall Config

    Now we need to modify firewalld

    [root@dns ~]#  firewall-cmd --add-service=dns --permanent
    success
    [root@dns ~]# firewall-cmd --add-service=dhcp --permanent
    success
    [root@dns ~]# firewall-cmd --reload
    success

    Testing

    I prefer to use nslookup for testing, so lets install it

    # dnf -y install bind-utils

    Now let’s make sure we are able to resolve addresses, using our local instance of dnsmasq.

    # nslookup
    > server localhost
    Default server: localhost
    Address: ::1#53
    Default server: localhost
    Address: 127.0.0.1#53
    > google.com
    Server:		localhost
    Address:	::1#53
    
    Non-authoritative answer:
    Name:	google.com
    Address: 142.251.40.174
    Name:	google.com
    Address: 2607:f8b0:4006:821::200e
    

    Install and Configure the Webui

    For the webui – we are going to use a simple one that I found via the link below. https://github.com/akivajp/dnsmasq-webconf

    First, we need to install git and pip

    # dnf -y install git
    # dnf -y install pip

    Then we follow the directions which I will repeat here.

    # mkdir -p ~/git && cd ~/git
    # git clone https://github.com/akivajp/dnsmasq-webconf.git

    Then we use pip to install jinja2

    # pip install --user bottle jinja2

    Now change directory

    # cd dnsmasq-webconf/

    We now need to poke a hole for http traffic in our local firewall

    # firewall-cmd --add-service=http --permanent
    # firewall-cmd --reload

    Now start the front end

    #  python ~/git/dnsmasq-webconf/app/index.py 80 --leases /var/lib/dnsmasq/dnsmasq.leases --hosts /etc/hosts --config /etc/dnsmasq.conf

    Creating a Service

    So far we have dnsmasq configured and running, and we have installed a web front end and have been able to start it on the command line. Now we need to configure the front end to start as a service when the OS boots. So we now need to turn the webUI into a systemd service.

    First we create the following service file

    # vi /etc/systemd/system/dnsmasq-webconf.service

    The contents of which are below. Note that we have modified relative paths to be absolute paths.

    [Unit]
    Description=DNSMasq WebConf
    After=network.target
    
    [Service]
    ExecStart=/usr/bin/python3 /root/git/dnsmasq-webconf/app/index.py 80 --leases /var/lib/dnsmasq/dnsmasq.leases --hosts /etc/hosts --config /etc/dnsmasq.conf
    Restart=always
    User=root
    WorkingDirectory=/root/git/dnsmasq-webconf/app
    StandardOutput=journal
    StandardError=journal
    
    [Install]
    WantedBy=multi-user.target

    Now we need to reload systemd

    # systemctl daemon-reload

    Now we can start the service

    # systemctl start dnsmasq-webconf.service

  • Red Hat OpenShift 4.x: Adding a new user with admin rights

    Getting Started

    Once the initial baremetal OpenShift 4.6 IPI install is complete, you will see output similar to what is shown below.

    This output contains the URL for the OpenShift WebUI along with the “kubeadmin” password. You will be able to log into the WebUI using these credentials.

    If you forget your kubeadmin login (which you will,. because it’s ugly) you will be able to find in the auth directory.

    {ocptest}[kni@bastion ~]$ cat clusterconfigs/auth/kubeadmin-password 
    WMAWX-QkGc3-IQKy2-XJDI6

    Upon initial login to the WebUI you will see a blue bar at the near the top of the page – as shown below.

    Adding a New User with Admin Rights

    Bottom line, the Openshift “kubeadmin” password is not user friendly. In my lab I do not want to have to look it up each time I attempt to access the webui. I need simple and easy to remember credentials.

    So I am going to create a new admin user. In the example below I am creating a user named ‘admin‘ with the password of ‘admin‘.

    I am working in a homelab, so security is not exactly a priority.

    {ocp}[kni@bastion ~]$ htpasswd -c -B -b users.htpasswd admin admin
    Adding password for user admin
    

    Now I define a secret which uses the HTPasswd user file as shown below.

    {ocp}[kni@bastion ~]$ oc create secret generic htpass-secret --from-file=htpasswd=/home/kni/users.htpasswd -n openshift-config
    secret/htpass-secret created
    

    Next, I create a custom resource (htpasswd.cr) that defines the use of the HTPasswd identity provider. This file is straight out of the user doc and is unmodified.

    apiVersion: config.openshift.io/v1
    kind: OAuth
    metadata:
      name: cluster
    spec:
      identityProviders:
      - name: my_htpasswd_provider 
        mappingMethod: claim 
        type: HTPasswd
        htpasswd:
          fileData:
            name: htpass-secret 

    Now we apply the Custom Resource (CR) via oc apply.

    {ocp}[kni@bastion ~]$ oc apply -f /home/kni/htpasswd.cr
    oauth.config.openshift.io/cluster configured
    

    Now let’s test your new id and password on the CLI. If the step below fails, you will need to start all over again.

    {ocp}[kni@bastion ~]$ oc login -u admin
    Authentication required for https://api.ocp.lab.localdomain:6443 (openshift)
    Username: admin
    Password: 
    Login successful.
    
    You have access to 59 projects, the list has been suppressed. You can list all projects with ' projects'
    
    Using project "default".
    

    Confirm URL for WebUI.

    {ocp}[kni@bastion ~]$ oc whoami --show-console
    https://console-openshift-console.apps.ocp.lab.localdomain
    

    Navigate to the URL above and select my_htpasswd_provider. Enter your new credentials. At this point you should be able to login without issue. If not, please work through the process again.

    Troubleshooting

    If you do run into errors, the files that you created and run the command below to delete the secret.

    oc delete secret htpass-secret -n openshift-config

    At this point you should be able to repeat the procedure and check for errors.

    Reference

    https://docs.openshift.com/container-platform/4.5/authentication/identity_providers/configuring-htpasswd-identity-provider.html#identity-provider-creating-htpasswd-file-linux_configuring-htpasswd-identity-provider

  • RHEL 8/CentOS 8: Install and Enable Libvirt

    [root@bastion ~]$  dnf module install virt 
    
    [root@bastion ~]$  dnf install virt-install virt-viewer
    [root@bastion ~]$  systemctl start libvirtd
    [root@bastion ~]$  systemctl enable libvirtd
    [root@bastion ~]$  systemctl status libvirtd
    ● libvirtd.service - Virtualization daemon
       Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
       Active: active (running) since Mon 2020-12-14 15:16:44 EST; 8s ago
         Docs: man:libvirtd(8)
               https://libvirt.org
     Main PID: 33522 (libvirtd)
        Tasks: 19 (limit: 32768)
       Memory: 22.2M
       CGroup: /system.slice/libvirtd.service
               ├─33522 /usr/sbin/libvirtd --timeout 120
               ├─33653 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
               └─33654 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
    
    
    [root@bastion ~]$  virsh list --all
     Id   Name   State
    --------------------
    
    
  • Installing RHEL 8.1 on Dell R710/R610 with H700 Raid Controller

    sys_r710.png

    Introduction

    A large number of storage controller drivers have been removed from RHEL 8.x, which means that the Dell R710/R610 with the H700 RAID controller, no longer supports RHEL 8.x natively. While this fact is unfortunate for a lot of hobbyists with their own homelabs, you can still install RHEL 8.x on these machines with the use of a driver update disk (DUD).

    Setup

    Using the link below, navigate and download the deprecated drivers.

    https://elrepo.org/linux/dud/el8/x86_64

    For Dell the R710/R610, you are specifically looking for the megaraid_sas drivers. See output below from R610.

    # dmesg | grep raid
    [ 1.402339] megaraid_sas 0000:03:00.0: FW now in Ready state
    [ 1.402346] megaraid_sas 0000:03:00.0: 64 bit DMA mask and 32 bit consistent mask
    [ 1.402571] megaraid_sas 0000:03:00.0: irq 34 for MSI/MSI-X
    [ 1.402593] megaraid_sas 0000:03:00.0: firmware supports msix : (0)

    The specific DUD iso that you need for RHEL 8.1 is shown below.

    https://elrepo.org/linux/dud/el8/x86_64/dd-megaraid_sas-07.707.51.00-1.el8_1.elrepo.iso

    Installation

    The install process is as follows.

    1. Download RHEL 8.x media and burn to dvd or usb drive
    2. Download deprecated drivers in iso format and burn to usb drive
    3. Boot with both RHEL 8.x media and DUD mounted
    4. The installer should detect the DUD iso and install the proper drivers

    Special Note: In my testing (3 systems) the DUD was not automatically detected by Anaconda.  I suggest using the method below.

     

    Troubleshooting

    When you burn your DUD a usb stick, that the usb drive’s volume label is OEMDRV. This was the default when I burned the iso. The RHEL installer (anaconda) will look for this label on any and all available disks and should automatically recognize the DUD and mount the usb drive.

    However, if this does not occur during the install process, and the installer still does not see your disks, you may need to reboot and this time interrupt the installer with the TAB key and append the following to your boot options.

    inst.dd=/dev/sdb1

    On my system, the DUD was /dev/sdb1 and the RHEL 8.1 install media was /dev/sda.

    Resources:

    https://access.redhat.com/discussions/3722151

    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/performing_an_advanced_rhel_installation/index#updating-drivers-during-installation_installing-rhel-as-an-experienced-user

    https://elrepo.org/linux/dud/el8/x86_64

    https://serverfault.com/questions/908209/how-to-add-a-dud-driver-to-centos-rhel-7-media-without-network