Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

  • Fixing the OpenSSH Post-Quantum Warning on RHEL 9

    Fixing the OpenSSH Post-Quantum Warning on RHEL 9

    Fired up my RHEL9 server today, something that I have not done in a while due to the savage summer heat in the southern US. All my Dell servers have had the summer off for this reason.

    Upon login via ssh I saw a warning message, which I do not think I have seen before on this machine – all my other Dell servers are RHEL 10, and this one was built with RHEL 9 due to deprecated support for older processors in RHEL 10. Specifically the Intel Xeon E5-2697 (Ivy Bridge) v2.

    Anyway, below is the warning message.

    ** WARNING: connection is not using a post-quantum key exchange algorithm.
    ** This session may be vulnerable to "store now, decrypt later" attacks.
    ** The server may need to be upgraded. See https://openssh.com/pq.html

    This warning means that the SSH client supports post-quantum key exchange, but the RHEL 9 SSH server did not offer one. It does not indicate that the current connection is broken; it warns about possible future decryption of captured traffic.

    Hence the “store now, decrypt later message above.


    SNDL/HNDL

    “Store Now, Decrypt Later” (also called Harvest Now, Decrypt Later) is when an attacker intercepts encrypted traffic, not with the idea of decrypting it now, but rather once they have access to a more powerful quantum computer running advanced algorithms. Today’s standard SSH connections rely on classical asymmetric cryptography (like Elliptic Curve Diffie-Hellman) which quantum computers can theoretically break.

    Meaning, you may not be vulnerable now, but you could be later.

    Most newer OpenSSH clients warn when a connection does not negotiate a post-quantum key-exchange algorithm because long-lived or sensitive data captured today could remain valuable in the future.

    Honestly this is not a big concern in my lab, but the warning will drive me up a wall.


    The RHEL 9 Fix

    The good news is that RHEL 9.1 and later support the following hybrid post-quantum SSH key-exchange method. So we can remediate.

    sntrup761x25519-sha512@openssh.com

    This combines the post-quantum sntrup761 algorithm with the traditional X25519 elliptic-curve key exchange. Because it is hybrid, the connection remains protected as long as either component remains secure.

    Red Hat added support for this algorithm to the RHEL 9 system-wide cryptographic-policy framework, but it must be explicitly enabled through a custom subpolicy.


    Confirm the RHEL Version

    First, lets confirm that the server is running RHEL 9.1 or later:

    cat /etc/redhat-release

    You can also check the installed OpenSSH and crypto-policy package versions:

    rpm -q openssh-server crypto-policies

    Update OpenSSH and the Crypto-Policy Packages

    sudo dnf update openssh-server crypto-policies

    This ensures that the server has the RHEL packages that include support for the SNTRUP hybrid key-exchange algorithm.

    In my case, this step initially revealed a separate Red Hat subscription problem:

    Status code: 403 for https://cdn.redhat.com/...

    A 403 from the Red Hat CDN generally means that the server can reach the repository, but the system is not currently authorized to download its content. This may be caused by an expired developer subscription, stale registration data, an invalid entitlement, or a system identity left over from an earlier installation.

    Useful commands for checking the registration state include:

    sudo subscription-manager identity
    sudo subscription-manager status
    sudo subscription-manager repos --list-enabled

    As I said this machine has been powered off for a while, so lets refresh registration and repository metadata.

    sudo subscription-manager refresh
    sudo dnf clean all
    sudo rm -rf /var/cache/dnf
    sudo dnf makecache

    In my case, the system registration is stale, so I need to unregister and re-register.

    sudo subscription-manager unregister
    sudo subscription-manager clean
    sudo subscription-manager register

    After registration, we ensure that the standard RHEL 9 repositories are enabled:

    sudo subscription-manager repos \
    --enable=rhel-9-for-x86_64-baseos-rpms \
    --enable=rhel-9-for-x86_64-appstream-rpms

    Then rebuild the DNF cache:

    sudo dnf clean all
    sudo rm -rf /var/cache/dnf
    sudo dnf makecache

    Create a Custom Crypto-Policy Module

    Next we create a new policy module named SNTRUP:

    sudo tee /etc/crypto-policies/policies/modules/SNTRUP.pmod >/dev/null <<'EOF'
    key_exchange = +SNTRUP
    EOF

    The module name must use uppercase characters because crypto-policy module filenames are conventionally uppercase and end with .pmod.

    The policy adds the SNTRUP algorithm to the key-exchange methods permitted by the existing system-wide policy.


    Apply the Policy

    Apply the custom module on top of the normal RHEL DEFAULT policy:

    sudo update-crypto-policies --set DEFAULT:SNTRUP

    Confirm that the new policy is active:

    update-crypto-policies --show

    You are looking for expected output as shown below.

    DEFAULT:SNTRUP

    RHEL uses system-wide cryptographic policies to configure applications including OpenSSH, TLS libraries, IPsec, DNSSEC, and Kerberos. Using a crypto-policy module is therefore preferable to manually changing generated OpenSSH backend files.


    Restart SSH

    Next we restart sshd so that it reads the updated cryptographic policy:

    sudo systemctl restart sshd

    And we confirm that the service restarted without issue

    sudo systemctl status sshd --no-pager

    Verify the Server Configuration

    Run the command below to check the current SSH server key-exchange configuration:

    sudo sshd -T | grep -i '^kexalgorithms'

    The output should include:

    sntrup761x25519-sha512@openssh.com

    We can also inspect the generated OpenSSH server crypto-policy backend:

    grep -i kexalgorithms \
    /etc/crypto-policies/back-ends/opensshserver.config

    Files beneath /etc/crypto-policies/back-ends/ are generated by the crypto-policy framework and should not normally be edited directly. Manual changes may be overwritten the next time the system policy is updated.

    Now it’s time to test.

    7. Test a New SSH Connection

    Open a new terminal window, leaving the existing window open just in case we broke something. In the second window, connect with verbose logging:

    ssh -vv user@server

    Look for a line similar to:

    debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com

    The original post-quantum warning should no longer appear.

    A more focused test can be performed by explicitly requesting the algorithm:

    ssh \
    -o KexAlgorithms=sntrup761x25519-sha512@openssh.com \
    user@server

    If this succeeds, both the SSH client and server support the hybrid key exchange.

    We can check whether the local SSH client recognizes the algorithm with:

    ssh -Q kex | grep -i sntrup

    Expected output:

    sntrup761x25519-sha512@openssh.com

    Depending on the client version, another SNTRUP variant may also be listed.


    Troubleshooting

    If the algorithm does not appear in the output from sshd -T, first confirm that the custom policy is active:

    update-crypto-policies --show

    Then confirm that the policy module contains the correct entry:

    cat /etc/crypto-policies/policies/modules/SNTRUP.pmod

    It should contain:

    key_exchange = +SNTRUP

    Reapply the policy and restart SSH:

    sudo update-crypto-policies --set DEFAULT:SNTRUP
    sudo systemctl restart sshd

    Check the SSH service logs for errors:

    sudo journalctl -u sshd -b --no-pager

    You can also validate the SSH configuration before restarting the service:

    sudo sshd -t

    No output means that the configuration passed validation.


    Important FIPS Consideration

    Do not replace an existing FIPS policy with DEFAULT:SNTRUP on a system required to operate in FIPS mode.

    Check the current policy first:

    update-crypto-policies --show

    Also check whether the kernel is operating in FIPS mode:

    cat /proc/sys/crypto/fips_enabled

    A value of 1 means that FIPS mode is active.

    Algorithms permitted by the normal RHEL DEFAULT policy are not automatically permitted under the RHEL FIPS policy. Red Hat documents that certain Curve25519-based SSH key exchanges are not allowed in FIPS mode, so the SNTRUP hybrid method should not be enabled without validating the applicable compliance requirements.


    Rolling Back the Change

    To remove the SNTRUP subpolicy and return to the standard RHEL default policy:

    sudo update-crypto-policies --set DEFAULT
    sudo systemctl restart sshd

    Confirm the result:

    update-crypto-policies --show

    Expected output:

    DEFAULT

    The custom policy file can then be removed if it is no longer needed:

    sudo rm -f /etc/crypto-policies/policies/modules/SNTRUP.pmod

    Final Result

    After enabling the RHEL crypto-policy module, restarting sshd, and creating a new connection, the server and client should negotiate:

    sntrup761x25519-sha512@openssh.com

    The SSH session still uses familiar symmetric encryption and authentication mechanisms. The portion being changed is the initial key exchange used to establish the session keys.

    This does not make the entire SSH protocol “post-quantum.” It adds a hybrid post-quantum key-exchange mechanism designed to protect the session-establishment process against both traditional attacks and the future possibility of cryptographically capable quantum computers.

    This server that has a few years of useful service (specifically for my Infiniband testing), but cannot move to RHEL 10. So enabling the supported RHEL 9 hybrid key exchange is “good enough”.

  • The July 2026 Agentic Coding Cheat Sheet: Frontier Tools, Cost Models, and Quota Mechanics – V2

    The July 2026 Agentic Coding Cheat Sheet: Frontier Tools, Cost Models, and Quota Mechanics – V2

    The tool sprawl in AI engineering has officially outrun our collective memory, at least it has mine.

    Between native terminal agents, newly minted desktop clients, and entirely divergent API cost models, keeping the ecosystem straight is a full-time job.

    I built this reference guide for a very simple reason: I needed a single, no-nonsense source of truth to track what tools each frontier provider actually offers right now, and exactly how running autonomous developer loops will affect my wallet.

    Here is the state of the stack as I know it and use it. Sure there are plenty of options I am missing.


    The Tooling Matrix: Interfaces, CLIs, and IDEs

    This matrix maps out exactly where each provider expects you to interact with their models, highlighting the shift toward dedicated desktop environments and autonomous terminal agents.

    Provider / EcosystemFlagship Web / Workspace InterfaceOfficial Desktop AppNative CLI / Terminal AgentIDE Integration Strategy
    GitHub CopilotGitHub.com & Copilot Workspace (Issue-to-PR sandbox)GitHub Copilot App (Native agent-centric control center)GitHub Copilot CLI (TUI via gh copilot with local execution capabilities)Official Extension (VS Code, JetBrains, Visual Studio, Neovim)
    Anthropic (Claude)Claude.ai (with Artifacts for rendering code/UI side-by-side)Claude Desktop App (Supports local filesystem/tool execution)Claude Code (State-of-the-art terminal agent for codebase refactoring)No official extension; relies on CLI or third-party tools like Cline/Continue
    OpenAI (ChatGPT)ChatGPT (utilizes Canvas for inline text and code editing)ChatGPT Desktop App (Mac & Windows)OpenAI API CLI (Primarily for asset, file, and model management)No direct IDE tool; serves as the model layer for Cursor, Aider, etc.
    Google (Gemini)Gemini Web InterfaceAntigravity 2.0 Desktop ClientAntigravity CLI (Replaces old gemini-cli; orchestrates parallel agent tasks)Gemini Code Assist & Antigravity plugins (VS Code, Android Studio, IntelliJ)
    DeepSeekDeepSeek Chat (V3 & R1 reasoning models)Mobile Apps (Relies on web wrappers for desktop)None natively (Relies on community tools like Ollama or Aider)Native integration into open-source extensions like Continue and Cline
    PerplexityPerplexity.ai (Pro search & “Pages” workspace)Perplexity Desktop (Mac & Windows)None (Relies on API keys for third-party scripts)None (Primarily used as a data-grounding search API)

    The Economic Matrix: Quotas, Costs, and Loop Behavior

    This is the most critical matrix, IMHO. As someone who is experimenting with numerous paid frontier model services, each with a limited budge, I am not all in on any service yet, and want to maximize my low-paid token tiers.

    This table explains how the subscription models and APIs.

    Provider / ServiceDeveloper Plan & Cost (USD)IDE / CLI Billing MechanismQuota / Rate Limit MechanicsThe “Agentic Loop” Behavior (The Cost Trap)
    GitHub CopilotPro: $10/mo
    Max: $100/mo
    Flat-rate subscription for native tools.Hybrid Metered: Unlimited standard completions + a fixed pool of premium model credits.The Safest Route: Because it’s integrated, Microsoft absorbs a massive amount of the token cost, but heavy loop usage will hit a speed throttle unless on the Max tier.
    Anthropic (Claude)Pro: $20/mo
    Max 5x: $100/mo
    Billed via web account or separate Pay-As-You-Go API.5-Hour Rolling Window: Dynamic usage budget that resets every 5 hours based on server demand.High Risk for Pro: Running Claude Code on a large infrastructure repository will hit your 5-hour limit fast. Full-time agentic workflows require switching to Claude Max or the raw API.
    OpenAI (ChatGPT)API Account
    (Pay-As-You-Go)
    BYOK (Bring Your Own Key): Paid per token used in Cursor, Aider, etc.Volumetric Tiers: Limits depend on lifetime account funding (Tiers 1–5), scaling from 200k to millions of tokens/min.Predictable but Costly: Perfect for episodic troubleshooting, but an out-of-control recursive autonomous loop can easily rack up a $20–$50 bill in an afternoon.
    Google (Gemini)Code Assist Standard
    ~$19/mo
    Flat-rate per user or metered hourly via Google Cloud.Concurrently Throttled: Throttles users if background parallel multi-agent loops request too much compute at once.Great for Standard Dev: Highly stable for individual coding, but hard to hook into un-vetted, third-party autonomous command-line tools.
    DeepSeekPlatform API
    (Pay-As-You-Go)
    BYOK: Funded via a pre-paid developer wallet.Wallet Funded: Strictly bound by whatever dollar amount you load into your account.The Economy Champion: Because of their unified architecture and heavy 90%+ prompt caching discounts, running heavy agentic loops costs literal pennies compared to western models. $10 can last a month.
    PerplexityPerplexity Pro
    $20/mo
    Subscription includes a small monthly API credit.Fixed Credit Ceiling: API access stops or charges your card when your small monthly search token allowance runs dry.Not For Coding Loops: Great for pulling real-time documentation or API schemas into your agent, but financially unviable for writing or refactoring code blocks.

    Understanding the Layers of Agentic LLM Tool Execution

    To expand this post, it is useful to define several concepts that are now common in agentic AI systems: tools, skills, MCP servers, local execution layers, and IDE-integrated actions.

    Out of the box, an LLM is primarily a reasoning and text-generation system. You can ask it questions, request explanations, have it draft a script, or ask it to review configuration, and it will respond with generated text. By itself, however, the model does not inherently have access to your operating system, local files, terminal, IDE, logs, repositories, ticketing systems, monitoring platforms, or infrastructure.

    That distinction is important.

    A hosted model may appear to have additional capabilities, such as reading uploaded documents, searching the web, generating files, or analyzing code, but those capabilities are usually provided by the surrounding platform, client application, or agent runtime rather than by the model alone.

    This is where concepts such as tools, skills, MCP servers, and agent runtimes become important. These components extend an LLM from a passive chatbot into an interactive agentic system. The model can reason about what needs to happen, request the appropriate tool, inspect the result, and continue working through a task. The runtime around the model is responsible for exposing those tools, enforcing policy, executing approved actions, and returning results back to the model.

    When discussing LLMs that can inspect logs, modify files, run scripts, query systems, or interact with development environments, it is important to separate the layers involved. These capabilities are often described broadly as “agentic AI,” but agentic behavior is usually the result of several distinct components working together.

    The model provides reasoning and structured tool-call requests. The orchestration layer manages the workflow, state, prompts, and tool-calling loop. The local execution layer performs approved actions against the operating system, filesystem, terminal, repository, or other target environment. IDE-integrated actions expose workspace-specific capabilities such as file editing, diagnostics, Git state, and terminal access. Skills provide reusable task-specific instructions or workflows. MCP servers provide a standardized way to expose tools, resources, and prompts to compatible AI clients.

    In other words, the LLM is tool-aware, but the surrounding runtime is tool-authoritative.

    The model may decide that it needs to read a log file, edit a configuration file, run a script, or inspect a repository. However, the runtime determines which tools exist, which actions are allowed, where those actions execute, what policy controls apply, and what results are returned to the model.

    The table below provides a high-level comparison of the major concepts involved in LLM-driven local execution and tool integration.

    LLM Agent Execution Layers and Integration Concepts

    ConceptFormal NameWhat It MeansWhat It Controls or ProvidesExample in Practice
    Tool callingTool calling / function calling / structured tool invocationThe model’s ability to request an external action using a defined schema instead of free-form text.Provides the structured interface between the LLM and external capabilities.The model requests read_file(path="/var/log/app.log") or run_tests(command="pytest").
    ToolAgent tool / callable tool / local execution toolA specific function or action that can be invoked by the agent workflow.Provides a single capability, such as reading a file, tailing logs, running a script, or checking Git status.tail_log, write_file, run_shell, git_diff, restart_service.
    AgentLLM agent / tool-using agent / agentic workflowThe loop that allows the model to plan, call tools, observe results, and continue until the task is complete.Coordinates reasoning, tool requests, observations, and final responses.The agent reads an error log, edits a config file, restarts a service, checks the result, and reports back.
    Agent orchestration frameworkAgent framework / workflow orchestration frameworkA framework used to build and manage the agent loop, tools, prompts, state, and execution flow.Provides abstractions for models, prompts, tools, memory, state, routing, and workflow control.LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI.
    Local execution layerGoverned local execution layer / OS execution layer / tool execution runtimeThe runtime component that actually performs approved actions against the local operating system or workspace.Controls OS-level access, command execution, filesystem changes, script execution, and result capture.Running journalctl, editing a YAML file, executing a validation script, or applying a patch.
    IDE-local actionsIDE agent tools / workspace tools / editor-integrated actionsTool capabilities provided inside an IDE or local coding assistant environment.Gives the agent access to files, terminals, diagnostics, search, Git state, and editor context within a workspace.Cursor, Cline, Continue, Zed, VS Code Chat, Aider, OpenHands.
    SkillsAgent skills / repo-local skills / workflow skillsPackaged instructions, scripts, conventions, or workflows that teach an agent how to perform a specific task in a specific context.Provides repeatable task guidance and project-specific operating instructions.A repo skill that tells the agent how to validate changes using shellcheck, pytest, yamllint, and ./health-check.sh.
    MCPModel Context Protocol / MCP server / MCP tool / MCP resourceA standard protocol for exposing tools, resources, and prompts to compatible AI clients.Standardizes how external systems provide capabilities and context to AI applications.An MCP server exposes filesystem access, GitHub issues, database queries, lab inventory, or monitoring data.
    Policy and guardrailsExecution policy / tool governance / approval policyThe rules that determine what tools are available, what actions are allowed, and whether approval is required.Controls access boundaries, command restrictions, file path restrictions, approval gates, logging, and auditability.Allow reading logs but require approval before modifying /etc/ or restarting a service.
    Observability and audit trailAgent observability / execution logging / audit trailThe record of what the agent requested, what was approved, what was executed, and what changed.Provides troubleshooting, reviewability, compliance, and rollback support.Logging each tool call, command output, file diff, approval decision, and final result.
  • From Phrasing to Choreography: An Exceptionally Brief History of Agentic Coding

    From Phrasing to Choreography: An Exceptionally Brief History of Agentic Coding

    The future is now… which is now “then.”

    Do you remember where you were on November 30th, 2022. If you were in the US (or perhaps Liberia or parts of Mexico as I have recently come to learn), you may have been sitting around the house eating leftover turkey, or a nice ham sandwich (if you were fortunate to have both available on Thanksgiving).

    November 30th, 2022 was the launch date for ChatGPT.

    Although it was very rough around the edges and not ready for prime-time, many immediately saw into the future. I cannot say that I did immediately, but I was out there experimenting, watching it “lie” to me. Those “lies” were quickly labeled hallucinations and sounded much cooler. Anyway the race was on apparently. Not that I was fully paying attention at the time.

    Like most engineers across infrastructure, hardware, or software, I’ve spent the last four years just trying to keep my head above water. Figuring out what to learn and what to ignore has become almost as exhausting as the actual learning process. On one side, you have the performative hype from influencers desperate to prove how far ahead of the curve they are. On the other, there’s a deafening wall of noise from frontier model providers and newly christened startups, all frantically trying to dump a product before their razor-thin window of entrepreneurship slams shut.

    As a traditional infra-engineer, LLMs and agentic coding have been quick to deliver value. I have never identified as a software engineer, but rather a hardware and infra-centric practitioner. I’ve spent my work-life architecting, deploying, and supporting Unix/Linux, virtualization platforms, and enabling hardware functions such as SR-IOV for Telcos across North America.

    I’m the guy who knows the difference between raid 0+5 and 5+0.

    Suddenly, I am now a technologist who now has the tools to quickly deliver true infrastructure as code without having to become an evangelist for any specific automation platform. Suddenly, its much easier to deploy tools, supporting services, create dashboards, and troubleshoot and resolve system/hardware issues.

    Strangely I have always loved creating documentation, and have spent many years being frustrated with the lack of documentation that I have seen from others. To me, a measure of success is the fact that a solution that you built, is still relied upon once you are gone. Without proper documentation, and the ability for your successors to truly understand, operate, and enhance something that you built is dependent on great documentation. If you can not transfer the appropriate level of operational and design knowledge to those who replace you, they will replace your solution as well. With AI, everyone now has the power of documentation. “Review this repo and describe how it works, update the README.md”. It’s truly a wonderful thing.

    The Micro-Eras of Agentic Coding

    Having done my best to keep up via reading, experimenting, and building, I have learned quite a bit in a very short time. Those who share the same passions and beyond are often learning the same things at the same time. Separately, but also together, as they uniformly change the vernacular of agentic coding. First, we learn how to prompt from the perspective of a persona, defining outcomes. Next, we discover the need for guardrails and frameworks, especially when the word security rears its ugly head.

    Nothing below is unique or original, and I am not sure that there is actually a difference between 3 & 4, or possibly the Eras are in the wrong order, but here is my table for trying to keep straight concepts and terminologies that describe those contexts

    Era & FocusThe Primary Problem SolvedThe Technical Stumbling BlockThe AI Engineer’s Job
    1. Prompt Engineering
    (2022-2023)
    Expression: Shaping the tone, persona, and output formatting of the LLM.Hallucinations: Cannot conjure up data or facts it wasn’t trained on.Writing clear, contextual instructions and few-shot examples.
    2. Context Engineering
    (2023-2024)
    Information: Injecting real-time, proprietary data via RAG and Vector DBs.Passivity: The model can read your data, but cannot act on it or use tools.Managing token budgets, chunking data, and optimizing semantic retrieval.
    3. Harness Engineering
    (2024-2025)
    Environment: Giving the model tools, secure sandboxes, and safety guardrails.Rigidity: Linear chains break completely when the model makes a minor mistake.Building secure execution environments, MCP servers, and input/output parsers.
    4. Loop Engineering
    (2025-2026)
    Autonomy: Letting the system evaluate, self-correct, and iterate recursively.Compute Cost: High token consumption due to repetitive agent cycles.Designing state machines, testable termination conditions, and exit criteria.

    “It’s all vibecoding, right”

    The term “vibe-coding” was officially coined by Andrej Karpathy (former Director of AI at Tesla and co-founder of OpenAI) in a viral post on X (Twitter) on February 8, 2025.

    And it’s a term that I despise, despite fully understanding how fitting that term was at the time.

    Fast forward 1.5 years, and now “vibe-coding” sounds silly and very passe, as to me it implies sitting down with a chat window and an idea, and nothing more. This is what kids do in the basement to build a fun yet forgettable android game, right?

    And this is not where we are today. Today we are knee-deep in learning, and imagining what the next step is in this agentic evolution.

    Planning, context, guardrails, loops, governance, frameworks, and harnesses is where we are, at least mainly in theory, and sometimes in practice.

  • How to Reconfigure Claude Code CLI when changing subscriptions

    How to Reconfigure Claude Code CLI when changing subscriptions

    My employer recently put the kibosh on personal home lab usage.

    No longer can employees who have invested time and money into building out a lab to learn and test our own products utilize their home lab to connect to any internal tool or service. You can not even access your work gmail on anything other than your company provided workstation or your monitored phone.

    Which apparently means that I can no longer use my work provided Claude code account from a non-company supplied workstation, as its a company provided tool.

    This was my original workflow

    A flowchart illustrating a software development process with five steps: starting with a laptop labeled 'Claude/Cursor', followed by 'Test & Deploy in Home Lab', then 'Fix Bugs and Secure', next 'Commit', and finally 'Clone Repo to Work Env'.

    Now my choices are to either “Test & Deploy on Company Hardware” or utilize my own subscriptions and do my initial development on whatever workstation I happen to be in front of at the time. I chose the latter, as I want to establish my own agentic workflows that are not dependent on employer provided subscriptions, and I want to utilize my own hardware without having to deal with hardware resource scheduling.

    So that takes us to where we are today.

    A broken Claude Code CLI, that fails to authenticate as my initial corp provided config seems to be pervasive across multiple env vars and conf files. So lets clean that up.

    Screenshot of a terminal interface displaying welcome message for Claude Code v2.1.195, with tips for getting started and recent updates.

    First step is to remove any env vars you find in the files below

    grep -rE "CLAUDE_CODE_USE_VERTEX|ANTHROPIC_VERTEX_PROJECT_ID|CLOUD_ML_REGION|VERTEX|vertex" \
    ~/.bashrc \
    ~/.zshrc \
    ~/.bash_profile \
    ~/.profile \
    ~/.config/environment.d/* \
    /etc/environment \
    /etc/profile.d/* \
    ~/.claude/settings.json \
    ~/.claude/settings.local.json \
    2>/dev/null

    We must go deeper.

    find ~ -maxdepth 3 -name ".env" 2>/dev/null | xargs grep -l -iE "vertex|CLAUDE_CODE" 2>/dev/null

    Any vars found here, delete.

    Let’s make sure I cleaned up all shell vars. If you find any – unset them. Then source your .bashrc to make sure they are not set there on login.

    env | grep -iE "vertex|gcloud|google|claude|anthropic"
    PWD=/home/cpaquin/.claude

    Same with claude logging, we can check the latest debug log and look for errors around a broken config (use /debug to enable).

    cat ~/.claude/debug/latest

    Once you have removed any detritus from your config/env simply re-launch Claude and authenticate via your method of choice.

    Login screen for Claude Code with options for account types: Claude account with subscription, Anthropic Console account, and 3rd-party platform.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 3: RDMA Performance Testing

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 3: RDMA Performance Testing

    Before moving on to Part 3 of this project, lets review what we have accomplished thus far.

    In Part 1 and Part 2 we have…

    • Did a bit of planning and scoping
    • Built a 3-node GPU cluster (viper, columbia, prometheus)
    • Interconnected with InfiniBand Installed and validated ConnectX-4 NICs and RDMA stack (mlx5, ib_core, etc.)
    • Brought up the InfiniBand fabric using OpenSM (links active, LIDs assigned)
    • Verified topology and connectivity (ibstat, ibnetdiscover)
    • Configured IP over InfiniBand for basic networking between nodes Identified PCIe/NUMA limitations affecting optimal GPU↔NIC performance

    We are now ready to do some performance testing of our Infiniband network.


    Pre-Test Setup

    Before we can get started on our perf testing we have bit of work to do. We are going to install a few packages, and configure some tunables.

    Diagnostic Tools

    First lets make sure that we have a couple tools installed, so lets install some rpms.

    sudo dnf install infiniband-diags libibverbs-utils librdmacm-utils -y

    Kernel Modules

    InfiniBand and GPUDirect require specific modules to load at boot. So lets create hpc.conf in /etc/modules-load.d/. This creates (or overwrites) /etc/modules-load.d/hpc.conf. This file ensures each module loads automatically at boot via systemd-modules-load. Run this on each host.

    sudo tee /etc/modules-load.d/hpc.conf >/dev/null <<'EOF'
    ib_ipoib
    ib_umad
    ib_uverbs
    nvidia-peermem
    EOF

    Then force load the modules.

    sudo modprobe ib_ipoib ib_umad ib_uverbs nvidia-peermem

    Below is a short breakdown/description for each module.

    ModuleHow it’s used
    ib_ipoibProvides IP networking over InfiniBand (e.g., ib0) for SSH, NFS, TCP/IP
    ib_umadEnables userspace IB management tools (e.g., ibstat, fabric queries)
    ib_uverbsCore RDMA interface used by applications (MPI, NCCL, libibverbs)
    nvidia-peermemEnables GPUDirect RDMA for direct GPU ↔ NIC memory transfers (no CPU copy)

    Locked Memory Limits

    RDMA works by “pinning” memory so the OS cannot swap it to disk. So we need to create /etc/security/limits.d/99-hpc.conf as shown below.

    sudo tee /etc/security/limits.d/99-hpc.conf >/dev/null <<'EOF'
    * soft memlock unlimited
    * hard memlock unlimited
    EOF

    Performance & RDMA Benchmarking

    Health Check

    First lets run the following commands on any host under test, just to make sure the InfiniBand network is healthy before we start any testing. Run each line individually and make note of the output.

    hostname
    ibstat
    ibv_devinfo | egrep 'hca_id|transport|fw_ver|port:|link_layer|active_mtu|sm_lid|port_lid'

    You are specifically interesting in the following

    • Device Present (mlx5)
    • State: Active
    • Physical state: LinkUp
    • Link layer: InfiniBand

    Confirm HCA Name and Port Number

    Run on any device under test – we will need this for our test on our receiver and sender side.

    ibv_devices

    Output from columbia.lab.

     device          	   node GUID
     ------          	----------------
     mlx5_0          	248a070300ac5414
    
    

    Output from prometheus.lab

     device          	   node GUID
     ------          	----------------
     mlx5_0          	248a070300ac5610
    
    

    Run the RDMA Latency Test

    For our ib_send_lat (latency test) our device IP addresses are as follows.

    • columbia.lab – 172.16.50.12
    • prometheus.lab -172.16.50.11

    On our first device, columbia.lab, we run the following and leave it running.

    ib_send_lat -d mlx5_0 -i 1

    Now over on prometheus, run the command below. Insert the IP from columbia captured above. You will see a good bit of output in your terminal window.

    ib_send_lat -d mlx5_0 -i 1 <columbia_ip>

    Key configuration details

    So assuming the test did not fail, you are going to see some data spit out. Lets make sense of some of it.

    ParameterValueMeaning
    Devicemlx5_0ConnectX-4 (mlx5 driver)
    TransportIB (RC)Reliable Connection (standard RDMA mode)
    MTU4096Optimal for IB performance
    Queue Pairs1Single stream test
    Inline data236BSmall messages optimized
    Link typeInfiniBandCorrect mode

    What this test is actually doing

    ib_send_lat:

    • Registers memory with the NIC
    • Creates RDMA queue pairs
    • Sends messages using:
      • ibv_post_send()
    • Measures completion latency via completion queues (CQs)

    This is direct RDMA messaging, not IP networking.

    Our Overall results

    • Average latency: ~1.15 µs
    • Typical latency: ~1.14 µs
    • Minimum latency: 1.06 µs
    • Outliers: up to 11.41 µs
    • Conclusion: Healthy RDMA performance

    While InfiniBand ≠ RDMA test by default, our test ib_send_lat specifically uses RDMA verbs, so a successful result proves RDMA is working.

    In the output above, our average latency confirms that RDMA is functioning, as is kernel bypass. Note, that while we are using TCP/IP to setup the test, the actual data transfer is NIC to NIC and memory to memory. The queue pair exchange confirms RDMA session, as QPs were created on both nodes and transitioned through the required queue pair states shown below.

    StateNamePurposeAnalogy
    INITInitializeLocal QP setupPhone powered on
    RTRReady to ReceiveCan receive remote dataYou know the other person’s number
    RTSReady to SendFully operational (send + receive)Call connected and talking

    Run the RDMA Bandwidth Test

    For this test we will run ib_send_bw. This test measures the following.

    • Throughput (bandwidth) of RDMA send operations
    • NIC-to-NIC data transfer rate
    • Memory → NIC → fabric → NIC → memory

    Again this test uses IP to establish the initial connection between nodes, but make no mistake we are using RDMA verbs and are testing IB traffic (not IP traffic).

    So over on our first node (columbia.lab) we run the following.

    ib_send_bw -d mlx5_0 -i 1 -a

    Why these flags

    FlagPurpose
    -d mlx5_0Select your ConnectX-4 device
    -i 1Use IB port 1
    -aSweep all message sizes

    And on our second node we run the command shown below.

    ib_send_bw -d mlx5_0 -i 1 -a <columbia_ip>

    Assuming that this command does not fail, you will see a bunch of output that we need to interpret. This output is truncated, but I wanted to give you an idea of what to expect in the output.

    ib_send_bw -d mlx5_0 -i 1 -a 172.16.50.12
    ---------------------------------------------------------------------------------------
    Send BW Test
    Dual-port : OFF Device : mlx5_0
    Number of qps : 1 Transport type : IB
    Connection type : RC Using SRQ : OFF
    PCIe relax order: ON Lock-free : OFF
    WARNING: CPU is not PCIe relaxed ordering compliant.
    WARNING: You should disable PCIe RO with `--disable_pcie_relaxed` for both server and client.
    ibv_wr* API : ON Using DDP : OFF
    TX depth : 128
    CQ Moderation : 100
    CQE Poll Batch : 16
    Mtu : 4096[B]
    Link type : IB
    Max inline data : 0[B]
    rdma_cm QPs : OFF
    Data ex. method : Ethernet
    ---------------------------------------------------------------------------------------
    local address: LID 0x02 QPN 0x0107 PSN 0xed831e
    remote address: LID 0x01 QPN 0x0107 PSN 0xa2c511
    ---------------------------------------------------------------------------------------
    #bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
    Conflicting CPU frequency values detected: 1200.000000 != 1300.046000. CPU Frequency is not max.
    2 1000 7.79 7.40 3.879797

    Keep in mind that our IB bottleneck is our 40Gbe IB Switch. Here is what we can interpret from our test data.

    • Plateau was about: 3776.9 MiB/s which is about 31.7 Gbit/s
    • That is a normal practical result for a nominal 40 Gb InfiniBand-class link
    • Our plateau is consistent and stable, which is good

    Our InfiniBand link is healthy enough to sustain near-expected throughput, there are no obvious severe bottleneck or broken configuration. We are seeing some GPU frequency warnings, and some “PCIe relaxed ordering” warnings so lets fix those any try the test again.

    What is PCIe Relaxed Ordering? PCIe Relaxed Ordering is a performance feature where The CPU/NIC is allowed to reorder memory transactions. This can improve throughput by reducing stalls and increasing parallelism

    On both hosts, run the command below.

    cpupower frequency-set -g performance


    Now back on the first host, kick off the listen side of the test.

    ib_send_bw -d mlx5_0 -i 1 -a -q 4 --disable_pcie_relaxed

    And on the other server we kick off the test itself.

    ib_send_bw -d mlx5_0 -i 1 -a -q 4 --disable_pcie_relaxed <columbia_ip>

    Why these flags

    FlagPurpose
    -d mlx5_0Select your ConnectX-4 device
    -i 1Use IB port 1
    -aSweep all message sizes
    -q 4Use multiple queue pairs (better utilization)
    --disable_pcie_relaxedMatch your CPU capabilities and remove warning

    So lets summarize our output.

    • Almost identical throughput as initial test
    • Slight improvement in consistency
    • Cleaner test conditions (set cpu-frequency to performance)
    • Multiple QPs established (we see 4 QPNs)
    • We still see CPU Frequency is not max, however this is non issue as we have already saturated our links.


    Wrap Up

    In our previous post, we stood up our IB network, and performed some basic fabric tests. Today was all about performance testing and testing with the actual RDMA verb stack. We found that our fabric was pretty much performing as expected out of the box with minimal tuning, as we are hitting near-theoretical limits for our 40Gb hardware.

    We have a stable, low latency, high bandwidth IB Fabric.

    I was hoping to get to GPU direct testing today, however that looks like it might a bit of a beast and I think I will call it a day and do a bit more research on the topic.

  • The Homelab Dilemma: Living With Enterprise Servers

    The Homelab Dilemma: Living With Enterprise Servers

    Enterprise Servers are loud and hot, and while a couple of 13th generation Dell servers kept a room nice and toasty in the cold of winter, it a double whammy to your electric bill in the Spring/Summer.

    The noise can be problematic as well, especially since I no longer have a basement, where I relied on my servers to circulate and dehumidify the stuffy air. Now, I actually have to sit in a room with the rack. Which is why I moved it to the bedroom. Strange.

    So why the bedroom, well the white noise (when under control) helps me sleep and I can turn down the heat in the Winter and not freeze at night. And in the day, when I power everything up, I am in my home office (or my second home office – the dinning room), which is already loud enough with my workstations and old Cisco switches. That tiny home office, that is technically a small bedroom for your least favorite child, can get downright uncomfortably warm.


    Why So Much Noise

    As anyone who has worked in a datacenter can tell you. Servers are loud. Let compare fans in an Dell R630 to an R730.

    ServerForm FactorEffective fan sizeFan CountNotes
    Dell R6301U~40 mm blower7 Very high RPM, loud, high static pressure
    Dell R7302U~60 mm blower6Still loud, but more efficient airflow

    This boils down to the 1U server must spin its fans faster that then 2U to move the same amount of air through the server chassis, measured in CFMs (Cubic Feet per Minute).

    Additionally, you install a PCIe card in a server that the server itself does not recognize (like Telsa T4 GPUs) , and you may find your fans spinning at full bore, as the server would rather take flight than overheat.


    Enter the Dell Fan Susher

    Luckily, you can use ipmitools to override a Dell server’s fan speed, and quiet things down a great deal. However, you need to keep your eye on temperatures. For this I created Dell-Server-Fan-Shusher.

    Its a python script that monitors system temps (via sensors), and NVIDIA GPU temps (via nvidia-smi – when present), and sets fan speeds accordingly. It currently has seven threshold levels for temp and fan speeds, all of which can easily be modified.

    Installation is one command and its scheduled either via cron or systemd.

    sudo ./install.sh

    The deployed fan_control.py reads system temps in the following order.

    • Sysfs (/sys/class/hwmon)
    • sensors
    • ipmitool sdr list (last fallback)

    It also gets GPU temps via nvidia-smi and system temps via IPMI (e.g. Inlet, Exhaust, CPU packages, etc.).


    Its Getting Hot in Here

    The susher has been working fine for months, but as I mentioned its no longer Winter and the ambient air in my “server room” has been rising. See output from my Netbotz 450 below. Starting to get hotter than a grandparent’s Florida condo.

    Line graph showing temperature readings over time from a sensor pod. The y-axis represents temperature in Fahrenheit, ranging from 71 to 77 degrees. The x-axis shows time intervals. The maximum temperature recorded is 75.4 degrees, and the minimum is 72.3 degrees.

    So why I am posting all this? So this morning, when I attempted to log into my main hypervisor to start on my 3rd installment of my HPC GPU Cluster journey, I found the host unresponsive and throwing the errors below on the console. Time to reboot.

    Terminal screen showing error messages for a network adapter in Red Hat Enterprise Linux, indicating it has stopped due to overheating and suggesting to restart the computer or replace the adapter.

    Overheating NIC?

    So here is what happened, the ixgbe driver detected that the adapter overheated and disabled it to protect the hardware. When that happens, all ports on that card stop working. In the console, kernel logs are reporting a thermal shutdown of the Intel 10GbE network adapter. Specifically the Intel X540-AT2, which is a dual port 10Gbe copper (RJ45) adapter – with a large heatsink (some variants of this card have a fan.

    lspci -s 0000:82:00.0
    82:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

    A few minutes after a reboot the error returns, and the NIC goes offline again and is accessible only via the IDRAC console. At this point I cannot keep the system online long enough to do any meaningful troubleshooting. So we need to shush the susher. So from the console we disable it and then reboot again.

    # systemctl disable --now dell-r730-fan-control.timer
    Removed '/etc/systemd/system/timers.target.wants/dell-r730-fan-control.timer'.


    From our workstation we connect to the servers IDRAC and set manual fan control and set fan speeds to 100%. We do this while the target system is in the process or rebooting.

    ipmitool -I lanplus -H 10.1.10.20 -U root -P calvin raw 0x30 0x30 0x01 0x00
    ipmitool -I lanplus -H 10.1.10.20 -U root -P calvin raw 0x30 0x30 0x02 0xff 100

    Hopefully now we can keep the system online long enough to see what is really going on, as the server itself is not reporting any temperature issues on the idrac.


    Troubleshooting

    So first lets check the system logs and see how many times that this has occurred.

    grep -r "over heat\|overheat\|stopped because" /var/log/ 2>/dev/null | tail -50

    We see two events today, and a 3rd event a few days ago (probably patient zero).

    DateTimeEvent
    Mar 22, 202612:04:56Both Intel 10GbE NICs (enp130s0f0, enp130s0f1) stopped due to overheat
    Mar 22, 202613:17:24Same overheat again shortly before reboot at 13:24:59
    Mar 15, 202611:39:31Same overheat on both NICs

    Digging into logs a bit further, we see the following at around 12:04

    Fan speed: 15% (~3800 RPM)

    • GPU temp: 35°C
    • System temps: max 47°C (sensors 24–25 and 39 at 41–47°C)
    • Action: Fan control had just set fans to 15% for the “LOW” threshold.

    Intel X540-AT2 thermal specs

    A quick internet search yields the following. This NIC needs to get pretty hot to experience a thermal shutdown

    SpecificationValueNotes
    Operating temp (ambient)0°C to 55°CMarketing spec for 200 LFM airflow
    Extended ambient0°C to 70°CWith 300 LFM airflow and adequate heatsink
    Tcase max107°CMax case temp at heat spreader (die-level limit)

    NIC temperature visibility

    So the Intel ixgbe driver does not expose temperature to Linux:

    • No hwmon temperature sensor under /sys/...
    • ethtool does not show NIC temperature

    So there is no NIC temperature in system logs. The “over heated” message comes from the NIC’s internal thermal protection. The driver only reports the event; the actual NIC temperature is not logged. At this point we do no have a method to pull the temp from the NIC.

    Additional, there are no log messages like thermal warnings, throttling, or high-temperature notices before the shutdown. So there is really nothing that we can check for in the logs and use as a trigger for our Dell Fan Controller (Shusher).

    EventLast ixgbe message before overheatOverheat
    Mar 22, 12:04:5611:58:02 – NIC Link is Up 10 Gbps12:04:56 – overheat (about 7 minutes later)
    Mar 22, 13:17:2413:11:28 – NIC Link is Up 10 Gbps (after reboot)13:17:24 – overheat (about 6 minutes later)
    Mar 15, 11:39:31No ixgbe messages in preserved logs before this time11:39:31 – overheat

    Additionally, none of our reported temps from lm-sensors show any temperature issues, and we can see that the fan-susher had recently adjusted fans to 15% due to somewhat amenable temps.


    LM-Sensors

    Currently sensors detects the following temperatures on this R730.

    ChipSensorCurrentLimits
    coretemp-isa-0000 (CPU Package 0)Package id 028°Chigh 83°C, crit 93°C
    Core 0–28 (16 cores)22–25°Chigh 83°C, crit 93°C
    coretemp-isa-0001 (CPU Package 1)Package id 129°Chigh 83°C, crit 93°C
    Core 0–28 (16 cores)22–25°Chigh 83°C, crit 93°C

    Lets run sensors-detect and see if we can add any additional sensors that might help us get a better picture of temps across the system

    # sensors-detect

    We answer “Yes” at each prompt. Any new modules/sensors are added to /etc/sysconfig/lm_sensors

    Now we reload all the sensors

     . /etc/sysconfig/lm_sensors; for m in $HWMON_MODULES $BUS_MODULES; do [ -n "$m" ] && sudo modprobe -r $m 2>/dev/null; done; for m in $BUS_MODULES $HWMON_MODULES; do [ -n "$m" ] && sudo modprobe $m; done
    
    

    However, no new modules/temps are output by the sensors command, so no additional system temps to feed to the shusher.


    So What Now?

    At this point we have a few options, besides just cranking down the AC.

    We know that our overheat events occurred when system temps were ~47°C with fans at 15%. So, lets start with an overhaul of the susher and increase fan speeds by 10% for each of the 7 temp thresholds as configured in our .env file, and redeploy.

    Updated Fan speed levels (7 tiers)

    LevelTemp thresholdFan speedTrigger
    Very-Low< 20°C20%Below LOW
    Low≥ 20°C25%GPU LOW / System LOW
    Medium-Low≥ 35°C35%MED_LOW
    Medium≥ 50°C45%MED
    Medium-High≥ 60°C60%MED_HIGH
    High≥ 60°C (system) / 70°C (GPU)75%HIGH
    Very-High≥ 75°C90%Auto mode → iDRAC

    Our minimum fan speed is now 20% instead of 10%, which should improve airflow over the NICs and reduce overheat risk.

    Inlet/Exhaust Differential Logging

    For the NIC overheat events (Mar 22 12:04, 13:17 and Mar 15 11:39), inlet/exhaust were not logged, so let’s start logging them on each scheduled run of fan_control.py , so if/when something goes wrong, we can see how hot the air was going in and how much it warmed up inside the chassis.

    Safety Floor

    Additionally lets setup login in fan_control.py if inlet or exhaust gets too high:

    • Inlet ≥ 40°C → minimum fan speed set to 35%
    • Exhaust ≥ 50°C → minimum fan speed set to 35%

    Even if GPU/CPU temps look fine, we still ramp fans to protect things like the NIC when chassis air is hot.


    Next Steps

    For now we are going to let things ride. We have adjusted our fan speeds up, and we setup our safety floor.

    If we continue to see issues, we may need to take additional action. This r730 is running a number of Virtual machines which I use as “lab infrastructure”, so I want it running 24/7. And while I could just keep jacking up fan speeds, I would rather take a more proactive approach.

    1. Remove heatsink and apply fresh thermal paste. This card is long in the tooth. Could be dried up.
    2. Move to an earlier revision of the Intel X540-AT2 which came with active fans. (Cheap)
    3. Move to a card who’s driver can expose temperatures, like the Broadcom NetXtreme-E (e.g. BCM57416 – not as cheap)

    In theory, I like the 3rd option, as it would be nice to be able to pull temperatures from the the NIC, and allow fan_control.py to actually adjust fan speeds intelligently. In practice, however, replacing the NIC is probably overkill unless I run into a dual port actively cooled Intel with both long and short brackets (I like to keep my options open slot-wise). Although I do like getting packages in the mail.

  • Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 2: Infiniband Setup

    Project “NVIDIA HPC Infiniband Homelab GPU Cluster”: Part 2: Infiniband Setup

    Black silhouette of a cat with an arched back.

    Part 1: Of this Project Log can be found here

    Now that the 3x Mellanox MCX455A-ECAT ConnectX-4 Adapters have arrived, its time to install them into their respective servers (columbia.lab, prometheus.lab, and viper.lab)


    Verify Mellanox CX4s are Detected

    Once installed, log into the IDRAC of each host and verify that the CX-4 appears in system inventory. Sample output below from one of the hosts.

    Note that you may need to boot the system for the CX-4 to appear in the IDRAC inventory (as Collect System Inventory on Restart” (CSIOR) will run when starting up)

    InfiniBand.Slot.1-1 - PCI Device
    BusNumber 129
    DataBusWidth 16x or x16
    Description ConnectX-4 VPI IB EDR/100 GbE Single Port QSFP28 Adapter
    Device Type PCIDevice
    DeviceDescription InfiniBand.Slot.1-1
    DeviceNumber 0
    FQDD InfiniBand.Slot.1-1
    FunctionNumber 0
    InstanceID InfiniBand.Slot.1-1
    LastSystemInventoryTime 2026-03-14T22:33:28
    LastUpdateTime 2026-03-15T03:33:07
    Manufacturer Mellanox Technologies
    PCIDeviceID 1013
    PCISubDeviceID 0033
    PCISubVendorID 15B3
    PCIVendorID 15B3
    SlotLength Long Length
    SlotType PCI Express Gen 3

    Once each machine has booted to the running OS, you can confirm that the RHEL properly detects the CX-4 with lspci

    lspci | grep -i mel
    44:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

    Verify Numa Topology Via nvidia-smi

    Ideally, for best performance, your GPUS and InfiniBand adapters will be NUMA local to each other. If you were deploying a similar setup in a production environment, NUMA alignment would be critical.

    Our lab setup is less than ideal due to the limited number of PCI slots. In many Dell servers, PCIe risers for GPUs have only one PCIe slot. Stick two of these risers in a single server, and you end up with only 3 slots free on riser 1 (half lenght) which is where we had to install our CX-4s.

    All this being said, we “should” be fine for functional testing. Lets review each of our 3 nodes below. Since NVIDIA Drivers are already installed on all three systems, we can run nvidia-smi and confirm that the CX-4 is in the output and review the topology


    Nvidia-smi output on host viper.lab

    This server viper.lab is a Dell R720 running RHEL 9, and has 2x Nvidia Telsa P4 GPUs installed along with the CX-4.

    nvidia-smi topo -m
    GPU0 GPU1 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
    GPU0 X PHB SYS 0,2,4,6,8,10 0 N/A
    GPU1 PHB X SYS 0,2,4,6,8,10 0 N/A
    NIC0 SYS SYS X

    In the output above. Both GPUs are local to NUMA node 0 and connected to each other through a PCIe host bridge (PHB), while the ConnectX NIC (mlx5_0) is topologically remote from both GPUs (SYS), making the setup workable but not ideal for GPUDirect RDMA performance. This should be ok for our lab as we are performing functional tests, and performance is secondary. Time will tell.

    Nvidia SMI output on host columbia.lab

    Columbia.lab is a Dell R730, with 1x Nvidia Telsa T4 installed along with one CX-4 (possible to move to another slot, if we had the full length bracket for the CX-4, or had half-length brackets for our NICs on Riser 2.

    nvidia-smi topo -m
    GPU0 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
    GPU0 X SYS 0,2,4,6,8,10 0 N/A
    NIC0 SYS X

    In the output above, we can see that this system has one GPU (T4) on NUMA node 0 and and CX-4, but the GPU-to-NIC path is SYS, indicating a topologically distant connection that is usable but sub-optimal for GPUDirect RDMA performance. Again, may be fine for functional testing.

    Nvidia SMI output on host prometheus.lab

    This Dell R730 has 2x NVIDIA Tesla T4s installed, as well as the recently installed CX-4

    nvidia-smi topo -m
    GPU0 GPU1 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
    GPU0 X PHB SYS 0,2,4,6,8,10 0 N/A
    GPU1 PHB X SYS 0,2,4,6,8,10 0 N/A
    NIC0 SYS SYS X

    This output shows two GPUs on the same NUMA node with a moderate GPU-to-GPU path (PHB) and relatively distant NIC connectivity (SYS), which is acceptable for many workloads but not ideal for high-performance GPU-to-NIC or GPUDirect-style traffic.


    Verify Drivers Loaded Properly

    In the first step we saw the CX-4 in the IDRAC, and in the output of lspci. We will now check that the driver has loaded properly. Rinse and repeat on each host.

    lsmod | egrep 'mlx5_core|mlx5_ib|ib_core'
    mlx5_ib 561152 0
    macsec 73728 1 mlx5_ib
    mlx5_core 3153920 2 mlx5_fwctl,mlx5_ib
    mlxfw 49152 1 mlx5_core
    psample 20480 1 mlx5_core
    tls 159744 2 bonding,mlx5_core
    pci_hyperv_intf 12288 1 mlx5_core
    ib_uverbs 217088 2 rdma_ucm,mlx5_ib
    ib_core 573440 12 rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm

    Lets review the imporant/relevant output below

    mlx5_ib 561152 0
    mlx5_core 3153920 2 mlx5_fwctl,mlx5_ib
    ib_uverbs 217088 2 rdma_ucm,mlx5_ib
    ib_core 573440 12 ...
    • mlx5_core – main low-level kernel driver for Mellanox/NVIDIA ConnectX-4/5-class adapters.
      • the kernel sees the adapter family and has loaded the base driver
      • this is required for the card to function at all
      • other Mellanox modules are depending on it
    • mlx5_ib – This is the InfiniBand/RDMA driver layer for mlx5 devices.
      • the adapter is not just using the generic Ethernet driver path
      • the system has the RDMA / InfiniBand-capable driver loaded
      • the kernel is prepared to expose the card as an IB/RDMA device
    • ib_core – the core InfiniBand subsystem in the kernel.
      • the Linux IB stack is loaded
      • multiple RDMA/IB-related modules are attached to it
      • the host is set up for InfiniBand/RDMA functionality, not just plain NIC support
    • ib_uverbs – This is the userspace verbs interface.
      • userspace RDMA tools and libraries should be able to talk to the device
      • commands like ibv_devinfo, ibstat, and RDMA applications have the proper kernel interface available

    Show Devices and Port State

    First we need to install some prerequesits

    sudo dnf install rdma-core infiniband-diags libibverbs-utils -y

    ibv_devices

    We can show InfiniBand devices with ibv_devices, which shows local RDMA/InfiniBand devices that the OS can see on that host. It does not enumerate remote hosts, switches, or the rest of the IB fabric.

    We will run this command on each host and capture the output.

    On columbia.lab

    [root@columbia ~]# ibv_devices
    device node GUID
    ------ ----------------
    mlx5_0 248a070300ac5414

    On viper.lab

    root@viper:~# ibv_devices
    device node GUID
    ------ ----------------
    mlx5_0 248a070300ac5f6c

    On prometheus.lab

    [root@prometheus ~]$ ibv_devices
    device node GUID
    ------ ----------------
    mlx5_0 248a070300ac5610
    [root@prometheus ~]$

    ibstat

    Now that we have confirmed all devices are present and accounted for lets check for links. In the output below you can see that we have link “Physical state: LinkUp“, but since we have not configured subnet manager on any of our nodes, the logical fabric is “State: Initializing“.

    ibstat
    CA 'mlx5_0'
    CA type: MT4115
    Number of ports: 1
    Firmware version: 12.28.4512
    Hardware version: 0
    Node GUID: 0x248a070300ac5610
    System image GUID: 0x248a070300ac5610
    Port 1:
    State: Initializing
    Physical state: LinkUp
    Rate: 40
    Base lid: 65535
    LMC: 0
    SM lid: 0
    Capability mask: 0x2659e848
    Port GUID: 0x248a070300ac5610
    Link layer: InfiniBand

    Run ibstat on any of your remaining nodes. Ensure that you see “Physical state: LinkUp”. You may also want to make notes of “Firmware version: 12.28.4512“. We have the same firmware on all three CX-4s.


    ibv_definfo

    We can also run ibv_devinfo, which gives a detailed view of the local RDMA / InfiniBand device and its ports. It is more detailed than ibv_devices and overlaps somewhat with ibstat, but from the verbs / RDMA stack perspective.

    Example output below:

     ibv_devinfo
    hca_id:	mlx5_0
    	transport:			InfiniBand (0)
    	fw_ver:				12.28.4512
    	node_guid:			248a:0703:00ac:5414
    	sys_image_guid:			248a:0703:00ac:5414
    	vendor_id:			0x02c9
    	vendor_part_id:			4115
    	hw_ver:				0x0
    	board_id:			DEL2180110032
    	phys_port_cnt:			1
    		port:	1
    			state:			PORT_INIT (2)
    			max_mtu:		4096 (5)
    			active_mtu:		4096 (5)
    			sm_lid:			0
    			port_lid:		65535
    			port_lmc:		0x00
    			link_layer:		InfiniBand
    
    

    This output shows us the following…

    • Local RDMA devices
      • Example: mlx5_0, mlx5_1
    • Port state
      • Example: PORT_ACTIVE, PORT_DOWN, PORT_INIT
    • Physical link state
      • Example: LINK_UP, POLLING, DISABLED
    • Negotiated link details
      • Speed and link width
    • Fabric info
      • Local LID and SM LID
    • Device identifiers
      • Node GUID, port GUID, system image GUID
    • Transport / firmware details
      • Transport type and device-specific details
    • RDMA capabilities
      • Limits such as QPs, CQs, MR size, atomic support, GID table size

    rdma link

    The current output of “rdma link” shows use that our InfiniBand ports are connected but as we know the fabric is not initialized.

    rdma link
    link mlx5_0/1 subnet_prefix fe80:0000:0000:0000 lid 65535 sm_lid 0 lmc 0 state INIT physical_state LINK_UP

    Specifically the output shows us the following.

    • mlx5_0/1
      • Device mlx5_0, port 1
    • subnet_prefix fe80:0000:0000:0000
      • Normal default InfiniBand subnet prefix
    • lid 65535
      • The port does not have a valid assigned LID yet
    • sm_lid 0
      • No subnet manager is detected
    • lmc 0
      • LID mask control is 0; not important here
    • state INIT
      • The port is not fully active yet
    • physical_state LINK_UP
      • The physical link is up and the cable/port side is working

    Setup Subnet Manager on one Host

    For our lab, we are going to only setup subnet manager on one host. Pick your always-on host. Multiple instances of subnet manager can be used, but again, not needed for our current objective.

    On the selected host run the following to install required packages.

     dnf install -y rdma-core opensm infiniband-diags

    Next, start and enable the service

    sudo systemctl enable --now opensm

    Then check to ensure that the service started without error.

    sudo systemctl status opensm --no-pager
    journalctl -u opensm -b --no-pager

    Now we can re-check the fabric on each host. Now we see the Fabric status is “State: Active”

     ibstat
    CA 'mlx5_0'
    	CA type: MT4115
    	Number of ports: 1
    	Firmware version: 12.28.4512
    	Hardware version: 0
    	Node GUID: 0x248a070300ac5f6c
    	System image GUID: 0x248a070300ac5f6c
    	Port 1:
    		State: Active
    		Physical state: LinkUp
    		Rate: 40
    		Base lid: 4
    		LMC: 0
    		SM lid: 1
    		Capability mask: 0x2659e848
    		Port GUID: 0x248a070300ac5f6c
    		Link layer: InfiniBand
    root@viper:~# 
    
    

    rdma link shows similar output.

    rdma link
    link mlx5_0/1 subnet_prefix fe80:0000:0000:0000 lid 4 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP

    Confirm Infiniband Fabric Topology

    You can run the following commands to confirm that your fabric is up and running

    Run the “ibnetdiscover” to see host adapters, links, GUIDs, port relationships, and switches. Example output below

    ibnetdiscover
    #
    # Topology file: generated on Sat Mar 14 21:15:43 2026
    #
    # Initiated from node 248a070300ac5f6c port 248a070300ac5f6c
    vendid=0x2c9
    devid=0xbd36
    sysimgguid=0x2c902004cf11b
    switchguid=0x2c902004cf118(2c902004cf118)
    Switch 8 "S-0002c902004cf118" # "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0
    [1] "H-248a070300ac5f6c"[1](248a070300ac5f6c) # "viper mlx5_0" lid 4 4xQDR
    [2] "H-248a070300ac5414"[1](248a070300ac5414) # "columbia mlx5_0" lid 1 4xQDR
    [3] "H-248a070300ac5610"[1](248a070300ac5610) # "prometheus mlx5_0" lid 2 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5414
    caguid=0x248a070300ac5414
    Ca 1 "H-248a070300ac5414" # "columbia mlx5_0"
    [1](248a070300ac5414) "S-0002c902004cf118"[2] # lid 1 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5610
    caguid=0x248a070300ac5610
    Ca 1 "H-248a070300ac5610" # "prometheus mlx5_0"
    [1](248a070300ac5610) "S-0002c902004cf118"[3] # lid 2 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR
    vendid=0x2c9
    devid=0x1013
    sysimgguid=0x248a070300ac5f6c
    caguid=0x248a070300ac5f6c
    Ca 1 "H-248a070300ac5f6c" # "viper mlx5_0"
    [1](248a070300ac5f6c) "S-0002c902004cf118"[1] # lid 4 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 3 4xQDR

    In the output above we can see the following

    • One Mellanox InfiniScale-IV switch is present in the fabric
      • Switch GUID: 0x2c902004cf118
      • Switch LID: 3
      • Model family shown as Infiniscale-IV Mellanox Technologies
      • It is an 8-port switch
    • Three hosts are connected to the switch
      • columbia on switch port 2, LID 1
      • prometheus on switch port 3, LID 2
      • viper on switch port 1, LID 4
    • All three hosts are being seen as CA / HCA nodes
      • columbia mlx5_0
      • prometheus mlx5_0
      • viper mlx5_0
    • All discovered links are running at:
      • 4xQDR
      • That means a 4-lane QDR InfiniBand link, which aligns with a 40 Gb/s class IB link

    This output confirms that OpenSM is working properly, that the switch is visible, and all three nodes are connected to the fabric.

    Run “ibnodes” which provides a similar output to ibdiscover, albeit a bit less verbose.

    ibnodes
    Ca : 0x248a070300ac5610 ports 1 "prometheus mlx5_0"
    Ca : 0x248a070300ac5414 ports 1 "columbia mlx5_0"
    Ca : 0x248a070300ac5f6c ports 1 "viper mlx5_0"
    Switch : 0x0002c902004cf118 ports 8 "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0

    Run “ibswitches” to see switches only.

     ibswitches
    Switch	: 0x0002c902004cf118 ports 8 "Infiniscale-IV Mellanox Technologies" base port 0 lid 3 lmc 0
    
    

    iblinkinfo will show you InfiniBand topology info

     iblinkinfo
    CA: viper mlx5_0:
          0x248a070300ac5f6c      4    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    1[  ] "Infiniscale-IV Mellanox Technologies" ( )
    CA: columbia mlx5_0:
          0x248a070300ac5414      1    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    2[  ] "Infiniscale-IV Mellanox Technologies" ( )
    Switch: 0x0002c902004cf118 Infiniscale-IV Mellanox Technologies:
               3    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       4    1[  ] "viper mlx5_0" ( )
               3    2[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       1    1[  ] "columbia mlx5_0" ( )
               3    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    1[  ] "prometheus mlx5_0" ( )
               3    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
               3    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
    CA: prometheus mlx5_0:
          0x248a070300ac5610      2    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    3[  ] "Infiniscale-IV Mellanox Technologies" ( )
    
    

    In the output above, we see …

    • 4x 10Gbps per lane
    • InfiniBand speed class is QDR
    • aggregate raw signaling rate is about 40 Gb/s
    • Down/Polling – unused/not-connected switch ports

    And finally, “sminfo” will show you info on subnet manager.

    Specifically (below) we see that subnet manager is reachable on LID1, with a GUID of 0x248a070300ac5414 (which belongs to columbia mlx5_0). We also see “activity count 446” which shows that subnet manager has processed fabric-management activity 466 times since startup (basically a liveness/activty counter).

    Additionally the output below show us the priority of the subnet manager instance (0 in this case), while state 3 SMINFO_MASTER shows us that this instance of subnet manager is in the master state and is the active controller in our IB fabric (assigning LIDS, managing paths/routing)

    sminfo
    sminfo: sm lid 1 sm guid 0x248a070300ac5414, activity count 446 priority 0 state 3 SMINFO_MASTER

    Configuring IP over InfiniBand

    IP over InfiniBand, or IPoIB, allows an InfiniBand fabric to carry normal IP traffic between hosts. That means systems connected by InfiniBand can use familiar network tools and services such as ping, ssh, scp, NFS, and other TCP/IP-based applications over the IB link instead of only using native RDMA-aware software.

    IPoIB is not required for RDMA itself, and it is also not inherently required for technologies like GPUDirect RDMA. RDMA and GPUDirect RDMA operate through the RDMA/verbs stack and the InfiniBand fabric, not through the IP emulation layer that IPoIB provides. NVIDIA’s current networking/operator docs describe RDMA and GPUDirect RDMA enablement separately from IPoIB, and they also document IPoIB as an optional deployment pattern rather than a prerequisite.

    We use IPoIB when we want the simplicity of standard IP networking on top of the higher-speed, low-latency InfiniBand fabric. In a small lab or cluster, this is useful for private host-to-host traffic, storage traffic, migration traffic, testing, or other east-west communication, while leaving the normal Ethernet interfaces in place for management access, internet access, and general connectivity.

    IPoIB Addresses for our lab

    Our lab uses 10.1.x.x for its existing IP scheme, so to avoid any confusion, we will use 172.16.x.x addresses for our small private subnet on the IB network. Note that we do not need a gateway.

    HOSTIPoIB AddressINTERFACE
    prometheus.lab172.16.50.11/24ibp129s0
    columbia.lab172.16.50.12/24ibp129s0
    viper.lab172.16.50.13/24ibp68s0

    As part of our initial temporary test, we will apply the IPoIB addresses to the indicated interfaces on each host (all as outlined above. Example temporary config will be for one host. However we will run the command (modified) for each host in our cluster.

    sudo ip link set ibp129s0 up
    sudo ip addr add 172.16.50.11/24 dev ibp129s0

    As you go host to host, verify that the address was assigned correctly.

    9: ibp129s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 1000
    link/infiniband 00:00:03:f2:fe:80:00:00:00:00:00:00:24:8a:07:03:00:ac:56:10 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 172.16.50.11/24 scope global ibp129s0
    valid_lft forever preferred_lft forever

    Also verify your routing table.

     netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.10.1       0.0.0.0         UG        0 0          0 bridge0
    10.1.10.0       0.0.0.0         255.255.255.0   U         0 0          0 bridge0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 idrac
    172.16.50.0     0.0.0.0         255.255.255.0   U         0 0          0 ibp129s0
    
    

    Now perform ping tests from each host and ensure that they can hit the remaining hosts in your cluster. For example.

    ping -I ibp129s0 -c 2 172.16.50.12
    ping -I ibp129s0 -c 2 172.16.50.13

    Once you have tested all three hosts, we can move forward with configuring persistent network configs.


    Persistent RHEL 10 / NetworkManager setup

    prometheus.lab

    sudo nmcli connection add type infiniband ifname ibp129s0 con-name ib-ibp129s0
    sudo nmcli connection modify ib-ibp129s0 ipv4.method manual ipv4.addresses 172.16.50.11/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp129s0
    
    
    
    
    

    columbia.lab

    sudo nmcli connection add type infiniband ifname ibp129s0 con-name ib-ibp129s0
    sudo nmcli connection modify ib-ibp129s0 ipv4.method manual ipv4.addresses 172.16.50.12/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp129s0

    viper.lab

    sudo nmcli connection add type infiniband ifname ibp68s0 con-name ib-ibp68s0
    sudo nmcli connection modify ib-ibp68s0 ipv4.method manual ipv4.addresses 172.16.50.13/24 ipv6.method disabled
    sudo nmcli connection up ib-ibp68s0

    Confirm Routing

    Use “ip route” to ensure that we have the proper route in place for our IPoIB network

    ip route
    default via 10.1.10.1 dev bridge0 proto static metric 425
    10.1.10.0/24 dev bridge0 proto kernel scope link src 10.1.10.25 metric 425
    169.254.0.0/16 dev idrac proto kernel scope link src 169.254.0.2 metric 100
    172.16.50.0/24 dev ibp68s0 proto kernel scope link src 172.16.50.13 metric 150

    Also confirm that NetworkManger sees our IB devices correctly (as Infiniband)

    nmcli device status
    DEVICE TYPE STATE CONNECTION
    bridge0 bridge connected bridge0
    idrac ethernet connected idrac
    ibp68s0 infiniband connected ib-ibp68s0
    bond0 bond connected bond0
    enp65s0f0 ethernet connected bond0-port0
    enp65s0f1 ethernet connected bond0-port1
    
    
    
    
    

    IP, IPoIB, and RDMA Usage Matrix

    We can use this simple decision matrix to ensure that we understand when to use traditional IP for host to host communication, vs when to use IPoIB, and when native RDMA/IB

    Use caseEthernetIPoIBNative RDMA / IB
    Host management, SSH, web UI, package installsBest choicePossible, but usually unnecessaryNo
    Internet access / default routeBest choiceNoNo
    General admin traffic between hostsBest choiceGood for isolated lab trafficNo
    Simple host-to-host testing with ping, ssh, scp, rsync over IB fabricNoBest choiceNo
    NFS/SMB using normal IP networking over the IB fabricNoBest choiceNo
    Fast private storage or migration traffic using standard TCP/IP appsNoBest choiceNo
    RDMA-aware apps using verbs/libibverbsNoNoBest choice
    MPI or cluster workloads built for native IB/RDMANoSometimes, if app specifically uses IPBest choice
    GPUDirect RDMA / high-performance GPU-to-network workflowsNoNoBest choice
    Lowest latency / highest efficiency IB data pathNoNoBest choice
    Easiest troubleshooting and least risk of routing mistakesBest choiceGood if kept isolatedMore specialized

    Next Steps.

    So in Part 2 of our project, we focused on getting InfiniBand up and running, as well as IPoIB. We validated connectivity and setup subnet manager and made sure that our fabric was initialized. We leared a number of IB related command and learned how to read their output. Good Stuff.

    In our next post we will start working with the various NVIDIA tools and projects, many of which will rely on our IB network. Additionally we may try to update firmware on our CX-4s and our IB Switch, however I may skip this step or circle back to it later.