Goal here was to create 3 LACP Port-Channels on a tp-link SX3008F 10gbe switch connecting to RHEL 9/10 hosts also configured to use LACP.

Specific Requirements

• Jumbo Frames (mtu 9216)
• LACP (802.3ad)
• RHEL side = “xmit_hash_policy=layer3+4”
• Switch side = src-dst-ip

End result should be 20Gbe connectivity between hosts for fast NFS backups of Virtual Machines (SSD and HDD NFS) shares). Additional tuning was performed on the NFS host for optimum throughput

Switch Side Config

The tplink is Cisco like, but not exactly so commands were a bit of a challenge to nail down specific syntax.

Jumbo Frames

On this switch either jumbo frames in on or off for the switch, there is no per port config

jumbo-size 9216 

Switch Port Config

1st interface. Both ports added to channel-group 1 which we will define in a later step

interface ten-gigabitEthernet 1/0/6
description "columbia bond0 member"
switchport general allowed vlan 10 untagged
switchport pvid 10
no switchport general allowed vlan 1
channel-group 1 mode active
exit

2nd interface

interface ten-gigabitEthernet 1/0/7
description "columbia bond0 member"
switchport general allowed vlan 10 untagged
switchport pvid 10
no switchport general allowed vlan 1
channel-group 1 mode active
exit

Define Port-Channel (channel-group)

interface port-channel 1
description "columbia bond0"
switchport general allowed vlan 10 untagged
switchport pvid 10
no switchport general allowed vlan 1
exit

Load Balance Mode

The port-channel load-balance src-dst-ip command configures a network switch to distribute traffic across aggregated links (EtherChannel) based on a mathematical hash of both the source and destination IP addresses.

This method ensures that traffic between the same two IP addresses consistently uses the same link while providing more even distribution compared to relying on MAC addresses alone.

We will match this config on the server side via “xmit_hash_policy=layer3+4”

port-channel load-balance src-dst-ip

Don’t forget to save your work

write memory

Health Check

# show etherchannel summary

Here we can see both ports 6 and 7 are in PO1

# show lacp neighbor

# show vlan id 10

Here we can see both ports 6 and 7 are in the correct vlan

# show interface status ten-gigabitEthernet 1/0/6

Here we can see that we have 10G link and both ports have correct description

# show lacp internal

Configuring the RHEL Side

Here we need to have the following

• 2 interfaces in a bond
• 1 bond
• 1 bridge
  • IP is on bridge
  • bond is connected to bridge

The Logical Order of Operations

1. The Bridge (bridge0): The “Top Level” virtual switch that holds the IP address.
2. The Bond (bond0): The logical aggregation of physical NICS. It is a “Port” of the bridge.
3. The Bond Ports (enp...): The physical wires. These are “Ports” of the bond.

Step 1: Create the Bridge (The Anchor)

You create the bridge first because the bond needs a “controller (aka bond)” to point to.

nmcli connection add type bridge con-name bridge0 ifname bridge0 \
ipv4.method manual ipv4.addresses 10.1.10.21/24 ipv4.gateway 10.1.10.1 \
ipv4.dns 10.1.10.74 ipv6.method disabled \
802-3-ethernet.mtu 9216

Step 2: Create the Bond (The Controller)

Note that the controller is the bridge we just made. We specify LACP (802.3ad) and the hashing policy here.

nmcli connection add type bond con-name bond0 ifname bond0 \
connection.controller bridge0 connection.port-type bridge \
bond.options "mode=802.3ad,xmit_hash_policy=layer3+4" \
802-3-ethernet.mtu 9216

Step 3: Attach the Physical Interfaces (The Slaves)

Now we tell the physical hardware to report to the bond. Crucial: The MTU must be set at this level so the hardware buffers are sized correctly for jumbo frames.

# First Port
nmcli connection add type ethernet con-name bond0-port1 ifname enp130s0f0 \
connection.controller bond0 connection.port-type bond \
802-3-ethernet.mtu 9216
# Second Port
nmcli connection add type ethernet con-name bond0-port2 ifname enp130s0f1 \
connection.controller bond0 connection.port-type bond \
802-3-ethernet.mtu 9216

Step 4: Verification Checklist

Once created, we bring the stack up from the top. NetworkManager will automatically trigger the underlying ports.

1. Bring it up: nmcli connection up bridge0
2. Verify MTU Consistency: Every device in the chain must match.
   • ip link show | grep 9216
3. Verify LACP Sync: The switch must see the server.
   • cat /proc/net/bonding/bond0 (Look for “Partner Mac Address”)
4. Verify Bridge Membership:
   • bridge link show (The bond should be listed as a member of the bridge).

Why this order matters

• If you define the physical ports first without a controller, they might try to get a DHCP address on their own.
• By setting MTU 9216 at every single stage of the nmcli command, you prevent the kernel from defaulting any segment to 1500, which causes the exact “packet loss” issue you experienced during the jumbo ping tests.