Goal here was to create 3 LACP Port-Channels on a tp-link SX3008F 10gbe switch connecting to RHEL 9/10 hosts also configured to use LACP.
Specific Requirements
- Jumbo Frames (mtu 9216)
- LACP (802.3ad)
- RHEL side = “xmit_hash_policy=layer3+4”
- Switch side = src-dst-ip
End result should be 20Gbe connectivity between hosts for fast NFS backups of Virtual Machines (SSD and HDD NFS) shares). Additional tuning was performed on the NFS host for optimum throughput
Switch Side Config
The tplink is Cisco like, but not exactly so commands were a bit of a challenge to nail down specific syntax.
Jumbo Frames
On this switch either jumbo frames in on or off for the switch, there is no per port config
jumbo-size 9216
Switch Port Config
1st interface. Both ports added to channel-group 1 which we will define in a later step
interface ten-gigabitEthernet 1/0/6description "columbia bond0 member"switchport general allowed vlan 10 untaggedswitchport pvid 10no switchport general allowed vlan 1channel-group 1 mode activeexit
2nd interface
interface ten-gigabitEthernet 1/0/7description "columbia bond0 member"switchport general allowed vlan 10 untaggedswitchport pvid 10no switchport general allowed vlan 1channel-group 1 mode activeexit
Define Port-Channel (channel-group)
interface port-channel 1description "columbia bond0"switchport general allowed vlan 10 untaggedswitchport pvid 10no switchport general allowed vlan 1exit
Load Balance Mode
The port-channel load-balance src-dst-ip command configures a network switch to distribute traffic across aggregated links (EtherChannel) based on a mathematical hash of both the source and destination IP addresses.
This method ensures that traffic between the same two IP addresses consistently uses the same link while providing more even distribution compared to relying on MAC addresses alone.
We will match this config on the server side via “xmit_hash_policy=layer3+4”
port-channel load-balance src-dst-ip
Don’t forget to save your work
write memoryHealth Check
# show etherchannel summary
Here we can see both ports 6 and 7 are in PO1
# show lacp neighbor
# show vlan id 10
Here we can see both ports 6 and 7 are in the correct vlan
# show interface status ten-gigabitEthernet 1/0/6
Here we can see that we have 10G link and both ports have correct description
# show lacp internal
Configuring the RHEL Side
Here we need to have the following
- 2 interfaces in a bond
- 1 bond
- 1 bridge
- IP is on bridge
- bond is connected to bridge
The Logical Order of Operations
- The Bridge (
bridge0): The “Top Level” virtual switch that holds the IP address. - The Bond (
bond0): The logical aggregation of physical NICS. It is a “Port” of the bridge. - The Bond Ports (
enp...): The physical wires. These are “Ports” of the bond.
Step 1: Create the Bridge (The Anchor)
You create the bridge first because the bond needs a “controller (aka bond)” to point to.
nmcli connection add type bridge con-name bridge0 ifname bridge0 \ipv4.method manual ipv4.addresses 10.1.10.21/24 ipv4.gateway 10.1.10.1 \ipv4.dns 10.1.10.74 ipv6.method disabled \802-3-ethernet.mtu 9216
Step 2: Create the Bond (The Controller)
Note that the controller is the bridge we just made. We specify LACP (802.3ad) and the hashing policy here.
nmcli connection add type bond con-name bond0 ifname bond0 \connection.controller bridge0 connection.port-type bridge \bond.options "mode=802.3ad,xmit_hash_policy=layer3+4" \802-3-ethernet.mtu 9216
Step 3: Attach the Physical Interfaces (The Slaves)
Now we tell the physical hardware to report to the bond. Crucial: The MTU must be set at this level so the hardware buffers are sized correctly for jumbo frames.
# First Portnmcli connection add type ethernet con-name bond0-port1 ifname enp130s0f0 \connection.controller bond0 connection.port-type bond \802-3-ethernet.mtu 9216# Second Portnmcli connection add type ethernet con-name bond0-port2 ifname enp130s0f1 \connection.controller bond0 connection.port-type bond \802-3-ethernet.mtu 9216
Step 4: Verification Checklist
Once created, we bring the stack up from the top. NetworkManager will automatically trigger the underlying ports.
- Bring it up:
nmcli connection up bridge0 - Verify MTU Consistency: Every device in the chain must match.
ip link show | grep 9216
- Verify LACP Sync: The switch must see the server.
cat /proc/net/bonding/bond0(Look for “Partner Mac Address”)
- Verify Bridge Membership:
bridge link show(The bond should be listed as a member of the bridge).
Why this order matters
- If you define the physical ports first without a controller, they might try to get a DHCP address on their own.
- By setting MTU 9216 at every single stage of the
nmclicommand, you prevent the kernel from defaulting any segment to 1500, which causes the exact “packet loss” issue you experienced during the jumbo ping tests.
