[Markdown] 

Linux hints: linux multihomed routing (part2)

linux multipath routing with automatic load balancing with iproute2 and nftables

for better understanding of this article i suggest to read part1 first, because i will omit info explained where.
assuming what you have read mentioned earlier article and understand basics of multiple routing tables and ip rule command.

goals:

  • utilize all available uplinks to balance outgoing traffic

implementation:

let’s define some variables:

uplink1_device = tap0
uplink1_gw_address = 10.0.0.1
uplink1_own_address = 10.0.0.2
uplink2_device = tap1
uplink2_gw_address = 10.0.1.1
uplink2_own_address = 10.0.1.2

multipath_table_name = multipath
multipath_table_num = 10
multipath_table_mark = 0xa
  1. add additional routing table for multipath routing.
    in this article table will be:
10 multipath

and we will use 0xa mark for this table

  1. add all required routes.

you can just duplicate everything except default route from default route table
and interesting part:

ip route add default table $multipath_table_num nexthop via $uplink1_gw_address dev $uplink1_device weight 1 onlink nexthop via $uplink1_gw_address dev $uplink1_device weight 2 onlink

weight is hop priority, more is higher

  1. add rule to route traffic via our new multipath table
ip rule add from all fwmark $multipath_table_mark lookup $multipath_table_name
  1. nftables rules nftables is replacement for iptables, you can read more in official docs.
    here i will show commented config for the rest:
#clear existing ruleset
flush ruleset

#define variables (nftables does support variables which is very useful)
define uplink1_dev = "tap0"
define uplink2_dev = "tap1"
define uplink1_local_ip = 10.0.0.2
define uplink2_local_ip = 10.0.1.2

define uplink_devs = { $uplink1_dev,$uplink2_dev}
define uplink_local_ips = { $uplink1_local_ip, $uplink2_local_ip}

define uplink1_mark = 0x1
define uplink2_mark = 0x2

define multipath_mark = 0xa

table mangle {
    chain output        {
        type route hook output priority -150
        policy accept

        #this is already marked connection already associated with some uplink, let packet go, do not need any handling
        ct mark $uplink1_mark counter accept
        ct mark $uplink2_mark counter accept

        #mark packets and connections with uplink addresses
        ip saddr $uplink1_local_ip meta mark set $uplink1_mark ct mark set $uplink1_mark counter accept
        ip saddr $uplink2_local_ip meta mark set $uplink2_mark ct mark set $uplink2_mark counter accept

        #use load balancing for traffic from ```mail``` user
        ip saddr != $uplink_local_ips skuid mail meta mark set $multipath_mark ct mark set $multipath_mark counter accept
        #use load balancing for udp traffic
        ip saddr != $uplink_local_ips udp dport 53 meta mark set $multipath_mark ct mark set $multipath_mark counter accept
    }
}


table filter {
    chain input     {
        type filter hook input priority 0
        policy accept

        #if you using accept policy, you can just accept marked packets here to avoid further processing and save a bit of cpu )


        #mark packets/connections going through uplink devices
        iifname $uplink1_dev meta mark set $uplink1_mark ct mark set $uplink1_mark counter
        iifname $uplink2_dev meta mark set $uplink2_mark ct mark set $uplink2_mark counter

    }
    chain forward       {
        type filter hook forward priority 0
        policy accept
        tcp flags & (syn|rst) == syn tcp option maxseg size set rt mtu

        #this is already marked connection already associated with some uplink, let packet go, do not need any handling
        ct mark $uplink1_mark counter accept
        ct mark $uplink2_mark counter accept

        #mark packets/connections going through uplink devices
        ip saddr $uplink1_local_ip meta mark set $uplink1_mark ct mark set $uplink1_mark counter accept
        ip saddr $uplink2_local_ip meta mark set $uplink2_mark ct mark set $uplink2_mark counter accept
    }
    chain output        {
        type filter hook output priority 0
        policy accept

        #this is already marked connection already associated with some uplink, let packet go, do not need any handling
        ct mark $uplink1_mark counter accept
        ct mark $uplink2_mark counter accept

        #mark packets/connections going from uplink addresses 
        ip saddr $uplink1_local_ip meta mark set $uplink1_mark ct mark set $uplink1_mark counter accept
        ip saddr $uplink2_local_ip meta mark set $uplink2_mark ct mark set $uplink2_mark counter accept
    }
}

little explanation:

  • we marking traffic which we want send via load-balanced multipath route table with multipath mark
  • we changing mark from multipath to uplink dedicated mark by source ip or outgoing device
  • we just let go all packets marked with uplink dedicated mark (prevent further packet/connection handling)