Advanced Routing HOWTO: policy routing, two uplinks and tc on Linux

Updated November 2026

Normal Linux routing looks only at the destination address. With iproute2 you can also route by source address, incoming interface or a firewall mark, keep several routing tables, split traffic over two Internet links, and control queuing so a busy upload does not ruin everyone's latency. This page shows each technique with commands tested on a current kernel.

Start with what you have

The ip command from iproute2 replaced ifconfig and route long ago. These commands show the current state:

ip -br addr            # addresses, one line per interface
ip route               # the main routing table
ip route get 192.0.2.10              # which route a packet would take
ip route get 192.0.2.10 from 203.0.113.2   # ...if it had this source
ip rule show           # the routing policy database

On a fresh system, ip rule show prints three rules:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The kernel checks rules from the lowest number up. Rule 0 looks in the local table (ID 255: the machine's own and broadcast addresses). Rule 32766 looks in main (ID 254: the routes you normally see). Rule 32767 looks in default (ID 253, normally empty). Policy routing means adding your own rules between 0 and 32766 that send some packets to other tables.

Working over SSH. A wrong ip rule or default route can send your own SSH session out of the wrong link and cut it off. Have console access, or schedule an undo before you start, for example sudo systemd-run --on-active=5min /usr/sbin/ip rule del priority 1000, and cancel it once a new SSH login still works.

You can name tables in an rt_tables file, but numbers work everywhere and avoid differences between iproute2 versions in where that file lives. This page uses table 200.

A common case: a router or server with two providers.

Without extra rules, a connection that arrives on ISP B gets its reply sent out through ISP A, with ISP B's address as the source. ISP A drops it, and the connection fails. The fix is a second table for ISP B and a rule that sends packets from ISP B's address to it:

sudo ip route add 203.0.113.0/24 dev eth1 src 203.0.113.2 table 200
sudo ip route add default via 203.0.113.1 dev eth1 table 200
sudo ip rule add from 203.0.113.2 table 200 priority 1000

Check it:

$ ip route get 192.0.2.10 from 203.0.113.2
192.0.2.10 from 203.0.113.2 via 203.0.113.1 dev eth1 table 200 uid 0
    cache

Traffic from 198.51.100.2, and traffic with no particular source, still uses the main table and ISP A.

Reverse path filtering

The kernel's rp_filter setting drops packets that arrive on an interface the kernel would not use to reply. In strict mode (1), asymmetric setups like this one can lose packets. Loose mode (2) only checks that the source is reachable through some interface. The kernel uses the higher of the all and per-interface values:

sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth1.rp_filter
echo 'net.ipv4.conf.all.rp_filter = 2' | sudo tee /etc/sysctl.d/90-rpfilter.conf
sudo sysctl --system

If a LAN sits behind this router, masquerade on each uplink so packets leave with the right address. In the NAT chain from the IP Masquerade HOWTO, use oifname { "eth0", "eth1" } masquerade, and allow LAN traffic to both interfaces in the forward chain.

To send one LAN subnet through ISP B, add a rule by source network:

sudo ip rule add from 192.168.60.0/24 table 200 priority 1100

To pick by port or protocol, let nftables put a mark on the packets and route by that mark. This sends web traffic from the LAN on eth2 through ISP B:

table inet routemark {
    chain prerouting {
        type filter hook prerouting priority mangle; policy accept;
        iifname "eth2" tcp dport { 80, 443 } meta mark set 2
    }
}
sudo nft -f routemark.nft
sudo ip rule add fwmark 2 table 200 priority 900
ip rule show

The mark exists only inside the kernel; it is never sent on the wire. After changing many rules at once, the ip-rule manual suggests sudo ip route flush cache.

Load balancing with multipath routes

A route can have several next hops. The kernel picks one per flow (connection), not per packet, so a single download does not get faster, but many connections spread over both links:

sudo ip route replace default \
     nexthop via 198.51.100.1 dev eth0 weight 1 \
     nexthop via 203.0.113.1 dev eth1 weight 1

By default the kernel chooses the path from the source and destination addresses only (net.ipv4.fib_multipath_hash_policy = 0). Setting it to 1 also uses the ports, which spreads traffic between the same two hosts more evenly. Multipath does not notice when a provider fails behind a working modem; for real failover you need a monitoring tool that removes the dead route.

Make it permanent

Commands typed with ip are gone after a reboot or when the interface restarts. Put them in your network configuration.

NetworkManager (Fedora, RHEL, desktops): routes and rules belong to the connection profile.

sudo nmcli connection modify isp-b \
     ipv4.method manual ipv4.addresses 203.0.113.2/24 \
     ipv4.never-default yes \
     ipv4.routes "203.0.113.0/24 table=200, 0.0.0.0/0 203.0.113.1 table=200" \
     ipv4.routing-rules "priority 1000 from 203.0.113.2 table 200"
sudo nmcli connection up isp-b

systemd-networkd (many servers; also what netplan uses on Ubuntu Server): /etc/systemd/network/20-isp-b.network:

[Match]
Name=eth1

[Network]
Address=203.0.113.2/24

[Route]
Destination=203.0.113.0/24
Scope=link
PreferredSource=203.0.113.2
Table=200

[Route]
Gateway=203.0.113.1
Table=200

[RoutingPolicyRule]
From=203.0.113.2
Table=200
Priority=1000
sudo networkctl reload
ip rule show

VRF: separate routing worlds

A VRF (virtual routing and forwarding) device ties interfaces to their own routing table, as if they were in a separate router. It is useful for a management network that must never mix with production traffic.

sudo ip link add vrf-mgmt type vrf table 10
sudo ip link set dev vrf-mgmt up
sudo ip link set dev eth3 master vrf-mgmt
ip vrf show
ip route show vrf vrf-mgmt

The kernel adds the needed l3mdev rule automatically when the first VRF is created (priority 1000 on current kernels; if you also use 1000 for your own rule, as above, both work, but distinct numbers make ip rule show easier to read). Run a program inside the VRF with sudo ip vrf exec vrf-mgmt ssh admin@10.0.0.5; that needs cgroup v2 (stat -fc %T /sys/fs/cgroup prints cgroup2fs). Servers listening on all addresses only accept connections arriving through a VRF if net.ipv4.tcp_l3mdev_accept = 1.

Traffic control: queues and bufferbloat

When a link is full, packets wait in a queue. A big, dumb queue adds hundreds of milliseconds of delay to everything, a problem known as bufferbloat. The tc command sets the queuing discipline (qdisc) per interface.

tc qdisc show dev eth0
tc -s qdisc show dev eth0      # with drop and backlog counters

fq_codel keeps a separate queue per flow and drops or ECN-marks packets that wait too long (target 5 ms, interval 100 ms by default). Your distribution may already use it; check with sysctl net.core.default_qdisc. To set it on one interface:

sudo tc qdisc replace dev eth0 root fq_codel

fq_codel only helps if the queue builds up in Linux. On a home connection the queue usually builds in the modem, because the modem's link is slower than your Ethernet port. Then you need a shaper that sends slightly slower than the line, so the queue moves into Linux where it can be managed. cake does shaping, fair queuing and AQM in one qdisc:

# upload: shape just below the measured upload speed
sudo tc qdisc replace dev eth0 root cake bandwidth 18Mbit nat
# download: on a router, shape the LAN side (eth2) just below the download speed
sudo tc qdisc replace dev eth2 root cake bandwidth 90Mbit ingress
tc -s qdisc show dev eth0

The figures are an example for a 100/20 Mbit/s line; start a little below your measured speeds and adjust while running a latency test under load. nat makes cake look up the real inside addresses on a NAT router, so it shares bandwidth fairly between LAN hosts. Overhead keywords such as ethernet, docsis, pppoe-ptm or bridged-ptm tell cake how much framing your line type adds. The default priority mode is diffserv3; besteffort turns priorities off.

Remove a qdisc with sudo tc qdisc del dev eth0 root. Like ip commands, tc settings do not survive a reboot. systemd-networkd can set them in a .network file (it has [FairQueueingControlledDelay] and [CAKE] sections); otherwise run them from a small systemd service.

Old recipes. The CBQ, ATM and dsmark qdiscs and the RSVP and tcindex classifiers used in 2000-era guides have been deleted from the kernel. Use HTB for class-based shaping, and cake or fq_codel for latency.

Common mistakes

The classic Linux Advanced Routing & Traffic Control HOWTO, written for kernel 2.4, is kept by the LDP at tldp.org.

Sources

The ip, tc and nft commands on this page were tested in a network namespace with iproute2 6.15 and kernel 6.12.