Multi-Path Routing

Multi-Path routing with OpenBSD/FreeBSD using GIF interfaces and ifstated.

Multi-Path routing allows you to do several different concepts and methodologies. This article will be focused on fault tolerance. You can certainly use public address space.. no one can stop you... yet this article is geared towards private gif interfaces. A few questions come to mind;

1.) Server outage and you have a ZFS mirror in another location, how can we get there?
2.) Core router that allows other routers to talk to one another has a fiber cut or hardware failure.
3.) Corporate wide redundancy

The first thing to do is install ifstated;

FreeBSD: pkg add ifstated
OpenBSD: pkg_add ifstated

FreeBSD: echo ifstated_enable="YES" >> /etc/rc.conf && service start ifstated 
OpenBSD: rcctl enable ifstated && rcctl start ifstated

Assuming you have gif tunnels already established?

RouterA

1. ifconfig gif0 create
2. ifconfig gif0 internal1 internal2
3. ifconfig gif0 tunnel external1 external2

RouterB

1. ifconfig gif0 create
2. ifconfig gif0 internal1 internal2
3. ifconfig gif0 tunnel external1 external2

With ifstated installed we can go through the configuration file;

$ more /usr/local/etc/ifstated.conf

init-state auto

pf_check = '( "ping -q -c 1 -W 1 10.0.11.9 > /dev/null" every 10)'


state auto {
        if ! $pf_check {
                set-state int_demoted
        }

        if $pf_check {
                set-state int_promoted
        }
}

state int_demoted {
        init {
                run "sh /etc/netstart"
                run "sh /etc/rc.local2"

        }

        if $pf_check {
                set-state int_promoted
        }
}

state int_promoted {
        init {
                run "sh /etc/netstart"
                run "sh /etc/rc.local" 
                
        }
        if ! $pf_check {
                set-state int_demoted
        }
}

To break this file down so you can understand, the line pf_check is pinging another router every ten seconds. If the ping works, keep things promoted. If ping fails then initiate set-state int_demoted. Very basic concept really.

So the default rc.local looks like this;

ifconfig gif0 tunnel external1 external2

/sbin/route add -net 10.0.1.0/24 10.0.11.9
/sbin/route add -net 10.0.2.0/24 10.0.11.9
/sbin/route add -net 10.0.3.0/24 10.0.11.9

and rc.local2

ifconfig gif1 tunnel external1 external2

/sbin/route add -net 10.0.1.0/24 10.0.11.1
/sbin/route add -net 10.0.2.0/24 10.0.20.1
/sbin/route add -net 10.0.3.0/24 10.0.20.1

Note the two different gif interfaces (gif1 and gif2). The difference here is we're taking routes and redirecting things to another router. There are a lot of things we can do, we can grep the ping time.. if we notice high latency we could redirect to another router that isn't working as hard. In this scenario we're pinging the router, when the ping fails it will redirect until it can ping the main router again. Routes will be remove and added as needed through netstart or a route del.

At the main routers make sure you route those properly in itself;

Route A/B: /sbin/route add -net 10.0.2.0/24 10.0.2.254 /sbin/route add -net 10.0.3.0/24 10.0.3.254 /sbin/route add -net 10.0.4.0/24 10.0.4.254

The benefit to ifstated is that it can be built into pretty much any router and co-exist with diversity(cisco,juniper,linux,BSD.. etc). You're not bound to any specific brand. OSPF would be a better alternative but can get very sophisticated especially when you start adding areas.

If you're looking for local interface redundancy, I would dig into CARP/PF/PFSYNC although ifstated can be used for this as well.