Multi-Path Routing

Multi-Path routing with OpenBSD/FreeBSD using GIF interfaces and ifstated.

Multi-Path routing allows you to do several different concepts and methodologies. This article will be focused on fault tolerance. You can certainly use public address space.. no one can stop you... yet this article is geared towards private gif interfaces.

The concept is a central router that allows a network administrator ease of access and maintenance. It certainly doesn't need to be setup this way but does reduce overhead. With this central router you have 100+ routers connecting to it..with in those 100+ routers they have the option or ability to connect to another core router if the primary every crashes. There are many ways and options to insure the primary stays online and redundant (generators, redundancy) but this is outside the boundary of that location.

Some key thoughts

1.) Server outage and you have a ZFS mirror in another location, how can we get there?
2.) Core router that allows other routers to talk to one another has a fiber cut or hardware failure or act of nature.
3.) Corporate wide redundancy for all

The first thing to do is install ifstated;

FreeBSD: pkg add ifstated
OpenBSD: pkg_add ifstated

FreeBSD: echo ifstated_enable="YES" >> /etc/rc.conf && service start ifstated 
OpenBSD: rcctl enable ifstated && rcctl start ifstated

Assuming you have gif tunnels already established?

RouterC (main)
#To Router A
# ifconfig gif0 create
# ifconfig gif0 2.2.2.2 1.1.1.1
# ifconfig gif0 tunnel 111.111.111.111 123.123.123.123

#To Router B
# ifconfig gif1 create
# ifconfig gif1 4.4.4.4 3.3.3.3
# ifconfig gif1 tunnel 111.111.111.111 122.122.122.122

RouterD (backup main)
#To Router A
# ifconfig gif1 create
# ifconfig gif1 2.2.2.3 1.1.1.1
# ifconfig gif1 tunnel 222.222.222.222 123.123.123.123
#to Router B
# ifconfig gif1 create
# ifconfig gif1 2.2.2.4 3.3.3.3
# ifconfig gif1 tunnel 222.222.222.222 122.122.122.122 


RouterA
# ifconfig gif0 create
# ifconfig gif0 1.1.1.1 2.2.2.2
# ifconfig gif0 tunnel 123.123.123.123 111.111.111.111
#Gif1 to backup router
# ifconfig gif1 create
# ifconfig gif1 1.1.1.1 2.2.2.3
# ifconfig gif1 tunnel 123.123.123.123 222.222.222.222


RouterB
# ifconfig gif0 create
# ifconfig gif0 3.3.3.3 4.4.4.4
# ifconfig gif0 tunnel 122.122.122.122 111.111.111.111
#Gif1 to backup router
# ifconfig gif1 create
# ifconfig gif1 3.3.3.3 2.2.2.4
# ifconfig gif1 tunnel 122.122.122.122 222.222.222.222

With ifstated installed we can go through the configuration file;

$ more /usr/local/etc/ifstated.conf

init-state auto

pf_check = '( "ping -q -c 1 -W 1 10.0.11.9 > /dev/null" every 10)'


state auto {
        if ! $pf_check {
                set-state int_demoted
        }

        if $pf_check {
                set-state int_promoted
        }
}

state int_demoted {
        init {
                run "sh /etc/netstart"
                run "sh /etc/rc.local2"

        }

        if $pf_check {
                set-state int_promoted
        }
}

state int_promoted {
        init {
                run "sh /etc/netstart"
                run "sh /etc/rc.local" 
                
        }
        if ! $pf_check {
                set-state int_demoted
        }
}

To break this file down so you can understand, the line pf_check is pinging another router every ten seconds. If the ping works, keep things promoted. If ping fails then initiate set-state int_demoted. Very basic concept really.

So the default rc.local looks like this in the promoted state;

ifconfig gif0 tunnel external1 external2

/sbin/route add -net 10.0.1.0/24 10.0.11.9
/sbin/route add -net 10.0.2.0/24 10.0.11.9
/sbin/route add -net 10.0.3.0/24 10.0.11.9

and rc.local2 looks like this in the demoted state

ifconfig gif1 tunnel external1 external2

/sbin/route add -net 10.0.1.0/24 10.0.11.1
/sbin/route add -net 10.0.2.0/24 10.0.20.1
/sbin/route add -net 10.0.3.0/24 10.0.20.1

Note the two different gif interfaces (gif1 and gif2). The difference here is we're taking routes and redirecting things to another router. In this scenario we're pinging the router, when the ping fails it will redirect until it can ping the main router again. Routes will be remove and added as needed through netstart or a route del. Remote routers will need to have two gif interfaces, main routers will have many gif interfaces.

At the main router(s) make sure you route those properly in itself;

Router C/D:
/sbin/route add -net 10.0.2.0/24 10.0.2.254
/sbin/route add -net 10.0.3.0/24 10.0.3.254
/sbin/route add -net 10.0.4.0/24 10.0.4.254

The benefit to ifstated is that it can be built into pretty much any router and co-exist with diversity(cisco,juniper,linux,BSD.. etc). You're not bound to any specific brand. There are a lot of things we can do, we can grep the ping time.. if we notice high latency we could redirect to another router that isn't working as hard. OSPF would be a better alternative but can get very sophisticated especially when you start adding areas.

If you're looking for local interface redundancy, I would dig into CARP/PF/PFSYNC although ifstated can be used for this as well. Something to note, if you have at least two boxes and you do a ton of PF/Fire-walling/work there is a risk that you will lock yourself out.. it happens. You can easily setup two gif tunnels and have ifstated load a generic pf.conf file when it can no longer ping the primary tunnel.