HomeCurrent ProjectsInterestsResumeE-mail

<< Back to Soekris net4501 FreeBSD Router Project Page

How to Aggregate the Bandwidth of Multiple Internet Connections using FreeBSD 4.x
  By Michael R. Brumm

Introduction

Often an organization will require more bandwidth than a single DSL or cable Internet connection can provide, but higher bandwidth connections (T1 or T3) are beyond the reach of their budget. An alternative (and much less costly) solution is to combine multiple slower Internet connections using some form of bandwidth aggregation. This is often called "load balancing" or "site multihoming", and the specific type of aggregation I describe in this tutorial is often called "teaming" or "connection teaming".

Another possible benefit of bandwidth aggregation might be increased reliability. Because multiple separate connections are used, if one of them becomes unavailable, the others can takeover (at an obvious cost to overall bandwidth).

Ideally bandwidth aggregation would be supported by your ISP, and performed at the link layer (using multilink PPP) or at least with BGP routing support. However, most ISPs are selling their lines with only the most basic of services, so customers are left with basically only one choice: connection teaming using network address translation (NAT).

Here are the project requirements:

  • Allow NAT clients to share bandwidth from multiple Internet connections with dynamic IP addresses (cheap DSL, cable, or other Internet service) and no upstream routing support from the ISPs
  • FreeBSD 4.x
  • ipf for packet filtering
  • ipnat for network address and port translation
  • ipfw for traffic shaping
  • No natd (ipnat provides vastly better performance)
  • Minimize disruption to NAT clients if one or more Internet connections are unavailable
  • Use only a single Ethernet port on the router for all the Internet connections

It might be hard to believe, but all of these things are achievable using FreeBSD 4.x and no other additional software. You just have to learn how to make all its various pieces and parts work smoothly together.

Newcomers to FreeBSD might be wondering where I found a bandwidth aggregator in the packages library (nope, there isn't one). Experienced users of FreeBSD will wonder how I got ipnat to handle translation to multiple dynamic IP addresses on a single Ethernet port. Hardcore users will want to know how I allowed the Internet connections to be disconnected with minimal disruption to the NAT clients. Well, step right up ladies and gentlemen. The show is about to start.

The Secrets to My Success

The first thing to recognize is that policy routing can be used together with network address translation to handle bandwidth aggregation. Policy routing allows you to route packets based on policies rather than route just based on the destination address and the routing tables. This isn't a tutorial on policy routing, so if you are unfamiliar about policy routing, read the FreeBSD man pages for ipf and ipfw. ipf uses the "fastroute" or "to" command. ipfw uses the "fwd" or "forward" command.

The second thing to realize is that it is impossible to use ipnat's network address translation with multiple external IP addresses on the same Ethernet interface. The ipnat configuration doesn't even have a way to describe such a situation. However, you can create multiple virtual Ethernet interfaces on the same Ethernet port, and have ipnat use each interface independently.

Also, keep in mind that dissimilar Internet services use dissimilar authentication and connection protocols. Many DSL connections use PPPoE, some cable ISPs authenticate using a MAC address, and a few of the cheapest ISPs require you to use their router/modems. Supporting all these different environments on the FreeBSD router can be done, however it would also make this tutorial unnecessarily complicated. Instead, I'm going to use the ISP supplied NAT router/modem and perform a second network address translation.

Topology

In this tutorial, I've configured the ISP supplied routers to translate class C (/24) subnets within the unroutable 192.168.0.0/16 subnet. For the clients, I've used a class B subnet (10.0.0.0/16) in the unroutable 10.0.0.0/8 subnet. This means the FreeBSD router will translate the NAT clients in 10.0.0.0/16 to an IP address on one of its virtual Ethernet interfaces (each of which have an IP address within 192.168.0.0/16). The corresponding ISP supplied NAT router will then translate the 192.168.0.0/16 address into its public dynamic IP address.

Many organizations need at least one static IP address for providing services, so I've also included a static public IP address on sis0 (12.34.56.7). This is also configured as a backup for the NAT clients in case the connections on sis1 are down.

Of course, a simple diagram is worth a thousand words:


Requirements

Your kernel must have been built with support for ipf, ipfw, ipnat, dummynet, and netgraph. This includes (but is not limited to the following kernel options):


    options IPFIREWALL
    options IPFIREWALL_FORWARD
    options IPFILTER
    options DUMMYNET
    options NETGRAPH


In addition, you will have to build and install the following kernel module: ng_eiface.ko (not a kernel option as far as I know).

Finally, you will have to fix the order in which ipf and ipfw handle packets by following this tutorial: How to Reorder the FreeBSD 4.x ipf and ipfw Packet Filters

Create Multiple Virtual Ethernet Interfaces

We need to create and configure virtual ethernet interfaces on boot. To have the interfaces automatically created, create a file named /etc/start_if.sis1 and add this:


    ngctl mkpeer . eiface hook ether
    ngctl mkpeer . eiface hook ether
    ifconfig ngeth0 link 00:5c:12:34:56:78
    ifconfig ngeth1 link 00:5c:12:34:56:79
    ifconfig ngeth0 up
    ifconfig ngeth1 up


To enable bridging on these two new interfaces, install the following script in /usr/local/etc/rc.d: ether.bridge.0.sh

Also, configure the interfaces (and a few other things) in /etc/rc.conf with these lines:


    ipfilter_enable="YES"
    firewall_enable="YES"
    ipnat_enable="YES"

    ifconfig_sis0="inet 12.34.56.7/24"
    ifconfig_ngeth0="inet 192.168.0.100/24"
    ifconfig_ngeth1="inet 192.168.1.100/24"
    ifconfig_sis2="inet 10.0.0.1/24"
    gateway_enabled="YES"
    defaultroute="12.34.56.1"

    firewall_type="/etc/ipfw.rules"


Configure ipnat to Perform Network Address Translation

To allow network address translation on all the Internet connections, add these lines to /etc/ipnat.rules:


    map sis0 10.0.0.0/24 -> 12.34.56.7/32 proxy port ftp ftp/tcp mssclamp 1360
    map sis0 10.0.0.0/24 -> 12.34.56.7/32 portmap tcp/udp auto mssclamp 1360
    map sis0 10.0.0.0/24 -> 12.34.56.7/32 mssclamp 1360

    map ngeth0 10.0.0.0/24 -> 192.168.0.100/32 proxy port ftp ftp/tcp mssclamp 1360
    map ngeth0 10.0.0.0/24 -> 192.168.0.100/32 portmap tcp/udp auto mssclamp 1360
    map ngeth0 10.0.0.0/24 -> 192.168.0.100/32 mssclamp 1360

    map ngeth1 10.0.0.0/24 -> 192.168.1.100/32 proxy port ftp ftp/tcp mssclamp 1360
    map ngeth1 10.0.0.0/24 -> 192.168.1.100/32 portmap tcp/udp auto mssclamp 1360
    map ngeth1 10.0.0.0/24 -> 192.168.1.100/32 mssclamp 1360


Configure ipfw with Policy Routing

Policy routing is used to balance the NAT client requests between the Internet connections. The contents of /etc/ipfw.rules:


    flush
    # ipfw is used only for traffic shaping (ipf is used for filtering)

    # Everything gets passed by default
    add 65000 pass all from any to any

    # None of the checks below are for these interfaces
    add 100 pass all from any to any via lo0
    add 120 pass all from any to any via ngeth0
    add 130 pass all from any to any via ngeth1

    # Policy routing
    add 500 forward 192.168.0.1 all from 192.168.0.100 to any out xmit sis0
    add 510 forward 192.168.1.1 all from 192.168.1.100 to any out xmit sis0

    # None of the checks below are for these interfaces
    add 700 pass all from any to any via sis0


    # Incoming: coming from the Internet.
    # Outgoing: going to the Internet.
    #
    # sis2 is the LAN side, so
    #  Incoming packets out xmit to sis2
    #  Outgoing packets in recv on sis2


    # Load balance between the dynamic IP connections

     # Use state for existing connections
    add 1500 check-state

     # Allows use of the regular routing when load balancing cannot be done
      # rule 1799 reserved for check-loadbalance
    add 1800 skipto 2000 all from any to any in recv sis2
    add 1900 skipto 10000 all from any to any in recv sis2 keep-state

     # HTTP/HTTPS balanced using probability
      # rule 2000 reserved for check-loadbalance
      # rule 2001 reserved for check-loadbalance
    add 2100 prob 0.5 forward 192.168.0.1 tcp from any to any 80,443 in recv sis2 keep-state
    add 2200 forward 192.168.1.1 tcp from any to any 80,443 in recv sis2 keep-state
    add 2300 forward 192.168.0.1 tcp from any to any 80,443 in recv sis2 keep-state

     # Other protocols balanced based on source IP address
      # rule 5000 reserved for check-loadbalance
      # rule 5001 reserved for check-loadbalance
    add 5100 forward 192.168.0.1 all from 0.0.0.0:0.0.0.1 to any in recv sis2 keep-state
    add 5200 forward 192.168.1.1 all from 0.0.0.1:0.0.0.1 to any in recv sis2 keep-state
    add 5300 forward 192.168.0.1 all from any to any in recv sis2 keep-state
    add 5400 forward 192.168.1.1 all from any to any in recv sis2 keep-state


The method of balancing based on services could be improved, but this gives you a good general idea of how to balance (both based on probability and on odd/even IP addresses). Stateless protocols like HTTP and HTTPS tend to be very forgiving about receiving the same client requests from different IP addresses. Other protocols, like FTP, are not.

Additional Internet connections can be added, but if you want to balance them evenly, the number of rules and probability values get complicated (especially with check-loadbalance script enabling and disabling the rules ). Remember that the probabilities must be in the series [1/n, 1/(n-1), 1/(n-2), ... , 1]. For example, with four connections you would want to use these lines:

add 2100 prob 0.25 forward ...
add 2200 prob 0.3333 forward ...
add 2300 prob 0.5 forward ...
add 2400 forward ...
...

Also, with four Internet connections, balancing with odd and even source IP addresses won't work, but you can use mod 4 (least signifigant bits: 1 + 2) with these lines:

add 5100 forward 192.168.0.1 all from 0.0.0.0:0.0.0.3 ...
add 5200 forward 192.168.1.1 all from 0.0.0.1:0.0.0.3 ...
add 5300 forward 192.168.0.1 all from 0.0.0.2:0.0.0.3 ...
add 5400 forward 192.168.1.1 all from 0.0.0.3:0.0.0.3 ...
...

Script Watchdog for Internet Connection Outages

Now that you've got your clients connection teaming these multiple Internet connections, you'll want to examine reliability. Imagine one of the cheap Internet connections become unavailable. You'll get some unusual problems on your NAT clients: half of all HTTP/HTTPS connections will fail, and half of all NAT clients won't be able to use other protocols at all.

This would obviously not be acceptable. To handle this problem, I've scheduled a script in the crontab to watch these connections every few minutes and modify the policy routes when a connection becomes unavailable: check-loadbalance.sh

Change the variable definitions DST1_IP and DST2_IP to suit your situation. Ideally they should be ping-able IP addresses within the respective ISP's (like routers one hop away, or the ISP's DNS servers). They could be the same destination, if necessary.

This script is designed only for checking two Internet connections, but a similar idea could be applied to handle more connections. Try unplugging one of the Internet connections and running the script from the console to test it before scheduling it in the crontab.

Conclusion

As you can see, even without installing any additional software, FreeBSD can be a very powerful and flexible operating system for routing. It is always a surprise what you can do with it if you think outside the box.

 

<< Back to Soekris net4501 FreeBSD Router Project Page

 

All Material Copyright © Michael R. Brumm