Configuring SmartOS as a router with OSPF and DHCP

Published on 15th Jan, 2018 by David Young

This will be a reasonably long post about setting up a router with SmartOS. It is a fairly long process, and we finish with a script that wraps all of this work into a single command, because I am lazy.

The setup here is a backbone network on 172.20.99.0/24 and 2001:470:xxxx:2099::/64 with routes being advertised via OSPF, and we are configuring a router with DHCP server for the 172.24.200.0/23 and 2001:470:xxxx:5032::/64 network.

Manifest

The first step is to create a zone for the shiny new router to live in:

{
  "brand": "joyent",
  "quota": 20,
  "image_uuid": "23b267fc-ad02-11e7-94da-53e3d3884fe0",
  "max_physical_memory": 1024,
  "alias": "router",
  "hostname": "router",
  "resolvers": ["172.20.99.2", "2001:470:xxxx:2099::2", "9.9.9.9"],
  "nics": [
    {
      "nic_tag": "routernet",
      "ips": ["dhcp", "addrconf"],
      "vlan_id": 2099,
      "primary": 1,
      "allow_ip_spoofing": true
    },
    {
      "nic_tag": "wifi",
      "ips": ["172.24.200.1/23", "2001:470:xxxx:5032::1/64"],
      "allow_ip_spoofing": true,
      "allow_dhcp_spoofing": true,
      "vlan_id": 630
    }
  ]
}

There are a few important options to take note of in the zone manifest. The first is the "ips": ["dhcp", "addrconf"] line in the first NIC. This is necessary for the interface to pick up addresses via DHCP and SLAAC. As far as I'm aware DHCPv6 won't work without the addrconf option, and SmartOS is clever enough to allow the zone to communicate using the addresses it picks up this way (in earlier builds some of the addresses had to be explicitly specified in the vmadm config https://wiki.smartos.org/display/DOC/Setting+up+IPv6+in+a+Zone).

Secondly we need to set allow_ip_spoofing for the zone to function correctly as a router, since it will be passing packets with sources and destinations other than its own addresses. Similarly allow_dhcp_spoofing tells SmartOS to allow the new zone to act as a DHCP server for its VLAN.

The first two DNS resolvers live on the backbone network, and the third is a public external resolver as a failsafe.

Routing and Neighbour Discovery

Barring a few config files, IP routing "just works" in SmartOS, with only a few simple steps needed to get it going. Our first job is to install Quagga, which has the routing daemons for OSPF, BGP and RIP:

pkgin install quagga
svcadm enable quagga:zebra
svcadm enable quagga:ospf
svcadm enable quagga:ospf6

Zebra is the framework on top of which the OSPF and OSPF6 routing happens, it acts a bit like having a Cisco router running inside our zone and can be configured in essentially the same manner when we use vtysh to connect to its console. However configuring it directly is a bit cumbersome so we'll just edit the configuration files directly:

! /opt/local/etc/zebra/zebra.conf
log syslog
!
interface lo0
 no link-detect
!
interface net0
 no link-detect
 ipv6 nd suppress-ra
 ipv6 nd ra-interval 10
 ipv6 nd prefix 2001:470:xxxx:2099::/64
!
interface net1
 no link-detect
 ip address 172.24.200.1/23
 no ipv6 nd suppress-ra
 ipv6 nd managed-config-flag
 ipv6 address 2001:470:xxxx:5032::1/64
 ipv6 nd prefix 2001:470:xxxx:5032::/64
!
line vty
!

I'm not certain that we need the neighbour discovery parts in the zebra.conf, as there will be some similar config going into ndpd.conf later on, but it doesn't hurt to have it there. The OSPF config files are the interesting ones anyway:

! /opt/local/etc/zebra/ospfd.conf
!
log syslog
!
interface lo0
!
interface net0
!
interface net1
!
router ospf
 redistribute connected
 passive-interface net1
 network 172.20.99.0/24 area 0.0.0.0
 network 172.24.200.0/23 area 0.0.0.0
!
line vty

This is telling the router which networks it knows about and to advertise them on the backbone interface. We don't send any OSPF out of the other interface as we will be the default gateway for that network.

! /opt/local/etc/zebra/ospf6d.conf
log syslog
!
interface net0
!
interface net1
!
debug ospf6 lsa unknown
!
router ospf6
 redistribute static
 redistribute connected
 redistribute kernel
 area 0.0.0.0 range 2001:470:xxxx:2099::/64
 area 0.0.0.0 range 2001:470:xxxx:5032::/64
!
interface net0 area 0.0.0.0
!
line vty

Slightly different to the IPv4 configuration, here we are telling the router which networks it can see and then telling the interfaces which area they are in.

Next up is NDP. When a SmartOS zone boots with "addrconf" configured as one of its IPs the NDP daemon is automatically enabled. By default it only acts as a client. If we want to send out router advertisements then we need to create /etc/inet/ndpd.conf:

if net1 AdvSendAdvertisements True
if net1 AdvManagedFlag True
if net1 StatelessAddrConfig True
prefix 2001:470:xxxx:5032::/64 net1

With this we can configure NDPd to send out router advertisements and specify the relevant options, such as whether to permit stateless address config and whether we are using DHCPv6.

Finally, we restart the services we have updated and tell routeadm about our changes and to allow IP routing:

svcadm restart quagga:zebra
svcadm restart quagga:ospf
svcadm restart quagga:ospf6
svcadm restart ndp
routeadm -e ipv4-forwarding
routeadm -e ipv4-routing
routeadm -e ipv6-forwarding
routeadm -e ipv6-routing
routeadm -s routing-svcs="quagga:ospf quagga:ospf6"
routeadm -u

NAT

If you need NAT then that's easy enough to do:

# /etc/ipf/ipnat.conf
# outbound nat mappings

map net1 172.20.99.0/24 -> 0/32
map net1 172.24.0.0/16 -> 0/32

# inbound nat to irc server
rdr net1 0/0 port 6697 -> 172.24.3.12 port 6697 tcp
 # etc.

and svcadm enable ipfilter, remembering to set up the appropriate rules in /etc/ipf/ipf.conf and /etc/ipf/ipf6.conf.

DHCP

DHCP was the trickiest part of the setup as we had to configure a second instance of isc-dhcpd via svccfg. To start off with, we configure the IPv4 DHCP server, installing the package with pkgin in isc-dhcpd.

# /opt/local/etc/dhcp/dhcpd.conf
#
option domain-name "core.example.com";
option domain-name-servers 172.20.99.2, 9.9.9.9;

default-lease-time 600;
max-lease-time 7200;

authoritative;
log-facility local7;

subnet 172.20.99.0 netmask 255.255.255.0 {
        range 172.20.99.10 172.20.99.100;
        option routers 172.20.99.1;
}

Standard DHCP config stuff. Hit it with a svcadm enable isc-dhcpd and that's IPv4 done. Next job is IPv6. We make an SMF manifest by running svccfg export isc-dhcpd > isc-dhcpd6.xml and edit it so that it looks a bit more like the following:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='pkgsrc/isc-dhcpd6' type='service' version='0'>
    <create_default_instance enabled='false'/>
    <single_instance/>
    <dependency name='network' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/milestone/network:default'/>
    </dependency>
    <dependency name='filesystem-local' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local:default'/>
    </dependency>
    <dependency name='config-file' grouping='require_all' restart_on='refresh' type='path'>
      <service_fmri value='file:///opt/local/etc/dhcp/dhcpd6.conf'/>
    </dependency>
    <method_context>
      <method_credential group='root' user='root'/>
    </method_context>
    <exec_method name='start' type='method' exec='/opt/local/lib/svc/method/isc-dhcpd6 start' timeout_seconds='30'/>
    <exec_method name='stop' type='method' exec='/opt/local/lib/svc/method/isc-dhcpd6 stop' timeout_seconds='30'/>
    <exec_method name='refresh' type='method' exec='/opt/local/lib/svc/method/isc-dhcpd6 refresh' timeout_seconds='30'/>
    <template>
      <common_name>
        <loctext xml:lang='C'>ISC DHCP6 Server</loctext>
      </common_name>
    </template>
  </service>
</service_bundle>

Note that we will need to write a new init script for isc-dhcpd6 so that we can pass the -6 option to dhcpd:

#!/sbin/sh
# /opt/local/lib/svc/method/isc-dhcpd6
# $NetBSD: isc-dhcpd.sh,v 1.1 2014/03/12 14:29:31 jperkin Exp $
#
# Init script for isc-dhcpd.
#

case "$1" in
start)
        if [ ! -d /var/db/isc-dhcp ]; then
                mkdir -p /var/db/isc-dhcp
                touch /var/db/isc-dhcp/dhcpd6.leases
                chmod 0640 /var/db/isc-dhcp/dhcpd6.leases
        fi
        mkdir -p /var/run/isc-dhcp
        chmod 0770 /var/run/isc-dhcp
        /opt/local/sbin/dhcpd -6 -cf /opt/local/etc/dhcp/dhcpd6.conf
        ;;
stop)
        if [ -s /var/run/isc-dhcp/isc-dhcpd6.pid ]; then
                kill `cat /var/run/isc-dhcp/isc-dhcpd6.pid`
        fi
        ;;
refresh)
        $0 stop
        $0 start
        ;;
esac

Then it needs a configuration file:

# /opt/local/etc/dhcp/dhcpd.conf
#

option dhcp6.domain-search "core.example.com", "example.com";
option dhcp6.name-servers 2001:470:xxxx:2099::2;

default-lease-time 600;
max-lease-time 7200;

authoritative;

log-facility local7;

subnet6 2001:470:xxxx:2099::/64 {
        range6 2001:470:xxxx:2099:123:45:6:78 2001:470:xxxx:2099:678:444:222:111;
}

Notice that each of the options that can take v6 addresses now has a 6 in it.

Finally, we need to make the init script executable, touch the leases file, and import our service manifest into SMF:

# chmod +x /opt/local/lib/svc/method/isc-dhcpd6
# touch /var/db/isc-dhcp/dhcpd6.leases
# svccfg
> import isc-dhcpd6.xml
> end
# svcadm enable isc-dhcpd6

And with that we should have a fully functioning DHCP/DHCPv6 server. I always had to import the service manifest via the svccfg console rather than directly with svccfg import because the latter always gave me a "syntax error on line 1", while importing from the console worked. No idea why :P

Summary script

That was long and complicated. I have condensed this whole process into a single, massive Bash script to run on new routers; replacing a rather large amount of typing and remembering steps with a single command and a few questions and tweaks. As this post is already a shade on the long side I've uploaded the script onto github for perusal at your leisure.

https://gist.github.com/sciguy16/593814bca510dc88bedab067f1801178