All posts by Simon

JNCIP-SEC – Configuring Juniper Enhanced Web Filtering

Two more weeks left to take the JNCIP-SEC exam, or I will lose my JNCIS-SEC credentials. I have a fully loaded SRX320 cluster (with advanced feature licenses) and a vSRX in place so I need to lab up quickly. First feature on the list is Juniper Enhanced Web filtering.

Before you begin the configuration, make sure you meet the following prerequisites:

  • The device has a valid wf_key_websense_ewf license
  • The egress interface has internet connectivity
  • The device has working name-servers defined
  • If you are enabling the feature on a virtual router, be sure to define the VR under [edit security utm feature-profile web-filtering]

To activate the Juniper Enhanced Web Filtering feature, enter the following command:

[edit security utm feature-profile web-filtering]
root@NP-vSRX# show
type juniper-enhanced;

Next, enable the UTM feature profile for Juniper Enhanced Web Filtering and configure the Juniper-managed Websense server URL.

[edit security utm feature-profile web-filtering]
root@NP-vSRX# show
juniper-enhanced {
    server {
        host rp.cloud.threatseeker.com;
        port 80;
    }
}

Before we configure the Websense categories, we should define custom whitelists and blacklists, which can be used to overrule unexpected results from the automated Websense categorization.

Under the UTM custom-objects, we’ll create an URL pattern for one of our favorite networking-related forums and for a blacklisted adult site. These are then addded to custom black/whitelist categories.

[edit security utm custom-objects]
root@NP-vSRX# show
url-pattern {
    URL-networking-forumsdotcom {
        value [ http://networking-forums.com 64.90.58.116 ];
    }
    URL-sexdotcom {
        value [ sex.com 206.125.164.82 ];
    }
}
custom-url-category {
    EWF-Blacklist {
        value URL-sexdotcom;
    }
    EWF-Whitelist {
        value URL-networking-forumsdotcom;
    }
}

Our two custom URL categories are added under the global web-filtering configuration.

[edit security utm feature-profile web-filtering]
root@NP-vSRX# show
url-whitelist EWF-Whitelist;
url-blacklist EWF-Blacklist;

As for the Enhanced Web-filtering example, I’ll create one profile for standard internet access called “WF-DefaultInternetAccess”. Adult content and malicious URLs/IP addresses will be blocked by default and some ‘undesirable’ categories will be allowed, yet a security event will be logged. The default action will be set to “permit”.

[edit security utm feature-profile web-filtering juniper-enhanced]
root@NP-vSRX# show
cache;
server {
    host rp.cloud.threatseeker.com;
    port 80;
}
profile WF-DefaultInternetAccess {
    category {
        Enhanced_Adult_Content {
            action block;
        }
        Enhanced_Adult_Material {
            action block;
        }
        Enhanced_Advanced_Malware_Command_and_Control {
            action block;
        }
        Enhanced_Advanced_Malware_Payloads {
            action block;
        }
        Enhanced_Bot_Networks {
            action block;
        }
        Enhanced_Dynamic_DNS {
            action block;
        }
        Enhanced_Emerging_Exploits {
            action block;
        }
        Enhanced_Keyloggers {
            action block;
        }
        Enhanced_Malicious_Embedded_Link {
            action block;
        }
        Enhanced_Malicious_Embedded_iFrame {
            action block;
        }
        Enhanced_Malicious_Web_Sites {
            action block;
        }
        Enhanced_Mobile_Malware {
            action block;
        }
        Enhanced_Parked_Domain {
            action block;
        }
        Enhanced_Phishing_and_Other_Frauds {
            action block;
        }
        Enhanced_Proxy_Avoidance {
            action block;
        }
        Enhanced_Spyware {
            action block;
        }
        Enhanced_Abused_Drugs {
            action log-and-permit;
        }
        Enhanced_Alcohol_and_Tobacco {
            action log-and-permit;
        }
        Enhanced_Drugs {
            action log-and-permit;
        }
        Enhanced_Gambling {
            action log-and-permit;
        }
        Enhanced_Games {
            action log-and-permit;
        }
        Enhanced_Hacking {
            action log-and-permit;
        }
        Enhanced_Illegal_or_Questionable {
            action log-and-permit;
        }
        Enhanced_Nudity {
            action log-and-permit;
        }
        Enhanced_Racism_and_Hate {
            action log-and-permit;
        }
        Enhanced_Sex {
            action log-and-permit;
        }
    }
    default permit;
    fallback-settings {
        default log-and-permit;
        server-connectivity block;
        timeout block;
        too-many-requests block;
    }
}

In case a URL does not have a category assigned but only a Web Reputation Score (WBRS) we can define an action based on that score.

Note – the scores are weighted as follows:

  • 100-90–Site is considered very safe.
  • 80-89–Site is considered moderately safe.
  • 70-79–Site is considered fairly safe.
  • 60-69–Site is considered suspicious.
  • 0-59–Site is considered harmful.
[edit security utm feature-profile web-filtering juniper-enhanced profile WF-DefaultInternetAccess site-reputation-action]
root@NP-vSRX# show
very-safe permit;
moderately-safe permit;
fairly-safe log-and-permit;
suspicious quarantine;
harmful block;

For some special cases, in which the device might not be able to process the HTTP request, we can define fallback options. This basically equates to a fail-open/fail-closed decision.

[edit security utm feature-profile web-filtering juniper-enhanced profile WF-DefaultInternetAccess fallback-settings]
root@NP-vSRX# show
default log-and-permit;
server-connectivity block;
timeout block;
too-many-requests block;

With most pieces in place, the full web-filtering profile looks like this:

[edit security utm feature-profile]
root@NP-vSRX# show
web-filtering {
    url-whitelist EWF-Whitelist;
    url-blacklist EWF-Blacklist;
    type juniper-enhanced;
    juniper-enhanced {
        cache;
        server {
            host rp.cloud.threatseeker.com;
            port 80;
        }
        profile WF-DefaultInternetAccess {
            category {
                Enhanced_Adult_Content {
                    action block;
                }
                Enhanced_Adult_Material {
                    action block;
                }
                Enhanced_Advanced_Malware_Command_and_Control {
                    action block;
                }
                Enhanced_Advanced_Malware_Payloads {
                    action block;
                }
                Enhanced_Bot_Networks {
                    action block;
                }
                Enhanced_Dynamic_DNS {
                    action block;
                }
                Enhanced_Emerging_Exploits {
                    action block;
                }
                Enhanced_Keyloggers {
                    action block;
                }
                Enhanced_Malicious_Embedded_Link {
                    action block;
                }
                Enhanced_Malicious_Embedded_iFrame {
                    action block;
                }
                Enhanced_Malicious_Web_Sites {
                    action block;
                }
                Enhanced_Mobile_Malware {
                    action block;
                }
                Enhanced_Parked_Domain {
                    action block;
                }
                Enhanced_Phishing_and_Other_Frauds {
                    action block;
                }
                Enhanced_Proxy_Avoidance {
                    action block;
                }
                Enhanced_Spyware {
                    action block;
                }
                Enhanced_Abused_Drugs {
                    action log-and-permit;
                }
                Enhanced_Alcohol_and_Tobacco {
                    action log-and-permit;
                }
                Enhanced_Drugs {
                    action log-and-permit;
                }
                Enhanced_Gambling {
                    action log-and-permit;
                }
                Enhanced_Games {
                    action log-and-permit;
                }
                Enhanced_Hacking {
                    action log-and-permit;
                }
                Enhanced_Illegal_or_Questionable {
                    action log-and-permit;
                }
                Enhanced_Nudity {
                    action log-and-permit;
                }
                Enhanced_Racism_and_Hate {
                    action log-and-permit;
                }
                Enhanced_Sex {
                    action log-and-permit;
                }
            }
            site-reputation-action {
                very-safe permit;
                moderately-safe permit;
                fairly-safe log-and-permit;
                suspicious quarantine;
                harmful block;
            }
            default permit;
            fallback-settings {
                default log-and-permit;
                server-connectivity block;
                timeout block;
                too-many-requests block;
            }
        }
    }
}

Next, we need configure a new UTM policy which includes the Enhanced Web Filtering feature. This can be combined with other UTM features such as Sophos anti-virus or content filtering.

[edit security utm]
root@NP-vSRX# set utm-policy UTM-DefaultInternetAccess web-filtering http-profile WF-DefaultInternetAccess

The last step is to attach the UTM profile to a security policy by configuring the application-services action.

[edit security policies from-zone internal-np to-zone internal-transit policy FW-Internal-PermitAll]
root@NP-vSRX# show
match {
    source-address any;
    destination-address any;
    application any;
}
then {
    permit {
        application-services {
            utm-policy UTM-DefaultInternetAccess;
        }
    }
}

Configuring logging

To make webfilter troubleshooting easier, configure the following syslog files, or alternatively send them to a syslog server.

[edit system syslog]
root@NP-vSRX# show
file blocked-sites {
    any any;
    match WEBFILTER_URL_BLOCKED;
}
file allowed-sites {
    any any;
    match WEBFILTER_URL_PERMITTED;
}

Testing

Testing the custom blacklist by browsing to sex.com – HTTPS session was broken (based on destination IP) and the browser returned an error.

Oct 13 14:21:14  NP-vSRX RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.255.1.13(50278)->206.125.164.82(443) CATEGORY="EWF-Blacklist" REASON="BY_BLACK_LIST" PROFILE="WF-DefaultInternetAccess" URL=206.125.164.82 OBJ=/ username N/A roles N/A

Next, we test the URL in the whitelist (which was actually be blocked by Websense). Networking-forums loads fine now and we have this in the log:

Oct 13 14:23:43  NP-vSRX RT_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" 10.255.1.13(50303)->64.90.58.116(443) CATEGORY="EWF-Whitelist" REASON="BY_WHITE_LIST" PROFILE="WF-DefaultInternetAccess" URL=64.90.58.116 OBJ=/ username N/A roles N/A

As a last test, we initiate an HTTP session to http://www.penthouse.com
In the browser, we get the following message:

Juniper Networks Firewall has blocked the URL: 10.255.1.13(62175)->64.59.126.214(80) www.penthouse.com/ CATEGORY: Enhanced_Adult_Content REASON: BY_PRE_DEFINED

And in the syslog file, it looks like this:

Oct 13 14:33:29  NP-vSRX RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.255.1.13(62175)->64.59.126.214(80) CATEGORY="Enhanced_Adult_Content" REASON="BY_PRE_DEFINED" PROFILE="WF-DefaultInternetAccess" URL=www.penthouse.com OBJ=/ username N/A roles N/A

Verification Commands

To check connectivity between your SRX and the Juniper Websense server:

root@NP-vSRX> show security utm web-filtering status
 UTM web-filtering status:
    Server status: Juniper Enhanced using Websense server UP

Web-filtering statistics – this is useful when the service is not working as expected.

root@NP-vSRX> show security utm web-filtering statistics
 UTM web-filtering statistics:
    Total requests:                     17086
    white list hit:                     20
    Black list hit:                     20
    No license permit:                  0
    Queries to server:                  6472
    Server reply permit:                1868
    Server reply block:                 9
    Server reply quarantine:            0
    Server reply quarantine block:      0
    Server reply quarantine permit:     0
    Custom category permit:             0
    Custom category block:              0
    Custom category quarantine:         0
    Custom category qurantine block:    0
    Custom category quarantine permit:  0
    Site reputation permit:             10597
    Site reputation block:              2495
    Site reputation quarantine:         49
    Site reputation quarantine block:   0
    Site reputation quarantine permit:  1170
    Site reputation by Category         0
    Site reputation by Global           14311
    Cache hit permit:                   847
    Cache hit block:                    11
    Cache hit quarantine:               0
    Cache hit quarantine block:         0
    Cache hit quarantine permit:        0
    Safe-search redirect:               0
    Web-filtering sessions in total:    128000
    Web-filtering sessions in use:      2
    Fallback:                       log-and-permit           block
          Default                                 0               0
          Timeout                                 0               0
     Connectivity                                 0               0
Too-many-requests                                 0               0

That’s pretty much all there is to it for a basic EWF setup. There are many more nerdknobs you can tune but I’ll stick to the basics for my JNCIP-SEC studies.

Bear in mind that the URL filter only works for HTTP traffic and most of the web these day is on SSL and using CDN networks, so it’s very hard to get a good match on SSL traffic. This can be solved by using SSL Forward Proxy or relying more on Layer-7 AppID with custom application signatures.

Sources:

Juniper Enhanced Web Filtering feature guide.

DHCPv6 configuration for isc-dhcp-server

My network is running IPv6 with SLAAC and the JunOS version on my firewall does not support RDNSS, so I needed a full-featured DHCP server to hand out DNS information.

I already have a Raspberry Pi running isc-dhcp-server for my IPv4 network so I thought it would be simple to add IPv6 support. Unfortunately it was not – the ISC DHCP server does not support dual-stack natively so you need to run two independent services in parallel.

The official documentation for this is very limited and outdated, so it took a lot of blogs and forum posts to get it working, so perhaps this post will help others in their quest for DHCPv6 support 🙂

I am assuming that you have already installed and configured the DHCP server for IPv4. If not, you can do so by entering the following command.

pi@pi ~ $ sudo apt-get install isc-dhcp-server

After installation, you can start the service manually, or reboot.

pi@pi ~ $ sudo service isc-dhcp-server start
[ ok ] Starting ISC DHCP server: dhcpd.

This post focuses on the DHCPv6 part. For the actual configuration for standard DHCP, there are plenty of good guides available online.

Configuring a static IPv6 address

In case you haven’t done so, be sure to configure your device with a static IPv6 address.

pi@pi ~ $ cat /etc/network/interfaces

# iface eth0 inet dhcp

# Loopback interface

auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static

# Static IPv4

address 10.6.0.36
gateway 10.6.0.1
network 10.6.0.0
netmask 255.255.255.0
broadcast 10.6.0.255

# Static IPv6

iface eth0 inet6 static
address 2a02:1234:420a:100b::10
netmask 64
gateway 2a02:1234:420a:100b::1

Restart the network interface or reboot the device to apply it.

INIT file – /etc/init.d/isc-dhcp-server6

First step is to copy the working init script, which is used by the v4 server, to a new file.

pi@pi ~ $ sudo cp /etc/init.d/isc-dhcp-server /etc/init.d/isc-dhcp-server6

This will create a new file isc-dhcp-server6 – use nano or vi to edit it.

Some of the values in the ### INIT INFO ### section need to modified to represent the new dhcpv6 service. I’ve also modified the value for the default config file, DHCPD_DEFAULT, to be /etc/default/isc-dhcp-server6 – more on that file later.

The script below shows only the first part of the script so don’t copy-paste.

#!/bin/sh
#
#

### BEGIN INIT INFO
# Provides:          isc-dhcp-server6
# Required-Start:    $remote_fs $network $syslog
# Required-Stop:     $remote_fs $network $syslog
# Should-Start:      $local_fs slapd $named
# Should-Stop:       $local_fs slapd
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: DHCPv6 server
# Description:       Dynamic Host Configuration Protocol Server for IPv6
### END INIT INFO

PATH=/sbin:/bin:/usr/sbin:/usr/bin

test -f /usr/sbin/dhcpd || exit 0

DHCPD_DEFAULT="${DHCPD_DEFAULT:-/etc/default/isc-dhcp-server6}"

# It is not safe to start if we don't have a default configuration...
if [ ! -f "$DHCPD_DEFAULT" ]; then
        echo "$DHCPD_DEFAULT does not exist! - Aborting..."
        if [ "$DHCPD_DEFAULT" = "/etc/default/isc-dhcp-server" ]; then
                echo "Run 'dpkg-reconfigure isc-dhcp-server' to fix the problem."
        fi
        exit 0
fi

. /lib/lsb/init-functions

# Read init script configuration
[ -f "$DHCPD_DEFAULT" ] && . "$DHCPD_DEFAULT"

NAME=dhcpd
DESC="ISC DHCPv6 server"
# fallback to default config file
DHCPD_CONF=${DHCPD_CONF:-/etc/dhcp/dhcpd6.conf}
# try to read pid file name from config file, with fallback to /var/run/dhcpd.pid
if [ -z "$DHCPD_PID" ]; then
        DHCPD_PID=$(sed -n -e 's/^[ \t]*pid-file-name[ \t]*"(.*)"[ \t]*;.*$/\1/p' < "$DHCPD_CONF" 2>/dev/null | head -n 1)
fi
DHCPD_PID="${DHCPD_PID:-/var/run/dhcpd.pid}"


... rest of script omitted for brevity ...

Defaults file – /etc/default/isc-dhcp-server6

This file stores the values used by the INIT script, so it’s absolutely essential that they are correct, or the service will refuse to start or will not work as expected.

Again, you can copy the existing file and modify those parameters.

sudo cp /etc/default/isc-dhcp-server /etc/default/isc-dhcp-server6

The configuration file for this example has four different values.

  • DHCPD_CONF – this value identifies the configuration file which will hold our DHCPv6 attributes.
  • DHCPD_PID – when the service is started, the process ID for the service will be written to this file
  • OPTIONS – here we need to specify the server to run in v6 mode by using the -6 knob
  • INTERFACES – the interface(s) on which the DHCP server will be listening, in my case eth0
# Defaults for isc-dhcp-server6 initscript
# sourced by /etc/init.d/isc-dhcp-server6
# installed at /etc/default/isc-dhcp-server6 by the maintainer scripts

#
# This is a POSIX shell fragment
#

# Path to dhcpd's config file (default: /etc/dhcp/dhcpd.conf).
DHCPD_CONF=/etc/dhcp/dhcpd6.conf

# Path to dhcpd's PID file (default: /var/run/dhcpd.pid).
DHCPD_PID=/var/run/dhcpd6.pid

# Additional options to start dhcpd with.
#       Don't use options -cf or -pf here; use DHCPD_CONF/ DHCPD_PID instead
OPTIONS="-6"

# On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
#       Separate multiple interfaces with spaces, e.g. "eth0 eth1".
INTERFACES="eth0"

Config file – /etc/dhcp/dhcpd6.conf

This file, just like in the IPv4 version, contains the global DHCP options and the subnet statements, reservations, etc. I am using it in conjunction with SLAAC, so DHCPv6 is only used to hand out the DNS info.

Note – For IPv6 subnets, you need to use the subnet6 statement.

# The ddns-updates-style parameter controls whether or not the server will
# attempt to do a DNS update when a lease is confirmed.
ddns-update-style none;

# Option definitions common to all supported networks...
default-lease-time 600;
max-lease-time 7200;

# This DHCP server is the official DHCP server for the local network
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# Subnet declaration
subnet6 2a02:1234:420a:112d::/64  {
        option dhcp6.name-servers 2a02:1234:420a:100b::10, 2001:4860:4860::8888;
        option dhcp6.domain-search "netprobe.local";
}

subnet6 2a02:1234:420a:100b::/64 {
        option dhcp6.name-servers 2a02:1234:420a:100b::10, 2001:4860:4860::8888;
        option dhcp6.domain-search "netprobe.local";
}

The DHCP6 leases file

It’s possible the service will not start because this file does not exist:

Dec  5 21:37:00 pi dhcpd: Can't open lease database /var/lib/dhcp/dhcpd6.leases: No such file or directory --
Dec  5 21:37:00 pi dhcpd:   check for failed database rewrite attempt!

Just create an empty file and it should be OK.

pi@pi / $ sudo touch /var/lib/dhcp/dhcpd6.leases

Starting the Service and verifying it

If the configuration files are correct, we can now manually start the service.

pi@pi /etc/default $ sudo service isc-dhcp-server6 start
[ ok ] Starting ISC DHCPv6 server: dhcpd.

If you are getting an error, you will usually find a clue in the /var/log/syslog files…

Verify that the second service is now running:

pi@pi /etc/default $ ps aux | grep dhcp
root      2091  0.0  1.2   6824  5368 ?        Ss   20:52   0:00 /usr/sbin/dhcpd -q -cf /etc/dhcp/dhcpd.conf -pf /var/run/dhcpd.pid eth0
root      2751  0.0  0.6   5484  2868 ?        Ss   21:18   0:00 /usr/sbin/dhcpd -q -6 -cf /etc/dhcp/dhcpd6.conf -pf /var/run/dhcpd6.pid eth0
pi        2772  0.0  0.4   3572  1944 pts/0    S+   21:19   0:00 grep --color=auto dhcp

To validate the process ID as written at service start time.

pi@pi /etc/default $ cat /var/run/dhcpd6.pid
2751

To verify that the service is listening on UDP port 547:

pi@pi /etc/default $ netstat -an | grep 547
udp6       0      0 :::547                  :::*

If you have your DHCP relays set up correctly, you can also verify communication with tcpdump. Here we can see a client is requesting information from DHCP.

pi@pi /etc/default $ sudo tcpdump -i eth0 port 547
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:21:36.935479 IP6 2a02:1234:420a:112d::1.dhcpv6-server > 2a02:1234:420a:100b::10.dhcpv6-server: dhcp6 relay-fwd
21:21:36.942854 IP6 2a02:1234:420a:100b::10.dhcpv6-server > 2a02:1234:420a:112d::1.dhcpv6-server: dhcp6 relay-reply
21:21:36.951406 IP6 2a02:1234:420a:112d::1.dhcpv6-server > 2a02:1234:420a:100b::10.dhcpv6-server: dhcp6 relay-fwd
21:21:36.955437 IP6 2a02:1234:420a:100b::10.dhcpv6-server > 2a02:1234:420a:112d::1.dhcpv6-server: dhcp6 relay-reply
21:22:16.092006 IP6 2a02:1234:420a:112d::1.dhcpv6-server > 2a02:1234:420a:100b::10.dhcpv6-server: dhcp6 relay-fwd
21:22:16.094353 IP6 2a02:1234:420a:100b::10.dhcpv6-server > 2a02:1234:420a:112d::1.dhcpv6-server: dhcp6 relay-reply
21:22:16.169193 IP6 2a02:1234:420a:112d::1.dhcpv6-server > 2a02:1234:420a:100b::10.dhcpv6-server: dhcp6 relay-fwd
21:22:16.171748 IP6 2a02:1234:420a:100b::10.dhcpv6-server > 2a02:1234:420a:112d::1.dhcpv6-server: dhcp6 relay-reply

Auto-starting the service at boot

To add the new service to the initialization scripts, enter the following command.

pi@pi ~ $ sudo update-rc.d isc-dhcp-server6 defaults
update-rc.d: using dependency based boot sequencing
pi@pi ~ $

After a reboot, you should again have two running instances of the dhcpd service – one with the -6 option.

pi@pi ~ $ ps aux | grep dhcp
root      2100  0.0  1.1   6824  5120 ?        Ss   21:30   0:00 /usr/sbin/dhcpd -q -cf /etc/dhcp/dhcpd.conf -pf /var/run/dhcpd.pid eth0
root      2135  0.0  0.6   5484  3048 ?        Ss   21:30   0:00 /usr/sbin/dhcpd -q -6 -cf /etc/dhcp/dhcpd6.conf -pf /var/run/dhcpd6.pid eth0
pi        2356  0.0  0.4   3568  1836 pts/0    S+   21:32   0:00 grep --color=auto dhcp

If everything went well, your clients should now be picking up the DNS servers (or other DHCPv6 information) from your new DHCPv6 server.

If this guide does not work for you, please let me know in the comments!

IPv6 on JunOS – Static Routing

In my previous post I configured IPv6 with Prefix Delegation. IPv6-PD is alright for a simple topology but it is very, very limiting and seems quite complicated to implement when you have many subnets in a complex topology.

This post will cover static IPv6 routing. Ideally I would have implemented OSPFv3 to handle dynamic routing but sadly, my switches do not support that feature, even with the EFL license.

Lab Topology

Here is a small part of my network which I’ll be using to test it.

IPv6 Topology

Notice that the client VLAN is the only segment with a globally routed IPv6 address. With IPv6, it is perfectly fine to use only link-local (non-routed) addressing on your transit networks, and in fact it has several benefits! If you’re curious what those benefits are, you can read all about it in RFC7407 by Eric Vyncke and Michael Behringer.

Addressing configuration

My provider has assigned me two static prefixes to me. These are not the real ones btw 🙂

IPv6 LAN prefix 	2a02:1234:420a::/48
IPv6 WAN prefix 	2a02:1234:8401:9a00::/64

First, let’s configure our WAN interface with a static IP address in the WAN range.

[edit]
admin@NPFW01# set interfaces pp0 unit 0 family inet6 address 2a02:1234:8401:9a00::1/64

As I’m using PPPoE, I will need to add a static default route (::/0) with the pp0.0 interface as next-hop. Also, I’m using VRFs so I need to add it under the right routing instance.

[edit routing-instances VRF-Edge]
admin@NPFW01# set routing-options rib VRF-Edge.inet6.0 static route ::/0 next-hop pp0.0

A quick ping to Google to verify that we have IPv6 connectivity on the external interface.

admin@NPFW01> ping inet6 2001:4860:4860::8888 routing-instance VRF-Edge source 2a02:1234:8401:9a00::1 rapid count 10
PING6(56=40+8+8 bytes) 2a02:1234:8401:9a00::1 --> 2001:4860:4860::8888
!!!!!!!!!!
--- 2001:4860:4860::8888 ping6 statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 30.980/31.647/32.716/0.444 ms

So far, so good. Let’s configure the internal interfaces now.

Configuration for the link-local addresses on the Firewall:

[edit interfaces ae0 unit 90 family inet6]
admin@NPFW01# show
address fe80:90::2/64;
[edit interfaces ae1 unit 10 family inet6]
admin@NPFW01# show
address fe80:10::2/64;
[edit interfaces ae1 unit 100 family inet6]
admin@NPFW01# show
address fe80:100::2/64;

And for the switch:

{master:0}[edit interfaces vlan unit 10 family inet6]
admin@NPSWC01# show
address fe80:10::1/64;
{master:0}[edit interfaces vlan unit 90 family inet6]
admin@NPSWC01# show
address fe80:90::1/64;
{master:0}[edit interfaces vlan unit 100 family inet6]
admin@NPSWC01# show
address fe80:100::1/64;
{master:0}[edit interfaces vlan unit 101 family inet6]
admin@NPSWC01# show
address fe80:101::1/64;

In my case, the interfaces are already in the right routing instances so I don’t need to add them anymore. Don’t forget this step though, if you are recreating this in your own lab.

After a commit, we can confirm reachability between the devices inside their routing instances.

admin@NPSWC01> ping fe80:90::2 routing-instance VRF-Transit source fe80:90::1 rapid count 5
PING6(56=40+8+8 bytes) fe80:90::1 --> fe80:90::2
!!!!!
--- fe80:90::2 ping6 statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 3.981/4.689/6.566/0.947 ms
admin@NPSWC01> ping fe80:10::2 routing-instance VRF-Transit source fe80:10::1 rapid count 5
PING6(56=40+8+8 bytes) fe80:10::1 --> fe80:10::2
!!!!!
--- fe80:10::2 ping6 statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 4.181/5.018/6.296/0.731 ms
admin@NPSWC01> ping fe80:100::2 routing-instance VRF-NP source fe80:100::1 rapid count 5
PING6(56=40+8+8 bytes) fe80:100::1 --> fe80:100::2
!!!!!
--- fe80:100::2 ping6 statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 3.924/4.819/5.715/0.575 ms

Both devices should now have an entry for eachother in the IPv6 neighbour tables, which is the equivalent of the v4 ARP table.

admin@NPSWC01> show ipv6 neighbors
IPv6 Address                 Linklayer Address  State       Exp Rtr Secure Interface
fe80:10::2                   a8:d0:e5:d3:2a:81  stale       927 yes no      vlan.10
fe80:90::2                   a8:d0:e5:d3:2a:80  stale       896 yes no      vlan.90
fe80:100::2                  a8:d0:e5:d3:2a:81  stale       887 yes no      vlan.100

Client network addressing

Last step is to assign the VLAN201 interface with a publically routed interface. I will be configuring this address: 2a02:1234:420a:10c9::1/64

There are probably more elegant addressing designs but this is what I came up with:

  • The first 48 bits are our globally routed prefix.
  • For the next 16 bits, I’ve used the leading 4 bits to represent the site number. This is my first site, so number 1.
  • The last 12 bits represent the VLAN number in hexadecimal notation, as there are 4096 vlans available in the dot1q standard – VLAN201 is C9 in hex!

Granted, it’s not the most scalable solution, as it only gives me 15 more sites and it wastes a lot of addresses but hey, this is not a multinational yet! 🙂

Here’s the configuration for the VLAN201 SVI:

{master:0}[edit interfaces vlan unit 201 family inet6]
admin@NPSWC01# show
address 2a02:1234:420a:10c9::1/64;

Enabling Router Advertisements

To enable dynamic address assignments (SLAAC) for the clients behind it, we also need to enable router advertisements on this interface. The other-stateful-configuration command will add the O-Flag to the advertisements, so we can later add the additional DNS information via DHCPv6.

{master:0}[edit protocols router-advertisement interface vlan.201]
admin@NPSWC01# show
other-stateful-configuration;
prefix 2a02:1234:420a:10c9::/64;
}

After committing, the client has assigned itself an IPv6 based on the received router advertisements.

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : ***-networks.local
   IPv6 Address. . . . . . . . . . . : 2a02:1234:420a:10c9:b1d1:660c:ac69:c5f8
   Temporary IPv6 Address. . . . . . : 2a02:1234:420a:10c9:3c71:b6f2:e93f:67ab
   Link-local IPv6 Address . . . . . : fe80::b1d1:660c:ac69:c5f8%6
   IPv4 Address. . . . . . . . . . . : 10.255.1.13
   Subnet Mask . . . . . . . . . . . : 255.255.255.128
   Default Gateway . . . . . . . . . : fe80::5e45:27ff:fee7:af81%6
                                       10.255.1.1

The basics are now in place. Now we will add some routing to make this prefix reachable.

Configuring static routes

As mentioned, my EX2200 switches do not support OSPFv3 so static routing is the only option for now…

Note – If you are using link-local addresses as the next-hop, you must use the qualified-next-hop statement with the interface in question.

Static routes, using link-local addresses, added on the switch’s routing-instances:

{master:0}[edit routing-instances VRF-Transit routing-options]
admin@NPSWC01# show
rib VRF-Transit.inet6.0 {
    static {
        route ::/0 {
            qualified-next-hop fe80:90::2 {
                interface vlan.90;
            }
        }
        route 2a02:1234:420a:10c9::/64 {
            qualified-next-hop fe80:10::2 {
                interface vlan.10;
            }
        }
    }
}
{master:0}[edit routing-instances VRF-NP routing-options]
admin@NPSWC01# show
rib VRF-NP.inet6.0 {
    static {
        route ::/0 {
            qualified-next-hop fe80:100::2 {
                interface vlan.100;
            }
        }
    }
}

And the routes on the firewall:

[edit routing-instances VRF-Internal routing-options]
admin@NPFW01# show
rib VRF-Internal.inet6.0 {
    static {
        route ::/0 {
            qualified-next-hop fe80:10::1 {
                interface ae1.10;
            }
        }
        route 2a02:1234:420a:10c9::/64 {
            qualified-next-hop fe80:100::1 {
                interface ae1.100;
            }
        }
    }
}
admin@NPFW01# show
rib VRF-Edge.inet6.0 {
    static {
        route ::/0 next-hop pp0.0;
        route 2a02:1234:420a:10c9::/64 {
            qualified-next-hop fe80:90::1 {
                interface ae0.90;
            }
        }
    }
}

After adding this configuration, and assuming that the necessary firewall policies are in place, the client can now ping outbound to the internet.

C:\Users\user>ping 2001:4860:4860::8888

Pinging 2001:4860:4860::8888 with 32 bytes of data:
Reply from 2001:4860:4860::8888: time=30ms
Reply from 2001:4860:4860::8888: time=30ms
Reply from 2001:4860:4860::8888: time=31ms
Reply from 2001:4860:4860::8888: time=30ms

Ping statistics for 2001:4860:4860::8888:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 30ms, Maximum = 31ms, Average = 30ms

That’s all it takes to implement static routing for IPv6, easy-peasy!

IPv6 on Juniper SRX – Prefix Delegation & DHCPv6

I’m currently setting up IPv6 in my own office so I thought it would be worth documenting the configuration I’m adding on my Juniper equipment to make it all work.

I have an internet subscription with a static IPv4 address and also included with it is a /48 static IPv6 prefix. There are two ways I can set it up: static configuration or “semi-automatic” by using Prefix Delegation. This article discusses the latter and in my next post I will document the static configuration, as that’s what I will ultimately be using.

First Things First – Enabling IPv6 Forwarding

Before you can start using IPv6 on the SRX, the device needs to be explicitly configured to forward V6 traffic. This because, by default, the forwarding mode for IPv6 is set to drop. You can verify this on the command line – as shown below, the default forwarding mode for Inet6 traffic is drop:

np@NP-FW01> show security flow status
  Flow forwarding mode:
    Inet forwarding mode: flow based
    Inet6 forwarding mode: drop
    MPLS forwarding mode: drop
    ISO forwarding mode: drop
  Flow trace status
    Flow tracing status: off
  Flow session distribution
    Distribution mode: RR-based
  Flow ipsec performance acceleration: off
  Flow packet ordering
    Ordering mode: Hardware

To activate it, you must change the forwarding mode. There are three modes to choose from: drop (default), flow-based and packet-based.

[edit]
np@NP-FW01# set security forwarding-options family inet6 mode ?
Possible completions:
  drop                 Disable forwarding
  flow-based           Enable flow-based forwarding
  packet-based         Enable packet-based forwarding

I’m using the SRX as a stateful firewall, so I’ll configure flow-based and commit. JunOS will warn you that the change is active only after a device reboot, so I’ll reboot the SRX and continue below.

[edit]
np@NP-FW01# set security forwarding-options family inet6 mode flow-based
[edit]
np@NP-FW01# show | compare
[edit security]
+   forwarding-options {
+       family {
+           inet6 {
+               mode flow-based;
+           }
+       }
+   }
[edit]
np@NP-FW01# commit
warning: You have enabled/disabled inet6 flow.
You must reboot the system for your change to take effect.
If you have deployed a cluster, be sure to reboot all nodes.
commit complete

After the reboot, the forwarding mode is set to flow based and we can start implementing IPv6.

np@NP-FW01> show security flow status
  Flow forwarding mode:
    Inet forwarding mode: flow based
    Inet6 forwarding mode: flow based
    MPLS forwarding mode: drop
    ISO forwarding mode: drop
  Flow trace status
    Flow tracing status: off
  Flow session distribution
    Distribution mode: RR-based
  Flow ipsec performance acceleration: off
  Flow packet ordering
    Ordering mode: Hardware

Configuring IPv6 with Prefix Delegation

Prefix delegation is a simple method to automate the assignment of IPv6 prefixes on subscriber equipment. The CPE (in my case the SRX) informs the upstream router, called the Broadband Network Gateway (BNG) that it wants to use Prefix Delegation. The BNG retrieves the prefix for the subscriber and announces the /48 back to the CPE, after which both sides confirm the prefix allocation. The CPE will then automatically divide the /48 into smaller /64 subnets on the LAN interfaces it has configured for PD Router Advertisements.

If you want to find out exactly how it works, you can find more information in RFC3633

First, we configure the outside interface, pp0.0 in my case. Notice that we are configuring this under family inet6.

The DHCPv6 client is activated on the outside, ISP-facing interface. Some PD-specific configuration is added and we will request the DNS server addresses from the provider.

[edit interfaces pp0 unit 0 family inet6]
np@NP-FW01# show
dhcpv6-client {
    client-type statefull;
    client-ia-type ia-pd;
    rapid-commit;
    client-identifier duid-type duid-ll;
    req-option dns-server;
    retransmission-attempt 0;
}

The LAN interface, in my case ae0.311, on which I will further delegate the prefix, is specified. Showing this separately as it’s important not to miss this one! 🙂

[edit interfaces pp0 unit 0 family inet6]
np@NP-FW01# set dhcpv6-client update-router-advertisement interface ae0.311

The last step is to allow DHCPv6 packets on the security zone inbound traffic. Could be that you need to add it under the interface, if you specified it further down on that level.

[edit security zones security-zone edge-untrust]
np@NP-FW01# set host-inbound-traffic system-services dhcpv6

If you want to use this on the LAN segment, you will also need to enable DHCPv6 on that security zone and interface. I’m using a guest VLAN for testing so I’ll enable it there.

[edit]
np@NP-FW01# set security zones security-zone edge-guest host-inbound-traffic system-services dhcpv6

After a commit, we see that interface ae0.311 now has a public IPv6 address with a /64 mask.

np@NP-FW01> show interfaces terse
Interface               Admin Link Proto    Local                 Remote

 ae0.311                 up    up   inet     192.168.200.1/24
                                   inet6    2a02:1234:420a:1::1/64
                                            fe80::aad0:e501:37d3:2a80/64
...

Notice that the SRX has taken the original /48 prefix and automatically derived a /64 subnet from it for interface ae0.311: 2a02:1234:420a:1::1/64 .

Note – real prefix modified for obvious reasons 🙂

Before we can test reachability, we will need to configure the default route via pp0.0. I’m using routing-instances (VRFs) so I will add it on my external edge instance.

[edit routing-instances VRF-Edge routing-options]
np@NP-FW01# show
rib VRF-Edge.inet6.0 {
    static {
        route ::/0 next-hop pp0.0;
    }
}

With the default route in place, we should be able to ping a public v6 address from this VRF. Below, we are pinging Google DNS, from the interface with Prefix Delegation.

np@NP-FW01> ping 2001:4860:4860::8888 routing-instance VRF-Edge rapid count 10 source 2a02:1234:420a:1::1
PING6(56=40+8+8 bytes) 2a02:1234:420a:1::1 --> 2001:4860:4860::8888
!!!!!!!!!!
--- 2001:4860:4860::8888 ping6 statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 31.811/32.841/35.861/1.163 ms

Success! This shows that IPv6-PD is working as expected and that the delegated prefix is globally reachable.

After connecting a test machine inside the VLAN, we can confirm that the router advertisements are working as expected. The client has taken the /64 prefix and assigned itself an EUI-64 address.

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : netprobe.guest
   Description . . . . . . . . . . . : Intel(R) Ethernet Connection (4) I219-V
   Physical Address. . . . . . . . . : C8-5B-76-BC-04-6E
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2a02:1234:420a:1:103f:efd8:f2f7:d0cf(Preferred)
   Temporary IPv6 Address. . . . . . : 2a02:1234:420a:1:293c:fd60:c216:8123(Preferred)
   Link-local IPv6 Address . . . . . : fe80::103f:efd8:f2f7:d0cf%14(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.200.13(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : zondag 12 november 2017 15:23:41
   Lease Expires . . . . . . . . . . : maandag 13 november 2017 15:26:16
   Default Gateway . . . . . . . . . : fe80::aad0:e501:37d3:2a80%14
                                       192.168.200.1
   DHCP Server . . . . . . . . . . . : 192.168.200.1
   DHCPv6 IAID . . . . . . . . . . . : 63462262
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-20-35-58-1B-C8-5B-76-BC-04-6E
   DNS Servers . . . . . . . . . . . : 208.67.220.123
                                       208.67.222.123
   NetBIOS over Tcpip. . . . . . . . : Enabled

Now, this part seems to be working fine, but you may notice that the client has not received any IPv6 name servers.
This is because the client uses SLAAC to configure its v6 address based on the Router Advertisements, but it still relies on DHCPv6 to receive any special information such as DNS servers.

To make this work, we need to add a few more bits of configuration to use the SRX as a a DHCPv6 server.

Handing out DNS servers via DHCPv6

First, configure the DHCPv6 pool with the prefix and the DNS server you want to hand out. I’m using Google DNS here.

np@NP-FW01# show routing-instances VRF-Edge access address-assignment
pool vlan311-dhcpv6 {
    family inet6 {
        prefix 2a02:1234:420a:1::/64;
        dhcp-attributes {
            dns-server {
                2001:4860:4860::8888;
                2001:4860:4860::8844;
            }
        }
    }
}

Next, configure the internal DHCPv6 server under system services, referring to the pool you just created – add it under routing-instances if required!

np@NP-FW01# show routing-instances VRF-Edge system services dhcp-local-server
dhcpv6 {
    overrides {
        interface-client-limit 100;
        process-inform {
            pool vlan311-dhcpv6;
        }
    }
    group ipv6 {
        interface ae0.311;
    }
}
group group-vlan311 {
    interface ae0.311;
}

Activate Router Advertisements for your outside interface. It doesn’t seem to work for me without this.

[edit]
np@NP-FW01# show protocols router-advertisement
interface pp0.0;

The following bit controls the Router Advertisements on the delegated interface and activates the O-Flag (Other Information) on the RAs.

[edit interfaces pp0 unit 0 family inet6 dhcpv6-client update-router-advertisement interface ae0.311]
+         other-stateful-configuration;
+         max-advertisement-interval 6;
+         min-advertisement-interval 3;

After adding this configuration, we see the client has now successfully picked up the new DNS settings.

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : netprobe.guest
   Description . . . . . . . . . . . : Intel(R) Ethernet Connection (4) I219-V
   Physical Address. . . . . . . . . : C8-5B-76-BC-04-6E
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2a02:1234:420a:1:103f:efd8:f2f7:d0cf(Preferred)
   Temporary IPv6 Address. . . . . . : 2a02:1234:420a:1:488c:f549:1e67:254c(Preferred)
   Link-local IPv6 Address . . . . . : fe80::103f:efd8:f2f7:d0cf%14(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.200.13(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : zondag 12 november 2017 15:23:41
   Lease Expires . . . . . . . . . . : maandag 13 november 2017 17:08:57
   Default Gateway . . . . . . . . . : fe80::aad0:e501:37d3:2a80%14
                                       192.168.200.1
   DHCP Server . . . . . . . . . . . : 192.168.200.1
   DHCPv6 IAID . . . . . . . . . . . : 63462262
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-20-35-58-1B-C8-5B-76-BC-04-6E
   DNS Servers . . . . . . . . . . . : 2001:4860:4860::8888
                                       2001:4860:4860::8844
                                       208.67.220.123
                                       208.67.222.123
   NetBIOS over Tcpip. . . . . . . . : Enabled

After adding a security policy that allows IPv6 traffic, the client can succesfully communicate over IPv6.

C:\Users\Simon>nslookup - 2001:4860:4860::8888
Default Server:  google-public-dns-a.google.com
Address:  2001:4860:4860::8888

> google.com
Server:  google-public-dns-a.google.com
Address:  2001:4860:4860::8888

Non-authoritative answer:
Name:    google.com
Addresses:  2a00:1450:400e:80b::200e
          172.217.20.110

C:\Users\Simon>ping -6 www.google.com

Pinging www.google.com [2a00:1450:400e:806::2004] with 32 bytes of data:
Reply from 2a00:1450:400e:806::2004: time=30ms
Reply from 2a00:1450:400e:806::2004: time=31ms
Reply from 2a00:1450:400e:806::2004: time=31ms
Reply from 2a00:1450:400e:806::2004: time=30ms

Ping statistics for 2a00:1450:400e:806::2004:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 30ms, Maximum = 31ms, Average = 30ms

Verification Commands

Checking which prefix was allocated to you by the ISP – don’t forget to specify the routing-instance if you are using them.

np@NP-FW01> show dhcpv6 client binding routing-instance VRF-Edge

IP/prefix                       Expires     State      ClientType    Interface       Client DUID
2a02:1234:420a::/48              2590728     BOUND      STATEFUL      pp0.0           LL0x29-a8:d0:e5:d3:2a:47

You can renew the prefix lease this way:

np@NP-FW01> request dhcpv6 client renew routing-instance VRF-Edge interface pp0.0

Displaying the IPv6 neighbours – this is similar to your ARP table in IPv4.

np@NP-FW01> show ipv6 neighbors
IPv6 Address                 Linklayer Address  State       Exp Rtr Secure Interface
2a02:1234:420a:1:488c:f549:1e67:254c
                             c8:5b:76:bc:04:6e  stale       1017 no no      ae0.311
fe80::103f:efd8:f2f7:d0cf    c8:5b:76:bc:04:6e  stale       363 no  no      ae0.311

Statistics for the DHCPv6 server:

np@NP-FW01> show dhcpv6 server statistics routing-instance VRF-Edge
Dhcpv6 Packets dropped:
    Total               0

Messages received:
    DHCPV6_DECLINE             1
    DHCPV6_SOLICIT             6
    DHCPV6_INFORMATION_REQUEST 60
    DHCPV6_RELEASE             0
    DHCPV6_REQUEST             3
    DHCPV6_CONFIRM             0
    DHCPV6_RENEW               0
    DHCPV6_REBIND              0
    DHCPV6_RELAY_FORW          0
    DHCPV6_RELAY_REPL          0

Messages sent:
    DHCPV6_ADVERTISE           4
    DHCPV6_REPLY               8
    DHCPV6_RECONFIGURE         0
    DHCPV6_RELAY_REPL          0

Now, if I was running a fairly simple network with just a few subnets, I would be using Prefix Delegation but I also have a couple of L3 devices behind this firewall. Extending PD all the way down would make it overly complex, so I’ll go for static configuration with OSPFv3 in the next article.

Please let me know in the comments if this configuration did not work for you!

Virtual Chassis on EX2200 switches

The Juniper Virtual Chassis technology allows you to combine multiple physical switches into one logical switch stack, which reduces the management overhead of dealing with many switches. Because all members are acting as a single device, with a proprietary control protocol underneath, there is no need for Spanning Tree and its blocked links. It also has dual routing engine support, albeit with some feature limitations on the EX2200 platform.

Unlike the high-end Juniper switches, the EX2200 does not have dedicated virtual-chassis ports either, so you are limited to using the on-board ethernet ports. The cross-member bandwidth should be OK for basic access deployments though, so here’s how to set it up.

First, gather the serial numbers of all your switches. There are several commands that will show you the serial. I’ve used show chassis hardware

root> show chassis hardware
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                CW0210160462      EX2200-24T-4G
Routing Engine 0 REV 11   750-026468   CW0210160462      EX2200-24T-4G
FPC 0            REV 11   750-026468   CW0210160462      EX2200-24T-4G
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           24x 10/100/1000 Base-T
  PIC 1          REV 11   750-026468   CW0210160462      4x GE SFP
Power Supply 0                                           PS 100W AC
Fan Tray                                                 Fan Tray

Do this for all your switches and copy them to notepad. You will need them later. Mine are indicated on the diagram below.

EX2200 Virtual Chassis Topology

Next, we need to configure our Ethernet ports for virtual-chassis functionality. The EX2200 does not have dedicated VCP ports so we need to convert some ethernet ports. As shown on the drawing, I will use the last four ports on each switch. To activate the ports for VC functionality, enter the following commands.

{master:0}
root> request virtual-chassis vc-port set pic-slot 0 port 20

{master:0}
root> request virtual-chassis vc-port set pic-slot 0 port 21

{master:0}
root> request virtual-chassis vc-port set pic-slot 0 port 22

{master:0}
root> request virtual-chassis vc-port set pic-slot 0 port 23

Rinse and repeat on switch two and three. You can verify the ports were added correctly to the virtual chassis with the following command.

root> show virtual-chassis vc-port
fpc0:
--------------------------------------------------------------------------
Interface   Type              Trunk  Status       Speed        Neighbor
or                             ID                 (mbps)       ID  Interface
PIC / Port
0/20        Configured         -1    Down         1000
0/21        Configured         -1    Down         1000
0/22        Configured         -1    Down         1000
0/23        Configured         -1    Down         1000

Before we can move on to connecting the cables, we have to set up the virtual chassis parameters on our primary routing engine.

We will set up the virtual-chassis in pre-provisioned mode, where we manually tie the serial ids to the member id. The first two switches will be configured as routing engines, one master and one backup, and the last switch will serve as a dumb old line-card. In my case, the virtual chassis configuration on routing engine 0 looks like this:

{master:0}[edit virtual-chassis]
root# show
preprovisioned;
member 0 {
    role routing-engine;
    serial-number CW0210160462;
}
member 1 {
    role routing-engine;
    serial-number CW0210300705;
}
member 2 {
    role line-card;
    serial-number CW0210300694;
}

Now that this is set up, I have connected the cables and the virtual chassis has formed. You can verify VC state and inter-switch connectivity using the show virtual-chassis status command.

root> show virtual-chassis status

Preprovisioned Virtual Chassis
Virtual Chassis ID: fd81.5b7b.172c
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    CW0210160462 ex2200-24t-4g 129 Master*   NA  1  vcp-255/0/20
                                                                 1  vcp-255/0/21
                                                                 2  vcp-255/0/22
                                                                 2  vcp-255/0/23
1 (FPC 1)  Prsnt    CW0210300705 ex2200-24t-4g 129 Backup    NA  2  vcp-255/0/20
                                                                 2  vcp-255/0/21
                                                                 0  vcp-255/0/22
                                                                 0  vcp-255/0/23
2 (FPC 2)  Prsnt    CW0210300694 ex2200-24t-4g   0 Linecard  NA  0  vcp-255/0/20
                                                                 0  vcp-255/0/21
                                                                 1  vcp-255/0/22
                                                                 1  vcp-255/0/23

This command shows us the IDs, serials and roles of all the members. The far-right columns also show on which ports the neighbours are detected. You can use this to verify everything is properly cabled up.

A quick terse command also shows us all interfaces in the stack are available, except for the ones we dedicated as VCP ports.

root> show interfaces terse | no-more
Interface               Admin Link Proto    Local                 Remote
vcp-255/0/20            up    up
vcp-255/0/20.32768      up    up
vcp-255/0/21            up    up
vcp-255/0/21.32768      up    up
vcp-255/0/22            up    up
vcp-255/0/22.32768      up    up
vcp-255/0/23            up    up
vcp-255/0/23.32768      up    up
ge-0/0/0                up    down
ge-0/0/0.0              up    down eth-switch
ge-0/0/1                up    down
ge-0/0/1.0              up    down eth-switch
ge-0/0/2                up    down
ge-0/0/2.0              up    down eth-switch
ge-0/0/3                up    down
ge-0/0/3.0              up    down eth-switch
ge-0/0/4                up    down
ge-0/0/4.0              up    down eth-switch
ge-0/0/5                up    down
ge-0/0/5.0              up    down eth-switch
ge-0/0/6                up    down
ge-0/0/6.0              up    down eth-switch
ge-0/0/7                up    down
ge-0/0/7.0              up    down eth-switch
ge-0/0/8                up    down
ge-0/0/8.0              up    down eth-switch
ge-0/0/9                up    down
ge-0/0/9.0              up    down eth-switch
ge-0/0/10               up    down
ge-0/0/10.0             up    down eth-switch
ge-0/0/11               up    down
ge-0/0/11.0             up    down eth-switch
ge-0/0/12               up    down
ge-0/0/12.0             up    down eth-switch
ge-0/0/13               up    down
ge-0/0/13.0             up    down eth-switch
ge-0/0/14               up    down
ge-0/0/14.0             up    down eth-switch
ge-0/0/15               up    down
ge-0/0/15.0             up    down eth-switch
ge-0/0/16               up    down
ge-0/0/16.0             up    down eth-switch
ge-0/0/17               up    down
ge-0/0/17.0             up    down eth-switch
ge-0/0/18               up    down
ge-0/0/18.0             up    down eth-switch
ge-0/0/19               up    down
ge-0/0/19.0             up    down eth-switch
ge-1/0/0                up    down
ge-1/0/0.0              up    down eth-switch
ge-1/0/1                up    down
ge-1/0/1.0              up    down eth-switch
ge-1/0/2                up    down
ge-1/0/2.0              up    down eth-switch
ge-1/0/3                up    down
ge-1/0/3.0              up    down eth-switch
ge-1/0/4                up    down
ge-1/0/4.0              up    down eth-switch
ge-1/0/5                up    down
ge-1/0/5.0              up    down eth-switch
ge-1/0/6                up    down
ge-1/0/6.0              up    down eth-switch
ge-1/0/7                up    down
ge-1/0/7.0              up    down eth-switch
ge-1/0/8                up    down
ge-1/0/8.0              up    down eth-switch
ge-1/0/9                up    down
ge-1/0/9.0              up    down eth-switch
ge-1/0/10               up    down
ge-1/0/10.0             up    down eth-switch
ge-1/0/11               up    down
ge-1/0/11.0             up    down eth-switch
ge-1/0/12               up    down
ge-1/0/12.0             up    down eth-switch
ge-1/0/13               up    down
ge-1/0/13.0             up    down eth-switch
ge-1/0/14               up    down
ge-1/0/14.0             up    down eth-switch
ge-1/0/15               up    down
ge-1/0/15.0             up    down eth-switch
ge-1/0/16               up    down
ge-1/0/16.0             up    down eth-switch
ge-1/0/17               up    down
ge-1/0/17.0             up    down eth-switch
ge-1/0/18               up    down
ge-1/0/18.0             up    down eth-switch
ge-1/0/19               up    down
ge-1/0/19.0             up    down eth-switch
ge-2/0/0                up    down
ge-2/0/0.0              up    down eth-switch
ge-2/0/1                up    down
ge-2/0/1.0              up    down eth-switch
ge-2/0/2                up    down
ge-2/0/2.0              up    down eth-switch
ge-2/0/3                up    down
ge-2/0/3.0              up    down eth-switch
ge-2/0/4                up    down
ge-2/0/4.0              up    down eth-switch
ge-2/0/5                up    down
ge-2/0/5.0              up    down eth-switch
ge-2/0/6                up    down
ge-2/0/6.0              up    down eth-switch
ge-2/0/7                up    down
ge-2/0/7.0              up    down eth-switch
ge-2/0/8                up    up
ge-2/0/8.0              up    up   eth-switch
ge-2/0/9                up    down
ge-2/0/9.0              up    down eth-switch
ge-2/0/10               up    down
ge-2/0/10.0             up    down eth-switch
ge-2/0/11               up    down
ge-2/0/11.0             up    down eth-switch
ge-2/0/12               up    down
ge-2/0/12.0             up    down eth-switch
ge-2/0/13               up    down
ge-2/0/13.0             up    down eth-switch
ge-2/0/14               up    down
ge-2/0/14.0             up    down eth-switch
ge-2/0/15               up    down
ge-2/0/15.0             up    down eth-switch
ge-2/0/16               up    down
ge-2/0/16.0             up    down eth-switch
ge-2/0/17               up    down
ge-2/0/17.0             up    down eth-switch
ge-2/0/18               up    down
ge-2/0/18.0             up    down eth-switch
ge-2/0/19               up    down
ge-2/0/19.0             up    down eth-switch
bme0                    up    up
bme0.32768              up    up   inet     128.0.0.1/2
                                            128.0.0.16/2
                                            128.0.0.32/2
                                   tnp      0x10
bme0.32770              down  up   eth-switch
bme0.32771              down  up   eth-switch
dsc                     up    up
gre                     up    up
ipip                    up    up
lo0                     up    up
lo0.16384               up    up   inet     127.0.0.1           --> 0/0
lsi                     up    up
me0                     up    down
me0.0                   up    down inet
mtun                    up    up
pimd                    up    up
pime                    up    up
tap                     up    up
vlan                    up    up
vlan.0                  up    up   inet     192.168.1.1/24
vme                     up    down

We can login to other chassis member for troubleshooting by using the following command. When you log in to a line card JunOS will alert you that it shouldn’t be used for configuration.

root> request session member 2

--- JUNOS 12.3R11.2 built 2015-09-24 11:14:53 UTC
root@:LC:2%
root@:LC:2% cli
warning: This chassis is operating in a non-master role as part of a virtual-chassis (VC) system.
warning: Use of interactive commands should be limited to debugging and VC Port operations.
warning: Full CLI access is provided by the Virtual Chassis Master (VC-M) chassis.
warning: The VC-M can be identified through the show virtual-chassis status command executed at this console.
warning: Please logout and log into the VC-M to use CLI.
{linecard:2}
root>

The out of band management interface, on the back of the switches, can use the same shared IP address across all VC members. To do so, you need to configure the vme interface with the desired management IP. In the background, all member’s management ports (otherwise known as me interfaces) are added to a special management VLAN which is tied to the layer3 vme interface.

{master:0}[edit interfaces]
root# show vme
unit 0 {
    family inet {
        address 10.6.66.25/24;
    }
}

All physical management interfaces can be connected to the OOB management VLAN, and the virtual chassis can respond to ARP requests on all of them.

Note – the VME interface will not work if you already have conflicting configuration on the me interface.

Because we have two routing engines, it’s also recommended to configure the commit synchronize option. When the configuration is committed on the primary routing engine, it will copy over to the backup routing engine. Both routing engines will then validate and activate the new configuration during the commit operation.

{master:0}[edit]
root# set system commit synchronize

Under the hood, the switches are running VCCP, a Juniper proprietary protocol largely based on IS-IS. If you’re interested, you can verify much of the protocol state from the command line, but I won’t go into detail.

root> show virtual-chassis protocol ?
Possible completions:
  adjacency            Show virtual chassis adjacency database
  database             Show virtual chassis link-state database
  interface            Show virtual chassis protocol interface information
  route                Show virtual chassis routing table
  statistics           Show virtual chassis performance statistics

Quoting Juniper here, “most high availability features, including Graceful Routing Engine switchover (GRES), Nonstop bridging (NSB), Nonstop active routing (NSR), and Nonstop software upgrade (NSSU), are not supported on an EX2200 Virtual Chassis.” Therefore, I would not rely an EX2200 virtual chassis as a core switch HA solution.. It does make sense using EX2200 virtual chassis in the access layer though, for simplified management and cross-switch etherchannel connectivity.

Removing Virtual-Chassis Configuration

A switch that was previously member of a virtual chassis may give you some headache when you’re trying to use it as a standard switch again. For example, VC ports can not be returned to their original state, or you might see that it stays in the previous role, even though you’ve removed the configuration. Here’s what seems to work for me.

If you haven’t done so, delete all virtual-chassis configuration

{linecard:2}[edit]
root# delete virtual-chassis
root# commit
warning: Could not connect to fpc0 : Can't assign requested address
warning: Cannot connect to other RE, ignoring it
commit complete

Go into shell mode (start shell) and delete all virtual chassis configuration files.

root@:LC:0% cd /config/vchassis/
root@:LC:0% ls
vc.db                   vc.tlv.db.0             vclocal.conf.tlv
vc.param                vc.tlv.db.1             vclocal.conf.tlv.0
vc.tlv.db               vclocal.conf            vclocal.conf.tlv.1
root@:LC:0% rm *.*

Then, return to operational mode with the cli command and do a system reboot to flush all stale virtual-chassis configuration from memory.

After the reboot, the switch will be in the default state again

Amnesiac (ttyu0)

login: root
Password:

--- JUNOS 12.3R11.2 built 2015-09-24 11:14:53 UTC
root@:RE:0% cli
{master:0}
root> show virtual-chassis

Virtual Chassis ID: 7c29.6cd0.a3b3
Virtual Chassis Mode: Enabled
                                           Mstr           Mixed Neighbor List
Member ID  Status   Serial No    Model     prio  Role      Mode ID  Interface
0 (FPC 0)  Prsnt    CW0210300694 ex2200-24t-4g 128 Master*   NA

Member ID for next new member: 1 (FPC 1)

{master:0}
root> show virtual-chassis vc-port
fpc0:
--------------------------------------------------------------------------

{master:0}

Any questions or comments? Feel free to add them below!

JNCIS-ENT – OSPF LSA types on JunOS

Going through the JNCIS-ENT course, I realized that I had forgotten so much of the details about the different LSA types and all of its intricacies. Next to being able to answer trick exam questions, knowing the LSAs and their working helps a lot when dealing with complex OSPF problems or optimizing your network topologies. As I really needed a refresher for this exam and my upcoming NP refresh I created this simple topology and tried to document as much of my findings as possible.

Lab Topology

OSPF Topology

We have four routers interconnected via OSPF. The area configuration is as follows.

  • vMX1 – RID 1.1.1.1 – connected to area 12
  • vMX2 – RID 2.2.2.2 – connected to area 12 and backbone area 0
  • vMX3 – RID 3.3.3.3 – connected to backbone area 0 and area 34
  • vMX4 – RID 4.4.4.4 – connected to area 34

Below is the standard configuration that is running on router VMX2 – Router ID, area statements and active interfaces. The other routers have similar configuration for their respective interfaces and areas.

[edit]
root@vMX2# show
routing-options {
    router-id 2.2.2.2;
}
protocols {
    ospf {
        area 0.0.0.0 {
            interface ge-0/0/0.0;
            interface lo0.0 {
                passive;
            }
        }
        area 0.0.0.12 {
            interface ge-0/0/1.0;
        }
    }
}

For now, until we move on to the Type 7 LSA, all areas are configured as standard areas. Before we start, here is an output of the LSA database on router VMX2. As it’s connected to two areas, it’s keeping two LSDBs. We’ll be using this router’s database for the first set of LSAs.

root@vMX2> show ospf database

    OSPF database, Area 0.0.0.0
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router  *2.2.2.2          2.2.2.2          0x80000180  2034  0x22 0x4d8   48
Router   3.3.3.3          3.3.3.3          0x8000017f  2044  0x22 0x1ab6  48
Network  172.16.23.3      3.3.3.3          0x8000017b  1794  0x22 0x7651  32
Summary *1.1.1.1          2.2.2.2          0x80000070  1784  0x22 0x4a76  28
Summary  4.4.4.4          3.3.3.3          0x80000001    53  0x22 0x809f  28
Summary *172.16.12.0      2.2.2.2          0x8000017c  1034  0x22 0x539b  28
Summary  172.16.34.0      3.3.3.3          0x8000017d  1544  0x22 0x4093  28

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router   1.1.1.1          1.1.1.1          0x80000073   253  0x22 0xd03   48
Router  *2.2.2.2          2.2.2.2          0x80000074   784  0x22 0xe0b   36
Network *172.16.12.2      2.2.2.2          0x80000071   534  0x22 0xda10  32
Summary *2.2.2.2          2.2.2.2          0x80000071   284  0x22 0x10ac  28
Summary *3.3.3.3          2.2.2.2          0x80000071    34  0x22 0xebcb  28
Summary *4.4.4.4          2.2.2.2          0x80000001    52  0x22 0xa87a  28
Summary *172.16.23.0      2.2.2.2          0x80000070  2784  0x22 0xf4fb  28
Summary *172.16.34.0      2.2.2.2          0x80000070  2534  0x22 0x855f  28

LSA Type 1 – Router LSA

The first LSA type is generated by every router participating in OSPF and lists all of the router’s links together with their associated cost, as well the OSPF neigbhours it has inside the area. Every router will flood a Router LSA for each area it is active in.

Looking at our OSPF database from before, router 2.2.2.2 has received a router LSA from router 1.1.1.1 and router 3.3.3.3 (Adv Rtr). Those LSAs are only seen in their respective areas, which demonstrates that a router LSA never leaves an area. We can further drill down to display only routers LSAs with the following command.

root@vMX2> show ospf database router

    OSPF database, Area 0.0.0.0
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router  *2.2.2.2          2.2.2.2          0x80000181   149  0x22 0x2d9   48
Router   3.3.3.3          3.3.3.3          0x80000180    37  0x22 0x18b7  48

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router   1.1.1.1          1.1.1.1          0x80000073  1174  0x22 0xd03   48
Router  *2.2.2.2          2.2.2.2          0x80000074  1705  0x22 0xe0b   36

If we want to get more specific and view the contents of the actual LSA, we can do so by specifying the LSA ID and adding the extensive command. Let’s see what router 1.1.1.1 is telling us about itself.

root@vMX2> show ospf database router lsa-id 1.1.1.1 extensive

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router   1.1.1.1          1.1.1.1          0x80000073  1274  0x22 0xd03   48
  bits 0x0, link count 2
  id 172.16.12.2, data 172.16.12.1, Type Transit (2)
    Topology count: 0, Default metric: 1
  id 1.1.1.1, data 255.255.255.255, Type Stub (3)
    Topology count: 0, Default metric: 0
  Topology default (ID 0)
    Type: Transit, Node ID: 172.16.12.2
      Metric: 1, Bidirectional
  Aging timer 00:38:45
  Installed 00:21:13 ago, expires in 00:38:46, sent 3d 19:45:06 ago
  Last changed 00:21:13 ago, Change count: 4

The router is advertising its two links in the LSA (link count 2). The LSA identifier is the originating router’s RID. The first link is an OSPF type Transit where it has a neighbourship with router 172.16.12.2 (data). If we configured the interface-type to p2p, we would see this as the link type. The router also has a Stub link with prefix 1.1.1.1/32, which means that this network contains only one router.

LSA Type 2 – Network LSA

The second LSA type is the Network LSA, which is generated by the Designated Router on a multi-access (broadcast or non-broadcast) segment. It lists all of the routers connected to the segment. It is generated only by the Designated Router to prevent duplicate LSAs.

Our router 2.2.2.2 has received the following network LSAs in area 12:

root@vMX2> show ospf database area 0.0.0.12 network

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Network *172.16.12.2      2.2.2.2          0x8000001b  2085  0x22 0x87b9  32

Drilling down on this particular LSA, it contains the following information:

root@vMX2> ... 0.0.0.12 network lsa-id 172.16.12.2 extensive

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Network *172.16.12.2      2.2.2.2          0x8000001b  2099  0x22 0x87b9  32
  mask 255.255.255.0
  attached router 2.2.2.2
  attached router 1.1.1.1
  Topology default (ID 0)
    Type: Transit, Node ID: 1.1.1.1
      Metric: 0, Bidirectional
    Type: Transit, Node ID: 2.2.2.2
      Metric: 0, Bidirectional
  Gen timer 00:15:01
  Aging timer 00:25:01
  Installed 00:34:59 ago, expires in 00:25:01, sent 00:34:57 ago
  Last changed 22:03:21 ago, Change count: 1, Ours

From this LSA, we can tell that the network has a /24 mask and that it has router 1.1.1.1 and 2.2.2.2 attached to it. It does not include a cost or metric for the routers, because from the network’s point of view, the cost to the attached routers is zero. If we wanted to know more information about a particular router, we can use the attached router ID which in turn refers to the router LSA we looked at earlier.

LSA Type 3 – Network Summary

This LSA type is generated by Area Border Routers and flooded into a particular area to represent destinations outside of the area. In the opposite direction, the ABR will also advertise prefixes from a non-backbone area back into backbone area 0 with a network LSA. They are flooded into areas where the destination prefix was not found. One thing worth noting is that this only counts for inter-area destinations learned inside the OSPF AS. Routes external to OSPF will use a different LSA type which I’ll cover later.

Again, we will view the contents of router 2.2.2.2’s database for area 12.

root@vMX2> show ospf database area 0.0.0.12 netsummary

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Summary *2.2.2.2          2.2.2.2          0x800000ae  2289  0x22 0x95e9  28
Summary *3.3.3.3          2.2.2.2          0x800000ae  1689  0x22 0x7109  28
Summary *4.4.4.4          2.2.2.2          0x8000003e  1989  0x22 0x2eb7  28
Summary *172.16.23.0      2.2.2.2          0x800000ae  1389  0x22 0x783a  28
Summary *172.16.34.0      2.2.2.2          0x800000ae  1089  0x22 0x99d   28

Router 2.2.2.2, as the area border router, has generated these LSAs itself (*) and flooded them into area 12. Comparing this to the diagram and the full LSA database, we can tell that it has created network summaries for Router, Network and Summary LSAs inside area 0.

root@vMX2>show ospf database area 12 netsummary lsa-id 172.16.34.0 extensive

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Summary *172.16.34.0      2.2.2.2          0x800000ae  1527  0x22 0x99d   28
  mask 255.255.255.0
  Topology default (ID 0) -> Metric: 2
  Gen timer 00:24:33
  Aging timer 00:34:33
  Installed 00:25:27 ago, expires in 00:34:33, sent 00:25:25 ago
  Last changed 5d 22:58:56 ago, Change count: 2, Ours

The LSA ID matches the destination network’s prefix. In this case, router R2 knows about the network 172.16.34.0/24 and added a cost of 2 to reach it. Router 1.1.1.1 will add this value to its own cost to ABR 2.2.2.2 to determine the ultimate cost for the route.

LSA Type 4 – ASBR Summary

When a router is advertising routes from sources external to the OSPF AS, it is called an Autonomous System Border Router (ASBR). Like any other router, it will generate a router LSA for itself (Type 1) into its own area but with a special flag E set. When this LSA is received by the other ABR, the router LSA will be converted into a Type 4 LSA when it is flooded into other areas. The LSA is flooded throughout the entire OSPF Autonomous System and lets other routers know how to reach the ASBR.

To demonstrate, I have configured router 4.4.4.4 to redistribute route 10.4.0.0/24 from its loopback interface as an external route. First, define the export policy as a policy-statement.

root@vMX4# show policy-options
policy-statement redist-direct {
    term term1 {
        from {
            protocol direct;
            interface lo0.0;
        }
        then accept;
    }
}

Then, configure OSPF to use that statement as its export policy.

root@vMX4# show protocols ospf
export redist-direct;
area 0.0.0.34 {
    interface ge-0/0/0.0;
}

We will now see an external route for 10.4.0.0/24, and an ASBR summary for 4.4.4.4 in the database on router 2.2.2.2

root@vMX2> show ospf database area 12

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router   1.1.1.1          1.1.1.1          0x800000b6  2563  0x22 0x8646  48
Router  *2.2.2.2          2.2.2.2          0x800000b8   559  0x22 0x854f  36
Network *172.16.12.2      2.2.2.2          0x8000001d   259  0x22 0x83bb  32
Summary *2.2.2.2          2.2.2.2          0x800000af  2359  0x22 0x93ea  28
Summary *3.3.3.3          2.2.2.2          0x800000af  1759  0x22 0x6f0a  28
Summary *172.16.23.0      2.2.2.2          0x800000af  1459  0x22 0x763b  28
Summary *172.16.34.0      2.2.2.2          0x800000af  1159  0x22 0x79e   28
ASBRSum *4.4.4.4          2.2.2.2          0x80000001   986  0x22 0x9a87  28
    OSPF AS SCOPE link state database
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Extern   10.4.0.0         4.4.4.4          0x80000001    99  0x22 0xd5be  36

Let’s see what’s inside the ASBR summary…

root@vMX2> ... 12 asbrsummary lsa-id 4.4.4.4 extensive

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
ASBRSum *4.4.4.4          2.2.2.2          0x80000001   177  0x22 0x9a87  28
  mask 0.0.0.0
  Topology default (ID 0) -> Metric: 2
  Gen timer 00:47:03
  Aging timer 00:57:03
  Installed 00:02:57 ago, expires in 00:57:03, sent 00:02:57 ago
  Last changed 00:02:57 ago, Change count: 1, Ours

The LSA ID is the router ID of the ASBR, and router 2.2.2.2 has a cost of 2 to reach it.

For completeness’ sake, the command below shows that the LSA was originally a Type 1 in area 34 before being converted into an ASBR summary by router 3.3.3.3, when flooding it into area 0.

root@vMX3> show ospf database lsa-id 4.4.4.4

    OSPF database, Area 0.0.0.0
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
ASBRSum *4.4.4.4          3.3.3.3          0x80000001  1090  0x22 0x72ac  28

    OSPF database, Area 0.0.0.34
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router   4.4.4.4          4.4.4.4          0x800000b8  1091  0x22 0x68d   36

LSA Type 5 – AS External LSA

When the ASBR imports routes from outside the Autonomous System, either through static or protocol redistribution, it will flood them into its area as AS External LSAs. This type of LSA is flooded throughout the OSPF topology except for stub areas.

Router 2.2.2.2 has received the external route from ASBR router 4.4.4.4.

root@vMX2> show ospf database area 12 external
    OSPF AS SCOPE link state database
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Extern   10.4.0.0         4.4.4.4          0x80000001   577  0x22 0xd5be  36

The original ASBR’s RID is preserved in the Advertising Router field.

root@vMX2> show ospf database area 12 lsa-id 10.4.0.0 extensive
    OSPF AS SCOPE link state database
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Extern   10.4.0.0         4.4.4.4          0x80000001   675  0x22 0xd5be  36
  mask 255.255.255.0
  Topology default (ID 0)
    Type: 2, Metric: 0, Fwd addr: 0.0.0.0, Tag: 0.0.0.0
  Aging timer 00:48:44
  Installed 00:11:13 ago, expires in 00:48:45, sent 00:11:13 ago
  Last changed 00:11:13 ago, Change count: 1

Router 2.2.2.2 has now learned about the external route 10.4.0.0/24 and will need to recursively use ASBR 4.4.4.4 to reach it. Because the route was injected with the default Type E2, inter-area OSPF link cost is not taken into consideration, and the associated cost is zero as injected by router 4.4.4.4. Before we wrap up on this LSA type, the external route types might need some clarification.

E1 and E2 type routes

External routes or Type 5 LSAs can be imported as either Type 1 or Type 2 routes. When injecting as E2 routes, the autonomous system’s internal cost metrics are not taken into consideration when the LSA is flooded throughout the topology. This the default behaviour and it’s fine in simple stub topologies, but as OSPF is by design a cost-based protocol, using E1 routes makes more sense. This will make the routers combine the route’s original metric with the cost to reach the ASBR (Type 4 LSA), resulting in the total route cost. Let’s demonstrate…

The external route type is configured in the routing-policy. Here’s an example of my export policy before making any changes.

[edit policy-options policy-statement redist-direct]
root@vMX4# show
term term1 {
    from {
        protocol direct;
        interface lo0.0;
    }
    then accept;
}

Now, if we want to set the route as an E1, we define this an action in the policy term. I will import the route with a default cost of 5 and set them as type E1.

[edit policy-options policy-statement redist-direct]
root@vMX4# set term term1 then external type 1

[edit policy-options policy-statement redist-direct]
root@vMX4# set term term1 then metric 5

[edit policy-options policy-statement redist-direct]
root@vMX4# show
term term1 {
    from {
        protocol direct;
        interface lo0.0;
    }
    then {
        metric 5;
        external {
            type 1;
        }
        accept;
    }
}

After a commit, we have the following LSA in our database.

root@vMX2> show ospf database external lsa-id 10.4.0.0 extensive
    OSPF AS SCOPE link state database
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Extern   10.4.0.0         4.4.4.4          0x8000003d    86  0x22 0xcc7   36
  mask 255.255.255.0
  Topology default (ID 0)
    Type: 1, Metric: 5, Fwd addr: 0.0.0.0, Tag: 0.0.0.0
  Aging timer 00:58:33
  Installed 00:01:24 ago, expires in 00:58:34, sent 00:01:24 ago
  Last changed 00:01:24 ago, Change count: 4

The same route is now received as E1 (Type: 1). The cost was injected with a cost of 5 but when we validate in the RIB, we see it has a metric of 7 for the route.

root@vMX2> show route 10.4.0.0 | match metric
10.4.0.0/24        *[OSPF/150] 00:02:09, metric 7, tag 0

This confirms that Total Cost = Cost of Route (5) + Cost to ASBR (2)

LSA Type 7 – NSSA External

When an area is configured as a stub, external routes or Type 5 LSAs will not be allowed in and commonly replaced by one default route. This reduces the LSDB size on the stub routers and keeps the topology simple, but there might be cases where you do need external routes to originate in those stub areas. For these cases, the Not-So-Stubby Area was designed. This allows you to place an ASBR in the stub area and still import external routes, but this time with a special LSA type. When it traverses to the backbone area, the stub-to-backbone ABR will convert the NSSA External LSA into a standard Type-5 External LSA.

To demonstrate, I have converted area 34 to a NSSA on both router 3 and router 4.

root@vMX4# show
export redist-direct;
area 0.0.0.34 {
    nssa;
    interface ge-0/0/0.0;
}

The Type 5 LSA that was previously flooded by router 4.4.4.4 will now be shown as an NSSA External LSA.

root@vMX4> show ospf database

    OSPF database, Area 0.0.0.34
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Router   3.3.3.3          3.3.3.3          0x80000005   227  0x20 0xcc84  36
Router  *4.4.4.4          4.4.4.4          0x80000004   226  0x20 0x8dbc  36
Network *172.16.34.4      4.4.4.4          0x80000002   226  0x20 0x3dee  32
Summary  1.1.1.1          3.3.3.3          0x80000002   227  0x20 0x31fa  28
Summary  2.2.2.2          3.3.3.3          0x80000002   227  0x20 0xf830  28
Summary  3.3.3.3          3.3.3.3          0x80000002   227  0x20 0xc065  28
Summary  172.16.12.0      3.3.3.3          0x80000002   227  0x20 0x5512  28
Summary  172.16.23.0      3.3.3.3          0x80000002   227  0x20 0xd18b  28
NSSA    *10.4.0.0         4.4.4.4          0x80000002   226  0x28 0x2cf7  36

On the last line, we have our Type 7 LSA. Inside it we find the following information…

root@vMX4> show ospf database lsa-id 10.4.0.0 extensive

    OSPF database, Area 0.0.0.34
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
NSSA    *10.4.0.0         4.4.4.4          0x80000002   262  0x28 0x2cf7  36
  mask 255.255.255.0
  Topology default (ID 0)
    Type: 1, Metric: 5, Fwd addr: 172.16.34.4, Tag: 0.0.0.0
  Gen timer 00:45:37
  Aging timer 00:55:37
  Installed 00:04:22 ago, expires in 00:55:38, sent 00:04:22 ago
  Last changed 00:04:22 ago, Change count: 2, Ours

The LSA contents are almost identical to the previous one, except that the router’s LAN IP is now in the Fwd address field.

Hopping back to router 2, we see the same prefix has been received as an External, Type 5, LSA.

root@vMX2> show ospf database lsa-id 10.4.0.0 extensive
    OSPF AS SCOPE link state database
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
Extern   10.4.0.0         3.3.3.3          0x80000002   715  0x22 0xc06f  36
  mask 255.255.255.0
  Topology default (ID 0)
    Type: 1, Metric: 5, Fwd addr: 172.16.34.4, Tag: 0.0.0.0
  Aging timer 00:48:05
  Installed 00:11:52 ago, expires in 00:48:05, sent 00:11:50 ago
  Last changed 00:12:16 ago, Change count: 1

Interestingly, the advertising router is no longer 4.4.4.4 but 3.3.3.3, as this was the one that generated the Type 5 LSA. Because this router is now also acting as an ASBR, it advertised its own Router LSA with the E bit set which is into an ASBR summary LSA by router 2.2.2.2 when it is sent into area 12.

root@vMX2> show ospf database asbrsummary

    OSPF database, Area 0.0.0.12
 Type       ID               Adv Rtr           Seq      Age  Opt  Cksum  Len
ASBRSum *3.3.3.3          2.2.2.2          0x80000001  1006  0x22 0xbe68  28

From the perspective of the other routers, this device is now the source of the external route and the same logic we saw earlier is applied.

There you go, some of the things I’ve learned labbing LSAs. If you have anything to add, or spotted some inaccuracy, I’m always happy to hear in the comments!

Consistent software versions on dual-partition JunOS devices

Modern Junos devices have dual boot partitions, each with their own copy of the operating system. This ensures that the device will boot if storage or other boot-related issues are detected on the primary boot partition.

However, when you manually update the software, it is only updated on one of the partitions which is then set as the boot partition. The second, original, partition will keep running on the previous version until it is manually updated as well. This might be an advantage if you are testing a new software version and want to quickly roll back to the old one. It could also wreak havoc if you are inadvertently falling back to a pre-historic software version that is missing or not compatible with some of the features you’ve since enabled. If you have found a stable software version, it’s best to keep both partitions in sync. Here’s how to do it…

Identifying a software mismatch

The first hint is given at boot time. Right before the login prompt and banner, the console will alert you of the software version disparity.

kern.securelevel: -1 -> 1
Creating JAIL MFS partition...
JAIL MFS partition created
Boot media /dev/ad0 has dual root support


WARNING: JUNOS versions running on dual partitions are not same


** /dev/ad0s1a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 242154 free (34 frags, 30265 blocks, 0.0% fragmentation)

You can also find out from operational mode. The command below will show you the software versions on both of the partitions.

netprobe@netfilter> show system snapshot media internal
Information for snapshot on       internal (/dev/ad0s1a) (backup)
Creation date: Jul 31 11:13:12 2014
JUNOS version on snapshot:
  junos  : 12.1X44-D35.5-domestic
Information for snapshot on       internal (/dev/ad0s2a) (primary)
Creation date: Mar 4 19:53:11 2016
JUNOS version on snapshot:
  junos  : 12.1X46-D40.2-domestic

As you can see from the output, we are currently running on partition /dev/ad0s2a, which has a newer software version than the first partition /dev/ad0s1a.

Cloning the primary partition

To get the software version from our now-primary partition over to the backup, the system will first format the backup partition and then clone the contents. This process is initiated with the command below.

netprobe@netfilter> request system snapshot slice alternate
Formatting alternate root (/dev/ad0s1a)...
Copying '/dev/ad0s2a' to '/dev/ad0s1a' .. (this may take a few minutes)
The following filesystems were archived: /

After the cloning process you might need to reboot the device, depending on the model. If all went well, you will no longer see the warning prompt for version mismatch:

Creating JAIL MFS partition...
JAIL MFS partition created
Boot media /dev/ad0 has dual root support
** /dev/ad0s1a
FILE SYSTEM CLEAN; SKIPPING CHECKS

And voila, the snapshot command will now show the same SW version on both partitions.

netprobe@netfilter> show system snapshot media internal
Information for snapshot on       internal (/dev/ad0s1a) (backup)
Creation date: Mar 4 21:45:37 2016
JUNOS version on snapshot:
  junos  : 12.1X46-D40.2-domestic
Information for snapshot on       internal (/dev/ad0s2a) (primary)
Creation date: Mar 4 21:48:51 2016
JUNOS version on snapshot:
  junos  : 12.1X46-D40.2-domestic

On EX switches, you can alternate between boot partitions by entering this command.

request system reboot slice alternate media internal

Unfortunately this doesn’t seem to work on SRX devices, at least not on the branch devices I’ve worked with so far. If anyone knows how to make this work on these SRXs I would love to hear about it!

JNCIS-SEC – Exam JN0-332 passed!

Can’t believe it has already been two months since I last posted on my blog… The last two weeks of October I had to go into full-on study mode for the JN0-332 exam so I had to pause on the write-ups for a while. Fortunately the hard work paid off and after a good six months of study and labbing, with a hot summer in between, I sat the JN0-332 on the 26th of October and passed it with 83 percent.

The exam itself was very fair and covered all the topics on the blueprint and in the freely available Juniper Study Guides. I have sat a few “other vendor” exams where multiple questions caught me off-guard and were definitely not included in the related cert guides. Not really the case with this Juniper exam though, so hats off to Juniper for delivering a consistent test.

Another great thing is that Juniper lets you go back through the questions when you’re done. You can also mark questions, which you’re uncertain about, for review. When you’re done, and if you still have some time on the counter, you can run through it once more to fill in the gaps. Or perhaps one of the later questions made you rethink your answer on an earlier question. I’m certain this approach will earn test-takers some points.

The Score Report is pretty straight-forward. It breaks the exam down into the main topics, the total number of questions, number of correct answers and your total percentage. My worst scores were on Firewall User Authentication, which to be fair only had a couple of questions, and UTM, which I couldn’t fully simulate in my SRX100 and vSRX lab. Fair to say I wasn’t surprised with the outcome, but still very satisfied overall.

For anyone interested in the JNCIS-SEC certification, here is what I used to pass the exam…

JNCIS-SEC Study Guides

Probably the most important source of information for this exam, Juniper is offering free Study Guides for all Specialist tracks on their site. You will need an account but registration is free so there really is no excuse. Follow this link

The first PDF covers SRX basics, Security Zones, Policies, User Authentication, NAT, IPsec and Clustering. The second PDF is dedicated to the UTM features.

Juniper SRX Series – By Brad Woodberg, Rob Cameron

Juniper SRX Series - O'Reilly

I bought this book early on, when I first encountered the SRX at a new job. Weighing in at about 1000 pages, it’s the perfect reference for anyone dealing with the SRX on a daily basis. It’s not something you’ll read front to back though, and I’ve found myself reading through the chapters for whatever feature I need at a certain point. For example, the chapter on Screens gives you an in-depth review of each of the features, the attacks for which they were written, and so on. Highly recommended! I’ve also found that you can read the book online

Juniper vSRX – Firefly

The virtual edition of the SRX firewall. You can run a trial version on your Hypervisor and even try the Advanced Features for 60 days. More information here.

Three SRX100Bs

I bought three of these boxes for cheap on eBay. They don’t have the high memory so don’t support UTM or IPS, but they are great for configurations that are hard to do on the virtual appliance, like aggregated interfaces and clustering. Added bonus is that they also support routing and switching so they can be used for the ENT track also.

Junos Genius app

The official Juniper app for JNCIA and Specialist level exams. Whenever I had a few minutes off I would go through some practice questions. Very good to keep your mind on the content and memorise some of the technical details. I just wish they had a PC edition! 🙂

JUNOS GENIUS – Android version
JUNOS GENIUS – IOS version

Next challenge?

For a few weeks I was working on the Brocade Certified vRouter Engineer certification. I already touched some Vyatta/VyOS routers so figured I might as well try the free exam. Unfortunately, when I tried to book the exam, I found that the Brocade voucher had expired. I tried mailing Brocade but to no avail – they confirmed that the promotion was no longer running. That sort of halted the BCVRE endeavour…

For a good two weeks now I’ve been going through the JNCIS-ENT study guides. Bought myself a couple EX switches which will be complementing the SRX and vSRX I already have in the lab. It should also be a good refresh of the routing & switching topics as my CCNP is up for renewal in December 2016. As I did for JNCIS-SEC, I will be writing up my ENT labs on the blog.

If you are interested in the JNCIS-SEC certification, and have any other questions, feel free to post them in the comments section. Thank you for reading!

Allowing inbound DHCP requests on a Cisco ZBFW

I came across an interesting one today, where a Cisco Zone-Based Firewall needed to be reconfigured to serve DHCP for a segment connected to it in a zone called “Guest”. It already had a policy-map configured for traffic from Guest to Self, which had ACLs for SSH management.

First I tried adding these two lines to that ACL, in the existing class-map

 permit udp any any eq bootpc
 permit udp any any eq bootps

Although I did see the ACL match counters increment, DHCP was not handing out addresses yet. A quick search led me to this page on the Cisco site. In the last paragraph, they state the following:

If the routers’ inside interface is acting as a DHCP server and if the clients that connect to the inside interface are the DHCP clients, this DHCP traffic is allowed by default if there is no inside-to-self or self-to-inside zone policy.
However, if either of those policies does exist, you need to configure a pass action for the traffic of interest (UDP port 67 or UDP port 68) in the zone pair service policy.

In my case, there was a policy configured but with the action set to inspect. To fix it, I had to add a new ACL and class-map to the Guest-Self policy-map.

New ACL that matches the DHCP traffic. The source and destination is set to any because of the DHCP request format.

ip access-list extended Guest-Self-DHCP-ACL
 permit udp any any eq bootpc
 permit udp any any eq bootps

Tie the ACL to a new inspect class map:

class-map type inspect match-any Guest-Self-DHCP-CMap
 match access-group name Guest-Self-DHCP-ACL

And finally, add the class-map to the policy-map with the pass action

policy-map type inspect Guest-Self-PMap
 class type inspect Guest-Self-CMap
  inspect
 class type inspect Guest-Self-DHCP-CMap
  pass log
 class class-default
  drop

After that the clients started receciving IP addresses again.

ZBFW-ROUTER#show ip dhcp binding
Bindings from all pools not associated with VRF:
IP address          Client-ID/              Lease expiration        Type
                    Hardware address/
                    User name
192.168.200.201     014d.970e.4136.af       Oct 21 2015 10:43 AM    Automatic

Juniper SRX Clustering with LACP

Most deployment guides for SRX clusters out there focus on standard two-port deployments, where you have one port in, one port out and a couple of cluster links that interconnect and control the cluster. Unfortunately, in that design, one simple link failure will usually make the cluster fail over. Coming from the R&S realm, I am very careful when it comes to physical redundancy so I wanted to figure out a way to get this working with Etherchannels.

After a lot of reading and some trial and error, I ended up with a working solution. Probably not perfect, but definitely more redundant. So, in this post we will get the topology below configured and afterwards do some failover testing.

Physical connections

Basic Cluster Setup

The commands below will get the basic cluster up and running, assuming you have already configured the cluster and node ids.

set system root-authentication encrypted-password "$1$FQl4d.NC$l25c0bDGr5aPq9ZHx0R.S."
set groups node0 system host-name FW01A
set groups node1 system host-name FW01B
set apply-groups "${node}"
set chassis cluster reth-count 2
set chassis cluster redundancy-group 0 node 0 priority 120
set chassis cluster redundancy-group 0 node 1 priority 1
set chassis cluster redundancy-group 1 node 0 priority 120
set chassis cluster redundancy-group 1 node 1 priority 1
set chassis cluster redundancy-group 1 preempt
set interfaces fab0 fabric-options member-interfaces fe-0/0/5
set interfaces fab1 fabric-options member-interfaces fe-1/0/5

Layer3 Etherchannel configuration on the SRX

The physical ports will be bundled in two reth interface. Per reth interface, we will add two physical ports per cluster node, which yields a total of four ports. However only two links will ever be actively forwarding traffic – those on the active RG.

Physical connections

It is important that the links from every cluster nodes are terminated on separate etherchannels on the switches. Otherwise the switches would also load-balance the traffic over the two non-forwarding ports, as documented here…
In my topology, both switches have a Port-Channel 11 and 12, going to their own cluster node.

{primary:node0}
root@FW01A> show configuration interfaces
fe-0/0/0 {
    fastether-options {
        redundant-parent reth0;
    }
}
fe-0/0/1 {
    fastether-options {
        redundant-parent reth0;
    }
}
fe-0/0/2 {
    fastether-options {
        redundant-parent reth1;
    }
}
fe-0/0/3 {
    fastether-options {
        redundant-parent reth1;
    }
}
fe-1/0/0 {
    fastether-options {
        redundant-parent reth0;
    }
}
fe-1/0/1 {
    fastether-options {
        redundant-parent reth0;
    }
}
fe-1/0/2 {
    fastether-options {
        redundant-parent reth1;
    }
}
fe-1/0/3 {
    fastether-options {
        redundant-parent reth1;
    }
}

Pay special attention to the configuration of the reth interfaces. First, we tie the redundant interfaces to our redundancy-group 1, in which we will later control the failover conditions.

The minimum-links command determines how many interfaces must be up before the LACP bundle is considered up. Although we have two interfaces per cluster, we still want traffic forwarded in a worst-case scenario. Setting minimum-links 1 will keep the Etherchannel up even in the unlikely event of having three physical ports down.

reth0 {
    vlan-tagging;
    redundant-ether-options {
        redundancy-group 1;
        minimum-links 1;
        lacp {
            active;
            periodic fast;
        }
    }
    unit 111 {
        vlan-id 111;
        family inet {
            address 1.1.1.1/28;
        }
    }
}
reth1 {
    vlan-tagging;
    redundant-ether-options {
        redundancy-group 1;
        minimum-links 1;
        lacp {
            active;
            periodic fast;
        }
    }
    unit 255 {
        vlan-id 255;
        family inet {
            address 10.255.255.254/28;
        }
    }
}

By adding the vlan-tagging command, and adding a logical subinterface (unit 255) with vlan-id 255 specified, we are creating a tagged L3 Etherchannel. In other words, the SRX is expecting packets for sub-interface reth1.255 to be tagged with a dot1q value of 255. The unit number can be any number you like, but it’s best to keep the unit number and the dot1q value aligned – for your own sanity! 🙂

Security Zones and Policies

To check basic connectivity and later run some failover tests, I have added the following configuration for the Security Zones, Policies and Source NAT. In a production environment, you probably won’t be allowing ping.

root@FW01A# show zones
security-zone trust {
    address-book {
        address Net-10.255.255.240-28 10.255.255.240/28;
    }
    host-inbound-traffic {
        system-services {
            ping;
        }
    }
    interfaces {
        reth1.255;
    }
}
security-zone untrust {
    address-book {
        address Net-1.1.1.0-28 1.1.1.0/28;
    }
    interfaces {
        reth0.111 {
            host-inbound-traffic {
                system-services {
                    ping;
                }
            }
        }
    }
}
{primary:node0}[edit security]
root@FW01A# show policies
from-zone trust to-zone untrust {
    policy Test-Policy {
        match {
            source-address Net-10.255.255.240-28;
            destination-address Net-1.1.1.0-28;
            application any;
        }
        then {
            permit;
        }
    }
}
{primary:node0}[edit security]
root@FW01A# show nat
source {
    rule-set SNAT-Trust-to-Untrust {
        from zone trust;
        to zone untrust;
        rule HideNAT-1 {
            match {
                source-address 10.255.255.240/28;
            }
            then {
                source-nat {
                    interface;
                }
            }
        }
    }
}

Preparing the switches for L2

I’m running two Cisco 3550 switches as my layer3 core switches. For now, I will just add the Layer 1 and 2 stuff.

vlan 111
 name untrust
!
vlan 255
 name transit


interface range fastEthernet 0/1 - 2
 channel-group 11 mode active

interface range fastEthernet 0/3 - 4
 channel-group 12 mode active

int po 11
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport nonegotiate
 switchport trunk allowed vlan 111

int po 12
 switchport mode trunk
 switchport nonegotiate
 switchport trunk allowed vlan 255

For CS2, we can copy paste the exact same config.

A second trunk link is added between the core switches, which will carry inter-switch traffic for VLAN 111 and 255.

interface FastEthernet0/23
 channel-group 1 mode active
!
interface FastEthernet0/24
 channel-group 1 mode active

...

interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 111,255
 switchport mode trunk
 switchport nonegotiate

The LACP configuration is now complete, and after cabling everything up we see the Port-channels are in the bundled state:

NP-CS1>show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 3
Number of aggregators:           3

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/23(P)   Fa0/24(P)
11     Po11(SU)        LACP      Fa0/1(P)    Fa0/2(P)
12     Po12(SU)        LACP      Fa0/3(P)    Fa0/4(P)

The SRX is bit more detailed with the information in the lacp command

root@FW01A> show lacp interfaces
Aggregated interface: reth0
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      fe-0/0/0       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      fe-0/0/0     Partner    No    No   Yes  Yes  Yes   Yes     Slow    Active
      fe-0/0/1       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      fe-0/0/1     Partner    No    No   Yes  Yes  Yes   Yes     Slow    Active
      fe-1/0/0       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      fe-1/0/0     Partner    No    No   Yes  Yes  Yes   Yes     Slow    Active
      fe-1/0/1       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      fe-1/0/1     Partner    No    No   Yes  Yes  Yes   Yes     Slow    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      fe-0/0/0                  Current   Slow periodic Collecting distributing
      fe-0/0/1                  Current   Slow periodic Collecting distributing
      fe-1/0/0                  Current   Slow periodic Collecting distributing
      fe-1/0/1                  Current   Slow periodic Collecting distributing

Layer3 Switch Configuration

Now that we have our layer 2 connectivity, we can move on to the IP addressing. On the L3 switches, I will configure the SVIs in the transit network, and run HSRP between them with hello/hold timers of 1 and 3 seconds. This would allow for a reasonable failover time in case of an outage.

NP-CS1:

interface Vlan255
 ip address 10.255.255.242 255.255.255.240
 standby 255 ip 10.255.255.241
 standby 255 timers 1 3
 standby 255 priority 110
 standby 255 preempt

NP-CS2:

NP-CS2#sh run int vlan 255
!
interface Vlan255
 ip address 10.255.255.243 255.255.255.240
 standby 255 ip 10.255.255.241
 standby 255 timers 1 3
 standby 255 preempt

The VLAN interfaces come up immediately and as a final test for basic L3 connectivity, we try pinging the SRX from the core switch.

NP-CS1#ping 10.255.255.254 repeat 10

Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 10.255.255.254, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/2/4 ms

Checking the ARP table, we can see the Cluster MAC address, with the 4th last digit (1) being the cluster ID and the last two (01) the reth number.

NP-CS1#show ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.255.255.254          1   0010.dbff.1001  ARPA   Vlan255
Internet  10.255.255.242          -   0012.7f81.8f00  ARPA   Vlan255
Internet  10.255.255.243          8   000d.ed6f.9680  ARPA   Vlan255
Internet  10.255.255.241          -   0000.0c07.acff  ARPA   Vlan255

The default route on both core switches is pointed to 10.255.255.254

ip route 0.0.0.0 0.0.0.0 10.255.255.254

Internet segment configuration

I have connected two routers to the fa0/5 interfaces of the switches and added them to vlan 111. They are running HSRP with a VIP of 1.1.1.14 and hello/hold timers of 1 and 3 seconds.

ISP1#show standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Fa0/1       111  110   Active  local           1.1.1.13        1.1.1.14

Running a simple ping to test connectivity between the firewalls and the ISP router.

root@FW01A> ping 1.1.1.14 count 1
PING 1.1.1.14 (1.1.1.14): 56 data bytes
64 bytes from 1.1.1.14: icmp_seq=0 ttl=255 time=2.981 ms

*snip*

To test if traffic is being forwarded across the firewall, we finally try reaching the ISP routers from the core switches.

NP-CS1#ping 1.1.1.14 source vlan 255

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.14, timeout is 2 seconds:
Packet sent with a source address of 10.255.255.242
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

After many bits of configuration, our cluster is live so we can start implementing some failover mechanisms.

Configuring the interface-monitor

When the interface-monitor is configured, it sets a threshold value of 255 to each redundancy group. Each interface is assigned a custom weight for the redundancy group, which is detracted from the threshold if the interface goes physically down. When the threshold reaches zero, the redundancy group and all its objects fail over.

Note – a commmon mistake (trust me) is to mix up the interface monitor weights with the RG priorities. These are two separate values, the interface monitor threshold is always 255 by default.

In my lab scenario, I will give each port a value of 128. Two physical links down will trigger a failover to node 1. This is the final configuration for redundancy group 1:

{primary:node0}[edit chassis cluster]
root@FW01A# show
reth-count 2;
redundancy-group 0 {
    node 0 priority 120;
    node 1 priority 1;
}
redundancy-group 1 {
    node 0 priority 120;
    node 1 priority 1;
    preempt;
    interface-monitor {
        fe-0/0/0 weight 128;
        fe-0/0/1 weight 128;
        fe-1/0/0 weight 128;
        fe-1/0/1 weight 128;
        fe-0/0/2 weight 128;
        fe-0/0/3 weight 128;
        fe-1/0/3 weight 128;
        fe-1/0/2 weight 128;
    }

Failover scenarios

  • First, we will physically disconnect fe-0/0/0 which should not impact our regular traffic flow.

Disconnecting fe-0/0/0/

{primary:node0}
root@FW01A> show interfaces terse | match fe-0/0/0
fe-0/0/0                up    down
fe-0/0/0.111            up    down aenet    --> reth0.111
fe-0/0/0.32767          up    down aenet    --> reth0.32767

The JSRPD daemon logged the following entries in which we can see it decrement the value of 128 of the global threshold of 255.

root@FW01A> show log jsrpd | last | match 19:30
Oct  6 19:30:31 Interface fe-0/0/0 is going down
Oct  6 19:30:31 fe-0/0/0 interface monitored by RG-1 changed state from Up to Down
Oct  6 19:30:31 intf failed, computed-weight -128
Oct  6 19:30:31 LED changed from Green to Amber, reason is Monitored objects are down
Oct  6 19:30:31 Current threshold for rg-1 is 127. Failures: interface-monitoring
Oct  6 19:30:42 printing fpc_num h0
Oct  6 19:30:42 jsrpd_ifd_msg_handler: Interface reth0 is up
Oct  6 19:30:42 reth0 from  jsrpd_ssam_reth_read reth_rg_id=1

The chassis cluster interfaces also shows us the interface as down and how much weight it had:

root@FW01A> show chassis cluster interfaces | find Monitoring
Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    fe-1/0/2          128       Up        1
    fe-1/0/3          128       Up        1
    fe-0/0/3          128       Up        1
    fe-0/0/2          128       Up        1
    fe-1/0/1          128       Up        1
    fe-1/0/0          128       Up        1
    fe-0/0/1          128       Up        1
    fe-0/0/0          128       Down      1

Because our threshold value is still at 127 (255 – 128), RG1 is still active on node0 and no failover event was triggered.

Redundancy group: 1 , Failover count: 0
    node0                   120         primary        yes      no
    node1                   1           secondary      yes      no
  • As a second test, we will also unplug fe-0/0/1. Without the interface monitoring, this would halt the forwarding of traffic to the untrust zone as reth0 would have no more interfaces on node0.

Disconnecting fe-0/0/1 as well

To get a basic idea of the failover time, I’m running a ping test from the core switch.

Sending 1000, 100-byte ICMP Echos to 1.1.1.14, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*snip*

We lost one packet during the failover with a timeout of two seconds, which is not bad at all. The switchover is stateful so TCP connections should be able to handle this just fine. Let’s see what happened on the SRX…

Our interface monitor shows two interfaces down that had a weight of 128. The threshold should now have reached zero which retired node0 from the cluster.

root@FW01A> show chassis cluster interfaces | find Monitoring
Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    fe-1/0/2          128       Up        1
    fe-1/0/3          128       Up        1
    fe-0/0/3          128       Up        1
    fe-0/0/2          128       Up        1
    fe-1/0/1          128       Up        1
    fe-1/0/0          128       Up        1
    fe-0/0/1          128       Down      1
    fe-0/0/0          128       Down      1

Redundancy group 1 is now primary on node1. The routing engine is still active on node0.

root@FW01A> show chassis cluster status
Cluster ID: 1
Node                  Priority          Status    Preempt  Manual failover

Redundancy group: 0 , Failover count: 0
    node0                   120         primary        no       no
    node1                   1           secondary      no       no

Redundancy group: 1 , Failover count: 1
    node0                   0           secondary      yes      no
    node1                   1           primary        yes      no

And the JSRPD logged documented the entire event:

Oct  6 19:43:01 Interface fe-0/0/1 is going down
Oct  6 19:43:01 fe-0/0/1 interface monitored by RG-1 changed state from Up to Down
Oct  6 19:43:01 intf failed, computed-weight -256
Oct  6 19:43:01 RG(1) priority changed on node0 120->0 Priority is set to 0, Monitoring objects are down
Oct  6 19:43:01 Successfully sent an snmp-trap due to priority change from 120 to 0 on RG-1 on cluster 1 node 0. Reason: Priority is set to 0, Monitoring objects are down
Oct  6 19:43:01 Current threshold for rg-1 is -1. Setting priority to 0. Failures: interface-monitoring
Oct  6 19:43:01 Both the nodes are primary. RG-1 PRIMARY->SECONDARY_HOLD due to preempt/yield, my priority 0 is worse than other node's priority 1
Oct  6 19:43:01 Successfully sent an snmp-trap due to a failover from primary to secondary-hold on RG-1 on cluster 1 node 0. Reason: Monitor failed: IF
Oct  6 19:43:01 updated rg_info for RG-1 with failover-cnt 1 state: secondary-hold into ssam. Result = success, error: 0
Oct  6 19:43:01 reth0 ifd state changed from node0-primary -> node1-primary for RG-1
Oct  6 19:43:01 reth1 ifd state changed from node0-primary -> node1-primary for RG-1
Oct  6 19:43:01 updating primary-node as node1 for RG-1 into ssam. Previous primary was node0. Result = success, 0
Oct  6 19:43:01 Successfully sent an snmp-trap due to a failover from primary to secondary-hold on RG-1 on cluster 1 node 0. Reason: Monitor failed: IF
Oct  6 19:43:01 printing fpc_num h0
Oct  6 19:43:01 jsrpd_ifd_msg_handler: Interface reth0 is up
Oct  6 19:43:01 reth0 from  jsrpd_ssam_reth_read reth_rg_id=1
Oct  6 19:43:01 printing fpc_num h1
Oct  6 19:43:01 jsrpd_ifd_msg_handler: Interface reth1 is up
Oct  6 19:43:01 reth1 from  jsrpd_ssam_reth_read reth_rg_id=1
Oct  6 19:43:02 SECONDARY_HOLD->SECONDARY due to back to back failover timer expiry for RG-1
Oct  6 19:43:02 Successfully sent an snmp-trap due to a failover from secondary-hold to secondary on RG-1 on cluster 1 node 0. Reason: Back to back failover interval expired
Oct  6 19:43:02 updated rg_info for RG-1 with failover-cnt 1 state: secondary into ssam. Result = success, error: 0
Oct  6 19:43:02 Successfully sent an snmp-trap due to a failover from secondary-hold to secondary on RG-1 on cluster 1 node 0. Reason: Back to back failover interval expired

What is interesting is that the SRX is explicitly sending an SNMP trap for these events, so make sure you have a good monitoring tool and trap receiver in place, preferably with alerting.

  • As the ultimate test, I will unplug interface fe-1/0/0, the first port of node1, which will make the whole LACP bundle reth0 run on just one link. If we had configured minimum-links 2 this action would bring down the whole reth bundle

We are riding on one wheel now!

After pulling the cable, reth0 is strolling along on just one link:

root@FW01A> show lacp interfaces reth0 | find protocol
    LACP protocol:        Receive State  Transmit State          Mux State
      fe-0/0/0            Port disabled     No periodic           Detached
      fe-0/0/1            Port disabled     No periodic           Detached
      fe-1/0/0            Port disabled     No periodic           Detached
      fe-1/0/1                  Current   Slow periodic Collecting distributing

And the Etherchannel is still up:

root@FW01A> show interfaces terse | match reth0
reth0                   up    up
reth0.111               up    up   inet     1.1.1.1/28
reth0.32767             up    up
root@FW01A> show chassis cluster interfaces | find Monitoring
Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    fe-1/0/2          128       Up        1
    fe-1/0/3          128       Up        1
    fe-0/0/3          128       Up        1
    fe-0/0/2          128       Up        1
    fe-1/0/1          128       Up        1
    fe-1/0/0          128       Down      1
    fe-0/0/1          128       Down      1
    fe-0/0/0          128       Down      1

For anyone designing a network solution with high-availability in mind, this is all very promising. Even with 75% of our physical links down, the reth will stay functional and forward traffic.

I have now reconnected all cabling for a last question I’ve been pondering about – how does the interface monitor react when physical ports go down on both nodes? Suppose we are running our trunks to a chassis or VSS and lose a linecard or stack-member? Does the interface weight count globally and trigger a failover? Let’s find out…

  • Disconnecting fe-0/0/0 and fe-1/0/0, the first port on each node which is in reth0
root@FW01A> show interfaces terse | match "fe-0/0/0|fe-1/0/0"
fe-0/0/0                up    down
fe-0/0/0.111            up    down aenet    --> reth0.111
fe-0/0/0.32767          up    down aenet    --> reth0.32767
fe-1/0/0                up    down
fe-1/0/0.111            up    down aenet    --> reth0.111
fe-1/0/0.32767          up    down aenet    --> reth0.32767

Both links, which both had a weight of 128, are now down but RG1 is still rolling along fine on node0 and the priority is not set to 0. This does prove that the interface-monitor value is tied to the node on which the interface resides.

Redundancy group: 1 , Failover count: 2
    node0                   120         primary        yes      no
    node1                   1           secondary      yes      no

Let’s disconnect fe-0/0/2 and fe-1/0/2 to simulate a really bad day. Our interface monitor now shows half our revenue ports down:

Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    fe-1/0/2          128       Down      1
    fe-1/0/3          128       Up        1
    fe-0/0/3          128       Up        1
    fe-0/0/2          128       Down      1
    fe-1/0/1          128       Up        1
    fe-1/0/0          128       Down      1
    fe-0/0/1          128       Up        1
    fe-0/0/0          128       Down      1

Interestingly though, both nodes have their priorities now set to zero, but node0 is still primary for RG1.

Redundancy group: 0 , Failover count: 0
    node0                   120         primary        no       no
    node1                   1           secondary      no       no

Redundancy group: 1 , Failover count: 2
    node0                   0           primary        yes      no
    node1                   0           secondary      yes      no

Is there still traffic being sent through though? Running a quick ping from CS2:

NP-CS1#ping 1.1.1.14

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.14, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms

Conclusion

This turned into quite a lengthy post but hopefully demonstrated the benefits of having an Etherchannel over the standard single-port configurations you’ll find in most documentation. Bundling your physical interfaces in a LAG gives you that extra layer of physical redundancy with the added benefit of load sharing on the links. We were able to “lose” 75 percent of our revenue ports without any significant impact. Adding the tagged Layer3 interfaces also gives us the option to add more logical units in the future, which can in turn be assigned to their own routing-instances and zones if you’re dealing with a large-scale or multi-tenant environment.

If this was helpful for you or if you have a remark, please let me know below in the comments! Thanks for reading 🙂

External links that added to my understanding of this topic

Failure is optional – Reth Interfaces and LACP
Juniper KB – Link aggregation (LACP) supported/non-supported configurations on SRX and EX
The free JNCIS-SEC Study Guides at Juniper.net