Intro
Welcome to Part 16 of the NSX-T Lab Series. In the previous post, we configured our Tier 1 Gateway.
In this post we will setup our Tier 0 Gateway and configure dynamic routing to the physical network and link our Tier 1 Gateway to our Tier 0 Gateway so the test app can be accessed from the physical network.
This will be a long post so strap in!
The diagram below shows how our configuration will look after we are done, the RouterLink is automatically configured with an IP when we connect the Tier-0 to the Tier-1 Gateway.
Logically we have two Edge Nodes which each run the SR component of our Tier-0 Gateway, the diagram below shows how the Tier-0 DR which is distributed across our hosts, relates to the SR component running on the Edge nodes which then connects to the physical network.
Remember all the Tier-0 components the DR and the two SR’s are still part of the one Tier-0 Gateway but the SR has to run on an Edge node.
The Build
Go to ‘Networking’ ‘Tier-0 Gateways’ and click on ‘ADD TIER-0 GATEWAY’
Give it a name, for my lab initially I’m going to configure an Active Active configuration which is selected by default, the other option would be Active Standby. Finally click the drop down menu for the Edge Cluster and select the Edge-Cluster we configured earlier.
Now hit the ‘Save’ button
You’ll now be asked if you was to continue configuring the Tier-0 Gateway.
For the Tier-0 we click on ‘YES’
Click the Arrow next to ‘INTERFACES’ to expand the section and then click ‘Set’ We are going to configure our uplinks to the physical network.
We need to configure 4 x uplink interfaces which will be two per Edge Node, each uplink on an Edge node will go to a separate physical router (or in my case a virtual router).
Click ‘ADD INTERFACE’ give it a name, the Type box will be External, configure the IP address in CIDR format for the interface. Select the UplinkA-LS as the Connected To(Segment).
Next select the Edge Node this interface is for in this case thats NSXT-ESG01 finally set the MTU I’m using 1600 here but 1700 or up is a good option.
then Click ‘Save’
Repeat this process another 3 times for the other interfaces.
I’ll configure the following.
ESG01-UplinkB, Connected To(Segment) UplinkB-LS, Edge Node NSXT-ESG01
ESG02-UplinkA, Connected To(Segment) UplinkA-LS, Edge Node NSXT-ESG02
ESG02-UplinkB, Connected To(Segment) UplinkB-LS, Edge Node NSXT-ESG02
Now that we have added the Uplink interfaces we need to set the URPF mode to Node. This is the same as reverse path filter that we have in NSX-V and for an ECMP Edge we always set that to disabled.
Close the Set interfaces screen then click on ‘Close Editing’
Now click ‘ADVANCED CONFIGURATION’
This will switch the view to the Advanced Networking & Security page.
From here Click on ‘Configuration’ and ‘Router Ports’
We need to edit all four ports so select each one and run the following configuration. You’ll notice on my lab the naming for the ports on ESG01 and ESG02 are different this I can only assume is due to the different deployment methods we used for the Edge nodes I’ll do some testing later to try and confirm if this is the reason but it matters not lets turn off URPF.
Select a port and click Edit.
Change URPF Mode from Strict to None then click save.
Repeat for all interfaces.
At this stage I need to explain how I am planning to connect my Edges to the routers. As this lab is nested and I only have a single Cisco 3750 physical Switch which cannot do BGP I needed a way to have virtual routers in the lab. Also to match a normal customer deployment I need two virtual routers to allow me to pair each Edge node uplink with a separate router for ECMP.
For this I am using Sophos XG virtual Routers, I’m not going to cover how to setup Sophos XG there are plenty of other blogs that cover that in more detail that I can do here but I will briefly show you the necessary steps to get the dynamic routing working.
First off I need the VLANs created on the routers.
Router 1 will host VLAN 160 while Router 2 will host VLAN 170. Each VLAN will be assigned the relevant gateway IP 10.160.1.1/24 for VLAN 160 and 10.170.1.1/24 for VLAN 170
Router 1
Router 2
Next I need to configure the BGP Router ID and Local AS for each router.
For Router 1 I set the ID to 10.160.1.1 and the local AS to 65000.
For Router 2 I set the ID to 10.170.1.1 and the local AS to 65000.
We’ll come back to the Sophos XG’s later when we add the BGP neighbours i.e. the Edge nodes and confirm we are getting routes.
We are going to keep the Routing configuration very basic I’m not going to configure Static Routes, IP Prefix Lists, Community Lists or Route maps that’s something I’ll blog about later on.
First off let’s configure our BGP routing on the Tier-0 gateway.
As we edited the uplinks earlier to turn of URPF mode we need to navigate back to our Tier-0 router so go to ‘Networking’ ‘Tier-0 Gateways’ click the three dots to the left of our Tier-0 Gateway and click Edit.
Click the arrow next to BGP and enter the Local AS. I set 65100.
Leave all other settings as they are. Graceful restart would need to be enabled if we were deploying an active standby Tier-0 but since we are active active we want it off.
Click ‘Save’
Now next to BGP Neighbors click on ‘Set’
Click ‘ADD BGP Neighbours’ set the IP of the Router 1 Gateway, this is for VLAN 160. Set the Remote AS Number, for me that’s 65000 and click ‘Save’
Repeat the process for the Router 2 Gateway.
Then Click ‘Close’
Next click the Arrow next to Route Re-distribution and click ‘Set’
I’ve set Tier-0 Subnets to redistribute Static Routes (As I may add them later) , Connected Interfaces and & Segments / All selected.
Also Advertised Tier-1 Subnets Connected Subnets and Static Routes.
Click ‘Apply’
Then Click ‘Save’ on the main screen under Route Re-distribution
OK so we have now configured our Tier-0 to peer via BGP with our physical routers so the next step is to configure the routers to peer with our Tier-0 once that is done we will link the Tier-1 with the Tier-0 and we should see the routers to our test app networks appear on the routers route table.
Back on my Sophos XG router 1 under ‘Routing’ ‘BGP’ ‘Neighbours’ I click on Add and enter the IP of ESG01-UplinkA 10.160.1.11 and set the Remote AS to 65100.
Now if I go to information BGP Neighbours and expand it I can see that I have peered with the Tier-0
BGP neighbor is 10.160.1.11, remote AS 65100, local AS 65000, external link
BGP version 4, remote router ID 10.170.1.11
BGP state = Established, up for 06:56:18
Last read 00:00:19, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
Route refresh: advertised and received(old & new)
Address family IPv4 Unicast: advertised and received
Graceful Restart Capabilty: received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart informations:
End-of-RIB send: IPv4 Unicast
End-of-RIB received:
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 0
Notifications: 0 0
Updates: 2 2
Keepalives: 418 416
Route Refresh: 0 0
Capability: 0 0
Total: 421 418
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
Community attribute sent to this neighbor(both)
3 accepted prefixes
Connections established 1; dropped 0
Last reset never
Local host: 10.160.1.1, Local port: 179
Foreign host: 10.160.1.11, Foreign port: 39107
Nexthop: 10.160.1.1
Read thread: on Write thread: off
If we login to the command line of our NSXT-ESG01 Edge node we can also check the BGP Peering.
Once connected run get logical-router to get the router VRF ID for the Service Router.
NSXT-ESG01> get logical-router Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 a39901b6-00c0-42b6-bcb5-7564141e4329 1 1026 SR-Tier-0-GW01 SERVICE_ROUTER_TIER0 7
Next run vfr vfrid where vrfid is the ID of you SR for me that’s 1 so the command is vrf 1
I can now run the command get bgp neighbor This will give me a lot of detail so to reduce it I add summary to the end of the command.
The 10.160.1.1 Neighbor has a state of Estab which is Established so the peering is working. The 169.254.0.131 is also established, this is the Inter-SR routing, in other words the connection between the two SR components running on each of the Edge nodes.
10.170.1.1 Has a state of Active this is because we have set the neighbor on the Tier-0 but not on router 2 so the Tier-0 is ‘Actively’ trying to Peer. Lets go ahead and configure router 2 to peer with ESG01-UplinkB.
I repeat the process on Router 2 this time adding the neighbor as 10.170.1.11 using the Remote AS 65100
Again I check the neighbors on the router. and the peer is established.
BGP neighbor is 10.170.1.11, remote AS 65100, local AS 65000, external link
BGP version 4, remote router ID 10.170.1.11
BGP state = Established, up for 00:01:01
Last read 00:00:01, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
Route refresh: advertised and received(old & new)
Address family IPv4 Unicast: advertised and received
Graceful Restart Capabilty: received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart informations:
End-of-RIB send: IPv4 Unicast
End-of-RIB received:
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 1 2
Keepalives: 3 1
Route Refresh: 0 0
Capability: 0 0
Total: 5 4
Minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
Community attribute sent to this neighbor(both)
3 accepted prefixes
Connections established 1; dropped 0
Last reset never
Local host: 10.170.1.1, Local port: 34622
Foreign host: 10.170.1.11, Foreign port: 179
Nexthop: 10.170.1.1
Read thread: on Write thread: off
Running the ‘get bgp neighbor summary’ command now shows me 10.170.1.1 is Established.
From Router 1 I can see these routes.
The first three are the uplink A and B and the Inter-SR link with the next hop via the ESG01-UplinkA interface.
The 192.168.88.0 is my local home LAN
BGP table version is 0, local router ID is 10.160.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.160.1.0/24 10.160.1.11 0 0 65100 ?
*> 10.170.1.0/24 10.160.1.11 0 0 65100 ?
*> 169.254.0.128/25 10.160.1.11 0 0 65100 ?
*> 192.168.88.0 0.0.0.0 0 32768 i
Total number of prefixes 4
Router 2 shows the same but via the ESG01-UplinkB interface
BGP table version is 0, local router ID is 10.170.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.160.1.0/24 10.170.1.11 0 0 65100 ?
*> 10.170.1.0/24 10.170.1.11 0 0 65100 ?
*> 169.254.0.128/25 10.170.1.11 0 0 65100 ?
*> 192.168.88.0 0.0.0.0 0 32768 i
Total number of prefixes 4
running get route on the Edge node will show me all the routes. At this stage they are all directly connected routes.
NSXT-ESG01(tier0_sr)> get route Flags: t0c - Tier0-Connected, t0s - Tier0-Static, B - BGP, t0n - Tier0-NAT, t1s - Tier1-Static, t1c - Tier1-Connected, t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT, t1d: Tier1-DNS FORWARDER, > - selected route, * - FIB route Total number of routes: 5 t0c> * 10.170.1.0/24 is directly connected, uplink-271, 08:06:43 t0c> * 10.160.1.0/24 is directly connected, uplink-269, 08:06:37 t0c> * 169.254.0.128/25 is directly connected, inter-sr-266, 08:06:37 t0c> * fe80::/64 is directly connected, inter-sr-266, 08:06:37
I now repeat the process and add the NSXT-ESG02 Uplink A and B peers to the Router 1 and Router 2 configs.
Router 1 now shows both Edge nodes as does router 2
BGP router identifier 10.160.1.1, local AS number 65000 RIB entries 7, using 448 bytes of memory Peers 2, using 4968 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.160.1.11 4 65100 502 495 0 0 0 08:10:17 3 10.160.1.12 4 65100 3 6 0 0 0 00:00:14 3 Total number of neighbors 2
Routes on Router 1 and two show a next hop of 1.11 and 1.12 which are the uplinks on each of the Edge nodes.
BGP table version is 0, local router ID is 10.160.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 10.160.1.0/24 10.160.1.12 0 0 65100 ?
*> 10.160.1.11 0 0 65100 ?
* 10.170.1.0/24 10.160.1.12 0 0 65100 ?
*> 10.160.1.11 0 0 65100 ?
* 169.254.0.128/25 10.160.1.12 0 0 65100 ?
*> 10.160.1.11 0 0 65100 ?
*> 192.168.88.0 0.0.0.0 0 32768 i
So we now have the Tier-0 peering via BGP with the physical network great 🙂 now we need to connect our Tier-1 Gateway to the Tier-0 and do a tiny bit of configuration and we will see our test app networks on the Routers route table.
So let’s go back to ‘Networking’ and this time to ‘Tier-1 Gateways’ click the three dots next to our Tier-1 Gateway and click ‘Edit’.
Now from the dropdown menu under Linked Tier-0 Gateway select our Tier-0 Next click the Arrow next to Route Advertisement and select ‘All Connected Segments and Service Ports’ I’ve also checked All Static Routes for future testing’ now click Save.
Now let’s go back to our Edge nodes and our Routers and confirm that we can see the test app networks.
The route table on Router one shows….
BGP table version is 0, local router ID is 10.160.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.0.1.0/24 10.160.1.11 0 0 65100 ?
* 10.160.1.12 0 0 65100 ?
*> 10.0.2.0/24 10.160.1.11 0 0 65100 ?
* 10.160.1.12 0 0 65100 ?
*> 10.0.3.0/24 10.160.1.11 0 0 65100 ?
* 10.160.1.12 0 0 65100 ?
* 10.160.1.0/24 10.160.1.12 0 0 65100 ?
*> 10.160.1.11 0 0 65100 ?
* 10.170.1.0/24 10.160.1.12 0 0 65100 ?
*> 10.160.1.11 0 0 65100 ?
* 169.254.0.128/25 10.160.1.12 0 0 65100 ?
*> 10.160.1.11 0 0 65100 ?
*> 192.168.88.0 0.0.0.0 0 32768 i
Total number of prefixes 7
Awesome we can now see routes to 10.0.1.0/24, 10.0.2.0/24 and 10.0.3.0/24 which are our test app networks!
NSXT-ESG01 and 02 can also see the routes.
NSXT-ESG01(tier0_sr)> get route Flags: t0c - Tier0-Connected, t0s - Tier0-Static, B - BGP, t0n - Tier0-NAT, t1s - Tier1-Static, t1c - Tier1-Connected, t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT, t1d: Tier1-DNS FORWARDER, > - selected route, * - FIB route Total number of routes: 9 t1c> * 10.0.1.0/24 [3/0] via 100.64.160.1, linked-627, 00:05:23 t1c> * 10.0.2.0/24 [3/0] via 100.64.160.1, linked-627, 00:05:23 t0c> * 10.160.1.0/24 is directly connected, uplink-269, 08:27:33 t0c> * 169.254.0.128/25 is directly connected, inter-sr-266, 08:27:33 t0c> * 100.64.160.0/31 is directly connected, linked-627, 00:06:52 t0c> * 10.170.1.0/24 is directly connected, uplink-271, 08:27:39 t1c> * 10.0.3.0/24 [3/0] via 100.64.160.1, linked-627, 00:05:23 t0c> * fc85:875e:fcd8:3800::/64 is directly connected, linked-627, 00:06:52 t0c> * fe80::/64 is directly connected, inter-sr-266, 08:27:33
From our test app we can now Ping the uplink interfaces of our Tier-0 Gateway
We can also ping the Router Gateway IP’s
As a final test I want to ensure I can reach my test app from an external PC.
I have a Computer attached to my local home LAN, the virtual routers also have a network adapter in this local LAN.
On the external PC I need to add two static summary routes to cover my three test networks.
I add the following
route add 10.0.0.0 MASK 255.255.252.0 192.168.88.80
route add 10.0.0.0 MASK 255.255.252.0 192.168.88.90
192.168.88.80 is the home LAN IP for Router 1 192.168.88.90 is Router 2.
These cover my three test app networks which are 10.0.1.0/24 10.0.2.0/24 and 10.0.3.0/24 and the two routes go one to each of the Routers.
in normal conditions they tend to use .80 but shutting down that router will cause the traffic to flick to the other one.
So let’s test it, first a telnet to my web server
C:\WINDOWS\system32>tracert 10.0.1.11
Tracing route to 10.0.1.11 over a maximum of 30 hops
1 1 ms 2 ms 5 ms 192.168.88.90
2 4 ms 2 ms 2 ms 10.170.1.11
3 2 ms 2 ms 2 ms 100.64.160.1
4 5 ms 2 ms 2 ms 10.0.1.11
Trace complete.
So the first hop is to 192.168.88.90 which is the LAN IP for Router 2
Next hop is 10.170.1.11 which is the Uplink interface B of ESG01
Then 100.64.160.1 is the uplink interface of the Tier-1
Finally we hit 10.0.1.11 which is the web server.
If I ping the App server.
C:\WINDOWS\system32>tracert 10.0.2.11
Tracing route to 10.0.2.11 over a maximum of 30 hops
1 1 ms 1 ms 2 ms 192.168.88.80
2 5 ms 1 ms 1 ms 10.160.1.11
3 2 ms 1 ms 2 ms 100.64.160.1
4 4 ms 3 ms 3 ms 10.0.2.11
Trace complete.
The first hop is to 192.168.88.80 which is the LAN IP for Router 1
Next hop is 10.160.1.11 which is the Uplink interface A of ESG01
Then 100.64.160.1 is the uplink interface of the Tier-1
Finally we hit 10.0.2.11 which is the App server.
DB server is the same as the App server trace route.
If I now power off ESG01 1 and run the trace route again I get the following.
WEB
C:\WINDOWS\system32>tracert 10.0.1.11
Tracing route to 10.0.1.11 over a maximum of 30 hops
1 129 ms 1 ms 23 ms 192.168.88.90
2 3 ms 2 ms 2 ms 10.170.1.12
3 3 ms 2 ms 2 ms 100.64.160.1
4 74 ms 6 ms 9 ms 10.0.1.11
Trace complete.
APP
C:\WINDOWS\system32>tracert 10.0.2.11
Tracing route to 10.0.2.11 over a maximum of 30 hops
1 1 ms 1 ms 1 ms 192.168.88.80
2 2 ms 2 ms 2 ms 10.160.1.12
3 4 ms 9 ms 2 ms 100.64.160.1
4 17 ms 2 ms 2 ms 10.0.2.11
Trace complete.
Notice the changes hop two is now hitting 10.170.1.12 for web which is Uplink interface B of ESG02
For App hop 2 is now hitting 10.160.1.12 which is the Uplink interface A of ESG02
The rest of the hops are the same.
So there we have it dynamic routing is working and we can reach our test app from an external network 🙂
And that concludes my NSX-T Lab install series I hope you found it useful.
It’s taken me longer to write that I had planned due to work commitments and lack of time, as a result NSX-T 2.5 is now out so the next thing I’ll be doing is upgrading my lab to 2.5 and blogging it as I go.