NSX-T Lab: 2.4 – 2.5 Upgrade

Intro

Welcome to Part 17 of the NSX-T Lab Series.
This is somewhat of a bonus post as I’ve already completed the NSX-T Lab build series but since I now need to upgrade to 2.5 it makes sense to just add it as part of the original series.
This is an in place upgrade from 2.4 to 2.5.

The Steps

The upgrade process can be broken down into several steps.

  • Download the upgrade bundle
  • Upload the bundle to the NSX Manager
  • Upgrade the Upgrade Coordinator
  • Run Pre Checks
  • Upgrade the NSX Edge Cluster
  • Upgrade the hosts
  • Upgrade the Management Plane
  • Upgrade the Policy Manager

The Build

Download the Upgrade Bundle

The first step is to download the upgrade bundle.
This is not the full OVA file download although you should probably get that as well. What we need is the mub file as shown below.

For production deployments make sure you check the operational impact of the upgrade before proceeding the details are listed on the VMware website HERE
The two biggest changes are the NSX messaging channel port from all transport nodes to the NSX Manager has changed from port 5671 to port 1234, since my lab has no physical firewall this is not an issue.
The other big change is that the NSX Manager now requires more CPU and Memory so I’ll have to tweak the settings for my lab to get it to run without eating all my precious resources.

NSX-T Data Center 2.3 ApplianceMemoryvCPUNSX-T Data Center 2.5 ApplianceMemoryvCPU
N/AN/AN/ANSX Manager Extra Small VM8 GB2
NSX Manager Small VM8 GB2NSX Manager Small VM16 GB4
NSX Manager Medium VM16 GB4NSX Manager Medium VM24 GB6
NSX Manager Large VM32 GB8NSX Manager Large VM48 GB12

Before proceeding make sure the system is healthy by checking the Dashboards from the NSX Manager home page.
Also ensure you have a valid backup, I’ve not configured backups on my lab yet but for a production system make sure you have one before you proceed.

Upload the Upgrade Bundle

When you are ready to upgrade go to ‘System’ ‘ Upgrade’ and then ‘Browse’ to the MUB file.

Select the MUB file and then Hit ‘UPLOAD’

While the file is being processed login to the NSX manager console and run ‘get service’ check the listed services to ensure that all services that should be running are in fact running.

 Last login: Tue Dec  3 12:42:55 2019
 NSX CLI (Manager, Policy, Controller 2.4.1.0.0.13716579). Press ? for command list or enter: help
 NSXTMan01> get service
 Service name:                  cluster_manager     
 Service state:                 running             
 
 Service name:                  cm-inventory        
 Service state:                 running             
 
 Service name:                  controller          
 Service state:                 running             
 Listen address:                                    
 
 Service name:                  datastore           
 Service state:                 running             
 
 Service name:                  http                
 Service state:                 running             
 Session timeout:               1800                
 Connection timeout:            30                  
 Redirect host:                 (not configured)    
 Client API rate limit:         100 requests/sec    
 Client API concurrency limit:  40                  
 Global API concurrency limit:  199     
 
 Client API concurrency limit:  40                  
 Global API concurrency limit:  199                 
 
 Service name:                  install-upgrade     
 Service state:                 running             
 Enabled on:                    192.168.10.50       
 
 Service name:                  liagent             
 Service state:                 stopped             
 
 Service name:                  manager             
 Service state:                 running             
 Logging level:                 info                
 
 Service name:                  mgmt-plane-bus      
 Service state:                 running             
 
 Service name:                  migration-coordinator
 Service state:                 stopped             
 
 Service name:                  node-mgmt           
 Service state:                 running             
 
 Service name:                  node-stats          
 Service state:                 running             
 
 Service name:                  nsx-message-bus     
 Service state:                 running             
 
 Service name:                  nsx-upgrade-agent   
 Service state:                 running             
 
 Service name:                  ntp                 
 Service state:                 running             
 
 Service name:                  policy              
 Service state:                 running             
 Logging level:                 info                
 
 Service name:                  search              
 Service state:                 running             
 
 Service name:                  snmp                
 Service state:                 stopped             
 Start on boot:                 False               
 
 Service name:                  ssh                 
 Service state:                 running             
 Start on boot:                 True                
 
 Service name:                  syslog              
 Service state:                 running             
 
 Service name:                  telemetry           
 Service state:                 running             

 Service name:                  ui-service          
 Service state:                 running

Upgrade the Upgrade Coordinator

Once the file is uploaded and extracted (which may take a while) click on ‘BEGIN UPGRADE’

Accept the license agreement and then Click on ‘Continue’ to start the upgrade of the Upgrade Coordinator.

Run Pre Checks

Once the Upgrade Coordinator has been upgraded you will be returned to this screen. Now you can just click Next and carry on but you should really run the Pre Checks first. The pre checks can be run if you click Next as you will get a warning and the option to run them. But for simplicity we can just click ‘RUN PRE CHECKS’ from the upgrade screen.

Click ‘RUN PRE CHECKS’

Once the checks finish you can see the results in the three sections ‘Edges’ ‘Hosts’ and ‘Management Nodes’ if there are issues then click on the link for more details. As you can see I have an alert for the Management Nodes.

From the PreCheck Issues screen I can click on the warning and also change to the Edges and Hosts tab to view any issues they may have and then go ahead and resolve them. The issue I have with the Management Node is not really an issue but instead is a warning that the Communication port has changed to port 1234 and so if I have a physical firewall I should ensure that I have opened the port. Since I have no firewall on my lab I can ignore this warning. Click ‘OK’ to close the PreCheck window and return the the upgrade screen then click on ‘Next’ in the bottom right corner.

Upgrade Edge Clusters

The Edge upgrades are done in groups, each “Edge Group” consists of Edge Nodes that are part of an Edge Cluster. The obvious reason why you have to upgrade the group is so all Edge Nodes in the cluster end up running on the same version. If you have multiple Edge Clusters each one forms a group and the order in which the groups are upgraded can be changed. Since I only have a single Edge Cluster I don’t have the option to reorder it 😛

There are a couple of settings that can be changed.
Upgrade order across groups

SerialUpgrade all the Edge upgrade unit groups consecutively.
This menu item is selected by default. This selection is applied to the overall upgrade sequence.
ParallelUpgrade all the Edge upgrade unit groups simultaneously.For example, if the overall upgrade is set to the parallel order, the Edge upgrade unit groups are upgraded together and the NSX Edge nodes are upgraded one at a time.

Pause upgrade condition.

When an upgrade unit fails to upgradeSelected by default so that you can fix an error on the Edge node and continue the upgrade.You cannot deselect this setting.
After each group completesSelect to pause the upgrade process after each Edge upgrade unit group finishes upgrading.

I’m going to leave these at their default values, click ‘START’

The Edge upgrade will start and can be paused if needed but not recommended. Each Edge Node within the group will be upgraded one at a time.

Once the upgrade is complete the status will show successful (hopefully).
The post check status will state that No checks have been performed, we can click on ‘RUN POST CHECKS’

We can see that there are no issues so our next step is to click ‘Next’ and proceed to the hosts upgrade.

Upgrade Hosts

The upgrade options for hosts are the same as for Edges, however we can also edit the settings for each Host Group by selecting it and clicking ‘Edit’

With the edit group function we can change the order within the group to either Serial or Parallel and change the upgrade mode to put the hosts into Maintenance or not.
I won’t change the default settings here so I just click ‘Start’ from the Host upgrade screen.

The settings will upgrade one host at a time placing each into maintenance mode and moving any running VM’s to the other hosts.
Once complete we can again run the Post upgrade checks.

We can also verify the software on the hosts have been upgraded by logging into the console and running the command
esxcli software vib list | grep nsx

[root@Comp01:~] esxcli software vib list | grep nsx 

nsx-adf 2.5.0.0.0-6.7.14664072 VMware VMwareCertified   2019-12-03 
nsx-aggservice 2.5.0.0.0-6.7.14664087 VMware VMwareCertified   2019-12-03 
nsx-cli-libs 2.5.0.0.0-6.7.14664172   VMware  VMwareCertified   2019-12-03 
nsx-common-libs 2.5.0.0.0-6.7.14664172 VMware  VMwareCertified   2019-12-03 
nsx-context-mux 2.5.0.0.0esx67-14664127 VMware  VMwareCertified   2019-12-03 
nsx-esx-datapath     2.5.0.0.0-6.7.14663999                VMware  VMwareCertified   2019-12-03 nsx-exporter                   2.5.0.0.0-6.7.14664087                VMware  VMwareCertified   2019-12-03 nsx-host                       2.5.0.0.0-6.7.14663975                VMware  VMwareCertified   2019-12-03 nsx-metrics-libs               2.5.0.0.0-6.7.14664172                VMware  VMwareCertified   2019-12-03 nsx-mpa                        2.5.0.0.0-6.7.14664087                VMware  VMwareCertified   2019-12-03 nsx-nestdb-libs                2.5.0.0.0-6.7.14664172                VMware  VMwareCertified   2019-12-03 nsx-nestdb                     2.5.0.0.0-6.7.14664057                VMware  VMwareCertified   2019-12-03 nsx-netcpa                     2.5.0.0.0-6.7.14664120                VMware  VMwareCertified   2019-12-03 nsx-netopa                     2.5.0.0.0-6.7.14664047                VMware  VMwareCertified   2019-12-03 nsx-opsagent                   2.5.0.0.0-6.7.14664087                VMware  VMwareCertified   2019-12-03 nsx-platform-client            2.5.0.0.0-6.7.14664087                VMware  VMwareCertified   2019-12-03 nsx-profiling-libs             2.5.0.0.0-6.7.14664172                VMware  VMwareCertified   2019-12-03 nsx-proxy                      2.5.0.0.0-6.7.14664108                VMware  VMwareCertified   2019-12-03 nsx-python-gevent              1.1.0-9273114                         VMware  VMwareCertified   2019-06-25 nsx-python-greenlet            0.4.9-12819723                        VMware  VMwareCertified   2019-12-03 nsx-python-logging             2.5.0.0.0-6.7.14664072                VMware  VMwareCertified   2019-12-03 nsx-python-protobuf            2.6.1-12818951                        VMware  VMwareCertified   2019-12-03 nsx-rpc-libs                   2.5.0.0.0-6.7.14664172                VMware  VMwareCertified   2019-12-03 nsx-sfhc                       2.5.0.0.0-6.7.14664087                VMware  VMwareCertified   2019-12-03 nsx-shared-libs                2.5.0.0.0-6.7.14100719                VMware  VMwareCertified   2019-12-03 nsx-upm-libs                   2.5.0.0.0-6.7.14664172                VMware  VMwareCertified   2019-12-03 nsx-vdpi                       2.5.0.0.0-6.7.14664097                VMware  VMwareCertified   2019-12-03 nsxcli                         2.5.0.0.0-6.7.14663983                VMware  VMwareCertified   2019-12-03 [root@Comp01:~] 

Upgrade Management Plane

Next we will upgrade the NSX Manager, there are no configuration options here as I am only running a single manager due to the lab environment. if this was production there is still no options but it will show the three NSX managers and will upgrade them all. Now simply click ‘Start’

The NSX Manager console will be unavailable until the upgrade is complete.
During the upgrade you will receive the status message below as the NSX Manager reboots.

Once the NSX Manager has finished upgrading you can log back in and join or not join the VMware customer experience improvement program.

The upgrade page now shows the Management Nodes Upgrade Status as ‘Successful’ You can also login to the NSX manager console again and run ‘get service’ to confirm they are started. If you are running a cluster of NSX managers which you should be in production then also run get cluster status

Upgrade the Policy Manager

This is the easiest step of them all as its not required.
As we were running NSX 2.4 the Policy Manager is part of the NSX Manager and so has already been upgraded.

Summary

In my lab for some reason my NSX license key was lost as part of the upgrade to resolve this I simply added the key back in again.

The vCPU increased to 4 vCPU and the RAM to 12 GB so I will need to again reduce these down to an acceptable level for my lab, the upgrade however did not add the reservations back in again.
I’ll start with 2 vCPU and 10 GB of RAM and see how it behaves.

Leave a Reply

Your email address will not be published.