Configuring NSX-T SDN

NSX-T SDN and OKD

VMware NSX-T Data Center ™ provides advanced software-defined networking (SDN), security, and visibility to container environments that simplifies IT operations and extends native OKD networking capabilities.

NSX-T Data Center supports virtual machine, bare metal, and container workloads across multiple clusters. This allows organizations to have complete visibility using a single SDN across the entire environment.

For more information on how NSX-T integrates with OKD, see the NSX-T SDN in Available SDN plug-ins.

Example Topology

One typical use case is to have a Tier-0 (T0) router that connects the physical system with the virtual environment and a Tier-1 (T1) router to act as a default gateway for the OKD VMs.

Each VM has two vNICs: One vNIC connects to the Management Logical Switch for accessing the VMs. The other vNIC connects to a Dump Logical Switch and is used by nsx-node-agent to uplink the Pod networking. For further details, refer to NSX Container Plug-in for OpenShift.

The LoadBalancer used for configuring OKD Routes and all project T1 routers and Logical Switches are created automatically during the OKD installation.

In this topology, the default OKD HAProxy Router is used for all infrastructure components such as Grafana, Prometheus, Console, Service Catalog, and others. Ensure that the DNS records for the infrastructure components point to the infrastructure node IP addresses, because the HAProxy uses the host network namespace. This works for infrastructure routes, but in order to avoid exposing the infrastructure nodes management IPs to the outside world, deploy application-specific routes to the NSX-T LoadBalancer.

This example topology assumes you are using three OKD master virtual machines and four OKD worker virtual machines (two for infrastructure and two for compute).

Installing VMware NSX-T

Prerequisites:

  • ESXi hosts requirements:

    • ESXi servers that host OKD node VMs must be NSX-T Transport Nodes.

      NSX Transport Nodes

      Figure 1. NSX UI dislaying the Transport Nodes for a typical high availability environment:

  • DNS requirements:

    • You must add a new entry to your DNS server with a wildcard to the infrastructure nodes. This allows load balancing by NSX-T or other third-party LoadBalancer. In the hosts file below, the entry is defined by the openshift_master_default_subdomain variable.

    • You must update your DNS server with the openshift_master_cluster_hostname and openshift_master_cluster_public_hostname variables.

  • Virtual Machine requirements:

    • The OKD node VMs must have two vNICs:

    • A Management vNIC must be connected to the Logical Switch that is uplinked to the management T1 router.

    • The second vNIC on all VMs must be tagged in NSX-T so that the NSX Container Plug-in (NCP) knows which port needs to be used as a parent VIF for all Pods running in a particular OKD node. The tags must be the following:

      1. {'ncp/node_name': 'node_name'}
      2. {'ncp/cluster': 'cluster_name'}

      The following image shows how the tags in NSX UI for all nodes. For a large scale cluster, you can automate the tagging using API Call or by using Ansible.

      NSX VM tags

      Figure 2. NSX UI dislaying node tags

      The order of the tags in the NSX UI is opposite from the API. The node name must be exactly as kubelet expects and the cluster name must be the same as the nsx_openshift_cluster_name in the Ansible hosts file, as shown below. Ensure that the proper tags are applied on the second vNIC on every node.

  • NSX-T requirements:

    The following prerequisites need to be met in NSX:

    • A Tier-0 Router.

    • An Overlay Transport Zone.

    • An IP Block for POD networking.

    • Optionally, an IP Block for routed (NoNAT) POD networking.

    • An IP Pool for SNAT. By default the subnet given per Project from the Pod networking IP Block is routable only inside NSX-T. NCP uses this IP Pool to provide connectivity to the outside.

    • Optionally, the Top and Bottom firewall sections in a dFW (Distributed Firewall). NCP places the Kubernetes Network Policy rules between those two sections.

    • The Open vSwitch and CNI plug-in RPMs need to be hosted on a HTTP server reachable from the OKD Node VMs ([http://websrv.example.com](http://websrv.example.com) in this example). Those files are included in the NCP Tar file, which you can download from VMware at Download NSX Container Plug-in 2.4.0 .

  • OKD requirements:

    • Run the following command to install required software packages, if any, for OKD:

      1. $ ansible-playbook -i hosts openshift-ansible/playbooks/prerequisites.yml

    • Ensure that the NCP container image is downloaded locally on all nodes

    • After the prerequisites.yml playbook has successfully executed, run the following command on all nodes, replacing the xxx with the NCP build version:

      1. $ docker load -i nsx-ncp-rhel-xxx.tar

      For example:

      1. $ docker load -i nsx-ncp-rhel-2.4.0.12511604.tar

    • Get the image name and retag it:

      1. $ docker images
      2. $ docker image tag registry.local/xxxxx/nsx-ncp-rhel nsx-ncp (1)
      1Replace the xxx with the NCP build version. For example:
      1. docker image tag registry.local/2.4.0.12511604/nsx-ncp-rhel nsx-ncp

    • In the OKD Ansible hosts file, specify the following parameters to set up NSX-T as the network plug-in:

      1. [OSEv3:children]
      2. masters
      3. nodes
      4. etcd
      6. [OSEv3:vars]
      7. ansible_ssh_user=root
      8. openshift_deployment_type=origin
      9. openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
      10. openshift_master_htpasswd_users={"admin" : "$apr1$H0QeP6oX$HHdscz5gqMdtTcT5eoCJ20"}
      11. openshift_master_default_subdomain=demo.example.com
      12. openshift_use_nsx=true
      13. os_sdn_network_plugin_name=cni
      14. openshift_use_openshift_sdn=false
      15. openshift_node_sdn_mtu=1500
      16. openshift_master_cluster_method=native
      17. openshift_master_cluster_hostname=master01.example.com
      18. openshift_master_cluster_public_hostname=master01.example.com
      19. openshift_hosted_manage_registry=true
      20. openshift_hosted_manage_router=true
      21. openshift_enable_service_catalog=true
      22. openshift_cluster_monitoring_operator_install=true
      23. openshift_web_console_install=true
      24. openshift_console_install=true
      26. # NSX-T specific configuration
      27. #nsx_use_loadbalancer=false
      28. nsx_openshift_cluster_name='cluster01'
      29. nsx_api_managers='nsxmgr.example.com'
      30. nsx_api_user='nsx_admin'
      31. nsx_api_password='nsx_api_password_example'
      32. nsx_tier0_router='LR-Tier-0'
      33. nsx_overlay_transport_zone='TZ-Overlay'
      34. nsx_container_ip_block='pod-networking'
      35. nsx_no_snat_ip_block='pod-nonat'
      36. nsx_external_ip_pool='pod-external'
      37. nsx_top_fw_section='containers-top'
      38. nsx_bottom_fw_section='containers-bottom'
      39. nsx_ovs_uplink_port='ens224'
      40. nsx_cni_url='http://websrv.example.com/nsx-cni-buildversion.x86_64.rpm'
      41. nsx_ovs_url='http://websrv.example.com/openvswitch-buildversion.rhel75-1.x86_64.rpm'
      42. nsx_kmod_ovs_url='http://websrv.example.com/kmod-openvswitch-buildversion.rhel75-1.el7.x86_64.rpm'
      43. nsx_insecure_ssl=true
      44. # vSphere Cloud Provider
      45. #openshift_cloudprovider_kind=vsphere
      46. #openshift_cloudprovider_vsphere_username='administrator@example.com'
      47. #openshift_cloudprovider_vsphere_password='viadmin_password'
      48. #openshift_cloudprovider_vsphere_host='vcsa.example.com'
      49. #openshift_cloudprovider_vsphere_datacenter='Example-Datacenter'
      50. #openshift_cloudprovider_vsphere_cluster='example-Cluster'
      51. #openshift_cloudprovider_vsphere_resource_pool='ocp'
      52. #openshift_cloudprovider_vsphere_datastore='example-Datastore-name'
      53. #openshift_cloudprovider_vsphere_folder='ocp'
      55. [masters]
      56. master01.example.com
      57. master02.example.com
      58. master03.example.com
      60. [etcd]
      61. master01.example.com
      62. master02.example.com
      63. master03.example.com
      65. [nodes]
      66. master01.example.com ansible_ssh_host=192.168.220.2 openshift_node_group_name='node-config-master'
      67. master02.example.com ansible_ssh_host=192.168.220.3 openshift_node_group_name='node-config-master'
      68. master03.example.com ansible_ssh_host=192.168.220.4 openshift_node_group_name='node-config-master'
      69. node01.example.com ansible_ssh_host=192.168.220.5 openshift_node_group_name='node-config-infra'
      70. node02.example.com ansible_ssh_host=192.168.220.6 openshift_node_group_name='node-config-infra'
      71. node03.example.com ansible_ssh_host=192.168.220.7 openshift_node_group_name='node-config-compute'
      72. node04.example.com ansible_ssh_host=192.168.220.8 openshift_node_group_name='node-config-compute'

      For information on the OKD installation parameters, see Configuring Your Inventory File.

Procedure

After meeting all of the prerequisites, you can deploy NSX Data Center and OKD.

  1. Deploy the OKD cluster:

    1. $ ansible-playbook -i hosts openshift-ansible/playbooks/deploy_cluster.yml

    For more information on the OKD installation, see Installing OpenShift Container Platform.

  2. After the installation is complete, validate that the NCP and nsx-node-agent Pods are running:

    1. $ oc get pods -o wide -n nsx-system
    2. NAME                   READY     STATUS    RESTARTS   AGE       IP              NODE                                   NOMINATED NODE
    3. nsx-ncp-5sggt          1/1       Running   0          1h        192.168.220.8   node04.example.com     <none>
    4. nsx-node-agent-b8nkm   2/2       Running   0          1h        192.168.220.5   node01.example.com     <none>
    5. nsx-node-agent-cldks   2/2       Running   0          2h        192.168.220.8   node04.example.com     <none>
    6. nsx-node-agent-m2p5l   2/2       Running   28         3h        192.168.220.4   master03.example.com   <none>
    7. nsx-node-agent-pcfd5   2/2       Running   0          1h        192.168.220.7   node03.example.com     <none>
    8. nsx-node-agent-ptwnq   2/2       Running   26         3h        192.168.220.2   master01.example.com   <none>
    9. nsx-node-agent-xgh5q   2/2       Running   26         3h        192.168.220.3   master02.example.com   <none>

Check NSX-T after OKD deployment

After installing OKD and verifying the NCP and nsx-node-agent-* Pods:

  • Check the routing. Ensure that the Tier-1 routers were created during the installation and are linked to the Tier-0 router:

    NSX routing

    Figure 3. NSX UI dislaying showing the T1 routers

  • Observe the network traceflow and visibility. For example, check the connection between ‘console’ and ‘grafana’.

    For more information on securing and optimizing communications between Pods, Projects, virtual machines, and external services, see the following example:

    NSX visibility

    Figure 4. NSX UI dislaying showing network traceflow

  • Check the load balancing. NSX-T Data center offers Load Balancer and Ingress Controller capabilities, as shown in the following example:

    NSX loadbalancing

    Figure 5. NSX UI dislay showing the load balancers

For additional configuration and options, refer to the VMware NSX-T v2.4 OpenShift Plug-In documentation.