Multi-Region DC/OS on Azure using the Universal Installer

ENTERPRISE

Guide for DC/OS on Azure using the Universal Installer adding a remote region.

This guide expects that you already have a running DC/OS cluster based on Universal Installer 0.3. To learn more about running DC/OS with the Universal Installer have a look into the Guide for DC/OS on Azure using the Universal Installer.

You will learn how to place additional infrastructure into an Azure remote region. Remote regions will be connected to each other by using the peering functionality of Azure VNETs.

Prerequisites

  • A running DC/OS Enterprise cluster set up with Universal Installer 0.2 modules
  • A subnet range for your remote region

Getting started with remote region

We expect your already running DC/OS clusters main.tf will look similar to this example. To deploy a remote region we have to do some changes to your main.tf

  1. provider "azurerm" {
  2.   features {}
  3. }
  5. # Used to determine your public IP for forwarding rules
  6. data "http" "whatismyip" {
  7.   url = "http://whatismyip.akamai.com/"
  8. }
  10. module "dcos" {
  11.   source  = "dcos-terraform/dcos/aws"
  12.   version = "~> 0.3.0"
  14.   providers = {
  15.     azurerm = azurerm
  16.   }
  18.   location                          = "West US"
  19.   avset_platform_fault_domain_count = 3
  21.   cluster_name        = "my-dcos-demo"
  22.   ssh_public_key_file = "<path-to-public-key-file>"
  23.   admin_ips           = ["${data.http.whatismyip.body}/32"]
  25.   num_masters        = 3
  26.   num_private_agents = 2
  27.   num_public_agents  = 1
  29.   dcos_version = ""
  31.   dcos_variant              = "ee"
  32.   dcos_license_key_contents = "${file("./license.txt")}"
  34.   # Make sure to set your credentials if you do not want the default EE
  35.   # dcos_superuser_username      = "superuser-name"
  36.   # dcos_superuser_password_hash = "${file("./dcos_superuser_password_hash.sha512")}"
  38.   dcos_instance_os = "centos_7.6"
  39. }
  41. output "masters-ips" {
  42.   value = module.dcos.masters-ips
  43. }
  45. output "cluster-address" {
  46.   value = module.dcos.masters-loadbalancer
  47. }
  49. output "public-agents-loadbalancer" {
  50.   value = module.dcos.public-agents-loadbalancer
  51. }

Shared config options

To create the remote region and its infrastructure we will use the same underlying modules as in our master region. This also means there will be some information needed for both infrastructures like cluster_name, admin_ips and ssh_public_key_file. To make the operation easier you should define local variables in your main.tf that will be used in every module.

  1. #...
  3. // lets define variables which are shared between all regions
  4. locals {
  5.   ssh_public_key_file = "~/.ssh/id_rsa.pub"
  6.   cluster_name        = "my-dcos-demo"
  7.   admin_ips           = ["${data.http.whatismyip.body}/32"]
  8. }
  10. #...

Internal subnetworks

Part of the shared information is which internal subnets are used in your infrastructure. If you did not specify subnet_range, terraform uses the default which is 172.12.0.0/16. The remote region we want to specify needs its own subnet.

IMPORTANT: You should not take 172.17.0.0/16, it is dockers internal network default which will lead to problems.

To have a clear separation between our master and our remote regions we will take 10.128.0.0/16 as our remote regions subnet. Also, we will use a map variable to assign the networks to regions. This will make it easier when adding additional regions in the future.

The locals section will now look like this

  1. #...
  3. // lets define variables which are shared between all regions
  4. locals {
  5.   ssh_public_key_file       = "~/.ssh/id_rsa.pub"
  6.   cluster_name              = "my-dcos-demo"
  7.   admin_ips                 = ["${data.http.whatismyip.body}/32"]
  9.   region_networks = {
  10.     // dont use 172.17/26 as its used by docker.
  11.     "master"    = "172.12.0.0/16" // this is the default
  12.     "West US 2" = "10.128.0.0/16"
  13.   }
  14. }
  16. #...

The remote region

Before we start changing values within the dcos module we will append the infrastructure definition of the remote region to your main.tf. In our example case we only want to have private agents in our remote region, and both private and public agents can be put in a remote region.

IMPORTANT: Running master instances in remote regions is not supported.

To only start private agents we will set num_bootstrap = 0, num_masters = 0 and num_public_agents = 0.

Another important topic to mention is naming. To distinguish between instances of your main and your remote region we introduced the name_prefix variable which allows you to add a prefix to the name of every resource. In this example we set the name_prefix to the short name of the remote region.

In the following example you will also find the shared config options being used in the module call referenced by e.g. local.ssh_public_key_file.

  1. #...
  3. module "dcos-wus2" {
  4.   source  = "dcos-terraform/infrastructure/azurerm"
  5.   version = "~> 0.3.0"
  7.   providers = {
  8.     azurerm = azurerm
  9.   }
  11.   location                          = "West US 2"
  12.   avset_platform_fault_domain_count = 2
  13.   subnet_range                      = local.region_networks["West US 2"]
  15.   admin_ips   = local.admin_ips
  16.   name_prefix = "wus2"
  18.   cluster_name = local.cluster_name
  20.   num_bootstrap      = 0
  21.   num_masters        = 0
  22.   num_private_agents = 1
  23.   num_public_agents  = 0
  25.   ssh_public_key_file    = local.ssh_public_key_file
  26. }

Peering to the main DC/OS region

We now need to establish a connection between the two infrastructures. The Universal Installer provides a module for this task. In this module we reference data from both infrastructures the main region holding DC/OS masters and the remote region holding your remote private agents.

The only information this module needs to receive is the output of our dcos and dcos-wus2 modules. We will append this module to the end of your main.tf

Here is the example vnet-peering-section

  1. #...
  3. module "vnet-connection-master-wus2" {
  4.   source  = "dcos-terraform/vnet-peering/azurerm"
  5.   version = "~> 0.3.0"
  7.   providers = {
  8.     azurerm = azurerm
  9.   }
  11.   cluster_name               = local.cluster_name
  12.   local_region_network       = "master"
  13.   local_resource_group_name  = module.dcos.infrastructure.resource_group_name
  14.   local_vnet_name            = module.dcos.infrastructure.vnet_name
  15.   local_vnet_id              = module.dcos.infrastructure.vnet_id
  16.   remote_region_network      = "wus2"
  17.   remote_resource_group_name = module.dcos-wus2.resource_group_name
  18.   remote_vnet_name           = module.dcos-wus2.vnet_name
  19.   remote_vnet_id             = module.dcos-wus2.vnet_id
  20. }

Changes to dcos module

At this point its time to do changes to your dcos module so it knows about the remote region and is able to install the remote agents.

  1. Choose a subnet range. In general this change is not needed but we wanted to make your example pretty specific.

    subnet_range = "local.region_networks["master"]

  2. Change the cluster_name. As this is a shared resource we will make use of the local variable.

    cluster_name = "local.cluster_name

  3. List the SSH key. This is also a shared resource and we can make use of the local variable

    ssh_public_key_file = "local.ssh_public_key_file

  4. Add the admin IPs following the same pattern.

    `admin_ips = local.admin_ips

  5. Add the private agents. This nearly the most important new variable. This tells the DC/OS installation module which other agents need to be installed.

    additional_private_agent_ips = module.dcos-wus2.private_agents.private_ips

Example dcos module

After the changes above got applied your dcos module should look like this

  1. module "dcos" {
  2.   source  = "dcos-terraform/dcos/azurerm"
  3.   version = "~> 0.3.0"
  5.   providers = {
  6.     azure = "azure"
  7.   }
  9.   location                          = "West US"
  10.   avset_platform_fault_domain_count = 3
  11.   subnet_range                      = local.region_networks["master"]
  13.   cluster_name        = local.cluster_name
  14.   ssh_public_key_file = local.ssh_public_key_file
  15.   admin_ips           = local.admin_ips
  17.   num_masters        = 3
  18.   num_private_agents = 2
  19.   num_public_agents  = 1
  21.   dcos_version = ""
  23.   dcos_variant              = "ee"
  24.   dcos_license_key_contents = "${file("./license.txt")}"
  26.   # Make sure to set your credentials if you do not want the default EE
  27.   # dcos_superuser_username      = "superuser-name"
  28.   # dcos_superuser_password_hash = "${file("./dcos_superuser_password_hash.sha512")}"
  30.   dcos_instance_os = "centos_7.6"
  32.   additional_private_agent_ips = module.dcos-wus2.private_agents.private_ips
  33. }

Full main.tf example

Here is the complete main.tf you should see once you completed this guide.

  1. provider "azurerm" {
  2.   features {}
  3. }
  5. # Used to determine your public IP for forwarding rules
  6. data "http" "whatismyip" {
  7.   url = "http://whatismyip.akamai.com/"
  8. }
  11. // lets define variables which are shared between all regions
  12. locals {
  13.   ssh_public_key_file       = "~/.ssh/id_rsa.pub"
  14.   cluster_name              = "my-dcos-demo"
  15.   admin_ips                 = ["${data.http.whatismyip.body}/32"]
  17.   region_networks = {
  18.     // dont use 172.17/26 as its used by docker.
  19.     "master"    = "172.12.0.0/16" // this is the default
  20.     "West US 2" = "172.13.0.0/16"
  21.   }
  22. }
  24. module "dcos" {
  25.   source  = "dcos-terraform/dcos/azurerm"
  26.   version = "~> 0.3.0"
  28.   providers = {
  29.     azurerm = azurerm
  30.   }
  32.   location                          = "West US"
  33.   avset_platform_fault_domain_count = 3
  34.   subnet_range                      = local.region_networks["master"]
  36.   cluster_name        = local.cluster_name
  37.   ssh_public_key_file = local.ssh_public_key_file
  38.   admin_ips           = local.admin_ips
  40.   num_masters        = 3
  41.   num_private_agents = 2
  42.   num_public_agents  = 1
  44.   dcos_version = ""
  46.   dcos_variant              = "ee"
  47.   dcos_license_key_contents = "${file("./license.txt")}"
  49.   # Make sure to set your credentials if you do not want the default EE
  50.   # dcos_superuser_username      = "superuser-name"
  51.   # dcos_superuser_password_hash = "${file("./dcos_superuser_password_hash.sha512")}"
  53.   dcos_instance_os = "centos_7.6"
  55.   additional_private_agent_ips = module.dcos-usw2.private_agents_private_ips
  56. }
  58. output "masters-ips" {
  59.   value = module.dcos.masters-ips
  60. }
  62. output "cluster-address" {
  63.   value = module.dcos.masters-loadbalancer
  64. }
  66. output "public-agents-loadbalancer" {
  67.   value = module.dcos.public-agents-loadbalancer
  68. }
  70. module "dcos-usw2" {
  71.   source  = "dcos-terraform/infrastructure/azurerm"
  72.   version = "~> 0.3.0"
  74.   providers = {
  75.     azurerm = azurerm
  76.   }
  78.   location                          = "West US 2"
  79.   avset_platform_fault_domain_count = 2
  80.   subnet_range                      = local.region_networks["West US 2"]
  82.   admin_ips   = local.admin_ips
  83.   name_prefix = "usw2"
  85.   cluster_name = local.cluster_name
  87.   num_bootstrap      = 0
  88.   num_masters        = 0
  89.   num_private_agents = 1
  90.   num_public_agents  = 0
  92.   ssh_public_key_file    = local.ssh_public_key_file
  93. }
  95. module "vnet-connection-master-usw2" {
  96.   source  = "dcos-terraform/vnet-peering/azurerm"
  97.   version = "~> 0.3.0"
  99.   providers = {
  100.     azurerm = azurerm
  101.   }
  103.   cluster_name               = local.cluster_name
  104.   local_region_network       = "master"
  105.   local_resource_group_name  = module.dcos.infrastructure_resource_group_name
  106.   local_vnet_name            = module.dcos.infrastructure_vnet_name
  107.   local_vnet_id              = module.dcos.infrastructure_vnet_id
  108.   remote_region_network      = "usw2"
  109.   remote_resource_group_name = module.dcos-usw2.resource_group_name
  110.   remote_vnet_name           = module.dcos-usw2.vnet_name
  111.   remote_vnet_id             = module.dcos-usw2.vnet_id
  112. }