Replaceable masters on AWS using the Universal Installer

Replaceable Masters on AWS using the Universal Installer

This guide expects that you already have a running DC/OS cluster based on Universal Installer 0.3. To learn more about running DC/OS with the Universal Installer have a look into the Guide for DC/OS on AWS using the Universal Installer.

By default the Universal Installer uses a static master list to form the quorum needed by DC/OS. This adds some problems to dynamic cloud environments. The master IP addresses shouldn’t change, and most cases this will not happen as long as you do not destroy or taint a master instance. But in a complete lifecycle of a DC/OS cluster you might face situations where you want to be able to recreate a master instance without deep manual interaction. This feature will enable you simply taint and reinstall a minority of your master nodes without any dataloss as long as the majority of the initial masters is still alive.

Prerequisites

Fully Managed Replaceable Masters

IMPORTANT: Do not apply this change on an already running cluster.

With Universal Installer 0.3 we offer the ability of fully managed replaceable masters based on an AWS S3 Bucket. To enable this feature set with_replaceable_masters=true. Once set this option will lead to the creation of a S3 bucket in the location your cluster will be placed. The bucket name is cluster_name + a 16 Byte random hex string. As bucket names are global we have to attach a random string to your cluster name to avoid collisions. Beside creating a bucket we will inject these DC/OS config defaults.

  1. dcos_s3_prefix                 = "exhibitor"
  2. dcos_exhibitor_explicit_keys   = "false"
  3. dcos_aws_region                = <<the_current_region>>
  4. dcos_master_discovery          = "master_http_loadbalancer"
  5. dcos_exhibitor_address         = <<master_load_balancer_address>>
  6. dcos_num_masters               = <<the_number_of_masters_set>>
  7. dcos_exhibitor_storage_backend = "aws_s3"

Example

Here is an example using with_replaceable_masters:

  1. provider "aws" {
  2.   # Change your default region here
  3.   region = "us-east-1"
  4. }
  6. # Used to determine your public IP for forwarding rules
  7. data "http" "whatismyip" {
  8.   url = "http://whatismyip.akamai.com/"
  9. }
  11. module "dcos" {
  12.   source  = "dcos-terraform/dcos/aws"
  13.   version = "~> 0.3.0"
  15.   providers = {
  16.     aws = aws
  17.   }
  19.   cluster_name        = "my-dcos-demo"
  20.   ssh_public_key_file = "<path-to-public-key-file>"
  21.   admin_ips           = ["${data.http.whatismyip.body}/32"]
  23.   num_masters        = "3"
  24.   num_private_agents = "2"
  25.   num_public_agents  = "1"
  27.   dcos_version = "2.1"
  29.   dcos_variant              = "ee"
  30.   dcos_license_key_contents = "${file("./license.txt")}"
  32.   # Make sure to set your credentials if you do not want the default EE
  33.   # dcos_superuser_username      = "superuser-name"
  34.   # dcos_superuser_password_hash = "${file("./dcos_superuser_password_hash.sha512")}"
  36.   with_replaceable_masters = true
  37. }
  39. output "cluster-address" {
  40.   value = module.dcos.masters-loadbalancer
  41. }