Manual Installation of Consul on AWS Elastic Container Service (ECS)

The following instructions describe how to use the consul-ecs Docker image to manually create the ECS task definition without Terraform. If you prefer to use Terraform, refer to Consul ECS Terraform module.

If you intend to peer the service mesh to multiple Consul datacenters or partitions, you must use the Consul ECS Terraform module to install your service mesh on ECS. Manually configuring mesh gateways without using the gateway-task Terraform module is not supported.

This topic does not include instructions for creating all AWS resources necessary to install Consul, such as a VPC or the ECS cluster. Refer to the linked guides in the Getting Started section for complete, runnable examples.

Prerequisites

You should have some familiarity with AWS ECS. See What is Amazon Elastic Container Service for details.

Task Definition

Configure a task definition that creates the containers:

• Your application container
• An Envoy sidecar-proxy container
• A Consul client container
• A consul-ecs-mesh-init container for service mesh setup
• (Optional) A consul-ecs-health-sync container to sync ECS health checks into Consul

Top-level fields

Your task definition must include the following top-level fields.

The volumes list contains two bind mounts, named consul_data and consul_binary. Bind mounts are directories on the host which can be mounted into one or more containers in order to share files among containers. For Consul on ECS, certain binaries and configuration are shared among containers during task startup.

{
  "family": "my-example-client-app",
  "networkMode": "awsvpc",
  "volumes": [
    {
      "name": "consul_data"
    },
    {
      "name": "consul_binary"
    }
  ],
  "containerDefinitions": [...]
  "tags": [
    {
      "key": "consul.hashicorp.com/mesh",
      "value": "true"
    },
    {
      "key": "consul.hashicorp.com/service-name",
      "value":  "example-client-app"
    }
  ]
}

Task Tags

The tags list must include the following if you are using the ACL controller in a secure configuration. Without these tags, the ACL controller will be unable to provision a service token for the task.

Tag Key	Tag Value	Description
consul.hashicorp.com/mesh	true (string)	The ACL controller ignores tasks without this tag set to true.
consul.hashicorp.com/service-name	Consul service name	Specifies the Consul service associated with this task. Required if the service name is different than the task family.
consul.hashicorp.com/partition	Consul admin partition	EnterpriseEnterpriseSpecifies the Consul admin partition associated with this task. Defaults to the default admin partition if omitted.
consul.hashicorp.com/namespace	Consul namespace	EnterpriseEnterpriseSpecifies the Consul namespace associated with this task. Defaults to the default namespace if omitted.

Application container

First, include your application container in the containerDefinitions list in the task definition.

Ensure that the containerName and condition fields in the dependsOn list are specified as described in the following example. These are container dependencies, which must be used to enforce a specific startup order. By using the following settings, your application container will start after consul-ecs-mesh-init has completed task setup and after sidecar-proxy is ready to proxy traffic between this task and the service mesh.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      "essential": true,
      "dependsOn": [
        {
          "containerName": "consul-ecs-mesh-init",
          "condition": "SUCCESS"
        },
        {
          "containerName": "sidecar-proxy",
          "condition": "HEALTHY"
        }
      ],
      ...
    }
  ]
}

See the ECS Task Definition documentation for a complete reference.

sidecar-proxy container

The sidecar-proxy container runs Envoy proxy for Consul service mesh. In most cases, the container should contain the following parameters and values.

The mountPoints list must be set as shown in the following example. This will mount the shared consul_data volume into the sidecar-proxy container at the path /consul. This volume is where the consul-ecs-mesh-init container copies the envoy-bootstrap.json file and the consul-ecs binary, which are required to start Envoy. The dependsOn list must also be defined as follows to ensure the sidecar-proxy container starts after consul-ecs-mesh-init has successfully written these files to the shared volume.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<VERSION>",
      "essential": false,
      "dependsOn": [
        {
          "containerName": "consul-ecs-mesh-init",
          "condition": "SUCCESS"
        }
      ],
      "healthCheck": {
        "retries": 3,
        "command": ["nc", "-z", "127.0.0.1", "20000"],
        "timeout": 5,
        "interval": 30
      },
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/consul",
          "sourceVolume": "consul_data"
        }
      ],
      "ulimits": [
        {
          "name": "nofile",
          "softLimit": 1048576,
          "hardLimit": 1048576
        }
      ],
      "command": ["envoy", "--config-path", "/consul/envoy-bootstrap.json"],
      "entryPoint": ["/consul/consul-ecs", "envoy-entrypoint"],
    }
  ]
}

The following table describes the necessary configuration settings.

NOTE: Envoy and Consul must be compatible versions. See the supported versions of Envoy in the Consul documentation.

consul-client container

Each task must include a Consul client container in order for the task to join your Consul cluster.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
      ...
    }
    {
      "name": "consul-client",
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      "mountPoints": [
        {
          "readOnly": false,
          "containerPath": "/consul",
          "sourceVolume": "consul_data"
        },
        {
          "containerPath": "/bin/consul-inject",
          "sourceVolume": "consul_binary"
        }
      ],
      "entryPoint": ["/bin/sh", "-ec"],
      "command": [
        "cp /bin/consul /bin/consul-inject/consul\n\nECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')\n\n\ncat << EOF > /consul/agent-defaults.hcl\naddresses = {\n  dns = \"127.0.0.1\"\n  grpc = \"127.0.0.1\"\n  http = \"127.0.0.1\"\n}\nadvertise_addr = \"$ECS_IPV4\"\nadvertise_reconnect_timeout = \"15m\"\nclient_addr = \"0.0.0.0\"\ndatacenter = \"dc1\"\nenable_central_service_config = true\nleave_on_terminate = true\nports {\n  grpc = 8502\n}\nretry_join = [\n  \"<Consul server location>\",\n]\ntelemetry {\n  disable_compat_1.9 = true\n}\n\nEOF\n\ncat << EOF > /consul/agent-extra.hcl\naddresses = {\n  dns = \"0.0.0.0\"\n}\nlog_level = \"debug\"\n\nEOF\n\nexec consul agent \\\n    -data-dir /consul/data \\\n    -config-file /consul/agent-defaults.hcl \\\n    -config-file /consul/agent-extra.hcl\n"
      ]
    }
  ]
}

Consul client startup script

The following script is used to start the Consul client for Consul on ECS.

# Copy the consul binary to a shared volume for `consul-ecs-mesh-init` to use to generate Envoy configuration.
cp /bin/consul /bin/consul-inject/consul
# At runtime, determine the IP address assigned to this ECS Task.
ECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')
# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
addresses = {
  dns = "127.0.0.1"
  grpc = "127.0.0.1"
  http = "127.0.0.1"
}
advertise_addr = "$ECS_IPV4"
advertise_reconnect_timeout = "15m"
client_addr = "0.0.0.0"
datacenter = "dc1"
enable_central_service_config = true
leave_on_terminate = true
ports {
  grpc = 8502
}
retry_join = ["<consul server location>"]
telemetry {
  disable_compat_1.9 = true
}
EOF
# Start the consul agent.
exec consul agent \
    -data-dir /consul/data \
    -config-file /consul/agent-defaults.hcl

The following table describes the values that you should use to configure the command script:

NOTE: Use exec to start the Consul agent so that the Consul agent runs as PID 1. This ensures the Consul agent directly receives signals from ECS, which is important for graceful shutdown of the Consul agent.

Refer to the Consul Agent documentation for a complete reference of Consul agent configuration options.

consul-ecs-mesh-init container

The consul-ecs-mesh-init container runs at task startup to setup this instance for Consul service mesh. It registers the service and proxy for this task with Consul and writes Envoy bootstrap configuration to a shared volume.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
      ...
    },
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      ...
    },
    {
      "name": "consul-ecs-mesh-init",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
      "command": ["mesh-init"],
      "essential": false,
      "environment": [
        {
          "name": "CONSUL_ECS_CONFIG_JSON",
          "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
        }
      ],
      "mountPoints": [
        {
          "readOnly": false,
          "containerPath": "/consul",
          "sourceVolume": "consul_data"
        },
        {
          "readOnly": true,
          "containerPath": "/bin/consul-inject",
          "sourceVolume": "consul_binary"
        }
      ]
    }
  ]
}

CONSUL_ECS_CONFIG_JSON

Consul uses the CONSUL_ECS_CONFIG_JSON environment variable to passed configurations to the consul-ecs binary in JSON format.

The following example configures a service named example-client-app with one upstream service name example-server-app. The proxy and service blocks include information used by consul-ecs-mesh-init to register the service with Consul during task start up. The same configuration format is used for the consul-ecs-health-sync container.

{
  "bootstrapDir": "/consul",
  "healthSyncContainers": [],
  "proxy": {
    "upstreams": [
      {
        "destinationName": "example-server-app",
        "localBindPort": 1234
      }
    ]
  },
  "service": {
    "checks": [],
    "meta": {},
    "name": "example-client-app",
    "port": 9090,
    "tags": []
  }
}

See the Configuration Reference for a complete reference of fields.

consul-ecs-health-sync container

Optionally, Consul ECS can sync health checks for this task into Consul checks. This allows you to configure a health check for your application in one place and see a consistent health status in both ECS and Consul.

For example, the following defines an ECS health check command that runs curl localhost:9090/health:

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      "healthCheck": {
        "retries": 3,
        "command": ["CMD-SHELL", "curl localhost:9090/health"],
        "timeout": 5,
        "interval": 30
      },
      ...
    },
    ...
  ]
}

First, define which containers need their health status synced into Consul. To do this, add the container name(s) to the healthSyncContainers list of the CONSUL_ECS_CONFIG_JSON variable, as shown in the following example. This configuration must be passed to both the consul-ecs-mesh-init and consul-ecs-health-sync containers.

{
  "bootstrapDir": "/consul",
  "healthSyncContainers": ["example-client-app"],
  ...
}

Next, set the CONSUL_ECS_CONFIG_JSON variable for the consul-ecs-mesh-init container. The following example shows how the CONSUL_ECS_CONFIG_JSON variable should be formatted. The JSON configuration is compacted down to a single line and escaped.

{
  "containerDefinitions": [
    {
      "name": "consul-ecs-mesh-init",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<VERSION>",
      "environment": [
        {
          "name": "CONSUL_ECS_CONFIG_JSON",
          "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
        }
      ],
      ...
    },
    ...
  ]
}

Finally, include the consul-ecs-health-sync container in the containerDefinitions list. Pass the same value for CONSUL_ECS_CONFIG_JSON for both the consul-ecs-health-sync and consul-ecs-mesh-init containers.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
      ...
    },
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      ...
    },
    {
      "name": "consul-ecs-mesh-init",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
      ...
    },
    {
      "name": "consul-ecs-health-sync",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
      "command": ["health-sync"],
      "essential": false,
      "dependsOn": [
        {
          "containerName": "consul-ecs-mesh-init",
          "condition": "SUCCESS"
        }
      ],
      "environment": [
        {
          "name": "CONSUL_ECS_CONFIG_JSON",
          "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
        }
      ]
    }
  ]
}

Next Steps

• Create the task definition using the AWS Console or the AWS CLI, or another method of your choice.
• Create an ECS Service to start tasks using the task definition.
• Follow the Secure Configuration to get production-ready.