Manual Installation

The following instructions describe how to manually create the ECS task definition using the consul-ecs Docker image without Terraform. Refer to the Consul ECS Terraform module documentation for an alternative method for installing Consul on ECS.

This topic does not include instructions for creating all AWS resources necessary to install Consul, such as a VPC or the ECS cluster. Refer to the linked guides in the Getting Started section for complete, runnable examples.

Prerequisites

You should have some familiarity with AWS ECS. See What is Amazon Elastic Container Service for details.

Task Definition

You must create a task definition, which includes the following containers:

• Your application container
• An Envoy sidecar-proxy container
• A Consul client container
• A consul-ecs-mesh-init container for service mesh setup
• Optionally, a consul-ecs-health-sync container to sync ECS health checks into Consul

Top-level fields

Your task definition must include the following top-level fields.

The volumes list contains two bind mounts, named consul_data and consul_binary. Bind mounts are directories on the host which can be mounted into one or more containers in order to share files among containers. For Consul on ECS, certain binaries and configuration are shared among containers during task startup.

{
  "family": "my-example-client-app",
  "networkMode": "awsvpc",
  "volumes": [
    {
      "name": "consul_data",
    },
    {
      "name": "consul_binary",
    }
  ],
  "containerDefinitions": [...]
  "tags": [
    {
      "key": "consul.hashicorp.com/mesh",
      "value": "true"
    },
    {
      "key": "consul.hashicorp.com/service-name",
      "value":  "example-client-app"
    }
  ]
}

Task Tags

The tags list must include the following if you are using the ACL controller in a secure configuration. Without these tags, the ACL controller will be unable to provision a service token for the task.

Application container

First, include your application container in the containerDefinitions list in the task definition.

Ensure that the containerName and condition fields in the dependsOn list are specified as described in the following example. These are container dependencies, which must be used to enforce a specific startup order. By using the following settings, your application container will start after consul-ecs-mesh-init has completed task setup and after sidecar-proxy is ready to proxy traffic between this task and the service mesh.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      "essential": true,
      "dependsOn": [
        {
          "containerName": "consul-ecs-mesh-init",
          "condition": "SUCCESS"
        },
        {
          "containerName": "sidecar-proxy",
          "condition": "HEALTHY"
        }
      ],
      ...
    }
  ]
}

See the ECS Task Definition documentation for a complete reference.

sidecar-proxy container

The sidecar-proxy container runs Envoy proxy for Consul Connect. In most cases, the container should contain the following parameters and values.

The mountPoints list must be set as shown in the following example. This will mount the shared consul_data volume into the sidecar-proxy container at the path /consul. This volume is where the consul-ecs-mesh-init container copies the envoy-bootstrap.json file and the consul-ecs binary, which are required to start Envoy. The dependsOn list must also be defined as follows to ensure the sidecar-proxy container starts after consul-ecs-mesh-init has successfully written these files to the shared volume.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<VERSION>",
      "essential": false,
      "dependsOn": [
        {
          "containerName": "consul-ecs-mesh-init",
          "condition": "SUCCESS"
        }
      ],
      "healthCheck": {
        "retries": 3,
        "command": ["nc", "-z", "127.0.0.1", "20000"],
        "timeout": 5,
        "interval": 30
      },
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/consul",
          "sourceVolume": "consul_data"
        }
      ],
      "ulimits": [
        {
          "name": "nofile",
          "softLimit": 1048576,
          "hardLimit": 1048576
        }
      ],
      "command": ["envoy", "--config-path", "/consul/envoy-bootstrap.json"],
      "entryPoint": ["/consul/consul-ecs", "envoy-entrypoint"],
    }
  ]
}

The following table describes the necessary configuration settings.

NOTE: Envoy and Consul must be compatible versions. See the supported versions of Envoy in the Consul documentation.

consul-client container

Each task must include a Consul client container in order for the task to join your Consul cluster.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
      ...
    }
    {
      "name": "consul-client",
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      "mountPoints": [
        {
          "readOnly": false,
          "containerPath": "/consul",
          "sourceVolume": "consul_data"
        },
        {
          "containerPath": "/bin/consul-inject",
          "sourceVolume": "consul_binary"
        }
      ],
      "entryPoint": ["/bin/sh", "-ec"],
      "command": [
        "cp /bin/consul /bin/consul-inject/consul\n\nECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')\n\n\ncat << EOF > /consul/agent-defaults.hcl\naddresses = {\n  dns = \"127.0.0.1\"\n  grpc = \"127.0.0.1\"\n  http = \"127.0.0.1\"\n}\nadvertise_addr = \"$ECS_IPV4\"\nadvertise_reconnect_timeout = \"15m\"\nclient_addr = \"0.0.0.0\"\ndatacenter = \"dc1\"\nenable_central_service_config = true\nleave_on_terminate = true\nports {\n  grpc = 8502\n}\nretry_join = [\n  \"<Consul server location>",\n]\ntelemetry {\n  disable_compat_1.9 = true\n}\n\nEOF\n\ncat << EOF > /consul/agent-extra.hcl\naddresses = {\n  dns = \"0.0.0.0\"\n}\nlog_level = \"debug\"\n\nEOF\n\nexec consul agent \\\n    -data-dir /consul/data \\\n    -config-file /consul/agent-defaults.hcl \\\n    -config-file /consul/agent-extra.hcl\n"
      ]
    }
  ]
}

Consul client startup script

The following script is used to start the Consul client for Consul on ECS.

# Copy the consul binary to a shared volume for `consul-ecs-mesh-init` to use to generate Envoy configuration.
cp /bin/consul /bin/consul-inject/consul
# At runtime, determine the IP address assigned to this ECS Task.
ECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')
# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
addresses = {
  dns = "127.0.0.1"
  grpc = "127.0.0.1"
  http = "127.0.0.1"
}
advertise_addr = "$ECS_IPV4"
advertise_reconnect_timeout = "15m"
client_addr = "0.0.0.0"
datacenter = "dc1"
enable_central_service_config = true
leave_on_terminate = true
ports {
  grpc = 8502
}
retry_join = ["<consul server location>"]
telemetry {
  disable_compat_1.9 = true
}
EOF
# Start the consul agent.
exec consul agent \
    -data-dir /consul/data \
    -config-file /consul/agent-defaults.hcl

The following table describes the values that you should use to configure the command script:

NOTE: Use exec to start the Consul agent so that the Consul agent runs as PID 1. This ensures the Consul agent directly receives signals from ECS, which is important for graceful shutdown of the Consul agent.

Refer to the Consul Agent documentation for a complete reference of Consul agent configuration options.

consul-ecs-mesh-init container

The consul-ecs-mesh-init container runs at task startup to setup this instance for Consul service mesh. It registers the service and proxy for this task with Consul and writes Envoy bootstrap configuration to a shared volume.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
      ...
    },
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      ...
    },
    {
      "name": "consul-ecs-mesh-init",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
      "command": ["mesh-init"],
      "essential": false,
      "environment": [
        {
          "name": "CONSUL_ECS_CONFIG_JSON",
          "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
        }
      ],
      "mountPoints": [
        {
          "readOnly": false,
          "containerPath": "/consul",
          "sourceVolume": "consul_data"
        },
        {
          "readOnly": true,
          "containerPath": "/bin/consul-inject",
          "sourceVolume": "consul_binary"
        }
      ]
    }
  ]
}

CONSUL_ECS_CONFIG_JSON

Configuration is passed to the consul-ecs binary in JSON format using the CONSUL_ECS_CONFIG_JSON environment variable.

The following is an example of the configuration that might be used for a service named example-client-app with one upstream service name example-server-app. The proxy and service blocks include information used by consul-ecs-mesh-init to perform service registration with Consul during task startup. The same configuration format is used for the consul-ecs-health-sync container.

{
  "bootstrapDir": "/consul",
  "healthSyncContainers": [],
  "proxy": {
    "upstreams": [
      {
        "destinationName": "example-server-app",
        "localBindPort": 1234
      }
    ]
  },
  "service": {
    "checks": [],
    "meta": {},
    "name": "example-client-app",
    "port": 9090,
    "tags": []
  }
}

See the Configuration Reference for a complete reference of fields.

consul-ecs-health-sync container

Optionally, Consul ECS can sync health checks for this task into Consul checks. This allows you to configure a health check for your application in one place and see a consistent health status in both ECS and Consul.

For example, the following defines an ECS health check command that runs curl localhost:9090/health:

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      "healthCheck": {
        "retries": 3,
        "command": ["CMD-SHELL", "curl localhost:9090/health"],
        "timeout": 5,
        "interval": 30
      },
      ...
    },
    ...
  ]
}

First, define which containers need their health status synced into Consul. To do this, add the container name(s) to the healthSyncContainers list of the CONSUL_ECS_CONFIG_JSON variable, as shown in the following example. This configuration must be passed to both the consul-ecs-mesh-init and consul-ecs-health-sync containers.

{
  "bootstrapDir": "/consul",
  "healthSyncContainers": ["example-client-app"],
  ...
}

Next, set the CONSUL_ECS_CONFIG_JSON variable for the consul-ecs-mesh-init container. The following example shows how the CONSUL_ECS_CONFIG_JSON variable should be formatted. The JSON configuration is compacted down to a single line and escaped.

{
  "containerDefinitions": [
    {
      "name": "consul-ecs-mesh-init",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<VERSION>",
      "environment": [
        {
          "name": "CONSUL_ECS_CONFIG_JSON",
          "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
        }
      ],
      ...
    },
    ...
  ]
}

Finally, include the consul-ecs-health-sync container in the containerDefinitions list. Pass the same value for CONSUL_ECS_CONFIG_JSON for both the consul-ecs-health-sync and consul-ecs-mesh-init containers.

{
  "containerDefinitions": [
    {
      "name": "example-client-app",
      "image": "docker.io/org/my_task:v0.0.1",
      ...
    },
    {
      "name": "sidecar-proxy",
      "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
      ...
    },
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      ...
    },
    {
      "name": "consul-ecs-mesh-init",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
      ...
    },
    {
      "name": "consul-ecs-health-sync",
      "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
      "command": ["health-sync"],
      "essential": false,
      "dependsOn": [
        {
          "containerName": "consul-ecs-mesh-init",
          "condition": "SUCCESS"
        }
      ],
      "environment": [
        {
          "name": "CONSUL_ECS_CONFIG_JSON",
          "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
        }
      ]
    }
  ]
}

Next Steps

• Create the task definition using the AWS Console or the AWS CLI, or another method of your choice.
• Create an ECS Service to start tasks using the task definition.
• Follow the Secure Configration to get production-ready.