Secure Configuration

For a production-ready installation of Consul on ECS, you will need to make sure that the cluster is secured. A secure Consul cluster should include the following:

1. TLS Encryption for RPC communication between Consul clients and servers.
2. Gossip Encryption for encrypting gossip traffic.
3. Access Control (ACLs) for authentication and authorization for Consul clients and services on the mesh.

NOTE: In this topic, we assume that you have already configured your Consul server with the security-related features.

Prerequisites

• You should already have followed the installation instructions to understand how to define the necessary components of the task definition for Consul on ECS.
• You should be familiar with specifying sensitive data on ECS.
• You should be familiar with configuring Consul’s secure features, including how to create ACL tokens and policies. Refer to the following Learn Guides for an introduction and the ACL system documentation for more information.

ACL Tokens

You must create two types of ACL tokens for Consul on ECS:

• Client tokens: used by the consul-client containers to join the Consul cluster
• Service tokens: used by sidecar containers for service registration and health syncing

The following sections describe the ACL polices which must be associated with these token types.

NOTE: This section describes how operators would create ACL tokens by hand. To ease operator burden, the ACL Controller can automatically create ACL tokens for Consul on ECS. Refer to the ACL Controller page for installation details.

Create Consul client token

You must create a token for the Consul client. This is a shared token used by the consul-client containers to join the Consul cluster.

The following is the ACL policy needed for the Consul client token:

node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}

This policy allows node:write for any node name, which is necessary because the Consul node names on ECS are not known until runtime.

Create service tokens

Service tokens should be associated with a service identity. The service identity includes service:write permissions for the service and sidecar proxy.

The following example shows how to use the Consul CLI to create a service token for a service named example-client-app:

consul acl token create -service-identity=example-client-app ...

NOTE: You will need to create one service token for each registered Consul service in ECS, including when new services are added to the service mesh.

Secret storage

You should securely store the following secrets in order to make them available to ECS tasks.

1. Consul Server CA certificate
2. Consul gossip encryption key
3. Consul client ACL token
4. Consul service ACL tokens (one per service)

These secrets can be securely stored and passed to ECS tasks using either of the following AWS secret services:

• AWS Systems Manager Parameter Store
• AWS Secrets Manager

Once the secrets are stored they can be referenced using their ARN. The following shows example secret ARNs when using AWS Secrets Manager:

Configure consul-client

The following secrets must be passed to the consul-client container:

• Consul server CA certificate
• Gossip encryption key
• Consul client ACL token

The following example shows how to include these secrets in the task definition. The secrets list specifies environment variable names that will be set to the secret values for this container. ECS automatically fetches the secret values specified in the valueFrom fields during task provisioning.

{
  "containerDefinitions": [
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      "secrets": [
        {
          "name": "CONSUL_CACERT",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-ca-cert"
        },
        {
          "name": "CONSUL_GOSSIP_ENCRYPTION_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-gossip-key"
        },
        {
          "name": "AGENT_TOKEN",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-client-token"
        }
      ]
    },
    ...
  ]
}

Next, update Consul configuration options to pass the secrets to the Consul client.

The following is an example of the additional content to include in the consul-client startup script. Refer to the install page for the remainder of the startup script and how to pass this script to the container.

...
# Write the CA Cert to a file
echo "$CONSUL_CACERT" > /tmp/consul-ca-cert.pem
# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
...
# Configure gossip encryption key
encrypt = "$CONSUL_GOSSIP_ENCRYPTION_KEY"
# Configure TLS settings
auto_encrypt = {
  tls = true
  ip_san = ["$ECS_IPV4"]
}
ca_file = "/tmp/consul-ca-cert.pem"
verify_outgoing = true
# Configure ACLs
acl {
  enabled = true
  default_policy = "deny"
  down_policy = "async-cache"
  tokens {
    agent = "$AGENT_TOKEN"
  }
}
EOF

The following table describes the additional fields that must be included in the Consul client configuration file.

Configure consul-ecs-mesh-init and consul-ecs-health-sync

Both consul-ecs-mesh-init and consul-ecs-health-sync containers need to be configured with the service ACL token. This allows these containers to make HTTP API requests to the local Consul client for service registration and health syncing.

The following shows how to set the CONSUL_HTTP_TOKEN variable to the service token for the example-client-app service, if the token is stored in AWS Secrets Manager.

{
  "containerDefinitions": [
    {
      "secrets": [
        {
          "name": "CONSUL_HTTP_TOKEN",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-example-client-app-token"
        }
      ]
    },
    ...
  ],
  ...
}