The following guide shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster

Cluster bootstrapping in Container Linux is simplest with Ignition; coreos-metadata.service dynamically fetches the machine’s IP for discovery. Note that etcd’s discovery service protocol is only meant for bootstrapping, and cannot be used with runtime reconfiguration or cluster monitoring.

The Container Linux Config Transpiler compiles etcd configuration files into Ignition configuration files:

  1. etcd:
  2.   version: 3.2.0
  3.   name: s1
  4.   data_dir: /var/lib/etcd
  5.   advertise_client_urls:       http://{PUBLIC_IPV4}:2379
  6.   initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
  7.   listen_client_urls:          http://0.0.0.0:2379
  8.   listen_peer_urls:            http://{PRIVATE_IPV4}:2380
  9.   discovery:                   https://discovery.etcd.io/<token>

ct would produce the following Ignition Config:

  1. $ ct --platform=gce --in-file /tmp/ct-etcd.cnf
  2. {"ignition":{"version":"2.0.0","config"...
  1. {
  2.   "ignition":{"version":"2.0.0","config":{}},
  3.   "storage":{},
  4.   "systemd":{
  5.     "units":[{
  6.       "name":"etcd-member.service",
  7.       "enable":true,
  8.       "dropins":[{
  9.         "name":"20-clct-etcd-member.conf",
  10.         "contents":"[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.1.8\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --name=\"s1\" \\\n  --data-dir=\"/var/lib/etcd\" \\\n  --listen-peer-urls=\"http://${COREOS_GCE_IP_LOCAL_0}:2380\" \\\n  --listen-client-urls=\"http://0.0.0.0:2379\" \\\n  --initial-advertise-peer-urls=\"http://${COREOS_GCE_IP_LOCAL_0}:2380\" \\\n  --advertise-client-urls=\"http://${COREOS_GCE_IP_EXTERNAL_0}:2379\" \\\n  --discovery=\"https://discovery.etcd.io/\u003ctoken\u003e\""}]}]},
  11.       "networkd":{},
  12.       "passwd":{}}

To avoid accidental misconfiguration, the transpiler helpfully verifies etcd configurations when generating Ignition files:

  1. etcd:
  2.   version: 3.2.0
  3.   name: s1
  4.   data_dir_x: /var/lib/etcd
  5.   advertise_client_urls:       http://{PUBLIC_IPV4}:2379
  6.   initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
  7.   listen_client_urls:          http://0.0.0.0:2379
  8.   listen_peer_urls:            http://{PRIVATE_IPV4}:2380
  9.   discovery:                   https://discovery.etcd.io/<token>
  1. $ ct --platform=gce --in-file /tmp/ct-etcd.cnf
  2. warning at line 3, column 2
  3. Config has unrecognized key: data_dir_x

See Container Linux Provisioning for more details.

etcd 3.x service

Container Linux does not include etcd 3.x binaries by default. Different versions of etcd 3.x can be fetched via etcd-member.service.

Confirm unit file exists:

  1. systemctl cat etcd-member.service

Check if the etcd service is running:

  1. systemctl status etcd-member.service

Example systemd drop-in unit to override the default service settings:

  1. cat > /tmp/20-cl-etcd-member.conf <<EOF
  2. [Service]
  3. Environment="ETCD_IMAGE_TAG=v3.2.0"
  4. Environment="ETCD_DATA_DIR=/var/lib/etcd"
  5. Environment="ETCD_SSL_DIR=/etc/ssl/certs"
  6. Environment="ETCD_OPTS=--name s1 \
  7.   --listen-client-urls https://10.240.0.1:2379 \
  8.   --advertise-client-urls https://10.240.0.1:2379 \
  9.   --listen-peer-urls https://10.240.0.1:2380 \
  10.   --initial-advertise-peer-urls https://10.240.0.1:2380 \
  11.   --initial-cluster s1=https://10.240.0.1:2380,s2=https://10.240.0.2:2380,s3=https://10.240.0.3:2380 \
  12.   --initial-cluster-token mytoken \
  13.   --initial-cluster-state new \
  14.   --client-cert-auth \
  15.   --trusted-ca-file /etc/ssl/certs/etcd-root-ca.pem \
  16.   --cert-file /etc/ssl/certs/s1.pem \
  17.   --key-file /etc/ssl/certs/s1-key.pem \
  18.   --peer-client-cert-auth \
  19.   --peer-trusted-ca-file /etc/ssl/certs/etcd-root-ca.pem \
  20.   --peer-cert-file /etc/ssl/certs/s1.pem \
  21.   --peer-key-file /etc/ssl/certs/s1-key.pem \
  22.   --auto-compaction-retention 1"
  23. EOF
  24. mv /tmp/20-cl-etcd-member.conf /etc/systemd/system/etcd-member.service.d/20-cl-etcd-member.conf

Or use a Container Linux Config:

  1. systemd:
  2.   units:
  3.     - name: etcd-member.service
  4.       dropins:
  5.         - name: conf1.conf
  6.           contents: |
  7.             [Service]
  8.             Environment="ETCD_SSL_DIR=/etc/ssl/certs"
  10. etcd:
  11.   version: 3.2.0
  12.   name: s1
  13.   data_dir: /var/lib/etcd
  14.   listen_client_urls:          https://0.0.0.0:2379
  15.   advertise_client_urls:       https://{PUBLIC_IPV4}:2379
  16.   listen_peer_urls:            https://{PRIVATE_IPV4}:2380
  17.   initial_advertise_peer_urls: https://{PRIVATE_IPV4}:2380
  18.   initial_cluster:             s1=https://{PRIVATE_IPV4}:2380,s2=https://10.240.0.2:2380,s3=https://10.240.0.3:2380
  19.   initial_cluster_token:       mytoken
  20.   initial_cluster_state:       new
  21.   client_cert_auth:            true
  22.   trusted_ca_file:             /etc/ssl/certs/etcd-root-ca.pem
  23.   cert_file:                   /etc/ssl/certs/s1.pem
  24.   key_file:                    /etc/ssl/certs/s1-key.pem
  25.   peer_client_cert_auth:       true
  26.   peer_trusted_ca_file:        /etc/ssl/certs/etcd-root-ca.pem
  27.   peer_cert_file:              /etc/ssl/certs/s1.pem
  28.   peer_key_file:               /etc/ssl/certs/s1-key.pem
  29.   auto_compaction_retention:   1
  1. $ ct --platform=gce --in-file /tmp/ct-etcd.cnf
  2. {"ignition":{"version":"2.0.0","config"...

To see all runtime drop-in changes for system units:

  1. systemd-delta --type=extended

To enable and start:

  1. systemctl daemon-reload
  2. systemctl enable --now etcd-member.service

To see the logs:

  1. journalctl --unit etcd-member.service --lines 10

To stop and disable the service:

  1. systemctl disable --now etcd-member.service

etcd 2.x service

Container Linux includes a unit file etcd2.service for etcd 2.x, which will be removed in the near future. See Container Linux FAQ for more details.

Confirm unit file is installed:

  1. systemctl cat etcd2.service

Check if the etcd service is running:

  1. systemctl status etcd2.service

To stop and disable:

  1. systemctl disable --now etcd2.service