Demo

Procedures for working with an etcd cluster

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

01_etcd_clustering_2016050601

On each etcd node, specify the cluster members:

  1. TOKEN=token-01
  2. CLUSTER_STATE=new
  3. NAME_1=machine-1
  4. NAME_2=machine-2
  5. NAME_3=machine-3
  6. HOST_1=10.240.0.17
  7. HOST_2=10.240.0.18
  8. HOST_3=10.240.0.19
  9. CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

  1. # For machine 1
  2. THIS_NAME=${NAME_1}
  3. THIS_IP=${HOST_1}
  4. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  5.     --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  6.     --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  7.     --initial-cluster ${CLUSTER} \
  8.     --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  10. # For machine 2
  11. THIS_NAME=${NAME_2}
  12. THIS_IP=${HOST_2}
  13. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  14.     --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  15.     --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  16.     --initial-cluster ${CLUSTER} \
  17.     --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  19. # For machine 3
  20. THIS_NAME=${NAME_3}
  21. THIS_IP=${HOST_3}
  22. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  23.     --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  24.     --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  25.     --initial-cluster ${CLUSTER} \
  26.     --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

  1. curl https://discovery.etcd.io/new?size=3
  2. https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92
  4. # grab this token
  5. TOKEN=token-01
  6. CLUSTER_STATE=new
  7. NAME_1=machine-1
  8. NAME_2=machine-2
  9. NAME_3=machine-3
  10. HOST_1=10.240.0.17
  11. HOST_2=10.240.0.18
  12. HOST_3=10.240.0.19
  13. DISCOVERY=https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92
  15. THIS_NAME=${NAME_1}
  16. THIS_IP=${HOST_1}
  17. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  18.     --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  19.     --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  20.     --discovery ${DISCOVERY} \
  21.     --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  23. THIS_NAME=${NAME_2}
  24. THIS_IP=${HOST_2}
  25. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  26.     --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  27.     --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  28.     --discovery ${DISCOVERY} \
  29.     --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  31. THIS_NAME=${NAME_3}
  32. THIS_IP=${HOST_3}
  33. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  34.     --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  35.     --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  36.     --discovery ${DISCOVERY} \
  37.     --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Now etcd is ready! To connect to etcd with etcdctl:

  1. export ETCDCTL_API=3
  2. HOST_1=10.240.0.17
  3. HOST_2=10.240.0.18
  4. HOST_3=10.240.0.19
  5. ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
  7. etcdctl --endpoints=$ENDPOINTS member list

Access etcd

02_etcdctl_access_etcd_2016051001

put command to write:

  1. etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

get to read from etcd:

  1. etcdctl --endpoints=$ENDPOINTS get foo
  2. etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo

Get by prefix

03_etcdctl_get_by_prefix_2016050501

  1. etcdctl --endpoints=$ENDPOINTS put web1 value1
  2. etcdctl --endpoints=$ENDPOINTS put web2 value2
  3. etcdctl --endpoints=$ENDPOINTS put web3 value3
  5. etcdctl --endpoints=$ENDPOINTS get web --prefix

Delete

04_etcdctl_delete_2016050601

  1. etcdctl --endpoints=$ENDPOINTS put key myvalue
  2. etcdctl --endpoints=$ENDPOINTS del key
  4. etcdctl --endpoints=$ENDPOINTS put k1 value1
  5. etcdctl --endpoints=$ENDPOINTS put k2 value2
  6. etcdctl --endpoints=$ENDPOINTS del k --prefix

Transactional write

txn to wrap multiple requests into one transaction:

05_etcdctl_transaction_2016050501

  1. etcdctl --endpoints=$ENDPOINTS put user1 bad
  2. etcdctl --endpoints=$ENDPOINTS txn --interactive
  4. compares:
  5. value("user1") = "bad"
  7. success requests (get, put, delete):
  8. del user1
  10. failure requests (get, put, delete):
  11. put user1 good

Watch

watch to get notified of future changes:

06_etcdctl_watch_2016050501

  1. etcdctl --endpoints=$ENDPOINTS watch stock1
  2. etcdctl --endpoints=$ENDPOINTS put stock1 1000
  4. etcdctl --endpoints=$ENDPOINTS watch stock --prefix
  5. etcdctl --endpoints=$ENDPOINTS put stock1 10
  6. etcdctl --endpoints=$ENDPOINTS put stock2 20

Lease

lease to write with TTL:

07_etcdctl_lease_2016050501

  1. etcdctl --endpoints=$ENDPOINTS lease grant 300
  2. # lease 2be7547fbc6a5afa granted with TTL(300s)
  4. etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
  5. etcdctl --endpoints=$ENDPOINTS get sample
  7. etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
  8. etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
  9. # or after 300 seconds
  10. etcdctl --endpoints=$ENDPOINTS get sample

Distributed locks

lock for distributed lock:

08_etcdctl_lock_2016050501

  1. etcdctl --endpoints=$ENDPOINTS lock mutex1
  3. # another client with the same name blocks
  4. etcdctl --endpoints=$ENDPOINTS lock mutex1

Elections

elect for leader election:

09_etcdctl_elect_2016050501

  1. etcdctl --endpoints=$ENDPOINTS elect one p1
  3. # another client with the same name blocks
  4. etcdctl --endpoints=$ENDPOINTS elect one p2

Cluster status

Specify the initial cluster configuration for each machine:

10_etcdctl_endpoint_2016050501

  1. etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status
  3. +------------------+------------------+---------+---------+-----------+-----------+------------+
  4. |     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
  5. +------------------+------------------+---------+---------+-----------+-----------+------------+
  6. | 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.0.0   | 45 kB   | true      |         4 |      16726 |
  7. | 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.0.0   | 45 kB   | false     |         4 |      16726 |
  8. | 10.240.0.19:2379 | 94df724b66343e6c | 3.0.0   | 45 kB   | false     |         4 |      16726 |
  9. +------------------+------------------+---------+---------+-----------+-----------+------------+
  1. etcdctl --endpoints=$ENDPOINTS endpoint health
  3. 10.240.0.17:2379 is healthy: successfully committed proposal: took = 3.345431ms
  4. 10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.767967ms
  5. 10.240.0.18:2379 is healthy: successfully committed proposal: took = 4.025451ms

Snapshot

snapshot to save point-in-time snapshot of etcd database:

11_etcdctl_snapshot_2016051001

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

  1. ENDPOINTS=$HOST_1:2379
  2. etcdctl --endpoints=$ENDPOINTS snapshot save my.db
  4. Snapshot saved at my.db
  1. etcdctl --write-out=table --endpoints=$ENDPOINTS snapshot status my.db
  3. +---------+----------+------------+------------+
  4. |  HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
  5. +---------+----------+------------+------------+
  6. | c55e8b8 |        9 |         13 | 25 kB      |
  7. +---------+----------+------------+------------+

Migrate

migrate to transform etcd v2 to v3 data:

12_etcdctl_migrate_2016061602

  1. # write key in etcd version 2 store
  2. export ETCDCTL_API=2
  3. etcdctl --endpoints=http://$ENDPOINT set foo bar
  5. # read key in etcd v2
  6. etcdctl --endpoints=$ENDPOINTS --output="json" get foo
  8. # stop etcd node to migrate, one by one
  10. # migrate v2 data
  11. export ETCDCTL_API=3
  12. etcdctl --endpoints=$ENDPOINT migrate --data-dir="default.etcd" --wal-dir="default.etcd/member/wal"
  14. # restart etcd node after migrate, one by one
  16. # confirm that the key got migrated
  17. etcdctl --endpoints=$ENDPOINTS get /foo

Member

member to add,remove,update membership:

13_etcdctl_member_2016062301

  1. # For each machine
  2. TOKEN=my-etcd-token-1
  3. CLUSTER_STATE=new
  4. NAME_1=etcd-node-1
  5. NAME_2=etcd-node-2
  6. NAME_3=etcd-node-3
  7. HOST_1=10.240.0.13
  8. HOST_2=10.240.0.14
  9. HOST_3=10.240.0.15
  10. CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
  12. # For node 1
  13. THIS_NAME=${NAME_1}
  14. THIS_IP=${HOST_1}
  15. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  16.     --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  17.     --listen-peer-urls http://${THIS_IP}:2380 \
  18.     --advertise-client-urls http://${THIS_IP}:2379 \
  19.     --listen-client-urls http://${THIS_IP}:2379 \
  20.     --initial-cluster ${CLUSTER} \
  21.     --initial-cluster-state ${CLUSTER_STATE} \
  22.     --initial-cluster-token ${TOKEN}
  24. # For node 2
  25. THIS_NAME=${NAME_2}
  26. THIS_IP=${HOST_2}
  27. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  28.     --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  29.     --listen-peer-urls http://${THIS_IP}:2380 \
  30.     --advertise-client-urls http://${THIS_IP}:2379 \
  31.     --listen-client-urls http://${THIS_IP}:2379 \
  32.     --initial-cluster ${CLUSTER} \
  33.     --initial-cluster-state ${CLUSTER_STATE} \
  34.     --initial-cluster-token ${TOKEN}
  36. # For node 3
  37. THIS_NAME=${NAME_3}
  38. THIS_IP=${HOST_3}
  39. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  40.     --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  41.     --listen-peer-urls http://${THIS_IP}:2380 \
  42.     --advertise-client-urls http://${THIS_IP}:2379 \
  43.     --listen-client-urls http://${THIS_IP}:2379 \
  44.     --initial-cluster ${CLUSTER} \
  45.     --initial-cluster-state ${CLUSTER_STATE} \
  46.     --initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

  1. # get member ID
  2. export ETCDCTL_API=3
  3. HOST_1=10.240.0.13
  4. HOST_2=10.240.0.14
  5. HOST_3=10.240.0.15
  6. etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list
  8. # remove the member
  9. MEMBER_ID=278c654c9a6dfd3b
  10. etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 \
  11.     member remove ${MEMBER_ID}
  13. # add a new member (node 4)
  14. export ETCDCTL_API=3
  15. NAME_1=etcd-node-1
  16. NAME_2=etcd-node-2
  17. NAME_4=etcd-node-4
  18. HOST_1=10.240.0.13
  19. HOST_2=10.240.0.14
  20. HOST_4=10.240.0.16 # new member
  21. etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379 \
  22.     member add ${NAME_4} \
  23.     --peer-urls=http://${HOST_4}:2380

Next, start the new member with --initial-cluster-state existing flag:

  1. # [WARNING] If the new member starts from the same disk space,
  2. # make sure to remove the data directory of the old member
  3. #
  4. # restart with 'existing' flag
  5. TOKEN=my-etcd-token-1
  6. CLUSTER_STATE=existing
  7. NAME_1=etcd-node-1
  8. NAME_2=etcd-node-2
  9. NAME_4=etcd-node-4
  10. HOST_1=10.240.0.13
  11. HOST_2=10.240.0.14
  12. HOST_4=10.240.0.16 # new member
  13. CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_4}=http://${HOST_4}:2380
  15. THIS_NAME=${NAME_4}
  16. THIS_IP=${HOST_4}
  17. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  18.     --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  19.     --listen-peer-urls http://${THIS_IP}:2380 \
  20.     --advertise-client-urls http://${THIS_IP}:2379 \
  21.     --listen-client-urls http://${THIS_IP}:2379 \
  22.     --initial-cluster ${CLUSTER} \
  23.     --initial-cluster-state ${CLUSTER_STATE} \
  24.     --initial-cluster-token ${TOKEN}

Auth

auth,user,role for authentication:

14_etcdctl_auth_2016062301

  1. export ETCDCTL_API=3
  2. ENDPOINTS=localhost:2379
  4. etcdctl --endpoints=${ENDPOINTS} role add root
  5. etcdctl --endpoints=${ENDPOINTS} role grant-permission root readwrite foo
  6. etcdctl --endpoints=${ENDPOINTS} role get root
  8. etcdctl --endpoints=${ENDPOINTS} user add root
  9. etcdctl --endpoints=${ENDPOINTS} user grant-role root root
  10. etcdctl --endpoints=${ENDPOINTS} user get root
  12. etcdctl --endpoints=${ENDPOINTS} auth enable
  13. # now all client requests go through auth
  15. etcdctl --endpoints=${ENDPOINTS} --user=root:123 put foo bar
  16. etcdctl --endpoints=${ENDPOINTS} get foo
  17. etcdctl --endpoints=${ENDPOINTS} --user=root:123 get foo
  18. etcdctl --endpoints=${ENDPOINTS} --user=root:123 get foo1

Last modified June 14, 2021: Renaming content/en/next folder to content/en/v3.5. Updating redirects, links, and config as needed. (#363) (138926b)