Dynamic configuration (control plane)

Requirements

Sandbox environment

Setup your sandbox environment with Docker and Docker Compose, and clone the Envoy repository with Git.

curl

Used to make HTTP requests.

jq

Parse json output from the upstream echo servers.

This example walks through configuring Envoy using the Go Control Plane reference implementation.

It demonstrates how configuration provided to Envoy persists, even when the control plane is not available, and provides a trivial example of how to update Envoy’s configuration dynamically.

Step 1: Start the proxy container

Change directory to examples/dynamic-config-cp in the Envoy repository.

First build the containers and start the proxy container.

This should also start two upstream HTTP echo servers, service1 and service2.

The control plane has not yet been started.

  1. $ pwd
  2. envoy/examples/dynamic-config-cp
  3. $ docker-compose build --pull
  4. $ docker-compose up -d proxy
  5. $ docker-compose ps
  7.        Name                            Command                State                     Ports
  8. ------------------------------------------------------------------------------------------------------------------------
  9. dynamic-config-cp_proxy_1      /docker-entrypoint.sh /usr ... Up      0.0.0.0:10000->10000/tcp, 0.0.0.0:19000->19000/tcp
  10. dynamic-config-cp_service1_1   /bin/echo-server               Up      8080/tcp
  11. dynamic-config-cp_service2_1   /bin/echo-server               Up      8080/tcp

Step 2: Check initial config and web response

As we have not yet started the control plane, nothing should be responding on port 10000.

  1. $ curl http://localhost:10000
  2. curl: (56) Recv failure: Connection reset by peer

Dump the proxy’s static_clusters configuration and you should see the cluster named xds_cluster configured for the control plane:

  1. $ curl -s http://localhost:19000/config_dump  | jq '.configs[1].static_clusters'
  1. [
  2.   {
  3.     "cluster": {
  4.       "@type": "type.googleapis.com/envoy.api.v2.Cluster",
  5.       "name": "xds_cluster",
  6.       "type": "STRICT_DNS",
  7.       "connect_timeout": "1s",
  8.       "http2_protocol_options": {},
  9.       "load_assignment": {
  10.         "cluster_name": "xds_cluster",
  11.         "endpoints": [
  12.           {
  13.             "lb_endpoints": [
  14.               {
  15.                 "endpoint": {
  16.                   "address": {
  17.                     "socket_address": {
  18.                       "address": "go-control-plane",
  19.                       "port_value": 18000
  20.                     }
  21.                   }
  22.                 }
  23.               }
  24.             ]
  25.           }
  26.         ]
  27.       }
  28.     },
  29.     "last_updated": "2020-10-25T20:20:54.897Z"
  30.   }
  31. ]

No dynamic_active_clusters have been configured yet:

  1. $ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
  2. null

Step 3: Start the control plane

Start up the go-control-plane service.

You may need to wait a moment or two for it to become healthy.

  1. $ docker-compose up --build -d go-control-plane
  2. $ docker-compose ps
  4.         Name                                Command                  State                    Ports
  5. -------------------------------------------------------------------------------------------------------------------------------------
  6. dynamic-config-cp_go-control-plane_1  bin/example -debug             Up (healthy)
  7. dynamic-config-cp_proxy_1             /docker-entrypoint.sh /usr ... Up            0.0.0.0:10000->10000/tcp, 0.0.0.0:19000->19000/tcp
  8. dynamic-config-cp_service1_1          /bin/echo-server               Up            8080/tcp
  9. dynamic-config-cp_service2_1          /bin/echo-server               Up            8080/tcp

Step 4: Query the proxy

Once the control plane has started and is healthy, you should be able to make a request to port 10000, which will be served by service1.

  1. $ curl http://localhost:10000
  2. Request served by service1
  4. HTTP/1.1 GET /
  6. Host: localhost:10000
  7. Accept: */*
  8. X-Forwarded-Proto: http
  9. X-Request-Id: 1d93050e-f39c-4602-90f8-a124d6e78d26
  10. X-Envoy-Expected-Rq-Timeout-Ms: 15000
  11. Content-Length: 0
  12. User-Agent: curl/7.72.0

Step 5: Dump Envoy’s dynamic_active_clusters config

If you now dump the proxy’s dynamic_active_clusters configuration, you should see it is configured with the example_proxy_cluster pointing to service1, and a version of 1.

  1. $ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
  1. [
  2.   {
  3.     "version_info": "1",
  4.     "cluster": {
  5.       "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  6.       "name": "example_proxy_cluster",
  7.       "type": "LOGICAL_DNS",
  8.       "connect_timeout": "5s",
  9.       "dns_lookup_family": "V4_ONLY",
  10.       "load_assignment": {
  11.         "cluster_name": "example_proxy_cluster",
  12.         "endpoints": [
  13.           {
  14.             "lb_endpoints": [
  15.               {
  16.                 "endpoint": {
  17.                   "address": {
  18.                     "socket_address": {
  19.                       "address": "service1",
  20.                       "port_value": 8080
  21.                     }
  22.                   }
  23.                 }
  24.               }
  25.             ]
  26.           }
  27.         ]
  28.       }
  29.     },
  30.     "last_updated": "2020-10-25T20:37:05.838Z"
  31.   }
  32. ]

Step 6: Stop the control plane

Stop the go-control-plane service:

  1. $ docker-compose stop go-control-plane

The Envoy proxy should continue proxying responses from service1.

  1. $ curl http://localhost:10000 | grep "served by"
  2. Request served by service1

Step 7: Edit go file and restart the control plane

The example setup starts the go-control-plane service with a custom resource.go file which specifies the configuration provided to Envoy.

Update this to have Envoy proxy instead to service2.

Edit resource.go in the dynamic configuration example folder and change the UpstreamHost from service1 to service2:

  1. 33const (
  2. 34    ClusterName  = "example_proxy_cluster"
  3. 35    RouteName    = "local_route"
  4. 36    ListenerName = "listener_0"
  5. 37    ListenerPort = 10000
  6. 38    UpstreamHost = "service1"
  7. 39    UpstreamPort = 8080
  8. 40)

Further down in this file you must also change the configuration snapshot version number from "1" to "2" to ensure Envoy sees the configuration as newer:

  1. 164func GenerateSnapshot() cache.Snapshot {
  2. 165    return cache.NewSnapshot(
  3. 166        "1",
  4. 167        []types.Resource{}, // endpoints
  5. 168        []types.Resource{makeCluster(ClusterName)},
  6. 169        []types.Resource{makeRoute(RouteName, ClusterName)},
  7. 170        []types.Resource{makeHTTPListener(ListenerName, RouteName)},
  8. 171        []types.Resource{}, // runtimes
  9. 172        []types.Resource{}, // secrets
  10. 173        []types.Resource{}, // extensions configs
  11. 174    )

Now rebuild and restart the control plane:

  1. $ docker-compose up --build -d go-control-plane

You may need to wait a moment or two for the go-control-plane service to become healthy again.

Step 8: Check Envoy uses the updated configuration

Now when you make a request to the proxy it should be served by the service2 upstream service.

  1. $ curl http://localhost:10000 | grep "served by"
  2. Request served by service2

Dumping the dynamic_active_clusters you should see the cluster configuration now has a version of 2, and example_proxy_cluster is configured to proxy to service2:

  1. $ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
  1. [
  2.   {
  3.     "version_info": "2",
  4.     "cluster": {
  5.       "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  6.       "name": "example_proxy_cluster",
  7.       "type": "LOGICAL_DNS",
  8.       "connect_timeout": "5s",
  9.       "dns_lookup_family": "V4_ONLY",
  10.       "load_assignment": {
  11.         "cluster_name": "example_proxy_cluster",
  12.         "endpoints": [
  13.           {
  14.             "lb_endpoints": [
  15.               {
  16.                 "endpoint": {
  17.                   "address": {
  18.                     "socket_address": {
  19.                       "address": "service2",
  20.                       "port_value": 8080
  21.                     }
  22.                   }
  23.                 }
  24.               }
  25.             ]
  26.           }
  27.         ]
  28.       }
  29.     },
  30.     "last_updated": "2020-10-26T14:35:17.360Z"
  31.   }
  32. ]

See also

Dynamic configuration (control plane) quick start guide

Quick start guide to dynamic configuration of Envoy with a control plane.

Envoy admin quick start guide

Quick start guide to the Envoy admin interface.

Dynamic configuration (filesystem) sandbox

Configure Envoy using filesystem-based dynamic configuration.

Go control plane

Reference implementation of Envoy control plane written in go.