How to configure circuit breaking

Installing the demo program

If you haven’t installed the demo program yet, Please refer to Quick Start to install Aeraki, Istio, and the demo.

After installation, you can see that the following two NSs have been added to the cluster, and the Dubbo and Thrift demo applications are installed in these two NSs. You can choose either of them to test.

  1. ➜  ~ kubectl get ns|grep meta
  2. meta-dubbo        Active   16m
  3. meta-thrift       Active   16m

Simulate thrift service call failure

Create a thrift-sample-server-fake deployment by using the command below，The deployment wil create an endpoint under thrift-sample-server service. As you can see from the yaml file below, there is no thrift sample server in the deployment, but rather an nginx container.

  1. kubectl apply -f- <<EOF
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: thrift-sample-server-fake
  6.   namespace: meta-thrift
  7.   labels:
  8.     app: thrift-sample-server
  9. spec:
  10.   selector:
  11.     matchLabels:
  12.       app: thrift-sample-server
  13.   replicas: 1
  14.   template:
  15.     metadata:
  16.       annotations:
  17.         sidecar.istio.io/bootstrapOverride: aeraki-bootstrap-config
  18.         sidecar.istio.io/proxyImage: aeraki/meta-protocol-proxy:1.0.1
  19.         sidecar.istio.io/rewriteAppHTTPProbers: "false"
  20.       labels:
  21.         app: thrift-sample-server
  22.     spec:
  23.       containers:
  24.         - name: thrift-sample-server
  25.           image: nginx
  26.           ports:
  27.             - containerPort: 9090
  28. EOF

Now there are three endpoint in the thrift-sample-server service, among them the thrift-sample-server-fake deployment have no real thrift server, so the corresponding endpoint “172.19.0.102” can not handle client requests. From the client log, you can see that there is an error message once in every three requests:

  1. ➜  ~  aerakictl_app_log client meta-thrift --tail 0 -f
  2. Connected to thrift-sample-server
  3. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  4. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  5. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  6.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  7.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  8.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  9.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  10. Connected to thrift-sample-server
  11. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  12. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  13. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  14.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  15.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  16.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  17.     at org.aeraki.HelloClient.main(HelloClient.java:44)

Create a circuit break rule

Create a Destination Rule by using the command below: When there’re 5 consecutive errors, the corresponding upstream host will be removed from the cluster’s load balancing pool.

  1. kubectl apply -f- <<EOF
  2. apiVersion: networking.istio.io/v1beta1
  3. kind: DestinationRule
  4. metadata:
  5.   name: thrift-sample-server
  6.   namespace: meta-thrift
  7. spec:
  8.   host: thrift-sample-server
  9.   trafficPolicy:
  10.     outlierDetection:
  11.       baseEjectionTime: 15m
  12.       consecutive5xxErrors: 5
  13.       interval: 5m
  14. EOF

At this point, if you look at the output of the client, you can see that the client no longer sends requests to the failed endpoint “172.19.0.102” after the number of errors specified by the fusion rule.

  1. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  2.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  3.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  4.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  5.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  6. Connected to thrift-sample-server
  7. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  8. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  9. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  10.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  11.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  12.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  13.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  14. Connected to thrift-sample-server
  15. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  16. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  17. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  18.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  19.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  20.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  21.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  22. Connected to thrift-sample-server
  23. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  24. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  25. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  26.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  27.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  28.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  29.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  30. Connected to thrift-sample-server
  31. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  32. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  33. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  34.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  35.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  36.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  37.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  38. Connected to thrift-sample-server
  39. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  40. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  41. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  42. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  43. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  44. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  45. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  46. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  47. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  48. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  49. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10

Understand what happened

By looking at envoy’s stats output, you can see that thrift-sample-server had consecutive request errors, and the failed host has been removed from the cluster’s load balancing group.

  1. aerakictl_sidecar_stats client  meta-thrift|grep -i outlier
  2. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_active: 1
  3. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_consecutive_5xx: 1
  4. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_detected_consecutive_5xx: 1
  5. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_consecutive_5xx: 1
  6. luster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_total: 1
  7. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_total: 1

Last modified May 9, 2022: Update circuit-breaking.md (c6e6190)