如何设置熔断规则

安装示例程序

如果你还没有安装示例程序，请参照 快速开始 安装 Aeraki，Istio 及示例程序。

安装完成后，可以看到集群中增加了下面两个 NS，这两个 NS 中分别安装了基于 MetaProtocol 实现的 Dubbo 和 Thrift 协议的示例程序。 你可以选用任何一个程序进行测试。

  1. ➜  ~ kubectl get ns|grep meta
  2. meta-dubbo        Active   16m
  3. meta-thrift       Active   16m

模拟 thrift 服务调用失败

通过下面的命令创建一个 thrift-sample-server-fake deployment，该 deployment 将在 thrift-sample-server service 中增加一个 endpoint。从下面的 yaml 可以看到，该 deployment 中并没有 thrift sample server，而是一个 nginx 容器。

  1. kubectl apply -f- <<EOF
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: thrift-sample-server-fake
  6.   namespace: meta-thrift
  7.   labels:
  8.     app: thrift-sample-server
  9. spec:
  10.   selector:
  11.     matchLabels:
  12.       app: thrift-sample-server
  13.   replicas: 1
  14.   template:
  15.     metadata:
  16.       annotations:
  17.         sidecar.istio.io/bootstrapOverride: aeraki-bootstrap-config
  18.         sidecar.istio.io/proxyImage: aeraki/meta-protocol-proxy:1.0.1
  19.         sidecar.istio.io/rewriteAppHTTPProbers: "false"
  20.       labels:
  21.         app: thrift-sample-server
  22.     spec:
  23.       containers:
  24.         - name: thrift-sample-server
  25.           image: nginx
  26.           ports:
  27.             - containerPort: 9090
  28. EOF

现在 thrift-sample-server service 中有三个 endpoint，其中 thrift-sample-server-fake 这个 deployment 中没有 thrift sample server，因此其对应的 endpoint “172.19.0.102” 并不能处理客户端的请求。从客户端的日志中可以看到每三次请求中就有一次错误信息：

  1. ➜  ~  aerakictl_app_log client meta-thrift --tail 0 -f
  2. Connected to thrift-sample-server
  3. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  4. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  5. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  6.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  7.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  8.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  9.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  10. Connected to thrift-sample-server
  11. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  12. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  13. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  14.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  15.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  16.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  17.     at org.aeraki.HelloClient.main(HelloClient.java:44)

创建熔断规则

通过下面的命令创建一个 Destination Rule：当请求连续出现 5 次错误后，就将出错的 upstream host 从 cluster 的负载均衡分组中移除。

  1. kubectl apply -f- <<EOF
  2. apiVersion: networking.istio.io/v1beta1
  3. kind: DestinationRule
  4. metadata:
  5.   name: thrift-sample-server
  6.   namespace: meta-thrift
  7. spec:
  8.   host: thrift-sample-server
  9.   trafficPolicy:
  10.     outlierDetection:
  11.       baseEjectionTime: 15m
  12.       consecutive5xxErrors: 5
  13.       interval: 5m
  14. EOF

此时查看客户端的输出，可以看到客户端在熔断规则指定的错误次数后，不再将请求发送到出错的 endpoint “172.19.0.102”。

  1. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  2.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  3.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  4.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  5.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  6. Connected to thrift-sample-server
  7. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  8. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  9. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  10.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  11.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  12.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  13.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  14. Connected to thrift-sample-server
  15. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  16. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  17. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  18.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  19.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  20.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  21.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  22. Connected to thrift-sample-server
  23. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  24. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  25. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  26.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  27.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  28.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  29.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  30. Connected to thrift-sample-server
  31. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  32. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  33. org.apache.thrift.TApplicationException: meta protocol upstream request: remote connection failure '172.19.0.102:9090'
  34.     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
  35.     at org.aeraki.HelloService$Client.recv_sayHello(HelloService.java:61)
  36.     at org.aeraki.HelloService$Client.sayHello(HelloService.java:48)
  37.     at org.aeraki.HelloClient.main(HelloClient.java:44)
  38. Connected to thrift-sample-server
  39. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  40. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  41. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  42. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  43. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  44. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  45. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  46. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  47. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10
  48. Hello Aeraki, response from thrift-sample-server-v2-86db7567f-z8vwz/172.19.0.98
  49. Hello Aeraki, response from thrift-sample-server-v1-c5cccb876-jmhhf/172.19.0.10

理解原理

通过查看 envoy 的 stats 输出，可以看到 thrift-sample-server 发生了连续的请求错误，该出错的 host 被移除了该 cluster 的负载均衡分组。

  1. aerakictl_sidecar_stats client  meta-thrift|grep -i outlier
  2. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_active: 1
  3. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_consecutive_5xx: 1
  4. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_detected_consecutive_5xx: 1
  5. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_consecutive_5xx: 1
  6. luster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_enforced_total: 1
  7. cluster.outbound|9090||thrift-sample-server.meta-thrift.svc.cluster.local.outlier_detection.ejections_total: 1

最后修改 March 22, 2022: circuit breaking (f12858b)