Best Practice for Managing Log Collection Sidecar Containers

Kubernetes Container Log Collection

As an indispensable part of any system, K8S Official Documentation also introduces various forms of log collection, summarizing the following three main types: Native approach, DaemonSet approach and Sidecar approach. All three approaches have advantages and disadvantages, and none of them can perfectly solve 100% of the problems, so they have to be fitted according to the scenarios.

The Sidecar approach deploys a separate logging agent for each POD, which is relatively more resource intensive, but more flexible and multi-tenant isolated, and is recommended for large K8S clusters or clusters serving multiple business parties as PAAS platforms.

EFK Architecture

EFK (ElasticSearch, FileBeat, Kibana) is a very popular and widely used log collection solution in the community, architecture as follows: k8s log sidecar

K8S Sidecar Model Disadvantage

As shown above, the FileBeat container is deployed in Sidecar mode in the same Pod as the business app container, and the logs are collected and uploaded to ElasticSearch by means of a shared volume, configuration as follows:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. name: test-pod
  5. spec:
  6. containers:
  7. - name: nginx
  8. image: nginx:latest
  9. volumeMounts:
  10. # Share log directory with filebeat sidecar container via volumeMount
  11. - mountPath: /var/log/nginx
  12. name: log
  13. - name: filebeat
  14. image: docker.elastic.co/beats/filebeat:7.16.2
  15. volumeMounts:
  16. - mountPath: /var/log/nginx
  17. name: log
  18. volumes:
  19. - name: log
  20. emptyDir: {}

Pod Sidecar Model: By defining specialized containers in Pods to perform the auxiliary work required by business containers (e.g. log collection, traffic proxy). Advantage: decoupling the auxiliary capacity from the business container, realizing independent release and capacity reuse. But there are also some disadvantages, as follows:

  • Business Pod contains (e.g. operation and maintenance, proxy) multiple sidecar containers, increasing the complexity of Pod configuration and the learning cost of business developers.
  • Sidecar container upgrade will lead to business Pod rebuild, and since Sidecar containers are generally the responsibility of independent middleware teams, there will be great business-side resistance if upgraded.

A Powerful Tool for Sidecar Container Management

SidecarSet is an abstract concept for sidecar container management in OpenKruise, responsible for injecting and upgrading sidecar containers in k8s cluster, and is one of the core workloads of OpenKruise. For details, please refer to SidecarSet Document

  • Automatic Injection Of Sidecar Container: Decoupling sidecar container from business Pod configuration, simplifying business development usage cost and learning cost.
  • Upgrade Sidecar Container Independently: No rebuilding Pod, upgrade Sidecar container alone, no feeling to business service.

EFK + SidecarSet(FileBeat) Practice

Install EFK (ElasticSearch, Kibana)

There is a lot of documentation in the community to install EFK, this article is mainly deployed by way of Helm, refer to Elastic Helm Charts. First of all, K8S cluster needs StorageClass for ElasticSearch PVC, this article uses the already created alibabacloud-cnfs-nas, as follows:

  1. helm-charts% kubectl get storageclass
  2. NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
  3. alibabacloud-cnfs-nas nasplugin.csi.alibabacloud.com Delete Immediate true 6d2h

Helm deploy ElasticSearch, Kibana, as follows:

  1. helm-charts% helm repo add elastic https://helm.elastic.co
  2. helm-charts% helm repo update
  3. ## Install ElasticSearch,set storage-class is alibabacloud-cnfs-nas
  4. helm-charts% helm install elasticsearch elastic/elasticsearch --version 7.16.3 --set persistence.annotations."volume.beta.kubernetes.io/storage-class"=alibabacloud-cnfs-nas -n elastic-system
  5. ## Install Kibana
  6. helm-charts% helm install kibana elastic/kibana --version 7.16.3 --set service.type=LoadBalancer -n elastic-system

FileBeat SidecarSet CRD

Create FileBeat configuration (the ConfigMap be created under business namespace), as follows:

  1. apiVersion: v1
  2. data:
  3. filebeat.yml: |
  4. filebeat.inputs:
  5. - type: log
  6. paths:
  7. - /var/log/*
  8. output.elasticsearch:
  9. host: '${NODE_NAME}'
  10. hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master.elastic-system:9200}'
  11. kind: ConfigMap
  12. metadata:
  13. name: filebeat-config

FileBeat SidecarSet Configuration, as follows:

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: SidecarSet
  3. metadata:
  4. name: filebeat-sidecarset
  5. spec:
  6. selector:
  7. matchLabels:
  8. kruise.io/inject-filebeat: "true"
  9. containers:
  10. - args:
  11. - -c
  12. - /etc/filebeat.yml
  13. - -e
  14. env:
  15. - name: POD_NAMESPACE
  16. valueFrom:
  17. fieldRef:
  18. apiVersion: v1
  19. fieldPath: metadata.namespace
  20. image: docker.elastic.co/beats/filebeat:7.16.2
  21. livenessProbe:
  22. exec:
  23. command:
  24. - sh
  25. - -c
  26. - |
  27. #!/usr/bin/env bash -e
  28. curl --fail 127.0.0.1:5066
  29. name: filebeat
  30. readinessProbe:
  31. exec:
  32. command:
  33. - sh
  34. - -c
  35. - |
  36. #!/usr/bin/env bash -e
  37. filebeat test output
  38. resources:
  39. limits:
  40. cpu: "1"
  41. memory: 200Mi
  42. requests:
  43. cpu: 100m
  44. memory: 100Mi
  45. volumeMounts:
  46. - name: config
  47. mountPath: /etc/filebeat.yml
  48. readOnly: true
  49. subPath: filebeat.yml
  50. - name: varlog
  51. mountPath: /var/log
  52. readOnly: true
  53. volumes:
  54. - name: config
  55. configMap:
  56. name: filebeat-config
  57. - name: varlog
  58. emptyDir: {}

For the scenario where machine resources are not sufficient, in order to reduce Pod resource requests, you can set sidecar container request.cpu=0. In this case, the Qos of Pod will be Burstable.

Automatic Injection Of FileBeat Sidecar Container

Nginx Deployment, only contains nginx container configuration, as follows:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. labels:
  5. app: nginx
  6. name: nginx
  7. spec:
  8. replicas: 1
  9. selector:
  10. matchLabels:
  11. app: nginx
  12. template:
  13. metadata:
  14. labels:
  15. app: nginx
  16. # need injection filebeat sidecar label
  17. kruise.io/inject-filebeat: "true"
  18. spec:
  19. containers:
  20. - name: nginx
  21. image: nginx:latest
  22. volumeMounts:
  23. # Share log directory with filebeat sidecar container via volumeMount
  24. - mountPath: /var/log/nginx
  25. name: log
  26. volumes:
  27. - name: log
  28. emptyDir: {}

After applying the nginx deployment to the k8s cluster, it was found that the created Pods were injected with the filebeat sidecar container, as follows:

  1. helm-charts% kubectl get pods nginx-5674976569-zdr7l -o yaml
  2. status:
  3. containerStatuses:
  4. - containerID: containerd://5330c2b32262de83ed387e5a932f61acc52e3896ddfcb22d626c43d82638faf3
  5. image: docker.elastic.co/beats/filebeat:7.16.2
  6. name: filebeat
  7. state:
  8. running:
  9. startedAt: "2022-03-02T12:17:15Z"
  10. - containerID: containerd://1ad335f39c134f7a66a0370a275dd95f67f5fd3d3f1fe523c955408b14887229
  11. image: docker.io/library/nginx:latest
  12. name: nginx
  13. state:
  14. running:
  15. startedAt: "2022-03-02T12:17:16Z"

Upgrade FileBeat Sidecar Container Independently (Version 7.16.2 -> 7.16.3)

Below are two windows, and on the right is a client request to access the nginx service. At this point, after changing the image in filebeat sidecarSet from 7.16.2 to 7.16.3, we find that the Pod is not rebuilt. And the nginx service is not interrupted during the completion of filebeat sidecar container image upgrade 7.16.3 (the nginx service has only one Pod instance), as follows:

k8s log sidecar

This feature relies on the ability of Kruise InPlace Update. However, upgrading sidecar independently comes with a risk, if the sidecar upgrade process fails, it will make Pod Not Ready and potentially affects the business, so SidecarSet itself provides rich progressive delivery capability to mitigate the risk. Refer to Kruise SidecarSet, as follows:

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: SidecarSet
  3. metadata:
  4. name: sidecarset
  5. spec:
  6. # ...
  7. updateStrategy:
  8. type: RollingUpdate
  9. # Maximum unavailable quantity
  10. maxUnavailable: 20%
  11. # Release in batches
  12. partition: 90
  13. # Canary release, via pod labels
  14. selector:
  15. matchLabels:
  16. # Some Pods contain canary labels,
  17. # or any other labels where a small number of pods can be selected
  18. deploy-env: canary

In addition, if it is similar to the ServiceMesh Envoy Mesh Container, you need to use the SidecarSet hot upgrade feature, Refer to SidecarSet HotUpgrade.

Argo-cd Deploy SidecarSet (Optional)

If you use Argo-cd to deploy Kruise SidecarSet, you need to configure Custom CRD Health Checks. According to this configuration, Argo-cd can implement SidecarSet Health Check, such as whether the SidecarSet is published or not, and whether the Pod ready, etc., as follows:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. labels:
  5. app.kubernetes.io/name: argocd-cm
  6. app.kubernetes.io/part-of: argocd
  7. name: argocd-cm
  8. namespace: argocd
  9. data:
  10. resource.customizations.health.apps.kruise.io_SidecarSet: |
  11. hs = {}
  12. -- if paused
  13. if obj.spec.updateStrategy.paused then
  14. hs.status = "Suspended"
  15. hs.message = "SidecarSet is Suspended"
  16. return hs
  17. end
  18. -- check sidecarSet status
  19. if obj.status ~= nil then
  20. if obj.status.observedGeneration < obj.metadata.generation then
  21. hs.status = "Progressing"
  22. hs.message = "Waiting for rollout to finish: observed sidecarSet generation less then desired generation"
  23. return hs
  24. end
  25. if obj.status.updatedPods < obj.spec.matchedPods then
  26. hs.status = "Progressing"
  27. hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
  28. return hs
  29. end
  30. if obj.status.updatedReadyPods < obj.status.updatedPods then
  31. hs.status = "Progressing"
  32. hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
  33. return hs
  34. end
  35. hs.status = "Healthy"
  36. return hs
  37. end
  38. -- if status == nil
  39. hs.status = "Progressing"
  40. hs.message = "Waiting for sidecarSet"
  41. return hs

Summary

Pod containing multiple containers will be more and more accepted by more and more developers, and thus the K8S ecosystem urgently needs a way to manage sidecar containers effectively. Kruise SidecarSet is an exploration of sidecar container management, and there are many companies in the community using Kruise SidecarSet to manage different types of sidecar containers.

While SidecarSet brings convenience, it also brings some management costs, such as: what if the Sidecar container is released at the same time with the business app container, and who owns the Pod when the containers in the Pod belong to multiple teams? Therefore, we also hope to explore with more developers in the community, and welcome everyone to provide some ideas to prosper the K8S ecology together.