Use Filebeat to collect logs of Karmada member clusters

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or kafka for indexing.

This document demonstrates how to use the Filebeat to collect logs of Karmada member clusters.

Start up Karmada clusters

You just need to clone Karmada repo, and run the following script in Karmada directory.

  1. hack/local-up-karmada.sh

Start Filebeat

  1. Create resource objects of Filebeat, the content is as follows. You can specify a list of inputs in the filebeat.inputs section of the filebeat.yml. Inputs specify how Filebeat locates and processes input data, also you can configure Filebeat to write to a specific output by setting options in the Outputs section of the filebeat.yml config file. The example will collect the log information of each container and write the collected logs to a file. For more detailed information about the input and output configuration, please refer to: https://github.com/elastic/beats/tree/master/filebeat/docs

    1. apiVersion: v1
    2. kind: Namespace
    3. metadata:
    4.   name: logging
    5. ---
    6. apiVersion: v1
    7. kind: ServiceAccount
    8. metadata:
    9.   name: filebeat
    10.   namespace: logging
    11.   labels:
    12.     k8s-app: filebeat
    13. ---
    14. apiVersion: rbac.authorization.k8s.io/v1
    15. kind: ClusterRole
    16. metadata:
    17.   name: filebeat
    18. rules:
    19. - apiGroups: [""] # "" indicates the core API group
    20.   resources:
    21.   - namespaces
    22.   - pods
    23.   verbs:
    24.   - get
    25.   - watch
    26.   - list
    27. ---
    28. apiVersion: rbac.authorization.k8s.io/v1
    29. kind: ClusterRoleBinding
    30. metadata:
    31.   name: filebeat
    32. subjects:
    33. - kind: ServiceAccount
    34.   name: filebeat
    35.   namespace: kube-system
    36. roleRef:
    37.   kind: ClusterRole
    38.   name: filebeat
    39.   apiGroup: rbac.authorization.k8s.io
    40. ---
    41. apiVersion: v1
    42. kind: ConfigMap
    43. metadata:
    44.   name: filebeat-config
    45.   namespace: logging
    46.   labels:
    47.     k8s-app: filebeat
    48.     kubernetes.io/cluster-service: "true"
    49. data:
    50.   filebeat.yml: |-
    51.     filebeat.inputs:
    52.     - type: container
    53.       paths:
    54.         - /var/log/containers/*.log
    55.       processors:
    56.         - add_kubernetes_metadata:
    57.             host: ${NODE_NAME}
    58.             matchers:
    59.             - logs_path:
    60.                 logs_path: "/var/log/containers/"
    61.     # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    62.     #filebeat.autodiscover:
    63.     #  providers:
    64.     #    - type: kubernetes
    65.     #      node: ${NODE_NAME}
    66.     #      hints.enabled: true
    67.     #      hints.default_config:
    68.     #        type: container
    69.     #        paths:
    70.     #          - /var/log/containers/*${data.kubernetes.container.id}.log
    72.     processors:
    73.       - add_cloud_metadata:
    74.       - add_host_metadata:
    76.     #output.elasticsearch:
    77.     #  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
    78.     #  username: ${ELASTICSEARCH_USERNAME}
    79.     #  password: ${ELASTICSEARCH_PASSWORD}
    80.     output.file:
    81.         path: "/tmp/filebeat"
    82.         filename: filebeat
    83. ---
    84. apiVersion: apps/v1
    85. kind: DaemonSet
    86. metadata:
    87.   name: filebeat
    88.   namespace: logging
    89.   labels:
    90.     k8s-app: filebeat
    91. spec:
    92.   selector:
    93.     matchLabels:
    94.       k8s-app: filebeat
    95.   template:
    96.     metadata:
    97.       labels:
    98.         k8s-app: filebeat
    99.     spec:
    100.       serviceAccountName: filebeat
    101.       terminationGracePeriodSeconds: 30
    102.       tolerations:
    103.       - effect: NoSchedule
    104.         key: node-role.kubernetes.io/master
    105.       containers:
    106.       - name: filebeat
    107.         image: docker.elastic.co/beats/filebeat:8.0.0-beta1-amd64
    108.         imagePullPolicy: IfNotPresent
    109.         args: [  "-c", "/usr/share/filebeat/filebeat.yml",  "-e",]
    110.         env:
    111.         - name: NODE_NAME
    112.           valueFrom:
    113.             fieldRef:
    114.               fieldPath: spec.nodeName
    115.         securityContext:
    116.           runAsUser: 0
    117.         resources:
    118.           limits:
    119.             memory: 200Mi
    120.           requests:
    121.             cpu: 100m
    122.             memory: 100Mi
    123.         volumeMounts:
    124.         - name: config
    125.           mountPath: /usr/share/filebeat/filebeat.yml
    126.           readOnly: true
    127.           subPath: filebeat.yml
    128.         - name: inputs
    129.           mountPath: /usr/share/filebeat/inputs.d
    130.           readOnly: true
    131.         - name: data
    132.           mountPath: /usr/share/filebeat/data
    133.         - name: varlibdockercontainers
    134.           mountPath: /var/lib/docker/containers
    135.           readOnly: true
    136.         - name: varlog
    137.           mountPath: /var/log
    138.           readOnly: true
    139.       volumes:
    140.       - name: config
    141.         configMap:
    142.           defaultMode: 0600
    143.           name: filebeat-config
    144.       - name: varlibdockercontainers
    145.         hostPath:
    146.           path: /var/lib/docker/containers
    147.       - name: varlog
    148.         hostPath:
    149.           path: /var/log
    150.       - name: inputs
    151.         configMap:
    152.           defaultMode: 0600
    153.           name: filebeat-config
    154.       # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
    155.       - name: data
    156.         hostPath:
    157.           path: /var/lib/filebeat-data
    158.           type: DirectoryOrCreate

  2. Run the below command to execute Karmada PropagationPolicy and ClusterPropagationPolicy.

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: policy.karmada.io/v1alpha1
    3. kind: PropagationPolicy
    4. metadata:
    5.   name: filebeat-propagation
    6.   namespace: logging
    7. spec:
    8.   resourceSelectors:
    9.     - apiVersion: v1
    10.       kind: Namespace
    11.       name: logging
    12.     - apiVersion: v1
    13.       kind: ServiceAccount
    14.       name: filebeat
    15.       namespace: logging
    16.     - apiVersion: v1
    17.       kind: ConfigMap
    18.       name: filebeat-config
    19.       namespace: logging
    20.     - apiVersion: apps/v1
    21.       kind: DaemonSet
    22.       name: filebeat
    23.       namespace: logging
    24.   placement:
    25.     clusterAffinity:
    26.       clusterNames:
    27.         - member1
    28.         - member2
    29.         - member3
    30. EOF
    31. cat <<EOF | kubectl apply -f -
    32. apiVersion: policy.karmada.io/v1alpha1
    33. kind: ClusterPropagationPolicy
    34. metadata:
    35.   name: filebeatsrbac-propagation
    36. spec:
    37.   resourceSelectors:
    38.     - apiVersion: rbac.authorization.k8s.io/v1
    39.       kind: ClusterRole
    40.       name: filebeat
    41.     - apiVersion: rbac.authorization.k8s.io/v1
    42.       kind: ClusterRoleBinding
    43.       name: filebeat
    44.   placement:
    45.     clusterAffinity:
    46.       clusterNames:
    47.       - member1
    48.       - member2
    49.       - member3
    50. EOF

  3. Obtain the collected logs according to the output configuration of the filebeat.yml.

Reference