Add Fluentd as a Receiver

You can use Elasticsearch, Kafka and Fluentd as log receivers in KubeSphere. This tutorial demonstrates:

  • How to deploy Fluentd as a Deployment and create the corresponding Service and ConfigMap.
  • How to add Fluentd as a log receiver to receive logs sent from Fluent Bit and then output to stdout.
  • How to verify if Fluentd receives logs successfully.

Prerequisites

  • You need a user granted a role including the permission of Cluster Management. For example, you can log in to the console as admin directly or create a new role with the permission and assign it to a user.

  • Before adding a log receiver, you need to enable any of the logging, events, or auditing components. For more information, see Enable Pluggable Components. logging is enabled as an example in this tutorial.

Step 1: Deploy Fluentd as a Deployment

Usually, Fluentd is deployed as a DaemonSet in Kubernetes to collect container logs on each node. KubeSphere chooses Fluent Bit because of its low memory footprint. Besides, Fluentd features numerous output plugins. Hence, KubeSphere chooses to deploy Fluentd as a Deployment to forward logs it receives from Fluent Bit to more destinations such as S3, MongoDB, Cassandra, MySQL, syslog and Splunk.

Run the following commands:

Note

  • The following commands create the Fluentd Deployment, Service, and ConfigMap in the default namespace and add a filter to the Fluentd ConfigMap to exclude logs from the default namespace to avoid Fluent Bit and Fluentd loop log collections.
  • Change the namespace if you want to deploy Fluentd into a different namespace.
  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: fluentd-config
  6.   namespace: default
  7. data:
  8.   fluent.conf: |-
  9.     # Receive logs sent from Fluent Bit on port 24224
  10.     <source>
  11.       @type forward
  12.       port 24224
  13.     </source>
  15.     # Because this will send logs Fluentd received to stdout,
  16.     # to avoid Fluent Bit and Fluentd loop logs collection,
  17.     # add a filter here to avoid sending logs from the default namespace to stdout again
  18.     <filter **>
  19.       @type grep
  20.       <exclude>
  21.         key $.kubernetes.namespace_name
  22.         pattern /^default$/
  23.       </exclude>
  24.     </filter>
  26.     # Send received logs to stdout for demo/test purpose only
  27.     # Various output plugins are supported to output logs to S3, MongoDB, Cassandra, MySQL, syslog, Splunk, etc.
  28.     <match **>
  29.       @type stdout
  30.     </match>    
  31. ---
  32. apiVersion: apps/v1
  33. kind: Deployment
  34. metadata:
  35.   labels:
  36.     app: fluentd
  37.   name: fluentd
  38.   namespace: default
  39. spec:
  40.   replicas: 1
  41.   selector:
  42.     matchLabels:
  43.       app: fluentd
  44.   template:
  45.     metadata:
  46.       labels:
  47.         app: fluentd
  48.     spec:
  49.       containers:
  50.       - image: fluentd:v1.9.1-1.0
  51.         imagePullPolicy: IfNotPresent
  52.         name: fluentd
  53.         ports:
  54.         - containerPort: 24224
  55.           name: forward
  56.           protocol: TCP
  57.         - containerPort: 5140
  58.           name: syslog
  59.           protocol: TCP
  60.         volumeMounts:
  61.         - mountPath: /fluentd/etc
  62.           name: config
  63.           readOnly: true
  64.       volumes:
  65.       - configMap:
  66.           defaultMode: 420
  67.           name: fluentd-config
  68.         name: config
  69. ---
  70. apiVersion: v1
  71. kind: Service
  72. metadata:
  73.   labels:
  74.     app: fluentd
  75.   name: fluentd
  76.   namespace: default
  77. spec:
  78.   ports:
  79.   - name: forward
  80.     port: 24224
  81.     protocol: TCP
  82.     targetPort: forward
  83.   selector:
  84.     app: fluentd
  85.   sessionAffinity: None
  86.   type: ClusterIP
  87. EOF

Step 2: Add Fluentd as a Log Receiver

  1. Log in to KubeSphere as admin. Click Platform in the upper-left corner and select Cluster Management.

    Note

    If you have enabled the multi-cluster feature, you can select a specific cluster.

  2. On the Cluster Management page, go to Log Receivers in Cluster Settings.

  3. Click Add Log Receiver and choose Fluentd.

  4. Provide the Fluentd service address and port number.

  5. Fluentd will appear in the receiver list on the Log Receivers page, the status of which is Collecting.

Step 3: Verify Fluentd is Receiving Logs Sent from Fluent Bit

  1. Click Application Workloads on the Cluster Management page.

  2. Select Workloads and then select the default project on the Deployments tab.

  3. Click the fluentd item and then select the fluentd-xxxxxxxxx-xxxxx Pod.

  4. Click the fluentd container.

  5. On the fluentd container page, select the Container Logs tab.

  6. You can see logs begin to scroll up continuously.

Feedback

Was this page Helpful?

Yes No