Use Loggie Aggregator

Loggie can be deployed as an Agent, and also support independent deployment for aggregation, forwarding and processing.

Preparation: Choose Architecture

There are many ways to use the aggregator architecture. Common ones are:

  • Agent -> Aggregator: The Agent sends data directly to the Aggregator, and the Aggregator sends data to the backend storage.
  • Agent -> MQ -> Aggregator: Agent sends data to a message queue, such as Kafka, and then Aggregator consumes Kafka messages and sends them to the backend.

Whether to introduce message queues such as Kafka mainly depends on scenario requirements and magnitude of data.

Preparation: Deploy

  1. Deploy Agent

  2. Deploy Aggregator

When deploy Loggie as Aggregator, be sure to specify the cluster name in the Kubernetes configuration.

Configuration

Agent

There is no difference in collection configuration of the LogConfig. You only need to modify the sink to send data to Loggie Aggregator or Kafka.

For the creation of the containers (to be collected) and the matching LogConfig, please refer to Loggie collect container log。

Here we modify the sink:

Example

Aggregator

  1. apiVersion: loggie.io/v1beta1
  2. kind: Sink
  3. metadata:
  4.   name: aggregator
  5. spec:
  6.   sink: |
  7.     type: grpc
  8.     host: "loggie-aggregator.loggie-aggregator:6066"

Kafka

  1. apiVersion: loggie.io/v1beta1
  2. kind: Sink
  3. metadata:
  4.   name: kafka
  5. spec:
  6.   sink: |
  7.     type: kafka
  8.     brokers: ["127.0.0.1:6400"]
  9.     topic: "log-${fields.topic}"

Aggregator

Configuring LogConfig to be delivered to the Aggregator is essentially the same. Only selector needs to be modified.

Example

  1. apiVersion: loggie.io/v1beta1
  2. kind: ClusterLogConfig
  3. metadata:
  4.   name: aggre
  5. spec:
  6.   selector:
  7.     type: cluster
  8.     cluster: aggregator
  9.   pipeline:
  10.     sources: |
  11.       - type: grpc
  12.         name: rec1
  13.         port: 6066
  14.     sinkRef: dev
  1. apiVersion: loggie.io/v1beta1
  2. kind: Sink
  3. metadata:
  4.   name: dev
  5. spec:
  6.   sink: |
  7.     type: dev
  8.     printEvents: true
  9.     codec:
  10.       type: json
  11.       pretty: true

type: cluster indicates that the configuration is selected to be delivered to Loggie cluster specified by cluster, that is, the aggregator cluster we just deployed. If it is not filled in, the configuration will be assigned to the default agent cluster, which will not take effect.

The source here is Grpc, which receives the data sent by the Agent Grpc sink, and then forwards it to the sink specified by its own sinkRef. Here we create a dev sink to view the data output by the aggregator.

Logs

Use kubectl -nloggie-aggregator logs -f <podName> --tail=200 to view logs similar to the following on the aggregator node:

events

  1. 2021-12-20 09:58:50 INF go/src/loggie.io/loggie/pkg/sink/dev/sink.go:98 > event: {
  2.     "body": "14-Dec-2021 06:19:58.306 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [141] milliseconds",
  3.     "fields": {
  4.         "podname": "tomcat-684c698b66-gkrfs",
  5.         "containername": "tomcat",
  6.         "logconfig": "tomcat",
  7.         "namespace": "default",
  8.         "nodename": "kind-control-plane"
  9.     },
  10. }