Using Alibaba Cloud Observable Unified Storage SLS

Regardless of whether your service is deployed on Alibaba Cloud or using a hybrid cloud architecture, Loggie supports collecting log data and send it to Alibaba Cloud SLS. And you just need to use sls sink.

For the use of Loggie in Alibaba Cloud’s official documentation, please refer to: upload logs with Loggie.

Preparation: Create SLS Project

In order to use Alibaba Cloud Observable Unified Storage SLS, we need to create a corresponding Project first. As shown below, on SLS page, click Create Project, fill in the project name and the corresponding region, and create the corresponding Logstore.

sls-project

Then we can prepare sls sink configuration. The example is as follows. Please refer to sls sink configuration for details:

sls sink

  1.     sink:
  2.       type: sls
  3.       name: demo
  4.       endpoint: cn-hangzhou.log.aliyuncs.com
  5.       accessKeyId: xxxx
  6.       accessKeySecret: xxxx
  7.       project: test
  8.       logstore: test1
  9.       topic: myservice

Collect Logs on ECS

Refer to Deploy on hosts. We can modify the pipeline.yml and add the configuration of the sls sink.

The pipeline configurationexample is as follows:

sls pipeline

  1. pipelines:
  2.   - name: test
  3.     sources:
  4.       - type: file
  5.         name: demo
  6.         addonMeta: true
  7.         paths:
  8.           - /tmp/log/*.log
  9.     sink:
  10.       type: sls
  11.       endpoint: cn-hangzhou.log.aliyuncs.com
  12.       accessKeyId: ${accessKeyId}
  13.       accessKeySecret: ${accessKeySecret}
  14.       project: loggietest
  15.       logstore: demo1
  16.       topic: myservice

Collect Logs of ACK Kubernetes Cluster

Like the self-built Kubernetes cluster, Loggie can also be deployed in Alibaba Cloud ACK. We can use kubectl or helm to deploy according to the kubeconfig file provided by ACK. Refer to Deploy in Kubernetes.

Node:

  • If you want to collect Pod logs that are not mounted with emptyDir/hostPath, make sure that the system configuration in values.yaml discovery.kubernetes.rootFsCollectionEnabled set true.
  • Configure discovery.kubernetes.containerRuntime to match the Kubernetes cluster. The default is containerd.

After deployment, similar to the usage in conventional Kubernetes cluster, ClusterLogConfig/LogConfig/Interceptor/Sink CR can also be used.

ACK provides a CRD page, and we can create and modify it directly in this page. sls-crd

For an example of collecting logs of a service, just use sls sink. You can also create a sink CR storage sls sink configuration separately and refer to it through sinkRef in LogConfig.

logconfig

  1. apiVersion: loggie.io/v1beta1
  2. kind: LogConfig
  3. metadata:
  4.   name: tomcat
  5.   namespace: default
  6. spec:
  7.   selector:
  8.     labelSelector:
  9.       app: tomcat
  10.     type: pod
  12.   pipeline:
  13.     sources: |
  14.       - type: file
  15.         name: stdout
  16.         paths:
  17.           - stdout
  18.       - type: file
  19.         name: access
  20.         ignoreOlder: 3d
  21.         paths:
  22.           - /usr/local/tomcat/logs/*.log
  24.     sink: |
  25.       type: sls
  26.       endpoint: cn-hangzhou.log.aliyuncs.com
  27.       accessKeyId: ${accessKeyId}
  28.       accessKeySecret: ${accessKeySecret} 
  29.       project: loggietest
  30.       logstore: demo1
  31.       topic: myservice

View logs

We can view the collected logs in the specific Project of SLS, as shown in the following figure: sls-searchlog.png

For details on adding meta information and modifying log fields, refer to Log Segmentation and Add Meta Info。