Logging

Amazon EKS Control Plane and Node logging for Kubeflow

Amazon EKS control plane logging provides audit and diagnostic logs directly from the Amazon EKS control plane to Amazon CloudWatch logs in your account. These logs make it easy for you to monitor your clusters and troubleshoot issues if they arise. You can select which log types you need, and the logs are sent as log streams to a named group for each Amazon EKS cluster.

By default, cluster control plane logs and worker node group logs aren’t sent to CloudWatch. You must enable each log type individually to send logs for your cluster.

Control Plane Logging

You can easily use the aws utility to enable control plane logs and customize the components you want to collect logs from. Only following components are available.

  • api
  • audit
  • authenticator
  • controllerManager
  • scheduler

Run command to enable logs

  1. aws eks update-cluster-config \
  2.   --region ${AWS_REGION} \
  3.   --name ${AWS_CLUSTER_NAME} \
  4.   --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

If you prefer, you can also enable cluster logging with eksctl. For example, to enable all logs:

  1. eksctl utils update-cluster-logging \
  2.     --enable-types all \
  3.   --region ${AWS_REGION} \
  4.   --cluster ${AWS_CLUSTER_NAME} \
  5.   --approve

Finally, logging can also be configured from within the Amazon EKS console, under the Configuration panel for your cluster.

Once configured, open the CloudWatch Console. Choose the cluster that you want to view logs from. The log group name format is /aws/eks/${AWS_CLUSTER_NAME}/cluster.

Worker Node Group Logging

You can add fluentd-cloud-watch to addons to enable node group logs. With this configuration in place, all Pod logs are sent to CloudWatch.

  1.   plugins:
  2.   - kind: KfAwsPlugin
  3.     metadata:
  4.       name: aws
  5.     spec:
  6.       region: ${AWS_REGION}
  7.       addons:
  8.       - fluentd-cloud-watch

You will see three log groups in total, again organized under the cluster name.

  • /aws/containerinsights/${AWS_CLUSTER_NAME}/containers
  • /aws/containerinsights/${AWS_CLUSTER_NAME}/dataplane
  • /aws/containerinsights/${AWS_CLUSTER_NAME}/host

Last modified 04.05.2021: refactor and refresh aws docs (#2688) (ef4cda60)