Monitoring

Kubebuilder projects use controller-runtimeto implement controllers and admission webhooks. controller-runtime instruments several key metricsrelated to controllers and webhooks by default using kubernetes instrumentation guidelines.and makes them available via HTTP endpoint in prometheus metric format.

Following metrics are instrumented by default:

  • Total number of reconcilation errors per controller
  • Length of reconcile queue per controller
  • Reconcilation latency
  • Usual resource metrics such as CPU, memory usage, file descriptor usage
  • Go runtime metrics such as number of Go routines, GC duration

Metrics support

Please note that metrics support has been added in controller-runtime 0.1.8+release which is the default version for Kubebuilder 1.0.6+ releases. So if yourproject was created using 1.0.5 or older kubebuilder, then update thecontroller-runtime dependencies to 0.1.8 or higher.

To quickly examine metrics in your development environment, you can run thefollowing:

  1. # launch manager
  2. $ make run
  4. # in another terminal, access the metrics
  6. $ curl http://localhost:8080/metrics
  7. # HELP controller_runtime_reconcile_errors_total Total number of reconcile errors per controller
  8. # TYPE controller_runtime_reconcile_errors_total counter
  9. controller_runtime_reconcile_errors_total{controller="mysql-controller"} 10
  10. # HELP controller_runtime_reconcile_queue_length Length of reconcile queue per controller
  11. # TYPE controller_runtime_reconcile_queue_length gauge
  12. controller_runtime_reconcile_queue_length{controller="mysql-controller"} 0
  13. # HELP controller_runtime_reconcile_time_seconds Length of time per reconcile per controller
  14. # TYPE controller_runtime_reconcile_time_seconds histogram
  15. controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="0.005"} 10
  16. controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="0.01"} 10
  17. controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="0.025"} 10
  18. controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="10"} 10
  19. controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="+Inf"} 10
  20. controller_runtime_reconcile_time_seconds_sum{controller="mysql-controller"} 2.3416e-05
  21. controller_runtime_reconcile_time_seconds_count{controller="mysql-controller"} 10
  22. # HELP go_gc_duration_seconds A summary of the GC invocation durations.
  23. # TYPE go_gc_duration_seconds summary
  24. go_gc_duration_seconds{quantile="0"} 7.69e-05
  25. go_gc_duration_seconds{quantile="0.25"} 0.0001225
  26. go_gc_duration_seconds{quantile="0.5"} 0.000124351
  27. go_gc_duration_seconds{quantile="0.75"} 0.000236344
  28. go_gc_duration_seconds{quantile="1"} 0.000262102
  29. go_gc_duration_seconds_sum 0.000822197
  30. go_gc_duration_seconds_count 5
  31. # HELP go_goroutines Number of goroutines that currently exist.
  32. # TYPE go_goroutines gauge
  33. go_goroutines 39
  34. # HELP go_info Information about the Go environment.
  35. # TYPE go_info gauge
  36. go_info{version="go1.9.4"} 1
  37. # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
  38. .....
  39. ....

Is the metrics endpoint protected ?

Yes. By default, kubebuilder generated YAML manifests (under config/ dir) ensures that the access to metrics endpoint is authenticated and authorized usingan auth proxy which is deployed as sidecar container in the manager pod. You can read more details about the auth proxy based approach here.

If you want to disable the auth proxy, which is not recommended, you can followthe instructions in the Kustomization file located in config/default/kustomization.yaml

If your project was created using 1.0.5 or older kubebuilder, you need to modifythe following files as show in PR #513.

  • cmd/manager/main.go
  • config/default/kustomization.yaml
  • config/default/manager_auth_proxy_patch.yaml
  • config/rbac/auth_proxy_role.yaml
  • config/rbac/auth_proxy_role_binding.yaml
  • config/rbac/auth_proxy_service.yaml

How do I configure Prometheus Server to access the metrics?

Kubebuilder generated manifests for manager have annotations such asprometheus.io/scrape, prometheus.io/path on the metrics service sothat it can be easily discovered by the prometheus server deployed in yourkubernetes cluster.

Assuming auth is enabled, which is by default, you will have to add thefollowing to the job which is configured to scrap kubernetes service endpoints.

  1. tls_config:
  2.     insecure_skip_verify: true
  4. bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token