Expose metrics related to Kong and proxied Upstream services in Prometheus exposition format, which can be scraped by a Prometheus Server.

Configuration Reference

This plugin is compatible with DB-less mode.

In DB-less mode, you configure Kong Gateway declaratively. Therefore, the Admin API is mostly read-only. The only tasks it can perform are all related to handling the declarative config, including:

  • Setting a target’s health status in the load balancer
  • Validating configurations against schemas
  • Uploading the declarative configuration using the /config endpoint

The database will always be reported as reachable in Prometheus with DB-less. Additionally, the DB entity count metric (kong_db_entities_total) is not emitted in DB-less mode.

Enable the plugin on a service

Admin API

Kubernetes

Declarative (YAML)

Konnect Cloud

Kong Manager

For example, configure this plugin on a service by making the following request:

  1. curl -X POST http://{HOST}:8001/services/{SERVICE}/plugins \
  2.     --data "name=prometheus"

SERVICE is the id or name of the service that this plugin configuration will target.

First, create a KongPlugin resource:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongPlugin
  3. metadata:
  4.   name: <prometheus-example>
  5. config:
  6.   <optional_parameter>: <value>
  7. plugin: prometheus

Next, apply the KongPlugin resource to a Service by annotating the Service as follows:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: {SERVICE}
  5.   labels:
  6.     app: {SERVICE}
  7.   annotations:
  8.     konghq.com/plugins: <prometheus-example>
  9. spec:
  10.   ports:
  11.   - port: 80
  12.     targetPort: 80
  13.     protocol: TCP
  14.     name: {SERVICE}
  15.   selector:
  16.     app: {SERVICE}

{SERVICE} is the id or name of the service that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any service, consumer, or route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

For example, configure this plugin on a service by adding this section to your declarative configuration file:

  1. plugins:
  2. - name: prometheus
  3.   service: {SERVICE}
  4.   config:
  5.     <optional_parameter>: <value>

SERVICE is the id or name of the service that this plugin configuration will target.

Configure this plugin on a service:

  1. In Konnect Cloud, select the service on the ServiceHub page.
  2. Scroll down to Versions and select the version.
  3. Scroll down to Plugins and click New Plugin.
  4. Find and select the Prometheus plugin.
  5. Click Create.

Configure this plugin on a service:

  1. In Kong Manager, select the workspace.
  2. From the Dashboard, scroll down to Services and click View for the service row.
  3. Scroll down to plugins and click Add Plugin.

  4. Find and select the Prometheus plugin.

    Note: If the plugin is greyed out, then it is not available for your product tier. See Kong Gateway tiers.

  5. If the option is available, select Scoped.

  6. Add the service name and ID to the Service field if it is not already prefilled.
  7. Click Create.

Enable the plugin globally

A plugin which is not associated to any service, route, or consumer is considered global, and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

Admin API

Kubernetes

Declarative (YAML)

Kong Manager

For example, configure this plugin globally with:

  1. $ curl -X POST http://{HOST}:8001/plugins/ \
  2.     --data "name=prometheus"

Create a KongClusterPlugin resource and label it as global:

  1. apiVersion: configuration.konghq.com/v1
  2. kind: KongClusterPlugin
  3. metadata:
  4.   name: <global-prometheus>
  5.   annotations:
  6.     kubernetes.io/ingress.class: kong
  7.   labels:
  8.     global: \"true\"
  9. config:
  10.   <optional_parameter>: <value>
  11. plugin: prometheus

For example, configure this plugin using the plugins: entry in the declarative configuration file:

  1. plugins:
  2. - name: prometheus
  3.   config:
  4.     <optional_parameter>: <value>

Configure this plugin globally:

  1. In Kong Manager, select the workspace.
  2. From the Dashboard, select Plugins in the left navigation.
  3. Click New Plugin.

  4. Find and select the Prometheus plugin.

    Note: If the plugin is greyed out, then it is not available for your product tier. See Kong Gateway tiers.

  5. If the option is available, set the plugin scope to Global.

  6. Click Create.

Parameters

Here’s a list of all the parameters which can be used in this plugin’s configuration:

Form ParameterDescription
name
required

Type: string		The name of the plugin, in this case prometheus.
service.id

Type: string		The ID of the Service the plugin targets.
enabled
required

Type: boolean

Default value: true		Whether this plugin will be applied.
config.per_consumer
optional

Type: boolean

Default value: false

A boolean value that determines if per-consumer metrics should be collected. If enabled, a kong_http_consumer_status metric is added to exported metrics.

Metrics tracked by this plugin are available on both the Admin API and Status API at the http://localhost:<port>/metrics endpoint. Note that the URL to those APIs will be specific to your installation; see Accessing the metrics.

This plugin records and exposes metrics at the node level. Your Prometheus server will need to discover all Kong nodes via a service discovery mechanism, and consume data from each node’s configured /metrics endpoint.

Grafana dashboard

Metrics exported by the plugin can be graphed in Grafana using a drop in dashboard: https://grafana.com/dashboards/7424.

Available metrics

  • Status codes: HTTP status codes returned by Upstream services. These are available per service, across all services, and per route per consumer.
  • Latencies Histograms: Latency as measured at Kong:
    • Request: Total time taken by Kong and Upstream services to serve requests.
    • Kong: Time taken for Kong to route a request and run all configured plugins.
    • Upstream: Time taken by the Upstream service to respond to requests.
  • Bandwidth: Total Bandwidth (egress/ingress) flowing through Kong. This metric is available per service and as a sum across all services.
  • DB reachability: A gauge type with a value of 0 or 1, which represents whether DB can be reached by a Kong node.
  • Connections: Various Nginx connection metrics like active, reading, writing, and number of accepted connections.
  • Target Health: The healthiness status (healthchecks_off, healthy, unhealthy, or dns_error) of Targets belonging to a given Upstream as well as their subsystem (http or stream).
  • Dataplane Status: The last seen timestamp, config hash, config sync status and certificate expiration timestamp for data plane nodes is exported to control plane.
  • Enterprise License Information: The Kong Gateway license expiration date, features and license signature. Those metrics are only exported on Kong Gateway.
  • DB Entity Count : A gauge metric that measures the current number of database entities.
  • Number of Nginx timers : A gauge metric that measures the total number of Nginx timers, in Running or Pending state.

Here is an example of output you could expect from the /metrics endpoint:

  1. $ curl -i http://localhost:8001/metrics
  2. HTTP/1.1 200 OK
  3. Server: openresty/1.15.8.3
  4. Date: Tue, 7 Jun 2020 16:35:40 GMT
  5. Content-Type: text/plain; charset=UTF-8
  6. Transfer-Encoding: chunked
  7. Connection: keep-alive
  8. Access-Control-Allow-Origin: *
  10. # HELP kong_bandwidth Total bandwidth in bytes consumed per service/route in Kong
  11. # TYPE kong_bandwidth counter
  12. kong_bandwidth{type="egress",service="google",route="google.route-1"} 1277
  13. kong_bandwidth{type="ingress",service="google",route="google.route-1"} 254
  14. # HELP kong_nginx_timers Number of nginx timers
  15. # TYPE kong_nginx_timers gauge
  16. kong_nginx_timers{state="running"} 3
  17. kong_nginx_timers{state="pending"} 1
  18. # HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable
  19. # TYPE kong_datastore_reachable gauge
  20. kong_datastore_reachable 1
  21. # HELP kong_http_consumer_status HTTP status codes for customer per service/route in Kong
  22. # TYPE kong_http_consumer_status counter
  23. kong_http_consumer_status{service="s1",route="s1.route-1",code="200",consumer="<CONSUMER_USERNAME>"} 3
  24. # HELP kong_http_status HTTP status codes per service/route in Kong
  25. # TYPE kong_http_status counter
  26. kong_http_status{code="301",service="google",route="google.route-1"} 2
  27. # HELP kong_latency Latency added by Kong, total request time and upstream latency for each service in Kong
  28. # TYPE kong_latency histogram
  29. kong_latency_bucket{type="kong",service="google",route="google.route-1",le="00001.0"} 1
  30. kong_latency_bucket{type="kong",service="google",route="google.route-1",le="00002.0"} 1
  31. .
  32. .
  33. .
  34. kong_latency_bucket{type="kong",service="google",route="google.route-1",le="+Inf"} 2
  35. kong_latency_bucket{type="request",service="google",route="google.route-1",le="00300.0"} 1
  36. kong_latency_bucket{type="request",service="google",route="google.route-1",le="00400.0"} 1
  37. .
  38. .
  39. kong_latency_bucket{type="request",service="google",route="google.route-1",le="+Inf"} 2
  40. kong_latency_bucket{type="upstream",service="google",route="google.route-1",le="00300.0"} 2
  41. kong_latency_bucket{type="upstream",service="google",route="google.route-1",le="00400.0"} 2
  42. .
  43. .
  44. kong_latency_bucket{type="upstream",service="google",route="google.route-1",le="+Inf"} 2
  45. kong_latency_count{type="kong",service="google",route="google.route-1"} 2
  46. kong_latency_count{type="request",service="google",route="google.route-1"} 2
  47. kong_latency_count{type="upstream",service="google",route="google.route-1"} 2
  48. kong_latency_sum{type="kong",service="google",route="google.route-1"} 2145
  49. kong_latency_sum{type="request",service="google",route="google.route-1"} 2672
  50. kong_latency_sum{type="upstream",service="google",route="google.route-1"} 527
  51. # HELP kong_nginx_http_current_connections Number of HTTP connections
  52. # TYPE kong_nginx_http_current_connections gauge
  53. kong_nginx_http_current_connections{state="accepted"} 8
  54. kong_nginx_http_current_connections{state="active"} 1
  55. kong_nginx_http_current_connections{state="handled"} 8
  56. kong_nginx_http_current_connections{state="reading"} 0
  57. kong_nginx_http_current_connections{state="total"} 8
  58. kong_nginx_http_current_connections{state="waiting"} 0
  59. kong_nginx_http_current_connections{state="writing"} 1
  60. # HELP kong_memory_lua_shared_dict_bytes Allocated slabs in bytes in a shared_dict
  61. # TYPE kong_memory_lua_shared_dict_bytes gauge
  62. kong_memory_lua_shared_dict_bytes{shared_dict="kong",kong_subsystem="http"} 40960
  63. .
  64. .
  65. # HELP kong_memory_lua_shared_dict_total_bytes Total capacity in bytes of a shared_dict
  66. # TYPE kong_memory_lua_shared_dict_total_bytes gauge
  67. kong_memory_lua_shared_dict_total_bytes{shared_dict="kong",kong_subsystem="http"} 5242880
  68. .
  69. .
  70. # HELP kong_memory_workers_lua_vms_bytes Allocated bytes in worker Lua VM
  71. # TYPE kong_memory_workers_lua_vms_bytes gauge
  72. kong_memory_workers_lua_vms_bytes{pid="7281",kong_subsystem="http"} 41124353
  73. # HELP kong_data_plane_config_hash Config hash value of the data plane
  74. # TYPE kong_data_plane_config_hash gauge
  75. kong_data_plane_config_hash{node_id="d4e7584e-b2f2-415b-bb68-3b0936f1fde3",hostname="ubuntu-bionic",ip="127.0.0.1"} 1.7158931820287e+38
  76. # HELP kong_data_plane_last_seen Last time data plane contacted control plane
  77. # TYPE kong_data_plane_last_seen gauge
  78. kong_data_plane_last_seen{node_id="d4e7584e-b2f2-415b-bb68-3b0936f1fde3",hostname="ubuntu-bionic",ip="127.0.0.1"} 1600190275
  79. # HELP kong_data_plane_version_compatible Version compatible status of the data plane, 0 is incompatible
  80. # TYPE kong_data_plane_version_compatible gauge
  81. kong_data_plane_version_compatible{node_id="d4e7584e-b2f2-415b-bb68-3b0936f1fde3",hostname="ubuntu-bionic",ip="127.0.0.1",kong_version="2.4.1"} 1
  82. # HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors
  83. # TYPE kong_nginx_metric_errors_total counter
  84. kong_nginx_metric_errors_total 0
  85. # HELP kong_upstream_target_health Health status of targets of upstream. States = healthchecks_off|healthy|unhealthy|dns_error, value is 1 when state is populated.
  86. kong_upstream_target_health{upstream="<upstream_name>",target="<target>",address="<ip>:<port>",state="healthchecks_off",subsystem="http"} 0
  87. kong_upstream_target_health{upstream="<upstream_name>",target="<target>",address="<ip>:<port>",state="healthy",subsystem="http"} 1
  88. kong_upstream_target_health{upstream="<upstream_name>",target="<target>",address="<ip>:<port>",state="unhealthy",subsystem="http"} 0
  89. kong_upstream_target_health{upstream="<upstream_name>",target="<target>",address="<ip>:<port>",state="dns_error",subsystem="http"} 0
  90. # HELP kong_db_entities_total Total number of Kong db entities
  91. # TYPE kong_db_entities_total gauge
  92. kong_db_entities_total 42
  93. # HELP kong_db_entity_count_errors Errors during entity count collection
  94. # TYPE kong_db_entity_count_errors counter
  95. kong_db_entity_count_errors 0

Note: Upstream targets’ health information is exported once per subsystem. If both stream and HTTP listeners are enabled, targets’ health will appear twice. Health metrics have a subsystem label to indicate which subsystem the metric refers to.

Accessing the metrics

In most configurations, the Kong Admin API will be behind a firewall or would need to be set up to require authentication. Here are a couple of options to allow access to the /metrics endpoint to Prometheus:

  1. If the Status API is enabled, then its /metrics endpoint can be used. This is the preferred method.

  2. The /metrics endpoint is also available on the Admin API, which can be used if the Status API is not enabled. Note that this endpoint is unavailable when RBAC is enabled on the Admin API (Prometheus does not support Key-Auth to pass the token).

Changelog

1.6.x

  • Adds a new metric:
    • kong_nginx_timers (gauge): total number of Nginx timers, in Running or Pending state.
  • Add two new metrics:
    • kong_db_entities_total (gauge): total number of entities in the database
    • kong_db_entity_count_errors (counter): measures the number of errors encountered during the measurement of kong_db_entities_total

1.4.x

  • New data_plane_cluster_cert_expiry_timestamp metric
  • Added subsystem label to Upstream Target health metrics