prometheus

Description

This plugin exposes metrics in Prometheus Exposition format.

Attributes

NameTypeRequirementDefaultValidDescription
prefer_namebooleanoptionalfalseWhen set to true, would print route/service name instead of id in Prometheus metric.

API

This plugin will add /apisix/prometheus/metrics to expose the metrics.

The metrics are exposed via a separate Prometheus server address. By default, the address is 127.0.0.1:9091. You can change it in the conf/config.yaml, for example:

  1. plugin_attr:
  2.   prometheus:
  3.     export_addr:
  4.       ip: ${{INTRANET_IP}}
  5.       port: 9092

Assume environment variable INTRANET_IP is 172.1.1.1, now APISIX will export the metrics via 172.1.1.1:9092.

If you still want to expose the metrics via the data plane port (default: 9080), you can configure it like this:

  1. plugin_attr:
  2.   prometheus:
  3.     enable_export_server: false

And expose it by using public-api plugin.

How to enable it

prometheus plugin could be enable with empty table. Notice, name could be duplicated for multiple routes/services, so when set prefer_name to true, take care of naming format or it could be misleading.

For example:

  1. curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
  2. {
  3.     "uri": "/hello",
  4.     "plugins": {
  5.         "prometheus":{}
  6.     },
  7.     "upstream": {
  8.         "type": "roundrobin",
  9.         "nodes": {
  10.             "127.0.0.1:80": 1
  11.         }
  12.     }
  13. }'

You can use APISIX Dashboard to complete the above operations through the web console.

First, add a Route:

create a route

Then add prometheus plugin:

enable prometheus plugin

How to fetch the metric data

We fetch the metric data from the specified url /apisix/prometheus/metrics.

  1. curl -i http://127.0.0.1:9091/apisix/prometheus/metrics

Puts this URL address into prometheus, and it will automatically fetch these metric data.

For example like this:

  1. scrape_configs:
  2.   - job_name: 'apisix'
  3.     scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value.
  4.     metrics_path: '/apisix/prometheus/metrics'
  5.     static_configs:
  6.     - targets: ['127.0.0.1:9091']

And we can check the status at prometheus console:

checking status on prometheus dashboard

prometheus apisix in-depth metric view

How to specify export uri

We can change the default export uri in the plugin_attr section of conf/config.yaml.

NameTypeDefaultDescription
export_uristring“/apisix/prometheus/metrics”uri to get the prometheus metrics

Here is an example:

  1. plugin_attr:
  2.   prometheus:
  3.     export_uri: /apisix/metrics

Grafana dashboard

Metrics exported by the plugin can be graphed in Grafana using a drop in dashboard.

Downloads Grafana dashboard meta and imports it to Grafana。

Or you can goto Grafana official for Grafana meta data.

Grafana chart-1

Grafana chart-2

Grafana chart-3

Grafana chart-4

Available metrics

  • Status codes: HTTP status code returned from upstream services. These status code available per service and across all services.

    Attributes：

    NameDescription
    codeThe HTTP status code returned by the upstream service.
    routeThe route_id of the matched route is request. If it does not match, the default value is an empty string.
    matched_uriThe uri of the route matching the request, if it does not match, the default value is an empty string.
    matched_hostThe host of the route that matches the request. If it does not match, the default value is an empty string.
    serviceThe service_id of the route matched by the request. When the route lacks service_id, the default is $host.
    consumerThe consumer_name of the consumer that matches the request. If it does not match, the default value is an empty string.
    nodeThe ip of the upstream node.

  • Bandwidth: Total Bandwidth (egress/ingress) flowing through APISIX. The total bandwidth of per service can be counted.

    Attributes：

    NameDescription
    typeThe type of bandwidth(egress/ingress).
    routeThe route_id of the matched route is request. If it does not match, the default value is an empty string..
    serviceThe service_id of the route matched by the request. When the route lacks service_id, the default is $host.
    consumerThe consumer_name of the consumer that matches the request. If it does not match, the default value is an empty string.
    nodeThe ip of the upstream node.

  • etcd reachability: A gauge type with a value of 0 or 1, representing if etcd can be reached by a APISIX or not, where 1 is available, and 0 is unavailable.

  • Connections: Various Nginx connection metrics like active, reading, writing, and number of accepted connections.

  • Batch process entries: A gauge type, when we use plugins and the plugin used batch process to send data, such as: sys logger, http logger, sls logger, tcp logger, udp logger and zipkin, then the entries which hasn’t been sent in batch process will be counted in the metrics.

  • Latency: The per service histogram of request time in different dimensions.

    Attributes：

    NameDescription
    typeThe value can be apisix, upstream or request, which means http latency caused by apisix, upstream, or their sum.
    serviceThe service_id of the route matched by the request. When the route lacks service_id, the default is $host.
    consumerThe consumer_name of the consumer that matches the request. If it does not match, the default value is an empty string.
    nodeThe ip of the upstream node.

  • Info: the information of APISIX node.

Here is the original metric data of APISIX:

  1. $ curl http://127.0.0.1:9091/apisix/prometheus/metrics
  2. # HELP apisix_bandwidth Total bandwidth in bytes consumed per service in Apisix
  3. # TYPE apisix_bandwidth counter
  4. apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
  5. apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
  6. apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
  7. apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
  8. apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
  9. apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
  10. # HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
  11. # TYPE apisix_etcd_modify_indexes gauge
  12. apisix_etcd_modify_indexes{key="consumers"} 0
  13. apisix_etcd_modify_indexes{key="global_rules"} 0
  14. apisix_etcd_modify_indexes{key="max_modify_index"} 222
  15. apisix_etcd_modify_indexes{key="prev_index"} 35
  16. apisix_etcd_modify_indexes{key="protos"} 0
  17. apisix_etcd_modify_indexes{key="routes"} 222
  18. apisix_etcd_modify_indexes{key="services"} 0
  19. apisix_etcd_modify_indexes{key="ssls"} 0
  20. apisix_etcd_modify_indexes{key="stream_routes"} 0
  21. apisix_etcd_modify_indexes{key="upstreams"} 0
  22. apisix_etcd_modify_indexes{key="x_etcd_index"} 223
  23. # HELP apisix_batch_process_entries batch process remaining entries
  24. # TYPE apisix_batch_process_entries gauge
  25. apisix_batch_process_entries{name="http-logger",route_id="9",server_addr="127.0.0.1"} 1
  26. apisix_batch_process_entries{name="sls-logger",route_id="9",server_addr="127.0.0.1"} 1
  27. apisix_batch_process_entries{name="tcp-logger",route_id="9",server_addr="127.0.0.1"} 1
  28. apisix_batch_process_entries{name="udp-logger",route_id="9",server_addr="127.0.0.1"} 1
  29. apisix_batch_process_entries{name="sys-logger",route_id="9",server_addr="127.0.0.1"} 1
  30. apisix_batch_process_entries{name="zipkin_report",route_id="9",server_addr="127.0.0.1"} 1
  31. # HELP apisix_etcd_reachable Config server etcd reachable from Apisix, 0 is unreachable
  32. # TYPE apisix_etcd_reachable gauge
  33. apisix_etcd_reachable 1
  34. # HELP apisix_http_status HTTP status codes per service in Apisix
  35. # TYPE apisix_http_status counter
  36. apisix_http_status{code="200",route="1",matched_uri="/hello",matched_host="",service="",consumer="",node="127.0.0.1"} 4
  37. apisix_http_status{code="200",route="2",matched_uri="/world",matched_host="",service="",consumer="",node="127.0.0.1"} 4
  38. apisix_http_status{code="404",route="",matched_uri="",matched_host="",service="",consumer="",node=""} 1
  39. # HELP apisix_http_requests_total The total number of client requests
  40. # TYPE apisix_http_requests_total gauge
  41. apisix_http_requests_total 1191780
  42. # HELP apisix_nginx_http_current_connections Number of HTTP connections
  43. # TYPE apisix_nginx_http_current_connections gauge
  44. apisix_nginx_http_current_connections{state="accepted"} 11994
  45. apisix_nginx_http_current_connections{state="active"} 2
  46. apisix_nginx_http_current_connections{state="handled"} 11994
  47. apisix_nginx_http_current_connections{state="reading"} 0
  48. apisix_nginx_http_current_connections{state="waiting"} 1
  49. apisix_nginx_http_current_connections{state="writing"} 1
  50. # HELP apisix_nginx_metric_errors_total Number of nginx-lua-prometheus errors
  51. # TYPE apisix_nginx_metric_errors_total counter
  52. apisix_nginx_metric_errors_total 0
  53. # HELP apisix_http_latency HTTP request latency in milliseconds per service in APISIX
  54. # TYPE apisix_http_latency histogram
  55. apisix_http_latency_bucket{type="apisix",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1
  56. apisix_http_latency_bucket{type="apisix",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1
  57. apisix_http_latency_bucket{type="request",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1
  58. apisix_http_latency_bucket{type="request",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1
  59. apisix_http_latency_bucket{type="upstream",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1
  60. apisix_http_latency_bucket{type="upstream",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1
  61. ...
  62. # HELP apisix_node_info Info of APISIX node
  63. # TYPE apisix_node_info gauge
  64. apisix_node_info{hostname="desktop-2022q8f-wsl"} 1

Disable Plugin

Remove the corresponding json configuration in the plugin configuration to disable prometheus. APISIX plugins are hot-reloaded, therefore no need to restart APISIX.

  1. curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
  2. {
  3.     "uri": "/hello",
  4.     "plugins": {},
  5.     "upstream": {
  6.         "type": "roundrobin",
  7.         "nodes": {
  8.             "127.0.0.1:80": 1
  9.         }
  10.     }
  11. }'