Currently, you can use two types of interfaces to monitor the state of the TiKV cluster:

  • The component state interface: use the HTTP interface to get the internal information of a component, which is called the component state interface.
  • The metrics interface: use the Prometheus interface to record the detailed information of various operations in the components, which is called the metrics interface.

The component state interface

You can use this type of interface to monitor the basic information of components. This interface can get the details of the entire TiKV cluster and can act as the interface to monitor Keepalive.

The PD server

The API address of the Placement Driver (PD) is http://${host}:${port}/pd/api/v1/${api_name}

The default port number is 2379.

For detailed information about various API names, see PD API doc.

You can use the interface to get the state of all the TiKV instances and the information about load balancing. It is the most important and frequently-used interface to get the state information of all the TiKV nodes. See the following example for the information about a 3-instance TiKV cluster deployed on a single machine:

  1. curl http://127.0.0.1:2379/pd/api/v1/stores
  2. {
  3.   "count": 3,
  4.   "stores": [
  5.     {
  6.       "store": {
  7.         "id": 1,
  8.         "address": "127.0.0.1:20161",
  9.         "version": "2.1.0-rc.2",
  10.         "state_name": "Up"
  11.       },
  12.       "status": {
  13.         "capacity": "937 GiB",
  14.         "available": "837 GiB",
  15.         "leader_weight": 1,
  16.         "region_count": 1,
  17.         "region_weight": 1,
  18.         "region_score": 1,
  19.         "region_size": 1,
  20.         "start_ts": "2018-09-29T00:05:47Z",
  21.         "last_heartbeat_ts": "2018-09-29T00:23:46.227350716Z",
  22.         "uptime": "17m59.227350716s"
  23.       }
  24.     },
  25.     {
  26.       "store": {
  27.         "id": 2,
  28.         "address": "127.0.0.1:20162",
  29.         "version": "2.1.0-rc.2",
  30.         "state_name": "Up"
  31.       },
  32.       "status": {
  33.         "capacity": "937 GiB",
  34.         "available": "837 GiB",
  35.         "leader_weight": 1,
  36.         "region_count": 1,
  37.         "region_weight": 1,
  38.         "region_score": 1,
  39.         "region_size": 1,
  40.         "start_ts": "2018-09-29T00:05:47Z",
  41.         "last_heartbeat_ts": "2018-09-29T00:23:45.65292648Z",
  42.         "uptime": "17m58.65292648s"
  43.       }
  44.     },
  45.     {
  46.       "store": {
  47.         "id": 7,
  48.         "address": "127.0.0.1:20160",
  49.         "version": "2.1.0-rc.2",
  50.         "state_name": "Up"
  51.       },
  52.       "status": {
  53.         "capacity": "937 GiB",
  54.         "available": "837 GiB",
  55.         "leader_count": 1,
  56.         "leader_weight": 1,
  57.         "leader_score": 1,
  58.         "leader_size": 1,
  59.         "region_count": 1,
  60.         "region_weight": 1,
  61.         "region_score": 1,
  62.         "region_size": 1,
  63.         "start_ts": "2018-09-29T00:05:47Z",
  64.         "last_heartbeat_ts": "2018-09-29T00:23:44.853636067Z",
  65.         "uptime": "17m57.853636067s"
  66.       }
  67.     }
  68.   ]
  69. }

The metrics interface

You can use this type of interface to monitor the state and performance of the entire cluster. The metrics data is displayed in Prometheus and Grafana. See Use Prometheus and Grafana for how to set up the monitoring system.

You can get the following metrics for each component:

The PD server

  • the total number of times that the command executes
  • the total number of times that a certain command fails
  • the duration that a command succeeds
  • the duration that a command fails
  • the duration that a command finishes and returns result

The TiKV server

  • Garbage Collection (GC) monitoring
  • the total number of times that the TiKV command executes
  • the duration that Scheduler executes commands
  • the total number of times of the Raft propose command
  • the duration that Raft executes commands
  • the total number of times that Raft commands fail
  • the total number of times that Raft processes the ready state

Use Prometheus and Grafana

This section introduces the deployment architecture of Prometheus and Grafana in TiKV, and how to set up and configure the monitoring system.

The deployment architecture

See the following diagram for the deployment architecture:

Monitoring a Cluster - 图1Figure 1. Monitor architecture

Set up the monitoring system

You can use your existing Prometheus server to pull metrics from TiKV, or quickly bootstrap a test server below. This pairs well with Grafana for monitoring.

Use the following script to bootstrap Grafana and Prometheus on a local CentOS system:

  1. cat <<EOT > /etc/yum.repos.d/prometheus.repo
  2. [prometheus]
  3. name=prometheus
  4. baseurl=https://packagecloud.io/prometheus-rpm/release/el/\$releasever/\$basearch
  5. repo_gpgcheck=1
  6. enabled=1
  7. gpgkey=https://packagecloud.io/prometheus-rpm/release/gpgkey
  8.       https://raw.githubusercontent.com/lest/prometheus-rpm/master/RPM-GPG-KEY-prometheus-rpm
  9. gpgcheck=1
  10. metadata_expire=300
  11. EOT
  12. cat <<EOT > /etc/yum.repos.d/grafana.repo
  13. [grafana]
  14. name=grafana
  15. baseurl=https://packages.grafana.com/oss/rpm
  16. repo_gpgcheck=1
  17. enabled=1
  18. gpgcheck=1
  19. gpgkey=https://packages.grafana.com/gpg.key
  20. sslverify=1
  21. sslcacert=/etc/pki/tls/certs/ca-bundle.crt
  22. EOT
  23. yum install -y prometheus grafana
  24. systemctl enable --now prometheus grafana-server

Browse to port 3000 on the host and you should be able to log in with admin/admin.

Configure Prometheus

To configure Prometheus, edit the /etc/prometheus/prometheus.yml and the scrape_configs block:

  1. global:
  2.   scrape_interval:     15s
  3.   evaluation_interval: 15s
  4.   external_labels:
  5.     cluster: 'test-cluster'
  6.     monitor: "prometheus"
  8. scrape_configs:
  9.   - job_name: 'pd'
  10.     honor_labels: true
  11.     static_configs:
  12.       - targets:
  13.         - '127.0.0.1:2379'
  14.   - job_name: 'tikv'
  15.     honor_labels: true
  16.     static_configs:
  17.       - targets:
  18.         - '127.0.0.1:20160'

Restart Prometheus with systemctl restart prometheus.

Configure Grafana

Visiting Add a Data Source, choose Prometheus and add the URL or your Prometheus server (http://127.0.0.1:9090). Finally, save & test it.

  • Log in to the Grafana Web interface (default login admin/admin).
  • Visit Add a data source.
  • Specify the data source information:
    • Specify the name for the data source.
    • For Type, select Prometheus.
    • For Url, specify the Prometheus address.
    • Specify other fields as needed.
  • Click ‘Test & Save’ to save the new data source.

Grafana Dashboard Templates

The TiKV community has developed several useful dashboards to get you started with Grafana. You may need to adjust some settings, particularly if you configured the external_labels option in the prometheus.yml configuration above.