Prometheus writing and reading

Prometheus is a widespread open-source monitoring and alerting system. Prometheus joined the Cloud Native Computing Foundation (CNCF) in 2016 as the second incubated project after Kubernetes, which has a very active developer and user community.

Prometheus provides remote_write and remote_read interfaces to leverage other database products as its storage engine. To enable users of the Prometheus ecosystem to take advantage of TDengine’s efficient writing and querying, TDengine also provides support for these two interfaces.

Prometheus data can be stored in TDengine via the remote_write interface with proper configuration. Data stored in TDengine can be queried via the remote_read interface, taking full advantage of TDengine’s efficient storage query performance and clustering capabilities for time-series data.

Prerequisites

To write Prometheus data to TDengine requires the following preparations.

  • The TDengine cluster is deployed and functioning properly
  • taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
  • Prometheus has been installed. Please refer to the official documentation for installing Prometheus

Configuration steps

Configuring Prometheus is done by editing the Prometheus configuration file prometheus.yml (default location /etc/prometheus/prometheus.yml).

Configuring third-party database addresses

Point the remote_read url and remote_write url to the domain name or IP address of the server running the taosAdapter service, the REST service port (taosAdapter uses 6041 by default), and the name of the database you want to write to TDengine, and ensure that the corresponding URL form as follows.

  • remote_read url : http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_read/<database name>
  • remote_write url : http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_write/<database name>

Configure Basic authentication

  • username: <TDengine’s username>
  • password: <TDengine’s password>
  1. remote_write:
  2.   - url: "http://localhost:6041/prometheus/v1/remote_write/prometheus_data"
  3.     basic_auth:
  4.       username: root
  5.       password: taosdata
  7. remote_read:
  8.   - url: "http://localhost:6041/prometheus/v1/remote_read/prometheus_data"
  9.     basic_auth:
  10.       username: root
  11.       password: taosdata
  12.     remote_timeout: 10s
  13.     read_recent: true

Verification method

After restarting Prometheus, you can refer to the following example to verify that data is written from Prometheus to TDengine and can read out correctly.

Query and write data using TDengine CLI

  1. taos> show databases;
  2.               name              |      created_time       |   ntables   |   vgroups   | replica | quorum |  days  |           keep           |  cache(MB)  |   blocks    |   minrows   |   maxrows   | wallevel |    fsync    | comp | cachelast | precision | update |   status   |
  3. ====================================================================================================================================================================================================================================================================================
  4.  test                           | 2022-04-12 08:07:58.756 |           1 |           1 |       1 |      1 |     10 | 3650                     |          16 |           6 |         100 |        4096 |        1 |        3000 |    2 |         0 | ms        |      0 | ready      |
  5.  log                            | 2022-04-20 07:19:50.260 |           2 |           1 |       1 |      1 |     10 | 3650                     |          16 |           6 |         100 |        4096 |        1 |        3000 |    2 |         0 | ms        |      0 | ready      |
  6.  prometheus_data                | 2022-04-20 07:21:09.202 |         158 |           1 |       1 |      1 |     10 | 3650                     |          16 |           6 |         100 |        4096 |        1 |        3000 |    2 |         0 | ns        |      2 | ready      |
  7.  db                             | 2022-04-15 06:37:08.512 |           1 |           1 |       1 |      1 |     10 | 3650                     |          16 |           6 |         100 |        4096 |        1 |        3000 |    2 |         0 | ms        |      0 | ready      |
  8. Query OK, 4 row(s) in set (0.000585s)
  10. taos> use prometheus_data;
  11. Database changed.
  13. taos> show stables;
  14.               name              |      created_time       | columns |  tags  |   tables    |
  15. ============================================================================================
  16.  metrics                        | 2022-04-20 07:21:09.209 |       2 |      1 |        1389 |
  17. Query OK, 1 row(s) in set (0.000487s)
  19. taos> select * from metrics limit 10;
  20.               ts               |           value           |             labels             |
  21. =============================================================================================
  22.  2022-04-20 07:21:09.193000000 |               0.000024996 | {"__name__":"go_gc_duration... |
  23.  2022-04-20 07:21:14.193000000 |               0.000024996 | {"__name__":"go_gc_duration... |
  24.  2022-04-20 07:21:19.193000000 |               0.000024996 | {"__name__":"go_gc_duration... |
  25.  2022-04-20 07:21:24.193000000 |               0.000024996 | {"__name__":"go_gc_duration... |
  26.  2022-04-20 07:21:29.193000000 |               0.000024996 | {"__name__":"go_gc_duration... |
  27.  2022-04-20 07:21:09.193000000 |               0.000054249 | {"__name__":"go_gc_duration... |
  28.  2022-04-20 07:21:14.193000000 |               0.000054249 | {"__name__":"go_gc_duration... |
  29.  2022-04-20 07:21:19.193000000 |               0.000054249 | {"__name__":"go_gc_duration... |
  30.  2022-04-20 07:21:24.193000000 |               0.000054249 | {"__name__":"go_gc_duration... |
  31.  2022-04-20 07:21:29.193000000 |               0.000054249 | {"__name__":"go_gc_duration... |
  32. Query OK, 10 row(s) in set (0.011146s)

Use promql-cli to read data from TDengine via remote_read

Install promql-cli

  1.  go install github.com/nalbury/promql-cli@latest

Query Prometheus data in the running state of TDengine and taosAdapter services

  1. ubuntu@shuduo-1804 ~ $ promql-cli --host "http://127.0.0.1:9090" "sum(up) by (job)"
  2. JOB           VALUE    TIMESTAMP
  3. prometheus    1        2022-04-20T08:05:26Z
  4. node          1        2022-04-20T08:05:26Z

Stop taosAdapter service and query Prometheus data to verify

  1. ubuntu@shuduo-1804 ~ $ sudo systemctl stop taosadapter.service
  2. ubuntu@shuduo-1804 ~ $ promql-cli --host "http://127.0.0.1:9090" "sum(up) by (job)"
  3. VALUE    TIMESTAMP