CLUSTER_LOG

You can query cluster logs on the CLUSTER_LOG cluster log table. By pushing down query conditions to each instance, the impact of the query on cluster performance is less than that of the grep command.

CLUSTER_LOG - 图1

Note

This table is only applicable to TiDB Self-Hosted and not available on TiDB Cloud.

To get the logs of the TiDB cluster before v4.0, you need to log in to each instance to summarize logs. This cluster log table in 4.0 provides the global and time-ordered log search result, which makes it easier to track full-link events. For example, by searching logs according to the region id, you can query all logs in the life cycle of this Region. Similarly, by searching the full link log through the slow log’s txn id, you can query the flow and the number of keys scanned by this transaction at each instance.

  1. USE information_schema;
  2. DESC cluster_log;
  1. +----------+------------------+------+------+---------+-------+
  2. | Field    | Type             | Null | Key  | Default | Extra |
  3. +----------+------------------+------+------+---------+-------+
  4. | TIME     | varchar(32)      | YES  |      | NULL    |       |
  5. | TYPE     | varchar(64)      | YES  |      | NULL    |       |
  6. | INSTANCE | varchar(64)      | YES  |      | NULL    |       |
  7. | LEVEL    | varchar(8)       | YES  |      | NULL    |       |
  8. | MESSAGE  | var_string(1024) | YES  |      | NULL    |       |
  9. +----------+------------------+------+------+---------+-------+
  10. 5 rows in set (0.00 sec)

Field description:

  • TIME: The time to print the log.
  • TYPE: The instance type. The optional values are tidb, pd, and tikv.
  • INSTANCE: The service address of the instance.
  • LEVEL: The log level.
  • MESSAGE: The log content.

CLUSTER_LOG - 图2

Note

  • All fields of the cluster log table are pushed down to the corresponding instance for execution. To reduce the overhead of using the cluster log table, you must specify the keywords used for the search, the time range, and as many conditions as possible. For example, select * from cluster_log where message like '%ddl%' and time > '2020-05-18 20:40:00' and time<'2020-05-18 21:40:00' and type='tidb'.

  • The message field supports the like and regexp regular expressions, and the corresponding pattern is encoded as regexp. Specifying multiple message conditions is equivalent to the pipeline form of the grep command. For example, executing the select * from cluster_log where message like 'coprocessor%' and message regexp '.*slow.*' and time > '2020-05-18 20:40:00' and time<'2020-05-18 21:40:00' statement is equivalent to executing grep 'coprocessor' xxx.log | grep -E '.*slow.*' on all cluster instances.

The following example shows how to query the execution process of a DDL statement using the CLUSTER_LOG table:

  1. SELECT time,instance,left(message,150) FROM cluster_log WHERE message LIKE '%ddl%job%ID.80%' AND type='tidb' AND time > '2020-05-18 20:40:00' AND time < '2020-05-18 21:40:00'
  1. +-------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
  2. | time                    | instance       | left(message,150)                                                                                                                                      |
  3. +-------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
  4. | 2020/05/18 21:37:54.784 | 127.0.0.1:4002 | [ddl_worker.go:261] ["[ddl] add DDL jobs"] ["batch count"=1] [jobs="ID:80, Type:create table, State:none, SchemaState:none, SchemaID:1, TableID:79, Ro |
  5. | 2020/05/18 21:37:54.784 | 127.0.0.1:4002 | [ddl.go:477] ["[ddl] start DDL job"] [job="ID:80, Type:create table, State:none, SchemaState:none, SchemaID:1, TableID:79, RowCount:0, ArgLen:1, start |
  6. | 2020/05/18 21:37:55.327 | 127.0.0.1:4000 | [ddl_worker.go:568] ["[ddl] run DDL job"] [worker="worker 1, tp general"] [job="ID:80, Type:create table, State:none, SchemaState:none, SchemaID:1, Ta |
  7. | 2020/05/18 21:37:55.381 | 127.0.0.1:4000 | [ddl_worker.go:763] ["[ddl] wait latest schema version changed"] [worker="worker 1, tp general"] [ver=70] ["take time"=50.809848ms] [job="ID:80, Type: |
  8. | 2020/05/18 21:37:55.382 | 127.0.0.1:4000 | [ddl_worker.go:359] ["[ddl] finish DDL job"] [worker="worker 1, tp general"] [job="ID:80, Type:create table, State:synced, SchemaState:public, SchemaI |
  9. | 2020/05/18 21:37:55.786 | 127.0.0.1:4002 | [ddl.go:509] ["[ddl] DDL job is finished"] [jobID=80]                                                                                                  |
  10. +-------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+

The query results above show the process of executing a DDL statement:

  1. The request with a DDL JOB ID of 80 is sent to the 127.0.0.1:4002 TiDB instance.
  2. The 127.0.0.1:4000 TiDB instance processes this DDL request, which indicates that the 127.0.0.1:4000 instance is the DDL owner at that time.
  3. The request with a DDL JOB ID of 80 has been processed.