Tune TiKV Memory Parameter Performance

This document describes how to tune the TiKV parameters for optimal performance. You can find the default configuration file in etc/config-template.toml. To modify the configuration, you can use TiUP or modify TiKV dynamically for a limited set of configuration items. For the complete configuration, see TiKV configuration file.

TiKV uses RocksDB for persistent storage at the bottom level of the TiKV architecture. Therefore, many of the performance parameters are related to RocksDB. TiKV uses two RocksDB instances: the default RocksDB instance stores KV data, the Raft RocksDB instance (RaftDB) stores Raft logs.

TiKV implements Column Families (CF) from RocksDB.

  • The default RocksDB instance stores KV data in the default, write and lock CFs.

    • The default CF stores the actual data. The corresponding parameters are in [rocksdb.defaultcf].
    • The write CF stores the version information in Multi-Version Concurrency Control (MVCC) and index-related data. The corresponding parameters are in [rocksdb.writecf].
    • The lock CF stores the lock information. The system uses the default parameters.

  • The Raft RocksDB (RaftDB) instance stores Raft logs.

    • The default CF stores the Raft log. The corresponding parameters are in [raftdb.defaultcf].

After TiKV 3.0, by default, all CFs share one block cache instance. You can configure the size of the cache by setting the capacity parameter under [storage.block-cache]. The bigger the block cache, the more hot data can be cached, and the easier to read data, in the meantime, the more system memory is occupied.

Before TiKV 3.0, shared block cache is not supported, and you need to configure block cache for each CF individually.

Each CF also has a separate write buffer. You can configure the size by setting the write-buffer-size parameter.

Parameter specification

  1. # Log level: trace, debug, warn, error, info, off.
  2. log-level = "info"
  4. [server]
  5. # Set listening address
  6. # addr = "127.0.0.1:20160"
  8. # Size of thread pool for gRPC
  9. # grpc-concurrency = 4
  10. # The number of gRPC connections between each TiKV instance
  11. # grpc-raft-conn-num = 10
  13. # Most read requests from TiDB are sent to the coprocessor of TiKV. This parameter is used to set the number of threads
  14. # of the coprocessor. If many read requests exist, add the number of threads and keep the number within that of the
  15. # system CPU cores. For example, for a 32-core machine deployed with TiKV, you can even set this parameter to 30 in
  16. # repeatable read scenarios. If this parameter is not set, TiKV automatically sets it to CPU cores * 0.8.
  17. # end-point-concurrency = 8
  19. # Tag the TiKV instances to schedule replicas.
  20. # labels = {zone = "cn-east-1", host = "118", disk = "ssd"}
  22. [storage]
  23. # The data directory
  24. # data-dir = "/tmp/tikv/store"
  26. # In most cases, you can use the default value. When importing data, it is recommended to set the parameter to 1024000.
  27. # scheduler-concurrency = 102400
  28. # This parameter controls the number of write threads. When write operations occur frequently, set this parameter value
  29. # higher. Run `top -H -p tikv-pid` and if the threads named `sched-worker-pool` are busy, set the value of parameter
  30. # `scheduler-worker-pool-size` higher and increase the number of write threads.
  31. # scheduler-worker-pool-size = 4
  33. [storage.block-cache]
  34. ## Whether to create a shared block cache for all RocksDB column families.
  35. ##
  36. ## Block cache is used by RocksDB to cache uncompressed blocks. Big block cache can speed up read.
  37. ## It is recommended to turn on shared block cache. Since only the total cache size need to be
  38. ## set, it is easier to configure. In most cases, it should be able to auto-balance cache usage
  39. ## between column families with standard LRU algorithm.
  40. ##
  41. ## The rest of config in the storage.block-cache session is effective only when shared block cache
  42. ## is on.
  43. ## Starting from v6.6.0, the `shared` option is always enabled and cannot be disabled.
  44. # shared = true
  46. ## Size of the shared block cache. Normally it should be tuned to 30%-50% of system's total memory.
  47. ## When the config is not set, it is decided by the sum of the following fields or their default
  48. ## value:
  49. ##   * rocksdb.defaultcf.block-cache-size or 25% of system's total memory
  50. ##   * rocksdb.writecf.block-cache-size   or 15% of system's total memory
  51. ##   * rocksdb.lockcf.block-cache-size    or  2% of system's total memory
  52. ##   * raftdb.defaultcf.block-cache-size  or  2% of system's total memory
  53. ##
  54. ## To deploy multiple TiKV nodes on a single physical machine, configure this parameter explicitly.
  55. ## Otherwise, the OOM problem might occur in TiKV.
  56. # capacity = "1GB"
  58. [pd]
  59. # PD address
  60. # endpoints = ["127.0.0.1:2379","127.0.0.2:2379","127.0.0.3:2379"]
  62. [metric]
  63. # The interval of pushing metrics to Prometheus Pushgateway
  64. interval = "15s"
  65. # Prometheus Pushgateway address
  66. address = ""
  67. job = "tikv"
  69. [raftstore]
  70. # Raft RocksDB directory. The default value is Raft subdirectory of [storage.data-dir].
  71. # If there are multiple disks on the machine, store the data of Raft RocksDB on different disks to improve TiKV performance.
  72. # raftdb-path = "/tmp/tikv/store/raft"
  74. # When the data size change in a Region is larger than the threshold value, TiKV checks whether this Region needs split.
  75. # To reduce the costs of scanning data in the checking process, set the value to 32 MB during the data import process. In the normal operation status, set it to the default value.
  76. region-split-check-diff = "32MB"
  78. [coprocessor]
  79. ## If the size of a Region with the range of [a,e) is larger than the value of `region_max_size`, TiKV tries to split the Region to several Regions, for example, the Regions with the ranges of [a,b), [b,c), [c,d), and [d,e).
  80. ## After the Region split, the size of the split Regions is equal to the value of `region_split_size` (or slightly larger than the value of `region_split_size`).
  81. # region-max-size = "144MB"
  82. # region-split-size = "96MB"
  84. [rocksdb]
  85. # The maximum number of threads of RocksDB background tasks. The background tasks include compaction and flush.
  86. # For detailed information why RocksDB needs to implement compaction, see RocksDB-related materials. When write
  87. # traffic (like the importing data size) is big, it is recommended to enable more threads. But set the number of the enabled
  88. # threads smaller than that of CPU cores. For example, when importing data, for a machine with a 32-core CPU,
  89. # set the value to 28.
  90. # max-background-jobs = 8
  92. # The maximum number of file handles RocksDB can open
  93. # max-open-files = 40960
  95. # The file size limit of RocksDB MANIFEST. For more details, see https://github.com/facebook/rocksdb/wiki/MANIFEST
  96. max-manifest-file-size = "20MB"
  98. # The directory of RocksDB write-ahead logs. If there are two disks on the machine, store the RocksDB data and WAL logs
  99. # on different disks to improve TiKV performance.
  100. # wal-dir = "/tmp/tikv/store"
  102. # Use the following two parameters to deal with RocksDB archiving WAL.
  103. # For more details, see https://github.com/facebook/rocksdb/wiki/How-to-persist-in-memory-RocksDB-database%3F
  104. # wal-ttl-seconds = 0
  105. # wal-size-limit = 0
  107. # In most cases, set the maximum total size of RocksDB WAL logs to the default value.
  108. # max-total-wal-size = "4GB"
  110. # Use this parameter to enable the readahead feature during RocksDB compaction. If you are using mechanical disks, it is recommended to set the value to 2MB at least.
  111. # compaction-readahead-size = "2MB"
  113. [rocksdb.defaultcf]
  114. # The data block size. RocksDB compresses data based on the unit of block.
  115. # Similar to page in other databases, block is the smallest unit cached in block-cache.
  116. block-size = "64KB"
  118. # The compaction mode of each layer of RocksDB data. The optional values include no, snappy, zlib,
  119. # bzip2, lz4, lz4hc, and zstd. Note that the Snappy compressed file must be in the [official Snappy format](https://github.com/google/snappy). Other variants of Snappy compression are not supported.
  120. # "no:no:lz4:lz4:lz4:zstd:zstd" indicates there is no compaction of level0 and level1; lz4 compaction algorithm is used
  121. # from level2 to level4; zstd compaction algorithm is used from level5 to level6.
  122. # "no" means no compaction. "lz4" is a compaction algorithm with moderate speed and compaction ratio. The
  123. # compaction ratio of zlib is high. It is friendly to the storage space, but its compaction speed is slow. This
  124. # compaction occupies many CPU resources. Different machines deploy compaction modes according to CPU and I/O resources.
  125. # For example, if you use the compaction mode of "no:no:lz4:lz4:lz4:zstd:zstd" and find much I/O pressure of the
  126. # system (run the iostat command to find %util lasts 100%, or run the top command to find many iowaits) when writing
  127. # (importing) a lot of data while the CPU resources are adequate, you can compress level0 and level1 and exchange CPU
  128. # resources for I/O resources. If you use the compaction mode of "no:no:lz4:lz4:lz4:zstd:zstd" and you find the I/O
  129. # pressure of the system is not big when writing a lot of data, but CPU resources are inadequate. Then run the top
  130. # command and choose the -H option. If you find a lot of bg threads (namely the compaction thread of RocksDB) are
  131. # running, you can exchange I/O resources for CPU resources and change the compaction mode to "no:no:no:lz4:lz4:zstd:zstd".
  132. # In a word, it aims at making full use of the existing resources of the system and improving TiKV performance
  133. # in terms of the current resources.
  134. compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
  136. # The RocksDB memtable size
  137. write-buffer-size = "128MB"
  139. # The maximum number of the memtables. The data written into RocksDB is first recorded in the WAL log, and then inserted
  140. # into memtables. When the memtable reaches the size limit of `write-buffer-size`, it turns into read only and generates
  141. # a new memtable receiving new write operations. The flush threads of RocksDB will flush the read only memtable to the
  142. # disks to become an sst file of level0. `max-background-flushes` controls the maximum number of flush threads. When the
  143. # flush threads are busy, resulting in the number of the memtables waiting to be flushed to the disks reaching the limit
  144. # of `max-write-buffer-number`, RocksDB stalls the new operation.
  145. # "Stall" is a flow control mechanism of RocksDB. When importing data, you can set the `max-write-buffer-number` value
  146. # higher, like 10.
  147. max-write-buffer-number = 5
  149. # When the number of sst files of level0 reaches the limit of `level0-slowdown-writes-trigger`, RocksDB
  150. # tries to slow down the write operation, because too many sst files of level0 can cause higher read pressure of
  151. # RocksDB. `level0-slowdown-writes-trigger` and `level0-stop-writes-trigger` are for the flow control of RocksDB.
  152. # When the number of sst files of level0 reaches 4 (the default value), the sst files of level0 and the sst files
  153. # of level1 which overlap those of level0 implement compaction to relieve the read pressure.
  154. level0-slowdown-writes-trigger = 20
  156. # When the number of sst files of level0 reaches the limit of `level0-stop-writes-trigger`, RocksDB stalls the new
  157. # write operation.
  158. level0-stop-writes-trigger = 36
  160. # When the level1 data size reaches the limit value of `max-bytes-for-level-base`, the sst files of level1
  161. # and their overlap sst files of level2 implement compaction. The golden rule: the first reference principle
  162. # of setting `max-bytes-for-level-base` is guaranteeing that the `max-bytes-for-level-base` value is roughly equal to the
  163. # data volume of level0. Thus unnecessary compaction is reduced. For example, if the compaction mode is
  164. # "no:no:lz4:lz4:lz4:lz4:lz4", the `max-bytes-for-level-base` value is write-buffer-size * 4, because there is no
  165. # compaction of level0 and level1 and the trigger condition of compaction for level0 is that the number of the
  166. # sst files reaches 4 (the default value). When both level0 and level1 adopt compaction, it is necessary to analyze
  167. # RocksDB logs to know the size of an sst file compressed from an mentable. For example, if the file size is 32MB,
  168. # the proposed value of `max-bytes-for-level-base` is 32MB * 4 = 128MB.
  169. max-bytes-for-level-base = "512MB"
  171. # The sst file size. The sst file size of level0 is influenced by the compaction algorithm of `write-buffer-size`
  172. # and level0. `target-file-size-base` is used to control the size of a single sst file of level1-level6.
  173. target-file-size-base = "32MB"
  175. [rocksdb.writecf]
  176. # Set it the same as `rocksdb.defaultcf.compression-per-level`.
  177. compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
  179. # Set it the same as `rocksdb.defaultcf.write-buffer-size`.
  180. write-buffer-size = "128MB"
  181. max-write-buffer-number = 5
  182. min-write-buffer-number-to-merge = 1
  184. # Set it the same as `rocksdb.defaultcf.max-bytes-for-level-base`.
  185. max-bytes-for-level-base = "512MB"
  186. target-file-size-base = "32MB"
  188. [raftdb]
  189. # The maximum number of the file handles RaftDB can open
  190. # max-open-files = 40960
  192. # Enable the readahead feature in RaftDB compaction. If you are using mechanical disks, it is recommended to set
  193. # this value to 2MB at least.
  194. # compaction-readahead-size = "2MB"
  196. [raftdb.defaultcf]
  197. # Set it the same as `rocksdb.defaultcf.compression-per-level`.
  198. compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
  200. # Set it the same as `rocksdb.defaultcf.write-buffer-size`.
  201. write-buffer-size = "128MB"
  202. max-write-buffer-number = 5
  203. min-write-buffer-number-to-merge = 1
  205. # Set it the same as `rocksdb.defaultcf.max-bytes-for-level-base`.
  206. max-bytes-for-level-base = "512MB"
  207. target-file-size-base = "32MB"

TiKV memory usage

Besides block cache and write buffer which occupy the system memory, the system memory is occupied in the following scenarios:

  • Some of the memory is reserved as the system’s page cache.

  • When TiKV processes large queries such as select * from ..., it reads data, generates the corresponding data structure in the memory, and returns this structure to TiDB. During this process, TiKV occupies some of the memory.

  • In production environments, it is not recommended to deploy TiKV on the machine whose CPU cores are less than 8 or the memory is less than 32GB.

  • If you demand a high write throughput, it is recommended to use a disk with good throughput capacity.

  • If you demand a very low read-write latency, it is recommended to use SSD with high IOPS.