TiFlash 部署拓扑

本文介绍在部署最小拓扑集群的基础上，部署 TiFlash 的拓扑结构。TiFlash 是列式的存储引擎，已经成为集群拓扑的标配，适合 Real-Time HTAP 业务。

拓扑信息

实例个数物理机配置IP配置
TiDB316 VCore 32GB 110.0.1.7
10.0.1.8
10.0.1.9		默认端口
全局目录配置
PD34 VCore 8GB 110.0.1.4
10.0.1.5
10.0.1.6		默认端口
全局目录配置
TiKV316 VCore 32GB 2TB (nvme ssd) 110.0.1.1
10.0.1.2
10.0.1.3		默认端口
全局目录配置
TiFlash132 VCore 64 GB 2TB (nvme ssd) 110.0.1.11默认端口
全局目录配置
Monitoring & Grafana14 VCore 8GB * 1 500GB (ssd)10.0.1.10默认端口
全局目录配置

拓扑模版

简单 TiFlash 配置模版

  1. # # Global variables are applied to all deployments and used as the default value of
  2. # # the deployments if a specific deployment value is missing.
  3. global:
  4.   user: "tidb"
  5.   ssh_port: 22
  6.   deploy_dir: "/tidb-deploy"
  7.   data_dir: "/tidb-data"
  9. server_configs:
  10.   pd:
  11.     replication.enable-placement-rules: true
  13. pd_servers:
  14.   - host: 10.0.1.4
  15.   - host: 10.0.1.5
  16.   - host: 10.0.1.6
  18. tidb_servers:
  19.   - host: 10.0.1.7
  20.   - host: 10.0.1.8
  21.   - host: 10.0.1.9
  23. tikv_servers:
  24.   - host: 10.0.1.1
  25.   - host: 10.0.1.2
  26.   - host: 10.0.1.3
  28. tiflash_servers:
  29.   - host: 10.0.1.11
  30.     data_dir: /tidb-data/tiflash-9000
  31.     deploy_dir: /tidb-deploy/tiflash-9000
  33. monitoring_servers:
  34.   - host: 10.0.1.10
  36. grafana_servers:
  37.   - host: 10.0.1.10
  39. alertmanager_servers:
  40.   - host: 10.0.1.10
  41. ``` 详细 TiFlash 配置模版

# Global variables are applied to all deployments and used as the default value of

# the deployments if a specific deployment value is missing.

global: user: “tidb” ssh_port: 22 deploy_dir: “/tidb-deploy” data_dir: “/tidb-data”

# Monitored variables are applied to all the machines.

monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115

deploy_dir: “/tidb-deploy/monitored-9100”

data_dir: “/tidb-data/monitored-9100”

log_dir: “/tidb-deploy/monitored-9100/log”

# Server configs are used to specify the runtime configuration of TiDB components.

# All configuration items can be found in TiDB docs:

# - TiDB: https://docs.pingcap.com/zh/tidb/stable/tidb-configuration-file

# - TiKV: https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file

# - PD: https://docs.pingcap.com/zh/tidb/stable/pd-configuration-file

# All configuration items use points to represent the hierarchy, e.g:

# readpool.storage.use-unified-pool

#

# You can overwrite this configuration via the instance-level config field.

server_configs: tidb: log.slow-threshold: 300 tikv:

  1. # server.grpc-concurrency: 4
  2. # raftstore.apply-pool-size: 2
  3. # raftstore.store-pool-size: 2
  4. # rocksdb.max-sub-compactions: 1
  5. # storage.block-cache.capacity: "16GB"
  6. # readpool.unified.max-thread-count: 12
  7. readpool.storage.use-unified-pool: false
  8. readpool.coprocessor.use-unified-pool: true

pd: schedule.leader-schedule-limit: 4 schedule.region-schedule-limit: 2048 schedule.replica-schedule-limit: 64 replication.enable-placement-rules: true tiflash:

  1. # Maximum memory usage for processing a single query. Zero means unlimited.
  2. profiles.default.max_memory_usage: 0
  3. # Maximum memory usage for processing all concurrently running queries on the server. Zero means unlimited.
  4. profiles.default.max_memory_usage_for_all_queries: 0

tiflash-learner:

  1. # The allowable number of threads in the pool that flushes Raft data to storage.
  2. raftstore.apply-pool-size: 4
  3. # The allowable number of threads that process Raft, which is the size of the Raftstore thread pool.
  4. raftstore.store-pool-size: 4

pd_servers:

  • host: 10.0.1.4

    ssh_port: 22

    name: “pd-1”

    client_port: 2379

    peer_port: 2380

    deploy_dir: “/tidb-deploy/pd-2379”

    data_dir: “/tidb-data/pd-2379”

    log_dir: “/tidb-deploy/pd-2379/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.pd values.

    config:

    schedule.max-merge-region-size: 20

    schedule.max-merge-region-keys: 200000

  • host: 10.0.1.5
  • host: 10.0.1.6 tidb_servers:
  • host: 10.0.1.7

    ssh_port: 22

    port: 4000

    status_port: 10080

    deploy_dir: “/tidb-deploy/tidb-4000”

    log_dir: “/tidb-deploy/tidb-4000/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.tidb values.

    config:

    log.slow-query-file: tidb-slow-overwrited.log

  • host: 10.0.1.8
  • host: 10.0.1.9 tikv_servers:
  • host: 10.0.1.1

    ssh_port: 22

    port: 20160

    status_port: 20180

    deploy_dir: “/tidb-deploy/tikv-20160”

    data_dir: “/tidb-data/tikv-20160”

    log_dir: “/tidb-deploy/tikv-20160/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.tikv values.

    config:

    server.grpc-concurrency: 4

    server.labels:

    zone: “zone1”

    dc: “dc1”

    host: “host1”

  • host: 10.0.1.2
  • host: 10.0.1.3

tiflash_servers:

  • host: 10.0.1.11

    ssh_port: 22

    tcp_port: 9000

    flash_service_port: 3930

    flash_proxy_port: 20170

    flash_proxy_status_port: 20292

    metrics_port: 8234

    deploy_dir: “/tidb-deploy/tiflash-9000”

    The data_dir will be overwritten if you define storage.main.dir configurations in the config section.

    data_dir: “/tidb-data/tiflash-9000”

    log_dir: “/tidb-deploy/tiflash-9000/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.tiflash values.

    config:

    logger.level: “info”

    ## Multi-disk deployment introduced in v4.0.9

    ## Check https://docs.pingcap.com/tidb/stable/tiflash-configuration#multi-disk-deployment for more details.

    ## Example1:

    # storage.main.dir: [ “/nvme_ssd0_512/tiflash”, “/nvme_ssd1_512/tiflash” ]

    # storage.main.capacity: [ 536870912000, 536870912000 ]

    ## Example2:

    # storage.main.dir: [ “/sata_ssd0_512/tiflash”, “/sata_ssd1_512/tiflash”, “/sata_ssd2_512/tiflash” ]

    # storage.latest.dir: [ “/nvme_ssd0_150/tiflash” ]

    # storage.main.capacity: [ 536870912000, 536870912000, 536870912000 ]

    # storage.latest.capacity: [ 161061273600 ]

    learner_config:

    log-level: “info”

    server.labels:

    zone: “zone2”

    dc: “dc2”

    host: “host2”

    - host: 10.0.1.12

    - host: 10.0.1.13

monitoring_servers:

  • host: 10.0.1.10

    ssh_port: 22

    port: 9090

    deploy_dir: “/tidb-deploy/prometheus-8249”

    data_dir: “/tidb-data/prometheus-8249”

    log_dir: “/tidb-deploy/prometheus-8249/log”

grafana_servers:

  • host: 10.0.1.10

    port: 3000

    deploy_dir: /tidb-deploy/grafana-3000

alertmanager_servers:

  • host: 10.0.1.10

    ssh_port: 22

    web_port: 9093

    cluster_port: 9094

    deploy_dir: “/tidb-deploy/alertmanager-9093”

    data_dir: “/tidb-data/alertmanager-9093”

    log_dir: “/tidb-deploy/alertmanager-9093/log”# # Global variables are applied to all deployments and used as the default value of

    # the deployments if a specific deployment value is missing.

    global: user: “tidb” ssh_port: 22 deploy_dir: “/tidb-deploy” data_dir: “/tidb-data”

# Monitored variables are applied to all the machines.

monitored: node_exporter_port: 9100 blackbox_exporter_port: 9115

deploy_dir: “/tidb-deploy/monitored-9100”

data_dir: “/tidb-data/monitored-9100”

log_dir: “/tidb-deploy/monitored-9100/log”

# Server configs are used to specify the runtime configuration of TiDB components.

# All configuration items can be found in TiDB docs:

# - TiDB: https://docs.pingcap.com/zh/tidb/stable/tidb-configuration-file

# - TiKV: https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file

# - PD: https://docs.pingcap.com/zh/tidb/stable/pd-configuration-file

# All configuration items use points to represent the hierarchy, e.g:

# readpool.storage.use-unified-pool

#

# You can overwrite this configuration via the instance-level config field.

server_configs: tidb: log.slow-threshold: 300 tikv:

  1. # server.grpc-concurrency: 4
  2. # raftstore.apply-pool-size: 2
  3. # raftstore.store-pool-size: 2
  4. # rocksdb.max-sub-compactions: 1
  5. # storage.block-cache.capacity: "16GB"
  6. # readpool.unified.max-thread-count: 12
  7. readpool.storage.use-unified-pool: false
  8. readpool.coprocessor.use-unified-pool: true

pd: schedule.leader-schedule-limit: 4 schedule.region-schedule-limit: 2048 schedule.replica-schedule-limit: 64 replication.enable-placement-rules: true tiflash:

  1. # Maximum memory usage for processing a single query. Zero means unlimited.
  2. profiles.default.max_memory_usage: 0
  3. # Maximum memory usage for processing all concurrently running queries on the server. Zero means unlimited.
  4. profiles.default.max_memory_usage_for_all_queries: 0

tiflash-learner:

  1. # The allowable number of threads in the pool that flushes Raft data to storage.
  2. raftstore.apply-pool-size: 4
  3. # The allowable number of threads that process Raft, which is the size of the Raftstore thread pool.
  4. raftstore.store-pool-size: 4

pd_servers:

  • host: 10.0.1.4

    ssh_port: 22

    name: “pd-1”

    client_port: 2379

    peer_port: 2380

    deploy_dir: “/tidb-deploy/pd-2379”

    data_dir: “/tidb-data/pd-2379”

    log_dir: “/tidb-deploy/pd-2379/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.pd values.

    config:

    schedule.max-merge-region-size: 20

    schedule.max-merge-region-keys: 200000

  • host: 10.0.1.5
  • host: 10.0.1.6 tidb_servers:
  • host: 10.0.1.7

    ssh_port: 22

    port: 4000

    status_port: 10080

    deploy_dir: “/tidb-deploy/tidb-4000”

    log_dir: “/tidb-deploy/tidb-4000/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.tidb values.

    config:

    log.slow-query-file: tidb-slow-overwrited.log

  • host: 10.0.1.8
  • host: 10.0.1.9 tikv_servers:
  • host: 10.0.1.1

    ssh_port: 22

    port: 20160

    status_port: 20180

    deploy_dir: “/tidb-deploy/tikv-20160”

    data_dir: “/tidb-data/tikv-20160”

    log_dir: “/tidb-deploy/tikv-20160/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.tikv values.

    config:

    server.grpc-concurrency: 4

    server.labels:

    zone: “zone1”

    dc: “dc1”

    host: “host1”

  • host: 10.0.1.2
  • host: 10.0.1.3

tiflash_servers:

  • host: 10.0.1.11

    ssh_port: 22

    tcp_port: 9000

    flash_service_port: 3930

    flash_proxy_port: 20170

    flash_proxy_status_port: 20292

    metrics_port: 8234

    deploy_dir: “/tidb-deploy/tiflash-9000”

    The data_dir will be overwritten if you define storage.main.dir configurations in the config section.

    data_dir: “/tidb-data/tiflash-9000”

    log_dir: “/tidb-deploy/tiflash-9000/log”

    numa_node: “0,1”

    # The following configs are used to overwrite the server_configs.tiflash values.

    config:

    logger.level: “info”

    ## Multi-disk deployment introduced in v4.0.9

    ## Check https://docs.pingcap.com/tidb/stable/tiflash-configuration#multi-disk-deployment for more details.

    ## Example1:

    # storage.main.dir: [ “/nvme_ssd0_512/tiflash”, “/nvme_ssd1_512/tiflash” ]

    # storage.main.capacity: [ 536870912000, 536870912000 ]

    ## Example2:

    # storage.main.dir: [ “/sata_ssd0_512/tiflash”, “/sata_ssd1_512/tiflash”, “/sata_ssd2_512/tiflash” ]

    # storage.latest.dir: [ “/nvme_ssd0_150/tiflash” ]

    # storage.main.capacity: [ 536870912000, 536870912000, 536870912000 ]

    # storage.latest.capacity: [ 161061273600 ]

    learner_config:

    log-level: “info”

    server.labels:

    zone: “zone2”

    dc: “dc2”

    host: “host2”

    - host: 10.0.1.12

    - host: 10.0.1.13

monitoring_servers:

  • host: 10.0.1.10

    ssh_port: 22

    port: 9090

    deploy_dir: “/tidb-deploy/prometheus-8249”

    data_dir: “/tidb-data/prometheus-8249”

    log_dir: “/tidb-deploy/prometheus-8249/log”

grafana_servers:

  • host: 10.0.1.10

    port: 3000

    deploy_dir: /tidb-deploy/grafana-3000

alertmanager_servers:

  • host: 10.0.1.10

    ssh_port: 22

    web_port: 9093

    cluster_port: 9094

    deploy_dir: “/tidb-deploy/alertmanager-9093”

    data_dir: “/tidb-data/alertmanager-9093”

    log_dir: “/tidb-deploy/alertmanager-9093/log”

    ```

以上 TiDB 集群拓扑文件中，详细的配置项说明见通过 TiUP 部署 TiDB 集群的拓扑文件配置。

关键参数介绍

  • 需要将配置模板中 replication.enable-placement-rules 设置为 true，以开启 PD 的 Placement Rules 功能。

  • tiflash_servers 实例级别配置 "-host" 目前只支持 IP，不支持域名。

  • TiFlash 具体的参数配置介绍可参考 TiFlash 参数配置。

TiFlash 部署拓扑 - 图1

注意

  • 无需手动创建配置文件中的 tidb 用户，TiUP cluster 组件会在目标主机上自动创建该用户。可以自定义用户，也可以和中控机的用户保持一致。
  • 如果部署目录配置为相对路径，会部署在用户的 Home 目录下。