Use Logical Import Mode

This document introduces how to use the logical import mode in TiDB Lightning, including writing the configuration file and tuning performance.

Configure and use the logical import mode

You can use the logical import mode via the following configuration file to import data:

  1. [lightning]
  2. # log
  3. level = "info"
  4. file = "tidb-lightning.log"
  5. max-size = 128 # MB
  6. max-days = 28
  7. max-backups = 14
  9. # Checks the cluster minimum requirements before start.
  10. check-requirements = true
  12. [mydumper]
  13. # The local data source directory or the URI of the external storage. For more information about the URI of the external storage, see https://docs.pingcap.com/tidb/v6.6/backup-and-restore-storages#uri-format.
  14. data-source-dir = "/data/my_database"
  16. [tikv-importer]
  17. # Import mode. "tidb" means using the logical import mode.
  18. backend = "tidb"
  20. [tidb]
  21. # The information of the target cluster. The address of any tidb-server from the cluster.
  22. host = "172.16.31.1"
  23. port = 4000
  24. user = "root"
  25. # Configure the password to connect to TiDB. Either plaintext or Base64 encoded.
  26. password = ""
  27. # tidb-lightning imports the TiDB library, and generates some logs.
  28. # Set the log level of the TiDB library.
  29. log-level = "error"

For the complete configuration file, refer to TiDB Lightning Configuration.

Conflict detection

Conflicting data refers to two or more records with the same data in the PK or UK column. In the logical import mode, you can configure the strategy for handling conflicting data by setting the conflict.strategy configuration item. Based on the strategy, TiDB Lightning imports data with different SQL statements.

StrategyDefault behavior of conflicting dataThe corresponding SQL statement
“replace”Replacing existing data with new data.REPLACE INTO …
“ignore”Keeping existing data and ignoring new data.INSERT IGNORE INTO …
“error”Terminating the import when conflicting data is detected.INSERT INTO …
“”Converted to “error”, which means terminating the import when conflicting data is detected.None

When the strategy is "error", errors caused by conflicting data directly terminate the import task. When the strategy is "replace" or "ignore", you can control the maximum tolerant conflicts by configuring conflict.threshold. The default value is 10000, which means that 10000 errors are tolerant.

When the strategy is "ignore", conflicting data is recorded in the downstream conflict_records table. For further details, see Error report. Before v8.1.0, you can limit the records by configuring conflict.max-record-rows, and conflicting data that exceeds the limit is skipped and not recorded. Starting from v8.1.0, you need to configure conflict.threshold instead, because TiDB Lightning automatically assigns the value of max-record-rows with the value of threshold, regardless of the user input.

Performance tuning

  • In the logical import mode, the performance of TiDB Lightning largely depends on the write performance of the target TiDB cluster. If the cluster hits a performance bottleneck, refer to Highly Concurrent Write Best Practices.

  • If the target TiDB cluster does not hit a write bottleneck, consider increasing the value of region-concurrency in TiDB Lightning configuration. The default value of region-concurrency is the number of CPU cores. The meaning of region-concurrency is different between the physical import mode and the logical import mode. In the logical import mode, region-concurrency is the write concurrency.

    Example configuration:

    1. [lightning]
    2. region-concurrency = 32

  • Adjusting the raftstore.apply-pool-size and raftstore.store-pool-size configuration items in the target TiDB cluster might improve the import speed.