Quick Start Guide for TiDB Data Migration

This document describes how to migrate data from MySQL to TiDB using TiDB Data Migration (DM). This guide is a quick demo of DM features and is not recommended for any production environment.

Step 1: Deploy a DM cluster

  1. Install TiUP, and install dmctl using TiUP:

    1. curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
    2. tiup install dm dmctl

  2. Generate the minimal deployment topology file of a DM cluster:

    1. tiup dm template

  3. Copy the configuration information in the output, and save it as the topology.yaml file with the modified IP address. Deploy the DM cluster with the topology.yaml file using TiUP:

    1. tiup dm deploy dm-test 6.0.0 topology.yaml -p

Step 2: Prepare the data source

You can use one or multiple MySQL instances as an upstream data source.

  1. Create a configuration file for each data source as follows:

    1. source-id: "mysql-01"
    2. from:
    3.   host: "127.0.0.1"
    4.   user: "root"
    5.   password: "fCxfQ9XKCezSzuCD0Wf5dUD+LsKegSg="  # encrypt with `tiup dmctl --encrypt "123456"`
    6.   port: 3306

  2. Add the source to the DM cluster by running the following command. mysql-01.yaml is the configuration file created in the previous step.

    1. tiup dmctl --master-addr=127.0.0.1:8261 operate-source create mysql-01.yaml # use one of master_servers as the argument of --master-addr

If you do not have a MySQL instance for testing, you can create a MySQL instance in Docker by taking the following steps:

  1. Create a MySQL configuration file:

    1. mkdir -p /tmp/mysqltest && cd /tmp/mysqltest
    3. cat > my.cnf <<EOF
    4. [mysqld]
    5. bind-address     = 0.0.0.0
    6. character-set-server=utf8
    7. collation-server=utf8_bin
    8. default-storage-engine=INNODB
    9. transaction-isolation=READ-COMMITTED
    10. server-id        = 100
    11. binlog_format    = row
    12. log_bin          = /var/lib/mysql/mysql-bin.log
    13. show_compatibility_56 = ON
    14. EOF

  2. Start the MySQL instance using Docker:

    1. docker run --name mysql-01 -v /tmp/mysqltest:/etc/mysql/conf.d -e MYSQL_ROOT_PASSWORD=my-secret-pw -d -p 3306:3306 mysql:5.7

  3. After the MySQL instance is started, access the instance:

    Quick Start - 图1

    Note

    This command is only suitable for trying out data migration, and cannot be used in production environments or stress tests.

    1. mysql -uroot -p -h 127.0.0.1 -P 3306

Step 3: Prepare a downstream database

You can choose an existing TiDB cluster as a target for data migration.

If you do not have a TiDB cluster for testing, you can quickly build a demonstration environment by running the following command:

  1. tiup playground

Step 4: Prepare test data

Create a test table and data in one or multiple data sources. If you use an existing MySQL database, and the database contains available data, you can skip this step.

  1. drop database if exists `testdm`;
  2. create database `testdm`;
  3. use `testdm`;
  4. create table t1 (id bigint, uid int, name varchar(80), info varchar(100), primary key (`id`), unique key(`uid`)) DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
  5. create table t2 (id bigint, uid int, name varchar(80), info varchar(100), primary key (`id`), unique key(`uid`)) DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
  6. insert into t1 (id, uid, name) values (1, 10001, 'Gabriel García Márquez'), (2, 10002, 'Cien años de soledad');
  7. insert into t2 (id, uid, name) values (3, 20001, 'José Arcadio Buendía'), (4, 20002, 'Úrsula Iguarán'), (5, 20003, 'José Arcadio');

Step 5: Create a data migration task

  1. Create a task configuration file testdm-task.yaml:

    1. name: testdm
    2. task-mode: all
    4. target-database:
    5.   host: "127.0.0.1"
    6.   port: 4000
    7.   user: "root"
    8.   password: "" # If the password is not empty, it is recommended to use a password encrypted with dmctl.
    10. # Configure the information of one or multiple data sources
    11. mysql-instances:
    12.   - source-id: "mysql-01"
    13.     block-allow-list:  "ba-rule1"
    15. block-allow-list:
    16.   ba-rule1:
    17.     do-dbs: ["testdm"]

  2. Create the task using dmctl:

    1. tiup dmctl --master-addr 127.0.0.1:8261 start-task testdm-task.yaml

You have successfully created a task that migrates data from a mysql-01 database to TiDB.

Step 6: Check the status of the task

After the task is created, you can use the dmctl query-status command to check the status of the task:

  1. tiup dmctl --master-addr 127.0.0.1:8261 query-status testdm