Migrate Data Using Data Migration

Migrate Data Using Data Migration

This guide shows how to migrate data using the Data Migration (DM) tool.

Step 1: Deploy the DM cluster

It is recommended to deploy the DM cluster using TiUP. You can also deploy the DM cluster using binary for trial or test.

Note

For database passwords in all the DM configuration files, it is recommended to use the passwords encrypted by dmctl. If a database password is empty, it is unnecessary to encrypt it. See Encrypt the database password using dmctl.
The user of the upstream and downstream databases must have the corresponding read and write privileges.

Step 2: Check the cluster information

After the DM cluster is deployed using TiUP, the configuration information is like what is listed below.

The configuration information of related components in the DM cluster:

Component Host Port
dm_worker1 172.16.10.72 8262
dm_worker2 172.16.10.73 8262
dm_master 172.16.10.71 8261

Component	Host	Port
dm_worker1	172.16.10.72	8262
dm_worker2	172.16.10.73	8262
dm_master	172.16.10.71	8261

The information of upstream and downstream database instances:

Database instance	Host	Port	Username	Encrypted password
Upstream MySQL-1	172.16.10.81	3306	root	VjX8cEeTX+qcvZ3bPaO4h0C80pe/1aU=
Upstream MySQL-2	172.16.10.82	3306	root	VjX8cEeTX+qcvZ3bPaO4h0C80pe/1aU=
Downstream TiDB	172.16.10.83	4000	root

The list of privileges needed on the MySQL host can be found in the precheck documentation.

Step 3: Create data source

Write MySQL-1 related information to conf/source1.yaml:

# MySQL1 Configuration.
source-id: "mysql-replica-01"
# This indicates that whether DM-worker uses Global Transaction Identifier (GTID) to pull binlog. Before you use this configuration item, make sure that the GTID mode is enabled in the upstream MySQL.
enable-gtid: false
from:
  host: "172.16.10.81"
  user: "root"
  password: "VjX8cEeTX+qcvZ3bPaO4h0C80pe/1aU="
  port: 3306

Execute the following command in the terminal, and use tiup dmctl to load the MySQL-1 data source configuration to the DM cluster:
```
tiup dmctl --master-addr 172.16.10.71:8261 operate-source create conf/source1.yaml
```
For MySQL-2, modify the relevant information in the configuration file and execute the same dmctl command.

Step 4: Configure the data migration task

The following example assumes that you need to migrate all the test_table table data in the test_db database of both the upstream MySQL-1 and MySQL-2 instances, to the downstream test_table table in the test_db database of TiDB, in the full data plus incremental data mode.

Edit the task.yaml task configuration file as below:

# The task name. You need to use a different name for each of the multiple tasks that
# run simultaneously.
name: "test"
# The full data plus incremental data (all) migration mode.
task-mode: "all"
# The downstream TiDB configuration information.
target-database:
  host: "172.16.10.83"
  port: 4000
  user: "root"
  password: ""
# Configuration of all the upstream MySQL instances required by the current data migration task.
mysql-instances:
-
  # The ID of upstream instances or the migration group. You can refer to the configuration of `source_id` in the "inventory.ini" file or in the "dm-master.toml" file.
  source-id: "mysql-replica-01"
  # The configuration item name of the block and allow lists of the name of the
  # database/table to be migrated, used to quote the global block and allow
  # lists configuration that is set in the global block-allow-list below.
  block-allow-list: "global"  # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2.
  # The configuration item name of the dump processing unit, used to quote the global configuration of the dump unit.
  mydumper-config-name: "global"
-
  source-id: "mysql-replica-02"
  block-allow-list: "global"  # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2.
  mydumper-config-name: "global"
# The global configuration of block and allow lists. Each instance can quote it by the
# configuration item name.
block-allow-list:                     # Use black-white-list if the DM version is earlier than or equal to v2.0.0-beta.2.
  global:
    do-tables:                        # The allow list of upstream tables to be migrated.
    - db-name: "test_db"              # The database name of the table to be migrated.
      tbl-name: "test_table"          # The name of the table to be migrated.
# The global configuration of the dump unit. Each instance can quote it by the configuration item name.
mydumpers:
  global:
    extra-args: ""

Step 5: Start the data migration task

To detect possible errors of data migration configuration in advance, DM provides the precheck feature:

DM automatically checks the corresponding privileges and configuration while starting the data migration task.
You can also use the check-task command to manually precheck whether the upstream MySQL instance configuration satisfies the DM requirements.

For details about the precheck feature, see Precheck the upstream MySQL instance configuration.

Note

Before starting the data migration task for the first time, you should have got the upstream configured. Otherwise, an error is reported while you start the task.

Run the tiup dmctl command to start the data migration tasks. task.yaml is the configuration file that is edited above.

tiup dmctl --master-addr 172.16.10.71:8261 start-task ./task.yaml

If the above command returns the following result, it indicates the task is successfully started.

{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "172.16.10.72:8262",
            "msg": ""
        },
        {
            "result": true,
            "worker": "172.16.10.73:8262",
            "msg": ""
        }
    ]
}

If you fail to start the data migration task, modify the configuration according to the returned prompt and then run the start-task task.yaml command to restart the task.

Step 6: Check the data migration task

If you need to check the task state or whether a certain data migration task is running in the DM cluster, run the following command in tiup dmctl:

tiup dmctl --master-addr 172.16.10.71:8261 query-status

Step 7: Stop the data migration task

If you do not need to migrate data any more, run the following command in tiup dmctl to stop the task:

tiup dmctl --master-addr 172.16.10.71:8261 stop-task test

test is the task name that you set in the name configuration item of the task.yaml configuration file.

Step 8: Monitor the task and check logs

Assuming that Prometheus, Alertmanager, and Grafana are successfully deployed along with the DM cluster deployment using TiUP, and the Grafana address is 172.16.10.71. To view the alert information related to DM, you can open http://172.16.10.71:9093 in a browser and enter into Alertmanager; to check monitoring metrics, go to http://172.16.10.71:3000, and choose the DM dashboard.

While the DM cluster is running, DM-master, DM-worker, and dmctl output the monitoring metrics information through logs. The log directory of each component is as follows:

DM-master log directory: It is specified by the --log-file DM-master process parameter. If DM is deployed using TiUP, the log directory is {log_dir} in the DM-master node.
DM-worker log directory: It is specified by the --log-file DM-worker process parameter. If DM is deployed using TiUP, the log directory is {log_dir} in the DM-worker node.

Migrate Data Using DM