Deploy Data Migration Using DM Binary

This document introduces how to quickly deploy the Data Migration (DM) cluster using DM binary.

Use Binary - 图1

Note

In the production environment, it is recommended to use TiUP to deploy a DM cluster.

Download DM binary

The DM binary is included in the TiDB Toolkit. To download the TiDB Toolkit, see Download TiDB Tools.

Sample scenario

Suppose that you deploy a DM cluster based on this sample scenario:

Two DM-worker nodes and three DM-master nodes are deployed on five servers.

Here is the address of each node:

InstanceServer addressPort
DM-master1192.168.0.48261
DM-master2192.168.0.58261
DM-master3192.168.0.68261
DM-worker1192.168.0.78262
DM-worker2192.168.0.88262

Based on this scenario, the following sections describe how to deploy the DM cluster.

Use Binary - 图2

Note

  • If you deploy multiple DM-master or DM-worker instances in a single server, the port and working directory of each instance must be unique.

  • If you do not need to ensure high availability of the DM cluster, deploy only one DM-master node, and the number of deployed DM-worker nodes must be no less than the number of upstream MySQL/MariaDB instances to be migrated.

  • To ensure high availability of the DM cluster, it is recommended to deploy three DM-master nodes, and the number of deployed DM-worker nodes must be greater than the number of upstream MySQL/MariaDB instances to be migrated (for example, the number of DM-worker nodes is two more than the number of upstream instances).

  • Make sure that the ports among the following components are interconnected:

    • The 8291 ports among the DM-master nodes are interconnected.
    • Each DM-master node can connect to the 8262 ports of all DM-worker nodes.
    • Each DM-worker node can connect to the 8261 port of all DM-master nodes.

Deploy DM-master

You can configure DM-master by using command-line parameters or the configuration file.

DM-master command-line parameters

The following is the description of DM-master command-line parameters:

  1. ./dm-master --help
  1. Usage of dm-master:
  2.   -L string
  3.         log level: debug, info, warn, error, fatal (default "info")
  4.   -V    prints version and exit
  5.   -advertise-addr string
  6.         advertise address for client traffic (default "${master-addr}")
  7.   -advertise-peer-urls string
  8.         advertise URLs for peer traffic (default "${peer-urls}")
  9.   -config string
  10.         path to config file
  11.   -data-dir string
  12.         path to the data directory (default "default.${name}")
  13.   -initial-cluster string
  14.         initial cluster configuration for bootstrapping, e.g. dm-master=http://127.0.0.1:8291
  15.   -join string
  16.         join to an existing cluster (usage: cluster's "${master-addr}" list, e.g. "127.0.0.1:8261,127.0.0.1:18261"
  17.   -log-file string
  18.         log file path
  19.   -master-addr string
  20.         master API server and status addr
  21.   -name string
  22.         human-readable name for this DM-master member
  23.   -peer-urls string
  24.         URLs for peer traffic (default "http://127.0.0.1:8291")
  25.   -print-sample-config
  26.         print sample config file of dm-worker

Use Binary - 图3

Note

In some situations, you cannot use the above method to configure DM-master because some configurations are not exposed to the command line. In such cases, use the configuration file instead.

DM-master configuration file

The following is the configuration file of DM-master. It is recommended that you configure DM-master by using this method.

  1. Write the following configuration to conf/dm-master1.toml:

    1. # Master Configuration.
    2. name = "master1"
    4. # Log configurations.
    5. log-level = "info"
    6. log-file = "dm-master.log"
    8. # The listening address of DM-master.
    9. master-addr = "192.168.0.4:8261"
    11. # The peer URLs of DM-master.
    12. peer-urls = "192.168.0.4:8291"
    14. # The value of `initial-cluster` is the combination of the `advertise-peer-urls` value of all DM-master nodes in the initial cluster.
    15. initial-cluster = "master1=http://192.168.0.4:8291,master2=http://192.168.0.5:8291,master3=http://192.168.0.6:8291"

  2. Execute the following command in the terminal to run DM-master:

    1. ./dm-master -config conf/dm-master1.toml

    Use Binary - 图4

    Note

    The console does not output logs after this command is executed. If you want to view the runtime log, you can execute tail -f dm-master.log.

  3. For DM-master2 and DM-master3, change name in the configuration file to master2 and master3 respectively, and change peer-urls to 192.168.0.5:8291 and 192.168.0.6:8291 respectively. Then repeat Step 2.

Deploy DM-worker

You can configure DM-worker by using command-line parameters or the configuration file.

DM-worker command-line parameters

The following is the description of the DM-worker command-line parameters:

  1. ./dm-worker --help
  1. Usage of worker:
  2.   -L string
  3.         log level: debug, info, warn, error, fatal (default "info")
  4.   -V    prints version and exit
  5.   -advertise-addr string
  6.         advertise address for client traffic (default "${worker-addr}")
  7.   -config string
  8.         path to config file
  9.   -join string
  10.         join to an existing cluster (usage: dm-master cluster's "${master-addr}")
  11.   -keepalive-ttl int
  12.         dm-worker's TTL for keepalive with etcd (in seconds) (default 10)
  13.   -log-file string
  14.         log file path
  15.   -name string
  16.         human-readable name for DM-worker member
  17.   -print-sample-config
  18.         print sample config file of dm-worker
  19.   -worker-addr string
  20.         listen address for client traffic

Use Binary - 图5

Note

In some situations, you cannot use the above method to configure DM-worker because some configurations are not exposed to the command line. In such cases, use the configuration file instead.

DM-worker configuration file

The following is the DM-worker configuration file. It is recommended that you configure DM-worker by using this method.

  1. Write the following configuration to conf/dm-worker1.toml:

    1. # Worker Configuration.
    2. name = "worker1"
    4. # Log configuration.
    5. log-level = "info"
    6. log-file = "dm-worker.log"
    8. # DM-worker address.
    9. worker-addr = ":8262"
    11. # The master-addr configuration of the DM-master nodes in the cluster.
    12. join = "192.168.0.4:8261,192.168.0.5:8261,192.168.0.6:8261"

  2. Execute the following command in the terminal to run DM-worker:

    1. ./dm-worker -config conf/dm-worker1.toml

  3. For DM-worker2, change name in the configuration file to worker2. Then repeat Step 2.

Now, a DM cluster is successfully deployed.