Manage Data Source Configurations in TiDB Data Migration

This document introduces how to manage data source configurations, including encrypting the MySQL password, operating the data source, and changing the bindings between upstream MySQL instances and DM-workers using dmctl.

Encrypt the database password

In DM configuration files, it is recommended to use the password encrypted with dmctl. For one original password, the encrypted password is different after each encryption.

Manage Data Sources - 图1

Note

Starting from v8.0.0, you must configure secret-key-path for DM-master before using the dmctl encrypt command.

  1. ./dmctl encrypt 'abc!@#123'
  1. MKxn0Qo3m3XOyjCnhEMtsUCm83EhGQDZ/T4=

Operate data source

You can use the operate-source command to load, list or remove the data source configurations to the DM cluster.

  1. help operate-source
  1. `create`/`stop`/`show` upstream MySQL/MariaDB source.
  2. Usage:
  3. dmctl operate-source <operate-type> [config-file ...] [--print-sample-config] [flags]
  4. Flags:
  5. -h, --help help for operate-source
  6. -p, --print-sample-config print sample config file of source
  7. Global Flags:
  8. -s, --source strings MySQL Source ID

Flags description

  • create: Creates one or more upstream database sources. When creating multiple data sources fails, DM rolls back to the state where the command was not executed.

  • stop: Stops one or more upstream database sources. When stopping multiple data sources fails, some data sources might be stopped.

  • show: Shows the added data source and the corresponding DM-worker.

  • config-file: Specifies the file path of source.yaml and can pass multiple file paths.

  • --print-sample-config: Prints the sample configuration file. This parameter ignores other parameters.

Usage example

Use the following operate-source command to create a source configuration file:

  1. operate-source create ./source.yaml

For the configuration of source.yaml, refer to Upstream Database Configuration File Introduction.

The following is an example of the returned result:

  1. {
  2. "result": true,
  3. "msg": "",
  4. "sources": [
  5. {
  6. "result": true,
  7. "msg": "",
  8. "source": "mysql-replica-01",
  9. "worker": "dm-worker-1"
  10. }
  11. ]
  12. }

Check data source configurations

Manage Data Sources - 图2

Note

The config command is only supported in DM v6.0 and later versions. For earlier versions, you must use the get-config command.

If you know the source-id, you can run dmctl --master-addr <master-addr> config source <source-id> to get the data source configuration.

  1. config source mysql-replica-01
  1. {
  2. "result": true,
  3. "msg": "",
  4. "cfg": "enable-gtid: false
  5. flavor: mysql
  6. source-id: mysql-replica-01
  7. from:
  8. host: 127.0.0.1
  9. port: 8407
  10. user: root
  11. password: '******'
  12. }

If you don’t know the source-id, you can run dmctl --master-addr <master-addr> operate-source show to list all data sources first.

  1. operate-source show
  1. {
  2. "result": true,
  3. "msg": "",
  4. "sources": [
  5. {
  6. "result": true,
  7. "msg": "source is added but there is no free worker to bound",
  8. "source": "mysql-replica-02",
  9. "worker": ""
  10. },
  11. {
  12. "result": true,
  13. "msg": "",
  14. "source": "mysql-replica-01",
  15. "worker": "dm-worker-1"
  16. }
  17. ]
  18. }

Change the bindings between upstream MySQL instances and DM-workers

You can use the transfer-source command to change the bindings between upstream MySQL instances and DM-workers.

  1. help transfer-source
  1. Transfers an upstream MySQL/MariaDB source to a free worker.
  2. Usage:
  3. dmctl transfer-source <source-id> <worker-id> [flags]
  4. Flags:
  5. -h, --help help for transfer-source
  6. Global Flags:
  7. -s, --source strings MySQL Source ID.

Before transferring, DM checks whether the worker to be unbound still has running tasks. If the worker has any running tasks, you need to pause the tasks first, change the binding, and then resume the tasks.

Usage example

If you do not know the bindings of DM-workers, you can run dmctl --master-addr <master-addr> list-member --worker to list the current bindings of all workers.

  1. list-member --worker
  1. {
  2. "result": true,
  3. "msg": "",
  4. "members": [
  5. {
  6. "worker": {
  7. "msg": "",
  8. "workers": [
  9. {
  10. "name": "dm-worker-1",
  11. "addr": "127.0.0.1:8262",
  12. "stage": "bound",
  13. "source": "mysql-replica-01"
  14. },
  15. {
  16. "name": "dm-worker-2",
  17. "addr": "127.0.0.1:8263",
  18. "stage": "free",
  19. "source": ""
  20. }
  21. ]
  22. }
  23. }
  24. ]
  25. }

In the above example, mysql-replica-01 is bound to dm-worker-1. The below command transfers the binding worker of mysql-replica-01 to dm-worker-2.

  1. transfer-source mysql-replica-01 dm-worker-2
  1. {
  2. "result": true,
  3. "msg": ""
  4. }

Check whether the command takes effect by running dmctl --master-addr <master-addr> list-member --worker.

  1. list-member --worker
  1. {
  2. "result": true,
  3. "msg": "",
  4. "members": [
  5. {
  6. "worker": {
  7. "msg": "",
  8. "workers": [
  9. {
  10. "name": "dm-worker-1",
  11. "addr": "127.0.0.1:8262",
  12. "stage": "free",
  13. "source": ""
  14. },
  15. {
  16. "name": "dm-worker-2",
  17. "addr": "127.0.0.1:8263",
  18. "stage": "bound",
  19. "source": "mysql-replica-01"
  20. }
  21. ]
  22. }
  23. }
  24. ]
  25. }