Apply Hotfix to DM Clusters Online

If you need to dynamically replace the binaries of a service while the cluster is running (that is, to keep the cluster available during the replacement), you can use the tiup dm patch command. The command does the following:

  • Uploads the binary package for replacement to the target machine.
  • Takes the related nodes offline using the API.
  • Stops the target service.
  • Unpacks the binary package and replaces the service.
  • Starts the target service.

Syntax

  1. tiup dm patch <cluster-name> <package-path> [flags]
  • <cluster-name>: The name of the cluster to be operated
  • <package-path>: The path to the binary package used for replacement

Preparation

You need to pack the binary package required for this command in advance according to the following steps:

  • Determine the name ${component} of the component to be replaced (dm-master, dm-worker …), the ${version} of the component (v2.0.0, v2.0.1 …), and the operating system ${os} and platform ${arch} on which the component runs.
  • Download the current component package using the command wget https://tiup-mirrors.pingcap.com/${component}-${version}-${os}-${arch}.tar.gz -O /tmp/${component}-${version}-${os}-${arch}.tar.gz.
  • Run mkdir -p /tmp/package && cd /tmp/package to create a temporary directory to pack files.
  • Run tar xf /tmp/${component}-${version}-${os}-${arch}.tar.gz to unpack the original binary package.
  • Run find . to view the file structure in the temporary package directory.
  • Copy the binary files or configuration files to the corresponding locations in the temporary directory.
  • Run tar czf /tmp/${component}-hotfix-${os}-${arch}.tar.gz * to pack the files in the temporary directory.
  • Finally, you can use /tmp/${component}-hotfix-${os}-${arch}.tar.gz as the value of <package-path> in the tiup dm patch command.

Options

--overwrite

  • After you patch a certain component (such as dm-worker), when the tiup-dm scales out the component, tiup-dm uses the original component version by default. To use the version that you patch when the cluster scales out in the future, you need to specify the option --overwrite in the command.
  • Data type: BOOLEAN
  • This option is disabled by default with the false value. To enable this option, add this option to the command, and either pass the true value or do not pass any value.

-N, —node

  • Specifies the nodes to be replaced. The value of this option is a comma-separated list of node IDs. You can get the node IDs from the first column of the cluster status table returned by the [tiup dm display](/tiup/tiup-component-dm-display.md) command.
  • Data type: STRING
  • If this option is not specified, TiUP selects all nodes to replace by default.

tiup dm patch - 图1

Note

If the option -R, --role is specified at the same time, TiUP then replaces service nodes that match both the requirements of -N, --node and -R, --role.

-R, —role

  • Specifies the roles to be replaced. The value of this option is a comma-separated list of the roles of the nodes. You can get the roles of the nodes from the second column of the cluster status table returned by the [tiup dm display](/tiup/tiup-component-dm-display.md) command.
  • Data type: STRING
  • If this option is not specified, TiUP selects all roles to replace by default.

tiup dm patch - 图2

Note

If the option -N, --node is specified at the same time, TiUP then replaces service nodes that match both the requirements of -N, --node and -R, --role.

--offline

  • Declares that the current cluster is offline. When this option is specified, TiUP DM only replaces the binary files of the cluster components in place without restarting the service.

-h, —help

  • Prints help information.
  • Data type: BOOLEAN
  • This option is disabled by default with the false value. To enable this option, add this option to the command, and either pass the true value or do not pass any value.

Example

The following example shows how to apply v5.3.0-hotfix to the v5.3.0 cluster deployed using TiUP. The operations might vary if you deploy the cluster using other methods.

tiup dm patch - 图3

Note

Hotfix is used only for emergency fixes. Its daily maintenance is complicated. It is recommend that you upgrade the DM cluster to an official version as soon as it is released.

Preparations

Before applying a hotfix, prepare the hotfix package dm-linux-amd64.tar.gz and confirm the current DM software version:

  1. /home/tidb/dm/deploy/dm-master-8261/bin/dm-master/dm-master -V

Output:

  1. Release Version: v5.3.0
  3. Git Commit Hash: 20626babf21fc381d4364646c40dd84598533d66
  4. Git Branch: heads/refs/tags/v5.3.0
  5. UTC Build Time: 2021-11-29 08:29:49
  6. Go Version: go version go1.16.4 linux/amd64

Prepare the patch package and apply it to the DM cluster

  1. Prepare the DM software package that matches the current version:

    1. mkdir -p /tmp/package
    2. tar -zxvf /root/.tiup/storage/dm/packages/dm-master-v5.3.0-linux-amd64.tar.gz -C /tmp/package/
    3. tar -zxvf /root/.tiup/storage/dm/packages/dm-worker-v5.3.0-linux-amd64.tar.gz -C /tmp/package/

  2. Replace the binary file with the hotfix package:

    1. # Decompress the hotfix package and use it to replace the binary file.
    2. cd /root; tar -zxvf dm-linux-amd64.tar.gz
    3. cp /root/dm-linux-amd64/bin/dm-master /tmp/package/dm-master/dm-master
    4. cp /root/dm-linux-amd64/bin/dm-worker /tmp/package/dm-worker/dm-worker
    5. # Re-package the modified files.
    6. # Note that the packaging method might be different for other deployment methods.
    7. cd /tmp/package/ && tar -czvf dm-master-hotfix-linux-amd64.tar.gz dm-master/
    8. cd /tmp/package/ && tar -czvf dm-worker-hotfix-linux-amd64.tar.gz dm-worker/

  3. Apply the hotfix:

    Query the cluster status. The following uses the cluster named dm-test as an example:

    1. tiup dm display dm-test

    Output:

    1. Cluster type:       dm
    2. Cluster name:       dm-test
    3. Cluster version:    v5.3.0
    4. Deploy user:        tidb
    5. SSH type:           builtin
    6. ID                  Role                 Host           Ports      OS/Arch       Status     Data Dir                              Deploy Dir
    7. --                  ----                 ----           -----      -------       ------     --------                              ----------
    8. 172.16.100.21:9093  alertmanager         172.16.100.21  9093/9094  linux/x86_64  Up         /home/tidb/dm/data/alertmanager-9093  /home/tidb/dm/deploy/alertmanager-9093
    9. 172.16.100.21:8261  dm-master            172.16.100.21  8261/8291  linux/x86_64  Healthy|L  /home/tidb/dm/data/dm-master-8261     /home/tidb/dm/deploy/dm-master-8261
    10. 172.16.100.21:8262  dm-worker            172.16.100.21  8262       linux/x86_64  Free       /home/tidb/dm/data/dm-worker-8262     /home/tidb/dm/deploy/dm-worker-8262
    11. 172.16.100.21:3000  grafana              172.16.100.21  3000       linux/x86_64  Up         -                                     /home/tidb/dm/deploy/grafana-3000
    12. 172.16.100.21:9090  prometheus           172.16.100.21  9090       linux/x86_64  Up         /home/tidb/dm/data/prometheus-9090    /home/tidb/dm/deploy/prometheus-9090
    13. Total nodes: 5

    Apply the hotfix to the specified node or specified role. If both -R and -N are specified, the intersection will be taken.

    1. # Apply hotfix to a specified node.
    2. tiup dm patch dm-test dm-master-hotfix-linux-amd64.tar.gz -N 172.16.100.21:8261
    3. tiup dm patch dm-test dm-worker-hotfix-linux-amd64.tar.gz -N 172.16.100.21:8262
    4. # Apply hotfix to a specified role.
    5. tiup dm patch dm-test dm-master-hotfix-linux-amd64.tar.gz -R dm-master
    6. tiup dm patch dm-test dm-worker-hotfix-linux-amd64.tar.gz -R dm-worker

  4. Query the hotfix application result:

    1. /home/tidb/dm/deploy/dm-master-8261/bin/dm-master/dm-master -V

    Output:

    1. Release Version: v5.3.0-20211230
    2. Git Commit Hash: ca7070c45013c24d34bd9c1e936071253451d707
    3. Git Branch: heads/refs/tags/v5.3.0-20211230
    4. UTC Build Time: 2022-01-05 14:19:02
    5. Go Version: go version go1.16.4 linux/amd64

    The cluster information changes accordingly:

    1. tiup dm display dm-test

    Output:

    1. Starting component `dm`: /root/.tiup/components/dm/v1.8.1/tiup-dm display dm-test
    2. Cluster type:       dm
    3. Cluster name:       dm-test
    4. Cluster version:    v5.3.0
    5. Deploy user:        tidb
    6. SSH type:           builtin
    7. ID                  Role                 Host           Ports      OS/Arch       Status     Data Dir                              Deploy Dir
    8. --                  ----                 ----           -----      -------       ------     --------                              ----------
    9. 172.16.100.21:9093  alertmanager         172.16.100.21  9093/9094  linux/x86_64  Up         /home/tidb/dm/data/alertmanager-9093  /home/tidb/dm/deploy/alertmanager-9093
    10. 172.16.100.21:8261  dm-master (patched)  172.16.100.21  8261/8291  linux/x86_64  Healthy|L  /home/tidb/dm/data/dm-master-8261     /home/tidb/dm/deploy/dm-master-8261
    11. 172.16.100.21:8262  dm-worker (patched)  172.16.100.21  8262       linux/x86_64  Free       /home/tidb/dm/data/dm-worker-8262     /home/tidb/dm/deploy/dm-worker-8262
    12. 172.16.100.21:3000  grafana              172.16.100.21  3000       linux/x86_64  Up         -                                     /home/tidb/dm/deploy/grafana-3000
    13. 172.16.100.21:9090  prometheus           172.16.100.21  9090       linux/x86_64  Up         /home/tidb/dm/data/prometheus-9090    /home/tidb/dm/deploy/prometheus-9090
    14. Total nodes: 5

<< Back to the previous page - TiUP DM command list