Back up Data to S3-Compatible Storage Using Dumpling

This document describes how to back up the data of the TiDB cluster in Kubernetes to an S3-compatible storage. “Backup” in this document refers to full backup (ad-hoc full backup and scheduled full backup).

The backup method described in this document is implemented based on CustomResourceDefinition (CRD) in TiDB Operator v1.1 or later versions. For the underlying implementation, Dumpling is used to get the logic backup of the TiDB cluster, and then this backup data is sent to the S3-compatible storage.

Dumpling is a data export tool that exports data stored in TiDB/MySQL as SQL or CSV files and can be used to make a logical full backup or export.

Usage scenarios

You can use the backup method described in this document if you want to make an ad-hoc full backup or scheduled full backup of the TiDB cluster data to S3-compatible storages with the following needs:

  • To export SQL or CSV files
  • To limit the memory usage of a single SQL statement
  • To export the historical data snapshot of TiDB

Ad-hoc full backup to S3-compatible storage

Ad-hoc full backup describes the backup by creating a Backup custom resource (CR) object. TiDB Operator performs the specific backup operation based on this Backup object. If an error occurs during the backup process, TiDB Operator does not retry and you need to handle this error manually.

For the current S3-compatible storage types, Ceph and Amazon S3 work normally as tested. Therefore, this document shows examples in which the data of the demo1 TiDB cluster in the tidb-cluster Kubernetes namespace is backed up to Ceph and Amazon S3 respectively.

Prerequisites

Before you use Dumpling to back up the TiDB cluster data to the S3-compatible storage, make sure that you have the following privileges:

  • The SELECT and UPDATE privileges of the mysql.tidb table: Before and after the backup, the Backup CR needs a database account with these privileges to adjust the GC time.
  • The global privileges: SELECT, RELOAD, LOCK TABLES and REPLICATION CLIENT

An example for creating a backup user:

  1. CREATE USER 'backup'@'%' IDENTIFIED BY '...';
  2. GRANT
  3.   SELECT, RELOAD, LOCK TABLES, REPLICATION CLIENT
  4.   ON *.*
  5.   TO 'backup'@'%';
  6. GRANT
  7.   UPDATE, SELECT
  8.   ON mysql.tidb
  9.   TO 'backup'@'%';

Step 1: Prepare for ad-hoc full backup

  1. Execute the following command to create the role-based access control (RBAC) resources in the tidb-cluster namespace based on backup-rbac.yaml:

    1. kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.3.2/manifests/backup/backup-rbac.yaml -n tidb-cluster

  2. Grant permissions to the remote storage.

    To grant permissions to access S3-compatible remote storage, refer to AWS account permissions.

    If you use Ceph as the backend storage for testing, you can grant permissions by using AccessKey and SecretKey.

  3. Create the backup-demo1-tidb-secret secret which stores the root account and password needed to access the TiDB cluster:

    1. kubectl create secret generic backup-demo1-tidb-secret --from-literal=password=${password} --namespace=tidb-cluster

Step 2: Perform ad-hoc backup

Back Up Data Using Dumpling - 图1Note

Because of the rclone issue, if the backup data is stored in Amazon S3 and the AWS-KMS encryption is enabled, you need to add the following spec.s3.options configuration to the YAML file in the examples of this section:

  1. spec:
  2.   ...
  3.   s3:
  4.     ...
  5.     options:
  6.     - --ignore-checksum

This section lists multiple storage access methods. Only follow the method that matches your situation. The methods are as follows:

  • Amazon S3 by importing AccessKey and SecretKey
  • Ceph by importing AccessKey and SecretKey
  • Amazon S3 by binding IAM with Pod

  • Amazon S3 by binding IAM with ServiceAccount

  • Method 1: Create the Backup CR, and back up cluster data to Amazon S3 by importing AccessKey and SecretKey to grant permissions:

    1. kubectl apply -f backup-s3.yaml

    The content of backup-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: Backup
    4. metadata:
    5.   name: demo1-backup-s3
    6.   namespace: tidb-cluster
    7. spec:
    8.   from:
    9.     host: ${tidb_host}
    10.     port: ${tidb_port}
    11.     user: ${tidb_user}
    12.     secretName: backup-demo1-tidb-secret
    13.   s3:
    14.     provider: aws
    15.     secretName: s3-secret
    16.     region: ${region}
    17.     bucket: ${bucket}
    18.     # prefix: ${prefix}
    19.     # storageClass: STANDARD_IA
    20.     # acl: private
    21.     # endpoint:
    22. # dumpling:
    23. #  options:
    24. #  - --threads=16
    25. #  - --rows=10000
    26. #  tableFilter:
    27. #  - "test.*"
    28.   # storageClassName: local-storage
    29.   storageSize: 10Gi

  • Method 2: Create the Backup CR, and back up data to Ceph by importing AccessKey and SecretKey to grant permissions:

    1. kubectl apply -f backup-s3.yaml

    The content of backup-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: Backup
    4. metadata:
    5.   name: demo1-backup-s3
    6.   namespace: tidb-cluster
    7. spec:
    8.   from:
    9.     host: ${tidb_host}
    10.     port: ${tidb_port}
    11.     user: ${tidb_user}
    12.     secretName: backup-demo1-tidb-secret
    13.   s3:
    14.     provider: ceph
    15.     secretName: s3-secret
    16.     endpoint: ${endpoint}
    17.     # prefix: ${prefix}
    18.     bucket: ${bucket}
    19. # dumpling:
    20. #  options:
    21. #  - --threads=16
    22. #  - --rows=10000
    23. #  tableFilter:
    24. #  - "test.*"
    25.   # storageClassName: local-storage
    26.   storageSize: 10Gi

  • Method 3: Create the Backup CR, and back up data to Amazon S3 by binding IAM with Pod to grant permissions:

    1. kubectl apply -f backup-s3.yaml

    The content of backup-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: Backup
    4. metadata:
    5. name: demo1-backup-s3
    6. namespace: tidb-cluster
    7. annotations:
    8.     iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user
    9. spec:
    10. backupType: full
    11. from:
    12.     host: ${tidb_host}
    13.     port: ${tidb_port}
    14.     user: ${tidb_user}
    15.     secretName: backup-demo1-tidb-secret
    16. s3:
    17.     provider: aws
    18.     region: ${region}
    19.     bucket: ${bucket}
    20.     # prefix: ${prefix}
    21.     # storageClass: STANDARD_IA
    22.     # acl: private
    23.     # endpoint:
    24. # dumpling:
    25. #  options:
    26. #  - --threads=16
    27. #  - --rows=10000
    28. #  tableFilter:
    29. #  - "test.*"
    30.   # storageClassName: local-storage
    31.   storageSize: 10Gi

  • Method 4: Create the Backup CR, and back up data to Amazon S3 by binding IAM with ServiceAccount to grant permissions:

    1. kubectl apply -f backup-s3.yaml

    The content of backup-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: Backup
    4. metadata:
    5. name: demo1-backup-s3
    6. namespace: tidb-cluster
    7. spec:
    8. backupType: full
    9. serviceAccount: tidb-backup-manager
    10. from:
    11.     host: ${tidb_host}
    12.     port: ${tidb_port}
    13.     user: ${tidb_user}
    14.     secretName: backup-demo1-tidb-secret
    15. s3:
    16.     provider: aws
    17.     region: ${region}
    18.     bucket: ${bucket}
    19.     # prefix: ${prefix}
    20.     # storageClass: STANDARD_IA
    21.     # acl: private
    22.     # endpoint:
    23. # dumpling:
    24. #  options:
    25. #  - --threads=16
    26. #  - --rows=10000
    27. #  tableFilter:
    28. #  - "test.*"
    29.   # storageClassName: local-storage
    30.   storageSize: 10Gi

In the examples above, all data of the TiDB cluster is exported and backed up to Amazon S3 or Ceph. You can ignore the acl, endpoint, and storageClass fields in the Amazon S3 configuration. Other S3-compatible storages can also use a configuration similar to that of Amazon S3. You can also leave the fields empty if you do not need to configure them, as shown in the above Ceph configuration. For more information about S3-compatible storage configuration, refer to S3 storage fields.

spec.dumpling refers to Dumpling-related configuration. You can specify Dumpling’s operation parameters in the options field. See Dumpling Option list for more information. These configuration items of Dumpling can be ignored by default. When these items are not specified, the default values of options fields are as follows:

  1. options:
  2. - --threads=16
  3. - --rows=10000

For more information about the Backup CR fields, refer to Backup CR fields.

After creating the Backup CR, use the following command to check the backup status:

  1. kubectl get bk -n tidb-cluster -owide

To get detailed information on a backup job, use the following command. For $backup_job_name in the command, use the name from the output of the previous command.

  1. kubectl describe bk -n tidb-cluster $backup_job_name

To run ad-hoc backup again, you need to delete the backup CR and create it again.

Scheduled full backup to S3-compatible storage

You can set a backup policy to perform scheduled backups of the TiDB cluster, and set a backup retention policy to avoid excessive backup items. A scheduled full backup is described by a custom BackupSchedule CR object. A full backup is triggered at each backup time point. Its underlying implementation is the ad-hoc full backup.

Step 1: Prepare for scheduled backup

The prerequisites for the scheduled backup is the same as the prepare for ad-hoc full backup.

Step 2: Perform scheduled backup

Back Up Data Using Dumpling - 图2Note

Because of the rclone issue, if the backup data is stored in Amazon S3 and the AWS-KMS encryption is enabled, you need to add the following spec.backupTemplate.s3.options configuration to the YAML file in the examples of this section:

  1. spec:
  2.   ...
  3.   backupTemplate:
  4.     ...
  5.     s3:
  6.       ...
  7.       options:
  8.       - --ignore-checksum

  • Method 1: Create the BackupSchedule CR to enable the scheduled full backup to Amazon S3 by importing AccessKey and SecretKey to grant permissions:

    1. kubectl apply -f backup-schedule-s3.yaml

    The content of backup-schedule-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: BackupSchedule
    4. metadata:
    5.   name: demo1-backup-schedule-s3
    6.   namespace: tidb-cluster
    7. spec:
    8.   #maxBackups: 5
    9.   #pause: true
    10.   maxReservedTime: "3h"
    11.   schedule: "*/2 * * * *"
    12.   backupTemplate:
    13.     from:
    14.       host: ${tidb_host}
    15.       port: ${tidb_port}
    16.       user: ${tidb_user}
    17.       secretName: backup-demo1-tidb-secret
    18.     s3:
    19.       provider: aws
    20.       secretName: s3-secret
    21.       region: ${region}
    22.       bucket: ${bucket}
    23.       # prefix: ${prefix}
    24.       # storageClass: STANDARD_IA
    25.       # acl: private
    26.       # endpoint:
    27.   # dumpling:
    28.   #  options:
    29.   #  - --threads=16
    30.   #  - --rows=10000
    31.   #  tableFilter:
    32.   #  - "test.*"
    33.     # storageClassName: local-storage
    34.     storageSize: 10Gi

  • Method 2: Create the BackupSchedule CR to enable the scheduled full backup to Ceph by importing AccessKey and SecretKey to grant permissions:

    1. kubectl apply -f backup-schedule-s3.yaml

    The content of backup-schedule-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: BackupSchedule
    4. metadata:
    5.   name: demo1-backup-schedule-ceph
    6.   namespace: tidb-cluster
    7. spec:
    8.   #maxBackups: 5
    9.   #pause: true
    10.   maxReservedTime: "3h"
    11.   schedule: "*/2 * * * *"
    12.   backupTemplate:
    13.     from:
    14.       host: ${tidb_host}
    15.       port: ${tidb_port}
    16.       user: ${tidb_user}
    17.       secretName: backup-demo1-tidb-secret
    18.     s3:
    19.       provider: ceph
    20.       secretName: s3-secret
    21.       endpoint: ${endpoint}
    22.       bucket: ${bucket}
    23.       # prefix: ${prefix}
    24.   # dumpling:
    25.   #  options:
    26.   #  - --threads=16
    27.   #  - --rows=10000
    28.   #  tableFilter:
    29.   #  - "test.*"
    30.     # storageClassName: local-storage
    31.     storageSize: 10Gi

  • Method 3: Create the BackupSchedule CR to enable the scheduled full backup, and back up the cluster data to Amazon S3 by binding IAM with Pod to grant permissions:

    1. kubectl apply -f backup-schedule-s3.yaml

    The content of backup-schedule-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: BackupSchedule
    4. metadata:
    5.   name: demo1-backup-schedule-s3
    6.   namespace: tidb-cluster
    7.   annotations:
    8.     iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user
    9. spec:
    10.   #maxBackups: 5
    11.   #pause: true
    12.   maxReservedTime: "3h"
    13.   schedule: "*/2 * * * *"
    14.   backupTemplate:
    15.     from:
    16.       host: ${tidb_host}
    17.       port: ${tidb_port}
    18.       user: ${tidb_user}
    19.       secretName: backup-demo1-tidb-secret
    20.     s3:
    21.       provider: aws
    22.       region: ${region}
    23.       bucket: ${bucket}
    24.       # prefix: ${prefix}
    25.       # storageClass: STANDARD_IA
    26.       # acl: private
    27.       # endpoint:
    28.   # dumpling:
    29.   #  options:
    30.   #  - --threads=16
    31.   #  - --rows=10000
    32.   #  tableFilter:
    33.   #  - "test.*"
    34.     # storageClassName: local-storage
    35.     storageSize: 10Gi

  • Method 4: Create the BackupSchedule CR to enable the scheduled full backup, and back up the cluster data to Amazon S3 by binding IAM with ServiceAccount to grant permissions:

    1. kubectl apply -f backup-schedule-s3.yaml

    The content of backup-schedule-s3.yaml is as follows:

    1. ---
    2. apiVersion: pingcap.com/v1alpha1
    3. kind: BackupSchedule
    4. metadata:
    5.   name: demo1-backup-schedule-s3
    6.   namespace: tidb-cluster
    7. spec:
    8.   #maxBackups: 5
    9.   #pause: true
    10.   maxReservedTime: "3h"
    11.   schedule: "*/2 * * * *"
    12.   serviceAccount: tidb-backup-manager
    13.   backupTemplate:
    14.     from:
    15.       host: ${tidb_host}
    16.       port: ${tidb_port}
    17.       user: ${tidb_user}
    18.       secretName: backup-demo1-tidb-secret
    19.     s3:
    20.       provider: aws
    21.       region: ${region}
    22.       bucket: ${bucket}
    23.       # prefix: ${prefix}
    24.       # storageClass: STANDARD_IA
    25.       # acl: private
    26.       # endpoint:
    27.   # dumpling:
    28.   #  options:
    29.   #  - --threads=16
    30.   #  - --rows=10000
    31.   #  tableFilter:
    32.   #  - "test.*"
    33.     # storageClassName: local-storage
    34.     storageSize: 10Gi

After creating the scheduled full backup, you can use the following command to check the backup status:

  1. kubectl get bks -n tidb-cluster -owide

You can use the following command to check all the backup items:

  1. kubectl get bk -l tidb.pingcap.com/backup-schedule=demo1-backup-schedule-s3 -n tidb-cluster

From the example above, you can see that the backupSchedule configuration consists of two parts. One is the unique configuration of backupSchedule, and the other is backupTemplate.

backupTemplate specifies the configuration related to the cluster and remote storage, which is the same as the spec configuration of the Backup CR. For the unique configuration of backupSchedule, refer to BackupSchedule CR fields.

Back Up Data Using Dumpling - 图3Note

TiDB Operator creates a PVC used for both ad-hoc full backup and scheduled full backup. The backup data is stored in PV first and then uploaded to remote storage. If you want to delete this PVC after the backup is completed, you can refer to Delete Resource to delete the backup Pod first, and then delete the PVC.

If the backup data is successfully uploaded to remote storage, TiDB Operator automatically deletes the local data. If the upload fails, the local data is retained.

Delete the backup CR

After the backup, you might need to delete the backup CR. For details, refer to Delete the Backup CR.

Troubleshooting

If you encounter any problem during the backup process, refer to Common Deployment Failures.