Restore Data from S3-Compatible Storage Using TiDB Lightning

This document describes how to restore the TiDB cluster data backed up using TiDB Operator in Kubernetes.

The restore method described in this document is implemented based on CustomResourceDefinition (CRD) in TiDB Operator v1.1 or later versions. For the underlying implementation, TiDB Lightning TiDB-backend is used to perform the restore.

TiDB Lightning is a tool used for fast full import of large amounts of data into a TiDB cluster. It reads data from local disks, Google Cloud Storage (GCS) or Amazon S3. TiDB Lightning supports three backends: Importer-backend, Local-backend, and TiDB-backend. In this document, TiDB-backend is used. For the differences of these backends and how to choose backends, see TiDB Lightning Backends. To import data using Importer-backend or Local-backend, see Import Data.

This document shows an example in which the backup data stored in the specified path on the S3-compatible storage is restored to the TiDB cluster.

Usage scenarios

You can use the restore solution introduced in this document if you need to export the backup data from S3 to a TiDB cluster, with the following requirements:

  • To restore data with lower resource usage and lower network bandwidth usage. A restore speed of 50 GB/h is acceptable.
  • To import data into the cluster with ACID compliance.
  • The TiDB cluster can still provide services during the restore process.

Prerequisites

Before you perform the data restore, you need to prepare the restore environment and get the required database account privileges.

Prepare the restore environment

  1. Download backup-rbac.yaml and execute the following command to create the role-based access control (RBAC) resources in the test2 namespace:

    1. kubectl apply -f backup-rbac.yaml -n test2

  2. Grant permissions to the remote storage.

    To grant permissions to access S3-compatible remote storage, refer to AWS account permissions.

    If you use Ceph as the backend storage for testing, you can grant permissions by using AccessKey and SecretKey.

  3. Create the restore-demo2-tidb-secret secret which stores the root account and password needed to access the TiDB cluster:

    1. kubectl create secret generic restore-demo2-tidb-secret --from-literal=password=${password} --namespace=test2

Get the required database account privileges

Before you use TiDB Lightning to restore the backup data in S3 to the TiDB cluster, make sure that you have the following database account privileges:

PrivilegesScope
SELECTTables
INSERTTables
UPDATETables
DELETETables
CREATEDatabases, tables
DROPDatabases, tables
ALTERTables

Restore process

Restore Data Using TiDB Lightning - 图1Note

Because of the rclone issue, if the backup data is stored in Amazon S3 and the AWS-KMS encryption is enabled, you need to add the following spec.s3.options configuration to the YAML file in the examples of this section:

  1. spec:
  2.   ...
  3.   s3:
  4.     ...
  5.     options:
  6.     - --ignore-checksum

This section lists multiple storage access methods. Only follow the method that matches your situation. The methods are as follows:

  • Amazon S3 by importing AccessKey and SecretKey
  • Ceph by importing AccessKey and SecretKey
  • Amazon S3 by binding IAM with Pod
  • Amazon S3 by binding IAM with ServiceAccount

  1. Create Restore customer resource (CR) and restore the specified backup data to the TiDB cluster.

    • Method 1: Create the Restore CR, and restore the cluster data from Ceph by importing AccessKey and SecretKey to grant permissions:

      1. kubectl apply -f restore.yaml

      The content of restore.yaml is as follows:

      1. ---
      2. apiVersion: pingcap.com/v1alpha1
      3. kind: Restore
      4. metadata:
      5.   name: demo2-restore
      6.   namespace: test2
      7. spec:
      8.   backupType: full
      9.   to:
      10.     host: ${tidb_host}
      11.     port: ${tidb_port}
      12.     user: ${tidb_user}
      13.     secretName: restore-demo2-tidb-secret
      14.   s3:
      15.     provider: ceph
      16.     endpoint: ${endpoint}
      17.     secretName: s3-secret
      18.     path: s3://${backup_path}
      19.   # storageClassName: local-storage
      20.   storageSize: 1Gi

    • Method 2: Create the Restore CR, and restore the cluster data from Amazon S3 by importing AccessKey and SecretKey to grant permissions:

      1. kubectl apply -f restore.yaml

      The restore.yaml file has the following content:

      1. ---
      2. apiVersion: pingcap.com/v1alpha1
      3. kind: Restore
      4. metadata:
      5.   name: demo2-restore
      6.   namespace: test2
      7. spec:
      8.   backupType: full
      9.   to:
      10.     host: ${tidb_host}
      11.     port: ${tidb_port}
      12.     user: ${tidb_user}
      13.     secretName: restore-demo2-tidb-secret
      14.   s3:
      15.     provider: aws
      16.     region: ${region}
      17.     secretName: s3-secret
      18.     path: s3://${backup_path}
      19.   # storageClassName: local-storage
      20.   storageSize: 1Gi

    • Method 3: Create the Restore CR, and restore the cluster data from Amazon S3 by binding IAM with Pod to grant permissions:

      1. kubectl apply -f restore.yaml

      The content of restore.yaml is as follows:

      1. ---
      2. apiVersion: pingcap.com/v1alpha1
      3. kind: Restore
      4. metadata:
      5.   name: demo2-restore
      6.   namespace: test2
      7.   annotations:
      8.     iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user
      9. spec:
      10.   backupType: full
      11.   to:
      12.     host: ${tidb_host}
      13.     port: ${tidb_port}
      14.     user: ${tidb_user}
      15.     secretName: restore-demo2-tidb-secret
      16.   s3:
      17.     provider: aws
      18.     region: ${region}
      19.     path: s3://${backup_path}
      20.   # storageClassName: local-storage
      21.   storageSize: 1Gi

    • Method 4: Create the Restore CR, and restore the cluster data from Amazon S3 by binding IAM with ServiceAccount to grant permissions:

      1. kubectl apply -f restore.yaml

      The content of restore.yaml is as follows:

      1. ---
      2. apiVersion: pingcap.com/v1alpha1
      3. kind: Restore
      4. metadata:
      5.   name: demo2-restore
      6.   namespace: test2
      7. spec:
      8.   backupType: full
      9.   serviceAccount: tidb-backup-manager
      10.   to:
      11.     host: ${tidb_host}
      12.     port: ${tidb_port}
      13.     user: ${tidb_user}
      14.     secretName: restore-demo2-tidb-secret
      15.   s3:
      16.     provider: aws
      17.     region: ${region}
      18.     path: s3://${backup_path}
      19.   # storageClassName: local-storage
      20.   storageSize: 1Gi

  2. After creating the Restore CR, execute the following command to check the restore status:

    1. kubectl get rt -n test2 -owide

The example above restores data from the spec.s3.path path on S3-compatible storage to the spec.to.host TiDB cluster. For more information about S3-compatible storage configuration, refer to S3 storage fields.

For more information about the Restore CR fields, refer to Restore CR fields.

Restore Data Using TiDB Lightning - 图2Note

TiDB Operator creates a PVC for data recovery. The backup data is downloaded from the remote storage to the PV first, and then restored. If you want to delete this PVC after the recovery is completed, you can refer to Delete Resource to delete the recovery Pod first, and then delete the PVC.

Troubleshooting

If you encounter any problem during the restore process, refer to Common Deployment Failures.