Connecting to S3-compatible object storage

• Connection
• Authentication
  • Command line parameters
  • Environment variables
  • AWS authentication file
• Getting S3 connection parameters
  • Yandex.Cloud

The export s3 and import s3 commands for exporting data to and importing data from S3-compatible storage, respectively, use the same S3 connection and authentication parameters. For information about how to find out these parameters for some cloud providers, see the Getting S3 connection parameters section below.

Connection

To connect to S3, make sure to specify the endpoint and bucket:

--s3-endpoint HOST: S3 endpoint. HOST: Valid hostname such as storage.yandexcloud.net

--bucket STR: S3 bucket. STR: String with the bucket name.

Authentication

To establish a connection, except when importing data from a public bucket, you’ll need to authenticate under an account with write (for import) or read (for export) permission granted for this bucket.

To authenticate in S3, the following two parameters are required:

• Access key ID (access_key_id)
• Secret access key (secret_access_key)

The YDB CLI takes these parameter values from the following sources (in order of priority):

1. The command line.
2. Environment variables.
3. The ~/.aws/credentials file.

Command line parameters

--access-key STR: Access key ID --secret-key STR: Secret access key

Environment variables

If any authentication parameter is not specified in the command line, the YDB CLI tries to get it from the following environment variables:

AWS_ACCESS_KEY_ID: Access key ID AWS_SECRET_ACCESS_KEY: Secret access key

AWS authentication file

If any authentication parameter is not specified in the command line and the YDB CLI couldn’t fetch it from the environment variable, it tries to get it from the ~/.aws/credentials file that is used for authentication in the AWS CLI. You can create this file with the AWS CLI aws configure command.

Getting S3 connection parameters

Yandex.Cloud

Follow the instructions below to get Yandex.Cloud Object Storage access keys using the Yandex.Cloud CLI.

1. Install and configure the Yandex.Cloud CLI.

2. Run the following command to get the ID of your folder in the cloud (you’ll need to specify it in the commands below):

   yc config list

   

   In the command output, the cloud folder ID is in the folder-id: line:

   folder-id: b2ge70qdcff4bo9q6t19

   

3. Run the following command to create a service account:

   yc iam service-account create --name s3account

   

   You can specify any account name except s3account or use an existing one. In this case, you’ll also need to replace it when copying commands below via the clipboard.

4. Run the following command to assign roles for the service account according to the required S3 access level:

   Read (to import data to the YDB database)

   Write (to export data from the YDB database)

   yc resource-manager folder add-access-binding <folder-id> \
     --role storage.viewer --subject serviceAccount:s3account

   

   yc resource-manager folder add-access-binding <folder-id> \
     --role storage.editor --subject serviceAccount:s3account

   

   , where <folder-id> is the cloud folder ID obtained in step 2.

   You can also view a full list of Yandex.Cloud roles.

5. Run the following command to get static access keys:

   yc iam access-key create --service-account-name s3account

   

   If successful, the command returns access_key attributes and the secret value:

   access_key:
     id: aje6t3vsbj8lp9r4vk2u
     service_account_id: ajepg0mjt06siuj65usm
     created_at: "2018-11-22T14:37:51Z"
     key_id: 0n8X6WY6S24N7OjXQ0YQ
   secret: JyTRFdqw8t1kh2-OJNz4JX5ZTz9Dj1rI9hxtzMP1

   

   In this output:

   • access_key.key_id is the access key ID.
   • secret is the secret access key.