Tenant commands

For a higher level understanding of multi-tenancy architecture, see Multi-Tenancy feature.

All Multi-Tenancy subcommands are located under CLI ozone tenant.

The commands below assume a Kerberized Ozone cluster with Ranger install. Enabling HTTPS on S3 Gateway is optional but recommended.

The exit code of a successful tenant command should be 0. A non-zero exit code indicates failure, which should be accompanied an error message.

Quick Start

Setup

Follow the Multi-Tenancy Setup guide if you haven’t done so.

If the OzoneManagers are running in HA, append --om-service-id= accordingly to the commands.

Create a tenant

Create a new tenant in the current Ozone cluster. This operation requires Ozone cluster administrator privilege.

Apart from adding new OM DB entries, creating a tenant also does the following in the background:

  1. Creates a volume of the exact same name. Therefore, volume name restrictions apply to the tenant name as well. Specifying a custom volume name during tenant creation is not supported yet. Tenant volume cannot be changed once the tenant is created.
  2. Creates two new Ranger roles, tenantName-UserRole and tenantName-AdminRole.
  3. Creates new Ranger policies that allows all tenant users to list and create buckets by default under the tenant volume, but only bucket owners and tenant admins are allowed to access the bucket contents.
  1. ozone tenant [--verbose] create <TENANT_NAME>

Example:

  1. bash-4.2$ kinit -kt /etc/security/keytabs/om.keytab om/om@EXAMPLE.COM
  2. bash-4.2$ ozone tenant create tenantone
  3. 2022-02-16 00:00:00,000 [main] INFO rpc.RpcClient: Creating Tenant: 'tenantone', with new volume: 'tenantone'

Verbose output example:

  1. bash-4.2$ kinit -kt /etc/security/keytabs/om.keytab om/om@EXAMPLE.COM
  2. bash-4.2$ ozone tenant --verbose create tenantone
  3. 2022-02-16 00:00:00,000 [main] INFO rpc.RpcClient: Creating Tenant: 'tenantone', with new volume: 'tenantone'
  4. {
  5.   "tenantId": "tenantone"
  6. }

List tenants

List all tenants in an Ozone cluster. Optionally, use --json to print the detailed result in JSON.

  1. ozone tenant list [--json]

Example:

  1. bash-4.2$ ozone tenant list
  2. tenantone
  1. bash-4.2$ ozone tenant list --json
  2. [
  3.   {
  4.     "tenantId": "tenantone",
  5.     "bucketNamespaceName": "tenantone",
  6.     "userRoleName": "tenantone-UserRole",
  7.     "adminRoleName": "tenantone-AdminRole",
  8.     "bucketNamespacePolicyName": "tenantone-VolumeAccess",
  9.     "bucketPolicyName": "tenantone-BucketAccess"
  10.   }
  11. ]

Assign a user to a tenant

The first user in a tenant must be assigned by an Ozone cluster administrator.

By default when user testuser is assigned to tenant tenantone, the generated Access ID for the user in this tenant is tenantone$testuser.

  • Be sure to enclose the Access ID in single quotes in Bash when using it so it doesn’t get expanded as environment variables.

It is possible to assign a user to multiple tenants.

  1. ozone tenant [--verbose] user assign <USER_NAME> --tenant=<TENANT_NAME>

<USER_NAME> should be a short user name for a Kerberos principal, e.g. testuser when the Kerberos principal is testuser/scm@EXAMPLE.COM

Example:

  1. bash-4.2$ ozone tenant user assign testuser --tenant=tenantone
  2. export AWS_ACCESS_KEY_ID='tenantone$testuser'
  3. export AWS_SECRET_ACCESS_KEY='<GENERATED_SECRET>'
  5. bash-4.2$ ozone tenant user assign testuser --tenant=tenantone
  6. TENANT_USER_ACCESS_ID_ALREADY_EXISTS accessId 'tenantone$testuser' already exists!
  1. bash-4.2$ ozone tenant --verbose user assign testuser2 --tenant=tenantone
  2. export AWS_ACCESS_KEY_ID='tenantone$testuser2'
  3. export AWS_SECRET_ACCESS_KEY='<GENERATED_SECRET>'
  4. Assigned 'testuser2' to 'tenantone' with accessId 'tenantone$testuser2'.

Assign a user as a tenant admin

The first user in a tenant must be assigned by an Ozone cluster administrator.

Both delegated and non-delegated tenant admin can assign and revoke regular tenant users.

The only difference between delegated tenant admin and non-delegated tenant admin is that delegated tenant admin can assign and revoke tenant admins in the tenant, while non-delegated tenant admin can’t.

By default, ozone tenant assignadmin assigns a non-delegated tenant admin. To assign a delegated tenant admin, specify --delegated or -d.

It is possible to assign a user to be tenant admins in multiple tenants. Just a reminder, the user would have a different access ID under each tenant.

  1. ozone tenant user assignadmin <ACCESS_ID> [-d|--delegated] --tenant=<TENANT_NAME>

Example:

  1. bash-4.2$ ozone tenant user assignadmin 'tenantone$testuser' --tenant=tenantone

By default, if the command succeeds, it exits with 0 and prints nothing. Use --verbose to print the result in JSON.

  1. bash-4.2$ ozone tenant --verbose user assignadmin 'tenantone$testuser' --tenant=tenantone
  2. {
  3.   "accessId": "tenantone$testuser",
  4.   "tenantId": "tenantone",
  5.   "isAdmin": true,
  6.   "isDelegatedAdmin": true
  7. }

Once testuser becomes a tenant admin of tenantone, one can kinit as testuser and assign new users to the tenant, even new tenant admins (if delegated). Example commands for illustration:

  1. kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
  2. ozone tenant user assign testuser2 --tenant=tenantone
  3. ozone tenant user assignadmin 'tenantone$testuser2' --tenant=tenantone

List users in a tenant

  1. ozone tenant user list [--json] <TENANT_NAME>

Example:

  1. bash-4.2$ ozone tenant user list tenantone
  2. - User 'testuser' with accessId 'tenantone$testuser'
  3. - User 'testuser2' with accessId 'tenantone$testuser2'
  1. bash-4.2$ ozone tenant user list --json tenantone
  2. [
  3.   {
  4.     "user": "testuser",
  5.     "accessId": "tenantone$testuser"
  6.   },
  7.   {
  8.     "user": "testuser2",
  9.     "accessId": "tenantone$testuser2"
  10.   }
  11. ]

Get tenant user info

This command lists all tenants a user is assigned to.

  1. ozone tenant user info [--json] <USER_NAME>

Example:

  1. bash-4.2$ ozone tenant user info testuser
  2. User 'testuser' is assigned to:
  3. - Tenant 'tenantone' delegated admin with accessId 'tenantone$testuser'
  1. bash-4.2$ ozone tenant user info --json testuser
  2. {
  3.   "user": "testuser",
  4.   "tenants": [
  5.     {
  6.       "accessId": "tenantone$testuser",
  7.       "tenantId": "tenantone",
  8.       "isAdmin": true,
  9.       "isDelegatedAdmin": true
  10.     }
  11.   ]
  12. }

Revoke a tenant admin

  1. ozone tenant [--verbose] user revokeadmin <ACCESS_ID>

Example:

  1. bash-4.2$ ozone tenant user revokeadmin 'tenantone$testuser'
  1. bash-4.2$ ozone tenant --verbose user revokeadmin 'tenantone$testuser'
  2. {
  3.   "accessId": "tenantone$testuser",
  4.   "isAdmin": false,
  5.   "isDelegatedAdmin": false
  6. }

Revoke user access from a tenant

  1. ozone tenant [--verbose] user revoke <ACCESS_ID>

Example:

  1. bash-4.2$ ozone tenant user revoke 'tenantone$testuser'

With verbose output:

  1. bash-4.2$ ozone tenant --verbose user revoke 'tenantone$testuser'
  2. Revoked accessId 'tenantone$testuser'.

Delete a tenant

In order to be able to delete a tenant, the tenant has to be empty. i.e. All users need to be revoked before a tenant can be deleted. Otherwise OM will throw TENANT_NOT_EMPTY exception and refuse to delete the tenant.

Note that it is intentional by design that the volume created and associated with the tenant during tenant creation is not removed. An admin has to remove the volume manually as prompt in the CLI, if deemed necessary.

Verbose option, in addition, will print the Ozone Manager RAW response in JSON.

  1. ozone tenant [--verbose] delete <TENANT_NAME>

Example:

  1. bash-4.2$ ozone tenant delete tenantone
  2. Deleted tenant 'tenantone'.
  3. But the associated volume 'tenantone' is not removed. To delete it, run
  4.     ozone sh volume delete tenantone

With verbose output:

  1. bash-4.2$ ozone tenant --verbose delete tenantone
  2. Deleted tenant 'tenantone'.
  3. But the associated volume 'tenantone' is not removed. To delete it, run
  4.     ozone sh volume delete tenantone
  6. {
  7.   "tenantId": "tenantone",
  8.   "volumeName": "tenantone",
  9.   "volumeRefCount": 0
  10. }

If an Ozone cluster admin (or whoever has the permission to delete the volume in Ranger) tries delete a volume before the tenant is deleted using the command above, the ozone sh volume delete command would fail because the volume reference count is not zero:

  1. bash-4.2$ ozone sh volume delete tenantone
  2. VOLUME_IS_REFERENCED Volume reference count is not zero (1). Ozone features are enabled on this volume. Try `ozone tenant delete <tenantId>` first.

Bucket links can be used to allow access to buckets outside of the tenant volume.

Bucket (sym)links are a special type of bucket that points to other buckets in the same Ozone cluster. It is similar to POSIX symbolic links.

An example to create a bucket link:

  1. $ ozone tenant linkbucket /vol1/bucket1 /tenantone/linked-bucket1

The command above creates a bucket symlink linked-bucket1 in volume tenantone, which points to bucket1 in vol1.

As long as the user running this command has the permission to create a bucket in the target volume tenantone, the command will succeed.

  • The link bucket command itself does not check for permission to access the source volume and bucket.
  • The link bucket command will not even check if the source volume and bucket exists.
  • Permission check will be performed when the bucket symlink is actually accessed.
    • In order to grant a user in tenant tenantone access the bucket, a new policy should be added by a Ranger admin that allow that user intended permissions (READ, WRITE, LIST, CREATE, DELETE, ...) to the source bucket bucket1 in volume vol1.
  • At the moment, ozone tenant linkbucket command is equivalent to ozone sh bucket link command (see Expose any volume section in S3 protocol).

Example: Accessing a bucket in a tenant volume via S3 Gateway using S3 API

Here is an example of accessing the bucket using AWS CLI in the Docker Compose cluster, with tenant tenantone created and testuser assigned to the tenant.

Configure AWS CLI

  1. bash-4.2$ aws configure
  2. AWS Access Key ID [****************fslf]: tenantone$testuser
  3. AWS Secret Access Key [****************fslf]: <GENERATED_SECRET>
  4. Default region name [us-west-1]:
  5. Default output format [None]:

List buckets, create a bucket

  1. bash-4.2$ aws s3api --endpoint-url http://s3g:9878 list-buckets
  2. {
  3.     "Buckets": []
  4. }
  5. bash-4.2$ aws s3api --endpoint-url http://s3g:9878 create-bucket --bucket bucket-test1
  6. {
  7.     "Location": "http://s3g:9878/bucket-test1"
  8. }
  9. bash-4.2$ aws s3api --endpoint-url http://s3g:9878 list-buckets
  10. {
  11.     "Buckets": [
  12.         {
  13.             "Name": "bucket-test1",
  14.             "CreationDate": "2022-02-16T00:05:00.000Z"
  15.         }
  16.     ]
  17. }

In the Docker Compose cluster, the AWS CLI might report AccessDenied because it uses a mocked Ranger endpoint (which can’t be used to perform authorization). A production Ranger setup uses RangerOzoneAuthorizer in OM for authorization while the ozonesecure Docker Compose cluster still uses OzoneNativeAuthorizer. So a workaround is to set the volume owner to testuser to gain full access, as OzoneNativeAuthorizer grants the volume owner full permission:

  1. ozone sh volume update tenantone --user=testuser

The bucket created with aws s3api is also visible under Ozone CLI:

  1. bash-4.2$ ozone sh bucket list /tenantone
  2. [ {
  3.   "metadata" : { },
  4.   "volumeName" : "tenantone",
  5.   "name" : "bucket-test1",
  6.   "storageType" : "DISK",
  7.   "versioning" : false,
  8.   "usedBytes" : 0,
  9.   "usedNamespace" : 0,
  10.   "creationTime" : "2022-02-16T00:05:00.000Z",
  11.   "modificationTime" : "2022-02-16T00:05:00.000Z",
  12.   "quotaInBytes" : -1,
  13.   "quotaInNamespace" : -1,
  14.   "bucketLayout" : "OBJECT_STORE",
  15.   "owner" : "root",
  16.   "link" : false
  17. } ]

Put object (key) to a bucket, list objects

  1. bash-4.2$ aws s3api --endpoint-url http://s3g:9878 put-object --bucket bucket-test1 --key file1 --body README.md
  2. bash-4.2$ aws s3api --endpoint-url http://s3g:9878 list-objects --bucket bucket-test1
  3. {
  4.     "Contents": [
  5.         {
  6.             "Key": "file1",
  7.             "LastModified": "2022-02-16T00:10:00.000Z",
  8.             "ETag": "2022-02-16T00:10:00.000Z",
  9.             "Size": 3811,
  10.             "StorageClass": "STANDARD"
  11.         }
  12.     ]
  13. }

Get object (key) from a bucket

  1. bash-4.2$ aws s3api --endpoint-url http://s3g:9878 get-object --bucket bucket-test1 --key file1 file1-get.txt
  2. {
  3.     "AcceptRanges": "bytes",
  4.     "LastModified": "Wed, 16 Feb 2022 00:10:00 GMT",
  5.     "ContentLength": 3811,
  6.     "CacheControl": "no-cache",
  7.     "ContentType": "application/octet-stream",
  8.     "Expires": "Wed, 16 Feb 2022 00:15:00 GMT",
  9.     "Metadata": {}
  10. }
  11. bash-4.2$ diff file1-get.txt README.md