Command Line Interface

Ozone shell is the primary interface to interact with Ozone from the command line. Behind the scenes it uses the Java API.

There are some functionality which couldn’t be accessed without using ozone sh commands. For example:

  1. Creating volumes with quota
  2. Managing internal ACLs
  3. Creating buckets with encryption key

All of these are one-time, administration tasks. Applications can use Ozone without this CLI using other interface like Hadoop Compatible File System (o3fs or ofs) or S3 interface.

Ozone shell help can be invoked at object level or at action level.

For example:

  1. ozone sh volume --help

will show all possible actions for volumes.

Or it can be invoked to explain a specific action like:

  1. ozone sh volume create --help

which will print the command line options of the create command for volumes.

General Command Format

Ozone shell commands take the following form:

ozone sh object action url

ozone script is used to invoke all Ozone sub-commands. The ozone shell is invoked via sh command.

Object can be volume, bucket or key. Actions are various verbs like create, list, delete etc.

Depending on the action, Ozone URL can point to a volume, bucket or key in the following format:

[schema][server:port]/volume/bucket/key

Where,

  1. Schema - This should be o3 which is the native RPC protocol to access Ozone API. The usage of the schema is optional.

  2. Server:Port - This is the address of the Ozone Manager. If the port is omitted the default port from ozone-site.xml will be used.

Please see volume commands, bucket commands, and key commands section for more detail.

Volume operations

Volume is the top level element of the hierarchy, managed only by administrators. Optionally, quota and the owner user can be specified.

Example commands:

  1. $ ozone sh volume create /vol1
  1. $ ozone sh volume info /vol1
  2. {
  3.   "metadata" : { },
  4.   "name" : "vol1",
  5.   "admin" : "hadoop",
  6.   "owner" : "hadoop",
  7.   "creationTime" : "2020-07-28T12:31:50.112Z",
  8.   "modificationTime" : "2020-07-28T12:31:50.112Z",
  9.   "acls" : [ {
  10.     "type" : "USER",
  11.     "name" : "hadoop",
  12.     "aclScope" : "ACCESS",
  13.     "aclList" : [ "ALL" ]
  14.   }, {
  15.     "type" : "GROUP",
  16.     "name" : "users",
  17.     "aclScope" : "ACCESS",
  18.     "aclList" : [ "ALL" ]
  19.   } ],
  20.   "quota" : 1152921504606846976
  21. }
  1. $ ozone sh volume list /
  2. [ {
  3.   "metadata" : { },
  4.   "name" : "s3v",
  5.   "admin" : "hadoop",
  6.   "owner" : "hadoop",
  7.   "creationTime" : "2020-07-27T11:32:22.314Z",
  8.   "modificationTime" : "2020-07-27T11:32:22.314Z",
  9.   "acls" : [ {
  10.     "type" : "USER",
  11.     "name" : "hadoop",
  12.     "aclScope" : "ACCESS",
  13.     "aclList" : [ "ALL" ]
  14.   }, {
  15.     "type" : "GROUP",
  16.     "name" : "users",
  17.     "aclScope" : "ACCESS",
  18.     "aclList" : [ "ALL" ]
  19.   } ],
  20.   "quota" : 1152921504606846976
  21. }, {
  22.   ....
  23. } ]

If the volume is empty, we can delete the volume using the command below.

  1. $ ozone sh volume delete /vol1
  2. Volume vol1 is deleted

If the volume contains any buckets or keys, we can delete the volume recursively. This will delete all keys and buckets within the volume, and then delete the volume itself. After running this command there is no way to recover deleted contents.

  1. $ ozone sh volume delete -r /vol1
  2. This command will delete volume recursively.
  3. There is no recovery option after using this command, and no trash for FSO buckets.
  4. Delay is expected running this command.
  5. Enter 'yes' to proceed': yes
  6. Volume vol1 is deleted

Bucket operations

Bucket is the second level of the object hierarchy, and is similar to AWS S3 buckets. Users can create buckets in volumes, if they have the necessary permissions.

Command examples:

  1. $ ozone sh bucket create /vol1/bucket1
  1. $ ozone sh bucket info /vol1/bucket1
  2. {
  3.   "metadata" : { },
  4.   "volumeName" : "vol1",
  5.   "name" : "bucket1",
  6.   "storageType" : "DISK",
  7.   "versioning" : false,
  8.   "creationTime" : "2020-07-28T13:14:45.091Z",
  9.   "modificationTime" : "2020-07-28T13:14:45.091Z",
  10.   "encryptionKeyName" : null,
  11.   "sourceVolume" : null,
  12.   "sourceBucket" : null
  13. }

If the bucket is empty we can delete the bucket using the command below.

  1. $ ozone sh bucket delete /vol1/bucket1
  2. Bucket bucket1 is deleted

If the bucket contains any keys, we can delete the bucket recursively. This will delete all the keys within the bucket, and then the bucket itself. After running this command there is no way to recover deleted contents.

  1. $ ozone sh bucket delete -r /vol1/bucket1
  2. This command will delete bucket recursively.
  3. There is no recovery option after using this command, and deleted keys won't move to trash.
  4. Enter 'yes' to proceed': yes
  5. Bucket bucket1 is deleted

Transparent Data Encryption can be enabled at the bucket level.

Key operations

Key is the object which can store the data.

  1. $ ozone sh key put /vol1/bucket1/README.md README.md

In this case the standard ozone sh <object_type> <action> <url> scheme may be a bit confusing at first, as it results in the syntax ozone sh key put <destination> <source> instead of the arguably more natural order of <source> <destination>.

  1. $ ozone sh key info /vol1/bucket1/README.md
  2. {
  3.   "volumeName" : "vol1",
  4.   "bucketName" : "bucket1",
  5.   "name" : "README.md",
  6.   "dataSize" : 3841,
  7.   "creationTime" : "2020-07-28T13:17:20.749Z",
  8.   "modificationTime" : "2020-07-28T13:17:21.979Z",
  9.   "replicationType" : "RATIS",
  10.   "replicationFactor" : 1,
  11.   "ozoneKeyLocations" : [ {
  12.     "containerID" : 1,
  13.     "localID" : 104591670688743424,
  14.     "length" : 3841,
  15.     "offset" : 0
  16.   } ],
  17.   "metadata" : { },
  18.   "fileEncryptionInfo" : null
  19. }
  1. $ ozone sh key get /vol1/bucket1/README.md /tmp/
  1. $ ozone sh key delete /vol1/bucket1/key1

If the key is in an FSO bucket it will be moved to the trash when deleted. Below is the trash location:

  1. $ /<volume>/<bucket>/.Trash/<user>

If the key is in an OBS bucket it will be permanently deleted.

Querying CLI Results

Ozone CLI returns JSON responses. jq is a command line JSON processor that can be used to filter CLI output for desired information.

For example:

  • List FSO buckets that are not links.
  1. $ ozone sh bucket list /s3v | jq '.[] | select(.link==false and .bucketLayout=="FILE_SYSTEM_OPTIMIZED")'
  2. {
  3.   "metadata": {},
  4.   "volumeName": "s3v",
  5.   "name": "fso-bucket",
  6.   "storageType": "DISK",
  7.   "versioning": false,
  8.   "usedBytes": 0,
  9.   "usedNamespace": 0,
  10.   "creationTime": "2023-02-01T05:18:46.974Z",
  11.   "modificationTime": "2023-02-01T05:18:46.974Z",
  12.   "quotaInBytes": -1,
  13.   "quotaInNamespace": -1,
  14.   "bucketLayout": "FILE_SYSTEM_OPTIMIZED",
  15.   "owner": "om",
  16.   "link": false
  17. }
  • List EC buckets with their replication config.
  1. $ ozone sh bucket list /vol1 | jq -r '.[] | select(.replicationConfig.replicationType == "EC") | {"name": .name, "replicationConfig": .replicationConfig}'
  2. {
  3.   "name": "ec5",
  4.   "replicationConfig": {
  5.     "data": 3,
  6.     "parity": 2,
  7.     "ecChunkSize": 1048576,
  8.     "codec": "RS",
  9.     "replicationType": "EC",
  10.     "requiredNodes": 5
  11.   }
  12. }
  13. {
  14.   "name": "ec9",
  15.   "replicationConfig": {
  16.     "data": 6,
  17.     "parity": 3,
  18.     "ecChunkSize": 1048576,
  19.     "codec": "RS",
  20.     "replicationType": "EC",
  21.     "requiredNodes": 9
  22.   }
  23. }
  • List names of encrypted buckets and their encryption key names in tab-separated-value format.
  2. $ ozone sh bucket list /vol1 | jq -r '.[] | select(.encryptionKeyName != null) | [.name, .encryptionKeyName] | @tsv'
  3. ec5     key1
  4. encrypted-bucket        key1