Object Storage

Object storage exposes an S3 API to the storage cluster for applications to put and get data.

Prerequisites

This guide assumes a Rook cluster as explained in the Quickstart.

Configure an Object Store

Rook has the ability to either deploy an object store in Kubernetes or to connect to an external RGW service. Most commonly, the object store will be configured locally by Rook. Alternatively, if you have an existing Ceph cluster with Rados Gateways, see the external section to consume it from Rook.

Create a Local Object Store

The below sample will create a CephObjectStore that starts the RGW service in the cluster with an S3 API.

NOTE: This sample requires at least 3 bluestore OSDs, with each OSD located on a different node.

The OSDs must be located on different nodes, because the failureDomain is set to host and the erasureCoded chunk settings require at least 3 different OSDs (2 dataChunks + 1 codingChunks).

See the Object Store CRD, for more detail on the settings available for a CephObjectStore.

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephObjectStore
  3. metadata:
  4.   name: my-store
  5.   namespace: rook-ceph
  6. spec:
  7.   metadataPool:
  8.     failureDomain: host
  9.     replicated:
  10.       size: 3
  11.   dataPool:
  12.     failureDomain: host
  13.     erasureCoded:
  14.       dataChunks: 2
  15.       codingChunks: 1
  16.   preservePoolsOnDelete: true
  17.   gateway:
  18.     type: s3
  19.     sslCertificateRef:
  20.     port: 80
  21.     securePort:
  22.     instances: 1
  23.   healthCheck:
  24.     bucket:
  25.       enabled: true
  26.       interval: 60s

After the CephObjectStore is created, the Rook operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.

  1. # Create the object store
  2. kubectl create -f object.yaml
  4. # To confirm the object store is configured, wait for the rgw pod to start
  5. kubectl -n rook-ceph get pod -l app=rook-ceph-rgw

Connect to an External Object Store

Rook can connect to existing RGW gateways to work in conjunction with the external mode of the CephCluster CRD. If you have an external CephCluster CR, you can instruct Rook to consume external gateways with the following:

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephObjectStore
  3. metadata:
  4.   name: external-store
  5.   namespace: rook-ceph
  6. spec:
  7.   gateway:
  8.     port: 8080
  9.     externalRgwEndpoints:
  10.       - ip: 192.168.39.182
  11.   healthCheck:
  12.     bucket:
  13.       enabled: true
  14.       interval: 60s

You can use the existing object-external.yaml file. When ready the ceph-object-controller will output a message in the Operator log similar to this one:

  1. ceph-object-controller: ceph object store gateway service running at 10.100.28.138:8080

You can now get and access the store via:

  1. kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
  2. NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
  3. rook-ceph-rgw-my-store   ClusterIP   10.100.28.138   <none>        8080/TCP   6h59m

Any pod from your cluster can now access this endpoint:

  1. curl 10.100.28.138:8080
  2. <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

It is also possible to use the internally registered DNS name:

  1. curl rook-ceph-rgw-my-store.rook-ceph:8080
  2. <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

The DNS name is created with the following schema rook-ceph-rgw-$STORE_NAME.$NAMESPACE.

Create a Bucket

Now that the object store is configured, next we need to create a bucket where a client can read and write objects. A bucket can be created by defining a storage class, similar to the pattern used by block and file storage. First, define the storage class that will allow object clients to create a bucket. The storage class defines the object storage system, the bucket retention policy, and other properties required by the administrator. Save the following as storageclass-bucket-delete.yaml (the example is named as such due to the Delete reclaim policy).

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.    name: rook-ceph-bucket
  5. provisioner: rook-ceph.ceph.rook.io/bucket
  6. reclaimPolicy: Delete
  7. parameters:
  8.   objectStoreName: my-store
  9.   objectStoreNamespace: rook-ceph
  10.   region: us-east-1
  1. kubectl create -f storageclass-bucket-delete.yaml

Based on this storage class, an object client can now request a bucket by creating an Object Bucket Claim (OBC). When the OBC is created, the Rook-Ceph bucket provisioner will create a new bucket. Notice that the OBC references the storage class that was created above. Save the following as object-bucket-claim-delete.yaml (the example is named as such due to the Delete reclaim policy):

  1. apiVersion: objectbucket.io/v1alpha1
  2. kind: ObjectBucketClaim
  3. metadata:
  4.   name: ceph-bucket
  5. spec:
  6.   generateBucketName: ceph-bkt
  7.   storageClassName: rook-ceph-bucket
  1. kubectl create -f object-bucket-claim-delete.yaml

Now that the claim is created, the operator will create the bucket as well as generate other artifacts to enable access to the bucket. A secret and ConfigMap are created with the same name as the OBC and in the same namespace. The secret contains credentials used by the application pod to access the bucket. The ConfigMap contains bucket endpoint information and is also consumed by the pod. See the Object Bucket Claim Documentation for more details on the CephObjectBucketClaims.

Client Connections

The following commands extract key pieces of information from the secret and configmap:”

  1. #config-map, secret, OBC will part of default if no specific name space mentioned
  2. export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o yaml | grep BUCKET_HOST | awk '{print $2}')
  3. export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o yaml | grep AWS_ACCESS_KEY_ID | awk '{print $2}' | base64 --decode)
  4. export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o yaml | grep AWS_SECRET_ACCESS_KEY | awk '{print $2}' | base64 --decode)

Consume the Object Storage

Now that you have the object store configured and a bucket created, you can consume the object storage from an S3 client.

This section will guide you through testing the connection to the CephObjectStore and uploading and downloading from it. Run the following commands after you have connected to the Rook toolbox.

Connection Environment Variables

To simplify the s3 client commands, you will want to set the four environment variables for use by your client (ie. inside the toolbox). See above for retrieving the variables for a bucket created by an ObjectBucketClaim.

  1. export AWS_HOST=<host>
  2. export AWS_ENDPOINT=<endpoint>
  3. export AWS_ACCESS_KEY_ID=<accessKey>
  4. export AWS_SECRET_ACCESS_KEY=<secretKey>
  • Host: The DNS host name where the rgw service is found in the cluster. Assuming you are using the default rook-ceph cluster, it will be rook-ceph-rgw-my-store.rook-ceph.
  • Endpoint: The endpoint where the rgw service is listening. Run kubectl -n rook-ceph get svc rook-ceph-rgw-my-store, then combine the clusterIP and the port.
  • Access key: The user’s access_key as printed above
  • Secret key: The user’s secret_key as printed above

The variables for the user generated in this example might be:

  1. export AWS_HOST=rook-ceph-rgw-my-store.rook-ceph
  2. export AWS_ENDPOINT=10.104.35.31:80
  3. export AWS_ACCESS_KEY_ID=XEZDB3UJ6X7HVBE7X7MA
  4. export AWS_SECRET_ACCESS_KEY=7yGIZON7EhFORz0I40BFniML36D2rl8CQQ5kXU6l

The access key and secret key can be retrieved as described in the section above on client connections or below in the section creating a user if you are not creating the buckets with an ObjectBucketClaim.

Install s3cmd

To test the CephObjectStore we will install the s3cmd tool into the toolbox pod.

  1. yum --assumeyes install s3cmd

PUT or GET an object

Upload a file to the newly created bucket

  1. echo "Hello Rook" > /tmp/rookObj
  2. s3cmd put /tmp/rookObj --no-ssl --host=${AWS_HOST} --host-bucket=  s3://rookbucket

Download and verify the file from the bucket

  1. s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_HOST} --host-bucket=
  2. cat /tmp/rookObj-download

Access External to the Cluster

Rook sets up the object storage so pods will have access internal to the cluster. If your applications are running outside the cluster, you will need to setup an external service through a NodePort.

First, note the service that exposes RGW internal to the cluster. We will leave this service intact and create a new service for external access.

  1. $ kubectl -n rook-ceph get service rook-ceph-rgw-my-store
  2. NAME                     CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
  3. rook-ceph-rgw-my-store   10.3.0.177   <none>        80/TCP      2m

Save the external service as rgw-external.yaml:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: rook-ceph-rgw-my-store-external
  5.   namespace: rook-ceph
  6.   labels:
  7.     app: rook-ceph-rgw
  8.     rook_cluster: rook-ceph
  9.     rook_object_store: my-store
  10. spec:
  11.   ports:
  12.   - name: rgw
  13.     port: 80
  14.     protocol: TCP
  15.     targetPort: 80
  16.   selector:
  17.     app: rook-ceph-rgw
  18.     rook_cluster: rook-ceph
  19.     rook_object_store: my-store
  20.   sessionAffinity: None
  21.   type: NodePort

Now create the external service.

  1. kubectl create -f rgw-external.yaml

See both rgw services running and notice what port the external service is running on:

  1. $ kubectl -n rook-ceph get service rook-ceph-rgw-my-store rook-ceph-rgw-my-store-external
  2. NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
  3. rook-ceph-rgw-my-store            ClusterIP   10.104.82.228    <none>        80/TCP         4m
  4. rook-ceph-rgw-my-store-external   NodePort    10.111.113.237   <none>        80:31536/TCP   39s

Internally the rgw service is running on port 80. The external port in this case is 31536. Now you can access the CephObjectStore from anywhere! All you need is the hostname for any machine in the cluster, the external port, and the user credentials.

Create a User

If you need to create an independent set of user credentials to access the S3 endpoint, create a CephObjectStoreUser. The user will be used to connect to the RGW service in the cluster using the S3 API. The user will be independent of any object bucket claims that you might have created in the earlier instructions in this document.

See the Object Store User CRD for more detail on the settings available for a CephObjectStoreUser.

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephObjectStoreUser
  3. metadata:
  4.   name: my-user
  5.   namespace: rook-ceph
  6. spec:
  7.   store: my-store
  8.   displayName: "my display name"

When the CephObjectStoreUser is created, the Rook operator will then create the RGW user on the specified CephObjectStore and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the CephObjectStoreUser.

  1. # Create the object store user
  2. kubectl create -f object-user.yaml
  4. # To confirm the object store user is configured, describe the secret
  5. kubectl -n rook-ceph describe secret rook-ceph-object-user-my-store-my-user
  7. Name:        rook-ceph-object-user-my-store-my-user
  8. Namespace:    rook-ceph
  9. Labels:            app=rook-ceph-rgw
  10.                   rook_cluster=rook-ceph
  11.                   rook_object_store=my-store
  12. Annotations:    <none>
  14. Type:    kubernetes.io/rook
  16. Data
  17. ====
  18. AccessKey:    20 bytes
  19. SecretKey:    40 bytes

The AccessKey and SecretKey data fields can be mounted in a pod as an environment variable. More information on consuming kubernetes secrets can be found in the K8s secret documentation

To directly retrieve the secrets:

  1. kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep AccessKey | awk '{print $2}' | base64 --decode
  2. kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml | grep SecretKey | awk '{print $2}' | base64 --decode

Object Multisite

Multisite is a feature of Ceph that allows object stores to replicate its data over multiple Ceph clusters.

Multisite also allows object stores to be independent and isloated from other object stores in a cluster.

For more information on multisite please read the ceph multisite overview for how to run it.