Qiniu Kodo

This guide describes how to configure Alluxio with Qiniu Kodo as the under storage system. Qiniu Object Storage Service (Kodo) is a massive, secure and highly reliable cloud storage service.

Initial Setup

To run an Alluxio cluster on a set of machines, you must deploy Alluxio binaries to each of these machines.You can compile the binaries from Alluxio source code, or download the precompiled binaries directly.

A Qiniu Kodo bucket is necessary before using Kodo with Alluxio. In this guide, the Qiniu Kodo bucket is called KODO_BUCKET, and the directory in the bucket is called KODO_DIRECTORY. In addition, you should provide a domain to identify the specified bucket, which is called KODO_DOWNLOAD_HOST. Through the KODO_DOWNLOAD_HOST you can get objects from the bucket.

Mounting Kodo

Alluxio unifies access to different storage systems through the unified namespace feature. The root of Alluxio namespace or its subdirectories are all available for the mount point of Kodo.

Root Mount

If you want to use Qiniu Kodo as its under storage system in Alluxio, conf/alluxio-site.properties must be modified. In the beginning, an existing Kodo bucket and its directory should be specified for storage by the following code:

alluxio.master.mount.table.root.ufs=kodo://<KODO_BUCKET>/<KODO_DIRECTORY>/

Next, some settings must be added to conf/alluxio-site.properties:

fs.kodo.accesskey=<KODO_ACCESS_KEY>
fs.kodo.secretkey=<KODO_SECRET_KET>
alluxio.underfs.kodo.downloadhost=<KODO_DOWNLOAD_HOST>
alluxio.underfs.kodo.endpoint=<KODO_ENDPOINT>

AccessKey/SecretKey can be found in Qiniu Console - Key Management

alluxio.underfs.kodo.downloadhost can be found in Qiniu Console - Kodo

alluxio.underfs.kodo.endpoint is the endpoint of this bucket, which can be found in the bucket in this table:

Nested Mount

An Kodo location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s mount command can be used for this purpose. For example, the following command mounts a directory inside an Kodo bucket into Alluxio directory

$ ./bin/alluxio fs mount --option fs.kodo.accessKey=<KODO_ACCESS_KEY> \
  --option fs.kodo.secretkey=<KODO_SECRET_KET> \
  --option alluxio.underfs.kodo.downloadhost=<KODO_DOWNLOAD_HOST> \
  --option alluxio.underfs.kodo.endpoint=<KODO_ENDPOINT> \
  kodo/ kodo://<KODO_BUCKET>/<KODO_DIRECTORY>/

Running Alluxio Locally with Kodo

After everything is configured, you can start up Alluxio locally to see that everything works.

$ ./bin/alluxio format
$ ./bin/alluxio-start.sh local

This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.

Next, you can run a simple example program:

$ ./bin/alluxio runTests

After this succeeds, you can visit your Kodo directory kodo://<KODO_BUCKET>/<KODO_DIRECTORY> to verify the files and directories mounted by Alluxio exist. For this test, you should see files named like KODO_BUCKET/KODO_DIRECTORY/default_tests_files/BasicFile_CACHE_PROMOTE_MUST_CACHE.

To stop Alluxio, you can run:

$ ./bin/alluxio-stop.sh local

Contributed by the Alluxio Community

Qiniu KODO UFS integration is contributed and maintained by the Alluxio community. The source code is located here. Feel free submit pull requests to improve the integration and update the documentation here if any information is missing or out of date.