WEB

Slack Docker Pulls GitHub edit source

This guide describes the instructions to configure WEB as Alluxio’s under storage system.

Initial Setup

The Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.

Configuring Alluxio

Alluxio unifies access to different storage systems through the unified namespace feature. An WEB location can be either mounted at the root of the Alluxio namespace or at a nested directory.

ROOT MOUNT

Configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

  1. $ cp conf/alluxio-site.properties.template conf/alluxio-site.properties

the following environment variable assignment needs to be added to conf/alluxio-site.properties, and we actually support both http:// and https:// protocols.

  1. alluxio.master.hostname=localhost
  2. # alluxio.master.mount.table.root.ufs=[https|http]://<HOSTNAME>:<PORT>/DIRECTORY/
  3. # A sample for this article
  4. alluxio.master.mount.table.root.ufs=https://downloads.alluxio.io/downloads/files/

Specify the settings for parsing a WEB page(Optional):

  1. alluxio.underfs.web.connnection.timeout=<WEB_CONNECTION_TIMEOUT>
  2. alluxio.underfs.web.header.last.modified=<WEB_HEADER_LAST_MODIFIED>
  3. alluxio.underfs.web.parent.names=<WEB_PARENT_NAMES>
  4. alluxio.underfs.web.titles=<WEB_TITLES>

Here, alluxio.underfs.web.connnection.timeout is the timeout setting for a http connection(unit: second, default: 60s). alluxio.underfs.web.header.last.modified represents the format to parse the last modified field for a directory or a file from a http response header (default: “EEE, dd MMM yyyy HH:mm:ss zzz”). alluxio.underfs.web.parent.names indicates the start row index of the files list, which can be set as multiple flags seperated by a comma (default: “Parent Directory,..,../”). alluxio.underfs.web.titles means the “web page title” flags to check it is a directory or not which can be also set with multiple values seperated by a comma (default: “Index of ,Directory listing for “).

NESTED MOUNT

An WEB location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Mount Command can be used for this purpose. For example, the following command mounts a directory inside an WEB directory into Alluxio directory /web:

  1. $ ./bin/alluxio fs mount \
  2.   --option alluxio.underfs.web.connnection.timeout=<WEB_CONNECTION_TIMEOUT> \
  3.   --option alluxio.underfs.web.header.last.modified=<WEB_HEADER_LAST_MODIFIED> \
  4.   --option alluxio.underfs.web.parent.names=<WEB_PARENT_NAMES> \
  5.   --option alluxio.underfs.web.titles=<WEB_TITLES> \
  6.   /web [https|http]://<HOSTNAME>:<PORT>/DIRECTORY/

Running Alluxio with WEB

Run the following command to start Alluxio filesystem.

  1. $ ./bin/alluxio format
  2. $ ./bin/alluxio-start.sh local

To verify that Alluxio is running, you can visit http://localhost:19999, or see the log in the logs folder.

Visit your WEB volume by running the following command:

  1. $ ./bin/alluxio fs ls /

After waiting for a while, you can see the following result:

  1. dr--r-----                                              0       PERSISTED 05-21-2019 12:53:22:000  DIR /1.4.0
  2. dr--r-----                                              0       PERSISTED 05-21-2019 12:54:23:000  DIR /1.5.0
  3. dr--r-----                                              0       PERSISTED 05-21-2019 12:55:06:000  DIR /1.6.0
  4. dr--r-----                                              0       PERSISTED 05-21-2019 12:55:38:000  DIR /1.6.1
  5. dr--r-----                                              0       PERSISTED 05-21-2019 12:57:00:000  DIR /1.7.0
  6. dr--r-----                                              0       PERSISTED 05-21-2019 12:57:57:000  DIR /1.7.1
  7. dr--r-----                                              0       PERSISTED 05-21-2019 13:00:25:000  DIR /1.8.0
  8. dr--r-----                                              0       PERSISTED 05-21-2019 13:02:07:000  DIR /1.8.1
  9. dr--r-----                                              0       PERSISTED 05-24-2019 05:16:31:000  DIR /2.0.0
  10. dr--r-----                                              0       PERSISTED 05-21-2019 13:02:11:000  DIR /2.0.0-preview

Stop Alluxio by running:

  1. $ ./bin/alluxio-stop.sh local