FUSE-based POSIX API

The Alluxio POSIX API is a feature that allows mounting an Alluxio File System as a standard file system on most flavors of Unix. By using this feature, standard tools (for example, ls, cat or mkdir) will have basic access to the Alluxio namespace. More importantly, with the POSIX API integration applications can interact with the Alluxio no matter what language (C, C++, Python, Ruby, Perl, or Java) they are written in without any Alluxio library integrations.

Note that Alluxio-FUSE is different from projects like s3fs, mountableHdfs which mount specific storage services like S3 or HDFS to the local filesystem. The Alluxio POSIX API is a generic solution for the many storage systems supported by Alluxio. Data orchestration and caching features from Alluxio speed up I/O access to frequently used data.

The Alluxio POSIX API is based on the Filesystem in Userspace (FUSE) project. Most basic file system operations are supported. However, given the intrinsic characteristics of Alluxio, like its write-once/read-many-times file data model, the mounted file system does not have full POSIX semantics and contains some limitations. Please read the section of limitations for details.

Choose POSIX API Implementation

The Alluxio POSIX API has two implementations for users to choose from:

• Alluxio JNR-Fuse Alluxio’s first generation Fuse implementation that uses JNR-Fuse for FUSE on Java. JNR-Fuse targets for low concurrency scenarios and has some known limitations in performance.
• Alluxio JNI-Fuse Alluxio’s default in-house implementation based on JNI (Java Native Interface) which targets more performance-sensitve applications (like model training workloads) and initiated by researchers from Nanjing University and engineers from Alibaba Inc.

Here is a guideline To choose between the default JNR-Fuse and experimental JNI-Fuse:

• Workloads: If your data access pattern is highly concurrent (e.g., deep learning training), JNI-Fuse is better and more stable.
• Maintenance: JNI-Fuse is experimental but under active development (checkout our developer meeting notes). Alluxio community will focus more on developing JNI-Fuse and deprecate Alluxio JNR-Fuse eventually.

Requirements

The followings are the basic requirements running Alluxio FUSE integration to support POSIX API in a standalone way. Installing Alluxio using Docker and Kubernetes can further simplify the setup.

• Install JDK 1.8 or newer
• Install libfuse
  • On Linux, install libfuse 2.9.3 or newer (2.8.3 has been reported to also work - with some warnings). For example on a Redhat, run yum install fuse fuse-devel
  • On MacOS, install osxfuse 3.7.1 or newer. For example, run brew install osxfuse
• JNI-Fuse is enabled by default for better performance. If JNR-Fuse is needed, set alluxio.fuse.jnifuse.enabled to false in ${ALLUXIO_HOME}/conf/alluxio-site.properties:

alluxio.fuse.jnifuse.enabled=false

Basic Setup

Mount Alluxio as a FUSE mount point

After properly configuring and starting an Alluxio cluster; Run the following command on the node where you want to create the mount point:

$ ${ALLUXIO_HOME}/integration/fuse/bin/alluxio-fuse mount \
  <mount_point> [<alluxio_path>]

This will spawn a background user-space java process (alluxio-fuse) that will mount the Alluxio path specified at <alluxio_path> to the local file system on the specified <mount_point>.

For example, running the following commands from the ${ALLUXIO_HOME} directory will mount the Alluxio path /people to the folder /mnt/people on the local file system.

$ ./bin/alluxio fs mkdir /people
$ sudo mkdir -p /mnt/people
$ sudo chown $(whoami) /mnt/people
$ chmod 755 /mnt/people
$ integration/fuse/bin/alluxio-fuse mount /mnt/people /people

When <alluxio_path> is not given, the value defaults to the root (/). Note that the <mount_point> must be an existing and empty path in your local file system hierarchy and that the user that runs the integration/fuse/bin/alluxio-fuse script must own the mount point and have read and write permissions on it. Multiple Alluxio FUSE mount points can be created in the same node. All the AlluxioFuse processes share the same log output at $ALLUXIO_HOME\logs\fuse.log, which is useful for troubleshooting when errors happen on operations under the filesystem.

Unmount Alluxio from FUSE

To unmount a previously mounted Alluxio-FUSE file system, on the node where the file system is mounted run:

$ ${ALLUXIO_HOME}/integration/fuse/bin/alluxio-fuse unmount mount_point

This unmounts the file system at the mount point and stops the corresponding Alluxio-FUSE process. For example,

$ ${ALLUXIO_HOME}/integration/fuse/bin/alluxio-fuse unmount /mnt/people
Unmount fuse at /mnt/people (PID:97626).

Check the Alluxio POSIX API mounting status

To list the mount points; on the node where the file system is mounted run:

$ ${ALLUXIO_HOME}/integration/fuse/bin/alluxio-fuse stat

This outputs the pid, mount_point, alluxio_path of all the running Alluxio-FUSE processes.

For example, the output could be:

pid mount_point alluxio_path
80846 /mnt/people /people
80847 /mnt/sales  /sales

Advanced Setup

Configure Alluxio fuse options

These are the configuration parameters for Alluxio POSIX API.

Configure mount point options

You can use -o [mount options] to set mount options. If you want to set multiple mount options, you can pass in comma separated mount options as the value of -o. The -o [mount options] must follow the mount command.

Different versions of libfuse and osxfuse may support different mount options. The available Linux mount options are listed here. The mount options of MacOS with osxfuse are listed here . Some mount options (e.g. allow_other and allow_root) need additional set-up and the set up process may be different depending on the platform.

$ ${ALLUXIO_HOME}integration/fuse/bin/alluxio-fuse mount \
  -o [comma separated mount options] mount_point [alluxio_path]

Tuning mount options

A special mount option is the max_idle_threads=N which defines the maximum number of idle fuse daemon threads allowed. If the value is too small, FUSE may frequently create and destroy threads which will introduce extra performance overhead. Note that, libfuse introduce this mount option in 3.2 while JNI-Fuse supports 2.9.X during experimental stage. The Alluxio Fuse docker image alluxio/alluxio-fuse enables this property by modifying the libfuse source code.

If you are using alluxio fuse docker image, set the MAX_IDLE_THREADS via environment variable:

$ docker run --rm \
    ...
    --env MAX_IDLE_THREADS=64 \
    alluxio/alluxio-fuse fuse

Example: `allow_other` and `allow_root`

By default, Alluxio-FUSE mount point can only be accessed by the user mounting the Alluxio namespace to the local filesystem.

For Linux, add the following line to file /etc/fuse.conf to allow other users or allow root to access the mounted folder:

user_allow_other

This option allow non-root users to specify the allow_other or allow_root mount options.

For MacOS, follow the osxfuse allow_other instructions to allow other users to use the allow_other and allow_root mount options.

After setting up, pass the allow_other or allow_root mount options when mounting Alluxio-FUSE:

# All users (including root) can access the files.
$ integration/fuse/bin/alluxio-fuse mount -o allow_other mount_point [alluxio_path]
# The user mounting the filesystem and root can access the files.
$ integration/fuse/bin/alluxio-fuse mount -o allow_root mount_point [alluxio_path]

Note that only one of the allow_other or allow_root could be set.

Assumptions and Limitations

Currently, most basic file system operations are supported. However, due to Alluxio implicit characteristics, please be aware that:

• Files can be written only once, only sequentially, and never be modified. That means overriding a file is not allowed, and an explicit combination of delete and then create is needed. For example, the cp command will fail when the destination file exists. vi and vim commands will succeed because the underlying system do create, delete, and rename operation combinations.
• Alluxio does not have hard-links or soft-links, so commands like ln are not supported. The hardlinks number is not displayed in ll output.
• The user and group are mapped to the Unix user and group only when Alluxio POSIX API is configured to use shell user group translation service, by setting alluxio.fuse.user.group.translation.enabled to true. Otherwise chown and chgrp are no-ops, and ll will return the user and group of the user who started the Alluxio-FUSE process. The translation service does not change the actual file permission when running ll.

Performance Optimization

Due to the conjunct use of FUSE, the performance of the mounted file system is expected to lower compared to using the Alluxio Java client directly.

Most of the overheads come from the fact that there are several memory copies going on for each call for read or write operations. FUSE caps the maximum granularity of writes to 128KB. This could be probably improved by a large extent by leveraging the FUSE cache write-backs feature introduced in the 3.15 Linux Kernel (supported by libfuse 3.x but not yet supported in jnr-fuse/jni-fuse).

The following client options are useful when running deep learning workloads against Alluxio JNI-Fuse based on our experience. If you find other options useful, please share with us via Alluxio community slack channel or pull request. Note that these changes should be done before the mounting steps.

Enable Metadata Caching

Alluxio Fuse process can cache file metadata locally to reduce the overhead of repeatedly requesting metadata of the same file from Alluxio Master. Enable when the workload repeatedly getting information of numerous files.

For example, a workload that repeatedly gets information of 1 million files and runs for 50 minutes can set the following configuration:

alluxio.user.metadata.cache.enabled=true
alluxio.user.metadata.cache.max.size=1000000
alluxio.user.metadata.cache.expiration.time=1h

The metadata size of 1 million files usually is around 25MB to 100MB. Enable metadata cache may also introduce some overhead, but may not be as big as client data cache.

Other Performance or Debugging Tips

The following client options may affect the training performance or provides more training information.

Increase Direct Memory Size

When encountering the out of direct memory issue, add the following JVM opts to ${ALLUXIO_HOME}/conf/alluxio-env.sh to increase the max amount of direct memory.

ALLUXIO_FUSE_JAVA_OPTS+=" -XX:MaxDirectMemorySize=8G"