Running Alluxio on EMR

Slack Docker Pulls GitHub edit source

This guide describes how to configure Alluxio to run on AWS EMR.

Overview

AWS EMR provides great options for running clusters on-demand to handle compute workloads. It manages the deployment of various Hadoop Services and allows for hooks into these services for customizations. Alluxio can run on EMR to provide functionality above what EMRFS currently provides. Aside from the added performance benefits of caching, Alluxio also enables users to run compute workloads against on-premise storage or even a different cloud provider’s storage i.e. GCS, Azure Blob Store.

Prerequisites

  • Account with AWS
  • IAM Account with the default EMR Roles
  • Key Pair for EC2
  • An S3 Bucket
  • AWS CLI: Make sure that the AWS CLI is set up and ready with the required AWS Access/Secret key

The majority of the pre-requisites can be found by going through the AWS EMR Getting Started guide. An S3 bucket is needed as Alluxio’s root Under File System and to serve as the location for the bootstrap script. If required, the root UFS can be reconfigured to be HDFS.

Basic Setup

Set up the required IAM roles for the account to be able to use the EMR service

Open a terminal and use the AWS CLI to create the necessary IAM roles on your account.

  1. $ aws emr create-default-roles

Run the aws create-cluster command with the bootstrap action

The create-cluster command requires passing in multiple flags to successfully execute:

  • release-label: The version of EMR to install with. The current version of Alluxio is compatible with emr-5.25.0.
  • instance-count: The number of nodes to provision for the cluster.
  • instance-type: The instance type to provision with. Note that your account is limited in the number of instances you can launch in each region; check your instance limits here. A good instance type to start off with is r4.4xlarge.
  • applications: Specify Name=Spark Name=Presto Name=Hive to bootstrap the three additional services
  • name: The EMR cluster name
  • bootstrap-actions:
    • Path: The path to the bootstrap script, hosted in a publicly readable S3 bucket: s3://alluxio-public/emr/2.2.2/alluxio-emr.sh
    • Args: The arguments passed to the bootstrap script.
      • The first argument, the root UFS URI, is required. This S3 URI designates the root mount of the Alluxio file system and should be of the form s3://bucket-name/mount-point. The mount point should be a folder; follow these instructions to create a folder in S3.
      • Specify the path to a publicly accessible Alluxio tarball with the -d flag. For example, you can use the URL: https://downloads.alluxio.io/downloads/files/2.2.2/alluxio-2.2.2-bin.tar.gz
      • You can also specify additional Alluxio properties as a delimited list of key-value pairs in the format key=value. For example, alluxio.user.file.writetype.default=CACHE_THROUGH instructs Alluxio to write files synchronously to the underlying storage system. See more about write type options.
  • configurations: The path to the configuration json file, also hosted in a publicly readable S3 bucket: s3://alluxio-public/emr/2.2.2/alluxio-emr.json
  • ec2-attributes: EC2 settings to provide, most notably the name of the key pair to use to connect to the cluster

Below is a sample command with all of the above flags populated:

Note that this command’s formatting is designed for the bash interpreter.

  1. $ aws emr create-cluster \
  2. --release-label emr-5.25.0 \
  3. --instance-count 3 \
  4. --instance-type r4.4xlarge \
  5. --applications Name=Spark Name=Presto Name=Hive \
  6. --name try-alluxio \
  7. --bootstrap-actions \
  8. Path=s3://alluxio-public/emr/2.2.2/alluxio-emr.sh,\
  9. Args=[s3://myBucketName/mountPointFolder,\
  10. -d,"https://downloads.alluxio.io/downloads/files/2.2.2/alluxio-2.2.2-bin.tar.gz",\
  11. -p,"alluxio.user.block.size.bytes.default=122M|alluxio.user.file.writetype.default=CACHE_THROUGH",\
  12. -s,"|"] \
  13. --configurations https://alluxio-public.s3.amazonaws.com/emr/2.2.2/alluxio-emr.json \
  14. --ec2-attributes KeyName=myKeyPairName

where s3://myBucketName/mountPointFolder should be replaced with a S3 URI that your AWS account can read and write to and myKeyPairName should be replaced with the name of your EC2 key pair.

SSH into the EMR cluster’s master node

Log into the EMR console.

Once the cluster is in the Waiting stage, click on the cluster details to get the Master public DNS if available or click on the Hardware tab to see the master and worker details.

emr_console_waiting

Clicking on the master instance group will show you the public DNS.

emr_console_public_dns

SSH into the master instance using the key pair provided in the previous command. Use hadoop as the username.

  1. $ ssh -i /path/to/keypair.pem hadoop@<masterPublicDns>

If a security group isn’t specified in the create-cluster command, the default EMR security group created for you will not allow inbound SSH. In order to continue, you will need to edit the security group and open port 22. See more details here.

Test that Alluxio is running

Once inside the master instance, run the following command to run a series of basic tests to ensure Alluxio can read and write files.

  1. $ sudo runuser -l alluxio -c "/opt/alluxio/bin/alluxio runTests"

Cluster details

Using this boostrap script, Alluxio is installed in /opt/alluxio/ by default. Hive and Presto are already configured to connect to Alluxio. The cluster also uses AWS Glue as the default metastore for both Presto and Hive. This will allow you to maintain table definitions between multiple runs of the Alluxio cluster. By default, the Alluxio worker is allocated one third of the instance’s maximum available memory.

Creating a Table

The simplest step to using EMR with Alluxio is to create a table on Alluxio and query it using Presto/Hive.

SSH into the master node

From your terminal, SSH into the master instance using the key pair provided in the create-cluster command.

  1. $ ssh -i /path/to/keypair.pem hadoop@<masterPublicDns>

Note that we are connecting as the hadoop user. All subsequent commands assume they are being executed from within the instance.

Create a directory in Alluxio to be the external location of your table

Create the /testTable directory in Alluxio, then set the hadoop user to be the directory owner. Note that these commands are being executed as the alluxio user.

  1. $ sudo runuser -l alluxio -c "/opt/alluxio/bin/alluxio fs mkdir /testTable"
  2. $ sudo runuser -l alluxio -c "/opt/alluxio/bin/alluxio fs chown hadoop:hadoop /testTable"

Create a new database in AWS GLUE

Open the Hive CLI.

  1. $ hive

Create a database, then check in the Glue console to see if the database is created.

  1. CREATE DATABASE glue;

Use the newly created database and define a table.

  1. USE glue;
  2. create external table test1 (userid INT,
  3. age INT,
  4. gender CHAR(1),
  5. occupation STRING,
  6. zipcode STRING)
  7. ROW FORMAT DELIMITED
  8. FIELDS TERMINATED BY '|'
  9. LOCATION 'alluxio:///testTable';

Exit the Hive CLI.

  1. $ exit

Create the Presto /tmp directory

Similar to before, create a /tmp directory in Alluxio. Then set the directory permissions to 777.

  1. $ sudo runuser -l alluxio -c "/opt/alluxio/bin/alluxio fs mkdir /tmp"
  2. $ sudo runuser -l alluxio -c "/opt/alluxio/bin/alluxio fs chmod 777 /tmp"

Interact with the table using Presto

Open the Presto CLI, specifying hive as the catalog.

  1. $ presto-cli --catalog hive

Use the created database and insert some values into the table.

  1. USE glue;
  2. INSERT INTO test1 VALUES (1, 24, 'F', 'Developer', '12345');

Read back all values in the table with a SELECT statement.

  1. SELECT * FROM test1;

Customization

Tuning of Alluxio properties can be done in a few different locations. Depending on which service needs tuning, EMR offers different ways of modifying the service settings/environment variables.

Bootstrap Script Usage

The following describes all the possible flags that can be passed into the bootstrap script. In the above example, we used the -p and -s flags to specify additional Alluxio properties and the delimiting string between properties.

  1. Usage: alluxio-emr.sh <root-ufs-uri>
  2.                              [-b <backup_uri>]
  3.                              [-d <alluxio-download-uri>]
  4.                              [-f <file_uri>]
  5.                              [-i <journal_backup_uri>]
  6.                              [-n <storage percentage>]
  7.                              [-p <delimited_properties>]
  8.                              [-s <property_delimiter>]
  10. alluxio-emr.sh is a script which can be used to bootstrap an AWS EMR cluster
  11. with Alluxio. It can download and install Alluxio as well as add properties
  12. specified as arguments to the script.
  14. By default, if the environment this script executes in does not already contain
  15. an Alluxio install at ${ALLUXIO_HOME} then it will download, untar, and configure
  16. the environment at ${ALLUXIO_HOME}. If an install already exists at ${ALLUXIO_HOME},
  17. nothing will be installed over it, even if -d is specified.
  19. If a specific Alluxio tarball is desired, see the -d option.
  21.   <root-ufs-uri>    (Required) The URI of the root UFS in the Alluxio
  22.                     namespace.
  24.   -b                An s3:// URI that the Alluxio master will write a backup
  25.                     to upon shutdown of the EMR cluster. The backup and and
  26.                     upload MUST be run within 60 seconds. If the backup cannot
  27.                     finish within 60 seconds, then an incomplete journal may
  28.                     be uploaded. This option is not recommended for production
  29.                     or mission critical use cases where the backup is relied
  30.                     upon to restore cluster state after a previous shutdown.
  33.   -d                An s3:// or http(s):// URI which points to an Alluxio
  34.                     tarball. This script will download and untar the
  35.                     Alluxio tarball and install Alluxio at ${ALLUXIO_HOME} if an
  36.                     Alluxio installation doesn't already exist at that location.
  39.   -f                An s3:// or http(s):// URI to any remote file. This property
  40.                     can be specified multiple times. Any file specified through
  41.                     this property will be downloaded and stored with the same
  42.                     name to ${ALLUXIO_HOME}/conf/
  45.   -i                An s3:// or http(s):// URI which represents the URI of a
  46.                     previous Alluxio journal backup. If supplied, the backup
  47.                     will be downloaded, and upon Alluxio startup, the Alluxio
  48.                     master will read and restore the backup.
  50.   -n                Automatically configure NVMe storage for Alluxio workers at
  51.                     tier 0 instead of MEM. When present, the script will attempt
  52.                     to locate mounted NVMe storage locations and configure them
  53.                     to be used with Alluxio. The argument provided is an
  54.                     integer between 1 and 100 that represents the percentage of
  55.                     each disk that will be allocated to Alluxio.                    
  57.   -p                A string containing a delimited set of properties which
  58.                     should be added to the
  59.                     ${ALLUXIO_HOME}/conf/alluxio-site.properties file. The
  60.                     delimiter by default is a semicolon ";". If a different
  61.                     delimiter is desired use the [-s] argument.
  64.   -s                A string containing a single character representing what
  65.                     delimiter should be used to split the Alluxio properties
  66.                     provided in the [-p] argument.

Alluxio service

Making configuration changes to the Alluxio service can be done in a few different ways via the bootstrap script. The -p flag allows users to pass in a set of delimited key-value properties to be set on all of the Alluxio nodes. An alternative would be to pass in a custom file using the -f flag named alluxio-site.properties. The bootstrap will make sure to overwrite any user-provided configs while retaining any defaults that are not overwritten. The bootstrap also allows users to install previous versions of Alluxio (>=2.0) by specifying a download URL (HTTP or S3 only).

Alluxio client

Generic client-side properties can also be edited via the bootstrap script as mentioned above. This is mostly for the native client (CLI). Property changes for a specific service like Presto/Hive should be done in the respective section of the EMR JSON configuration file i.e. core-site.xml or hive.catalog.