S3 Filesystem

In this page, we explain how to get your Hudi spark job to store into AWS S3.

AWS configs

There are two configurations required for Hudi-S3 compatibility:

  • Adding AWS Credentials for Hudi
  • Adding required Jars to classpath

AWS Credentials

Simplest way to use Hudi with S3, is to configure your SparkSession or SparkContext with S3 credentials. Hudi will automatically pick this up and talk to S3.

Alternatively, add the required configs in your core-site.xml from where Hudi can fetch them. Replace the fs.defaultFS with your S3 bucket name and Hudi should be able to read/write from the bucket.

  1.   <property>
  2.       <name>fs.defaultFS</name>
  3.       <value>s3://ysharma</value>
  4.   </property>
  6.   <property>
  7.       <name>fs.s3.impl</name>
  8.       <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
  9.   </property>
  11.   <property>
  12.       <name>fs.s3.awsAccessKeyId</name>
  13.       <value>AWS_KEY</value>
  14.   </property>
  16.   <property>
  17.        <name>fs.s3.awsSecretAccessKey</name>
  18.        <value>AWS_SECRET</value>
  19.   </property>
  21.   <property>
  22.        <name>fs.s3n.awsAccessKeyId</name>
  23.        <value>AWS_KEY</value>
  24.   </property>
  26.   <property>
  27.        <name>fs.s3n.awsSecretAccessKey</name>
  28.        <value>AWS_SECRET</value>
  29.   </property>

Utilities such as hudi-cli or deltastreamer tool, can pick up s3 creds via environmental variable prefixed with HOODIE_ENV_. For e.g below is a bash snippet to setup such variables and then have cli be able to work on datasets stored in s3

  1. export HOODIE_ENV_fs_DOT_s3a_DOT_access_DOT_key=$accessKey
  2. export HOODIE_ENV_fs_DOT_s3a_DOT_secret_DOT_key=$secretKey
  3. export HOODIE_ENV_fs_DOT_s3_DOT_awsAccessKeyId=$accessKey
  4. export HOODIE_ENV_fs_DOT_s3_DOT_awsSecretAccessKey=$secretKey
  5. export HOODIE_ENV_fs_DOT_s3n_DOT_awsAccessKeyId=$accessKey
  6. export HOODIE_ENV_fs_DOT_s3n_DOT_awsSecretAccessKey=$secretKey
  7. export HOODIE_ENV_fs_DOT_s3n_DOT_impl=org.apache.hadoop.fs.s3a.S3AFileSystem

AWS Libs

AWS hadoop libraries to add to our classpath

  • com.amazonaws:aws-java-sdk:1.10.34
  • org.apache.hadoop:hadoop-aws:2.7.3

AWS glue data libraries are needed if AWS glue data is used

  • com.amazonaws.glue:aws-glue-datacatalog-hive2-client:1.11.0
  • com.amazonaws:aws-java-sdk-glue:1.11.475