IBM Cloud Object Storage Filesystem

In this page, we explain how to get your Hudi spark job to store into IBM Cloud Object Storage.

IBM COS configs

There are two configurations required for Hudi-IBM Cloud Object Storage compatibility:

  • Adding IBM COS Credentials for Hudi
  • Adding required Jars to classpath

IBM Cloud Object Storage Credentials

Simplest way to use Hudi with IBM Cloud Object Storage, is to configure your SparkSession or SparkContext with IBM Cloud Object Storage credentials using Stocator storage connector for Spark. Hudi will automatically pick this up and talk to IBM Cloud Object Storage.

Alternatively, add the required configs in your core-site.xml from where Hudi can fetch them. Replace the fs.defaultFS with your IBM Cloud Object Storage bucket name and Hudi should be able to read/write from the bucket.

For example, using HMAC keys and service name myCOS:

  1.   <property>
  2.       <name>fs.defaultFS</name>
  3.       <value>cos://myBucket.myCOS</value>
  4.   </property>
  6.   <property>
  7.       <name>fs.cos.flat.list</name>
  8.       <value>true</value>
  9.   </property>
  11.   <property>
  12.       <name>fs.stocator.scheme.list</name>
  13.       <value>cos</value>
  14.   </property>
  16.   <property>
  17.       <name>fs.cos.impl</name>
  18.       <value>com.ibm.stocator.fs.ObjectStoreFileSystem</value>
  19.   </property>
  21.   <property>
  22.       <name>fs.stocator.cos.impl</name>
  23.       <value>com.ibm.stocator.fs.cos.COSAPIClient</value>
  24.   </property>
  26.   <property>
  27.       <name>fs.stocator.cos.scheme</name>
  28.       <value>cos</value>
  29.   </property>
  31.   <property>
  32.       <name>fs.cos.myCos.access.key</name>
  33.       <value>ACCESS KEY</value>
  34.   </property>
  36.   <property>
  37.       <name>fs.cos.myCos.endpoint</name>
  38.       <value>http://s3-api.us-geo.objectstorage.softlayer.net</value>
  39.   </property>
  41.   <property>
  42.       <name>fs.cos.myCos.secret.key</name>
  43.       <value>SECRET KEY</value>
  44.   </property>

For more options see Stocator documentation.

IBM Cloud Object Storage Libs

IBM Cloud Object Storage hadoop libraries to add to our classpath

  • com.ibm.stocator:stocator:1.1.3