HDFS is optimized for large files. The scalability of the single HDFS namenode is limited by the number of files. It is hard for HDFS to store lots of small files.

SeaweedFS excels on small files and has no issue to store large files. Now it is possible to enable Hadoop jobs to read from and write to SeaweedFS.

Build SeaweedFS Hadoop Client Jar

  1. $cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/client
  2. $ mvn install
  4. # build for hadoop2
  5. $cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/hdfs2
  6. $ mvn package
  7. $ ls -al target/seaweedfs-hadoop2-client-*.jar
  9. # build for hadoop3
  10. $cd $GOPATH/src/github.com/chrislusf/seaweedfs/other/java/hdfs3
  11. $ mvn package
  12. $ ls -al target/seaweedfs-hadoop3-client-*.jar

Or you can download the latest version from MavenCentral

Test SeaweedFS on Hadoop

Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run.

You would need to start a weed filer first, build the seaweedfs-hadoop2-client-xxx.jaror seaweedfs-hadoop3-client-xxx.jar, and do the following:

  1. $ cd ${HADOOP_HOME}
  2. # create etc/hadoop/mapred-site.xml, just to satisfy hdfs dfs. skip this if the file already exists.
  3. $ echo "<configuration></configuration>" > etc/hadoop/mapred-site.xml
  5. # on hadoop2
  6. $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
  7.                -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
  8.                -libjars ./seaweedfs-hadoop2-client-x.x.x.jar \
  9.                -ls /
  10. # or on hadoop3
  11. $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
  12.                -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
  13.                -libjars ./seaweedfs-hadoop3-client-x.x.x.jar \
  14.                -ls /

Both reads and writes are working fine.

Installation for Hadoop

  • Configure Hadoop to use SeaweedFS in etc/hadoop/conf/core-site.xml. core-site.xml resides on each node in the Hadoop cluster. You must add the same properties to each instance of core-site.xml. There are several properties to modify:
    • fs.seaweedfs.impl: This property defines the Seaweed HCFS implementation classes that are contained in the SeaweedFS HDFS client JAR. It is required.
    • fs.defaultFS: This property defines the default file system URI to use. It is optional if a path always has prefix seaweedfs://localhost:8888.
  1. <configuration>
  2.     <property>
  3.         <name>fs.seaweedfs.impl</name>
  4.         <value>seaweed.hdfs.SeaweedFileSystem</value>
  5.     </property>
  6.     <property>
  7.         <name>fs.defaultFS</name>
  8.         <value>seaweedfs://localhost:8888</value>
  9.     </property>
  10. </configuration>
  • Deploy the SeaweedFS HDFS client jar
  1. # Run the classpath command to get the list of directories in the classpath
  2. $ bin/hadoop classpath
  4. # Copy SeaweedFS HDFS client jar to one of the folders
  5. $ cd ${HADOOP_HOME}
  6. # for hadoop2
  7. $ cp ./seaweedfs-hadoop2-client-x.x.x.jar share/hadoop/common/lib/
  8. # or for hadoop3
  9. $ cp ./seaweedfs-hadoop3-client-x.x.x.jar share/hadoop/common/lib/

Now you can do this:

  1. $ cd ${HADOOP_HOME}
  2. $ bin/hdfs dfs -ls /
  4. # if you did not set fs.defaultFS in etc/hadoop/core-site.xml
  5. # or you want to access a different SeaweedFS filer
  6. $ bin/hdfs dfs -ls seaweedfs://localhost:8888/

Supported HDFS Operations

  1.     bin/hdfs dfs -appendToFile README.txt /weedfs/weedfs.txt
  2.     bin/hdfs dfs -cat /weedfs/weedfs.txt
  3.     bin/hdfs dfs -rm -r /uber
  4.     bin/hdfs dfs -chown -R chris:chris /weedfs
  5.     bin/hdfs dfs -chmod -R 755 /weedfs
  6.     bin/hdfs dfs -copyFromLocal README.txt /weedfs/README.txt.2
  7.     bin/hdfs dfs -copyToLocal /weedfs/README.txt.2 .
  8.     bin/hdfs dfs -count /weedfs/README.txt.2
  9.     bin/hdfs dfs -cp /weedfs/README.txt.2 /weedfs/README.txt.3
  10.     bin/hdfs dfs -du -h /weedfs
  11.     bin/hdfs dfs -find /weedfs -name "*.txt" -print
  12.     bin/hdfs dfs -get /weedfs/weedfs.txt
  13.     bin/hdfs dfs -getfacl /weedfs
  14.     bin/hdfs dfs -getmerge -nl /weedfs w.txt
  15.     bin/hdfs dfs -ls /
  16.     bin/hdfs dfs -mkdir /tmp
  17.     bin/hdfs dfs -mkdir -p /tmp/x/y
  18.     bin/hdfs dfs -moveFromLocal README.txt.2 /tmp/x/
  19.     bin/hdfs dfs -mv /tmp/x/y/README.txt.2 /tmp/x/y/README.txt.3
  20.     bin/hdfs dfs -mv /tmp/x /tmp/z
  21.     bin/hdfs dfs -put README.txt /tmp/z/y/
  22.     bin/hdfs dfs -rm /tmp/z/y/*
  23.     bin/hdfs dfs -rmdir /tmp/z/y
  24.     bin/hdfs dfs -stat /weedfs
  25.     bin/hdfs dfs -tail /weedfs/weedfs.txt
  26.     bin/hdfs dfs -test -f /weedfs/weedfs.txt
  27.     bin/hdfs dfs -text /weedfs/weedfs.txt
  28.     bin/hdfs dfs -touchz /weedfs/weedfs.txtx

Operations Plan to Support

  1.   getfattr
  2.   setfacl
  3.   setfattr
  4.   truncate
  5.   createSnapshot
  6.   deleteSnapshot
  7.   renameSnapshot
  8.   setrep

Notes

Atomicity

SeaweedFS satisfies the HCFS requirements that the following operations to be atomic, when using MySql/Postgres database transactions.

  • Creating a file. If the overwrite parameter is false, the check and creation MUST be atomic.
  • Deleting a file.
  • Renaming a file.
  • Renaming a directory.

  • Creating a single directory with mkdir().Among these, except file or directory renaming, the following operations are all atomic for any filer store.

  • Creating a file

  • Deleting a file
  • Creating a single directory with mkdir().

No native shared libraries

The SeaweedFS hadoop client is a pure java library. There are no native libraries to install if you already have Hadoop running.

This is different from many other HCFS options. If native shared libraries are needed, these libraries need to be install on all hadoop nodes. This is quite some task.

Shaded Fat Jar

One of the headache with complicated Java systems is the jar runtime dependency problem, which is resolved by Go's build time dependency resolution. For this SeaweedFS hadoop client, the required jars are mostly shaded and packaged as one fat jar, so there are no extra jar files needed.

Note