HadoopClient

A utility to wrap Hadoop console command

class bigflow.util.hadoop_client.HadoopClient(client_path, config_path)

A wrapper class of Hadoop console command

内部类，请用户不要使用该类，未来不保证接口兼容，请直接使用hadoop-client。

参数:
  • client_path (str) — path of Hadoop client(usually ‘hadoop’ executable file)
  • config_path (str) — path of Hadoop configuration

  • fs_dus(path, args=None)

    Wraps console command ‘hadoop fs -dus ‘

    参数:path (str) — path to get size
    返回:path size
    返回类型:long

  • fs_get(source, target, args=None)

    Wraps console command ‘hadoop fs -get ‘

    参数:
    • source (str) — path of source
    • target (str) — path of target

  • fs_mkdir(path, args=None)

    Wraps console command ‘hadoop fs -mkdir -p ‘

    参数:path (str) — path to be created

  • fs_mv(source, target, args=None)

    Wraps console command ‘hadoop fs -mv ‘

    参数:
    • source (str) — path of source
    • target (str) — path of target

  • fs_put(source, target, args=None, need_mkdir=True)

    Wraps console command ‘hadoop fs -put ‘

    参数:
    • source (str) — path of source
    • target (str) — path of target

  • fs_rmr(path, args=None)

    Wraps console command ‘hadoop fs -rmr ‘

    参数:path (str) — path to be removed

  • fs_test(path, args=None)

    Wraps console command ‘hadoop fs -test -e ‘

    参数:path (str) — path to test
    返回:if path exist
    返回类型:bool

bigflow.util.hadoop_client.extract_fs_name_from_path(path)

Get fs.defaultFS from path like “hdfs://abcde:22222/a/b/c”