HDFS

This plugin based on WebHDFS api of Hadoop, it just POST data to WebHDFS port. So, it’s a native Ruby code.

  1. output {
  2.     hadoop_webhdfs {
  3.         workers => 2
  4.         server => "your.nameno.de:14000"
  5.         user => "flume"
  6.         path => "/user/flume/logstash/dt=%{+Y}-%{+M}-%{+d}/logstash-%{+H}.log"
  7.         flush_size => 500
  8.         compress => "snappy"
  9.         idle_flush_time => 10
  10.         retry_interval => 0.5
  11.     }
  12. }

This plugin based on HDFS api of Hadoop, it import java classes like org.apache.hadoop.fs.FileSystem etc.

Configuration

  1. output {
  2.     hdfs {
  3.         path => "/path/to/output_file.log"
  4.         enable_append => true
  5.     }
  6. }

Howto run

  1. CLASSPATH=$(find /path/to/hadoop -name '*.jar' | tr '\n' ':'):/etc/hadoop/conf:/path/to/logstash-1.1.7-monolithic.jar java logstash.runner agent -f conf/hdfs-output.conf -p /path/to/cloned/logstash-hdfs