Configuring a multi agent flow

To setup a multi-tier flow, you need to have an avro/thrift sink of first hoppointing to avro/thrift source of the next hop. This will result in the firstFlume agent forwarding events to the next Flume agent. For example, if you areperiodically sending files (1 file per event) using avro client to a localFlume agent, then this local agent can forward it to another agent that has themounted for storage.

Weblog agent config:

  1. # list sources, sinks and channels in the agent
  2. agent_foo.sources = avro-AppSrv-source
  3. agent_foo.sinks = avro-forward-sink
  4. agent_foo.channels = file-channel
  5.  
  6. # define the flow
  7. agent_foo.sources.avro-AppSrv-source.channels = file-channel
  8. agent_foo.sinks.avro-forward-sink.channel = file-channel
  9.  
  10. # avro sink properties
  11. agent_foo.sinks.avro-forward-sink.type = avro
  12. agent_foo.sinks.avro-forward-sink.hostname = 10.1.1.100
  13. agent_foo.sinks.avro-forward-sink.port = 10000
  14.  
  15. # configure other pieces
  16. #...

HDFS agent config:

  1. # list sources, sinks and channels in the agent
  2. agent_foo.sources = avro-collection-source
  3. agent_foo.sinks = hdfs-sink
  4. agent_foo.channels = mem-channel
  5.  
  6. # define the flow
  7. agent_foo.sources.avro-collection-source.channels = mem-channel
  8. agent_foo.sinks.hdfs-sink.channel = mem-channel
  9.  
  10. # avro source properties
  11. agent_foo.sources.avro-collection-source.type = avro
  12. agent_foo.sources.avro-collection-source.bind = 10.1.1.100
  13. agent_foo.sources.avro-collection-source.port = 10000
  14.  
  15. # configure other pieces
  16. #...

Here we link the avro-forward-sink from the weblog agent to theavro-collection-source of the hdfs agent. This will result in the events comingfrom the external appserver source eventually getting stored in HDFS.