Adding multiple flows in an agent

A single Flume agent can contain several independent flows. You can listmultiple sources, sinks and channels in a config. These components can belinked to form multiple flows:

  1. # list the sources, sinks and channels for the agent
  2. <Agent>.sources = <Source1> <Source2>
  3. <Agent>.sinks = <Sink1> <Sink2>
  4. <Agent>.channels = <Channel1> <Channel2>

Then you can link the sources and sinks to their corresponding channels (forsources) of channel (for sinks) to setup two different flows. For example, ifyou need to setup two flows in an agent, one going from an external avro clientto external HDFS and another from output of a tail to avro sink, then here’s aconfig to do that:

  1. # list the sources, sinks and channels in the agent
  2. agent_foo.sources = avro-AppSrv-source1 exec-tail-source2
  3. agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
  4. agent_foo.channels = mem-channel-1 file-channel-2
  5.  
  6. # flow #1 configuration
  7. agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1
  8. agent_foo.sinks.hdfs-Cluster1-sink1.channel = mem-channel-1
  9.  
  10. # flow #2 configuration
  11. agent_foo.sources.exec-tail-source2.channels = file-channel-2
  12. agent_foo.sinks.avro-forward-sink2.channel = file-channel-2