Load Balancing Log4J Appender

Appends Log4j events to a list of flume agent’s avro source. A client using thisappender must have the flume-ng-sdk in the classpath (eg,flume-ng-sdk-1.9.0.jar). This appender supports a round-robin and randomscheme for performing the load balancing. It also supports a configurable backofftimeout so that down agents are removed temporarily from the set of hostsRequired properties are in bold.

Property NameDefaultDescription
Hosts–A space-separated list of host:port at which Flume (throughan AvroSource) is listening for events
SelectorROUND_ROBINSelection mechanism. Must be either ROUND_ROBIN,RANDOM or custom FQDN to class that inherits fromLoadBalancingSelector.
MaxBackoff–A long value representing the maximum amount of time inmilliseconds the Load balancing client will backoff from anode that has failed to consume an event. Defaults to no backoff
UnsafeModefalseIf true, the appender will not throw exceptions on failure tosend the events.
AvroReflectionEnabledfalseUse Avro Reflection to serialize Log4j events.
AvroSchemaUrl–A URL from which the Avro schema can be retrieved.

Sample log4j.properties file configured using defaults:

  1. #...
  2. log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
  3. log4j.appender.out2.Hosts = localhost:25430 localhost:25431
  4.  
  5. # configure a class's logger to output to the flume appender
  6. log4j.logger.org.example.MyClass = DEBUG,flume
  7. #...

Sample log4j.properties file configured using RANDOM load balancing:

  1. #...
  2. log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
  3. log4j.appender.out2.Hosts = localhost:25430 localhost:25431
  4. log4j.appender.out2.Selector = RANDOM
  5.  
  6. # configure a class's logger to output to the flume appender
  7. log4j.logger.org.example.MyClass = DEBUG,flume
  8. #...

Sample log4j.properties file configured using backoff:

  1. #...
  2. log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
  3. log4j.appender.out2.Hosts = localhost:25430 localhost:25431 localhost:25432
  4. log4j.appender.out2.Selector = ROUND_ROBIN
  5. log4j.appender.out2.MaxBackoff = 30000
  6.  
  7. # configure a class's logger to output to the flume appender
  8. log4j.logger.org.example.MyClass = DEBUG,flume
  9. #...