Log4J Appender

Appends Log4j events to a flume agent’s avro source. A client using thisappender must have the flume-ng-sdk in the classpath (eg,flume-ng-sdk-1.9.0.jar).Required properties are in bold.

Property NameDefaultDescription
Hostname–The hostname on which a remote Flume agent is running with anavro source.
Port–The port at which the remote Flume agent’s avro source islistening.
UnsafeModefalseIf true, the appender will not throw exceptions on failure tosend the events.
AvroReflectionEnabledfalseUse Avro Reflection to serialize Log4j events. (Do not use when users log strings)
AvroSchemaUrl–A URL from which the Avro schema can be retrieved.

Sample log4j.properties file:

  1. #...
  2. log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
  3. log4j.appender.flume.Hostname = example.com
  4. log4j.appender.flume.Port = 41414
  5. log4j.appender.flume.UnsafeMode = true
  6.  
  7. # configure a class's logger to output to the flume appender
  8. log4j.logger.org.example.MyClass = DEBUG,flume
  9. #...

By default each event is converted to a string by calling toString(),or by using the Log4j layout, if specified.

If the event is an instance oforg.apache.avro.generic.GenericRecord, org.apache.avro.specific.SpecificRecord,or if the property AvroReflectionEnabled is set to true then the event will beserialized using Avro serialization.

Serializing every event with its Avro schema is inefficient, so it is good practice toprovide a schema URL from which the schema can be retrieved by the downstream sink,typically the HDFS sink. If AvroSchemaUrl is not specified,then the schema will be included as a Flume header.

Sample log4j.properties file configured to use Avro serialization:

  1. #...
  2. log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
  3. log4j.appender.flume.Hostname = example.com
  4. log4j.appender.flume.Port = 41414
  5. log4j.appender.flume.AvroReflectionEnabled = true
  6. log4j.appender.flume.AvroSchemaUrl = hdfs://namenode/path/to/schema.avsc
  7.  
  8. # configure a class's logger to output to the flume appender
  9. log4j.logger.org.example.MyClass = DEBUG,flume
  10. #...