Tools

File Channel Integrity Tool

File Channel Integrity tool verifies the integrity of individual Events in the File channeland removes corrupted Events.

The tools can be run as follows:

$bin/flume-ng tool --conf ./conf FCINTEGRITYTOOL -l ./datadir

where datadir is the comma separated list of data directory to be verified.

Following are the options available

Event Validator Tool

Event validator tool can be used to validate the File Channel Event’s in application specific way.The tool applies the user provider validation login on each event and drop the event which do notconfirm to the logic.

The tools can be run as follows:

$bin/flume-ng tool --conf ./conf FCINTEGRITYTOOL -l ./datadir -e org.apache.flume.MyEventValidator -DmaxSize 2000

where datadir is the comma separated list of data directory to be verified.

Following are the options available

The Event validator implementation must implement EventValidator interface. It’s recommendednot to throw any exception from the implementation as they are treated as invalid events.Additional parameters can be passed to EventValitor implementation via -D options.

Let’s see an example of simple size based Event Validator, which shall reject event’s largerthan maximum size specified.

public static class MyEventValidator implements EventValidator {
 
  private int value = 0;
 
  private MyEventValidator(int val) {
    value = val;
  }
 
  @Override
  public boolean validateEvent(Event event) {
    return event.getBody() <= value;
  }
 
  public static class Builder implements EventValidator.Builder {
 
    private int sizeValidator = 0;
 
    @Override
    public EventValidator build() {
      return new DummyEventVerifier(sizeValidator);
    }
 
    @Override
    public void configure(Context context) {
      binaryValidator = context.getInteger("maxSize");
    }
  }
}