Manager Custom Plugin

Overview

This article is aimed at InLong-Manager plugin developers, trying to explain the process of developing an Manager plugin as comprehensively as possible, and strive to eliminate the confusion of developers and make plugin development easier.

Before Development

  • Inlong is stream processing framework constructed with a Group + Stream architecture.
  • An Inlong Group contains more than one Inlong Stream, each Inlong Stream is capable of a single individual dataflow.
  • Inlong Group is responsible for physical resource definition shared by all Inlong Streams included, especially middleware clusters and sort functions.
  • In order to create Inlong Group, Inlong Manager use CreateGroupWorkflowDefinition to init all necessary physical resources, a workflow definition contains several individual service tasks. When it’s created and processed, service tasks will be executed one after another.
  • Service task is constructed in observer pattern, which also known as the publish-subscribe pattern, each service task will register several task listeners. Listener accepts workflow context and controls execution logic on physical resources.
  • As a developer, you need to develop specific Listener with personalized logic.

Demonstration

  • The Inlong Manager plugin mechanism can be represented by the following figure:

Manager Custom Plugin - 图1

  • When Inlong Manager is deployed, plugins must be located under installation directory, then Manager Process will find the plugin jar and install the plugin automatically.

Manager Custom Plugin - 图2

  • As a developer, you can confirm your plugin be loaded successfully by searching logs below:

Manager Custom Plugin - 图3

Reference Demo

  • For helping all Inlong developers. We hava provide manager-plugins in Inlong Manager Module, which provide FlinkSortProcessPlugin as an example; or you can create ProcessPlugin as below;
  1. public class EmptyProcessPlugin implements ProcessPlugin {
  3.     @Override
  4.     public Map<DataSourceOperateListener, EventSelector> createSourceOperateListeners() {
  5.         return new LinkedHashMap<>();
  6.     }
  8.     @Override
  9.     public Map<QueueOperateListener, EventSelector> createQueueOperateListeners() {
  10.         return new LinkedHashMap<>();
  11.     }
  13.     @Override
  14.     public Map<SortOperateListener, EventSelector> createSortOperateListeners() {
  15.         return ProcessPlugin.super.createSortOperateListeners();
  16.     }
  18.     @Override
  19.     public Map<SinkOperateListener, EventSelector> createSinkOperateListeners() {
  20.         return ProcessPlugin.super.createSinkOperateListeners();
  21.     }
  23. }
  • DataSourceOperateListener,QueueOperateListener,SortOperateListener,SinkOperateListener are child_classes extended from TaskEventListener. Then EventSelector decides whether the listener should be triggered.
  1. public interface EventSelector {
  3.     boolean accept(WorkflowContext context);
  5. }
  • After developing you plugin, you should prepare plugin definition file in Yaml, and put it under resources/META-INF.
  1. name: example
  2. description: example for manager plugin
  3. javaVersion: 1.8
  4. pluginClass: org.apache.inlong.manager.plugin.EmptyProcessPlugin
  • To develop available Listeners , you can refer to the native Listeners in org.apache.inlong.manager.service.workflow.listener.GroupTaskListenerFactory

Last but not Least

We provide the plugin mechanism in Inlong Manager make it easier and more convenient for developers to customize personalized logic when Inlong is not supported. Plugin mechanism is far from perfect now and we will continuously devote to improve it.