Modules Beta

Modules allow users to extend Flink’s built-in objects, such as defining functions that behave like Flink built-in functions. They are pluggable, and while Flink provides a few pre-built modules, users can write their own.

For example, users can define their own geo functions and plug them into Flink as built-in functions to be used in Flink SQL and Table APIs. Another example is users can load an out-of-shelf Hive module to use Hive built-in functions as Flink built-in functions.

• Module Types
  • CoreModule
  • HiveModule
  • User-Defined Module
• Namespace and Resolution Order
• Module API
  • Loading and unloading a Module
  • List Available Modules

Module Types

CoreModule

CoreModule contains all of Flink’s system (built-in) functions and is loaded by default.

HiveModule

The HiveModule provides Hive built-in functions as Flink’s system functions to SQL and Table API users. Flink’s Hive documentation provides full details on setting up the module.

User-Defined Module

Users can develop custom modules by implementing the Module interface. To use custom modules in SQL CLI, users should develop both a module and its corresponding module factory by implementing the ModuleFactory interface.

A module factory defines a set of properties for configuring the module when the SQL CLI bootstraps. Properties are passed to a discovery service where the service tries to match the properties to a ModuleFactory and instantiate a corresponding module instance.

Namespace and Resolution Order

Objects provided by modules are considered part of Flink’s system (built-in) objects; thus, they don’t have any namespaces.

When there are two objects of the same name residing in two modules, Flink always resolves the object reference to the one in the 1st loaded module.

Module API

Loading and unloading a Module

Users can load and unload modules in an existing Flink session.

tableEnv.loadModule("myModule", new CustomModule());
tableEnv.unloadModule("myModule");

All modules defined using YAML must provide a type property that specifies the type. The following types are supported out of the box.

modules:
   - name: core
     type: core
   - name: myhive
     type: hive

List Available Modules

tableEnv.listModules();

Flink SQL> SHOW MODULES;