EngineConn Metrics Reporting Feature

The reported information lacks engine information, and the reported resources and progress interfaces are redundant, which reduces performance. It needs to be aligned for optimization and adjustment, and an extension module should be added to the reporting protocol.

• Added RPC protocol containing resources, progress, and additional information, supporting reporting of these information in one request
• Reconstruct existing resources and progress reporting links, and combine the actions of reporting related information into one request

This requirement involves the linkis-entrance, linkis-computation-orchestrator, linkis-orchestrator-ecm-plugin, linkis-computation-engineconn modules. Add and refactor the reporting information in the computation-engineconn module, and parse the information and store it on the entry side.

The engine information reporting architecture is shown in the figure. After the user submits the task to the entry, the entry applies to the linkismanager for an engine. After applying to the engine, submit tasks to the application, and receive regular reports of tasks (resources, progress, status). Until the task is executed, the entry returns the final result when the user queries. For this modification, the engine metrics information needs to be added to the entry into the database; Combine Resource and Progress interface information in Orchestrator, and add additional information such as metrics; On the ComputationEngineConn side of the interactive engine, the reported resources and progress information are combined, and engine statistics are additionally reported.

This feature involves the following function point modules:

-[input] The input is the interactive engine computation-engineconn. When the engine executes a task, it reports the running information TaskRunningInfo, including the original TaskProgressInfo and TaskResourceInfo, and adds the engine example information and the information about the number of existing tasks of the engine.

• [Processing process] orchestrator-plugin-ecm is responsible for monitoring the reporting information when the engine runs tasks, receiving the reporting information, and generating the TaskRunningInfoEvent asynchronous message, Sent to OrchestratorAsyncListenerBus for processing. The TaskRunningInfoListener registered to the OrchestratorAsyncListener receives the message, triggers the listener method, and calls back to the TaskRunningInfo callback method of the Entrance Job. The callback method parses the resource, progress, and engine metrancs information in TaskRunningInfo and persists them respectively.

RPC protocol TaskRunningInfo has been added to the requirement, no db table has been added

No external interface

RPC interface internal authentication, does not involve external security issues

Combined two RPC interfaces to reduce the number of reports and improve performance

Less metrics information, no impact

not involving