Asynchronous Log Live Push

Interservice Asynchronous Push Scheme

1 Background

With the use of the microservice architecture, multiple microservices will be deployed on different servers, and the logs generated by each microservice will be distributed among the servers. While the ELK is able to filter the logs to be viewed by users, there is a lack of time and customization.

If log disks before collection logs are used, it will be difficult to classify logs according to the user’s running information, first because of the excess time spent on disk, and then because of the loss of user information during the running phase after the log disk.

2 Ideas

Design an Appender based on log4j2 and add to the configuration of the microservice log, then use the listener design mode and the RPC’s service call to implement the log in real time.

3 Implementation

3.1 Design implementation of Appender components

The current background development of the more popular log framework uses slf4j as the faculty for logic, logback or log4j2, and now the open source project will prefer log4j2 as a framework for log printing, owing to its performance advantages and open source license.

Appender is a target (destination) printed in log4j2 logs. Once a log event is generated during the microservice running, all the Appenders registered in the configuration will get this log event.

The Linkis designed Appender will cache the acquired log event in the queue after the log microservice log is generated, and we have a listener in the Appender.

3.2 The design implementation of listeners

The listener mode is a common design mode and is a common method of implementing asynchronous callback programming.

The listener needs to listen for the log event queue and if the log event queue is full, the log will be removed from the queue and sent via HTTP.

The listener also needs to listen to the task state and after the task has completed all the execution steps in this microservice, it will be necessary to send all the logs cached in the queue to prevent the log loss.

3.3 Design implementation for the cache queue

The reason for using the cache queue is that if a log event is generated it will push the microservice to receive the log because HTTP requests are too frequent, so it must be cached and the general cache queue can be designed as a blockqueue with the maximum number of caches.

3.4 Logging collection

The log will be sent to the microservice collected by the log, which will then sort and encapsulated the log, which will be pushed to the user interface, and the log will be asynchronous and backups of a task log for each user.