Basic Concept

NameDescriptionOther
Standard ArchitectureContains all InLong components such as InLong Agent/Manager/MQ/Sort/DashboardSuitable for massive data and large-scale production environments
Lightweight ArchitectureContains only one component of InLong Sort, which can be used with Manager/DashboardThe lightweight architecture is simple and flexible, suitable for small-scale data
GroupData Streams Group, it contains multiple data streams, and one Group represents one data ingestion.Group has attributes such as ID and Name.
StreamData Stream, a stream has a specific flow direction.Stream has attributes such as ID, Name, and data fields.
NodeData Node, including Extract Node and Load Node, stands for the data source and sink types separately.
InLongMsgInLong data format, if you consume message directly from the message queue, you need to perform InLongMsg parsing first.
AgentThe standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities.It contains File Agent, SQL Agent, Binlog Agent, etc.
DataProxyForward received data to different message queues.Supports data transmission blocking, placing retransmission.
SortData stream sorting.Sort-flink based on Flink, sort-standalone for local sorting.
TubeMQInLong’s self-developed message queuing serviceIt can also be called Tube, with low-cost, high-performance features.
PulsarApache Pulsar, a high-performance, high-consistency message queue service