Basic Concept

NameDescriptionOther
StandardStandard Architecture, contains all InLong components such as InLong Agent, Manager, MQ, Sort, DashboardSuitable for massive data and large-scale production environments
LightweightLightweight Architecture, contains only one component of InLong Sort, which also can be used with Manager, DashboardThe lightweight architecture is simple and flexible, suitable for small-scale data
GroupData Streams Group, it contains multiple data streams, and one Group represents one data business unit.Group has attributes such as ID and Name.
StreamData Stream, a stream has a specific data source, data format and data sink.Stream has attributes such as ID, Name, and data fields.
NodeData Node, including Extract Node and Load Node, stands for the data source and sink types separately.
InLongMsgInLong data format, if you consume message directly from the message queue, you need to perform InLongMsg parsing first.
ClusterEach component can form a single cluster.Contains cluster name, label, necessary information for each component, etc.
TagClusters of different components can use the same tag to represent a set of data stream execution units.Currently tag are only available for clusters
AgentThe standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities.It contains File Agent, SQL Agent, Binlog Agent, etc.
DataProxyForward received data to different message queues.Supports data transmission blocking, placing retransmission.
SortData stream sorting.Sort-flink based on Flink, sort-standalone for local sorting.
TubeMQInLong’s self-developed message queuing serviceIt can also be called Tube, with low-cost, high-performance features.
PulsarApache Pulsar, a high-performance, high-consistency message queue service