Data to HStreamDB

HStreamDB is a streaming database designed for streaming data, with complete lifecycle management for accessing, storing, processing, and distributing large-scale real-time data streams. It uses standard SQL (and its stream extensions) as the primary interface language, with real-time as the main feature, and aims to simplify the operation and management of data streams and the development of real-time applications.

image

For more details, please refer to HStream official websiteSave data to HStreamDB - 图2 (opens new window)。

Create HStreamDB Server

Refer to the following documentsSave data to HStreamDB - 图3 (opens new window) for deployment instructions, which provide tutorials on deploying with docker locally or on the cloud.

Use the command to start an hstream-client

  1. docker run -it --rm --name some-hstream-cli --network host hstreamdb/hstream:v0.8.0 hstream-client --port 6570 --client-id 1

When entering the console, help info like these will be shown:

  1.       __  _________________  _________    __  ___
  2.      / / / / ___/_  __/ __ \/ ____/   |  /  |/  /
  3.     / /_/ /\__ \ / / / /_/ / __/ / /| | / /|_/ /
  4.    / __  /___/ // / / _, _/ /___/ ___ |/ /  / /
  5.   /_/ /_//____//_/ /_/ |_/_____/_/  |_/_/  /_/
  8. Command
  9.   :h                           To show these help info
  10.   :q                           To exit command line interface
  11.   :help [sql_operation]        To show full usage of sql statement

Create stream:

  1. > CREATE STREAM demo_stream;
  2. demo_stream
  3. > SHOW STREAMS;
  4. demo_stream
  5. >

Create HStreamDB resource

Access EMQX Dashboard, Click on Rules Engine, Resources, Create, select HStreamDB Resources, enter the resource address and link pool.

image

Create Rule

Click on Rules Engine > Resources > Create, then select resources type HStreamDB.

Edit the rule SQL.

  1. SELECT
  3.   payload
  5. FROM
  7.   "#"

The SQL rules in the document are for demonstrations only, please write the SQL according to the business design.

Click Add Action and select Data Persistence to save the data to HSTreamDB. Select the resource created in the previous step and enter the parameters, which are defined in the following table.

ParameterDefinitionType
StreamStream Name, Cannot be placeholdersString
Ordering KeyCannot be placeholdersString
Enable Batch AppendEnable or disable batch appending, default is trueBoolean
Max Batch Append CountMaximum number of message entries in a batchInteger
Max Batch Interval(ms)Batch maximum interval, in millisecondsInteger
Payload TemplateContent of the data message writtenBinary

Click Confirm to create it.

image

Now use the state-of-the-art MQTT desktop client MQTTX to connect to EMQX and send a message.

image

See rule monitor.

image

At this point the data is written to HStreamDB and the message is consumed using any consumption method. The documentation uses a simple consumption tool based on the HStream golang SDK, and the reader is free to write the consumption side according to a familiar programming language. The consumption log can be seen as follows. Test tool see fetcherSave data to HStreamDB - 图8 (opens new window).

  1. ./fetcher -s f1 -n demo_stream -p 127.0.0.1:6570 -c cons1 -v
  1. {"level":"info","ts":1656311005.5250711,"msg":"[f1]","recordId":"[BatchId: 8589934593, BatchIndex: 0, ShardId: 1317059070792293]","payload":"Hello HSreamDB !"}

The message has been written successfully and is consumed to completion.