Hive Example

Here we use a simple example to help you experience InLong by Docker.

Install Hive

Hive is the necessary component. If you don’t have Hive in your machine, we recommand using Docker to install it. Details can be found here.

Note that if you use Docker, you need to add a port mapping 8020:8020, because it’s the port of HDFS DefaultFS, and we need to use it later.

Install InLong

Before we begin, we need to install InLong. Here we provide two ways:

  1. Install InLong with Docker by according to the instructions here.(Recommanded)
  2. Install InLong binary according to the instructions here.

Create a data access

After deployment, we first enter the “Data Access” interface, click “Create an Access” in the upper right corner to create a new date access, and fill in the data streams group information as shown in the figure below.

Create Group

Then we click the next button, and fill in the stream information as shown in the figure below.

Create Stream

Note that the message source is “File”, and we don’t need to create a message source manually.

Then we fill in the following information in the “data information” column below.

Data Information

Then we select Hive in the data flow and click “Add” to add Hive configuration

Hive Config

Note that the target table does not need to be created in advance, as InLong Manager will automatically create the table for us after the access is approved. Also, please use connection test to ensure that InLong Manager can connect to your Hive.

Then we click the “Submit for Approval” button, the connection will be created successfully and enter the approval state.

Approve the data access

Then we enter the “Approval Management” interface and click “My Approval” to approve the data access that we just applied for.

At this point, the data access has been created successfully. We can see that the corresponding table has been created in Hive, and we can see that the corresponding topic has been created successfully in the management GUI of TubeMQ.

Configure the agent

Create a collect job by using curl to make a request.

  1. curl --location --request POST 'http://localhost:8008/config/job' \
  2. --header 'Content-Type: application/json' \
  3. --data '{
  4. "job": {
  5. "dir": {
  6. "path": "",
  7. "pattern": "/data/collect-data/test.log"
  8. },
  9. "trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
  10. "id": 1,
  11. "thread": {
  12. "running": {
  13. "core": "4"
  14. }
  15. },
  16. "name": "fileAgentTest",
  17. "source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
  18. "sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
  19. "channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
  20. },
  21. "proxy": {
  22. "inlongGroupId": "b_test_group",
  23. "inlongStreamId": "test_stream"
  24. },
  25. "op": "add"
  26. }'

At this point, the agent is configured successfully. Then we need to create a new file ./collect-data/test.log and add content to it to trigger the agent to send data to the dataproxy.

  1. mkdir collect-data
  2. END=100000
  3. for ((i=1;i<=END;i++)); do
  4.     sleep 3
  5.     echo "name_$i | $i" >> ./collect-data/test.log
  6. done

Then we can observe the logs of agent and dataproxy, and we can see that the relevant data has been sent successfully.

  1. $ docker logs agent
  2. $ docker logs dataproxy