Deployment

All deploying files at inlong-audit directory, if you use MySQL to store audit data, you need to first create the database through sql/apache_inlong_audit.sql.

  1. # initialize database
  2. mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_audit.sql

If you use ClickHouse to store audit data, you need to first create the database through sql/apache_inlong_audit_clickhouse.sql.

  1. # initialize database
  2. clickhouse client -u DB_USER --password DB_PASSWD < sql/apache_inlong_audit_clickhouse.sql

Audit Proxy

Configure MessageQueue

You can choose Apache Pulsar or InLong TubeMQ as your MessageQueue service:

  • If using Pulsar, the configuration file is conf/audit-proxy-pulsar.conf. Change the Pulsar service url for next configuration.
  1. agent1.sources.tcp-source.port = 10081
  2. agent1.sinks.pulsar-sink-msg1.pulsar_server_url= pulsar://localhost:6650
  3. agent1.sinks.pulsar-sink-msg1.topic = persistent://public/default/inlong-audit
  4. agent1.sinks.pulsar-sink-msg2.pulsar_server_url = pulsar://localhost:6650
  5. agent1.sinks.pulsar-sink-msg2.topic = persistent://public/default/inlong-audit
  • If using TubeMQ, the configuration file is conf/audit-proxy-tube.conf. Change the TubeMQ master address for next configuration.
  1. agent1.sources.tcp-source.port = 10081
  2. agent1.sinks.tube-sink-msg1.master-host-port-list = localhost:8715
  3. agent1.sinks.tube-sink-msg1.topic = inlong-audit
  4. agent1.sinks.tube-sink-msg2.master-host-port-list = localhost:8715
  5. agent1.sinks.tube-sink-msg2.topic = inlong-audit

Start

  1. # By default, pulsar is used as the MessageQueue, and the audit-proxy-pulsar.conf configuration file is loaded.
  2. bash +x ./bin/proxy-start.sh [pulsar｜tube]

Audit Store

Configure

The configuration file is conf/application.properties.

  1. # proxy.type: pulsar / tube
  2. audit.config.proxy.type=pulsar
  4. # store.server: mysql / elasticsearch / clickhouse 
  5. audit.config.store.mode=mysql
  7. # audit pulsar config (optional), replace PULSAR_BROKER_LIST with your Pulsar service url
  8. audit.pulsar.server.url=pulsar://PULSAR_BROKER_LIST
  9. audit.pulsar.topic=persistent://public/default/inlong-audit
  10. audit.pulsar.consumer.sub.name=sub-audit
  12. # audit tube config (optional), replace TUBE_LIST with your TubeMQ master address
  13. audit.tube.masterlist=TUBE_LIST
  14. audit.tube.topic=inlong-audit
  15. audit.tube.consumer.group.name=inlong-audit-consumer
  17. # mysql config
  18. spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
  19. spring.datasource.druid.username=root
  20. spring.datasource.druid.password=inlong
  22. # es config
  23. elasticsearch.host=127.0.0.1
  24. elasticsearch.port=9200
  25. elasticsearch.authEnable=false
  26. elasticsearch.username=elastic
  27. elasticsearch.password=inlong
  28. elasticsearch.shardsNum=5
  29. elasticsearch.replicaNum=1
  30. elasticsearch.indexDeleteDay=5
  31. elasticsearch.enableDocId=true
  32. elasticsearch.bulkInterval=10
  33. elasticsearch.bulkThreshold=5000
  34. elasticsearch.auditIdSet=1,2
  36. # clickhouse config
  37. clickhouse.driver=ru.yandex.clickhouse.ClickHouseDriver
  38. clickhouse.url=jdbc:clickhouse://127.0.0.1:8123/default
  39. clickhouse.username=default
  40. clickhouse.password=default
  41. clickhouse.batchIntervalMs=1000
  42. clickhouse.batchThreshold=500
  43. clickhouse.processIntervalMs=100

Dependencies

  • If the backend database is MySQL, please download mysql-connector-java-8.0.27.jar and put it into lib/ directory.

  • If the backend database is PostgreSQL, there’s no need for additional dependencies.

Start

  1. bash +x ./bin/store-start.sh

The default listen port is 10081.