Deployment

All installation files are in the inlong-audit directory, if you use MySQL to store audit data, you need to first create the database through sql/apache_inlong_audit_mysql.sql.

  1. # initialize database
  2. mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_audit_mysql.sql

If you use StarRocks to store audit data, you need to first create the database through sql/apache_inlong_audit_starrocks.sql.

  1. # initialize database
  2. mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_audit_starrocks.sql

Audit Proxy

Configure MessageQueue

You can choose Apache Pulsar, Apache Kafka or InLong TubeMQ as your MessageQueue service:

  • If using Pulsar, the configuration file is conf/audit-proxy-pulsar.conf. Change the Pulsar Topic info.
  1. agent1.sinks.pulsar-sink-msg1.topic = persistent://public/default/inlong-audit
  2. agent1.sinks.pulsar-sink-msg2.topic = persistent://public/default/inlong-audit
  • If using Kafka, the configuration file is conf/audit-proxy-kafka.conf. Change the Kafka Topic info.
  1. agent1.sinks.kafka-sink-msg1.topic = inlong-audit
  2. agent1.sinks.kafka-sink-msg2.topic = inlong-audit

Start

  1. # By default, pulsar is used as the MessageQueue, and the audit-proxy-pulsar.conf configuration file is loaded.
  2. bash +x ./bin/proxy-start.sh [pulsar｜kafka]

The default listen port is 10081.

Audit Store

Configure

The configuration file is conf/application.properties.

  1. # the MQ type for audit proxy: pulsar / kafka
  2. audit.config.proxy.type=pulsar
  4. # manger config
  5. manager.hosts=127.0.0.1:8083
  7. # Get Kafka or Pulsar address from the cluster tag
  8. default.mq.cluster.tag=default_cluster
  10. # pulsar config
  11. audit.pulsar.topic=persistent://public/default/inlong-audit
  12. audit.pulsar.consumer.sub.name=inlong-audit-subscription
  13. audit.pulsar.token=
  14. audit.pulsar.enable.auth=false
  16. # kafka config
  17. audit.kafka.topic=inlong-audit
  18. audit.kafka.topic.numPartitions=3
  19. audit.kafka.topic.replicationFactor=2
  20. audit.kafka.consumer.name=inlong-audit-consumer
  21. audit.kafka.group.id=audit-consumer-group
  23. # Generic jdbc storage
  24. jdbc.driver=com.mysql.cj.jdbc.Driver
  25. jdbc.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
  26. jdbc.username=root
  27. jdbc.password=inlong

Dependencies

  • If the backend database is MySQL, please download mysql-connector-java-8.0.28.jar and put it into lib/ directory.

  • If the backend database is PostgreSQL, there’s no need for additional dependencies.

Start

  1. bash +x ./bin/store-start.sh

Audit Service

Configure

The configuration file is conf/audit-service.properties

  1. mysql.jdbc.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useUnicode=true&rewriteBatchedStatements=true
  2. mysql.username=root
  3. mysql.password=inlong

  • (optional) Configure Audit Data Sources In the audit_source_config table used by the Audit Service, configure the data source for audit storage. By default, the same MySQL configuration is used as the Audit Service

  • (optional) Configure Audit Items In the audit_id_config table used by the Audit Service, configure the audit items that need to be cached. By default, Agent is used to receive successfully, Agent is sent successfully, DataProxy is received successfully, and DataProxy is sent successfully.

Dependencies

  • If the backend database is MySQL, please download mysql-connector-java-8.0.28.jar and put it into lib/ directory.
  • If the backend database is PostgreSQL, there’s no need for additional dependencies.

Start

  1. bash +x ./bin/service-start.sh

The default listen port is 10080.