DataHub

DataHub is a rich metadata platform that supports features like data discovery, data obeservability, federated governance, etc.

Since Hudi 0.11.0, you can now sync to a DataHub instance by setting DataHubSyncTool as one of the sync tool classes for HoodieStreamer.

The target Hudi table will be sync’ed to DataHub as a Dataset. The Hudi table’s avro schema will be sync’ed, along with the commit timestamp when running the sync.

Configurations

DataHubSyncTool makes use of DataHub’s Java Emitter to send the metadata via HTTP REST APIs. It is required to set hoodie.meta.sync.datahub.emitter.server to the URL of the DataHub instance for sync.

If needs auth token, set hoodie.meta.sync.datahub.emitter.token.

If needs customized creation of the emitter object, implement org.apache.hudi.sync.datahub.config.DataHubEmitterSupplier and supply the implementation’s FQCN to hoodie.meta.sync.datahub.emitter.supplier.class.

By default, the sync config’s database name and table name will be used to make the target Dataset‘s URN. Subclass HoodieDataHubDatasetIdentifier and set it to hoodie.meta.sync.datahub.dataset.identifier.class to customize the URN creation.

Example

The following shows an example configuration to run HoodieStreamer with DataHubSyncTool.

In addition to hudi-utilities-slim-bundle that contains HoodieStreamer, you also add hudi-datahub-sync-bundle to the classpath.

  1. spark-submit --master yarn \
  2. --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0 \
  3. --jars /opt/hudi-datahub-sync-bundle-1.0.0.jar \
  4. --class org.apache.hudi.utilities.streamer.HoodieStreamer \
  5. /opt/hudi-utilities-slim-bundle_2.12-1.0.0.jar \
  6. --target-table mytable \
  7. # ... other HoodieStreamer's configs
  8. --enable-sync \
  9. --sync-tool-classes org.apache.hudi.sync.datahub.DataHubSyncTool \
  10. --hoodie-conf hoodie.meta.sync.datahub.emitter.server=http://url-to-datahub-instance:8080 \
  11. --hoodie-conf hoodie.datasource.hive_sync.database=mydb \
  12. --hoodie-conf hoodie.datasource.hive_sync.table=mytable \