Docker Demo

A Demo using Docker containers

Let’s use a real world example to see how Hudi works end to end. For this purpose, a self contained data infrastructure is brought up in a local Docker cluster within your computer. It requires the Hudi repo to have been cloned locally.

The steps have been tested on a Mac laptop

Prerequisites

  • Clone the Hudi repository to your local machine.

  • Docker Setup : For Mac, Please follow the steps as defined in Install Docker Desktop on Mac. For running Spark-SQL queries, please ensure atleast 6 GB and 4 CPUs are allocated to Docker (See Docker -> Preferences -> Advanced). Otherwise, spark-SQL queries could be killed because of memory issues.

  • kcat : A command-line utility to publish/consume from kafka topics. Use brew install kcat to install kcat.

  • /etc/hosts : The demo references many services running in container by the hostname. Add the following settings to /etc/hosts

    1. 127.0.0.1 adhoc-1
    2. 127.0.0.1 adhoc-2
    3. 127.0.0.1 namenode
    4. 127.0.0.1 datanode1
    5. 127.0.0.1 hiveserver
    6. 127.0.0.1 hivemetastore
    7. 127.0.0.1 kafkabroker
    8. 127.0.0.1 sparkmaster
    9. 127.0.0.1 zookeeper

  • Java : Java SE Development Kit 8.

  • Maven : A build automation tool for Java projects.

  • jq : A lightweight and flexible command-line JSON processor. Use brew install jq to install jq.

Also, this has not been tested on some environments like Docker on Windows.

Setting up Docker Cluster

Build Hudi

The first step is to build Hudi. Note This step builds Hudi on default supported scala version - 2.11.

NOTE: Make sure you’ve cloned the Hudi repository first.

  1. cd <HUDI_WORKSPACE>
  2. mvn clean package -Pintegration-tests -DskipTests

Bringing up Demo Cluster

The next step is to run the Docker compose script and setup configs for bringing up the cluster. These files are in the Hudi repository which you should already have locally on your machine from the previous steps.

This should pull the Docker images from Docker hub and setup the Docker cluster.

  • Default
  • Mac AArch64
  1. cd docker
  2. ./setup_demo.sh
  3. ....
  4. ....
  5. ....
  6. [+] Running 10/13
  7. ⠿ Container zookeeper             Removed                 8.6s
  8. ⠿ Container datanode1             Removed                18.3s
  9. ⠿ Container trino-worker-1        Removed                50.7s
  10. ⠿ Container spark-worker-1        Removed                16.7s
  11. ⠿ Container adhoc-2               Removed                16.9s
  12. ⠿ Container graphite              Removed                16.9s
  13. ⠿ Container kafkabroker           Removed                14.1s
  14. ⠿ Container adhoc-1               Removed                14.1s
  15. ⠿ Container presto-worker-1       Removed                11.9s
  16. ⠿ Container presto-coordinator-1  Removed                34.6s
  17. .......
  18. ......
  19. [+] Running 17/17
  20. ⠿ adhoc-1 Pulled                                          2.9s
  21. ⠿ graphite Pulled                                         2.8s
  22. ⠿ spark-worker-1 Pulled                                   3.0s
  23. ⠿ kafka Pulled                                            2.9s
  24. ⠿ datanode1 Pulled                                        2.9s
  25. ⠿ hivemetastore Pulled                                    2.9s
  26. ⠿ hiveserver Pulled                                       3.0s
  27. ⠿ hive-metastore-postgresql Pulled                        2.8s
  28. ⠿ presto-coordinator-1 Pulled                             2.9s
  29. ⠿ namenode Pulled                                         2.9s
  30. ⠿ trino-worker-1 Pulled                                   2.9s
  31. ⠿ sparkmaster Pulled                                      2.9s
  32. ⠿ presto-worker-1 Pulled                                  2.9s
  33. ⠿ zookeeper Pulled                                        2.8s
  34. ⠿ adhoc-2 Pulled                                          2.9s
  35. ⠿ historyserver Pulled                                    2.9s
  36. ⠿ trino-coordinator-1 Pulled                              2.9s
  37. [+] Running 17/17
  38. ⠿ Container zookeeper                  Started           41.0s
  39. ⠿ Container kafkabroker                Started           41.7s
  40. ⠿ Container graphite                   Started           41.5s
  41. ⠿ Container hive-metastore-postgresql  Running            0.0s
  42. ⠿ Container namenode                   Running            0.0s
  43. ⠿ Container hivemetastore              Running            0.0s
  44. ⠿ Container trino-coordinator-1        Runni...           0.0s
  45. ⠿ Container presto-coordinator-1       Star...           42.1s
  46. ⠿ Container historyserver              Started           41.0s
  47. ⠿ Container datanode1                  Started           49.9s
  48. ⠿ Container hiveserver                 Running            0.0s
  49. ⠿ Container trino-worker-1             Started           42.1s
  50. ⠿ Container sparkmaster                Started           41.9s
  51. ⠿ Container spark-worker-1             Started           50.2s
  52. ⠿ Container adhoc-2                    Started           38.5s
  53. ⠿ Container adhoc-1                    Started           38.5s
  54. ⠿ Container presto-worker-1            Started           38.4s
  55. Copying spark default config and setting up configs
  56. Copying spark default config and setting up configs
  57. $ docker ps

Docker Demo - 图1Please note the following for Mac AArch64 users

  • The demo must be built and run using the master branch. We currently plan to include support starting with the 0.13.0 release.
  • Presto and Trino are not currently supported in the demo.
  1. cd docker
  2. ./setup_demo.sh --mac-aarch64
  3. .......
  4. ......
  5. [+] Running 12/12
  6. ⠿ adhoc-1 Pulled                                          2.9s
  7. ⠿ spark-worker-1 Pulled                                   3.0s
  8. ⠿ kafka Pulled                                            2.9s
  9. ⠿ datanode1 Pulled                                        2.9s
  10. ⠿ hivemetastore Pulled                                    2.9s
  11. ⠿ hiveserver Pulled                                       3.0s
  12. ⠿ hive-metastore-postgresql Pulled                        2.8s
  13. ⠿ namenode Pulled                                         2.9s
  14. ⠿ sparkmaster Pulled                                      2.9s
  15. ⠿ zookeeper Pulled                                        2.8s
  16. ⠿ adhoc-2 Pulled                                          2.9s
  17. ⠿ historyserver Pulled                                    2.9s
  18. [+] Running 12/12
  19. ⠿ Container zookeeper                  Started           41.0s
  20. ⠿ Container kafkabroker                Started           41.7s
  21. ⠿ Container hive-metastore-postgresql  Running            0.0s
  22. ⠿ Container namenode                   Running            0.0s
  23. ⠿ Container hivemetastore              Running            0.0s
  24. ⠿ Container historyserver              Started           41.0s
  25. ⠿ Container datanode1                  Started           49.9s
  26. ⠿ Container hiveserver                 Running            0.0s
  27. ⠿ Container sparkmaster                Started           41.9s
  28. ⠿ Container spark-worker-1             Started           50.2s
  29. ⠿ Container adhoc-2                    Started           38.5s
  30. ⠿ Container adhoc-1                    Started           38.5s
  31. Copying spark default config and setting up configs
  32. Copying spark default config and setting up configs
  33. $ docker ps

At this point, the Docker cluster will be up and running. The demo cluster brings up the following services

  • HDFS Services (NameNode, DataNode)
  • Spark Master and Worker
  • Hive Services (Metastore, HiveServer2 along with PostgresDB)
  • Kafka Broker and a Zookeeper Node (Kafka will be used as upstream source for the demo)
  • Containers for Presto setup (Presto coordinator and worker)
  • Containers for Trino setup (Trino coordinator and worker)
  • Adhoc containers to run Hudi/Hive CLI commands

Demo

Stock Tracker data will be used to showcase different Hudi query types and the effects of Compaction.

Take a look at the directory docker/demo/data. There are 2 batches of stock data - each at 1 minute granularity. The first batch contains stocker tracker data for some stock symbols during the first hour of trading window (9:30 a.m to 10:30 a.m). The second batch contains tracker data for next 30 mins (10:30 - 11 a.m). Hudi will be used to ingest these batches to a table which will contain the latest stock tracker data at hour level granularity. The batches are windowed intentionally so that the second batch contains updates to some of the rows in the first batch.

Step 1 : Publish the first batch to Kafka

Upload the first batch to Kafka topic ‘stock ticks’

cat docker/demo/data/batch_1.json | kcat -b kafkabroker -t stock_ticks -P

To check if the new topic shows up, use

  1. kcat -b kafkabroker -L -J | jq .
  2. {
  3.   "originating_broker": {
  4.     "id": 1001,
  5.     "name": "kafkabroker:9092/1001"
  6.   },
  7.   "query": {
  8.     "topic": "*"
  9.   },
  10.   "brokers": [
  11.     {
  12.       "id": 1001,
  13.       "name": "kafkabroker:9092"
  14.     }
  15.   ],
  16.   "topics": [
  17.     {
  18.       "topic": "stock_ticks",
  19.       "partitions": [
  20.         {
  21.           "partition": 0,
  22.           "leader": 1001,
  23.           "replicas": [
  24.             {
  25.               "id": 1001
  26.             }
  27.           ],
  28.           "isrs": [
  29.             {
  30.               "id": 1001
  31.             }
  32.           ]
  33.         }
  34.       ]
  35.     }
  36.   ]
  37. }

Step 2: Incrementally ingest data from Kafka topic

Hudi comes with a tool named Hudi Streamer. This tool can connect to variety of data sources (including Kafka) to pull changes and apply to Hudi table using upsert/insert primitives. Here, we will use the tool to download json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool automatically initializes the tables in the file-system if they do not exist yet.

  1. docker exec -it adhoc-2 /bin/bash
  3. # Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_cow table in HDFS
  4. spark-submit \
  5.   --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
  6.   --table-type COPY_ON_WRITE \
  7.   --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  8.   --source-ordering-field ts  \
  9.   --target-base-path /user/hive/warehouse/stock_ticks_cow \
  10.   --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
  11.   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
  13. # Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_mor table in HDFS
  14. spark-submit \
  15.   --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
  16.   --table-type MERGE_ON_READ \
  17.   --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  18.   --source-ordering-field ts \
  19.   --target-base-path /user/hive/warehouse/stock_ticks_mor \
  20.   --target-table stock_ticks_mor \
  21.   --props /var/demo/config/kafka-source.properties \
  22.   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
  23.   --disable-compaction
  25. # As part of the setup (Look at setup_demo.sh), the configs needed for Hudi Streamer is uploaded to HDFS. The configs
  26. # contain mostly Kafa connectivity settings, the avro-schema to be used for ingesting along with key and partitioning fields.
  28. exit

You can use HDFS web-browser to look at the tables http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow.

You can explore the new partition folder created in the table along with a “commit” / “deltacommit” file under .hoodie which signals a successful commit.

There will be a similar setup when you browse the MOR table http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor

Step 3: Sync with Hive

At this step, the tables are available in HDFS. We need to sync with Hive to create new Hive tables and add partitions inorder to run Hive queries against those tables.

  1. docker exec -it adhoc-2 /bin/bash
  3. # This command takes in HiveServer URL and COW Hudi table location in HDFS and sync the HDFS state to Hive
  4. /var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
  5.   --jdbc-url jdbc:hive2://hiveserver:10000 \
  6.   --user hive \
  7.   --pass hive \
  8.   --partitioned-by dt \
  9.   --base-path /user/hive/warehouse/stock_ticks_cow \
  10.   --database default \
  11.   --table stock_ticks_cow \
  12.   --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
  13. .....
  14. 2020-01-25 19:51:28,953 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_cow
  15. .....
  17. # Now run hive-sync for the second data-set in HDFS using Merge-On-Read (MOR table type)
  18. /var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
  19.   --jdbc-url jdbc:hive2://hiveserver:10000 \
  20.   --user hive \
  21.   --pass hive \
  22.   --partitioned-by dt \
  23.   --base-path /user/hive/warehouse/stock_ticks_mor \
  24.   --database default \
  25.   --table stock_ticks_mor \
  26.   --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
  27. ...
  28. 2020-01-25 19:51:51,066 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_ro
  29. ...
  30. 2020-01-25 19:51:51,569 INFO  [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(129)) - Sync complete for stock_ticks_mor_rt
  31. ....
  33. exit

After executing the above command, you will notice

  1. A hive table named stock_ticks_cow created which supports Snapshot and Incremental queries on Copy On Write table.
  2. Two new tables stock_ticks_mor_rt and stock_ticks_mor_ro created for the Merge On Read table. The former supports Snapshot and Incremental queries (providing near-real time data) while the later supports ReadOptimized queries.

Step 4 (a): Run Hive Queries

Run a hive query to find the latest timestamp ingested for stock symbol ‘GOOG’. You will notice that both snapshot (for both COW and MOR _rt table) and read-optimized queries (for MOR _ro table) give the same value “10:29 a.m” as Hudi create a parquet file for the first batch of data.

  1. docker exec -it adhoc-2 /bin/bash
  2. beeline -u jdbc:hive2://hiveserver:10000 \
  3.   --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
  4.   --hiveconf hive.stats.autogather=false
  6. # List Tables
  7. 0: jdbc:hive2://hiveserver:10000> show tables;
  8. +---------------------+--+
  9. |      tab_name       |
  10. +---------------------+--+
  11. | stock_ticks_cow     |
  12. | stock_ticks_mor_ro  |
  13. | stock_ticks_mor_rt  |
  14. +---------------------+--+
  15. 3 rows selected (1.199 seconds)
  16. 0: jdbc:hive2://hiveserver:10000>
  19. # Look at partitions that were added
  20. 0: jdbc:hive2://hiveserver:10000> show partitions stock_ticks_mor_rt;
  21. +----------------+--+
  22. |   partition    |
  23. +----------------+--+
  24. | dt=2018-08-31  |
  25. +----------------+--+
  26. 1 row selected (0.24 seconds)
  29. # COPY-ON-WRITE Queries:
  30. =========================
  33. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
  34. +---------+----------------------+--+
  35. | symbol  |         _c1          |
  36. +---------+----------------------+--+
  37. | GOOG    | 2018-08-31 10:29:00  |
  38. +---------+----------------------+--+
  40. Now, run a projection query:
  42. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG';
  43. +----------------------+---------+----------------------+---------+------------+-----------+--+
  44. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  45. +----------------------+---------+----------------------+---------+------------+-----------+--+
  46. | 20180924221953       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  47. | 20180924221953       | GOOG    | 2018-08-31 10:29:00  | 3391    | 1230.1899  | 1230.085  |
  48. +----------------------+---------+----------------------+---------+------------+-----------+--+
  51. # Merge-On-Read Queries:
  52. ==========================
  54. Lets run similar queries against M-O-R table. Lets look at both 
  55. ReadOptimized and Snapshot(realtime data) queries supported by M-O-R table
  57. # Run ReadOptimized Query. Notice that the latest timestamp is 10:29
  58. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  59. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  60. +---------+----------------------+--+
  61. | symbol  |         _c1          |
  62. +---------+----------------------+--+
  63. | GOOG    | 2018-08-31 10:29:00  |
  64. +---------+----------------------+--+
  65. 1 row selected (6.326 seconds)
  68. # Run Snapshot Query. Notice that the latest timestamp is again 10:29
  70. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
  71. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  72. +---------+----------------------+--+
  73. | symbol  |         _c1          |
  74. +---------+----------------------+--+
  75. | GOOG    | 2018-08-31 10:29:00  |
  76. +---------+----------------------+--+
  77. 1 row selected (1.606 seconds)
  80. # Run Read Optimized and Snapshot project queries
  82. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  83. +----------------------+---------+----------------------+---------+------------+-----------+--+
  84. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  85. +----------------------+---------+----------------------+---------+------------+-----------+--+
  86. | 20180924222155       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  87. | 20180924222155       | GOOG    | 2018-08-31 10:29:00  | 3391    | 1230.1899  | 1230.085  |
  88. +----------------------+---------+----------------------+---------+------------+-----------+--+
  90. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG';
  91. +----------------------+---------+----------------------+---------+------------+-----------+--+
  92. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  93. +----------------------+---------+----------------------+---------+------------+-----------+--+
  94. | 20180924222155       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  95. | 20180924222155       | GOOG    | 2018-08-31 10:29:00  | 3391    | 1230.1899  | 1230.085  |
  96. +----------------------+---------+----------------------+---------+------------+-----------+--+
  98. exit

Step 4 (b): Run Spark-SQL Queries

Hudi support Spark as query processor just like Hive. Here are the same hive queries running in spark-sql

  1. docker exec -it adhoc-1 /bin/bash
  2. $SPARK_INSTALL/bin/spark-shell \
  3.   --jars $HUDI_SPARK_BUNDLE \
  4.   --master local[2] \
  5.   --driver-class-path $HADOOP_CONF_DIR \
  6.   --conf spark.sql.hive.convertMetastoreParquet=false \
  7.   --deploy-mode client \
  8.   --driver-memory 1G \
  9.   --executor-memory 3G \
  10.   --num-executors 1
  11. ...
  13. Welcome to
  14.       ____              __
  15.      / __/__  ___ _____/ /__
  16.     _\ \/ _ \/ _ `/ __/  '_/
  17.    /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
  18.       /_/
  20. Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
  21. Type in expressions to have them evaluated.
  22. Type :help for more information.
  24. scala> spark.sql("show tables").show(100, false)
  25. +--------+------------------+-----------+
  26. |database|tableName         |isTemporary|
  27. +--------+------------------+-----------+
  28. |default |stock_ticks_cow   |false      |
  29. |default |stock_ticks_mor_ro|false      |
  30. |default |stock_ticks_mor_rt|false      |
  31. +--------+------------------+-----------+
  33. # Copy-On-Write Table
  35. ## Run max timestamp query against COW table
  37. scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
  38. [Stage 0:>                                                          (0 + 1) / 1]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  39. SLF4J: Defaulting to no-operation (NOP) logger implementation
  40. SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
  41. +------+-------------------+
  42. |symbol|max(ts)            |
  43. +------+-------------------+
  44. |GOOG  |2018-08-31 10:29:00|
  45. +------+-------------------+
  47. ## Projection Query
  49. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG'").show(100, false)
  50. +-------------------+------+-------------------+------+---------+--------+
  51. |_hoodie_commit_time|symbol|ts                 |volume|open     |close   |
  52. +-------------------+------+-------------------+------+---------+--------+
  53. |20180924221953     |GOOG  |2018-08-31 09:59:00|6330  |1230.5   |1230.02 |
  54. |20180924221953     |GOOG  |2018-08-31 10:29:00|3391  |1230.1899|1230.085|
  55. +-------------------+------+-------------------+------+---------+--------+
  57. # Merge-On-Read Queries:
  58. ==========================
  60. Lets run similar queries against M-O-R table. Lets look at both
  61. ReadOptimized and Snapshot queries supported by M-O-R table
  63. # Run ReadOptimized Query. Notice that the latest timestamp is 10:29
  64. scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
  65. +------+-------------------+
  66. |symbol|max(ts)            |
  67. +------+-------------------+
  68. |GOOG  |2018-08-31 10:29:00|
  69. +------+-------------------+
  72. # Run Snapshot Query. Notice that the latest timestamp is again 10:29
  74. scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
  75. +------+-------------------+
  76. |symbol|max(ts)            |
  77. +------+-------------------+
  78. |GOOG  |2018-08-31 10:29:00|
  79. +------+-------------------+
  81. # Run Read Optimized and Snapshot project queries
  83. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG'").show(100, false)
  84. +-------------------+------+-------------------+------+---------+--------+
  85. |_hoodie_commit_time|symbol|ts                 |volume|open     |close   |
  86. +-------------------+------+-------------------+------+---------+--------+
  87. |20180924222155     |GOOG  |2018-08-31 09:59:00|6330  |1230.5   |1230.02 |
  88. |20180924222155     |GOOG  |2018-08-31 10:29:00|3391  |1230.1899|1230.085|
  89. +-------------------+------+-------------------+------+---------+--------+
  91. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG'").show(100, false)
  92. +-------------------+------+-------------------+------+---------+--------+
  93. |_hoodie_commit_time|symbol|ts                 |volume|open     |close   |
  94. +-------------------+------+-------------------+------+---------+--------+
  95. |20180924222155     |GOOG  |2018-08-31 09:59:00|6330  |1230.5   |1230.02 |
  96. |20180924222155     |GOOG  |2018-08-31 10:29:00|3391  |1230.1899|1230.085|
  97. +-------------------+------+-------------------+------+---------+--------+

Step 4 (c): Run Presto Queries

Here are the Presto queries for similar Hive and Spark queries.

Docker Demo - 图2note

  • Currently, Presto does not support snapshot or incremental queries on Hudi tables.
  • This section of the demo is not supported for Mac AArch64 users at this time.
  1. docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
  2. presto> show catalogs;
  3.   Catalog
  4. -----------
  5.  hive
  6.  jmx
  7.  localfile
  8.  system
  9. (4 rows)
  11. Query 20190817_134851_00000_j8rcz, FINISHED, 1 node
  12. Splits: 19 total, 19 done (100.00%)
  13. 0:04 [0 rows, 0B] [0 rows/s, 0B/s]
  15. presto> use hive.default;
  16. USE
  17. presto:default> show tables;
  18.        Table
  19. --------------------
  20.  stock_ticks_cow
  21.  stock_ticks_mor_ro
  22.  stock_ticks_mor_rt
  23. (3 rows)
  25. Query 20190822_181000_00001_segyw, FINISHED, 2 nodes
  26. Splits: 19 total, 19 done (100.00%)
  27. 0:05 [3 rows, 99B] [0 rows/s, 18B/s]
  30. # COPY-ON-WRITE Queries:
  31. =========================
  34. presto:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
  35.  symbol |        _col1
  36. --------+---------------------
  37.  GOOG   | 2018-08-31 10:29:00
  38. (1 row)
  40. Query 20190822_181011_00002_segyw, FINISHED, 1 node
  41. Splits: 49 total, 49 done (100.00%)
  42. 0:12 [197 rows, 613B] [16 rows/s, 50B/s]
  44. presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
  45.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close
  46. ---------------------+--------+---------------------+--------+-----------+----------
  47.  20190822180221      | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02
  48.  20190822180221      | GOOG   | 2018-08-31 10:29:00 |   3391 | 1230.1899 | 1230.085
  49. (2 rows)
  51. Query 20190822_181141_00003_segyw, FINISHED, 1 node
  52. Splits: 17 total, 17 done (100.00%)
  53. 0:02 [197 rows, 613B] [109 rows/s, 341B/s]
  56. # Merge-On-Read Queries:
  57. ==========================
  59. Lets run similar queries against M-O-R table. 
  61. # Run ReadOptimized Query. Notice that the latest timestamp is 10:29
  62.     presto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  63.  symbol |        _col1
  64. --------+---------------------
  65.  GOOG   | 2018-08-31 10:29:00
  66. (1 row)
  68. Query 20190822_181158_00004_segyw, FINISHED, 1 node
  69. Splits: 49 total, 49 done (100.00%)
  70. 0:02 [197 rows, 613B] [110 rows/s, 343B/s]
  73. presto:default>  select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  74.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close
  75. ---------------------+--------+---------------------+--------+-----------+----------
  76.  20190822180250      | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02
  77.  20190822180250      | GOOG   | 2018-08-31 10:29:00 |   3391 | 1230.1899 | 1230.085
  78. (2 rows)
  80. Query 20190822_181256_00006_segyw, FINISHED, 1 node
  81. Splits: 17 total, 17 done (100.00%)
  82. 0:02 [197 rows, 613B] [92 rows/s, 286B/s]
  84. presto:default> exit

Step 4 (d): Run Trino Queries

Here are the similar queries with Trino.

Docker Demo - 图3note

  • Currently, Trino does not support snapshot or incremental queries on Hudi tables.
  • This section of the demo is not supported for Mac AArch64 users at this time.
  1. docker exec -it adhoc-2 trino --server trino-coordinator-1:8091
  2. trino> show catalogs;
  3.  Catalog 
  4. ---------
  5.  hive    
  6.  system  
  7. (2 rows)
  9. Query 20220112_055038_00000_sac73, FINISHED, 1 node
  10. Splits: 19 total, 19 done (100.00%)
  11. 3.74 [0 rows, 0B] [0 rows/s, 0B/s]
  13. trino> use hive.default;
  14. USE
  15. trino:default> show tables;
  16.        Table        
  17. --------------------
  18.  stock_ticks_cow    
  19.  stock_ticks_mor_ro 
  20.  stock_ticks_mor_rt 
  21. (3 rows)
  23. Query 20220112_055050_00003_sac73, FINISHED, 2 nodes
  24. Splits: 19 total, 19 done (100.00%)
  25. 1.84 [3 rows, 102B] [1 rows/s, 55B/s]
  27. # COPY-ON-WRITE Queries:
  28. =========================
  30. trino:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
  31.  symbol |        _col1        
  32. --------+---------------------
  33.  GOOG   | 2018-08-31 10:29:00 
  34. (1 row)
  36. Query 20220112_055101_00005_sac73, FINISHED, 1 node
  37. Splits: 49 total, 49 done (100.00%)
  38. 4.08 [197 rows, 442KB] [48 rows/s, 108KB/s]
  40. trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
  41.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close   
  42. ---------------------+--------+---------------------+--------+-----------+----------
  43.  20220112054822108   | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02 
  44.  20220112054822108   | GOOG   | 2018-08-31 10:29:00 |   3391 | 1230.1899 | 1230.085 
  45. (2 rows)
  47. Query 20220112_055113_00006_sac73, FINISHED, 1 node
  48. Splits: 17 total, 17 done (100.00%)
  49. 0.40 [197 rows, 450KB] [487 rows/s, 1.09MB/s]
  51. # Merge-On-Read Queries:
  52. ==========================
  54. Lets run similar queries against MOR table.
  56. # Run ReadOptimized Query. Notice that the latest timestamp is 10:29
  58. trino:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  59.  symbol |        _col1        
  60. --------+---------------------
  61.  GOOG   | 2018-08-31 10:29:00 
  62. (1 row)
  64. Query 20220112_055125_00007_sac73, FINISHED, 1 node
  65. Splits: 49 total, 49 done (100.00%)
  66. 0.50 [197 rows, 442KB] [395 rows/s, 888KB/s]
  68. trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  69.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close   
  70. ---------------------+--------+---------------------+--------+-----------+----------
  71.  20220112054844841   | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02 
  72.  20220112054844841   | GOOG   | 2018-08-31 10:29:00 |   3391 | 1230.1899 | 1230.085 
  73. (2 rows)
  75. Query 20220112_055136_00008_sac73, FINISHED, 1 node
  76. Splits: 17 total, 17 done (100.00%)
  77. 0.49 [197 rows, 450KB] [404 rows/s, 924KB/s]
  79. trino:default> exit

Step 5: Upload second batch to Kafka and run Hudi Streamer to ingest

Upload the second batch of data and ingest this batch using Hudi Streamer. As this batch does not bring in any new partitions, there is no need to run hive-sync

  1. cat docker/demo/data/batch_2.json | kcat -b kafkabroker -t stock_ticks -P
  3. # Within Docker container, run the ingestion command
  4. docker exec -it adhoc-2 /bin/bash
  6. # Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_cow table in HDFS
  7. spark-submit \
  8.   --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
  9.   --table-type COPY_ON_WRITE \
  10.   --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  11.   --source-ordering-field ts \
  12.   --target-base-path /user/hive/warehouse/stock_ticks_cow \
  13.   --target-table stock_ticks_cow \
  14.   --props /var/demo/config/kafka-source.properties \
  15.   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
  17. # Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_mor table in HDFS
  18. spark-submit \
  19.   --class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
  20.   --table-type MERGE_ON_READ \
  21.   --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
  22.   --source-ordering-field ts \
  23.   --target-base-path /user/hive/warehouse/stock_ticks_mor \
  24.   --target-table stock_ticks_mor \
  25.   --props /var/demo/config/kafka-source.properties \
  26.   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
  27.   --disable-compaction
  29. exit

With Copy-On-Write table, the second ingestion by Hudi Streamer resulted in a new version of Parquet file getting created. See http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow/2018/08/31

With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file. Take a look at the HDFS filesystem to get an idea: http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor/2018/08/31

Step 6 (a): Run Hive Queries

With Copy-On-Write table, the Snapshot query immediately sees the changes as part of second batch once the batch got committed as each ingestion creates newer versions of parquet files.

With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file. This is the time, when ReadOptimized and Snapshot queries will provide different results. ReadOptimized query will still return “10:29 am” as it will only read from the Parquet file. Snapshot query will do on-the-fly merge and return latest committed data which is “10:59 a.m”.

  1. docker exec -it adhoc-2 /bin/bash
  2. beeline -u jdbc:hive2://hiveserver:10000 \
  3.   --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
  4.   --hiveconf hive.stats.autogather=false
  6. # Copy On Write Table:
  8. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
  9. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  10. +---------+----------------------+--+
  11. | symbol  |         _c1          |
  12. +---------+----------------------+--+
  13. | GOOG    | 2018-08-31 10:59:00  |
  14. +---------+----------------------+--+
  15. 1 row selected (1.932 seconds)
  17. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG';
  18. +----------------------+---------+----------------------+---------+------------+-----------+--+
  19. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  20. +----------------------+---------+----------------------+---------+------------+-----------+--+
  21. | 20180924221953       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  22. | 20180924224524       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  23. +----------------------+---------+----------------------+---------+------------+-----------+--+
  25. As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
  28. # Merge On Read Table:
  30. # Read Optimized Query
  31. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  32. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  33. +---------+----------------------+--+
  34. | symbol  |         _c1          |
  35. +---------+----------------------+--+
  36. | GOOG    | 2018-08-31 10:29:00  |
  37. +---------+----------------------+--+
  38. 1 row selected (1.6 seconds)
  40. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  41. +----------------------+---------+----------------------+---------+------------+-----------+--+
  42. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  43. +----------------------+---------+----------------------+---------+------------+-----------+--+
  44. | 20180924222155       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  45. | 20180924222155       | GOOG    | 2018-08-31 10:29:00  | 3391    | 1230.1899  | 1230.085  |
  46. +----------------------+---------+----------------------+---------+------------+-----------+--+
  48. # Snapshot Query
  49. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
  50. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  51. +---------+----------------------+--+
  52. | symbol  |         _c1          |
  53. +---------+----------------------+--+
  54. | GOOG    | 2018-08-31 10:59:00  |
  55. +---------+----------------------+--+
  57. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG';
  58. +----------------------+---------+----------------------+---------+------------+-----------+--+
  59. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  60. +----------------------+---------+----------------------+---------+------------+-----------+--+
  61. | 20180924222155       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  62. | 20180924224537       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  63. +----------------------+---------+----------------------+---------+------------+-----------+--+
  65. exit

Step 6 (b): Run Spark SQL Queries

Running the same queries in Spark-SQL:

  1. docker exec -it adhoc-1 /bin/bash
  2. $SPARK_INSTALL/bin/spark-shell \
  3.   --jars $HUDI_SPARK_BUNDLE \
  4.   --driver-class-path $HADOOP_CONF_DIR \
  5.   --conf spark.sql.hive.convertMetastoreParquet=false \
  6.   --deploy-mode client \
  7.   --driver-memory 1G \
  8.   --master local[2] \
  9.   --executor-memory 3G \
  10.   --num-executors 1
  12. # Copy On Write Table:
  14. scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
  15. +------+-------------------+
  16. |symbol|max(ts)            |
  17. +------+-------------------+
  18. |GOOG  |2018-08-31 10:59:00|
  19. +------+-------------------+
  21. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG'").show(100, false)
  23. +----------------------+---------+----------------------+---------+------------+-----------+--+
  24. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  25. +----------------------+---------+----------------------+---------+------------+-----------+--+
  26. | 20180924221953       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  27. | 20180924224524       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  28. +----------------------+---------+----------------------+---------+------------+-----------+--+
  30. As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
  33. # Merge On Read Table:
  35. # Read Optimized Query
  36. scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
  37. +---------+----------------------+
  38. | symbol  |         _c1          |
  39. +---------+----------------------+
  40. | GOOG    | 2018-08-31 10:29:00  |
  41. +---------+----------------------+
  42. 1 row selected (1.6 seconds)
  44. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG'").show(100, false)
  45. +----------------------+---------+----------------------+---------+------------+-----------+
  46. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  47. +----------------------+---------+----------------------+---------+------------+-----------+
  48. | 20180924222155       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  49. | 20180924222155       | GOOG    | 2018-08-31 10:29:00  | 3391    | 1230.1899  | 1230.085  |
  50. +----------------------+---------+----------------------+---------+------------+-----------+
  52. # Snapshot Query
  53. scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
  54. +---------+----------------------+
  55. | symbol  |         _c1          |
  56. +---------+----------------------+
  57. | GOOG    | 2018-08-31 10:59:00  |
  58. +---------+----------------------+
  60. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG'").show(100, false)
  61. +----------------------+---------+----------------------+---------+------------+-----------+
  62. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  63. +----------------------+---------+----------------------+---------+------------+-----------+
  64. | 20180924222155       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  65. | 20180924224537       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  66. +----------------------+---------+----------------------+---------+------------+-----------+
  68. exit

Step 6 (c): Run Presto Queries

Running the same queries on Presto for ReadOptimized queries.

Docker Demo - 图4note

This section of the demo is not supported for Mac AArch64 users at this time.

  1. docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
  2. presto> use hive.default;
  3. USE
  5. # Copy On Write Table:
  7. presto:default>select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
  8.  symbol |        _col1
  9. --------+---------------------
  10.  GOOG   | 2018-08-31 10:59:00
  11. (1 row)
  13. Query 20190822_181530_00007_segyw, FINISHED, 1 node
  14. Splits: 49 total, 49 done (100.00%)
  15. 0:02 [197 rows, 613B] [125 rows/s, 389B/s]
  17. presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG';
  18.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close
  19. ---------------------+--------+---------------------+--------+-----------+----------
  20.  20190822180221      | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02
  21.  20190822181433      | GOOG   | 2018-08-31 10:59:00 |   9021 | 1227.1993 | 1227.215
  22. (2 rows)
  24. Query 20190822_181545_00008_segyw, FINISHED, 1 node
  25. Splits: 17 total, 17 done (100.00%)
  26. 0:02 [197 rows, 613B] [106 rows/s, 332B/s]
  28. As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
  31. # Merge On Read Table:
  33. # Read Optimized Query
  34. presto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  35.  symbol |        _col1
  36. --------+---------------------
  37.  GOOG   | 2018-08-31 10:29:00
  38. (1 row)
  40. Query 20190822_181602_00009_segyw, FINISHED, 1 node
  41. Splits: 49 total, 49 done (100.00%)
  42. 0:01 [197 rows, 613B] [139 rows/s, 435B/s]
  44. presto:default>select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  45.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close
  46. ---------------------+--------+---------------------+--------+-----------+----------
  47.  20190822180250      | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02
  48.  20190822180250      | GOOG   | 2018-08-31 10:29:00 |   3391 | 1230.1899 | 1230.085
  49. (2 rows)
  51. Query 20190822_181615_00010_segyw, FINISHED, 1 node
  52. Splits: 17 total, 17 done (100.00%)
  53. 0:01 [197 rows, 613B] [154 rows/s, 480B/s]
  55. presto:default> exit

Step 6 (d): Run Trino Queries

Running the same queries on Trino for Read-Optimized queries.

Docker Demo - 图5note

This section of the demo is not supported for Mac AArch64 users at this time.

  1. docker exec -it adhoc-2 trino --server trino-coordinator-1:8091
  2. trino> use hive.default;
  3. USE
  5. # Copy On Write Table:
  7. trino:default> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
  8.  symbol |        _col1        
  9. --------+---------------------
  10.  GOOG   | 2018-08-31 10:59:00 
  11. (1 row)
  13. Query 20220112_055443_00012_sac73, FINISHED, 1 node
  14. Splits: 49 total, 49 done (100.00%)
  15. 0.63 [197 rows, 442KB] [310 rows/s, 697KB/s]
  17. trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG';
  18.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close   
  19. ---------------------+--------+---------------------+--------+-----------+----------
  20.  20220112054822108   | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02 
  21.  20220112055352654   | GOOG   | 2018-08-31 10:59:00 |   9021 | 1227.1993 | 1227.215 
  22. (2 rows)
  24. Query 20220112_055450_00013_sac73, FINISHED, 1 node
  25. Splits: 17 total, 17 done (100.00%)
  26. 0.65 [197 rows, 450KB] [303 rows/s, 692KB/s]
  28. As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
  30. # Merge On Read Table:
  31. # Read Optimized Query
  33. trino:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  34.  symbol |        _col1        
  35. --------+---------------------
  36.  GOOG   | 2018-08-31 10:29:00 
  37. (1 row)
  39. Query 20220112_055500_00014_sac73, FINISHED, 1 node
  40. Splits: 49 total, 49 done (100.00%)
  41. 0.59 [197 rows, 442KB] [336 rows/s, 756KB/s]
  43. trino:default> select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  44.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close   
  45. ---------------------+--------+---------------------+--------+-----------+----------
  46.  20220112054844841   | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02 
  47.  20220112054844841   | GOOG   | 2018-08-31 10:29:00 |   3391 | 1230.1899 | 1230.085 
  48. (2 rows)
  50. Query 20220112_055506_00015_sac73, FINISHED, 1 node
  51. Splits: 17 total, 17 done (100.00%)
  52. 0.35 [197 rows, 450KB] [556 rows/s, 1.24MB/s]
  54. trino:default> exit

Step 7 (a): Incremental Query for COPY-ON-WRITE Table

With 2 batches of data ingested, lets showcase the support for incremental queries in Hudi Copy-On-Write tables

Lets take the same projection query example

  1. docker exec -it adhoc-2 /bin/bash
  2. beeline -u jdbc:hive2://hiveserver:10000 \
  3.   --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
  4.   --hiveconf hive.stats.autogather=false
  6. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG';
  7. +----------------------+---------+----------------------+---------+------------+-----------+--+
  8. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  9. +----------------------+---------+----------------------+---------+------------+-----------+--+
  10. | 20180924064621       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  11. | 20180924065039       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  12. +----------------------+---------+----------------------+---------+------------+-----------+--+

As you notice from the above queries, there are 2 commits - 20180924064621 and 20180924065039 in timeline order. When you follow the steps, you will be getting different timestamps for commits. Substitute them in place of the above timestamps.

To show the effects of incremental-query, let us assume that a reader has already seen the changes as part of ingesting first batch. Now, for the reader to see effect of the second batch, he/she has to keep the start timestamp to the commit time of the first batch (20180924064621) and run incremental query

Hudi incremental mode provides efficient scanning for incremental queries by filtering out files that do not have any candidate rows using hudi-managed metadata.

  1. docker exec -it adhoc-2 /bin/bash
  2. beeline -u jdbc:hive2://hiveserver:10000 \
  3.   --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
  4.   --hiveconf hive.stats.autogather=false
  6. 0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;
  7. No rows affected (0.009 seconds)
  8. 0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.max.commits=3;
  9. No rows affected (0.009 seconds)
  10. 0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.start.timestamp=20180924064621;

With the above setting, file-ids that do not have any updates from the commit 20180924065039 is filtered out without scanning. Here is the incremental query :

  1. 0: jdbc:hive2://hiveserver:10000>
  2. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow where  symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064621';
  3. +----------------------+---------+----------------------+---------+------------+-----------+--+
  4. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  5. +----------------------+---------+----------------------+---------+------------+-----------+--+
  6. | 20180924065039       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  7. +----------------------+---------+----------------------+---------+------------+-----------+--+
  8. 1 row selected (0.83 seconds)
  9. 0: jdbc:hive2://hiveserver:10000>

Step 7 (b): Incremental Query with Spark SQL:

  1. docker exec -it adhoc-1 /bin/bash
  2. $SPARK_INSTALL/bin/spark-shell \
  3.   --jars $HUDI_SPARK_BUNDLE \
  4.   --driver-class-path $HADOOP_CONF_DIR \
  5.   --conf spark.sql.hive.convertMetastoreParquet=false \
  6.   --deploy-mode client \
  7.   --driver-memory 1G \
  8.   --master local[2] \
  9.   --executor-memory 3G \
  10.   --num-executors 1
  12. Welcome to
  13.       ____              __
  14.      / __/__  ___ _____/ /__
  15.     _\ \/ _ \/ _ `/ __/  '_/
  16.    /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
  17.       /_/
  19. Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
  20. Type in expressions to have them evaluated.
  21. Type :help for more information.
  23. scala> import org.apache.hudi.DataSourceReadOptions
  24. import org.apache.hudi.DataSourceReadOptions
  26. # In the below query, 20180925045257 is the first commit's timestamp
  27. scala> val hoodieIncViewDF =  spark.read.format("org.apache.hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL).option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY, "20180924064621").load("/user/hive/warehouse/stock_ticks_cow")
  28. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  29. SLF4J: Defaulting to no-operation (NOP) logger implementation
  30. SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
  31. hoodieIncViewDF: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 15 more fields]
  33. scala> hoodieIncViewDF.registerTempTable("stock_ticks_cow_incr_tmp1")
  34. warning: there was one deprecation warning; re-run with -deprecation for details
  36. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_cow_incr_tmp1 where  symbol = 'GOOG'").show(100, false);
  37. +----------------------+---------+----------------------+---------+------------+-----------+
  38. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  39. +----------------------+---------+----------------------+---------+------------+-----------+
  40. | 20180924065039       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  41. +----------------------+---------+----------------------+---------+------------+-----------+

Step 8: Schedule and Run Compaction for Merge-On-Read table

Lets schedule and run a compaction to create a new version of columnar file so that read-optimized readers will see fresher data. Again, You can use Hudi CLI to manually schedule and run compaction

  1. docker exec -it adhoc-1 /bin/bash
  2. root@adhoc-1:/opt# /var/hoodie/ws/hudi-cli/hudi-cli.sh
  3. ...
  4. Table command getting loaded
  5. HoodieSplashScreen loaded
  6. ===================================================================
  7. *         ___                          ___                        *
  8. *        /\__\          ___           /\  \           ___         *
  9. *       / /  /         /\__\         /  \  \         /\  \        *
  10. *      / /__/         / /  /        / /\ \  \        \ \  \       *
  11. *     /  \  \ ___    / /  /        / /  \ \__\       /  \__\      *
  12. *    / /\ \  /\__\  / /__/  ___   / /__/ \ |__|     / /\/__/      *
  13. *    \/  \ \/ /  /  \ \  \ /\__\  \ \  \ / /  /  /\/ /  /         *
  14. *         \  /  /    \ \  / /  /   \ \  / /  /   \  /__/          *
  15. *         / /  /      \ \/ /  /     \ \/ /  /     \ \__\          *
  16. *        / /  /        \  /  /       \  /  /       \/__/          *
  17. *        \/__/          \/__/         \/__/    Apache Hudi CLI    *
  18. *                                                                 *
  19. ===================================================================
  21. Welcome to Apache Hudi CLI. Please type help if you are looking for help.
  22. hudi->connect --path /user/hive/warehouse/stock_ticks_mor
  23. 18/09/24 06:59:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  24. 18/09/24 06:59:35 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
  25. 18/09/24 06:59:35 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://namenode:8020], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1261652683_11, ugi=root (auth:SIMPLE)]]]
  26. 18/09/24 06:59:35 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
  27. 18/09/24 06:59:36 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
  28. Metadata for table stock_ticks_mor loaded
  29. hoodie:stock_ticks_mor->compactions show all
  30. 20/02/10 03:41:32 INFO timeline.HoodieActiveTimeline: Loaded instants [[20200210015059__clean__COMPLETED], [20200210015059__deltacommit__COMPLETED], [20200210022758__clean__COMPLETED], [20200210022758__deltacommit__COMPLETED], [==>20200210023843__compaction__REQUESTED]]
  31. ___________________________________________________________________
  32. | Compaction Instant Time| State    | Total FileIds to be Compacted|
  33. |==================================================================|
  35. # Schedule a compaction. This will use Spark Launcher to schedule compaction
  36. hoodie:stock_ticks_mor->compaction schedule --hoodieConfigs hoodie.compact.inline.max.delta.commits=1
  37. ....
  38. Compaction successfully completed for 20180924070031
  40. # Now refresh and check again. You will see that there is a new compaction requested
  42. hoodie:stock_ticks_mor->refresh
  43. 18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
  44. 18/09/24 07:01:16 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
  45. 18/09/24 07:01:16 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
  46. Metadata for table stock_ticks_mor loaded
  48. hoodie:stock_ticks_mor->compactions show all
  49. 18/09/24 06:34:12 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924041125__clean__COMPLETED], [20180924041125__deltacommit__COMPLETED], [20180924042735__clean__COMPLETED], [20180924042735__deltacommit__COMPLETED], [==>20180924063245__compaction__REQUESTED]]
  50. ___________________________________________________________________
  51. | Compaction Instant Time| State    | Total FileIds to be Compacted|
  52. |==================================================================|
  53. | 20180924070031         | REQUESTED| 1                            |
  55. # Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query
  56. hoodie:stock_ticks_mor->compaction run --compactionInstant  20180924070031 --parallelism 2 --sparkMemory 1G  --schemaFilePath /var/demo/config/schema.avsc --retry 1  
  57. ....
  58. Compaction successfully completed for 20180924070031
  60. ## Now check if compaction is completed
  62. hoodie:stock_ticks_mor->refresh
  63. 18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
  64. 18/09/24 07:03:00 INFO table.HoodieTableConfig: Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
  65. 18/09/24 07:03:00 INFO table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1) from /user/hive/warehouse/stock_ticks_mor
  66. Metadata for table stock_ticks_mor loaded
  68. hoodie:stock_ticks_mor->compactions show all
  69. 18/09/24 07:03:15 INFO timeline.HoodieActiveTimeline: Loaded instants [[20180924064636__clean__COMPLETED], [20180924064636__deltacommit__COMPLETED], [20180924065057__clean__COMPLETED], [20180924065057__deltacommit__COMPLETED], [20180924070031__commit__COMPLETED]]
  70. ___________________________________________________________________
  71. | Compaction Instant Time| State    | Total FileIds to be Compacted|
  72. |==================================================================|
  73. | 20180924070031         | COMPLETED| 1                            |

Step 9: Run Hive Queries including incremental queries

You will see that both ReadOptimized and Snapshot queries will show the latest committed data. Lets also run the incremental query for MOR table. From looking at the below query output, it will be clear that the fist commit time for the MOR table is 20180924064636 and the second commit time is 20180924070031

  1. docker exec -it adhoc-2 /bin/bash
  2. beeline -u jdbc:hive2://hiveserver:10000 \
  3.   --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
  4.   --hiveconf hive.stats.autogather=false
  6. # Read Optimized Query
  7. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  8. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  9. +---------+----------------------+--+
  10. | symbol  |         _c1          |
  11. +---------+----------------------+--+
  12. | GOOG    | 2018-08-31 10:59:00  |
  13. +---------+----------------------+--+
  14. 1 row selected (1.6 seconds)
  16. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  17. +----------------------+---------+----------------------+---------+------------+-----------+--+
  18. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  19. +----------------------+---------+----------------------+---------+------------+-----------+--+
  20. | 20180924064636       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  21. | 20180924070031       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  22. +----------------------+---------+----------------------+---------+------------+-----------+--+
  24. # Snapshot Query
  25. 0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
  26. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  27. +---------+----------------------+--+
  28. | symbol  |         _c1          |
  29. +---------+----------------------+--+
  30. | GOOG    | 2018-08-31 10:59:00  |
  31. +---------+----------------------+--+
  33. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG';
  34. +----------------------+---------+----------------------+---------+------------+-----------+--+
  35. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  36. +----------------------+---------+----------------------+---------+------------+-----------+--+
  37. | 20180924064636       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  38. | 20180924070031       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  39. +----------------------+---------+----------------------+---------+------------+-----------+--+
  41. # Incremental Query:
  43. 0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.mode=INCREMENTAL;
  44. No rows affected (0.008 seconds)
  45. # Max-Commits covers both second batch and compaction commit
  46. 0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.max.commits=3;
  47. No rows affected (0.007 seconds)
  48. 0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.start.timestamp=20180924064636;
  49. No rows affected (0.013 seconds)
  50. # Query:
  51. 0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG' and `_hoodie_commit_time` > '20180924064636';
  52. +----------------------+---------+----------------------+---------+------------+-----------+--+
  53. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  54. +----------------------+---------+----------------------+---------+------------+-----------+--+
  55. | 20180924070031       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  56. +----------------------+---------+----------------------+---------+------------+-----------+--+
  58. exit

Step 10: Read Optimized and Snapshot queries for MOR with Spark-SQL after compaction

  1. docker exec -it adhoc-1 /bin/bash
  2. $SPARK_INSTALL/bin/spark-shell \
  3.   --jars $HUDI_SPARK_BUNDLE \
  4.   --driver-class-path $HADOOP_CONF_DIR \
  5.   --conf spark.sql.hive.convertMetastoreParquet=false \
  6.   --deploy-mode client \
  7.   --driver-memory 1G \
  8.   --master local[2] \
  9.   --executor-memory 3G \
  10.   --num-executors 1
  12. # Read Optimized Query
  13. scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
  14. +---------+----------------------+
  15. | symbol  |        max(ts)       |
  16. +---------+----------------------+
  17. | GOOG    | 2018-08-31 10:59:00  |
  18. +---------+----------------------+
  19. 1 row selected (1.6 seconds)
  21. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG'").show(100, false)
  22. +----------------------+---------+----------------------+---------+------------+-----------+
  23. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  24. +----------------------+---------+----------------------+---------+------------+-----------+
  25. | 20180924064636       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  26. | 20180924070031       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  27. +----------------------+---------+----------------------+---------+------------+-----------+
  29. # Snapshot Query
  30. scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
  31. +---------+----------------------+
  32. | symbol  |     max(ts)          |
  33. +---------+----------------------+
  34. | GOOG    | 2018-08-31 10:59:00  |
  35. +---------+----------------------+
  37. scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG'").show(100, false)
  38. +----------------------+---------+----------------------+---------+------------+-----------+
  39. | _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
  40. +----------------------+---------+----------------------+---------+------------+-----------+
  41. | 20180924064636       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
  42. | 20180924070031       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
  43. +----------------------+---------+----------------------+---------+------------+-----------+

Step 11: Presto Read Optimized queries on MOR table after compaction

Docker Demo - 图6note

This section of the demo is not supported for Mac AArch64 users at this time.

  1. docker exec -it presto-worker-1 presto --server presto-coordinator-1:8090
  2. presto> use hive.default;
  3. USE
  5. # Read Optimized Query
  6. resto:default> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
  7.   symbol |        _col1
  8. --------+---------------------
  9.  GOOG   | 2018-08-31 10:59:00
  10. (1 row)
  12. Query 20190822_182319_00011_segyw, FINISHED, 1 node
  13. Splits: 49 total, 49 done (100.00%)
  14. 0:01 [197 rows, 613B] [133 rows/s, 414B/s]
  16. presto:default> select "_hoodie_commit_time", symbol, ts, volume, open, close  from stock_ticks_mor_ro where  symbol = 'GOOG';
  17.  _hoodie_commit_time | symbol |         ts          | volume |   open    |  close
  18. ---------------------+--------+---------------------+--------+-----------+----------
  19.  20190822180250      | GOOG   | 2018-08-31 09:59:00 |   6330 |    1230.5 |  1230.02
  20.  20190822181944      | GOOG   | 2018-08-31 10:59:00 |   9021 | 1227.1993 | 1227.215
  21. (2 rows)
  23. Query 20190822_182333_00012_segyw, FINISHED, 1 node
  24. Splits: 17 total, 17 done (100.00%)
  25. 0:02 [197 rows, 613B] [98 rows/s, 307B/s]
  27. presto:default>

This brings the demo to an end.

Testing Hudi in Local Docker environment

You can bring up a Hadoop Docker environment containing Hadoop, Hive and Spark services with support for Hudi.

  1. $ mvn pre-integration-test -DskipTests

The above command builds Docker images for all the services with current Hudi source installed at /var/hoodie/ws and also brings up the services using a compose file. We currently use Hadoop (v2.8.4), Hive (v2.3.3) and Spark (v2.4.4) in Docker images.

To bring down the containers

  1. $ cd hudi-integ-test
  2. $ mvn docker-compose:down

If you want to bring up the Docker containers, use

  1. $ cd hudi-integ-test
  2. $ mvn docker-compose:up -DdetachedMode=true

Hudi is a library that is operated in a broader data analytics/ingestion environment involving Hadoop, Hive and Spark. Interoperability with all these systems is a key objective for us. We are actively adding integration-tests under hudi-integ-test/src/test/java that makes use of this docker environment (See hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieSanity.java )

Building Local Docker Containers:

The Docker images required for demo and running integration test are already in docker-hub. The Docker images and compose scripts are carefully implemented so that they serve dual-purpose

  1. The Docker images have inbuilt Hudi jar files with environment variable pointing to those jars (HUDI_HADOOP_BUNDLE, …)
  2. For running integration-tests, we need the jars generated locally to be used for running services within docker. The docker-compose scripts (see docker/compose/docker-compose_hadoop284_hive233_spark244.yml) ensures local jars override inbuilt jars by mounting local Hudi workspace over the Docker location
  3. As these Docker containers have mounted local Hudi workspace, any changes that happen in the workspace would automatically reflect in the containers. This is a convenient way for developing and verifying Hudi for developers who do not own a distributed environment. Note that this is how integration tests are run.

This helps avoid maintaining separate Docker images and avoids the costly step of building Hudi Docker images locally. But if users want to test Hudi from locations with lower network bandwidth, they can still build local images run the script docker/build_local_docker_images.sh to build local Docker images before running docker/setup_demo.sh

Here are the commands:

  1. cd docker
  2. ./build_local_docker_images.sh
  3. .....
  5. [INFO] Reactor Summary:
  6. [INFO]
  7. [INFO] Hudi ............................................... SUCCESS [  2.507 s]
  8. [INFO] hudi-common ........................................ SUCCESS [ 15.181 s]
  9. [INFO] hudi-aws ........................................... SUCCESS [  2.621 s]
  10. [INFO] hudi-timeline-service .............................. SUCCESS [  1.811 s]
  11. [INFO] hudi-client ........................................ SUCCESS [  0.065 s]
  12. [INFO] hudi-client-common ................................. SUCCESS [  8.308 s]
  13. [INFO] hudi-hadoop-mr ..................................... SUCCESS [  3.733 s]
  14. [INFO] hudi-spark-client .................................. SUCCESS [ 18.567 s]
  15. [INFO] hudi-sync-common ................................... SUCCESS [  0.794 s]
  16. [INFO] hudi-hive-sync ..................................... SUCCESS [  3.691 s]
  17. [INFO] hudi-spark-datasource .............................. SUCCESS [  0.121 s]
  18. [INFO] hudi-spark-common_2.11 ............................. SUCCESS [ 12.979 s]
  19. [INFO] hudi-spark2_2.11 ................................... SUCCESS [ 12.516 s]
  20. [INFO] hudi-spark_2.11 .................................... SUCCESS [ 35.649 s]
  21. [INFO] hudi-utilities_2.11 ................................ SUCCESS [  5.881 s]
  22. [INFO] hudi-utilities-bundle_2.11 ......................... SUCCESS [ 12.661 s]
  23. [INFO] hudi-cli ........................................... SUCCESS [ 19.858 s]
  24. [INFO] hudi-java-client ................................... SUCCESS [  3.221 s]
  25. [INFO] hudi-flink-client .................................. SUCCESS [  5.731 s]
  26. [INFO] hudi-spark3_2.12 ................................... SUCCESS [  8.627 s]
  27. [INFO] hudi-dla-sync ...................................... SUCCESS [  1.459 s]
  28. [INFO] hudi-sync .......................................... SUCCESS [  0.053 s]
  29. [INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [  5.652 s]
  30. [INFO] hudi-hive-sync-bundle .............................. SUCCESS [  1.623 s]
  31. [INFO] hudi-spark-bundle_2.11 ............................. SUCCESS [ 10.930 s]
  32. [INFO] hudi-presto-bundle ................................. SUCCESS [  3.652 s]
  33. [INFO] hudi-timeline-server-bundle ........................ SUCCESS [  4.804 s]
  34. [INFO] hudi-trino-bundle .................................. SUCCESS [  5.991 s]
  35. [INFO] hudi-hadoop-docker ................................. SUCCESS [  2.061 s]
  36. [INFO] hudi-hadoop-base-docker ............................ SUCCESS [ 53.372 s]
  37. [INFO] hudi-hadoop-base-java11-docker ..................... SUCCESS [ 48.545 s]
  38. [INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [  6.098 s]
  39. [INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [  4.825 s]
  40. [INFO] hudi-hadoop-history-docker ......................... SUCCESS [  3.829 s]
  41. [INFO] hudi-hadoop-hive-docker ............................ SUCCESS [ 52.660 s]
  42. [INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [01:02 min]
  43. [INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [ 12.661 s]
  44. [INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [  4.350 s]
  45. [INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [ 59.083 s]
  46. [INFO] hudi-hadoop-presto-docker .......................... SUCCESS [01:31 min]
  47. [INFO] hudi-hadoop-trinobase-docker ....................... SUCCESS [02:40 min]
  48. [INFO] hudi-hadoop-trinocoordinator-docker ................ SUCCESS [ 14.003 s]
  49. [INFO] hudi-hadoop-trinoworker-docker ..................... SUCCESS [ 12.100 s]
  50. [INFO] hudi-integ-test .................................... SUCCESS [ 13.581 s]
  51. [INFO] hudi-integ-test-bundle ............................. SUCCESS [ 27.212 s]
  52. [INFO] hudi-examples ...................................... SUCCESS [  8.090 s]
  53. [INFO] hudi-flink_2.11 .................................... SUCCESS [  4.217 s]
  54. [INFO] hudi-kafka-connect ................................. SUCCESS [  2.966 s]
  55. [INFO] hudi-flink-bundle_2.11 ............................. SUCCESS [ 11.155 s]
  56. [INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 12.369 s]
  57. [INFO] ------------------------------------------------------------------------
  58. [INFO] BUILD SUCCESS
  59. [INFO] ------------------------------------------------------------------------
  60. [INFO] Total time:  14:35 min
  61. [INFO] Finished at: 2022-01-12T18:41:27-08:00
  62. [INFO] ------------------------------------------------------------------------