Reading & Writing Hive Tables

Using the HiveCatalog and Flink’s connector to Hive, Flink can read and write from Hive data as an alternative to Hive’s batch engine. Be sure to follow the instructions to include the correct dependencies in your application.

Reading From Hive

Assume Hive contains a single table in its default database, named people that contains several rows.

  1. hive> show databases;
  2. OK
  3. default
  4. Time taken: 0.841 seconds, Fetched: 1 row(s)
  6. hive> show tables;
  7. OK
  8. Time taken: 0.087 seconds
  10. hive> CREATE TABLE mytable(name string, value double);
  11. OK
  12. Time taken: 0.127 seconds
  14. hive> SELECT * FROM mytable;
  15. OK
  16. Tom   4.72
  17. John  8.0
  18. Tom   24.2
  19. Bob   3.14
  20. Bob   4.72
  21. Tom   34.9
  22. Mary  4.79
  23. Tiff  2.72
  24. Bill  4.33
  25. Mary  77.7
  26. Time taken: 0.097 seconds, Fetched: 10 row(s)

With the data ready your can connect to Hive connect to an existing Hive installation and begin querying.

  1. Flink SQL> show catalogs;
  2. myhive
  3. default_catalog
  5. # ------ Set the current catalog to be 'myhive' catalog if you haven't set it in the yaml file ------
  7. Flink SQL> use catalog myhive;
  9. # ------ See all registered database in catalog 'mytable' ------
  11. Flink SQL> show databases;
  12. default
  14. # ------ See the previously registered table 'mytable' ------
  16. Flink SQL> show tables;
  17. mytable
  19. # ------ The table schema that Flink sees is the same that we created in Hive, two columns - name as string and value as double ------ 
  20. Flink SQL> describe mytable;
  21. root
  22.  |-- name: name
  23.  |-- type: STRING
  24.  |-- name: value
  25.  |-- type: DOUBLE
  28. Flink SQL> SELECT * FROM mytable;
  30.    name      value
  31. __________ __________
  33.     Tom      4.72
  34.     John     8.0
  35.     Tom      24.2
  36.     Bob      3.14
  37.     Bob      4.72
  38.     Tom      34.9
  39.     Mary     4.79
  40.     Tiff     2.72
  41.     Bill     4.33
  42.     Mary     77.7

Writing To Hive

Similarly, data can be written into hive using an INSERT INTO clause.

  1. Flink SQL> INSERT INTO mytable (name, value) VALUES ('Tom', 4.72);

Limitations

The following is a list of major limitations of the Hive connector. And we’re actively working to close these gaps.

  • INSERT OVERWRITE is not supported.
  • Inserting into partitioned tables is not supported.
  • ACID tables are not supported.
  • Bucketed tables are not supported.
  • Some data types are not supported. See the limitations for details.
  • Only a limited number of table storage formats have been tested, namely text, SequenceFile, ORC, and Parquet.
  • Views are not supported.