Iceberg

Overview

Apache Iceberg is a high-performance format for huge analytic tables.

Version

Dependencies

<dependency>
    <groupId>org.apache.inlong</groupId>
    <artifactId>sort-connector-iceberg</artifactId>
    <version>2.1.0-SNAPSHOT</version>
</dependency>

Usage

Before creating the Iceberg task, we need a Flink environment integrated with Hadoop.

• Download Apache Hadoop.
• Modify jobmanager.sh and taskmanager.sh and add Hadoop environment variables. For commands, please refer to Apache Flink.

export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`

• Modify docker-compose.yml in the docker/docker-compose and mount Hadoop and Flink startup commands into the container:

  jobmanager:
    image: apache/flink:1.15-scala_2.12
    container_name: jobmanager
    user: root
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
    volumes:
      # Mount Hadoop
      - HADOOP_HOME:HADOOP_HOME
      # Mount the modified jobmanager.sh which adds the HADOOP_HOME env correctly
      - /jobmanager.sh:/opt/flink/bin/jobmanager.sh
    ports:
      - "8081:8081"
    command: jobmanager
  taskmanager:
    image: apache/flink:1.15-scala_2.12
    container_name: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2
    volumes:
      # Mount Hadoop
      - HADOOP_HOME:HADOOP_HOME
      # Mount the modified taskmanager.sh which adds the HADOOP_HOME env correctly
      - /taskmanager.sh:/opt/flink/bin/taskmanager.sh
    command: taskmanager

Flink SQL API

Before using Flink sql client, sql-client.sh also needs to add Hadoop environment variables and mounted to the container. For commands, please refer to Apache Flink.

export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`

使用 Flink sql cli：

CREATE TABLE `iceberg_table_source`(
    PRIMARY KEY (`_id`) NOT ENFORCED,
    `_id` STRING,
    `id` INT,
    `name` STRING,
    `age` INT)
    WITH (
    'connector' = 'iceberg-inlong',
    'catalog-database' = 'DATABASES',
    'catalog-table' = 'TABLE',
    'catalog-type' = 'HIVE',
    'catalog-name' = 'HIVE',
    'streaming' = 'true',
    'uri' = 'thrift://127.0.0.1:9083'
);

Dashboard

Source → Create → Iceberg

Manager Client

TODO

Options

Data Type Mapping