Roadmap

Native Format IO

Integrate native Parquet & ORC reader & writer.

Deletion Vectors (Merge On Write)

  1. Primary Key Table Deletion Vectors Mode supports async compaction.
  2. Append Table supports DELETE & UPDATE with Deletion Vectors Mode. (Now only Spark SQL)
  3. Optimize lookup performance for HDD disk.

Support Flink Custom Data Distribution Lookup Join to reach large-scale data lookup join.

Produce Iceberg snapshots

Introduce a mode to produce Iceberg snapshots.

Branch

Branch production ready.

Changelog life cycle decouple

Changelog life cycle decouple supports none changelog-producer.

Partition Mark Done

Support partition mark done.

Default File Format

  • Default compression is ZSTD with level 1.
  • Parquet supports filter push down.
  • Parquet supports arrow with row type element.
  • Parquet becomes default file format.

Variant Type

Support Variant Type with Spark 4.0 and Flink 2.0. Unlocking support for semi-structured data.

Bucketed Join

Support Bucketed Join with Spark SQL to reduce shuffler in Join.

File Index

Add more index:

  1. Bitmap
  2. Inverse

Column Family

Support Column Family for super Wide Table.

View & Function support

Paimon Catalog supports views and functions.

Files Schema Evolution Ingestion

Introduce a files Ingestion with Schema Evolution.

Foreign Key Join

Explore Foreign Key Join solution.