Python/Rust Quick Start (Hudi-rs)

This guide will help you get started with hudi-rs, a native Rust library for Apache Hudi with Python bindings. Learn how to install, set up, and perform basic operations using both Python and Rust interfaces.

Installation

  1. # Python
  2. pip install hudi
  4. # Rust
  5. cargo add hudi

Basic Usage

Python/Rust Quick Start (Hudi-rs) - 图1note

Currently, write capabilities and reading from MOR tables are not supported.

The examples below expect a Hudi table exists at /tmp/trips_table, created using the quick start guide.

Python Example

  1. from hudi import HudiTableBuilder
  2. import pyarrow as pa
  4. hudi_table = (
  5.     HudiTableBuilder
  6.     .from_base_uri("/tmp/trips_table")
  7.     .build()
  8. )
  10. # Read with partition filters
  11. records = hudi_table.read_snapshot(filters=[("city", "=", "san_francisco")])
  13. # Convert to PyArrow table
  14. arrow_table = pa.Table.from_batches(records)
  15. result = arrow_table.select(["rider", "city", "ts", "fare"])

Rust Example (with DataFusion)

  1. Set up your project:
  1. cargo new my_project --bin && cd my_project
  2. cargo add tokio@1 datafusion@42
  3. cargo add hudi --features datafusion
  1. Add code to src/main.rs:
  1. use std::sync::Arc;
  2. use datafusion::error::Result;
  3. use datafusion::prelude::{DataFrame, SessionContext};
  4. use hudi::HudiDataSource;
  6. #[tokio::main]
  7. async fn main() -> Result<()> {
  8.     let ctx = SessionContext::new();
  9.     let hudi = HudiDataSource::new_with_options("/tmp/trips_table", []).await?;
  10.     ctx.register_table("trips_table", Arc::new(hudi))?;
  11.     // Read with partition filters
  12.     let df: DataFrame = ctx.sql("SELECT * from trips_table where city = 'san_francisco'").await?;
  13.     df.show().await?;
  14.     Ok(())
  15. }

Cloud Storage Integration

Python

  1. from hudi import HudiTableBuilder
  3. hudi_table = (
  4.     HudiTableBuilder
  5.     .from_base_uri("s3://bucket/trips_table")
  6.     .with_option("aws_region", "us-west-2")
  7.     .build()
  8. )

Rust

  1. use hudi::HudiDataSource;
  3. let hudi = HudiDataSource::new_with_options(
  4.     "s3://bucket/trips_table",
  5.     [("aws_region", "us-west-2")]
  6. ).await?;

Supported Cloud Storage

  • AWS S3 (s3://)
  • Azure Storage (az://)
  • Google Cloud Storage (gs://)

Set appropriate environment variables (AWS_*, AZURE_*, or GOOGLE_*) for authentication, or pass through the option() API.

Read with Timestamp

Add timestamp option for time-travel queries:

  1. .with_option("hoodie.read.as.of.timestamp", "20241122010827898")