iceberg Documentation

Technical Documentation: apache/iceberg

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

Background and documentation is available at https://iceberg.apache.org

Status Copy Chapter

Iceberg is under active development at the Apache Software Foundation.

The Iceberg format specification is stable and new features are added with each version.

The core Java library is located in this repository and is the reference implementation for other libraries.

Documentation is available for all libraries and integrations.

Collaboration Copy Chapter

Iceberg tracks issues in GitHub and prefers to receive contributions as pull requests.

Community discussions happen primarily on the dev mailing list or on specific issues.

Building Copy Section

Iceberg is built using Gradle with Java 17 or 21.

• To invoke a build and run tests: ./gradlew build
• To skip tests: ./gradlew build -x test -x integrationTest
• To fix code style for default versions: ./gradlew spotlessApply
• To fix code style for all versions of Spark/Hive/Flink:./gradlew spotlessApply -DallModules

Iceberg table support is organized in library modules:

• iceberg-common contains utility classes used in other modules
• iceberg-api contains the public Iceberg API
• iceberg-core contains implementations of the Iceberg API and support for Avro data files, this is what processing engines should depend on
• iceberg-parquet is an optional module for working with tables backed by Parquet files
• iceberg-arrow is an optional module for reading Parquet into Arrow memory
• iceberg-orc is an optional module for working with tables backed by ORC files
• iceberg-hive-metastore is an implementation of Iceberg tables backed by the Hive metastore Thrift client
• iceberg-data is an optional module for working with tables directly from JVM applications

Iceberg also has modules for adding Iceberg support to processing engines:

• iceberg-spark is an implementation of Spark's Datasource V2 API for Iceberg with submodules for each Spark version (use runtime jars for a shaded version to avoid dependency conflicts)
• iceberg-flink contains classes for integrating with Apache Flink (use iceberg-flink-runtime for a shaded version)
• iceberg-mr contains an InputFormat and other classes for integrating with Apache Hive

NOTE

The tests require Docker to execute. On macOS (with Docker Desktop), you might need to create a symbolic name to the docker socket in order to be detected by the tests:

sudo ln -s $HOME/.docker/run/docker.sock /var/run/docker.sock

In some cases the testcontainer may exit with an initialization error because of an illegal state exception in the GenericContainer. One work around for this problem is to set selinux into permissive mode before running the tests.

sudo setenforce Permissive
./gradlew ...
sudo setenforce Enforcing

Documentation

For information about building the documentation, see here.

Engine Compatibility Copy Section

See the Multi-Engine Support page to know about Iceberg compatibility with different Spark, Flink and Hive versions.
For other engines such as Presto or Trino, please visit their websites for Iceberg integration details.

Implementations Copy Section

This repository contains the Java implementation of Iceberg. Other implementations can be found at:

• Go: iceberg-go
• PyIceberg (Python): iceberg-python
• Rust: iceberg-rust
• C++: iceberg-cpp

Overview (`open-api/README.md`) Copy Section

Open API spec

The rest-catalog-open-api.yaml defines the REST catalog interface.

Lint Copy Chapter

To make sure that the open-api definition is valid, you can run the lint command:

make install
make lint

Generate Python code Copy Chapter

When updating rest-catalog-open-api.yaml, make sure to update rest-catalog-open-api.py with the spec changes by running the following commands:

make install
make generate

The generated code is not being used in the project, but helps to see what the changes in the open-API definition are in the generated code.

REST Compatibility Kit (RCK)

The REST Compatibility Kit (RCK) is a Technology Compatibility Kit (TCK) implementation for the
Iceberg REST Specification. This includes a series of tests based on the Java reference
implementation of the REST Catalog that can be executed against any REST server that implements the
spec.

Test Configuration Copy Chapter

The RCK can be configured using either environment variables or java system properties and allows
for configuring both the tests and the REST client. Environment variables prefixed by CATALOG_
are passed through the catalog configuring with the following mutations:

1. The CATALOG_ prefix is stripped from the key name
2. Single underscore (_) is replaced with a dot (.)
3. Double underscore (__) is replaced with a dash (-)
4. The key names are converted to lowercase

A basic environment configuration would look like the following:

CATALOG_URI=https://my_rest_server.io/                    ## -> uri=https://my_rest_server.io/
CATALOG_WAREHOUSE=test_warehouse                          ## -> warehouse=test_warehouse
CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO       ## -> io-impl=org.apache.iceberg.aws.s3.S3FileIO
CATALOG_CREDENTIAL=<oauth_key>:<oauth_secret>             ## -> credential=<oauth_key>:<oauth_secret>

Java properties passed to the test must be prefixed with rck., which can be used to configure some
test configurations described below and any catalog client properties.

An example of the same configuration using java system properties would look like the following:

rck.uri=https://my_rest_server.io/                    ## -> uri=https://my_rest_server.io/
rck.warehouse=test_warehouse                          ## -> warehouse=test_warehouse
rck.io-impl=org.apache.iceberg.aws.s3.S3FileIO        ## -> io-impl=org.apache.iceberg.aws.s3.S3FileIO
rck.credential=<oauth_key>:<oauth_secret>             ## -> credential=<oauth_key>:<oauth_secret>

Some test behaviors are configurable depending on the catalog implementations. Not all behaviors
are strictly defined by the REST Specification. The following are currently configurable:

Running Compatibility Tests Copy Chapter

The compatibility tests can be invoked via gradle with the following:

Note: The default behavior is to run a local http server with a jdbc backend for testing purposes,
so -Drck.local=false must be set to point to an external REST server.

./gradlew :iceberg-open-api:test --tests RESTCompatibilityKitSuite \
 -Drck.local=false \
 -Drck.requires-namespace-create=true \
 -Drck.uri=https://my_rest_server.io/ \
 -Drck.warehouse=test_warehouse \
 -Drck.credential=<oauth_key>:<oauth_secret>

Overview (`docker/iceberg-flink-quickstart/README.md`) Copy Section

Iceberg Flink Quickstart Docker Image

A pre-configured Apache Flink image with Apache Iceberg dependencies for quickly getting started with Iceberg on Flink.

See the Flink quickstart documentation for details.

Overview Copy Chapter

This Docker image extends the official Apache Flink image to include:

• Iceberg Flink runtime
• Iceberg AWS bundle for S3/Glue support
• Minimal Hadoop dependencies necessary for Flink

Build Arguments Copy Chapter

The following build arguments can be customized when building the image:

Building Locally Copy Chapter

To build the image locally with default versions:

docker build -t apache/iceberg-flink-quickstart docker/iceberg-flink-quickstart/

To build with custom versions:

docker build \
  --build-arg FLINK_VERSION=2.0 \
  --build-arg ICEBERG_VERSION=1.10.1 \
  -t apache/iceberg-flink-quickstart \
  docker/iceberg-flink-quickstart/

Usage Copy Chapter

See the Flink quickstart documentation for details.

Test Script Copy Chapter

A test script (test.sql) is provided to validate the Iceberg-Flink integration and future changes to the Docker image.

Start up the Docker containers:

docker compose -f docker/iceberg-flink-quickstart/docker-compose.yml up -d --build

Execute the test script directly from the host:

docker exec -i jobmanager ./bin/sql-client.sh < docker/iceberg-flink-quickstart/test.sql

Expected behavior:

• Exit code: 0 (success)
• Creates: 1 catalog (iceberg_catalog), 1 database (nyc), 1 table (taxis)
• Inserts: 4 records
• Final state: Table iceberg_catalog.nyc.taxis contains 4 rows

To stop the stack:

docker compose -f docker/iceberg-flink-quickstart/docker-compose.yml down

Overview (`docker/iceberg-rest-fixture/README.md`) Copy Section

Iceberg REST Catalog Adapter Test Fixture

For converting different catalog implementations into a rest one.
Adapter for wrapping the existing catalog backends over REST.

Configuration Copy Chapter

All configuration is provided via environment variables.

Backend catalog properties Copy Section

Catalog properties can be set via CATALOG_* environment variables. The
CATALOG_ prefix is stripped; single underscores become dots (.); double
underscores become dashes (-). Names are lowercased.

If catalog-impl and uri are unset, the fixture defaults to a SQLite
JdbcCatalog at /tmp/iceberg_catalog.db opened in WAL mode
(jdbc:sqlite:/tmp/iceberg_catalog.db?journal_mode=WAL), so concurrent
clients don't fail with SQLITE_BUSY.

Keep the ?journal_mode=WAL suffix if you override CATALOG_URI with another
SQLite path.

Catalog name Copy Section

By default, the fixture serves a catalog named rest_backend. To match a
name expected by a specific engine (for example, a catalog created via Trino
or PyIceberg), override the catalog.name property:

docker run -e CATALOG_CATALOG_NAME=mycatalog -p 8181:8181 apache/iceberg-rest-fixture

Logging Copy Section

By default, the fixture logs at INFO. To reduce log output, set LOG_LEVEL
to another slf4j-simple level such as WARN, ERROR, or OFF:

docker run -e LOG_LEVEL=WARN -p 8181:8181 apache/iceberg-rest-fixture

To use a full slf4j-simple properties file, mount a directory containing
simplelogger.properties and set LOG_CONFIG_DIR to that directory:

docker run \
  -v "$PWD/simplelogger.properties:/etc/iceberg-rest/simplelogger.properties:ro" \
  -e LOG_CONFIG_DIR=/etc/iceberg-rest \
  -p 8181:8181 \
  apache/iceberg-rest-fixture

If both LOG_LEVEL and LOG_CONFIG_DIR are set, LOG_LEVEL overrides the
default log level from simplelogger.properties; the properties file can still
configure other slf4j-simple settings.

Build the Docker Image Copy Chapter

When making changes to the local files and test them out, you can build the image locally:

# Build the project from iceberg root directory
./gradlew :iceberg-open-api:shadowJar

# Rebuild the docker image
docker image rm -f apache/iceberg-rest-fixture && docker build -t apache/iceberg-rest-fixture -f docker/iceberg-rest-fixture/Dockerfile .

Browse Copy Chapter

To browse the catalog, you can use pyiceberg:

/* Detailed source-code truncated for AI context efficiency. */

--- METRICS ---

• Files Extracted: 4
• Estimated Token Budget: ~3222 tokens
• Canonical Reference: https://codewiki.google/github.com/apache/iceberg