# Technical Documentation: apache/iceberg

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/apache/iceberg)

### Overview (`README.md`)

[](https://github.com/apache/iceberg/actions/workflows/java-ci.yml)
[](https://apache-iceberg.slack.com/)

Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

Background and documentation is available at <https://iceberg.apache.org>


## Status

Iceberg is under active development at the Apache Software Foundation.

The [Iceberg format specification][iceberg-spec] is stable and new features are added with each version.

The core Java library is located in this repository and is the reference implementation for other libraries.

[Documentation][iceberg-docs] is available for all libraries and integrations.

[iceberg-docs]: https://iceberg.apache.org/docs/latest/
[iceberg-spec]: https://iceberg.apache.org/spec/

## Collaboration

Iceberg tracks issues in GitHub and prefers to receive contributions as pull requests.

Community discussions happen primarily on the [dev mailing list][dev-list] or on specific issues.

[dev-list]: mailto:dev@iceberg.apache.org


### Building

Iceberg is built using Gradle with Java 17 or 21.

* To invoke a build and run tests: `./gradlew build`
* To skip tests: `./gradlew build -x test -x integrationTest`
* To fix code style for default versions: `./gradlew spotlessApply`
* To fix code style for all versions of Spark/Hive/Flink:`./gradlew spotlessApply -DallModules`

Iceberg table support is organized in library modules:

* `iceberg-common` contains utility classes used in other modules
* `iceberg-api` contains the public Iceberg API
* `iceberg-core` contains implementations of the Iceberg API and support for Avro data files, **this is what processing engines should depend on**
* `iceberg-parquet` is an optional module for working with tables backed by Parquet files
* `iceberg-arrow` is an optional module for reading Parquet into Arrow memory
* `iceberg-orc` is an optional module for working with tables backed by ORC files
* `iceberg-hive-metastore` is an implementation of Iceberg tables backed by the Hive metastore Thrift client
* `iceberg-data` is an optional module for working with tables directly from JVM applications

Iceberg also has modules for adding Iceberg support to processing engines:

* `iceberg-spark` is an implementation of Spark's Datasource V2 API for Iceberg with submodules for each Spark version (use [runtime jars](https://iceberg.apache.org/multi-engine-support/#runtime-jar) for a shaded version to avoid dependency conflicts)
* `iceberg-flink` contains classes for integrating with Apache Flink (use [iceberg-flink-runtime](https://iceberg.apache.org/multi-engine-support/#runtime-jar) for a shaded version)
* `iceberg-mr` contains an InputFormat and other classes for integrating with Apache Hive

---
**NOTE**

The tests require Docker to execute. On macOS (with Docker Desktop), you might need to create a symbolic name to the docker socket in order to be detected by the tests:

```
sudo ln -s $HOME/.docker/run/docker.sock /var/run/docker.sock
```

In some cases the testcontainer may exit with an initialization error because of an illegal state exception in the GenericContainer.  One work around for this problem is to set `selinux` into permissive mode before running the tests. 

```
sudo setenforce Permissive
./gradlew ...
sudo setenforce Enforcing
```

---

#### Documentation

For information about building the documentation, see [here](site/README.md).

### Engine Compatibility

See the [Multi-Engine Support](https://iceberg.apache.org/multi-engine-support/) page to know about Iceberg compatibility with different Spark, Flink and Hive versions.
For other engines such as Presto or Trino, please visit their websites for Iceberg integration details.

### Implementations

This repository contains the Java implementation of Iceberg. Other implementations can be found at:

* **Go**: [iceberg-go](https://github.com/apache/iceberg-go)
* **PyIceberg** (Python): [iceberg-python](https://github.com/apache/iceberg-python)
* **Rust**: [iceberg-rust](https://github.com/apache/iceberg-rust)
* **C++**: [iceberg-cpp](https://github.com/apache/iceberg-cpp)

---

### Overview (`open-api/README.md`)

# Open API spec

The `rest-catalog-open-api.yaml` defines the REST catalog interface.

## Lint

To make sure that the open-api definition is valid, you can run the `lint` command:

```sh
make install
make lint
```

## Generate Python code

When updating `rest-catalog-open-api.yaml`, make sure to update `rest-catalog-open-api.py` with the spec changes by running the following commands:

```sh
make install
make generate
```

The generated code is not being used in the project, but helps to see what the changes in the open-API definition are in the generated code.

# REST Compatibility Kit (RCK)

The REST Compatibility Kit (RCK) is a Technology Compatibility Kit (TCK) implementation for the 
Iceberg REST Specification.  This includes a series of tests based on the Java reference
implementation of the REST Catalog that can be executed against any REST server that implements the
spec.

## Test Configuration

The RCK can be configured using either environment variables or java system properties and allows
for configuring both the tests and the REST client.  Environment variables prefixed by `CATALOG_`
are passed through the catalog configuring with the following mutations:

1. The `CATALOG_` prefix is stripped from the key name
2. Single underscore (`_`) is replaced with a dot (`.`)
3. Double underscore (`__`) is replaced with a dash (`-`)
4. The key names are converted to lowercase

A basic environment configuration would look like the following:

```shell
CATALOG_URI=https://my_rest_server.io/                    ## -> uri=https://my_rest_server.io/
CATALOG_WAREHOUSE=test_warehouse                          ## -> warehouse=test_warehouse
CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO       ## -> io-impl=org.apache.iceberg.aws.s3.S3FileIO
CATALOG_CREDENTIAL=<oauth_key>:<oauth_secret>             ## -> credential=<oauth_key>:<oauth_secret>
```

Java properties passed to the test must be prefixed with `rck.`, which can be used to configure some
test configurations described below and any catalog client properties.

An example of the same configuration using java system properties would look like the following:
```shell
rck.uri=https://my_rest_server.io/                    ## -> uri=https://my_rest_server.io/
rck.warehouse=test_warehouse                          ## -> warehouse=test_warehouse
rck.io-impl=org.apache.iceberg.aws.s3.S3FileIO        ## -> io-impl=org.apache.iceberg.aws.s3.S3FileIO
rck.credential=<oauth_key>:<oauth_secret>             ## -> credential=<oauth_key>:<oauth_secret>
```

Some test behaviors are configurable depending on the catalog implementations.  Not all behaviors
are strictly defined by the REST Specification.  The following are currently configurable:

| config                           | default |
|----------------------------------|---------|
| rck.requires-namespace-create    | true    |
| rck.supports-serverside-retry    | true    |
| rck.overrides-requested-location | false   |
| rck.supports-names-with-dot      | false   |
| rck.supports-variant             | false   |


## Running Compatibility Tests

The compatibility tests can be invoked via gradle with the following:

Note:  The default behavior is to run a local http server with a jdbc backend for testing purposes,
so `-Drck.local=false` must be set to point to an external REST server.

```shell
./gradlew :iceberg-open-api:test --tests RESTCompatibilityKitSuite \
 -Drck.local=false \
 -Drck.requires-namespace-create=true \
 -Drck.uri=https://my_rest_server.io/ \
 -Drck.warehouse=test_warehouse \
 -Drck.credential=<oauth_key>:<oauth_secret>
```

---

### Overview (`docker/iceberg-flink-quickstart/README.md`)

# Iceberg Flink Quickstart Docker Image

A pre-configured Apache Flink image with Apache Iceberg dependencies for quickly getting started with Iceberg on Flink.

See the [Flink quickstart documentation](https://iceberg.apache.org/flink-quickstart/) for details.

## Overview

This Docker image extends the official Apache Flink image to include:

- Iceberg Flink runtime
- Iceberg AWS bundle for S3/Glue support
- Minimal Hadoop dependencies necessary for Flink

## Build Arguments

The following build arguments can be customized when building the image:

| Argument | Default | Description |
|----------|---------|-------------|
| `FLINK_VERSION` | `2.0` | Apache Flink version |
| `ICEBERG_FLINK_RUNTIME_VERSION` | `2.0` | Iceberg Flink runtime version |
| `ICEBERG_VERSION` | `1.10.1` | Apache Iceberg version |
| `HADOOP_VERSION` | `3.4.2` | Apache Hadoop version |

## Building Locally

To build the image locally with default versions:

```bash
docker build -t apache/iceberg-flink-quickstart docker/iceberg-flink-quickstart/
```

To build with custom versions:

```bash
docker build \
  --build-arg FLINK_VERSION=2.0 \
  --build-arg ICEBERG_VERSION=1.10.1 \
  -t apache/iceberg-flink-quickstart \
  docker/iceberg-flink-quickstart/
```

## Usage

See the [Flink quickstart documentation](https://iceberg.apache.org/flink-quickstart/) for details.


## Test Script

A test script (`test.sql`) is provided to validate the Iceberg-Flink integration and future changes to the Docker image. 

Start up the Docker containers:

```sh
docker compose -f docker/iceberg-flink-quickstart/docker-compose.yml up -d --build
```

Execute the test script directly from the host:

```bash
docker exec -i jobmanager ./bin/sql-client.sh < docker/iceberg-flink-quickstart/test.sql
```

**Expected behavior:**
- Exit code: 0 (success)
- Creates: 1 catalog (`iceberg_catalog`), 1 database (`nyc`), 1 table (`taxis`)
- Inserts: 4 records
- Final state: Table `iceberg_catalog.nyc.taxis` contains 4 rows

To stop the stack:

```bash
docker compose -f docker/iceberg-flink-quickstart/docker-compose.yml down
```

---

### Overview (`docker/iceberg-rest-fixture/README.md`)

# Iceberg REST Catalog Adapter Test Fixture

For converting different catalog implementations into a rest one.
Adapter for wrapping the existing catalog backends over REST.


## Configuration

All configuration is provided via environment variables.

### Backend catalog properties

Catalog properties can be set via `CATALOG_*` environment variables. The
`CATALOG_` prefix is stripped; single underscores become dots (`.`); double
underscores become dashes (`-`). Names are lowercased.

| Env var | Catalog property |
|---|---|
| `CATALOG_CATALOG_NAME` | `catalog.name` |
| `CATALOG_WAREHOUSE` | `warehouse` |
| `CATALOG_URI` | `uri` |
| `CATALOG_CATALOG__IMPL` | `catalog-impl` |
| `CATALOG_IO__IMPL` | `io-impl` |
| `CATALOG_JDBC_USER` | `jdbc.user` |

If `catalog-impl` and `uri` are unset, the fixture defaults to a SQLite
`JdbcCatalog` at `/tmp/iceberg_catalog.db` opened in WAL mode
(`jdbc:sqlite:/tmp/iceberg_catalog.db?journal_mode=WAL`), so concurrent
clients don't fail with `SQLITE_BUSY`.

Keep the `?journal_mode=WAL` suffix if you override `CATALOG_URI` with another
SQLite path.

### Catalog name

By default, the fixture serves a catalog named `rest_backend`. To match a
name expected by a specific engine (for example, a catalog created via Trino
or PyIceberg), override the `catalog.name` property:

```bash
docker run -e CATALOG_CATALOG_NAME=mycatalog -p 8181:8181 apache/iceberg-rest-fixture
```


### Logging

By default, the fixture logs at `INFO`. To reduce log output, set `LOG_LEVEL`
to another slf4j-simple level such as `WARN`, `ERROR`, or `OFF`:

```bash
docker run -e LOG_LEVEL=WARN -p 8181:8181 apache/iceberg-rest-fixture
```

To use a full slf4j-simple properties file, mount a directory containing
`simplelogger.properties` and set `LOG_CONFIG_DIR` to that directory:

```bash
docker run \
  -v "$PWD/simplelogger.properties:/etc/iceberg-rest/simplelogger.properties:ro" \
  -e LOG_CONFIG_DIR=/etc/iceberg-rest \
  -p 8181:8181 \
  apache/iceberg-rest-fixture
```

If both `LOG_LEVEL` and `LOG_CONFIG_DIR` are set, `LOG_LEVEL` overrides the
default log level from `simplelogger.properties`; the properties file can still
configure other slf4j-simple settings.

## Build the Docker Image

When making changes to the local files and test them out, you can build the image locally:

```bash
# Build the project from iceberg root directory
./gradlew :iceberg-open-api:shadowJar

# Rebuild the docker image
docker image rm -f apache/iceberg-rest-fixture && docker build -t apache/iceberg-rest-fixture -f docker/iceberg-rest-fixture/Dockerfile .
```

## Browse

To browse the catalog, you can use `pyiceberg`:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

--- METRICS ---
- Files Extracted: 4
- Estimated Token Budget: ~3222 tokens
- Canonical Reference: https://codewiki.google/github.com/apache/iceberg