# Technical Documentation: linkedin/ambry

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/linkedin/ambry)

### Overview (`README.md`)

# Ambry

[](https://github.com/linkedin/ambry/actions/workflows/github-actions.yml)
[](https://codecov.io/github/linkedin/ambry)
[](LICENSE)

Ambry is a distributed object store that supports storage of trillions of small immutable objects (50K -100K) as well as billions of large objects. It was specifically designed to store and serve media objects in web companies. However, it can be used as a general purpose storage system to store DB backups, search indexes or business reports. The system has the following characterisitics: 

1. Highly available and horizontally scalable
2. Low latency and high throughput
3. Optimized for both small and large objects
4. Cost effective
5. Easy to use

Requires JDK version 1.11 - 1.14.

## Documentation
Detailed documentation is available at https://github.com/linkedin/ambry/wiki

## Research
Paper introducing Ambry at [SIGMOD 2016](http://sigmod2016.org/) -> http://dprg.cs.uiuc.edu/data/files/2016/ambry.pdf

Reach out to us at ambrydev@googlegroups.com if you would like us to list a paper that is based off of research on Ambry.

## Getting Started
##### Step 1: Download the code, build it and prepare for deployment.
To get the latest code and build it, do

    $ git clone https://github.com/linkedin/ambry.git 
    $ cd ambry
    $ ./gradlew allJar
    $ cd target
    $ mkdir logs
Ambry uses files that provide information about the cluster to route requests from the frontend to servers and for replication between servers. We will use a simple clustermap that contains a single server with one partition. The partition will use `/tmp` as the mount point.
##### Step 2: Deploy a server.
    $ nohup java -Dlog4j2.configurationFile=file:../config/log4j2.xml -jar ambry.jar --serverPropsFilePath ../config/server.properties --hardwareLayoutFilePath ../config/HardwareLayout.json --partitionLayoutFilePath ../config/PartitionLayout.json > logs/server.log &

Through this command, we configure the log4j properties, provide the server with configuration options and cluster definitions and redirect output to a log. Note down the process ID returned (`serverProcessID`) because it will be needed for shutdown.  
The log will be available at `logs/server.log`. Alternately, you can change the log4j properties to write the log messages to a file instead of standard output.
##### Step 3: Deploy a frontend.
    $ nohup java -Dlog4j2.configurationFile=file:../config/log4j2.xml -cp "*" com.github.ambry.frontend.AmbryFrontendMain --serverPropsFilePath ../config/frontend.properties --hardwareLayoutFilePath ../config/HardwareLayout.json --partitionLayoutFilePath ../config/PartitionLayout.json > logs/frontend.log &

Note down the process ID returned (`frontendProcessID`) because it will be needed for shutdown. Make sure that the frontend is ready to receive requests.

    $ curl http://localhost:1174/healthCheck
    GOOD
The log will be available at `logs/frontend.log`. Alternately, you can change the log4j properties to write the log messages to a file instead of standard output.
##### Step 4: Interact with Ambry !
We are now ready to store and retrieve data from Ambry. Let us start by storing a simple image. For demonstration purposes, we will use an image `demo.gif` that has been copied into the `target` folder.
###### POST
    $ curl -i -H "x-ambry-service-id:CUrlUpload"  -H "x-ambry-owner-id:`whoami`" -H "x-ambry-content-type:image/gif" -H "x-ambry-um-description:Demonstration Image" http://localhost:1174/ --data-binary @demo.gif
    HTTP/1.1 201 Created
    Location: AmbryID
    Content-Length: 0
The CUrl command creates a `POST` request that contains the binary data in demo.gif. Along with the file data, we provide headers that act as blob properties. These include the size of the blob, the service ID, the owner ID and the content type.  
In addition to these properties, Ambry also has a provision for arbitrary user defined metadata. We provide `x-ambry-um-description` as user metadata. Ambry does not interpret this data and it is purely for user annotation.
The `Location` header in the response is the blob ID of the blob we just uploaded.
###### GET - Blob Info
Now that we stored a blob, let us verify some properties of the blob we uploaded.

    $ curl -i http://localhost:1174/AmbryID/BlobInfo
    HTTP/1.1 200 OK
    x-ambry-blob-size: {Blob size}
    x-ambry-service-id: CUrlUpload
    x-ambry-creation-time: {Creation time}
    x-ambry-private: false
    x-ambry-content-type: image/gif
    x-ambry-owner-id: {username}
    x-ambry-um-desc: Demonstration Image
    Content-Length: 0
###### GET - Blob
Now that we have verified that Ambry returns properties correctly, let us obtain the actual blob.

    $ curl http://localhost:1174/AmbryID > demo-downloaded.gif
    $ diff demo.gif demo-downloaded.gif 
    $
This confirms that the data that was sent in the `POST` request matches what we received in the `GET`. If you would like to see the image, simply point your browser to `http://localhost:1174/AmbryID` and you should see the image that was uploaded !
###### DELETE
Ambry is an immutable store and blobs cannot be updated but they can be deleted in order to make them irretrievable. Let us go ahead and delete the blob we just created.

    $ curl -i -X DELETE http://localhost:1174/AmbryID
    HTTP/1.1 202 Accepted
    Content-Length: 0
You will no longer be able to retrieve the blob properties or data.

    $ curl -i http://localhost:1174/AmbryID/BlobInfo
    HTTP/1.1 410 Gone
    Content-Type: text/plain; charset=UTF-8
    Content-Length: 17
    Connection: close

    Failure: 410 Gone
##### Step 5: Stop the frontend and server.
    $ kill -15 frontendProcessID
    $ kill -15 serverProcessID
You can confirm that the services have been shut down by looking at the logs.
##### Additional information:
In addition to the simple APIs demonstrated above, Ambry provides support for `GET` of only user metadata and `HEAD`. In addition to the `POST` of binary data that was demonstrated, Ambry also supports `POST` of `multipart/form-data` via CUrl or web forms.
Other features of interest include:
* **Time To Live (TTL)**: During `POST`, a TTL in seconds can be provided through the addition of a header named `x-ambry-ttl`. This means that Ambry will stop serving the blob after the TTL has expired. On `GET`, expired blobs behave the same way as deleted blobs.
* **Private**: During `POST`, providing a header named `x-ambry-private` with the value `true` will mark the blob as private. API behavior can be configured based on whether a blob is public or private.

## Testing
>**WARNING**: Tests currently can take upwards of 40 minutes to run

Ambry requires [azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=npm%2Cblob-storage)
and MySQL for testing. To install on MacOS:

azurite:
```
$ npm install azurite
$ azurite
```

mysql:
```
$ brew install mysql
$ brew services start mysql
$ mysql -uroot
mysql> CREATE USER 'travis'@'localhost';
mysql> GRANT ALL PRIVILEGES ON *.* to 'travis'@'localhost';
mysql> FLUSH PRIVILEGES;
mysql> CREATE DATABASE AmbryRepairRequests;
mysql> USE AmbryRepairRequests; SOURCE ./ambry-mysql/src/main/resources/AmbryRepairRequests.ddl;
```

Then run `./gradlew build` to build and run all unit tests.

## Named-blob cleaner purge (review proposal)

Normal named-blob cleanup retains the latest version. The optional
`NamedBlobsCleanupRunner` constructor accepting a `Predicate<Container> purgeEligibility`
and `boolean purgeDryRun` can instead delete every unexpired, non-deleted mapping in
selected containers, including latest READY versions and recent IN_PROGRESS versions.
Existing constructors never enable purge. Container/account exclusions and disabled
named-blob mode still take precedence; container lifecycle status is not changed.

The caller owns the eligibility policy. It must require explicit per-container opt-in,
verify that the source data is no longer needed, and prevent source reads and writes
before returning `true`. For a migration, write routing alone is not proof of read
cutover or destination completeness. Keep this policy live so removing approval
acts as a kill switch; it is checked before every page and every blob deletion.
A check cannot cancel an already-issued deletion, so it does not replace fencing
source traffic. Start with `purgeDryRun=true`: this reports only the first bounded page
per eligible container, not a full-container count, and performs no blob or metadata
deletion for those containers. Other containers continue normal cleanup.

`MySqlNamedBlobDb` scans purge candidates in ascending primary-key order rather than
normal cleanup's descending version order. The last blob name remains an inclusive
cursor so a name with more versions than a page is revisited. Purge waits for metadata
soft-deletion before fetching another page and stops on deletion or metadata failure,
or if the next page repeats versions that should have been removed;
the next scheduled run restarts its purge sweep. Unlike normal cleanup, it never skips
a name after repeated query kills. Primary-key ordering avoids the mixed-order filesort,
but filtering already-deleted rows can still require scanning many index entries:
this is not a fixed query-latency guarantee or a fix for normal-cleanup scans.

Database implementations without purge support fail explicitly through the new default
`NamedBlobDb.pullStaleBlobs` overload. Blob deletion continues to use the supplied
router's existing offline `deleteBlob(blobId, serviceId)` path. This OSS change does not
guarantee primary-only routing in a composite router; that must be verified by its
integration. Migration-state/completeness checks, deployment configuration, composite
database delegation, and any secondary metadata cleanup are deliberately outside this
OSS proposal. Expired/already-deleted mappings remain under their existing cleanup
mechanisms; metadata soft-deletion does not itself immediately reclaim storage bytes.

`PurgePreviewBlobCount` counts dry-run candidates and `PurgedBlobCount` counts versions
processed after metadata cleanup succeeds. Failures use the existing container failure
counter and logs. These are cumulative counters, not distinct-blob inventories.

Focused validation (the integration tests require the local MySQL setup above and the
`AmbryNamedBlobs` schema from `ambry-named-mysql/src/main/resources/NamedBlobsSchema.ddl`):

```bash
./gradlew -PdisableShipkit :ambry-frontend:test --tests com.github.ambry.frontend.NamedBlobsCleanupRunnerTest \
  :ambry-named-mysql:test --tests com.github.ambry.named.MySqlNamedBlobDbTest
./gradlew -PdisableShipkit :ambry-named-mysql:intTest \
  --tests 'com.github.ambry.named.MySqlNamedBlobDbIntegrationTest.testPurge*'
```

---

### Overview (`ambry-benchmarks/README.md`)

# Ambry Benchmarks

JMH (Java Microbenchmark Harness) performance benchmarks for Ambry components.

## Running Benchmarks

```bash
### Run all benchmarks
./gradlew jmh

### Run specific benchmarks
./gradlew jmh -Pjmh.includes='.*java8.*'
```
## Output

Results saved to: `ambry-benchmarks/build/reports/jmh/results.txt`

---

### Overview (`ambry-cloud/README.md`)

# ambry-cloud

This package contains classes that enable a Virtual Cloud Replica (VCR)
to reflect partition-level replication changes to a remote cloud backup.
Currently Azure is supported; other vendors will be added later.

Components

* CloudBlobStore: a Store implementation that works with a CloudDestination
 to reflect blob operations to a cloud store.
* CloudDestination: interface for transmitting blob data and metadata between the
local cluster and a remote cloud storage engine.
* AzureCloudDestination: a CloudDestination implementation that stores blob data
in Azure Blob Storage, and metadata in Azure Cosmos DB.

---

### Overview (`ambry-tools/src/main/java/com/github/ambry/Readme.md`)

## Terminologies

HardwareLayout   - Has information about the nodes available in the cluster. disks for those nodes and their states
PartitionLayout  - Has information about the partitions and its replicas
Blobid           - BlobId of the blob/object which was generated by ambry on put


## Set up
To execute tools, ./gradlew allJar in the root directory.
This will generate ambry.jar in "target" folder under root directory.
This jar will assist using these tools.


## DumpLogTool, DumpIndexTool, DumpReplicaTokenTool and DumpDataTool
These tools are used to dump the data parsing the storage files.

#### Operations supported
* Dumping log segment
* Dumping index / segments
* Dumping replicaToken
* Compare index entries to log contents

## BlobValidator
This tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas.

#### Operations supported:
* Get blob from all replicas and compare the record across replicas for a list of blob ids.
* Get blob from all replicas in a datacenter and compare the record across replicas for a list of blob ids.
* Get blob from a specific replica and ensure that it deserializes correctly for a list of blob ids.

## ServerAdminTool
This tool can be used to operate on a single ambry server. Please refer to the config documentation and the class 
documentation for usage details.

#### Operations supported
* GetBlobProperties
* GetBlobUserMetadata
* GetBlob
* TriggerCompaction

#### Dump Index
This tool can be used to dump or verify whole indexes or index segments. The operations can also be done on a subset of blobs.
Please refer to the file for more details on what the operations are and the configuration supported.
#### Operations supported
* DumpIndex
* DumpIndexSegment
* VerifyIndex

### Dumping Log

#### Dump log file
```bash
java -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
file.to.read=[LogFile]
silent=false
```

#### Dump log file with throttling
```bash
java -cp ambry.jar com.github.ambry.store.DumpLogTool --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
file.to.read=[LogFile]
blobs.per.sec=100
silent=false
```

#### Dump log starting at offset x and ending at offset y
```bash
java -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
file.to.read=[LogFile]
log.start.offset=x
log.end.offset=y
silent=false
```

#### Dump log filtering with a set of blobs
```bash
java -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
file.to.read=[LogFile]
blobId.list=blobId1,blobId2,blobId3
silent=false
```

### Comparison operations

#### Comparing index entries to log entries(or in other words, trying to read a log record based on index info for
#### every entry in the index)
```bash
java -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
type.of.operation=CompareIndexToLog
file.to.read=[indexFile]
```

#### Comparing all index entries for a replica to log entries
```bash
java -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
type.of.operation=CompareReplicaIndexesToLog
replica.root.directory=[replicaRootDirecotry]
index.files.per.sec=1 // number of index files to process per sec
```

### Dumping ReplicaToken file
```bash
java -cp ambry.jar com.github.ambry.store.DumpReplicaTokenTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
file.to.read=[replicaTokenFile]
```

### BlobValidator
This tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas. Please
refer to the config documentation and the class  documentation for usage details.

```bash
java -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]
```

### ServerAdminTool
```bash
java -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]
```

### ConsistencyCheckerTool

#### Check for consistency among replicas for a partition
```bash
java -cp ambry.jar com.github.ambry.store.ConsistencyCheckerTool  --propsFile [Config file path]
//Contents of config file
hardware.layout.file.path=[HardwareLayoutFile]
partition.layout.file.path=[PartitionLayoutFile]
partition.root.directory=[Root directory which contains one directory per replica. Each replica's directory is expected 
to contain all index files for the respective replica. Diretory's name is considered to be replica name for logging 
purposes]
```


```
Root Directory for Partition should contains N sub-directories pertaining to N replicas. Each sub-directory's name
should be the same as name of the replica. Each sub directory(replica) in turn should have all the index segments within it.

A sample structure of a root Directory:

rootDir
  ela4-app8763.prod.linkedin.com
    0_index
    10264262199_index
    10821344563_index
  lva1-app6774.prod.linkedin.com
    0_index
    10267816698_index
    10892574781_index
  lva1-app6775.prod.linkedin.com
    0_index
    10266323380_index
    10888037904_index
```


###ConcurrencyTestTool
####Sample test run

Ensure that ambry servers in the HardwareLayout are up running

```bash
java -cp ambry.jar com.github.ambry.tools.admin.ConcurrencyTestTool --hardwareLayout [HardwareLayoutFile]
 --partitionLayout [PartitionLayoutFile] --putGetHelperFactory com.github.ambry.tools.admin.RouterPutGetHelperFactory
 --maxParallelPutCount 2 --parallelGetCount 4 --burstCountForGet 4 --maxGetCountPerBlob 10
 --totalPutBlobCount 1000 --maxBlobSizeInBytes 1000 --minBlobSizeInBytes 100 --routerPropsFilePath [Path to router props
 file] --deleteAndValidate true
```

### ServerPerformanceTool

Testing for GET blobs
This tool will read the log file for blobs and keep making get requests for each blob until time out
```bash
java -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance  -Dlog4j2.configurationFile=file:../config/log4j2.xml --propsFile [Config file path]
//Contents of config file
server.performance.test.type=GET_BLOB
server.performance.hardware.layout.file.path=[HardwareLayoutFile]
server.performance.partition.layout.file.path=[PartitionLayoutFile]
server.performance.max.parallel.requests=[number of paraller requests we want to keep]
server.performance.hostname=[host name to which we want to make requests]
server.performance.port=[port of the host above]
server.performance.time.out.seconds=[time after which tool will stop]
server.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]
server.performance.get.test.blob.id.file.path=[File which contains blobids separated by new line]
```
We will be able to see logs 

```
[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 4 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)
[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)
[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 6 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)
[2025-02-06 20:57:45,654] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 1 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)
[2025-02-06 20:57:45,653] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)
[2025-02-06 20:57:45,655] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 3 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)

```


Testing for PUT blobs

This tool will try to put blobs to one partition per disk of the target host in round-robin format until
time out or it has reached limit of total number of bytes
```bash
java -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance -Dlog4j2.configurationFile=file:../config/log4j2.xml  --propsFile [Config file path]
//Contents of config file
server.performance.test.type=PUT_BLOB
server.performance.hardware.layout.file.path=[HardwareLayoutFile]
server.performance.partition.layout.file.path=[PartitionLayoutFile]
server.performance.max.parallel.requests=[number of paraller requests we want to keep]
server.performance.hostname=[host name to which we want to make requests]
server.performance.port=[port of the host above]
server.performance.time.out.seconds=[time after which tool will stop]
server.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]
server.performance.put.test.data.limit.bytes=[maximum number of bytes which we want to send from the tool]
```
We will be able to see logs

```
[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 14, blob id AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)
[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 15, blob id AAYQAv____8AAQAAAAAAAAB-_lPPvQMoSnSJMG0i-f4WEA , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)
[2025-02-06 21:11:42,005] INFO Submitting put request correlationId 16, blob id AAYQAv____8AAQAAAAAAAALgA7Dmj4g-Qa2ilxHxltHuCQ , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)
[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 11 AAYQAv____8AAQAAAAAAAAIvYd-809X8S5OJ5OUXE9amLA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)
[2025-02-06 21:11:42,417] INFO received success response and decoded response successfully for correlation id 13 AAYQAv____8AAQAAAAAAAALB72n194MWSz2bLaq2LRCPDA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)
[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 14 AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)
```

We can also exit the process using CTRL+C command for both the tests

Final output for the both tests is network client related metrics

```
[2025-02-06 21:11:43,235] INFO HTTP2 error count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)
[2025-02-06 21:11:43,236] INFO HTTP2 dropped request count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)
[2025-02-06 21:11:43,236] INFO HTTP2 send Mean rate 8.958314373706395 (com.github.ambry.tools.perf.serverperf.ServerPerformance)
[2025-02-06 21:11:43,251] INFO HTTP2 stream median read time 383.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)
[2025-02-06 21:11:43,251] INFO HTTP2 stream median acquire time, 171.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)
```

---

--- METRICS ---
- Files Extracted: 4
- Estimated Token Budget: ~5936 tokens
- Canonical Reference: https://codewiki.google/github.com/linkedin/ambry