{"owner":"linkedin","repo":"ambry","hasSkills":true,"totalSkillsCount":2,"totalTokensCount":6531,"categories":["plugin-manifest","claude-rule"],"hasMcp":false,"mcpConfig":null,"found":["ambry-tools/src/main/java/com/github/ambry/Readme.md","CLAUDE.md"],"skills":{"ambry-tools/src/main/java/com/github/ambry/Readme.md":"## Terminologies\n\nHardwareLayout   - Has information about the nodes available in the cluster. disks for those nodes and their states\nPartitionLayout  - Has information about the partitions and its replicas\nBlobid           - BlobId of the blob/object which was generated by ambry on put\n\n\n## Set up\nTo execute tools, ./gradlew allJar in the root directory.\nThis will generate ambry.jar in \"target\" folder under root directory.\nThis jar will assist using these tools.\n\n\n## DumpLogTool, DumpIndexTool, DumpReplicaTokenTool and DumpDataTool\nThese tools are used to dump the data parsing the storage files.\n\n#### Operations supported\n* Dumping log segment\n* Dumping index / segments\n* Dumping replicaToken\n* Compare index entries to log contents\n\n## BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas.\n\n#### Operations supported:\n* Get blob from all replicas and compare the record across replicas for a list of blob ids.\n* Get blob from all replicas in a datacenter and compare the record across replicas for a list of blob ids.\n* Get blob from a specific replica and ensure that it deserializes correctly for a list of blob ids.\n\n## ServerAdminTool\nThis tool can be used to operate on a single ambry server. Please refer to the config documentation and the class \ndocumentation for usage details.\n\n#### Operations supported\n* GetBlobProperties\n* GetBlobUserMetadata\n* GetBlob\n* TriggerCompaction\n\n#### Dump Index\nThis tool can be used to dump or verify whole indexes or index segments. The operations can also be done on a subset of blobs.\nPlease refer to the file for more details on what the operations are and the configuration supported.\n#### Operations supported\n* DumpIndex\n* DumpIndexSegment\n* VerifyIndex\n\n### Dumping Log\n\n#### Dump log file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nsilent=false\n```\n\n#### Dump log file with throttling\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobs.per.sec=100\nsilent=false\n```\n\n#### Dump log starting at offset x and ending at offset y\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nlog.start.offset=x\nlog.end.offset=y\nsilent=false\n```\n\n#### Dump log filtering with a set of blobs\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobId.list=blobId1,blobId2,blobId3\nsilent=false\n```\n\n### Comparison operations\n\n#### Comparing index entries to log entries(or in other words, trying to read a log record based on index info for\n#### every entry in the index)\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareIndexToLog\nfile.to.read=[indexFile]\n```\n\n#### Comparing all index entries for a replica to log entries\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareReplicaIndexesToLog\nreplica.root.directory=[replicaRootDirecotry]\nindex.files.per.sec=1 // number of index files to process per sec\n```\n\n### Dumping ReplicaToken file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpReplicaTokenTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[replicaTokenFile]\n```\n\n### BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas. Please\nrefer to the config documentation and the class  documentation for usage details.\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ServerAdminTool\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ConsistencyCheckerTool\n\n#### Check for consistency among replicas for a partition\n```bash\njava -cp ambry.jar com.github.ambry.store.ConsistencyCheckerTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\npartition.root.directory=[Root directory which contains one directory per replica. Each replica's directory is expected \nto contain all index files for the respective replica. Diretory's name is considered to be replica name for logging \npurposes]\n```\n\n\n```\nRoot Directory for Partition should contains N sub-directories pertaining to N replicas. Each sub-directory's name\nshould be the same as name of the replica. Each sub directory(replica) in turn should have all the index segments within it.\n\nA sample structure of a root Directory:\n\nrootDir\n  ela4-app8763.prod.linkedin.com\n    0_index\n    10264262199_index\n    10821344563_index\n  lva1-app6774.prod.linkedin.com\n    0_index\n    10267816698_index\n    10892574781_index\n  lva1-app6775.prod.linkedin.com\n    0_index\n    10266323380_index\n    10888037904_index\n```\n\n\n###ConcurrencyTestTool\n####Sample test run\n\nEnsure that ambry servers in the HardwareLayout are up running\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ConcurrencyTestTool --hardwareLayout [HardwareLayoutFile]\n --partitionLayout [PartitionLayoutFile] --putGetHelperFactory com.github.ambry.tools.admin.RouterPutGetHelperFactory\n --maxParallelPutCount 2 --parallelGetCount 4 --burstCountForGet 4 --maxGetCountPerBlob 10\n --totalPutBlobCount 1000 --maxBlobSizeInBytes 1000 --minBlobSizeInBytes 100 --routerPropsFilePath [Path to router props\n file] --deleteAndValidate true\n```\n\n### ServerPerformanceTool\n\nTesting for GET blobs\nThis tool will read the log file for blobs and keep making get requests for each blob until time out\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance  -Dlog4j2.configurationFile=file:../config/log4j2.xml --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=GET_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.get.test.blob.id.file.path=[File which contains blobids separated by new line]\n```\nWe will be able to see logs \n\n```\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 4 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 6 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,654] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 1 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,653] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,655] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 3 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n\n```\n\n\nTesting for PUT blobs\n\nThis tool will try to put blobs to one partition per disk of the target host in round-robin format until\ntime out or it has reached limit of total number of bytes\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance -Dlog4j2.configurationFile=file:../config/log4j2.xml  --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=PUT_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.put.test.data.limit.bytes=[maximum number of bytes which we want to send from the tool]\n```\nWe will be able to see logs\n\n```\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 14, blob id AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 15, blob id AAYQAv____8AAQAAAAAAAAB-_lPPvQMoSnSJMG0i-f4WEA , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,005] INFO Submitting put request correlationId 16, blob id AAYQAv____8AAQAAAAAAAALgA7Dmj4g-Qa2ilxHxltHuCQ , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 11 AAYQAv____8AAQAAAAAAAAIvYd-809X8S5OJ5OUXE9amLA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,417] INFO received success response and decoded response successfully for correlation id 13 AAYQAv____8AAQAAAAAAAALB72n194MWSz2bLaq2LRCPDA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 14 AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n```\n\nWe can also exit the process using CTRL+C command for both the tests\n\nFinal output for the both tests is network client related metrics\n\n```\n[2025-02-06 21:11:43,235] INFO HTTP2 error count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 dropped request count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 send Mean rate 8.958314373706395 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median read time 383.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median acquire time, 171.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n```","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n---\n\n## 1. LLM Behavioral Guidelines\n\nBehavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.\n\n**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.\n\n### Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them — don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\nStorage systems punish assumptions. Read the relevant source files, understand the data flow, and trace the call path before proposing changes. If you are unsure about a component's behavior, read its implementation and tests before modifying it.\n\n### Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n### Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it — don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n### Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" → \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" → \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" → \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] → verify: [check]\n2. [Step] → verify: [check]\n3. [Step] → verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n\n**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.\n\n**Use subagents by default** where possible for parallel research, exploration, and independent subtasks.\n\n---\n\n## 2. PR Guidelines\n\nPRs must be logically isolated and easy to review: small, coherent diffs with one purpose per PR. Avoid drive-by refactors.\n\nPR descriptions **must** include a `## Testing Done` section (not `## Testing`).\n\nAim for as close to 100% unit test coverage as practical for touched logic. Cover risky paths and edge cases.\n\n---\n\n## 3. Ambry System-Level Invariants\n\n### Durability Paranoia\n\nAmbry is a production blob storage system. **\"You must never lose data\"** is a non-negotiable invariant. Every PR must include a durability risk analysis and tests/mitigations.\n\n**Tradeoff priority order** (when forced to choose):\n1. **Durability** — Data must never be lost or silently corrupted\n2. **Security** — ACLs, encryption, and auth must not be weakened\n3. **Availability** — The system must remain serving\n4. **Operability** — Changes must be observable and debuggable\n5. **Efficiency** — Resource usage matters but not at the cost of the above\n6. **Performance** — Latency and throughput matter but not at the cost of the above\n\nNo repo-specific guideline can override the durability invariant.\n\n### Engineering Principles\n\n- Explicit invariants and clear failure-mode handling\n- Safe retries and idempotency — operations must be safe to retry after partial failure\n- Defensive coding — validate inputs at system boundaries, handle nulls explicitly\n- Code should be readable by humans and LLM agents (future maintainers)\n- Strong bias toward correctness and operability over cleverness\n\n### Durability Risk Checklist\n\nApply to every PR. If any item is checked, the PR requires extra scrutiny and explicit testing.\n\n- [ ] **Write path** — Does this change touch PUT, named blob PUT, TTL update, or delete operations?\n- [ ] **Metadata** — Does this change modify how blob metadata is stored, read, or indexed?\n- [ ] **Ordering / Atomicity** — Could operations complete partially, leaving inconsistent state?\n- [ ] **Callback semantics** — Could success be reported to a client before data is durable?\n- [ ] **Resource cleanup** — Does this modify ByteBuf release, Closeable.close(), or stream lifecycle?\n- [ ] **Retries / Idempotency** — Are retries safe? Could a retry cause duplicate writes or skipped deletes?\n- [ ] **Partial failures** — What happens if this operation fails midway? Is the system left in a recoverable state?\n- [ ] **Corruption handling** — If data corruption is detected (e.g., MD5 mismatch), is it logged and surfaced, not silently ignored?\n\n### Java Review Standards\n\n- **Do** use SLF4J parameterized logging: `logger.info(\"Processing blob {} for container {}\", blobId, containerName)`. **Avoid** string concatenation in log statements.\n- **Do** use correct log levels: `error` = unexpected failures requiring attention, `warn` = recoverable issues, `info` = important state changes, `debug`/`trace` = detailed debugging. **Avoid** logging expected conditions at `error` or `warn`.\n- **Do** ensure `LoggerFactory.getLogger()` references the enclosing class. **Avoid** copy-pasting logger declarations — this is a recurring source of bugs.\n- **Do** add explicit null checks before dereferencing, especially for values from `RestRequest.getArgs()` and enum comparisons. **Avoid** `enumValue != SOME_ENUM` without a null guard — it returns `true` when `enumValue` is `null`.\n- **Do** delete unused code outright. **Avoid** commenting out code blocks. Use version control to recover old code; use config flags for runtime control.\n- **Do** make defensive copies of mutable arrays (e.g., `byte[]`) in exactly one place — either the builder setter or the constructor. **Avoid** copying in both.\n- **Do** catch the narrowest exception type possible and log with context: `logger.error(\"Failed to process blob {}\", blobId, e)`. **Avoid** catching generic `Exception` without re-throwing.\n- **Do** use import statements. **Avoid** fully qualified names in code, unless there is a naming collision.\n- **Do** release Netty `ByteBuf` in `finally` blocks. Implement `Closeable` for services holding network clients or other resources. Verify `refCnt() > 0` before calling `release()`.\n- **Do** mark static mutable fields as `volatile` or use proper synchronization. **Avoid** unprotected static mutable state — services are long-running and concurrent.\n- **Do** extract non-obvious numeric literals to named constants with explanatory comments. **Avoid** magic numbers like `9223372036854775806L` (use `Long.MAX_VALUE - 1` with a comment).\n- **Do** annotate with `@VisibleForTesting` when widening field or method visibility solely for test access. **Avoid** silently making fields package-private without documenting why.\n\n### Documentation and Test Update Requirement\n\nEvery PR that changes behavior, APIs, config, or architecture must update relevant documentation AND tests.\n\n---\n\n## 4. Repo-Specific Guidelines\n\n### Build and Development Commands\n\n```bash\n# Build\n./gradlew build -x test                       # Build without tests\n\n# Test\n./gradlew test                                # Run all tests\n\n# Run specific test class\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest\"\n\n# Run specific test method\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest.bootstrapBasicTest\"\n```\n\n### Architecture Overview\n\nAmbry is an immutable distributed blob storage system. See [GitHub](https://github.com/linkedin/ambry) for details.\n\n#### Key Modules\n\n| Module | Purpose |\n|--------|---------|\n| `ambry-api/` | Public interfaces and configs |\n| `ambry-clustermap/` | Cluster topology, Helix integration, state management |\n| `ambry-replication/` | Cross-datacenter replication engine |\n| `ambry-router/` | Client-side routing and operation tracking |\n| `ambry-server/` | Storage data layer |\n| `ambry-store/` | On-disk blob store implementation |\n| `ambry-frontend/` | REST API gateway |\n| `ambry-test-utils/` | Shared test mocks and utilities |\n\n#### Package Convention\n\n- `com.github.ambry.*` — All Ambry classes\n\n### CRITICAL: Schema Change Safety\n\nAny change to a schema — whether it is a JSON-serialized Java class, a MySQL DDL, a Protobuf definition, a ZooKeeper node structure, or a Helix property store format — must be treated as a cross-cluster, cross-fabric change. All Ambry clusters across all fabrics read shared data. A backward-incompatible change poisons every cluster simultaneously.\n\n**General rules for ALL schema changes:**\n\n1. **Every schema change MUST be backward and forward compatible.** Backward compatibility means new code can read data written by older schemas. Forward compatibility means old code can safely ignore unknown fields/columns written by newer schemas.\n\n2. **Deployment ordering:** First deploy the code that can **read** the new schema to ALL hosts. Only then deploy the code that **writes** the new schema. Never write new fields/columns before all readers can handle them.\n\n3. **Any PR that changes a schema MUST include compatibility tests:**\n   - Forward compatibility test: read data written by the **new** schema (with new fields/columns) using the **old** code path — old readers must tolerate new data\n   - Backward compatibility test: read data written by the **old** schema (without new fields/columns) using the **new** code path — new readers must handle missing data with sensible defaults\n\n4. **Key shared schemas to watch:**\n   - `Account`, `Container` (JSON in MySQL `account_metadata`)\n   - Any class serialized to ZooKeeper or Helix property stores\n   - MySQL DDL schemas (column additions, type changes, index changes)\n   - Any wire protocol or RPC message format changes\n\n5. **Fail-fast on startup, degrade gracefully at runtime:**\n   - If shared metadata (e.g., account data) cannot be loaded at startup, the frontend MUST fail to start — not serve with an empty cache\n   - At runtime, not all records are safe to skip. Records affecting access control, quotas, routing, or ownership metadata must fail the operation or raise an alert if they cannot be loaded — silently skipping them could change correctness or authorization. For non-critical record types, skip the failing record and continue, but emit structured logs and metrics with record identifiers so the issue is detectable.\n\n**Jackson-specific rules (for shared-store DTOs):**\n\n6. **Every DTO deserialized from a shared store (MySQL JSON metadata, ZooKeeper, Helix property stores) MUST have `@JsonIgnoreProperties(ignoreUnknown = true)`** at the class level. Without it, adding a new field to the JSON will cause deserialization failures on any host running older code that does not recognize the field. This requirement applies to shared-store DTOs, not necessarily to purely internal config objects or in-process-only types.\n\n7. **When reviewing PRs that touch Jackson-deserialized shared-store DTOs:**\n   - Check: Does the class have `@JsonIgnoreProperties(ignoreUnknown = true)`? If not, **block the PR**.\n   - Check: Are there tests for unknown field tolerance? If not, **request them**.\n   - Test round-trip: serialize → inject unknown fields into the JSON string → deserialize → verify no data loss of known fields.\n   - For critical records (account/container metadata), also test that required fields are validated — removing or renaming a required field should fail deserialization or be caught by validation logic.\n\n### Error Handling\n\n- **Do** wrap errors in `RestServiceException` with appropriate `RestServiceErrorCode`.\n- **Do** check for errors first in callback handlers before processing results.\n- **Do** guard against NPE in validation callbacks: if primaryResult or secondaryResult is null, record a metric and return early.\n\n### Style Rules\n\n- **Imports:** Static imports first, then java.*, third-party, com.github.ambry.*. No wildcards except for static utility constants.\n- **Line length:** ~120 characters max.\n- **Annotations:** Each on its own line, `@Override` first.\n- **Naming:** PascalCase classes, camelCase methods/variables, UPPER_SNAKE_CASE constants, `{Name}Factory`/`{Name}Config`/`{Name}Test` suffixes.\n- **Test methods:** `test{Scenario}` (e.g., `testSecondaryFailure`, `testResourceCleanup`).\n\n### Must-Not-Do List\n\n1. **Never change public APIs** (REST endpoints, account/container schemas) unless explicitly instructed\n2. **Never change default config values** without explicit instruction — this affects all deployments\n3. **Never remove or weaken existing tests** unless instructed\n4. **Never use `jdk.internal.*` imports** — they are unstable and fail across JDK versions\n5. **Never use `Thread.sleep()` for test synchronization** — use `CountDownLatch`, `CompletableFuture.get(timeout, ...)`, or polling with timeout\n6. **Never use `Random` for security-sensitive operations** — use `java.security.SecureRandom`\n7. **Never buffer entire blobs in memory** — use streaming patterns throughout\n8. **Never widen field visibility without `@VisibleForTesting`** annotation\n\n### Testing Conventions\n\n**Framework:** JUnit 4 (`@Test`, `@Before`, `@After`), Mockito.\n\n**Key patterns:**\n- **Do** use `InMemAccountService` for account/container tests, not mocks.\n- **Do** verify metric increments against the **actual** Metrics instance used by the class under test. **Avoid** creating a separate Metrics instance in the test — it won't observe real increments.\n- **Do** use `CompletableFuture` and `future.get(timeout, unit)` for async callback testing.\n- **Do** verify `ByteBuf.refCnt()` drops to 0 in resource cleanup tests.\n- **Do** reset static mutable state in `@Before` or `@After` methods to prevent test interdependencies.\n\n**Test file layout:**\n```\n{module}/src/test/java/       # Unit tests, mirroring main package structure\n{module}/src/test/resources/  # Test resources\n```\n\n---\n\n## 5. Review Culture\n\n### Copy-paste vigilance for constants\n\n- **Do** verify that when duplicating a constant declaration, BOTH the variable name AND the string literal value are updated.\n- **Avoid** the pattern where the variable name is changed but the value still points to the original.\n\n### Async callback race conditions\n\n- **Do** call `primaryFuture.get(timeout, ...)` before reading primary data from request args in secondary comparison callbacks.\n- **Avoid** assuming request args are populated before the primary callback has run — if a secondary call completes first, you get stale/null data.\n\n### Test metric verification\n\n- **Do** spy on or retrieve the actual `*Metrics` instance wired into the class under test (via `Whitebox.getInternalState` or the shared `MetricRegistry`).\n- **Avoid** creating a fresh `*Metrics` instance in the test and asserting against its counters — it won't observe real increments.\n"},"files":{"ambry-tools/src/main/java/com/github/ambry/Readme.md":"## Terminologies\n\nHardwareLayout   - Has information about the nodes available in the cluster. disks for those nodes and their states\nPartitionLayout  - Has information about the partitions and its replicas\nBlobid           - BlobId of the blob/object which was generated by ambry on put\n\n\n## Set up\nTo execute tools, ./gradlew allJar in the root directory.\nThis will generate ambry.jar in \"target\" folder under root directory.\nThis jar will assist using these tools.\n\n\n## DumpLogTool, DumpIndexTool, DumpReplicaTokenTool and DumpDataTool\nThese tools are used to dump the data parsing the storage files.\n\n#### Operations supported\n* Dumping log segment\n* Dumping index / segments\n* Dumping replicaToken\n* Compare index entries to log contents\n\n## BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas.\n\n#### Operations supported:\n* Get blob from all replicas and compare the record across replicas for a list of blob ids.\n* Get blob from all replicas in a datacenter and compare the record across replicas for a list of blob ids.\n* Get blob from a specific replica and ensure that it deserializes correctly for a list of blob ids.\n\n## ServerAdminTool\nThis tool can be used to operate on a single ambry server. Please refer to the config documentation and the class \ndocumentation for usage details.\n\n#### Operations supported\n* GetBlobProperties\n* GetBlobUserMetadata\n* GetBlob\n* TriggerCompaction\n\n#### Dump Index\nThis tool can be used to dump or verify whole indexes or index segments. The operations can also be done on a subset of blobs.\nPlease refer to the file for more details on what the operations are and the configuration supported.\n#### Operations supported\n* DumpIndex\n* DumpIndexSegment\n* VerifyIndex\n\n### Dumping Log\n\n#### Dump log file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nsilent=false\n```\n\n#### Dump log file with throttling\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobs.per.sec=100\nsilent=false\n```\n\n#### Dump log starting at offset x and ending at offset y\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nlog.start.offset=x\nlog.end.offset=y\nsilent=false\n```\n\n#### Dump log filtering with a set of blobs\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobId.list=blobId1,blobId2,blobId3\nsilent=false\n```\n\n### Comparison operations\n\n#### Comparing index entries to log entries(or in other words, trying to read a log record based on index info for\n#### every entry in the index)\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareIndexToLog\nfile.to.read=[indexFile]\n```\n\n#### Comparing all index entries for a replica to log entries\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareReplicaIndexesToLog\nreplica.root.directory=[replicaRootDirecotry]\nindex.files.per.sec=1 // number of index files to process per sec\n```\n\n### Dumping ReplicaToken file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpReplicaTokenTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[replicaTokenFile]\n```\n\n### BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas. Please\nrefer to the config documentation and the class  documentation for usage details.\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ServerAdminTool\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ConsistencyCheckerTool\n\n#### Check for consistency among replicas for a partition\n```bash\njava -cp ambry.jar com.github.ambry.store.ConsistencyCheckerTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\npartition.root.directory=[Root directory which contains one directory per replica. Each replica's directory is expected \nto contain all index files for the respective replica. Diretory's name is considered to be replica name for logging \npurposes]\n```\n\n\n```\nRoot Directory for Partition should contains N sub-directories pertaining to N replicas. Each sub-directory's name\nshould be the same as name of the replica. Each sub directory(replica) in turn should have all the index segments within it.\n\nA sample structure of a root Directory:\n\nrootDir\n  ela4-app8763.prod.linkedin.com\n    0_index\n    10264262199_index\n    10821344563_index\n  lva1-app6774.prod.linkedin.com\n    0_index\n    10267816698_index\n    10892574781_index\n  lva1-app6775.prod.linkedin.com\n    0_index\n    10266323380_index\n    10888037904_index\n```\n\n\n###ConcurrencyTestTool\n####Sample test run\n\nEnsure that ambry servers in the HardwareLayout are up running\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ConcurrencyTestTool --hardwareLayout [HardwareLayoutFile]\n --partitionLayout [PartitionLayoutFile] --putGetHelperFactory com.github.ambry.tools.admin.RouterPutGetHelperFactory\n --maxParallelPutCount 2 --parallelGetCount 4 --burstCountForGet 4 --maxGetCountPerBlob 10\n --totalPutBlobCount 1000 --maxBlobSizeInBytes 1000 --minBlobSizeInBytes 100 --routerPropsFilePath [Path to router props\n file] --deleteAndValidate true\n```\n\n### ServerPerformanceTool\n\nTesting for GET blobs\nThis tool will read the log file for blobs and keep making get requests for each blob until time out\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance  -Dlog4j2.configurationFile=file:../config/log4j2.xml --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=GET_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.get.test.blob.id.file.path=[File which contains blobids separated by new line]\n```\nWe will be able to see logs \n\n```\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 4 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 6 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,654] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 1 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,653] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,655] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 3 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n\n```\n\n\nTesting for PUT blobs\n\nThis tool will try to put blobs to one partition per disk of the target host in round-robin format until\ntime out or it has reached limit of total number of bytes\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance -Dlog4j2.configurationFile=file:../config/log4j2.xml  --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=PUT_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.put.test.data.limit.bytes=[maximum number of bytes which we want to send from the tool]\n```\nWe will be able to see logs\n\n```\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 14, blob id AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 15, blob id AAYQAv____8AAQAAAAAAAAB-_lPPvQMoSnSJMG0i-f4WEA , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,005] INFO Submitting put request correlationId 16, blob id AAYQAv____8AAQAAAAAAAALgA7Dmj4g-Qa2ilxHxltHuCQ , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 11 AAYQAv____8AAQAAAAAAAAIvYd-809X8S5OJ5OUXE9amLA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,417] INFO received success response and decoded response successfully for correlation id 13 AAYQAv____8AAQAAAAAAAALB72n194MWSz2bLaq2LRCPDA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 14 AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n```\n\nWe can also exit the process using CTRL+C command for both the tests\n\nFinal output for the both tests is network client related metrics\n\n```\n[2025-02-06 21:11:43,235] INFO HTTP2 error count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 dropped request count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 send Mean rate 8.958314373706395 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median read time 383.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median acquire time, 171.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n```","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n---\n\n## 1. LLM Behavioral Guidelines\n\nBehavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.\n\n**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.\n\n### Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them — don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\nStorage systems punish assumptions. Read the relevant source files, understand the data flow, and trace the call path before proposing changes. If you are unsure about a component's behavior, read its implementation and tests before modifying it.\n\n### Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n### Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it — don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n### Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" → \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" → \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" → \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] → verify: [check]\n2. [Step] → verify: [check]\n3. [Step] → verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n\n**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.\n\n**Use subagents by default** where possible for parallel research, exploration, and independent subtasks.\n\n---\n\n## 2. PR Guidelines\n\nPRs must be logically isolated and easy to review: small, coherent diffs with one purpose per PR. Avoid drive-by refactors.\n\nPR descriptions **must** include a `## Testing Done` section (not `## Testing`).\n\nAim for as close to 100% unit test coverage as practical for touched logic. Cover risky paths and edge cases.\n\n---\n\n## 3. Ambry System-Level Invariants\n\n### Durability Paranoia\n\nAmbry is a production blob storage system. **\"You must never lose data\"** is a non-negotiable invariant. Every PR must include a durability risk analysis and tests/mitigations.\n\n**Tradeoff priority order** (when forced to choose):\n1. **Durability** — Data must never be lost or silently corrupted\n2. **Security** — ACLs, encryption, and auth must not be weakened\n3. **Availability** — The system must remain serving\n4. **Operability** — Changes must be observable and debuggable\n5. **Efficiency** — Resource usage matters but not at the cost of the above\n6. **Performance** — Latency and throughput matter but not at the cost of the above\n\nNo repo-specific guideline can override the durability invariant.\n\n### Engineering Principles\n\n- Explicit invariants and clear failure-mode handling\n- Safe retries and idempotency — operations must be safe to retry after partial failure\n- Defensive coding — validate inputs at system boundaries, handle nulls explicitly\n- Code should be readable by humans and LLM agents (future maintainers)\n- Strong bias toward correctness and operability over cleverness\n\n### Durability Risk Checklist\n\nApply to every PR. If any item is checked, the PR requires extra scrutiny and explicit testing.\n\n- [ ] **Write path** — Does this change touch PUT, named blob PUT, TTL update, or delete operations?\n- [ ] **Metadata** — Does this change modify how blob metadata is stored, read, or indexed?\n- [ ] **Ordering / Atomicity** — Could operations complete partially, leaving inconsistent state?\n- [ ] **Callback semantics** — Could success be reported to a client before data is durable?\n- [ ] **Resource cleanup** — Does this modify ByteBuf release, Closeable.close(), or stream lifecycle?\n- [ ] **Retries / Idempotency** — Are retries safe? Could a retry cause duplicate writes or skipped deletes?\n- [ ] **Partial failures** — What happens if this operation fails midway? Is the system left in a recoverable state?\n- [ ] **Corruption handling** — If data corruption is detected (e.g., MD5 mismatch), is it logged and surfaced, not silently ignored?\n\n### Java Review Standards\n\n- **Do** use SLF4J parameterized logging: `logger.info(\"Processing blob {} for container {}\", blobId, containerName)`. **Avoid** string concatenation in log statements.\n- **Do** use correct log levels: `error` = unexpected failures requiring attention, `warn` = recoverable issues, `info` = important state changes, `debug`/`trace` = detailed debugging. **Avoid** logging expected conditions at `error` or `warn`.\n- **Do** ensure `LoggerFactory.getLogger()` references the enclosing class. **Avoid** copy-pasting logger declarations — this is a recurring source of bugs.\n- **Do** add explicit null checks before dereferencing, especially for values from `RestRequest.getArgs()` and enum comparisons. **Avoid** `enumValue != SOME_ENUM` without a null guard — it returns `true` when `enumValue` is `null`.\n- **Do** delete unused code outright. **Avoid** commenting out code blocks. Use version control to recover old code; use config flags for runtime control.\n- **Do** make defensive copies of mutable arrays (e.g., `byte[]`) in exactly one place — either the builder setter or the constructor. **Avoid** copying in both.\n- **Do** catch the narrowest exception type possible and log with context: `logger.error(\"Failed to process blob {}\", blobId, e)`. **Avoid** catching generic `Exception` without re-throwing.\n- **Do** use import statements. **Avoid** fully qualified names in code, unless there is a naming collision.\n- **Do** release Netty `ByteBuf` in `finally` blocks. Implement `Closeable` for services holding network clients or other resources. Verify `refCnt() > 0` before calling `release()`.\n- **Do** mark static mutable fields as `volatile` or use proper synchronization. **Avoid** unprotected static mutable state — services are long-running and concurrent.\n- **Do** extract non-obvious numeric literals to named constants with explanatory comments. **Avoid** magic numbers like `9223372036854775806L` (use `Long.MAX_VALUE - 1` with a comment).\n- **Do** annotate with `@VisibleForTesting` when widening field or method visibility solely for test access. **Avoid** silently making fields package-private without documenting why.\n\n### Documentation and Test Update Requirement\n\nEvery PR that changes behavior, APIs, config, or architecture must update relevant documentation AND tests.\n\n---\n\n## 4. Repo-Specific Guidelines\n\n### Build and Development Commands\n\n```bash\n# Build\n./gradlew build -x test                       # Build without tests\n\n# Test\n./gradlew test                                # Run all tests\n\n# Run specific test class\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest\"\n\n# Run specific test method\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest.bootstrapBasicTest\"\n```\n\n### Architecture Overview\n\nAmbry is an immutable distributed blob storage system. See [GitHub](https://github.com/linkedin/ambry) for details.\n\n#### Key Modules\n\n| Module | Purpose |\n|--------|---------|\n| `ambry-api/` | Public interfaces and configs |\n| `ambry-clustermap/` | Cluster topology, Helix integration, state management |\n| `ambry-replication/` | Cross-datacenter replication engine |\n| `ambry-router/` | Client-side routing and operation tracking |\n| `ambry-server/` | Storage data layer |\n| `ambry-store/` | On-disk blob store implementation |\n| `ambry-frontend/` | REST API gateway |\n| `ambry-test-utils/` | Shared test mocks and utilities |\n\n#### Package Convention\n\n- `com.github.ambry.*` — All Ambry classes\n\n### CRITICAL: Schema Change Safety\n\nAny change to a schema — whether it is a JSON-serialized Java class, a MySQL DDL, a Protobuf definition, a ZooKeeper node structure, or a Helix property store format — must be treated as a cross-cluster, cross-fabric change. All Ambry clusters across all fabrics read shared data. A backward-incompatible change poisons every cluster simultaneously.\n\n**General rules for ALL schema changes:**\n\n1. **Every schema change MUST be backward and forward compatible.** Backward compatibility means new code can read data written by older schemas. Forward compatibility means old code can safely ignore unknown fields/columns written by newer schemas.\n\n2. **Deployment ordering:** First deploy the code that can **read** the new schema to ALL hosts. Only then deploy the code that **writes** the new schema. Never write new fields/columns before all readers can handle them.\n\n3. **Any PR that changes a schema MUST include compatibility tests:**\n   - Forward compatibility test: read data written by the **new** schema (with new fields/columns) using the **old** code path — old readers must tolerate new data\n   - Backward compatibility test: read data written by the **old** schema (without new fields/columns) using the **new** code path — new readers must handle missing data with sensible defaults\n\n4. **Key shared schemas to watch:**\n   - `Account`, `Container` (JSON in MySQL `account_metadata`)\n   - Any class serialized to ZooKeeper or Helix property stores\n   - MySQL DDL schemas (column additions, type changes, index changes)\n   - Any wire protocol or RPC message format changes\n\n5. **Fail-fast on startup, degrade gracefully at runtime:**\n   - If shared metadata (e.g., account data) cannot be loaded at startup, the frontend MUST fail to start — not serve with an empty cache\n   - At runtime, not all records are safe to skip. Records affecting access control, quotas, routing, or ownership metadata must fail the operation or raise an alert if they cannot be loaded — silently skipping them could change correctness or authorization. For non-critical record types, skip the failing record and continue, but emit structured logs and metrics with record identifiers so the issue is detectable.\n\n**Jackson-specific rules (for shared-store DTOs):**\n\n6. **Every DTO deserialized from a shared store (MySQL JSON metadata, ZooKeeper, Helix property stores) MUST have `@JsonIgnoreProperties(ignoreUnknown = true)`** at the class level. Without it, adding a new field to the JSON will cause deserialization failures on any host running older code that does not recognize the field. This requirement applies to shared-store DTOs, not necessarily to purely internal config objects or in-process-only types.\n\n7. **When reviewing PRs that touch Jackson-deserialized shared-store DTOs:**\n   - Check: Does the class have `@JsonIgnoreProperties(ignoreUnknown = true)`? If not, **block the PR**.\n   - Check: Are there tests for unknown field tolerance? If not, **request them**.\n   - Test round-trip: serialize → inject unknown fields into the JSON string → deserialize → verify no data loss of known fields.\n   - For critical records (account/container metadata), also test that required fields are validated — removing or renaming a required field should fail deserialization or be caught by validation logic.\n\n### Error Handling\n\n- **Do** wrap errors in `RestServiceException` with appropriate `RestServiceErrorCode`.\n- **Do** check for errors first in callback handlers before processing results.\n- **Do** guard against NPE in validation callbacks: if primaryResult or secondaryResult is null, record a metric and return early.\n\n### Style Rules\n\n- **Imports:** Static imports first, then java.*, third-party, com.github.ambry.*. No wildcards except for static utility constants.\n- **Line length:** ~120 characters max.\n- **Annotations:** Each on its own line, `@Override` first.\n- **Naming:** PascalCase classes, camelCase methods/variables, UPPER_SNAKE_CASE constants, `{Name}Factory`/`{Name}Config`/`{Name}Test` suffixes.\n- **Test methods:** `test{Scenario}` (e.g., `testSecondaryFailure`, `testResourceCleanup`).\n\n### Must-Not-Do List\n\n1. **Never change public APIs** (REST endpoints, account/container schemas) unless explicitly instructed\n2. **Never change default config values** without explicit instruction — this affects all deployments\n3. **Never remove or weaken existing tests** unless instructed\n4. **Never use `jdk.internal.*` imports** — they are unstable and fail across JDK versions\n5. **Never use `Thread.sleep()` for test synchronization** — use `CountDownLatch`, `CompletableFuture.get(timeout, ...)`, or polling with timeout\n6. **Never use `Random` for security-sensitive operations** — use `java.security.SecureRandom`\n7. **Never buffer entire blobs in memory** — use streaming patterns throughout\n8. **Never widen field visibility without `@VisibleForTesting`** annotation\n\n### Testing Conventions\n\n**Framework:** JUnit 4 (`@Test`, `@Before`, `@After`), Mockito.\n\n**Key patterns:**\n- **Do** use `InMemAccountService` for account/container tests, not mocks.\n- **Do** verify metric increments against the **actual** Metrics instance used by the class under test. **Avoid** creating a separate Metrics instance in the test — it won't observe real increments.\n- **Do** use `CompletableFuture` and `future.get(timeout, unit)` for async callback testing.\n- **Do** verify `ByteBuf.refCnt()` drops to 0 in resource cleanup tests.\n- **Do** reset static mutable state in `@Before` or `@After` methods to prevent test interdependencies.\n\n**Test file layout:**\n```\n{module}/src/test/java/       # Unit tests, mirroring main package structure\n{module}/src/test/resources/  # Test resources\n```\n\n---\n\n## 5. Review Culture\n\n### Copy-paste vigilance for constants\n\n- **Do** verify that when duplicating a constant declaration, BOTH the variable name AND the string literal value are updated.\n- **Avoid** the pattern where the variable name is changed but the value still points to the original.\n\n### Async callback race conditions\n\n- **Do** call `primaryFuture.get(timeout, ...)` before reading primary data from request args in secondary comparison callbacks.\n- **Avoid** assuming request args are populated before the primary callback has run — if a secondary call completes first, you get stale/null data.\n\n### Test metric verification\n\n- **Do** spy on or retrieve the actual `*Metrics` instance wired into the class under test (via `Whitebox.getInternalState` or the shared `MetricRegistry`).\n- **Avoid** creating a fresh `*Metrics` instance in the test and asserting against its counters — it won't observe real increments.\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/linkedin/ambry/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n---\n\n## 1. LLM Behavioral Guidelines\n\nBehavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.\n\n**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.\n\n### Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them — don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\nStorage systems punish assumptions. Read the relevant source files, understand the data flow, and trace the call path before proposing changes. If you are unsure about a component's behavior, read its implementation and tests before modifying it.\n\n### Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n### Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it — don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n### Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" → \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" → \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" → \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] → verify: [check]\n2. [Step] → verify: [check]\n3. [Step] → verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n\n**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.\n\n**Use subagents by default** where possible for parallel research, exploration, and independent subtasks.\n\n---\n\n## 2. PR Guidelines\n\nPRs must be logically isolated and easy to review: small, coherent diffs with one purpose per PR. Avoid drive-by refactors.\n\nPR descriptions **must** include a `## Testing Done` section (not `## Testing`).\n\nAim for as close to 100% unit test coverage as practical for touched logic. Cover risky paths and edge cases.\n\n---\n\n## 3. Ambry System-Level Invariants\n\n### Durability Paranoia\n\nAmbry is a production blob storage system. **\"You must never lose data\"** is a non-negotiable invariant. Every PR must include a durability risk analysis and tests/mitigations.\n\n**Tradeoff priority order** (when forced to choose):\n1. **Durability** — Data must never be lost or silently corrupted\n2. **Security** — ACLs, encryption, and auth must not be weakened\n3. **Availability** — The system must remain serving\n4. **Operability** — Changes must be observable and debuggable\n5. **Efficiency** — Resource usage matters but not at the cost of the above\n6. **Performance** — Latency and throughput matter but not at the cost of the above\n\nNo repo-specific guideline can override the durability invariant.\n\n### Engineering Principles\n\n- Explicit invariants and clear failure-mode handling\n- Safe retries and idempotency — operations must be safe to retry after partial failure\n- Defensive coding — validate inputs at system boundaries, handle nulls explicitly\n- Code should be readable by humans and LLM agents (future maintainers)\n- Strong bias toward correctness and operability over cleverness\n\n### Durability Risk Checklist\n\nApply to every PR. If any item is checked, the PR requires extra scrutiny and explicit testing.\n\n- [ ] **Write path** — Does this change touch PUT, named blob PUT, TTL update, or delete operations?\n- [ ] **Metadata** — Does this change modify how blob metadata is stored, read, or indexed?\n- [ ] **Ordering / Atomicity** — Could operations complete partially, leaving inconsistent state?\n- [ ] **Callback semantics** — Could success be reported to a client before data is durable?\n- [ ] **Resource cleanup** — Does this modify ByteBuf release, Closeable.close(), or stream lifecycle?\n- [ ] **Retries / Idempotency** — Are retries safe? Could a retry cause duplicate writes or skipped deletes?\n- [ ] **Partial failures** — What happens if this operation fails midway? Is the system left in a recoverable state?\n- [ ] **Corruption handling** — If data corruption is detected (e.g., MD5 mismatch), is it logged and surfaced, not silently ignored?\n\n### Java Review Standards\n\n- **Do** use SLF4J parameterized logging: `logger.info(\"Processing blob {} for container {}\", blobId, containerName)`. **Avoid** string concatenation in log statements.\n- **Do** use correct log levels: `error` = unexpected failures requiring attention, `warn` = recoverable issues, `info` = important state changes, `debug`/`trace` = detailed debugging. **Avoid** logging expected conditions at `error` or `warn`.\n- **Do** ensure `LoggerFactory.getLogger()` references the enclosing class. **Avoid** copy-pasting logger declarations — this is a recurring source of bugs.\n- **Do** add explicit null checks before dereferencing, especially for values from `RestRequest.getArgs()` and enum comparisons. **Avoid** `enumValue != SOME_ENUM` without a null guard — it returns `true` when `enumValue` is `null`.\n- **Do** delete unused code outright. **Avoid** commenting out code blocks. Use version control to recover old code; use config flags for runtime control.\n- **Do** make defensive copies of mutable arrays (e.g., `byte[]`) in exactly one place — either the builder setter or the constructor. **Avoid** copying in both.\n- **Do** catch the narrowest exception type possible and log with context: `logger.error(\"Failed to process blob {}\", blobId, e)`. **Avoid** catching generic `Exception` without re-throwing.\n- **Do** use import statements. **Avoid** fully qualified names in code, unless there is a naming collision.\n- **Do** release Netty `ByteBuf` in `finally` blocks. Implement `Closeable` for services holding network clients or other resources. Verify `refCnt() > 0` before calling `release()`.\n- **Do** mark static mutable fields as `volatile` or use proper synchronization. **Avoid** unprotected static mutable state — services are long-running and concurrent.\n- **Do** extract non-obvious numeric literals to named constants with explanatory comments. **Avoid** magic numbers like `9223372036854775806L` (use `Long.MAX_VALUE - 1` with a comment).\n- **Do** annotate with `@VisibleForTesting` when widening field or method visibility solely for test access. **Avoid** silently making fields package-private without documenting why.\n\n### Documentation and Test Update Requirement\n\nEvery PR that changes behavior, APIs, config, or architecture must update relevant documentation AND tests.\n\n---\n\n## 4. Repo-Specific Guidelines\n\n### Build and Development Commands\n\n```bash\n# Build\n./gradlew build -x test                       # Build without tests\n\n# Test\n./gradlew test                                # Run all tests\n\n# Run specific test class\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest\"\n\n# Run specific test method\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest.bootstrapBasicTest\"\n```\n\n### Architecture Overview\n\nAmbry is an immutable distributed blob storage system. See [GitHub](https://github.com/linkedin/ambry) for details.\n\n#### Key Modules\n\n| Module | Purpose |\n|--------|---------|\n| `ambry-api/` | Public interfaces and configs |\n| `ambry-clustermap/` | Cluster topology, Helix integration, state management |\n| `ambry-replication/` | Cross-datacenter replication engine |\n| `ambry-router/` | Client-side routing and operation tracking |\n| `ambry-server/` | Storage data layer |\n| `ambry-store/` | On-disk blob store implementation |\n| `ambry-frontend/` | REST API gateway |\n| `ambry-test-utils/` | Shared test mocks and utilities |\n\n#### Package Convention\n\n- `com.github.ambry.*` — All Ambry classes\n\n### CRITICAL: Schema Change Safety\n\nAny change to a schema — whether it is a JSON-serialized Java class, a MySQL DDL, a Protobuf definition, a ZooKeeper node structure, or a Helix property store format — must be treated as a cross-cluster, cross-fabric change. All Ambry clusters across all fabrics read shared data. A backward-incompatible change poisons every cluster simultaneously.\n\n**General rules for ALL schema changes:**\n\n1. **Every schema change MUST be backward and forward compatible.** Backward compatibility means new code can read data written by older schemas. Forward compatibility means old code can safely ignore unknown fields/columns written by newer schemas.\n\n2. **Deployment ordering:** First deploy the code that can **read** the new schema to ALL hosts. Only then deploy the code that **writes** the new schema. Never write new fields/columns before all readers can handle them.\n\n3. **Any PR that changes a schema MUST include compatibility tests:**\n   - Forward compatibility test: read data written by the **new** schema (with new fields/columns) using the **old** code path — old readers must tolerate new data\n   - Backward compatibility test: read data written by the **old** schema (without new fields/columns) using the **new** code path — new readers must handle missing data with sensible defaults\n\n4. **Key shared schemas to watch:**\n   - `Account`, `Container` (JSON in MySQL `account_metadata`)\n   - Any class serialized to ZooKeeper or Helix property stores\n   - MySQL DDL schemas (column additions, type changes, index changes)\n   - Any wire protocol or RPC message format changes\n\n5. **Fail-fast on startup, degrade gracefully at runtime:**\n   - If shared metadata (e.g., account data) cannot be loaded at startup, the frontend MUST fail to start — not serve with an empty cache\n   - At runtime, not all records are safe to skip. Records affecting access control, quotas, routing, or ownership metadata must fail the operation or raise an alert if they cannot be loaded — silently skipping them could change correctness or authorization. For non-critical record types, skip the failing record and continue, but emit structured logs and metrics with record identifiers so the issue is detectable.\n\n**Jackson-specific rules (for shared-store DTOs):**\n\n6. **Every DTO deserialized from a shared store (MySQL JSON metadata, ZooKeeper, Helix property stores) MUST have `@JsonIgnoreProperties(ignoreUnknown = true)`** at the class level. Without it, adding a new field to the JSON will cause deserialization failures on any host running older code that does not recognize the field. This requirement applies to shared-store DTOs, not necessarily to purely internal config objects or in-process-only types.\n\n7. **When reviewing PRs that touch Jackson-deserialized shared-store DTOs:**\n   - Check: Does the class have `@JsonIgnoreProperties(ignoreUnknown = true)`? If not, **block the PR**.\n   - Check: Are there tests for unknown field tolerance? If not, **request them**.\n   - Test round-trip: serialize → inject unknown fields into the JSON string → deserialize → verify no data loss of known fields.\n   - For critical records (account/container metadata), also test that required fields are validated — removing or renaming a required field should fail deserialization or be caught by validation logic.\n\n### Error Handling\n\n- **Do** wrap errors in `RestServiceException` with appropriate `RestServiceErrorCode`.\n- **Do** check for errors first in callback handlers before processing results.\n- **Do** guard against NPE in validation callbacks: if primaryResult or secondaryResult is null, record a metric and return early.\n\n### Style Rules\n\n- **Imports:** Static imports first, then java.*, third-party, com.github.ambry.*. No wildcards except for static utility constants.\n- **Line length:** ~120 characters max.\n- **Annotations:** Each on its own line, `@Override` first.\n- **Naming:** PascalCase classes, camelCase methods/variables, UPPER_SNAKE_CASE constants, `{Name}Factory`/`{Name}Config`/`{Name}Test` suffixes.\n- **Test methods:** `test{Scenario}` (e.g., `testSecondaryFailure`, `testResourceCleanup`).\n\n### Must-Not-Do List\n\n1. **Never change public APIs** (REST endpoints, account/container schemas) unless explicitly instructed\n2. **Never change default config values** without explicit instruction — this affects all deployments\n3. **Never remove or weaken existing tests** unless instructed\n4. **Never use `jdk.internal.*` imports** — they are unstable and fail across JDK versions\n5. **Never use `Thread.sleep()` for test synchronization** — use `CountDownLatch`, `CompletableFuture.get(timeout, ...)`, or polling with timeout\n6. **Never use `Random` for security-sensitive operations** — use `java.security.SecureRandom`\n7. **Never buffer entire blobs in memory** — use streaming patterns throughout\n8. **Never widen field visibility without `@VisibleForTesting`** annotation\n\n### Testing Conventions\n\n**Framework:** JUnit 4 (`@Test`, `@Before`, `@After`), Mockito.\n\n**Key patterns:**\n- **Do** use `InMemAccountService` for account/container tests, not mocks.\n- **Do** verify metric increments against the **actual** Metrics instance used by the class under test. **Avoid** creating a separate Metrics instance in the test — it won't observe real increments.\n- **Do** use `CompletableFuture` and `future.get(timeout, unit)` for async callback testing.\n- **Do** verify `ByteBuf.refCnt()` drops to 0 in resource cleanup tests.\n- **Do** reset static mutable state in `@Before` or `@After` methods to prevent test interdependencies.\n\n**Test file layout:**\n```\n{module}/src/test/java/       # Unit tests, mirroring main package structure\n{module}/src/test/resources/  # Test resources\n```\n\n---\n\n## 5. Review Culture\n\n### Copy-paste vigilance for constants\n\n- **Do** verify that when duplicating a constant declaration, BOTH the variable name AND the string literal value are updated.\n- **Avoid** the pattern where the variable name is changed but the value still points to the original.\n\n### Async callback race conditions\n\n- **Do** call `primaryFuture.get(timeout, ...)` before reading primary data from request args in secondary comparison callbacks.\n- **Avoid** assuming request args are populated before the primary callback has run — if a secondary call completes first, you get stale/null data.\n\n### Test metric verification\n\n- **Do** spy on or retrieve the actual `*Metrics` instance wired into the class under test (via `Whitebox.getInternalState` or the shared `MetricRegistry`).\n- **Avoid** creating a fresh `*Metrics` instance in the test and asserting against its counters — it won't observe real increments.\n","isInternal":false,"tokens":3436,"sizeBytes":15604},{"name":"Readme.md","path":"ambry-tools/src/main/java/com/github/ambry/Readme.md","rawUrl":"https://raw.githubusercontent.com/linkedin/ambry/HEAD/ambry-tools/src/main/java/com/github/ambry/Readme.md","title":"ambry Documentation","category":"plugin-manifest","format":"markdown","content":"## Terminologies\n\nHardwareLayout   - Has information about the nodes available in the cluster. disks for those nodes and their states\nPartitionLayout  - Has information about the partitions and its replicas\nBlobid           - BlobId of the blob/object which was generated by ambry on put\n\n\n## Set up\nTo execute tools, ./gradlew allJar in the root directory.\nThis will generate ambry.jar in \"target\" folder under root directory.\nThis jar will assist using these tools.\n\n\n## DumpLogTool, DumpIndexTool, DumpReplicaTokenTool and DumpDataTool\nThese tools are used to dump the data parsing the storage files.\n\n#### Operations supported\n* Dumping log segment\n* Dumping index / segments\n* Dumping replicaToken\n* Compare index entries to log contents\n\n## BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas.\n\n#### Operations supported:\n* Get blob from all replicas and compare the record across replicas for a list of blob ids.\n* Get blob from all replicas in a datacenter and compare the record across replicas for a list of blob ids.\n* Get blob from a specific replica and ensure that it deserializes correctly for a list of blob ids.\n\n## ServerAdminTool\nThis tool can be used to operate on a single ambry server. Please refer to the config documentation and the class \ndocumentation for usage details.\n\n#### Operations supported\n* GetBlobProperties\n* GetBlobUserMetadata\n* GetBlob\n* TriggerCompaction\n\n#### Dump Index\nThis tool can be used to dump or verify whole indexes or index segments. The operations can also be done on a subset of blobs.\nPlease refer to the file for more details on what the operations are and the configuration supported.\n#### Operations supported\n* DumpIndex\n* DumpIndexSegment\n* VerifyIndex\n\n### Dumping Log\n\n#### Dump log file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nsilent=false\n```\n\n#### Dump log file with throttling\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobs.per.sec=100\nsilent=false\n```\n\n#### Dump log starting at offset x and ending at offset y\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nlog.start.offset=x\nlog.end.offset=y\nsilent=false\n```\n\n#### Dump log filtering with a set of blobs\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobId.list=blobId1,blobId2,blobId3\nsilent=false\n```\n\n### Comparison operations\n\n#### Comparing index entries to log entries(or in other words, trying to read a log record based on index info for\n#### every entry in the index)\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareIndexToLog\nfile.to.read=[indexFile]\n```\n\n#### Comparing all index entries for a replica to log entries\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareReplicaIndexesToLog\nreplica.root.directory=[replicaRootDirecotry]\nindex.files.per.sec=1 // number of index files to process per sec\n```\n\n### Dumping ReplicaToken file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpReplicaTokenTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[replicaTokenFile]\n```\n\n### BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas. Please\nrefer to the config documentation and the class  documentation for usage details.\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ServerAdminTool\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ConsistencyCheckerTool\n\n#### Check for consistency among replicas for a partition\n```bash\njava -cp ambry.jar com.github.ambry.store.ConsistencyCheckerTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\npartition.root.directory=[Root directory which contains one directory per replica. Each replica's directory is expected \nto contain all index files for the respective replica. Diretory's name is considered to be replica name for logging \npurposes]\n```\n\n\n```\nRoot Directory for Partition should contains N sub-directories pertaining to N replicas. Each sub-directory's name\nshould be the same as name of the replica. Each sub directory(replica) in turn should have all the index segments within it.\n\nA sample structure of a root Directory:\n\nrootDir\n  ela4-app8763.prod.linkedin.com\n    0_index\n    10264262199_index\n    10821344563_index\n  lva1-app6774.prod.linkedin.com\n    0_index\n    10267816698_index\n    10892574781_index\n  lva1-app6775.prod.linkedin.com\n    0_index\n    10266323380_index\n    10888037904_index\n```\n\n\n###ConcurrencyTestTool\n####Sample test run\n\nEnsure that ambry servers in the HardwareLayout are up running\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ConcurrencyTestTool --hardwareLayout [HardwareLayoutFile]\n --partitionLayout [PartitionLayoutFile] --putGetHelperFactory com.github.ambry.tools.admin.RouterPutGetHelperFactory\n --maxParallelPutCount 2 --parallelGetCount 4 --burstCountForGet 4 --maxGetCountPerBlob 10\n --totalPutBlobCount 1000 --maxBlobSizeInBytes 1000 --minBlobSizeInBytes 100 --routerPropsFilePath [Path to router props\n file] --deleteAndValidate true\n```\n\n### ServerPerformanceTool\n\nTesting for GET blobs\nThis tool will read the log file for blobs and keep making get requests for each blob until time out\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance  -Dlog4j2.configurationFile=file:../config/log4j2.xml --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=GET_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.get.test.blob.id.file.path=[File which contains blobids separated by new line]\n```\nWe will be able to see logs \n\n```\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 4 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 6 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,654] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 1 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,653] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,655] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 3 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n\n```\n\n\nTesting for PUT blobs\n\nThis tool will try to put blobs to one partition per disk of the target host in round-robin format until\ntime out or it has reached limit of total number of bytes\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance -Dlog4j2.configurationFile=file:../config/log4j2.xml  --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=PUT_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.put.test.data.limit.bytes=[maximum number of bytes which we want to send from the tool]\n```\nWe will be able to see logs\n\n```\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 14, blob id AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 15, blob id AAYQAv____8AAQAAAAAAAAB-_lPPvQMoSnSJMG0i-f4WEA , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,005] INFO Submitting put request correlationId 16, blob id AAYQAv____8AAQAAAAAAAALgA7Dmj4g-Qa2ilxHxltHuCQ , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 11 AAYQAv____8AAQAAAAAAAAIvYd-809X8S5OJ5OUXE9amLA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,417] INFO received success response and decoded response successfully for correlation id 13 AAYQAv____8AAQAAAAAAAALB72n194MWSz2bLaq2LRCPDA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 14 AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n```\n\nWe can also exit the process using CTRL+C command for both the tests\n\nFinal output for the both tests is network client related metrics\n\n```\n[2025-02-06 21:11:43,235] INFO HTTP2 error count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 dropped request count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 send Mean rate 8.958314373706395 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median read time 383.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median acquire time, 171.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n```","isInternal":false,"tokens":3095,"sizeBytes":11817}],"systemPromptSnippet":"<yakaai_skills repo=\"linkedin/ambry\">\n<!-- File: CLAUDE.md (Tokens: ~3436 | Category: claude-rule) -->\n# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n---\n\n## 1. LLM Behavioral Guidelines\n\nBehavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.\n\n**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.\n\n### Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them — don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\nStorage systems punish assumptions. Read the relevant source files, understand the data flow, and trace the call path before proposing changes. If you are unsure about a component's behavior, read its implementation and tests before modifying it.\n\n### Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n### Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it — don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n### Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" → \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" → \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" → \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] → verify: [check]\n2. [Step] → verify: [check]\n3. [Step] → verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n\n**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.\n\n**Use subagents by default** where possible for parallel research, exploration, and independent subtasks.\n\n---\n\n## 2. PR Guidelines\n\nPRs must be logically isolated and easy to review: small, coherent diffs with one purpose per PR. Avoid drive-by refactors.\n\nPR descriptions **must** include a `## Testing Done` section (not `## Testing`).\n\nAim for as close to 100% unit test coverage as practical for touched logic. Cover risky paths and edge cases.\n\n---\n\n## 3. Ambry System-Level Invariants\n\n### Durability Paranoia\n\nAmbry is a production blob storage system. **\"You must never lose data\"** is a non-negotiable invariant. Every PR must include a durability risk analysis and tests/mitigations.\n\n**Tradeoff priority order** (when forced to choose):\n1. **Durability** — Data must never be lost or silently corrupted\n2. **Security** — ACLs, encryption, and auth must not be weakened\n3. **Availability** — The system must remain serving\n4. **Operability** — Changes must be observable and debuggable\n5. **Efficiency** — Resource usage matters but not at the cost of the above\n6. **Performance** — Latency and throughput matter but not at the cost of the above\n\nNo repo-specific guideline can override the durability invariant.\n\n### Engineering Principles\n\n- Explicit invariants and clear failure-mode handling\n- Safe retries and idempotency — operations must be safe to retry after partial failure\n- Defensive coding — validate inputs at system boundaries, handle nulls explicitly\n- Code should be readable by humans and LLM agents (future maintainers)\n- Strong bias toward correctness and operability over cleverness\n\n### Durability Risk Checklist\n\nApply to every PR. If any item is checked, the PR requires extra scrutiny and explicit testing.\n\n- [ ] **Write path** — Does this change touch PUT, named blob PUT, TTL update, or delete operations?\n- [ ] **Metadata** — Does this change modify how blob metadata is stored, read, or indexed?\n- [ ] **Ordering / Atomicity** — Could operations complete partially, leaving inconsistent state?\n- [ ] **Callback semantics** — Could success be reported to a client before data is durable?\n- [ ] **Resource cleanup** — Does this modify ByteBuf release, Closeable.close(), or stream lifecycle?\n- [ ] **Retries / Idempotency** — Are retries safe? Could a retry cause duplicate writes or skipped deletes?\n- [ ] **Partial failures** — What happens if this operation fails midway? Is the system left in a recoverable state?\n- [ ] **Corruption handling** — If data corruption is detected (e.g., MD5 mismatch), is it logged and surfaced, not silently ignored?\n\n### Java Review Standards\n\n- **Do** use SLF4J parameterized logging: `logger.info(\"Processing blob {} for container {}\", blobId, containerName)`. **Avoid** string concatenation in log statements.\n- **Do** use correct log levels: `error` = unexpected failures requiring attention, `warn` = recoverable issues, `info` = important state changes, `debug`/`trace` = detailed debugging. **Avoid** logging expected conditions at `error` or `warn`.\n- **Do** ensure `LoggerFactory.getLogger()` references the enclosing class. **Avoid** copy-pasting logger declarations — this is a recurring source of bugs.\n- **Do** add explicit null checks before dereferencing, especially for values from `RestRequest.getArgs()` and enum comparisons. **Avoid** `enumValue != SOME_ENUM` without a null guard — it returns `true` when `enumValue` is `null`.\n- **Do** delete unused code outright. **Avoid** commenting out code blocks. Use version control to recover old code; use config flags for runtime control.\n- **Do** make defensive copies of mutable arrays (e.g., `byte[]`) in exactly one place — either the builder setter or the constructor. **Avoid** copying in both.\n- **Do** catch the narrowest exception type possible and log with context: `logger.error(\"Failed to process blob {}\", blobId, e)`. **Avoid** catching generic `Exception` without re-throwing.\n- **Do** use import statements. **Avoid** fully qualified names in code, unless there is a naming collision.\n- **Do** release Netty `ByteBuf` in `finally` blocks. Implement `Closeable` for services holding network clients or other resources. Verify `refCnt() > 0` before calling `release()`.\n- **Do** mark static mutable fields as `volatile` or use proper synchronization. **Avoid** unprotected static mutable state — services are long-running and concurrent.\n- **Do** extract non-obvious numeric literals to named constants with explanatory comments. **Avoid** magic numbers like `9223372036854775806L` (use `Long.MAX_VALUE - 1` with a comment).\n- **Do** annotate with `@VisibleForTesting` when widening field or method visibility solely for test access. **Avoid** silently making fields package-private without documenting why.\n\n### Documentation and Test Update Requirement\n\nEvery PR that changes behavior, APIs, config, or architecture must update relevant documentation AND tests.\n\n---\n\n## 4. Repo-Specific Guidelines\n\n### Build and Development Commands\n\n```bash\n# Build\n./gradlew build -x test                       # Build without tests\n\n# Test\n./gradlew test                                # Run all tests\n\n# Run specific test class\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest\"\n\n# Run specific test method\n./gradlew test --tests \"com.github.ambry.clustermap.AmbryReplicaSyncUpManagerTest.bootstrapBasicTest\"\n```\n\n### Architecture Overview\n\nAmbry is an immutable distributed blob storage system. See [GitHub](https://github.com/linkedin/ambry) for details.\n\n#### Key Modules\n\n| Module | Purpose |\n|--------|---------|\n| `ambry-api/` | Public interfaces and configs |\n| `ambry-clustermap/` | Cluster topology, Helix integration, state management |\n| `ambry-replication/` | Cross-datacenter replication engine |\n| `ambry-router/` | Client-side routing and operation tracking |\n| `ambry-server/` | Storage data layer |\n| `ambry-store/` | On-disk blob store implementation |\n| `ambry-frontend/` | REST API gateway |\n| `ambry-test-utils/` | Shared test mocks and utilities |\n\n#### Package Convention\n\n- `com.github.ambry.*` — All Ambry classes\n\n### CRITICAL: Schema Change Safety\n\nAny change to a schema — whether it is a JSON-serialized Java class, a MySQL DDL, a Protobuf definition, a ZooKeeper node structure, or a Helix property store format — must be treated as a cross-cluster, cross-fabric change. All Ambry clusters across all fabrics read shared data. A backward-incompatible change poisons every cluster simultaneously.\n\n**General rules for ALL schema changes:**\n\n1. **Every schema change MUST be backward and forward compatible.** Backward compatibility means new code can read data written by older schemas. Forward compatibility means old code can safely ignore unknown fields/columns written by newer schemas.\n\n2. **Deployment ordering:** First deploy the code that can **read** the new schema to ALL hosts. Only then deploy the code that **writes** the new schema. Never write new fields/columns before all readers can handle them.\n\n3. **Any PR that changes a schema MUST include compatibility tests:**\n   - Forward compatibility test: read data written by the **new** schema (with new fields/columns) using the **old** code path — old readers must tolerate new data\n   - Backward compatibility test: read data written by the **old** schema (without new fields/columns) using the **new** code path — new readers must handle missing data with sensible defaults\n\n4. **Key shared schemas to watch:**\n   - `Account`, `Container` (JSON in MySQL `account_metadata`)\n   - Any class serialized to ZooKeeper or Helix property stores\n   - MySQL DDL schemas (column additions, type changes, index changes)\n   - Any wire protocol or RPC message format changes\n\n5. **Fail-fast on startup, degrade gracefully at runtime:**\n   - If shared metadata (e.g., account data) cannot be loaded at startup, the frontend MUST fail to start — not serve with an empty cache\n   - At runtime, not all records are safe to skip. Records affecting access control, quotas, routing, or ownership metadata must fail the operation or raise an alert if they cannot be loaded — silently skipping them could change correctness or authorization. For non-critical record types, skip the failing record and continue, but emit structured logs and metrics with record identifiers so the issue is detectable.\n\n**Jackson-specific rules (for shared-store DTOs):**\n\n6. **Every DTO deserialized from a shared store (MySQL JSON metadata, ZooKeeper, Helix property stores) MUST have `@JsonIgnoreProperties(ignoreUnknown = true)`** at the class level. Without it, adding a new field to the JSON will cause deserialization failures on any host running older code that does not recognize the field. This requirement applies to shared-store DTOs, not necessarily to purely internal config objects or in-process-only types.\n\n7. **When reviewing PRs that touch Jackson-deserialized shared-store DTOs:**\n   - Check: Does the class have `@JsonIgnoreProperties(ignoreUnknown = true)`? If not, **block the PR**.\n   - Check: Are there tests for unknown field tolerance? If not, **request them**.\n   - Test round-trip: serialize → inject unknown fields into the JSON string → deserialize → verify no data loss of known fields.\n   - For critical records (account/container metadata), also test that required fields are validated — removing or renaming a required field should fail deserialization or be caught by validation logic.\n\n### Error Handling\n\n- **Do** wrap errors in `RestServiceException` with appropriate `RestServiceErrorCode`.\n- **Do** check for errors first in callback handlers before processing results.\n- **Do** guard against NPE in validation callbacks: if primaryResult or secondaryResult is null, record a metric and return early.\n\n### Style Rules\n\n- **Imports:** Static imports first, then java.*, third-party, com.github.ambry.*. No wildcards except for static utility constants.\n- **Line length:** ~120 characters max.\n- **Annotations:** Each on its own line, `@Override` first.\n- **Naming:** PascalCase classes, camelCase methods/variables, UPPER_SNAKE_CASE constants, `{Name}Factory`/`{Name}Config`/`{Name}Test` suffixes.\n- **Test methods:** `test{Scenario}` (e.g., `testSecondaryFailure`, `testResourceCleanup`).\n\n### Must-Not-Do List\n\n1. **Never change public APIs** (REST endpoints, account/container schemas) unless explicitly instructed\n2. **Never change default config values** without explicit instruction — this affects all deployments\n3. **Never remove or weaken existing tests** unless instructed\n4. **Never use `jdk.internal.*` imports** — they are unstable and fail across JDK versions\n5. **Never use `Thread.sleep()` for test synchronization** — use `CountDownLatch`, `CompletableFuture.get(timeout, ...)`, or polling with timeout\n6. **Never use `Random` for security-sensitive operations** — use `java.security.SecureRandom`\n7. **Never buffer entire blobs in memory** — use streaming patterns throughout\n8. **Never widen field visibility without `@VisibleForTesting`** annotation\n\n### Testing Conventions\n\n**Framework:** JUnit 4 (`@Test`, `@Before`, `@After`), Mockito.\n\n**Key patterns:**\n- **Do** use `InMemAccountService` for account/container tests, not mocks.\n- **Do** verify metric increments against the **actual** Metrics instance used by the class under test. **Avoid** creating a separate Metrics instance in the test — it won't observe real increments.\n- **Do** use `CompletableFuture` and `future.get(timeout, unit)` for async callback testing.\n- **Do** verify `ByteBuf.refCnt()` drops to 0 in resource cleanup tests.\n- **Do** reset static mutable state in `@Before` or `@After` methods to prevent test interdependencies.\n\n**Test file layout:**\n```\n{module}/src/test/java/       # Unit tests, mirroring main package structure\n{module}/src/test/resources/  # Test resources\n```\n\n---\n\n## 5. Review Culture\n\n### Copy-paste vigilance for constants\n\n- **Do** verify that when duplicating a constant declaration, BOTH the variable name AND the string literal value are updated.\n- **Avoid** the pattern where the variable name is changed but the value still points to the original.\n\n### Async callback race conditions\n\n- **Do** call `primaryFuture.get(timeout, ...)` before reading primary data from request args in secondary comparison callbacks.\n- **Avoid** assuming request args are populated before the primary callback has run — if a secondary call completes first, you get stale/null data.\n\n### Test metric verification\n\n- **Do** spy on or retrieve the actual `*Metrics` instance wired into the class under test (via `Whitebox.getInternalState` or the shared `MetricRegistry`).\n- **Avoid** creating a fresh `*Metrics` instance in the test and asserting against its counters — it won't observe real increments.\n\n<!-- File: ambry-tools/src/main/java/com/github/ambry/Readme.md (Tokens: ~3095 | Category: plugin-manifest) -->\n## Terminologies\n\nHardwareLayout   - Has information about the nodes available in the cluster. disks for those nodes and their states\nPartitionLayout  - Has information about the partitions and its replicas\nBlobid           - BlobId of the blob/object which was generated by ambry on put\n\n\n## Set up\nTo execute tools, ./gradlew allJar in the root directory.\nThis will generate ambry.jar in \"target\" folder under root directory.\nThis jar will assist using these tools.\n\n\n## DumpLogTool, DumpIndexTool, DumpReplicaTokenTool and DumpDataTool\nThese tools are used to dump the data parsing the storage files.\n\n#### Operations supported\n* Dumping log segment\n* Dumping index / segments\n* Dumping replicaToken\n* Compare index entries to log contents\n\n## BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas.\n\n#### Operations supported:\n* Get blob from all replicas and compare the record across replicas for a list of blob ids.\n* Get blob from all replicas in a datacenter and compare the record across replicas for a list of blob ids.\n* Get blob from a specific replica and ensure that it deserializes correctly for a list of blob ids.\n\n## ServerAdminTool\nThis tool can be used to operate on a single ambry server. Please refer to the config documentation and the class \ndocumentation for usage details.\n\n#### Operations supported\n* GetBlobProperties\n* GetBlobUserMetadata\n* GetBlob\n* TriggerCompaction\n\n#### Dump Index\nThis tool can be used to dump or verify whole indexes or index segments. The operations can also be done on a subset of blobs.\nPlease refer to the file for more details on what the operations are and the configuration supported.\n#### Operations supported\n* DumpIndex\n* DumpIndexSegment\n* VerifyIndex\n\n### Dumping Log\n\n#### Dump log file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nsilent=false\n```\n\n#### Dump log file with throttling\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobs.per.sec=100\nsilent=false\n```\n\n#### Dump log starting at offset x and ending at offset y\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nlog.start.offset=x\nlog.end.offset=y\nsilent=false\n```\n\n#### Dump log filtering with a set of blobs\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpLogTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[LogFile]\nblobId.list=blobId1,blobId2,blobId3\nsilent=false\n```\n\n### Comparison operations\n\n#### Comparing index entries to log entries(or in other words, trying to read a log record based on index info for\n#### every entry in the index)\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareIndexToLog\nfile.to.read=[indexFile]\n```\n\n#### Comparing all index entries for a replica to log entries\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpDataTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\ntype.of.operation=CompareReplicaIndexesToLog\nreplica.root.directory=[replicaRootDirecotry]\nindex.files.per.sec=1 // number of index files to process per sec\n```\n\n### Dumping ReplicaToken file\n```bash\njava -cp ambry.jar com.github.ambry.store.DumpReplicaTokenTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\nfile.to.read=[replicaTokenFile]\n```\n\n### BlobValidator\nThis tool can be used validate blobs in a single replica, in all replicas in a datacenter or across all replicas. Please\nrefer to the config documentation and the class  documentation for usage details.\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ServerAdminTool\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ServerAdminTool --propsFile [props file path]\n```\n\n### ConsistencyCheckerTool\n\n#### Check for consistency among replicas for a partition\n```bash\njava -cp ambry.jar com.github.ambry.store.ConsistencyCheckerTool  --propsFile [Config file path]\n//Contents of config file\nhardware.layout.file.path=[HardwareLayoutFile]\npartition.layout.file.path=[PartitionLayoutFile]\npartition.root.directory=[Root directory which contains one directory per replica. Each replica's directory is expected \nto contain all index files for the respective replica. Diretory's name is considered to be replica name for logging \npurposes]\n```\n\n\n```\nRoot Directory for Partition should contains N sub-directories pertaining to N replicas. Each sub-directory's name\nshould be the same as name of the replica. Each sub directory(replica) in turn should have all the index segments within it.\n\nA sample structure of a root Directory:\n\nrootDir\n  ela4-app8763.prod.linkedin.com\n    0_index\n    10264262199_index\n    10821344563_index\n  lva1-app6774.prod.linkedin.com\n    0_index\n    10267816698_index\n    10892574781_index\n  lva1-app6775.prod.linkedin.com\n    0_index\n    10266323380_index\n    10888037904_index\n```\n\n\n###ConcurrencyTestTool\n####Sample test run\n\nEnsure that ambry servers in the HardwareLayout are up running\n\n```bash\njava -cp ambry.jar com.github.ambry.tools.admin.ConcurrencyTestTool --hardwareLayout [HardwareLayoutFile]\n --partitionLayout [PartitionLayoutFile] --putGetHelperFactory com.github.ambry.tools.admin.RouterPutGetHelperFactory\n --maxParallelPutCount 2 --parallelGetCount 4 --burstCountForGet 4 --maxGetCountPerBlob 10\n --totalPutBlobCount 1000 --maxBlobSizeInBytes 1000 --minBlobSizeInBytes 100 --routerPropsFilePath [Path to router props\n file] --deleteAndValidate true\n```\n\n### ServerPerformanceTool\n\nTesting for GET blobs\nThis tool will read the log file for blobs and keep making get requests for each blob until time out\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance  -Dlog4j2.configurationFile=file:../config/log4j2.xml --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=GET_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.get.test.blob.id.file.path=[File which contains blobids separated by new line]\n```\nWe will be able to see logs \n\n```\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 4 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:41,227] INFO submitting the blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA to network queue correlation id 6 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,654] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 1 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,653] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 5 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n[2025-02-06 20:57:45,655] INFO blob id AAYQAgBoAAMAAQAAAAAAAAC5jpIA0niiQ62hJkNsJaxCFA blob size 1000000  correlation id 3 (com.github.ambry.tools.perf.serverperf.GetLoadProducerConsumer)\n\n```\n\n\nTesting for PUT blobs\n\nThis tool will try to put blobs to one partition per disk of the target host in round-robin format until\ntime out or it has reached limit of total number of bytes\n```bash\njava -cp ambry.jar com.github.ambry.tools.perf.serverperf.ServerPerformance -Dlog4j2.configurationFile=file:../config/log4j2.xml  --propsFile [Config file path]\n//Contents of config file\nserver.performance.test.type=PUT_BLOB\nserver.performance.hardware.layout.file.path=[HardwareLayoutFile]\nserver.performance.partition.layout.file.path=[PartitionLayoutFile]\nserver.performance.max.parallel.requests=[number of paraller requests we want to keep]\nserver.performance.hostname=[host name to which we want to make requests]\nserver.performance.port=[port of the host above]\nserver.performance.time.out.seconds=[time after which tool will stop]\nserver.performance.operations.time.out.sec=[time after which requests will be timedout if response not received]\nserver.performance.put.test.data.limit.bytes=[maximum number of bytes which we want to send from the tool]\n```\nWe will be able to see logs\n\n```\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 14, blob id AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,004] INFO Submitting put request correlationId 15, blob id AAYQAv____8AAQAAAAAAAAB-_lPPvQMoSnSJMG0i-f4WEA , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,005] INFO Submitting put request correlationId 16, blob id AAYQAv____8AAQAAAAAAAALgA7Dmj4g-Qa2ilxHxltHuCQ , blob size 4096 (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 11 AAYQAv____8AAQAAAAAAAAIvYd-809X8S5OJ5OUXE9amLA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,417] INFO received success response and decoded response successfully for correlation id 13 AAYQAv____8AAQAAAAAAAALB72n194MWSz2bLaq2LRCPDA (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n[2025-02-06 21:11:42,416] INFO received success response and decoded response successfully for correlation id 14 AAYQAv____8AAQAAAAAAAADPyZlGH92nT_-MtTuwjv1fTw (com.github.ambry.tools.perf.serverperf.PutLoadProducerConsumer)\n```\n\nWe can also exit the process using CTRL+C command for both the tests\n\nFinal output for the both tests is network client related metrics\n\n```\n[2025-02-06 21:11:43,235] INFO HTTP2 error count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 dropped request count 0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,236] INFO HTTP2 send Mean rate 8.958314373706395 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median read time 383.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n[2025-02-06 21:11:43,251] INFO HTTP2 stream median acquire time, 171.0 (com.github.ambry.tools.perf.serverperf.ServerPerformance)\n```\n</yakaai_skills>"}