{"owner":"uber","repo":"kraken","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md"],"skills":{"CLAUDE.md":"# Kraken Development Guide for Claude\n\n## Project Overview\n\nKraken is a P2P-powered Docker registry designed for scalability and availability in hybrid cloud environments. It distributes Docker images using a BitTorrent-inspired protocol with a tracker-coordinated peer network. Built in Go, it has been in production at Uber since 2018, distributing over 1 million blobs per day.\n\n**Key technologies**: Go 1.24+, Docker, Containerd, P2P networking, pluggable storage backends (S3, GCS, ECR, HDFS)\n\n## Quick Start\n\n### Build & Test\n```bash\n# Install dependencies\nmake vendor\n\n# Build all binaries (uses Docker for cross-compilation on macOS)\nmake bins\n\n# Build Docker images\nmake images\n\n# Run unit tests\nmake unit-test\n\n# Run integration tests (Python-based)\nmake integration\n\n# Start local development cluster (requires Docker-for-Mac)\nmake devcluster\n```\n\n### Git Hooks\nInstall pre-commit hooks that run `golangci-lint` automatically:\n```bash\nmake install-hooks\n```\n\n## Architecture Overview\n\nKraken consists of five main components forming a distributed system:\n\n- **Agent** - Runs on every host, implements Docker registry API, P2P client/server\n- **Origin** - Dedicated seeders, stores blobs backed by pluggable storage, forms hash ring\n- **Tracker** - Orchestrates peer connections, tracks content availability, forms hash ring\n- **Proxy** - Handles uploads, routes to origins via hash ring, publishes tags\n- **Build-Index** - Tag→digest mapping, powers cross-cluster replication, forms hash ring\n\n**Key concepts:**\n- Components use **self-healing hash rings** for distribution and HA\n- **P2P protocol** is custom (BitTorrent-inspired but optimized for stable clusters)\n- **Pseudo-random regular graphs** ensure high connectivity and fast distribution\n- See docs/ARCHITECTURE.md for details\n\n## Repository Structure\n\n```\n/agent           - Agent service (runs on every host)\n/build-index     - Build-index service (tag storage and replication)\n/origin          - Origin service (blob seeding)\n/proxy           - Proxy service (upload handling)\n/tracker         - Tracker service (peer coordination)\n/lib             - Shared libraries\n  /backend       - Storage backend implementations (S3, GCS, etc.)\n  /hashring      - Consistent hashing and hash ring implementation\n  /torrent       - P2P protocol implementation\n  /store         - Local storage abstractions\n  /dockerregistry - Docker registry API handling\n/tools           - CLI tools (puller, visualization, etc.)\n/docker          - Dockerfiles for each component\n/examples        - Example deployments (k8s, devcluster)\n/docs            - Documentation\n```\n\n## Code Conventions\n\n**Follow docs/STYLEGUIDE.md strictly.** Key points:\n\n- **Line length**: Code <100 chars, comments <80 chars\n- **Variable naming**: Follow Go naming conventions (short, contextual names)\n- **Comments**: Proper grammar, capitalization, punctuation. Avoid obvious/redundant comments.\n- **Whitespace**: Be conservative with blank lines, avoid vertical clutter\n- **Function signatures**: Break long signatures with one param per line\n- **Testing**: Use testify/require for assertions, suffix test files with `_test.go`\n\n### Long Function Example\n```go\nfunc Foo(\n    bar int,\n    baz bool,\n    blah []int) (string, error) {\n\n    ...\n}\n```\n\n## Testing\n\n- **Unit tests**: `make unit-test` - Fast, required before commits\n- **Integration tests**: `make integration` - Slower, Python-based\n- **Test tags**: Unit tests use `--tags \"unit\"`\n- **Coverage**: Unit tests generate `coverage.txt`\n\nAll new features and bug fixes must include tests.\n\n## Important Notes\n\n### Build System\n- **macOS**: Uses Docker for cross-compilation (cgo/sqlite3 limitation)\n- **Native tools**: puller, reload, visualization can build natively on macOS\n- **Linux bins**: agent, origin, tracker, proxy, build-index require Linux build\n- Binaries are built inside `golang:1.24.0` container\n\n### Hash Rings\n- Multiple components (origin, tracker, build-index) use hash rings for sharding\n- Hash rings are **self-healing** - handle member failures gracefully\n- Critical for horizontal scaling - understand before modifying\n\n### Storage Backends\n- Pluggable architecture in `lib/backend`\n- Each backend must implement common interface\n- See `lib/backend/shadowbackend` for example proxy backend\n- See `lib/backend/sqlbackend` for SQL-backed implementation\n\n### P2P Protocol\n- Custom protocol in `lib/torrent`\n- Optimized for stable data center environments (not adversarial like BitTorrent)\n- Tracker orchestrates connections but doesn't transfer data\n- Peers negotiate directly for chunks\n\n### Performance\n- Blob size limit: 20G recommended (theoretically unbounded)\n- Tag mutation supported but with caveats (Nginx caching, replication delays)\n- System designed for immutable content (unique tags)\n\n## Common Development Tasks\n\n### Adding a new feature\n1. Read relevant code first - understand patterns before changing\n2. Check docs/ARCHITECTURE.md for architectural context\n3. Follow existing patterns in the codebase\n4. Add tests alongside implementation\n5. Run `make unit-test` before committing\n6. Lint automatically runs via git hooks if installed\n\n### Debugging\n- Use `make devcluster` for local testing\n- Logs are your friend - check component logs\n- Visualization tool available: `tools/bin/visualization`\n\n#### Mutex contention profiling\n\nAll services support `--mutex-profile-fraction=N` (default 0, disabled). When\nenabled, ~1/N mutex contention events are recorded and exposed at\n`/debug/pprof/mutex`.\n\n```bash\n# Start any service with profiling enabled (1 = record every event)\nkraken-origin --mutex-profile-fraction=1 [other flags]\n\n# View the profile as text\ncurl \"http://localhost:<port>/debug/pprof/mutex?debug=1\"\n\n# Analyze interactively\ngo tool pprof http://localhost:<port>/debug/pprof/mutex\n```\n\nThe devcluster already passes `--mutex-profile-fraction=1` to all services.\nDevcluster ports: proxy=15000, origin=15002, tracker=15003, build-index=15004,\nagent-1=16002, agent-2=17002.\n\n### Modifying storage backends\n- Look at existing implementations in `lib/backend`\n- Maintain interface compatibility\n- Consider both read and write paths\n- Test with integration tests\n\n## Working with Claude\n\n### Commit Practices\n- Run tests before committing: `make unit-test`\n- Follow git hooks (golangci-lint will auto-run if installed)\n- Write clear commit messages that explain \"why\" not \"what\"\n- Don't commit until tests pass and code is linted\n\n### Code Changes\n- Always read files before modifying them\n- Prefer editing existing files over creating new ones\n- Don't add unnecessary features or abstractions\n- Keep changes focused on the task at hand\n- Follow STYLEGUIDE.md for all Go code\n\n### Questions & Alignment\n- Check ARCHITECTURE.md before major structural changes\n- For architectural decisions, ask before implementing\n- When multiple approaches exist, present options\n\n### Don't Do This\n- Don't create documentation files unless requested\n- Don't add features beyond what's asked\n- Don't use destructive git operations without asking\n- Don't skip tests (\"I'll add them later\")\n- Don't guess at build commands - use the Makefile\n\n## References\n\n- [Architecture](docs/ARCHITECTURE.md) - System design and component interactions\n- [Configuration](docs/CONFIGURATION.md) - How to configure each component\n- [Style Guide](docs/STYLEGUIDE.md) - Go code style requirements\n- [Contributing](docs/CONTRIBUTING.md) - Fork-and-pull workflow\n- [K8s Setup](examples/k8s/README.md) - Kubernetes deployment\n- [Devcluster](examples/devcluster/README.md) - Local development setup\n\n## Contact\n\n- GitHub Issues: https://github.com/uber/kraken/issues\n"},"files":{"CLAUDE.md":"# Kraken Development Guide for Claude\n\n## Project Overview\n\nKraken is a P2P-powered Docker registry designed for scalability and availability in hybrid cloud environments. It distributes Docker images using a BitTorrent-inspired protocol with a tracker-coordinated peer network. Built in Go, it has been in production at Uber since 2018, distributing over 1 million blobs per day.\n\n**Key technologies**: Go 1.24+, Docker, Containerd, P2P networking, pluggable storage backends (S3, GCS, ECR, HDFS)\n\n## Quick Start\n\n### Build & Test\n```bash\n# Install dependencies\nmake vendor\n\n# Build all binaries (uses Docker for cross-compilation on macOS)\nmake bins\n\n# Build Docker images\nmake images\n\n# Run unit tests\nmake unit-test\n\n# Run integration tests (Python-based)\nmake integration\n\n# Start local development cluster (requires Docker-for-Mac)\nmake devcluster\n```\n\n### Git Hooks\nInstall pre-commit hooks that run `golangci-lint` automatically:\n```bash\nmake install-hooks\n```\n\n## Architecture Overview\n\nKraken consists of five main components forming a distributed system:\n\n- **Agent** - Runs on every host, implements Docker registry API, P2P client/server\n- **Origin** - Dedicated seeders, stores blobs backed by pluggable storage, forms hash ring\n- **Tracker** - Orchestrates peer connections, tracks content availability, forms hash ring\n- **Proxy** - Handles uploads, routes to origins via hash ring, publishes tags\n- **Build-Index** - Tag→digest mapping, powers cross-cluster replication, forms hash ring\n\n**Key concepts:**\n- Components use **self-healing hash rings** for distribution and HA\n- **P2P protocol** is custom (BitTorrent-inspired but optimized for stable clusters)\n- **Pseudo-random regular graphs** ensure high connectivity and fast distribution\n- See docs/ARCHITECTURE.md for details\n\n## Repository Structure\n\n```\n/agent           - Agent service (runs on every host)\n/build-index     - Build-index service (tag storage and replication)\n/origin          - Origin service (blob seeding)\n/proxy           - Proxy service (upload handling)\n/tracker         - Tracker service (peer coordination)\n/lib             - Shared libraries\n  /backend       - Storage backend implementations (S3, GCS, etc.)\n  /hashring      - Consistent hashing and hash ring implementation\n  /torrent       - P2P protocol implementation\n  /store         - Local storage abstractions\n  /dockerregistry - Docker registry API handling\n/tools           - CLI tools (puller, visualization, etc.)\n/docker          - Dockerfiles for each component\n/examples        - Example deployments (k8s, devcluster)\n/docs            - Documentation\n```\n\n## Code Conventions\n\n**Follow docs/STYLEGUIDE.md strictly.** Key points:\n\n- **Line length**: Code <100 chars, comments <80 chars\n- **Variable naming**: Follow Go naming conventions (short, contextual names)\n- **Comments**: Proper grammar, capitalization, punctuation. Avoid obvious/redundant comments.\n- **Whitespace**: Be conservative with blank lines, avoid vertical clutter\n- **Function signatures**: Break long signatures with one param per line\n- **Testing**: Use testify/require for assertions, suffix test files with `_test.go`\n\n### Long Function Example\n```go\nfunc Foo(\n    bar int,\n    baz bool,\n    blah []int) (string, error) {\n\n    ...\n}\n```\n\n## Testing\n\n- **Unit tests**: `make unit-test` - Fast, required before commits\n- **Integration tests**: `make integration` - Slower, Python-based\n- **Test tags**: Unit tests use `--tags \"unit\"`\n- **Coverage**: Unit tests generate `coverage.txt`\n\nAll new features and bug fixes must include tests.\n\n## Important Notes\n\n### Build System\n- **macOS**: Uses Docker for cross-compilation (cgo/sqlite3 limitation)\n- **Native tools**: puller, reload, visualization can build natively on macOS\n- **Linux bins**: agent, origin, tracker, proxy, build-index require Linux build\n- Binaries are built inside `golang:1.24.0` container\n\n### Hash Rings\n- Multiple components (origin, tracker, build-index) use hash rings for sharding\n- Hash rings are **self-healing** - handle member failures gracefully\n- Critical for horizontal scaling - understand before modifying\n\n### Storage Backends\n- Pluggable architecture in `lib/backend`\n- Each backend must implement common interface\n- See `lib/backend/shadowbackend` for example proxy backend\n- See `lib/backend/sqlbackend` for SQL-backed implementation\n\n### P2P Protocol\n- Custom protocol in `lib/torrent`\n- Optimized for stable data center environments (not adversarial like BitTorrent)\n- Tracker orchestrates connections but doesn't transfer data\n- Peers negotiate directly for chunks\n\n### Performance\n- Blob size limit: 20G recommended (theoretically unbounded)\n- Tag mutation supported but with caveats (Nginx caching, replication delays)\n- System designed for immutable content (unique tags)\n\n## Common Development Tasks\n\n### Adding a new feature\n1. Read relevant code first - understand patterns before changing\n2. Check docs/ARCHITECTURE.md for architectural context\n3. Follow existing patterns in the codebase\n4. Add tests alongside implementation\n5. Run `make unit-test` before committing\n6. Lint automatically runs via git hooks if installed\n\n### Debugging\n- Use `make devcluster` for local testing\n- Logs are your friend - check component logs\n- Visualization tool available: `tools/bin/visualization`\n\n#### Mutex contention profiling\n\nAll services support `--mutex-profile-fraction=N` (default 0, disabled). When\nenabled, ~1/N mutex contention events are recorded and exposed at\n`/debug/pprof/mutex`.\n\n```bash\n# Start any service with profiling enabled (1 = record every event)\nkraken-origin --mutex-profile-fraction=1 [other flags]\n\n# View the profile as text\ncurl \"http://localhost:<port>/debug/pprof/mutex?debug=1\"\n\n# Analyze interactively\ngo tool pprof http://localhost:<port>/debug/pprof/mutex\n```\n\nThe devcluster already passes `--mutex-profile-fraction=1` to all services.\nDevcluster ports: proxy=15000, origin=15002, tracker=15003, build-index=15004,\nagent-1=16002, agent-2=17002.\n\n### Modifying storage backends\n- Look at existing implementations in `lib/backend`\n- Maintain interface compatibility\n- Consider both read and write paths\n- Test with integration tests\n\n## Working with Claude\n\n### Commit Practices\n- Run tests before committing: `make unit-test`\n- Follow git hooks (golangci-lint will auto-run if installed)\n- Write clear commit messages that explain \"why\" not \"what\"\n- Don't commit until tests pass and code is linted\n\n### Code Changes\n- Always read files before modifying them\n- Prefer editing existing files over creating new ones\n- Don't add unnecessary features or abstractions\n- Keep changes focused on the task at hand\n- Follow STYLEGUIDE.md for all Go code\n\n### Questions & Alignment\n- Check ARCHITECTURE.md before major structural changes\n- For architectural decisions, ask before implementing\n- When multiple approaches exist, present options\n\n### Don't Do This\n- Don't create documentation files unless requested\n- Don't add features beyond what's asked\n- Don't use destructive git operations without asking\n- Don't skip tests (\"I'll add them later\")\n- Don't guess at build commands - use the Makefile\n\n## References\n\n- [Architecture](docs/ARCHITECTURE.md) - System design and component interactions\n- [Configuration](docs/CONFIGURATION.md) - How to configure each component\n- [Style Guide](docs/STYLEGUIDE.md) - Go code style requirements\n- [Contributing](docs/CONTRIBUTING.md) - Fork-and-pull workflow\n- [K8s Setup](examples/k8s/README.md) - Kubernetes deployment\n- [Devcluster](examples/devcluster/README.md) - Local development setup\n\n## Contact\n\n- GitHub Issues: https://github.com/uber/kraken/issues\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# Kraken Development Guide for Claude\n\n## Project Overview\n\nKraken is a P2P-powered Docker registry designed for scalability and availability in hybrid cloud environments. It distributes Docker images using a BitTorrent-inspired protocol with a tracker-coordinated peer network. Built in Go, it has been in production at Uber since 2018, distributing over 1 million blobs per day.\n\n**Key technologies**: Go 1.24+, Docker, Containerd, P2P networking, pluggable storage backends (S3, GCS, ECR, HDFS)\n\n## Quick Start\n\n### Build & Test\n```bash\n# Install dependencies\nmake vendor\n\n# Build all binaries (uses Docker for cross-compilation on macOS)\nmake bins\n\n# Build Docker images\nmake images\n\n# Run unit tests\nmake unit-test\n\n# Run integration tests (Python-based)\nmake integration\n\n# Start local development cluster (requires Docker-for-Mac)\nmake devcluster\n```\n\n### Git Hooks\nInstall pre-commit hooks that run `golangci-lint` automatically:\n```bash\nmake install-hooks\n```\n\n## Architecture Overview\n\nKraken consists of five main components forming a distributed system:\n\n- **Agent** - Runs on every host, implements Docker registry API, P2P client/server\n- **Origin** - Dedicated seeders, stores blobs backed by pluggable storage, forms hash ring\n- **Tracker** - Orchestrates peer connections, tracks content availability, forms hash ring\n- **Proxy** - Handles uploads, routes to origins via hash ring, publishes tags\n- **Build-Index** - Tag→digest mapping, powers cross-cluster replication, forms hash ring\n\n**Key concepts:**\n- Components use **self-healing hash rings** for distribution and HA\n- **P2P protocol** is custom (BitTorrent-inspired but optimized for stable clusters)\n- **Pseudo-random regular graphs** ensure high connectivity and fast distribution\n- See docs/ARCHITECTURE.md for details\n\n## Repository Structure\n\n```\n/agent           - Agent service (runs on every host)\n/build-index     - Build-index service (tag storage and replication)\n/origin          - Origin service (blob seeding)\n/proxy           - Proxy service (upload handling)\n/tracker         - Tracker service (peer coordination)\n/lib             - Shared libraries\n  /backend       - Storage backend implementations (S3, GCS, etc.)\n  /hashring      - Consistent hashing and hash ring implementation\n  /torrent       - P2P protocol implementation\n  /store         - Local storage abstractions\n  /dockerregistry - Docker registry API handling\n/tools           - CLI tools (puller, visualization, etc.)\n/docker          - Dockerfiles for each component\n/examples        - Example deployments (k8s, devcluster)\n/docs            - Documentation\n```\n\n## Code Conventions\n\n**Follow docs/STYLEGUIDE.md strictly.** Key points:\n\n- **Line length**: Code <100 chars, comments <80 chars\n- **Variable naming**: Follow Go naming conventions (short, contextual names)\n- **Comments**: Proper grammar, capitalization, punctuation. Avoid obvious/redundant comments.\n- **Whitespace**: Be conservative with blank lines, avoid vertical clutter\n- **Function signatures**: Break long signatures with one param per line\n- **Testing**: Use testify/require for assertions, suffix test files with `_test.go`\n\n### Long Function Example\n```go\nfunc Foo(\n    bar int,\n    baz bool,\n    blah []int) (string, error) {\n\n    ...\n}\n```\n\n## Testing\n\n- **Unit tests**: `make unit-test` - Fast, required before commits\n- **Integration tests**: `make integration` - Slower, Python-based\n- **Test tags**: Unit tests use `--tags \"unit\"`\n- **Coverage**: Unit tests generate `coverage.txt`\n\nAll new features and bug fixes must include tests.\n\n## Important Notes\n\n### Build System\n- **macOS**: Uses Docker for cross-compilation (cgo/sqlite3 limitation)\n- **Native tools**: puller, reload, visualization can build natively on macOS\n- **Linux bins**: agent, origin, tracker, proxy, build-index require Linux build\n- Binaries are built inside `golang:1.24.0` container\n\n### Hash Rings\n- Multiple components (origin, tracker, build-index) use hash rings for sharding\n- Hash rings are **self-healing** - handle member failures gracefully\n- Critical for horizontal scaling - understand before modifying\n\n### Storage Backends\n- Pluggable architecture in `lib/backend`\n- Each backend must implement common interface\n- See `lib/backend/shadowbackend` for example proxy backend\n- See `lib/backend/sqlbackend` for SQL-backed implementation\n\n### P2P Protocol\n- Custom protocol in `lib/torrent`\n- Optimized for stable data center environments (not adversarial like BitTorrent)\n- Tracker orchestrates connections but doesn't transfer data\n- Peers negotiate directly for chunks\n\n### Performance\n- Blob size limit: 20G recommended (theoretically unbounded)\n- Tag mutation supported but with caveats (Nginx caching, replication delays)\n- System designed for immutable content (unique tags)\n\n## Common Development Tasks\n\n### Adding a new feature\n1. Read relevant code first - understand patterns before changing\n2. Check docs/ARCHITECTURE.md for architectural context\n3. Follow existing patterns in the codebase\n4. Add tests alongside implementation\n5. Run `make unit-test` before committing\n6. Lint automatically runs via git hooks if installed\n\n### Debugging\n- Use `make devcluster` for local testing\n- Logs are your friend - check component logs\n- Visualization tool available: `tools/bin/visualization`\n\n#### Mutex contention profiling\n\nAll services support `--mutex-profile-fraction=N` (default 0, disabled). When\nenabled, ~1/N mutex contention events are recorded and exposed at\n`/debug/pprof/mutex`.\n\n```bash\n# Start any service with profiling enabled (1 = record every event)\nkraken-origin --mutex-profile-fraction=1 [other flags]\n\n# View the profile as text\ncurl \"http://localhost:<port>/debug/pprof/mutex?debug=1\"\n\n# Analyze interactively\ngo tool pprof http://localhost:<port>/debug/pprof/mutex\n```\n\nThe devcluster already passes `--mutex-profile-fraction=1` to all services.\nDevcluster ports: proxy=15000, origin=15002, tracker=15003, build-index=15004,\nagent-1=16002, agent-2=17002.\n\n### Modifying storage backends\n- Look at existing implementations in `lib/backend`\n- Maintain interface compatibility\n- Consider both read and write paths\n- Test with integration tests\n\n## Working with Claude\n\n### Commit Practices\n- Run tests before committing: `make unit-test`\n- Follow git hooks (golangci-lint will auto-run if installed)\n- Write clear commit messages that explain \"why\" not \"what\"\n- Don't commit until tests pass and code is linted\n\n### Code Changes\n- Always read files before modifying them\n- Prefer editing existing files over creating new ones\n- Don't add unnecessary features or abstractions\n- Keep changes focused on the task at hand\n- Follow STYLEGUIDE.md for all Go code\n\n### Questions & Alignment\n- Check ARCHITECTURE.md before major structural changes\n- For architectural decisions, ask before implementing\n- When multiple approaches exist, present options\n\n### Don't Do This\n- Don't create documentation files unless requested\n- Don't add features beyond what's asked\n- Don't use destructive git operations without asking\n- Don't skip tests (\"I'll add them later\")\n- Don't guess at build commands - use the Makefile\n\n## References\n\n- [Architecture](docs/ARCHITECTURE.md) - System design and component interactions\n- [Configuration](docs/CONFIGURATION.md) - How to configure each component\n- [Style Guide](docs/STYLEGUIDE.md) - Go code style requirements\n- [Contributing](docs/CONTRIBUTING.md) - Fork-and-pull workflow\n- [K8s Setup](examples/k8s/README.md) - Kubernetes deployment\n- [Devcluster](examples/devcluster/README.md) - Local development setup\n\n## Contact\n\n- GitHub Issues: https://github.com/uber/kraken/issues\n","category":"root","tokens":1915}]}