{"owner":"grafana","repo":"pyroscope","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md"],"skills":{"AGENTS.md":"# Pyroscope - AI Agent Development Guide\n\nThis document provides context and guidance for AI coding assistants (Claude, Cursor, GitHub Copilot, etc.) working on the Pyroscope codebase.\n\n## What is Pyroscope?\n\nPyroscope is a horizontally scalable, highly available, multi-tenant continuous profiling aggregation system. \nIt's designed to store and query profiling data at scale, similar to how Prometheus works for metrics and Loki for logs.\n\n**Key Characteristics:**\n- Written in **Go**\n- Microservices-based architecture inspired by Cortex/Mimir/Loki\n- Stores profiling data in object storage (S3, GCS, Azure, etc.)\n- Multi-tenant by design\n\n## Architecture Overview\n\nPyroscope uses a **microservices architecture** where a single binary can run different components based on the `-target` parameter.\n\n### V1 Components\n\n**Write Path:**\n- **Distributor**: Receives profile ingestion requests, validates, and forwards to ingesters\n- **Ingester**: Stores profiles in memory, periodically flushes to disk as blocks, periodically uploads blocks to long-term object storage\n- **Compactor**: Merges blocks and removes duplicates\n\n**Read Path:**\n- **Query Frontend**: Entry point for queries, handles query splitting and caching\n- **Query Scheduler**: Manages query queue and ensures fair execution across tenants\n- **Querier**: Executes queries by fetching data from ingesters and store-gateways\n- **Store Gateway**: Indexes and serves blocks from long-term object storage\n\n### V2 Components\n\n**Write Path:**\n- **Distributor**: Receives profile ingestion requests, validates, and forwards to segment writers\n- **Segment Writer**: Writes block segments to long-term object storage and the block metadata to metastore\n- **Metastore**: Maintains an index for the block metadata and coordinates the block compaction process\n- **Compaction Worker**: Merges small segments into larger blocks\n\n**Read Path:**\n- **Query Frontend**: Entry point for queries, creates the query plan and executes it against query backends\n- **Query Backend**: Executes queries and merges query responses\n\n### Storage\n\n- **Block Format**: Profiles stored in Parquet tables, series data in a TSDB index, symbols in a custom format\n- **Multi-tenant**: Each tenant has isolated storage\n- **Object Storage**: Primary storage backend (S3, GCS, Azure, local filesystem)\n\n## Repository Structure\n\n```\n.\n├── cmd/\n│   ├── pyroscope/           # Main server binary\n│   └── profilecli/          # CLI tool for profile operations\n├── pkg/                     # Core Go packages\n│   ├── distributor/         # Distributor component\n│   ├── ingester/            # Ingester component\n│   ├── querier/             # Querier component\n│   ├── frontend/            # Query frontend component\n│   ├── compactor/           # Compactor component\n│   ├── metastore/           # Metadata component\n│   ├── phlaredb/            # V1 database storage engine\n│   ├── model/               # Data models and types\n│   ├── objstore/            # Object storage abstraction\n│   ├── api/                 # API definitions and handlers\n│   └── og/                  # Legacy code (original Pyroscope)\n├── ui/                      # React/TypeScript frontend (Vite) - has its own ui/CLAUDE.md\n├── api/                     # API definitions (protobuf, OpenAPI)\n├── docs/                    # Documentation\n├── operations/              # Deployment configs (jsonnet, helm)\n├── examples/                # Example applications and SDKs\n└── tools/                   # Development and build tools\n```\n\n## Tech Stack\n\n### Backend\n- **Language**: Go 1.25 (see `go.mod`)\n- **RPC**: gRPC with Connect protocol\n- **Storage**: Parquet, TSDB\n- **Hash Ring**: Consistent hashing with memberlist (gossip protocol)\n- **Observability**: Prometheus metrics, Structured logs, Distributed traces, pprof profiles\n\n### Frontend\nThe frontend lives in `ui/` and is a dependency-minimal rewrite of the old `public/app` UI.\n**`ui/CLAUDE.md` and `ui/DESIGN.md` are the authoritative guides — read them before touching the UI.** Summary:\n- **Stack**: React 19 + Vite 8 + TypeScript 5.9, package-managed with **Yarn 4 (Berry)** — use `yarn`, not `npm`\n- **No UI library**: styling via CSS custom properties / semantic tokens in `src/theme.css` (no Emotion, no Grafana UI)\n- **Resist new dependencies** — prefer a small local implementation; minimizing the dependency surface is the whole point of the rewrite\n\n### Testing\n- **Go**: Standard `testing` package, testify for assertions\n- **Frontend**: Vitest (`yarn test` from `ui/`)\n\n## Development Workflow\n\n### Setup & Build\n\n```bash\n# Prerequisites: Go 1.25, Docker, Node, Yarn 4 (Berry).\n# All other build tools auto-download to .tmp/bin/\n\n# Build backend\nmake go/bin\n\n# Run tests\nmake go/test\n\n# Frontend dev (run from ui/): Vite dev server on :5173, proxies API to :4040\ncd ui && yarn install && yarn dev\n# Production frontend build (Docker -> ui/dist; required before `make build`):\nmake frontend/build\n\n# Docker image\nmake GOOS=linux GOARCH=amd64 docker-image/pyroscope/build\n```\n\n### Code Generation\n\n**IMPORTANT**: After changing protobuf, configs, or flags:\n```bash\nmake generate\n```\nCommit the generated files with your changes.\n\n### Running Locally\n\n```bash\n# Run all components in monolithic mode with embedded Grafana\ngo run ./cmd/pyroscope --target all,embedded-grafana\n# Pyroscope: http://localhost:4040\n# Grafana: http://localhost:4041\n\n# Run with V2 architecture (segment writers, query backend, symbolizer)\n# -symbolizer.enabled=true opts into symbolization (off by default).\ngo run ./cmd/pyroscope -symbolizer.enabled=true\n```\n\n## Code Style & Conventions\n\n### Go Code\n\n1. **Imports**: Three groups separated by blank lines:\n   ```go\n   import (\n       // Standard library\n       \"context\"\n       \"fmt\"\n\n       // Third-party packages\n       \"github.com/prometheus/client_golang/prometheus\"\n       \"go.uber.org/atomic\"\n\n       // Internal packages\n       \"github.com/grafana/pyroscope/v2/pkg/model\"\n       \"github.com/grafana/pyroscope/v2/pkg/objstore\"\n   )\n   ```\n\n2. **Formatting**: Use `golangci-lint` (run via `make lint`)\n   - gofmt for formatting\n   - goimports with `-local github.com/grafana/pyroscope`\n\n3. **Linting**:\n   - Enabled: depguard, goconst, misspell, revive, unconvert, unparam\n   - Use `github.com/go-kit/log` (NOT `github.com/go-kit/kit/log`)\n\n4. **Error Handling**:\n   - Always check errors explicitly\n   - Wrap errors with context: `fmt.Errorf(\"failed to query: %w\", err)`\n   - Use structured logging: `level.Error(logger).Log(\"msg\", \"failed to process\", \"err\", err)`\n\n5. **Context**:\n   - Always pass `context.Context` as the first parameter\n   - Respect context cancellation in loops and long operations\n\n6. **Testing**:\n   - File naming: `*_test.go`\n   - Test function naming: `TestFunctionName` or `TestComponentName_Method`\n   - Use table-driven tests for multiple cases\n   - Prefer `t.Run()` for subtests\n   - Use `require` for fatal assertions, `assert` for non-fatal\n\n### TypeScript/React Code\n\nThe `ui/` frontend has its own authoritative guides — **follow `ui/CLAUDE.md` and `ui/DESIGN.md`**. In brief:\n\n1. **File Extensions**: `.tsx` for components, `.ts` for utilities\n2. **Components**: Functional components with hooks; keep state management simple (the rewrite deliberately dropped Redux)\n3. **Styling**: Use semantic CSS custom properties from `src/theme.css`; never reference primitive tokens directly, and don't add Emotion or Grafana UI\n4. **Props**: Define explicit TypeScript interfaces for all component props\n5. **Formatting**: Prettier + ESLint — `yarn format` / `yarn lint` from `ui/`\n\n## Common Patterns\n\n### Multi-tenancy\n\nAll requests must include a tenant ID in the `X-Scope-OrgID` header:\n\n```go\nimport \"github.com/grafana/pyroscope/v2/pkg/tenant\"\n\n// Extract tenant ID from context\ntenantID, err := tenant.ExtractTenantIDFromContext(ctx)\nif err != nil {\n    return err\n}\n```\n\n### Consistent Hashing\n\nComponents use a hash ring for sharding:\n\n```go\n// Get ingester for a given label set\nreplicationSet, err := ring.Get(key, op, bufDescs, bufHosts, bufZones)\n```\n\n### Object Storage\n\nAbstract object storage operations:\n\n```go\nimport \"github.com/grafana/pyroscope/v2/pkg/objstore\"\n\n// Use the Bucket interface\nbucket := objstore.NewBucket(cfg)\nreader, err := bucket.Get(ctx, \"path/to/object\")\n```\n\n### Configuration\n\nUse `github.com/grafana/dskit` for configuration:\n\n```go\ntype Config struct {\n    ListenPort int `yaml:\"listen_port\"`\n    // Use RegisterFlags pattern\n}\n\nfunc (cfg *Config) RegisterFlags(f *flag.FlagSet) {\n    f.IntVar(&cfg.ListenPort, \"server.http-listen-port\", 4040, \"HTTP listen port\")\n}\n```\n\nRun `make generate` after changing config definitions, to regenerate docs.\n\n## Testing Best Practices\n\n1. **Unit Tests**: Test individual functions/methods in isolation\n2. **Integration Tests**: Use build tags: `//go:build integration`\n3. **Mocking**: Use `mockery` for generating mocks from interfaces\n4. **Fixtures**: Store test data in `testdata/` directories\n5. **Parallel Tests**: Use `t.Parallel()` when tests are independent\n6. **Cleanup**: Always use `t.Cleanup()` for resource cleanup\n\nExample test:\n```go\nfunc TestDistributor_Push(t *testing.T) {\n    t.Parallel()\n\n    tests := []struct {\n        name    string\n        input   *pushv1.PushRequest\n        wantErr bool\n    }{\n        {name: \"valid request\", input: validRequest(), wantErr: false},\n        {name: \"invalid tenant\", input: invalidRequest(), wantErr: true},\n    }\n\n    for _, tt := range tests {\n        t.Run(tt.name, func(t *testing.T) {\n            d := setupDistributor(t)\n            err := d.Push(context.Background(), tt.input)\n            if tt.wantErr {\n                require.Error(t, err)\n            } else {\n                require.NoError(t, err)\n            }\n        })\n    }\n}\n```\n\n## Common Pitfalls & Things to Avoid\n\n1. **Don't** introduce dependencies on `pkg/og/` - this is legacy code being phased out\n2. **Don't** use `github.com/go-kit/kit/log` - use `github.com/go-kit/log`\n3. **Don't** forget to run `make generate` after changing protobuf/config definitions\n4. **Don't** hardcode tenant IDs – always extract from context\n5. **Don't** create unbounded goroutines – use worker pools or semaphores\n6. **Don't** ignore context cancellation in loops\n7. **Don't** log PII or sensitive data\n8. **Don't** use `fmt.Println` for logging - use structured logging\n9. **Don't** add imports within the three import groups (keep them separate)\n10. **Don't** commit changes to `node_modules/` or generated code without source changes\n\n## Security Considerations\n\n1. **Input Validation**: Always validate and sanitize user input\n2. **Path Traversal**: Validate object keys before storage operations\n3. **Rate Limiting**: Distributor implements per-tenant rate limiting\n4. **Authentication**: Multi-tenancy via `X-Scope-OrgID` header (authentication delegated to gateway)\n\n## Performance Considerations\n\n1. **Profiling**: This is a profiling system – profile your own changes!\n   ```bash\n   go test -cpuprofile=cpu.prof -memprofile=mem.prof -bench=.\n   go tool pprof cpu.prof\n   ```\n\n2. **Allocations**: Minimize allocations in hot paths\n   - Reuse buffers with `sync.Pool`\n   - Avoid string concatenation in loops\n   - Use `strings.Builder` for string building\n\n3. **Concurrency**:\n   - Use worker pools for bounded concurrency\n   - Prefer channels for coordination over mutexes when possible\n   - Always consider the scalability implications\n\n## Documentation\n\n- **User Docs**: `docs/sources/` - Published to grafana.com\n- **Contributing**: `docs/internal/contributing/README.md`\n- **Component Docs**: In `docs/sources/reference-pyroscope-architecture/components/`\n\n## Useful Make Targets\n\n```bash\nmake help              # Show all available targets\nmake lint              # Run linters\nmake go/test           # Run Go unit tests\nmake go/bin            # Build binaries\nmake go/mod            # Tidy go modules\nmake generate          # Generate code (protobuf, mocks, etc.)\nmake docker-image/pyroscope/build  # Build Docker image\n```\n\n## Key Dependencies\n\n- **dskit**: Grafana's distributed systems toolkit (ring, services, middleware)\n- **connect**: RPC framework (gRPC-compatible)\n- **parquet-go**: Parquet file format implementation\n- **go-kit/log**: Structured logging\n- **prometheus/client_golang**: Metrics instrumentation\n- **opentelemetry**: Distributed tracing\n\n## When Working on Features\n\n1. **Read Component Docs**: Check `docs/sources/reference-pyroscope-architecture/components/` for the component you're modifying\n2. **Understand the Ring**: If working on write/read path, understand consistent hashing\n3. **Multi-tenancy First**: Always consider multi-tenant implications\n4. **Check for Similar Code**: Pyroscope is inspired by Cortex/Mimir - similar patterns apply\n5. **Test Multi-tenancy**: Test with multiple tenants to catch isolation issues\n6. **Profile Your Changes**: Use `go test -bench` and verify performance impact\n7. **Update Documentation**: If changing user-facing behavior, update docs\n\n## Getting Help\n\n- **Contributing Guide**: `docs/internal/contributing/README.md`\n- **Code Comments**: The codebase has extensive comments – read them\n- **Git History**: Use `git blame` and `git log` to understand design decisions\n\n## Commit Guidelines\n\n- **Atomic Commits**: Each commit should be a logical unit\n- **Commit Messages**: Focus on \"why\" not just \"what\"\n- **Generated Code**: Include generated files in the same commit as source changes\n- **Format**: Follow existing commit message style (see `git log --oneline -20`)\n\n## Additional Notes for AI Agents\n\n- **Favor Simplicity**: Pyroscope values simple, maintainable code over clever abstractions\n- **Performance Matters**: This system handles high-throughput profiling data\n- **Multi-tenancy is Critical**: Tenant isolation bugs are severe – test thoroughly\n- **Consistency with Grafana Labs Style**: Follow patterns from dskit, Mimir, Loki\n- **Ask Before Large Refactors**: Propose significant architectural changes before implementing\n\n---\n\nFor detailed setup and contributing instructions, see:\n- `docs/internal/contributing/README.md` - Development setup and workflow\n- `docs/sources/reference-pyroscope-architecture/` - System architecture deep dive\n"},"files":{"AGENTS.md":"# Pyroscope - AI Agent Development Guide\n\nThis document provides context and guidance for AI coding assistants (Claude, Cursor, GitHub Copilot, etc.) working on the Pyroscope codebase.\n\n## What is Pyroscope?\n\nPyroscope is a horizontally scalable, highly available, multi-tenant continuous profiling aggregation system. \nIt's designed to store and query profiling data at scale, similar to how Prometheus works for metrics and Loki for logs.\n\n**Key Characteristics:**\n- Written in **Go**\n- Microservices-based architecture inspired by Cortex/Mimir/Loki\n- Stores profiling data in object storage (S3, GCS, Azure, etc.)\n- Multi-tenant by design\n\n## Architecture Overview\n\nPyroscope uses a **microservices architecture** where a single binary can run different components based on the `-target` parameter.\n\n### V1 Components\n\n**Write Path:**\n- **Distributor**: Receives profile ingestion requests, validates, and forwards to ingesters\n- **Ingester**: Stores profiles in memory, periodically flushes to disk as blocks, periodically uploads blocks to long-term object storage\n- **Compactor**: Merges blocks and removes duplicates\n\n**Read Path:**\n- **Query Frontend**: Entry point for queries, handles query splitting and caching\n- **Query Scheduler**: Manages query queue and ensures fair execution across tenants\n- **Querier**: Executes queries by fetching data from ingesters and store-gateways\n- **Store Gateway**: Indexes and serves blocks from long-term object storage\n\n### V2 Components\n\n**Write Path:**\n- **Distributor**: Receives profile ingestion requests, validates, and forwards to segment writers\n- **Segment Writer**: Writes block segments to long-term object storage and the block metadata to metastore\n- **Metastore**: Maintains an index for the block metadata and coordinates the block compaction process\n- **Compaction Worker**: Merges small segments into larger blocks\n\n**Read Path:**\n- **Query Frontend**: Entry point for queries, creates the query plan and executes it against query backends\n- **Query Backend**: Executes queries and merges query responses\n\n### Storage\n\n- **Block Format**: Profiles stored in Parquet tables, series data in a TSDB index, symbols in a custom format\n- **Multi-tenant**: Each tenant has isolated storage\n- **Object Storage**: Primary storage backend (S3, GCS, Azure, local filesystem)\n\n## Repository Structure\n\n```\n.\n├── cmd/\n│   ├── pyroscope/           # Main server binary\n│   └── profilecli/          # CLI tool for profile operations\n├── pkg/                     # Core Go packages\n│   ├── distributor/         # Distributor component\n│   ├── ingester/            # Ingester component\n│   ├── querier/             # Querier component\n│   ├── frontend/            # Query frontend component\n│   ├── compactor/           # Compactor component\n│   ├── metastore/           # Metadata component\n│   ├── phlaredb/            # V1 database storage engine\n│   ├── model/               # Data models and types\n│   ├── objstore/            # Object storage abstraction\n│   ├── api/                 # API definitions and handlers\n│   └── og/                  # Legacy code (original Pyroscope)\n├── ui/                      # React/TypeScript frontend (Vite) - has its own ui/CLAUDE.md\n├── api/                     # API definitions (protobuf, OpenAPI)\n├── docs/                    # Documentation\n├── operations/              # Deployment configs (jsonnet, helm)\n├── examples/                # Example applications and SDKs\n└── tools/                   # Development and build tools\n```\n\n## Tech Stack\n\n### Backend\n- **Language**: Go 1.25 (see `go.mod`)\n- **RPC**: gRPC with Connect protocol\n- **Storage**: Parquet, TSDB\n- **Hash Ring**: Consistent hashing with memberlist (gossip protocol)\n- **Observability**: Prometheus metrics, Structured logs, Distributed traces, pprof profiles\n\n### Frontend\nThe frontend lives in `ui/` and is a dependency-minimal rewrite of the old `public/app` UI.\n**`ui/CLAUDE.md` and `ui/DESIGN.md` are the authoritative guides — read them before touching the UI.** Summary:\n- **Stack**: React 19 + Vite 8 + TypeScript 5.9, package-managed with **Yarn 4 (Berry)** — use `yarn`, not `npm`\n- **No UI library**: styling via CSS custom properties / semantic tokens in `src/theme.css` (no Emotion, no Grafana UI)\n- **Resist new dependencies** — prefer a small local implementation; minimizing the dependency surface is the whole point of the rewrite\n\n### Testing\n- **Go**: Standard `testing` package, testify for assertions\n- **Frontend**: Vitest (`yarn test` from `ui/`)\n\n## Development Workflow\n\n### Setup & Build\n\n```bash\n# Prerequisites: Go 1.25, Docker, Node, Yarn 4 (Berry).\n# All other build tools auto-download to .tmp/bin/\n\n# Build backend\nmake go/bin\n\n# Run tests\nmake go/test\n\n# Frontend dev (run from ui/): Vite dev server on :5173, proxies API to :4040\ncd ui && yarn install && yarn dev\n# Production frontend build (Docker -> ui/dist; required before `make build`):\nmake frontend/build\n\n# Docker image\nmake GOOS=linux GOARCH=amd64 docker-image/pyroscope/build\n```\n\n### Code Generation\n\n**IMPORTANT**: After changing protobuf, configs, or flags:\n```bash\nmake generate\n```\nCommit the generated files with your changes.\n\n### Running Locally\n\n```bash\n# Run all components in monolithic mode with embedded Grafana\ngo run ./cmd/pyroscope --target all,embedded-grafana\n# Pyroscope: http://localhost:4040\n# Grafana: http://localhost:4041\n\n# Run with V2 architecture (segment writers, query backend, symbolizer)\n# -symbolizer.enabled=true opts into symbolization (off by default).\ngo run ./cmd/pyroscope -symbolizer.enabled=true\n```\n\n## Code Style & Conventions\n\n### Go Code\n\n1. **Imports**: Three groups separated by blank lines:\n   ```go\n   import (\n       // Standard library\n       \"context\"\n       \"fmt\"\n\n       // Third-party packages\n       \"github.com/prometheus/client_golang/prometheus\"\n       \"go.uber.org/atomic\"\n\n       // Internal packages\n       \"github.com/grafana/pyroscope/v2/pkg/model\"\n       \"github.com/grafana/pyroscope/v2/pkg/objstore\"\n   )\n   ```\n\n2. **Formatting**: Use `golangci-lint` (run via `make lint`)\n   - gofmt for formatting\n   - goimports with `-local github.com/grafana/pyroscope`\n\n3. **Linting**:\n   - Enabled: depguard, goconst, misspell, revive, unconvert, unparam\n   - Use `github.com/go-kit/log` (NOT `github.com/go-kit/kit/log`)\n\n4. **Error Handling**:\n   - Always check errors explicitly\n   - Wrap errors with context: `fmt.Errorf(\"failed to query: %w\", err)`\n   - Use structured logging: `level.Error(logger).Log(\"msg\", \"failed to process\", \"err\", err)`\n\n5. **Context**:\n   - Always pass `context.Context` as the first parameter\n   - Respect context cancellation in loops and long operations\n\n6. **Testing**:\n   - File naming: `*_test.go`\n   - Test function naming: `TestFunctionName` or `TestComponentName_Method`\n   - Use table-driven tests for multiple cases\n   - Prefer `t.Run()` for subtests\n   - Use `require` for fatal assertions, `assert` for non-fatal\n\n### TypeScript/React Code\n\nThe `ui/` frontend has its own authoritative guides — **follow `ui/CLAUDE.md` and `ui/DESIGN.md`**. In brief:\n\n1. **File Extensions**: `.tsx` for components, `.ts` for utilities\n2. **Components**: Functional components with hooks; keep state management simple (the rewrite deliberately dropped Redux)\n3. **Styling**: Use semantic CSS custom properties from `src/theme.css`; never reference primitive tokens directly, and don't add Emotion or Grafana UI\n4. **Props**: Define explicit TypeScript interfaces for all component props\n5. **Formatting**: Prettier + ESLint — `yarn format` / `yarn lint` from `ui/`\n\n## Common Patterns\n\n### Multi-tenancy\n\nAll requests must include a tenant ID in the `X-Scope-OrgID` header:\n\n```go\nimport \"github.com/grafana/pyroscope/v2/pkg/tenant\"\n\n// Extract tenant ID from context\ntenantID, err := tenant.ExtractTenantIDFromContext(ctx)\nif err != nil {\n    return err\n}\n```\n\n### Consistent Hashing\n\nComponents use a hash ring for sharding:\n\n```go\n// Get ingester for a given label set\nreplicationSet, err := ring.Get(key, op, bufDescs, bufHosts, bufZones)\n```\n\n### Object Storage\n\nAbstract object storage operations:\n\n```go\nimport \"github.com/grafana/pyroscope/v2/pkg/objstore\"\n\n// Use the Bucket interface\nbucket := objstore.NewBucket(cfg)\nreader, err := bucket.Get(ctx, \"path/to/object\")\n```\n\n### Configuration\n\nUse `github.com/grafana/dskit` for configuration:\n\n```go\ntype Config struct {\n    ListenPort int `yaml:\"listen_port\"`\n    // Use RegisterFlags pattern\n}\n\nfunc (cfg *Config) RegisterFlags(f *flag.FlagSet) {\n    f.IntVar(&cfg.ListenPort, \"server.http-listen-port\", 4040, \"HTTP listen port\")\n}\n```\n\nRun `make generate` after changing config definitions, to regenerate docs.\n\n## Testing Best Practices\n\n1. **Unit Tests**: Test individual functions/methods in isolation\n2. **Integration Tests**: Use build tags: `//go:build integration`\n3. **Mocking**: Use `mockery` for generating mocks from interfaces\n4. **Fixtures**: Store test data in `testdata/` directories\n5. **Parallel Tests**: Use `t.Parallel()` when tests are independent\n6. **Cleanup**: Always use `t.Cleanup()` for resource cleanup\n\nExample test:\n```go\nfunc TestDistributor_Push(t *testing.T) {\n    t.Parallel()\n\n    tests := []struct {\n        name    string\n        input   *pushv1.PushRequest\n        wantErr bool\n    }{\n        {name: \"valid request\", input: validRequest(), wantErr: false},\n        {name: \"invalid tenant\", input: invalidRequest(), wantErr: true},\n    }\n\n    for _, tt := range tests {\n        t.Run(tt.name, func(t *testing.T) {\n            d := setupDistributor(t)\n            err := d.Push(context.Background(), tt.input)\n            if tt.wantErr {\n                require.Error(t, err)\n            } else {\n                require.NoError(t, err)\n            }\n        })\n    }\n}\n```\n\n## Common Pitfalls & Things to Avoid\n\n1. **Don't** introduce dependencies on `pkg/og/` - this is legacy code being phased out\n2. **Don't** use `github.com/go-kit/kit/log` - use `github.com/go-kit/log`\n3. **Don't** forget to run `make generate` after changing protobuf/config definitions\n4. **Don't** hardcode tenant IDs – always extract from context\n5. **Don't** create unbounded goroutines – use worker pools or semaphores\n6. **Don't** ignore context cancellation in loops\n7. **Don't** log PII or sensitive data\n8. **Don't** use `fmt.Println` for logging - use structured logging\n9. **Don't** add imports within the three import groups (keep them separate)\n10. **Don't** commit changes to `node_modules/` or generated code without source changes\n\n## Security Considerations\n\n1. **Input Validation**: Always validate and sanitize user input\n2. **Path Traversal**: Validate object keys before storage operations\n3. **Rate Limiting**: Distributor implements per-tenant rate limiting\n4. **Authentication**: Multi-tenancy via `X-Scope-OrgID` header (authentication delegated to gateway)\n\n## Performance Considerations\n\n1. **Profiling**: This is a profiling system – profile your own changes!\n   ```bash\n   go test -cpuprofile=cpu.prof -memprofile=mem.prof -bench=.\n   go tool pprof cpu.prof\n   ```\n\n2. **Allocations**: Minimize allocations in hot paths\n   - Reuse buffers with `sync.Pool`\n   - Avoid string concatenation in loops\n   - Use `strings.Builder` for string building\n\n3. **Concurrency**:\n   - Use worker pools for bounded concurrency\n   - Prefer channels for coordination over mutexes when possible\n   - Always consider the scalability implications\n\n## Documentation\n\n- **User Docs**: `docs/sources/` - Published to grafana.com\n- **Contributing**: `docs/internal/contributing/README.md`\n- **Component Docs**: In `docs/sources/reference-pyroscope-architecture/components/`\n\n## Useful Make Targets\n\n```bash\nmake help              # Show all available targets\nmake lint              # Run linters\nmake go/test           # Run Go unit tests\nmake go/bin            # Build binaries\nmake go/mod            # Tidy go modules\nmake generate          # Generate code (protobuf, mocks, etc.)\nmake docker-image/pyroscope/build  # Build Docker image\n```\n\n## Key Dependencies\n\n- **dskit**: Grafana's distributed systems toolkit (ring, services, middleware)\n- **connect**: RPC framework (gRPC-compatible)\n- **parquet-go**: Parquet file format implementation\n- **go-kit/log**: Structured logging\n- **prometheus/client_golang**: Metrics instrumentation\n- **opentelemetry**: Distributed tracing\n\n## When Working on Features\n\n1. **Read Component Docs**: Check `docs/sources/reference-pyroscope-architecture/components/` for the component you're modifying\n2. **Understand the Ring**: If working on write/read path, understand consistent hashing\n3. **Multi-tenancy First**: Always consider multi-tenant implications\n4. **Check for Similar Code**: Pyroscope is inspired by Cortex/Mimir - similar patterns apply\n5. **Test Multi-tenancy**: Test with multiple tenants to catch isolation issues\n6. **Profile Your Changes**: Use `go test -bench` and verify performance impact\n7. **Update Documentation**: If changing user-facing behavior, update docs\n\n## Getting Help\n\n- **Contributing Guide**: `docs/internal/contributing/README.md`\n- **Code Comments**: The codebase has extensive comments – read them\n- **Git History**: Use `git blame` and `git log` to understand design decisions\n\n## Commit Guidelines\n\n- **Atomic Commits**: Each commit should be a logical unit\n- **Commit Messages**: Focus on \"why\" not just \"what\"\n- **Generated Code**: Include generated files in the same commit as source changes\n- **Format**: Follow existing commit message style (see `git log --oneline -20`)\n\n## Additional Notes for AI Agents\n\n- **Favor Simplicity**: Pyroscope values simple, maintainable code over clever abstractions\n- **Performance Matters**: This system handles high-throughput profiling data\n- **Multi-tenancy is Critical**: Tenant isolation bugs are severe – test thoroughly\n- **Consistency with Grafana Labs Style**: Follow patterns from dskit, Mimir, Loki\n- **Ask Before Large Refactors**: Propose significant architectural changes before implementing\n\n---\n\nFor detailed setup and contributing instructions, see:\n- `docs/internal/contributing/README.md` - Development setup and workflow\n- `docs/sources/reference-pyroscope-architecture/` - System architecture deep dive\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# Pyroscope - AI Agent Development Guide\n\nThis document provides context and guidance for AI coding assistants (Claude, Cursor, GitHub Copilot, etc.) working on the Pyroscope codebase.\n\n## What is Pyroscope?\n\nPyroscope is a horizontally scalable, highly available, multi-tenant continuous profiling aggregation system. \nIt's designed to store and query profiling data at scale, similar to how Prometheus works for metrics and Loki for logs.\n\n**Key Characteristics:**\n- Written in **Go**\n- Microservices-based architecture inspired by Cortex/Mimir/Loki\n- Stores profiling data in object storage (S3, GCS, Azure, etc.)\n- Multi-tenant by design\n\n## Architecture Overview\n\nPyroscope uses a **microservices architecture** where a single binary can run different components based on the `-target` parameter.\n\n### V1 Components\n\n**Write Path:**\n- **Distributor**: Receives profile ingestion requests, validates, and forwards to ingesters\n- **Ingester**: Stores profiles in memory, periodically flushes to disk as blocks, periodically uploads blocks to long-term object storage\n- **Compactor**: Merges blocks and removes duplicates\n\n**Read Path:**\n- **Query Frontend**: Entry point for queries, handles query splitting and caching\n- **Query Scheduler**: Manages query queue and ensures fair execution across tenants\n- **Querier**: Executes queries by fetching data from ingesters and store-gateways\n- **Store Gateway**: Indexes and serves blocks from long-term object storage\n\n### V2 Components\n\n**Write Path:**\n- **Distributor**: Receives profile ingestion requests, validates, and forwards to segment writers\n- **Segment Writer**: Writes block segments to long-term object storage and the block metadata to metastore\n- **Metastore**: Maintains an index for the block metadata and coordinates the block compaction process\n- **Compaction Worker**: Merges small segments into larger blocks\n\n**Read Path:**\n- **Query Frontend**: Entry point for queries, creates the query plan and executes it against query backends\n- **Query Backend**: Executes queries and merges query responses\n\n### Storage\n\n- **Block Format**: Profiles stored in Parquet tables, series data in a TSDB index, symbols in a custom format\n- **Multi-tenant**: Each tenant has isolated storage\n- **Object Storage**: Primary storage backend (S3, GCS, Azure, local filesystem)\n\n## Repository Structure\n\n```\n.\n├── cmd/\n│   ├── pyroscope/           # Main server binary\n│   └── profilecli/          # CLI tool for profile operations\n├── pkg/                     # Core Go packages\n│   ├── distributor/         # Distributor component\n│   ├── ingester/            # Ingester component\n│   ├── querier/             # Querier component\n│   ├── frontend/            # Query frontend component\n│   ├── compactor/           # Compactor component\n│   ├── metastore/           # Metadata component\n│   ├── phlaredb/            # V1 database storage engine\n│   ├── model/               # Data models and types\n│   ├── objstore/            # Object storage abstraction\n│   ├── api/                 # API definitions and handlers\n│   └── og/                  # Legacy code (original Pyroscope)\n├── ui/                      # React/TypeScript frontend (Vite) - has its own ui/CLAUDE.md\n├── api/                     # API definitions (protobuf, OpenAPI)\n├── docs/                    # Documentation\n├── operations/              # Deployment configs (jsonnet, helm)\n├── examples/                # Example applications and SDKs\n└── tools/                   # Development and build tools\n```\n\n## Tech Stack\n\n### Backend\n- **Language**: Go 1.25 (see `go.mod`)\n- **RPC**: gRPC with Connect protocol\n- **Storage**: Parquet, TSDB\n- **Hash Ring**: Consistent hashing with memberlist (gossip protocol)\n- **Observability**: Prometheus metrics, Structured logs, Distributed traces, pprof profiles\n\n### Frontend\nThe frontend lives in `ui/` and is a dependency-minimal rewrite of the old `public/app` UI.\n**`ui/CLAUDE.md` and `ui/DESIGN.md` are the authoritative guides — read them before touching the UI.** Summary:\n- **Stack**: React 19 + Vite 8 + TypeScript 5.9, package-managed with **Yarn 4 (Berry)** — use `yarn`, not `npm`\n- **No UI library**: styling via CSS custom properties / semantic tokens in `src/theme.css` (no Emotion, no Grafana UI)\n- **Resist new dependencies** — prefer a small local implementation; minimizing the dependency surface is the whole point of the rewrite\n\n### Testing\n- **Go**: Standard `testing` package, testify for assertions\n- **Frontend**: Vitest (`yarn test` from `ui/`)\n\n## Development Workflow\n\n### Setup & Build\n\n```bash\n# Prerequisites: Go 1.25, Docker, Node, Yarn 4 (Berry).\n# All other build tools auto-download to .tmp/bin/\n\n# Build backend\nmake go/bin\n\n# Run tests\nmake go/test\n\n# Frontend dev (run from ui/): Vite dev server on :5173, proxies API to :4040\ncd ui && yarn install && yarn dev\n# Production frontend build (Docker -> ui/dist; required before `make build`):\nmake frontend/build\n\n# Docker image\nmake GOOS=linux GOARCH=amd64 docker-image/pyroscope/build\n```\n\n### Code Generation\n\n**IMPORTANT**: After changing protobuf, configs, or flags:\n```bash\nmake generate\n```\nCommit the generated files with your changes.\n\n### Running Locally\n\n```bash\n# Run all components in monolithic mode with embedded Grafana\ngo run ./cmd/pyroscope --target all,embedded-grafana\n# Pyroscope: http://localhost:4040\n# Grafana: http://localhost:4041\n\n# Run with V2 architecture (segment writers, query backend, symbolizer)\n# -symbolizer.enabled=true opts into symbolization (off by default).\ngo run ./cmd/pyroscope -symbolizer.enabled=true\n```\n\n## Code Style & Conventions\n\n### Go Code\n\n1. **Imports**: Three groups separated by blank lines:\n   ```go\n   import (\n       // Standard library\n       \"context\"\n       \"fmt\"\n\n       // Third-party packages\n       \"github.com/prometheus/client_golang/prometheus\"\n       \"go.uber.org/atomic\"\n\n       // Internal packages\n       \"github.com/grafana/pyroscope/v2/pkg/model\"\n       \"github.com/grafana/pyroscope/v2/pkg/objstore\"\n   )\n   ```\n\n2. **Formatting**: Use `golangci-lint` (run via `make lint`)\n   - gofmt for formatting\n   - goimports with `-local github.com/grafana/pyroscope`\n\n3. **Linting**:\n   - Enabled: depguard, goconst, misspell, revive, unconvert, unparam\n   - Use `github.com/go-kit/log` (NOT `github.com/go-kit/kit/log`)\n\n4. **Error Handling**:\n   - Always check errors explicitly\n   - Wrap errors with context: `fmt.Errorf(\"failed to query: %w\", err)`\n   - Use structured logging: `level.Error(logger).Log(\"msg\", \"failed to process\", \"err\", err)`\n\n5. **Context**:\n   - Always pass `context.Context` as the first parameter\n   - Respect context cancellation in loops and long operations\n\n6. **Testing**:\n   - File naming: `*_test.go`\n   - Test function naming: `TestFunctionName` or `TestComponentName_Method`\n   - Use table-driven tests for multiple cases\n   - Prefer `t.Run()` for subtests\n   - Use `require` for fatal assertions, `assert` for non-fatal\n\n### TypeScript/React Code\n\nThe `ui/` frontend has its own authoritative guides — **follow `ui/CLAUDE.md` and `ui/DESIGN.md`**. In brief:\n\n1. **File Extensions**: `.tsx` for components, `.ts` for utilities\n2. **Components**: Functional components with hooks; keep state management simple (the rewrite deliberately dropped Redux)\n3. **Styling**: Use semantic CSS custom properties from `src/theme.css`; never reference primitive tokens directly, and don't add Emotion or Grafana UI\n4. **Props**: Define explicit TypeScript interfaces for all component props\n5. **Formatting**: Prettier + ESLint — `yarn format` / `yarn lint` from `ui/`\n\n## Common Patterns\n\n### Multi-tenancy\n\nAll requests must include a tenant ID in the `X-Scope-OrgID` header:\n\n```go\nimport \"github.com/grafana/pyroscope/v2/pkg/tenant\"\n\n// Extract tenant ID from context\ntenantID, err := tenant.ExtractTenantIDFromContext(ctx)\nif err != nil {\n    return err\n}\n```\n\n### Consistent Hashing\n\nComponents use a hash ring for sharding:\n\n```go\n// Get ingester for a given label set\nreplicationSet, err := ring.Get(key, op, bufDescs, bufHosts, bufZones)\n```\n\n### Object Storage\n\nAbstract object storage operations:\n\n```go\nimport \"github.com/grafana/pyroscope/v2/pkg/objstore\"\n\n// Use the Bucket interface\nbucket := objstore.NewBucket(cfg)\nreader, err := bucket.Get(ctx, \"path/to/object\")\n```\n\n### Configuration\n\nUse `github.com/grafana/dskit` for configuration:\n\n```go\ntype Config struct {\n    ListenPort int `yaml:\"listen_port\"`\n    // Use RegisterFlags pattern\n}\n\nfunc (cfg *Config) RegisterFlags(f *flag.FlagSet) {\n    f.IntVar(&cfg.ListenPort, \"server.http-listen-port\", 4040, \"HTTP listen port\")\n}\n```\n\nRun `make generate` after changing config definitions, to regenerate docs.\n\n## Testing Best Practices\n\n1. **Unit Tests**: Test individual functions/methods in isolation\n2. **Integration Tests**: Use build tags: `//go:build integration`\n3. **Mocking**: Use `mockery` for generating mocks from interfaces\n4. **Fixtures**: Store test data in `testdata/` directories\n5. **Parallel Tests**: Use `t.Parallel()` when tests are independent\n6. **Cleanup**: Always use `t.Cleanup()` for resource cleanup\n\nExample test:\n```go\nfunc TestDistributor_Push(t *testing.T) {\n    t.Parallel()\n\n    tests := []struct {\n        name    string\n        input   *pushv1.PushRequest\n        wantErr bool\n    }{\n        {name: \"valid request\", input: validRequest(), wantErr: false},\n        {name: \"invalid tenant\", input: invalidRequest(), wantErr: true},\n    }\n\n    for _, tt := range tests {\n        t.Run(tt.name, func(t *testing.T) {\n            d := setupDistributor(t)\n            err := d.Push(context.Background(), tt.input)\n            if tt.wantErr {\n                require.Error(t, err)\n            } else {\n                require.NoError(t, err)\n            }\n        })\n    }\n}\n```\n\n## Common Pitfalls & Things to Avoid\n\n1. **Don't** introduce dependencies on `pkg/og/` - this is legacy code being phased out\n2. **Don't** use `github.com/go-kit/kit/log` - use `github.com/go-kit/log`\n3. **Don't** forget to run `make generate` after changing protobuf/config definitions\n4. **Don't** hardcode tenant IDs – always extract from context\n5. **Don't** create unbounded goroutines – use worker pools or semaphores\n6. **Don't** ignore context cancellation in loops\n7. **Don't** log PII or sensitive data\n8. **Don't** use `fmt.Println` for logging - use structured logging\n9. **Don't** add imports within the three import groups (keep them separate)\n10. **Don't** commit changes to `node_modules/` or generated code without source changes\n\n## Security Considerations\n\n1. **Input Validation**: Always validate and sanitize user input\n2. **Path Traversal**: Validate object keys before storage operations\n3. **Rate Limiting**: Distributor implements per-tenant rate limiting\n4. **Authentication**: Multi-tenancy via `X-Scope-OrgID` header (authentication delegated to gateway)\n\n## Performance Considerations\n\n1. **Profiling**: This is a profiling system – profile your own changes!\n   ```bash\n   go test -cpuprofile=cpu.prof -memprofile=mem.prof -bench=.\n   go tool pprof cpu.prof\n   ```\n\n2. **Allocations**: Minimize allocations in hot paths\n   - Reuse buffers with `sync.Pool`\n   - Avoid string concatenation in loops\n   - Use `strings.Builder` for string building\n\n3. **Concurrency**:\n   - Use worker pools for bounded concurrency\n   - Prefer channels for coordination over mutexes when possible\n   - Always consider the scalability implications\n\n## Documentation\n\n- **User Docs**: `docs/sources/` - Published to grafana.com\n- **Contributing**: `docs/internal/contributing/README.md`\n- **Component Docs**: In `docs/sources/reference-pyroscope-architecture/components/`\n\n## Useful Make Targets\n\n```bash\nmake help              # Show all available targets\nmake lint              # Run linters\nmake go/test           # Run Go unit tests\nmake go/bin            # Build binaries\nmake go/mod            # Tidy go modules\nmake generate          # Generate code (protobuf, mocks, etc.)\nmake docker-image/pyroscope/build  # Build Docker image\n```\n\n## Key Dependencies\n\n- **dskit**: Grafana's distributed systems toolkit (ring, services, middleware)\n- **connect**: RPC framework (gRPC-compatible)\n- **parquet-go**: Parquet file format implementation\n- **go-kit/log**: Structured logging\n- **prometheus/client_golang**: Metrics instrumentation\n- **opentelemetry**: Distributed tracing\n\n## When Working on Features\n\n1. **Read Component Docs**: Check `docs/sources/reference-pyroscope-architecture/components/` for the component you're modifying\n2. **Understand the Ring**: If working on write/read path, understand consistent hashing\n3. **Multi-tenancy First**: Always consider multi-tenant implications\n4. **Check for Similar Code**: Pyroscope is inspired by Cortex/Mimir - similar patterns apply\n5. **Test Multi-tenancy**: Test with multiple tenants to catch isolation issues\n6. **Profile Your Changes**: Use `go test -bench` and verify performance impact\n7. **Update Documentation**: If changing user-facing behavior, update docs\n\n## Getting Help\n\n- **Contributing Guide**: `docs/internal/contributing/README.md`\n- **Code Comments**: The codebase has extensive comments – read them\n- **Git History**: Use `git blame` and `git log` to understand design decisions\n\n## Commit Guidelines\n\n- **Atomic Commits**: Each commit should be a logical unit\n- **Commit Messages**: Focus on \"why\" not just \"what\"\n- **Generated Code**: Include generated files in the same commit as source changes\n- **Format**: Follow existing commit message style (see `git log --oneline -20`)\n\n## Additional Notes for AI Agents\n\n- **Favor Simplicity**: Pyroscope values simple, maintainable code over clever abstractions\n- **Performance Matters**: This system handles high-throughput profiling data\n- **Multi-tenancy is Critical**: Tenant isolation bugs are severe – test thoroughly\n- **Consistency with Grafana Labs Style**: Follow patterns from dskit, Mimir, Loki\n- **Ask Before Large Refactors**: Propose significant architectural changes before implementing\n\n---\n\nFor detailed setup and contributing instructions, see:\n- `docs/internal/contributing/README.md` - Development setup and workflow\n- `docs/sources/reference-pyroscope-architecture/` - System architecture deep dive\n","category":"root","tokens":3566}]}