{"owner":"apache","repo":"iggy","hasSkills":true,"totalSkillsCount":10,"totalTokensCount":32844,"categories":["anthropic-skill","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":[".claude/skills/connector-sdk/SKILL.md",".claude/skills/connector-sink/SKILL.md",".claude/skills/connector-transform/SKILL.md",".claude/skills/connector-source/SKILL.md",".claude/skills/connector-runtime/SKILL.md",".claude/skills/connectors-overview/SKILL.md",".claude/skills/connector-testing/SKILL.md",".claude/skills/connector-source/TEMPLATE.md",".claude/skills/connector-sink/TEMPLATE.md","AGENTS.md"],"skills":{".claude/skills/connector-sdk/SKILL.md":"---\nname: connector-sdk\ndescription: Extend or modify the Apache Iggy Connectors SDK (`core/connectors/sdk/`). Use when adding a new `Schema` variant, a new `StreamDecoder`/`StreamEncoder`, a new `Error` variant, modifying the `Sink`/`Source`/`Transform` trait surface, the FFI macros, or the `retry`/`api`/`convert` modules. NOT for writing plugins - use `connector-sink` or `connector-source` for those.\n---\n\n# Extending the Apache Iggy Connectors SDK\n\nThe SDK (`core/connectors/sdk/`) is the **stable contract** between the\nruntime and every plugin. Changes ripple to every sink/source in-tree\nand every third-party plugin. Treat the public surface as a versioned API.\n\n> **Universal connector rules** (benchmark, SecretString, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers SDK-level changes only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Cardinal rules](#cardinal-rules)\n- [Adding a new `Schema` variant](#adding-a-new-schema-variant)\n- [Adding a `StreamDecoder` / `StreamEncoder`](#adding-a-streamdecoder--streamencoder)\n- [Adding a new `Error` variant](#adding-a-new-error-variant)\n- [Modifying `Sink` / `Source` / `Transform` traits](#modifying-sink--source--transform-traits)\n- [FFI macros (`sink_connector!`, `source_connector!`)](#ffi-macros-sink_connector-source_connector)\n- [`Payload::try_to_bytes(&self)` is non-negotiable for JSON](#payloadtry_to_bytesself-is-non-negotiable-for-json)\n- [Retry helpers (`retry.rs`)](#retry-helpers-retryrs)\n- [`ConnectorState`](#connectorstate)\n- [Tests for SDK changes](#tests-for-sdk-changes)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Bumping the SDK MAJOR version or changing any `#[repr(C)]` layout - misaligns every pre-built plugin `.so`.\n- Changing `Sink` / `Source` / `Transform` / `StreamDecoder` / `StreamEncoder` trait signatures - cascades through every plugin.\n- Removing or renaming a `Schema` variant or `Error` variant - decoders/encoders pinned to it. downstream pattern matches break.\n- Changing wire serialization (postcard / rmp_serde / serde_json) for any FFI / state / config payload.\n- Changing FFI macro return codes or function names (`iggy_*_open` / `consume` / `handle` / `close`) - runtime + plugins both need lockstep update.\n\n## File map\n\n```text\nsdk/src/\n├── lib.rs Traits (Sink, Source, StreamDecoder, StreamEncoder),\n│ Payload, Schema, ConnectorState, Error enum, FFI message structs.\n├── sink.rs SinkContainer + sink_connector! macro (FFI plumbing).\n├── source.rs SourceContainer + source_connector! macro (FFI plumbing).\n├── api.rs ConnectorStatus, ConnectorStats (feature = \"api\").\n├── convert.rs owned_value_into_serde_json / owned_value_to_serde_json (simd_json ⇄ serde_json bridge).\n├── log.rs CallbackLayer for tracing across FFI.\n├── retry.rs retry_async + RetryPolicy, CircuitBreaker, HttpRetryMiddleware.\n├── decoders/ One per schema: json, raw, text, proto, flatbuffer, avro.\n├── encoders/ Mirror of decoders.\n└── transforms/ add_fields, delete_fields, update_fields, filter_fields,\n unwrap_envelope, proto_convert, flatbuffer_convert, avro_convert.\n```\n\n## Cardinal rules\n\n1. **Apache 2.0 license header** on every new file.\n2. **`Send + Sync`** on every public trait (FFI runs across thread pools).\n3. **`#[repr(C)]`** on every type that crosses FFI (`Schema`, `TopicMetadata`, `MessagesMetadata`, `RawMessage`, `ProducedMessages`, `ConsumedMessage`, `DecodedMessage`, etc. in `lib.rs`). Adding fields requires bumping the SDK version - existing plugins built against the old layout will misalign on `postcard::from_bytes`.\n4. **postcard** for FFI message serialization (handled by `SinkContainer::consume`, `SourceContainer`'s `handle_messages`). **MessagePack (`rmp_serde`)** for `ConnectorState`. **JSON** (`serde_json`) only for human-editable config that crosses FFI. Don't mix.\n5. **`simd_json::OwnedValue`** for JSON payloads, not `serde_json::Value`. Use `convert::owned_value_into_serde_json` as a bridge when you own the value (the common case - it moves strings instead of copying them); fall back to the borrowing `convert::owned_value_to_serde_json` only when you still need the `OwnedValue` afterward.\n6. **`BTreeMap`** for headers - deterministic ordering. Never `HashMap` on the wire.\n7. **No breaking changes** to `Sink`/`Source`/`StreamDecoder`/`StreamEncoder`/`Transform` trait signatures without coordinating with all in-tree plugins in the same PR.\n8. **A `Schema` variant that changes meaning is as breaking as a new variant.** The discriminant crosses FFI unchanged, so a plugin built against the old SDK keeps decoding it the old way and misreads every batch with no error anywhere. Same process as a trait change: bump the SDK minor version, update every in-tree plugin in the same PR, and state in the PR description that plugins built against the previous SDK must be rebuilt.\n\n## Adding a new `Schema` variant\n\nThe `Schema` enum (in `lib.rs`) is `#[repr(C)]` - it crosses FFI. Touch points to update **in one PR**:\n\n1. `Schema` enum: add variant with `#[strum(to_string = \"...\")]` matching the `serde(rename_all=\"snake_case\")` form.\n2. `Payload` enum: add a matching variant carrying the deserialized form.\n3. `Payload::try_into_vec` - consuming bytes-out path.\n4. `Payload::try_to_bytes` - **borrowing** bytes-out path. For non-trivial payloads (parsed trees), implement a no-clone serialization, not a `clone() + serialize`. See the `Payload::Json` arm for the canonical optimization.\n5. `Payload::schema()` - variant → `Schema`. The runtime tags every sink batch from this, so a missing arm will not compile but a wrong one silently mislabels the batch.\n6. `Payload::Display`.\n7. `Schema::try_into_payload` - bytes → `Payload`, for a tag naming a wire format (the source path).\n8. `Payload::try_from_schema` - bytes → `Payload`, for a tag that came from `Payload::schema()` (the sink path). The exact inverse of item 5.\n9. `Schema::decoder()` - factory returning `Arc`.\n10. `Schema::encoder()` - factory returning `Arc`.\n11. New `decoders/.rs` and `encoders/.rs`.\n12. Update the schema list in `sdk/README.md`.\n13. Tests: round-trip encode/decode, error paths. `Payload::schema()` must round-trip through `Payload::try_from_schema`, with no exceptions: every variant survives that trip and the test is table-driven over all of them.\n\nMiss any of these → silent failures at runtime (typically `Error::InvalidPayloadType` or surprising decoder behavior in plugin code).\n\n## Adding a `StreamDecoder` / `StreamEncoder`\n\nPattern from `decoders/json.rs::JsonStreamDecoder`:\n\n```rust\npub struct MyStreamDecoder; // unit struct if stateless\n\nimpl StreamDecoder for MyStreamDecoder {\n fn schema(&self) -> Schema { Schema::MyFormat }\n fn decode(&self, payload: Vec) -> Result {\n // Decode payload bytes into Payload::MyFormat(...).\n // Errors: Error::CannotDecode(Schema::MyFormat), Error::InvalidJsonPayload, etc.\n }\n}\n```\n\nFor **stateful** decoders (proto/avro/flatbuffer) holding schema descriptors:\n\n- Store the schema state directly in the struct (`config`, `message_descriptor`, `schema`). The instance is built once and wrapped in `Arc` for sharing - it must be `Send + Sync` but does not need additional locking if construction-time loading is final.\n- Provide **two constructors** when schema loading can fail: lenient `new(config) -> Self` (logs + degrades) AND strict `try_new(config) -> Result` (fail-fast). Plus a `Default` impl for the no-schema case. Canonical example: `sdk/src/decoders/avro.rs` exports all three (`AvroStreamDecoder::new`, `::try_new`, `Default`).\n- If you genuinely need to mutate the schema after construction (e.g., a `update_config` method), use `&mut self` plus `std::mem::replace(&mut self.config, new_config)` to swap without cloning the old value. Live pattern: `sdk/src/decoders/avro.rs::AvroStreamDecoder::update_config` and `sdk/src/encoders/avro.rs::AvroStreamEncoder::update_config`.\n- A fresh decoder instance is created via `Schema::decoder()` on each call (it returns `Arc`), so per-decoder caching of shared mutable state is the wrong abstraction.\n\n## Adding a new `Error` variant\n\nThe `Error` enum (in `lib.rs`) is `Clone + PartialEq + Eq + Hash`. New variants must preserve these. Guidelines:\n\n- **Use existing variants first.** Only add a new one if the failure mode is distinct in handling, not just description. `InvalidConfigValue(String)` covers most \"bad config\" cases.\n- **Carry context as `String`** - don't add a struct unless multiple discrete fields are needed by callers programmatically.\n- **Document retry semantics in the docstring.** Existing variants (`PermanentHttpError`, `CatalogCommitError`, `TransactionApplyError`) document retry behavior - mirror that style.\n- **Place near related variants** to keep the enum readable top-to-bottom.\n\n## Modifying `Sink` / `Source` / `Transform` traits\n\n**Breaking changes.** Required process:\n\n1. Document migration in the SDK changelog (and `connectors/sdk/README.md`).\n2. Update every in-tree plugin in the same PR. CI builds them all.\n3. Bump SDK minor version.\n4. Note in PR description: dynamic plugins compiled against the old SDK version will fail to load with mismatched FFI symbols.\n\nNon-breaking additions (new default method, new struct field with `#[serde(default)]`, new variant on a non-exhaustive enum) are preferable.\n\n## FFI macros (`sink_connector!`, `source_connector!`)\n\nThe macros (in `sink.rs` and `source.rs`) generate:\n\n- `iggy__open(id, config_ptr, config_len, [state_ptr, state_len,] log_callback) -> i32`\n- `iggy_sink_consume(id, ...)` / `iggy_source_handle(id, callback) -> i32`\n- `iggy__close(id) -> i32`\n- `iggy__version() -> *const c_char` (static lifetime, from `env!(\"CARGO_PKG_VERSION\")`)\n\nInvariants:\n\n- **Duplicate-ID guard** returns `-1` if the caller reopens an ID without closing first. Don't remove this - it prevents silent buffered-data loss.\n- `INSTANCES: Lazy>>` (and `SourceContainer<$type>` mirror) is the only mutable global the macro introduces - keep it that way.\n- Return codes: `0` success, `-1` invalid call, `1` open failure. Don't repurpose.\n\nChanges to the FFI signature must update `runtime/src/main.rs::{SourceApi, SinkApi}` in the same PR.\n\n## `Payload::try_to_bytes(&self)` is non-negotiable for JSON\n\nPlugin authors call this on every consumed message. The implementation in `lib.rs::Payload::try_to_bytes` documents *why* it skips the deep `OwnedValue` clone - replacing `O(n) clone + O(n) serialize` with `O(n) serialize`. If you add a new variant requiring a parsed tree, do the same: serialize in place, don't clone.\n\n## Retry helpers (`retry.rs`)\n\n- `CircuitBreaker`: threshold + cooldown, `try_lock()` on the success path to avoid hot-path contention.\n- `retry_async(policy, context, should_retry, op)`: the retry loop for anything failing as `Err`. Owns attempt counting, backoff and the per-retry log; returns `RetryFailure { error, attempts, exhausted }` so callers write their own terminal log.\n- `retry_backoff(base, retry, max)`: backoff only, for a loop that computes its own delay. `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath and applies no jitter, so call it directly only when the delay must be exact (`source.rs::nack_retry_delay`, whose tests assert exact values).\n- `HttpRetryMiddleware`: integrates with `reqwest-middleware`. Retries 429 + 5xx + network errors. Honors `Retry-After` on 429 and 5xx as a floor over the computed backoff, capped at a one-hour `MAX_RETRY_AFTER`, not `max_delay`.\n- `max_retries` = **total attempts** including the first try, not extra retries. Document if you change this convention. `meilisearch_sink` is the standing exception: its `max_retries` / `max_open_retries` count retries *after* the first, as its README states.\n- New helpers must take `Duration` (not `u64 millis`) on the public API. Internal computation uses `humantime` parsing of `String`.\n\n## `ConnectorState`\n\n- Bytes are opaque - the wrapping type does not impose schema.\n- Serialization is MessagePack via `rmp_serde`. Compact, deterministic, well-supported in serde.\n- `serialize`/`deserialize` helpers return `Option` and log on failure. **Failures are non-fatal** - design downstream code to tolerate fresh state.\n- Don't change the underlying format without coordinating with all sources.\n\n## Tests for SDK changes\n\n- Unit tests for any new pure function in the changed module.\n- Round-trip tests for new schemas/transforms (encode → decode == identity for JSON-equivalent payloads).\n- `sdk/tests/` integration tests when the change crosses module boundaries.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n\n# Rebuild all plugins to catch breaking-change leaks:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n# If FFI signatures changed, also build the runtime:\ncargo build -p iggy-connectors\n```\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-sink/SKILL.md":"---\nname: connector-sink\ndescription: Author a new Apache Iggy connector sink plugin under core/connectors/sinks/. Sinks consume messages from Apache Iggy streams and push them to an external system (DB, search engine, HTTP endpoint, object store). Load when creating, modifying, or reviewing a sink crate. Use for sink plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Sink\n\nA **sink** is a Rust `cdylib` that implements `iggy_connector_sdk::Sink`\nand exposes FFI symbols via the `sink_connector!` macro. The runtime\nloads it as `.so`/`.dylib`/`.dll`, polls Apache Iggy on its behalf,\nruns the transform chain, decodes via the configured `Schema`, and\nhands the sink batches of `ConsumedMessage`. The sink is responsible\nfor getting them to the external system reliably and efficiently.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's sink-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Sink::open` / `consume` / `close`) - that's an SDK change, not a plugin one.\n- Touching default `batch_size`, retry caps, or backoff defaults baked into existing in-tree sinks.\n- Promoting an `Err` variant to retryable when it previously was not - retrying bad data trips circuit breakers.\n- Adding a new sink that introduces a backend the runtime doesn't yet integration-test for.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand when authoring).\n- Exemplars: `stdout_sink` (minimal), `postgres_sink` (DB + transient detection), `http_sink` (validation, batch modes, retry middleware), `mongodb_sink` (atomic counters), `elasticsearch_sink` / `iceberg_sink` (backend-specific idioms).\n\n## Hard rules\n\n### Lifecycle\n\n- `Sink::consume` takes **`&self`** - never `&mut self`. Mutable state goes behind `tokio::sync::Mutex` / `AtomicU64`. Prefer atomics for hot counters - `mongodb_sink` uses `messages_processed: AtomicU64` + `insertion_errors: AtomicU64` and avoids a state lock on the hot path entirely.\n- `Sink::open` builds the client AND tests connectivity. Fail fast with `Error::InitError`.\n- `Sink::close` flushes + drops resources. Take ownership via `client: Option` + `.take()`. sqlx pools have `.close().await`. reqwest/mongodb/elasticsearch clients just drop. Log final stats either way.\n\n### Config\n\n- Every runtime-tunable field is `Option` with a default applied in `new()`. Forward-compat: adding a field doesn't break existing TOML.\n- Durations: `Option`, parsed via `humantime::Duration` in `new()`. Two equivalent in-tree idioms - inline `HumanDuration::from_str(&raw).map(|d| *d).unwrap_or_else(..)` (`random_source`, `postgres_source`) or a local `parse_duration` helper (`http_sink`, `sdk/src/retry.rs`). Fall back to a default on parse failure with `warn!` - never panic. We do **not** use `humantime_serde`.\n- **Conflict resolution**: if two config fields can be in invalid relative ordering, fix it in `new()` + `warn!` - don't error. `http_sink` swaps `retry_delay` and `max_retry_delay` when reversed to avoid an `ExponentialBackoff` panic downstream.\n- Validation order: structural in `new()`, connectivity in `open()`, **never in `consume()`**.\n\n### Payloads (efficiency-critical)\n\n- Dispatch on the `Payload` variant. Handle `Json`, `Raw`, `Text` and `Proto` at minimum. `Proto` holds text that is a JSON document when `proto_convert` had no descriptor or could not encode, and arbitrary text otherwise: call `Payload::json_document()` first and fall back to the text handling when it returns `None`. Unsupported -> `Error::InvalidPayloadType` or fall back to base64 (`elasticsearch_sink`).\n- For `Payload::Json`, call `try_to_bytes(&self)` - single-pass serialization without cloning the `OwnedValue` tree (`sdk/src/lib.rs::Payload::try_to_bytes`). Never `payload.clone().try_into_vec()`.\n- To take a payload out of `ConsumedMessage` without cloning: `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))`. Standard pattern across `http_sink::send_individual` / `send_ndjson` / `send_json_array`.\n- Pre-allocate per-batch buffers with `Vec::with_capacity(messages.len())` (or `* factor` for serialized bodies). Used across `postgres_sink`, `mongodb_sink`, `quickwit_sink`, `elasticsearch_sink`, `http_sink`.\n\n### Headers\n\n- `message.headers: Option>`. Check `.is_empty()` before iterating - runtime treats empty as `None`.\n- For binary values use `v.as_raw()` + base64. for text use `v.to_string_value()`. See `http_sink::EncodedHeader`.\n- `BTreeMap` for deterministic ordering. Don't collect into `HashMap`.\n\n### Errors\n\n| Scenario | SDK variant | Retry? |\n| ------------------------------------------ | --------------------------------------------------------------------------- | ---------------------------------------- |\n| Init / connectivity failure | `Error::InitError` | n/a (fails open) |\n| Bad config value | `Error::InvalidConfigValue(field)` | n/a |\n| Single-message validation failure | `Error::InvalidRecordValue(reason)` | no - skip + log |\n| Transient backend (5xx, conn reset, ...) | `Error::HttpRequestFailed` / `Error::Connection` / `Error::CannotStoreData` | yes |\n| Permanent 4xx, schema mismatch, constraint | `Error::PermanentHttpError` / `Error::SchemaMismatch` | **no** - retrying trips circuit breakers |\n| I/O failure mid-write (parquet/iceberg) | `Error::WriteFailure` | caller decides |\n| Catalog commit failure (Iceberg) | `Error::CatalogCommitError` | **no - not idempotent** |\n\n### Retry\n\n- SDK helpers cover the simple case: `iggy_connector_sdk::retry::check_connectivity_with_retry(...)` for `open()`, `HttpRetryMiddleware` for default 429/5xx/network policy on reqwest clients.\n- Custom strategies: `reqwest-middleware` + `reqwest_retry::RetryTransientMiddleware::new_with_policy_and_strategy`. `http_sink` defines its own `HttpSinkRetryStrategy` (honors `success_status_codes`, per-status decisions).\n- Non-HTTP clients: write `is_transient_error(&e)` mapping driver-specific codes. `postgres_sink::is_transient_error` maps SQLSTATEs `40001`, `40P01`, `57P01-03`, `08000/03/06`.\n- Retry loop: new connectors use `iggy_connector_sdk::retry::retry_async(policy, context, should_retry, op)` for anything that fails as `Err`. It owns attempt counting, backoff and the per-retry log, and returns `RetryFailure { error, attempts, exhausted }`; the caller logs the terminal failure.\n- Backoff only, for a loop that computes its own delay (it retries on an `Ok` response, carries a deadline, or reconnects between attempts): `retry_backoff(base, retry, max)`, where `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath, and it applies no jitter; call it directly only when the delay must be exact, as `sdk/src/source.rs::nack_retry_delay` needs for its tests.\n- Cap retries at 3 total attempts.\n\n### Idempotency\n\nRuntime retries on any `Err` - same batch can be redelivered. In-tree guarantees:\n\n- **Postgres** (`postgres_sink::build_batch_insert_query`): PK on `id`, but plain `INSERT` (no `ON CONFLICT`). Duplicate retry triggers `23505`, `is_transient_error` does NOT mark it transient -> error propagates. Adding `ON CONFLICT (id) DO NOTHING` is a known improvement.\n- **MongoDB**: composite `_id` from `stream:topic:partition:message_id`, via `insert_many(docs).ordered(false)`. Per-document duplicate-key errors don't abort the batch - effective idempotency under retry.\n- **Elasticsearch** (`bulk_index_documents`): bulk action specifies only `_index`, no `_id` -> ES auto-generates ids -> NOT idempotent. Extensions should add `\"_id\"`.\n- **HTTP**: endpoint-dependent. Document in the plugin README.\n- **Iceberg**: transactional commit. `Error::CatalogCommitError` documents that the txn is consumed on `commit()`. Rebuild-then-retry can duplicate data - callers must check the catalog first.\n\nWhen writing a new sink, prefer dedup-on-write using the message `id` field. Mongo's upsert-style is the cleanest reference.\n\n### Batching\n\n- Default `batch_size: 100`. Tune per-backend.\n- Chunk via `messages.chunks(self.batch_size)`. Do not buffer across `consume()` - runtime already batches at the consumer-poll boundary.\n- For HTTP backends support multiple batch modes when applicable (individual / ndjson / json_array / raw). See `http_sink::BatchMode`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: \", self.id);\ndebug!(\"Processing batch of {} messages, offset: {}\", batch.len(), offset);\nwarn!(\"Retry {attempt}/{max} for ID: {}, reason: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, processed: {}\", self.id, count);\n```\n\nUse literal API field names as labels (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` - runtime can't dlopen a regular rlib.\n2. Cloning the whole `Payload::Json` to serialize - use `try_to_bytes(&self)`.\n3. Holding `Mutex` across `.await` when the locked region does I/O.\n4. `&mut self` on `consume` - won't compile. you need interior mutability.\n5. Returning the first error and dropping subsequent batches - track `last_err`, process all, return.\n6. Forgetting to map `PermanentHttpError` - retries hammer the backend with bad data.\n7. Mutating config in `consume()` - config is a snapshot from `new()`. Reload requires runtime restart.\n8. Logging the connection string. Redact at construction: `http_sink::sanitize_url_for_log` populates `HttpSink.log_url`.\n\n## Tests\n\nUnit tests at EOF of `src/lib.rs`. See [connector-testing](../connector-testing/SKILL.md) for BDD naming + `test_config()` helper.\n\nReal-infra integration tests live under `core/integration/tests/connectors//` using `#[iggy_harness]` + a backend fixture that spins up Docker via `testcontainers-modules`. Plugins integrating with external services must have an integration test. Reference: `core/integration/tests/connectors/postgres/postgres_sink.rs` + `fixtures/postgres/`.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__sink --all-targets -- -D warnings\ncargo test -p iggy_connector__sink\n\n# Integration tests (filter by name; integration crate has no per-area target):\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sinks/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-transform/SKILL.md":"---\nname: connector-transform\ndescription: Add a new transform under `core/connectors/sdk/src/transforms/`. Transforms mutate, filter, or convert messages between source/encode and decode/sink. Use when adding field-level transforms (add/delete/update/filter), format conversions, envelope unwrapping, or similar config-driven message manipulations for Apache Iggy connectors. Use for transform authoring. NOT for runtime internals or sink/source loops.\n---\n\n# Adding a New Apache Iggy Connector Transform\n\nA transform implements `iggy_connector_sdk::transforms::Transform`. It\nruns after decode (for sinks) or before encode (for sources), in a chain\nconfigured per-connector. Config-driven, composable.\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers transform authoring only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [The trait](#the-trait)\n- [Steps to add a new transform](#steps-to-add-a-new-transform)\n- [Rules](#rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Adding a new transform that mutates payload SHAPE (renaming PK fields, dropping schema-required columns) - breaks downstream sinks silently.\n- Introducing non-determinism (system time, RNG without seed) - breaks tests + replay.\n- Changing the `Transform` trait signature (sync vs async, return type) - cascades to every transform.\n- Making transforms hold mutable runtime state - they're constructed once and `Arc`-cloned across tasks. mutation requires interior locking and is rarely what you actually want.\n\n## File map\n\n```text\nsdk/src/transforms/\n├── mod.rs Transform trait, TransformType enum, from_config() factory,\n│ FieldValue + ComputedValue helpers\n├── add_fields.rs AddFields - append static or computed fields to JSON\n├── delete_fields.rs DeleteFields - remove fields by key\n├── update_fields.rs UpdateFields - conditional field replacement\n├── filter_fields.rs FilterFields - keep/drop messages by key/value pattern\n├── unwrap_envelope.rs UnwrapEnvelope - promote nested field to top-level payload\n├── proto_convert.rs ProtoConvert - bidirectional Proto ⇄ Json/Text/Raw/etc.\n├── flatbuffer_convert.rs FlatBufferConvert\n├── avro_convert.rs AvroConvert\n└── json/ Shared JSON helpers (path traversal, ops)\n```\n\n## The trait\n\n```rust\n// sdk/src/transforms/mod.rs - Transform\npub trait Transform: Send + Sync {\n fn r#type(&self) -> TransformType;\n fn transform(\n &self,\n metadata: &TopicMetadata,\n message: DecodedMessage,\n ) -> Result, Error>;\n}\n```\n\nSemantics:\n\n- **`&self`** - transforms are immutable post-construction. Internal state (compiled regex, parsed paths, schema descriptors) is computed in `new()` and stored as plain struct fields. No locks needed - the whole instance is shared via `Arc`.\n- Takes `message` **by value** - the transform owns the `DecodedMessage` and may mutate freely without cloning.\n- Returns `Ok(Some(msg))` to pass through (possibly modified), `Ok(None)` to **filter** (drop silently), `Err(e)` for actual failures.\n- `&TopicMetadata` is for routing-aware logic. Most transforms ignore it.\n\n## Steps to add a new transform\n\n### 1. Create the file\n\n`sdk/src/transforms/my_transform.rs`. Apache 2.0 license header. Code reads top to bottom - config struct, then transform struct, then `impl`, then helpers.\n\n```rust\n/* Apache 2.0 header */\n\nuse crate::transforms::{Transform, TransformType};\nuse crate::{DecodedMessage, Error, Payload, TopicMetadata};\nuse serde::{Deserialize, Serialize};\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\npub struct MyTransformConfig {\n pub key: String,\n pub option: Option,\n}\n\npub struct MyTransform {\n config: MyTransformConfig,\n}\n\nimpl MyTransform {\n pub fn new(config: MyTransformConfig) -> Result {\n if config.key.is_empty() {\n return Err(Error::InvalidConfigValue(\n \"my_transform.key must not be empty\".into(),\n ));\n }\n Ok(Self { config })\n }\n}\n\nimpl Transform for MyTransform {\n fn r#type(&self) -> TransformType {\n TransformType::MyTransform\n }\n\n fn transform(\n &self,\n _metadata: &TopicMetadata,\n mut message: DecodedMessage,\n ) -> Result, Error> {\n // Most transforms operate only on Payload::Json - non-JSON is a no-op.\n let Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n };\n\n // Mutate `value` in place via simd_json::OwnedValue API.\n // Return Ok(None) to filter, Ok(Some(message)) to pass through.\n\n Ok(Some(message))\n }\n}\n```\n\n### 2. Wire into `transforms/mod.rs`\n\nThree edits, all required:\n\n(a) Module declaration + re-export at the top:\n\n```rust\nmod my_transform;\npub use my_transform::{MyTransform, MyTransformConfig};\n```\n\n(b) Discriminant in `TransformType`:\n\n```rust\npub enum TransformType {\n AddFields,\n // ...\n MyTransform, // snake_case form in TOML: \"my_transform\"\n}\n```\n\n(c) Branch in `from_config()`:\n\n```rust\nTransformType::MyTransform => {\n // For new transforms, prefer Error::InvalidConfigValue with formatted context -\n // it preserves the underlying serde_json error. Existing code uses bare\n // Error::InvalidConfig for most variants (UnwrapEnvelope is the exception).\n let cfg: MyTransformConfig = serde_json::from_value(raw.clone())\n .map_err(|error| Error::InvalidConfigValue(format!(\"my_transform: {error}\")))?;\n Ok(Arc::new(MyTransform::new(cfg)?))\n}\n```\n\n`from_config()` returns `Arc` so the same instance is shared across the pipeline.\n\n### 3. Document\n\nAdd a TOML usage example to `sdk/README.md`:\n\n```toml\n[transforms.my_transform]\nenabled = true\nkey = \"some_field\"\noption = true\n```\n\n### 4. Tests\n\nBottom of `my_transform.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn given_matching_field_should_transform() { ... }\n\n #[test]\n fn given_non_json_payload_should_passthrough() { ... }\n\n #[test]\n fn given_filter_condition_should_drop_message() { /* assert Ok(None) */ }\n\n #[test]\n fn given_invalid_config_should_error_in_new() {\n let cfg = MyTransformConfig { key: \"\".into(), option: None };\n assert!(matches!(MyTransform::new(cfg), Err(Error::InvalidConfigValue(_))));\n }\n\n fn topic_meta() -> TopicMetadata {\n TopicMetadata { stream: \"s\".into(), topic: \"t\".into() }\n }\n\n fn json_message(json: serde_json::Value) -> DecodedMessage { /* ... */ }\n}\n```\n\n## Rules\n\n### Operate on JSON only (mostly)\n\nMost field-level transforms only make sense for `Payload::Json`. Convention (see `add_fields.rs::AddFields::transform`): silently pass through non-JSON:\n\n```rust\nlet Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n};\n```\n\nFormat-conversion transforms (`proto_convert`, `avro_convert`, `flatbuffer_convert`) are the exception - they explicitly convert between variants.\n\n### Mutate `OwnedValue` in place, don't clone-then-replace\n\n`simd_json::OwnedValue` is mutable in place:\n\n```rust\nif let OwnedValue::Object(ref mut obj) = value {\n obj.insert(key.into(), new_value);\n}\n```\n\nDo NOT clone the value, mutate the clone, then `message.payload = Payload::Json(new)`. The whole point of `&mut value` is in-place mutation.\n\n### Filter semantics\n\n`Ok(None)` drops the message **silently** - no error log, no metric. By design (filters succeed when they decide to drop). For visibility, log at `debug!` level.\n\n### Config validation\n\nValidate in `new()` and return `Err(Error::InvalidConfigValue(...))`. Don't `panic!`. Don't validate on the hot path.\n\nFor transforms with regex or compiled patterns, compile once in `new()` and store the compiled form.\n\n### Stateless except for config\n\nTransforms are constructed once at startup and `Arc`-cloned across tasks. Must be `Send + Sync`. No mutable runtime state - if you need accumulator-like behavior, that's not a `Transform`, that's a sink.\n\n### Compiled state goes in the struct, not behind a lock\n\nThings derived from config (compiled regex, parsed paths, schema descriptors) should be computed in `new()` and stored as plain struct fields. The whole instance is shared via `Arc`, so per-field locking is unnecessary. See `sdk/src/transforms/filter_fields.rs::FilterFields::new` for the canonical pattern: regexes compile in `new()`, get stored as struct fields, are then read via `&self`.\n\n### `transform()` is sync\n\nThe trait method is sync (not `async`). Don't propose making it async - that would require pervasive change across the runtime. Schemas and other external resources should be loaded in `new()` (potentially via a fallible `try_new`), not lazily during `transform()`.\n\n### Errors\n\n| Scenario | Variant |\n| ---------------------------------- | -------------------------------------------------- |\n| Config validation in `new()` | `Error::InvalidConfigValue(\"transform_name: ...\")` |\n| Per-message validation (bad shape) | `Error::InvalidRecordValue(\"...\")` |\n| Schema resolution failure | `Error::InvalidTransformer` |\n| Format conversion failure | `Error::InvalidPayloadType` |\n\nReturning `Err` from `transform()` stops the message and is logged. For non-fatal field-level issues (field missing when transform expects it), prefer pass-through (`Ok(Some(message))`) unless config explicitly says to fail.\n\n## Common pitfalls\n\n1. **Forgetting one of the three `mod.rs` edits** - module declaration, `TransformType` variant, `from_config()` branch.\n2. **`#[serde(rename_all = \"snake_case\")]` mismatch** between enum variant name and TOML key.\n3. **Cloning `OwnedValue`** when in-place mutation would work.\n4. **Validating config inside `transform()`** instead of `new()`.\n5. **`std::sync::Mutex` for internal state** on the hot path - precompute in `new()` and store as plain struct fields. Transforms run on every message. locks are wasted work.\n6. **Returning `Err` for a filter** - filters use `Ok(None)`.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n```\n\nVerify by adding the transform to `core/connectors/runtime/example_config/connectors/` and running the runtime - the transform should apply to each message.\n\nIf the transform changes payload shape sufficiently to break downstream sinks (e.g., renaming primary keys), call it out in the PR description.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-source/SKILL.md":"---\nname: connector-source\ndescription: Author a new Apache Iggy connector source plugin under core/connectors/sources/. Sources poll an external system (DB, API, queue) and produce messages into Apache Iggy streams. Load when creating, modifying, or reviewing a source crate. Use for source plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Source\n\nA **source** is a Rust `cdylib` that implements\n`iggy_connector_sdk::Source` and exposes FFI symbols via the\n`source_connector!` macro. The runtime calls `poll()` in a loop,\napplies transforms, encodes via the configured `Schema`, sends to\nApache Iggy, and persists the returned `ConnectorState` after every\nsuccessful send.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's source-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Source::open` / `poll` / `on_batch_result` / `close`) - that's an SDK change.\n- Adding a long-running side task in the plugin - the runtime owns lifecycle, and orphans survive `close()`. Sanctioned only where the source is itself a server the runtime cannot drive, as in `http_source`'s listener, and then only with an explicit shutdown in the last `close()` that awaits its tasks before returning.\n- Persisting unbounded state - `State` is rewritten every batch.\n- Adding a source that requires authoritative offsets external to Apache Iggy without coordinating retention.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand).\n- Exemplars: `random_source` (minimal + canonical state tests), `postgres_source` (cursor / delete-after-read / processed-column modes, restart-survives-state tests), `elasticsearch_source` (scroll cursor), `influxdb_source` (time-series scan).\n\n## Hard rules\n\n### `poll()` signature is `&self`\n\nThe macro shares the source as `Arc` across the FFI callback and forwarding loop. Signature: `async fn poll(&self) -> ...` - any mutable state behind `tokio::sync::Mutex`. **Single most common new-contributor mistake.**\n\n### Lock discipline\n\nNever hold the state `Mutex` across upstream I/O. Build a candidate from committed\nstate, then stage it until the runtime reports the batch result:\n\n```rust\nlet mut candidate = self.state.lock().await.clone();\nlet rows = client.query(&sql, &[&candidate.cursor]).await?;\ncandidate.cursor = Some(new_cursor);\nlet persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n*self.pending.lock().await = Some(candidate);\n```\n\n### Delivery acknowledgment\n\n`on_batch_result` (added by #3855) is how a source learns what happened to the batch it just\nreturned. The SDK keeps exactly one batch in flight: it will not call `poll()` again until this\nreturns. By default it stops after `MAX_CONSECUTIVE_NACKS` (5) consecutive NACKs; a source can\noverride `batch_policy()` to disable the limit when its accepted input cannot be replayed after\nrestart. A stop is reported to the runtime only when the plugin exports the current SDK's stop\ncallback, so rebuild older source plugins.\n\n- `Ack` means the runtime sent the batch **and** persisted its state. `Nack` means it could not\n confirm both, which is **not** the same as neither happening: a batch that reached the topic but\n whose state save failed is NACKed. A missing runtime result leaves the batch pending; it does not\n produce a synthetic NACK. A source that replays on `Nack` is at-least-once, not exactly-once.\n- The trait has a **default no-op**, which suits only a source with no staged cursor and no\n destructive work. If `poll()` advances a cursor, deletes rows, or drains an in-memory buffer,\n omitting this loses data silently and nothing will tell you. `random_source` and `http_source`\n implement it; the other shipped sources take the default and skip rows on a NACK.\n- Stage in `poll()`, apply on `Ack`, discard or replay on `Nack`. A source whose input is pushed to\n it, rather than re-readable upstream, has to hold the batch itself: see `http_source`'s staging.\n- Returning `Err` from it stops the source immediately, so it is not a retry signal.\n\n### State persistence\n\n- `ConnectorState` is `Vec` via MessagePack (`rmp_serde`). Use\n `ConnectorState::serialize(&state, NAME, id)` and\n `ConnectorState::deserialize::(NAME, id)`.\n- `poll()` must not commit cursors or destructive work. Return messages with candidate state and\n keep the corresponding work staged.\n- The runtime sends the batch, saves its candidate state to\n `{state_path}/source_{key}.state`, then calls `on_batch_result(Ack)`. Commit staged in-memory\n state and external delete or mark operations only on ACK. A NACK discards the candidate so the\n same data can be polled again.\n- A crash between `poll()` returning and state persistence leaves the prior cursor for the next\n poll, so downstream must tolerate at-least-once delivery.\n- Cursor sources that stage state until `on_batch_result` can attach state to the corresponding\n message batch. Control-plane state that is independent of a publish can ride an empty batch\n instead.\n- Return `state: None` for an empty poll when no watermark changed. If an empty poll advances a\n watermark, stage and return the new state through the same ACK handshake.\n- The runtime can still NACK an empty batch when state storage is latched or a pending checkpoint\n cannot resolve. Never treat hand-off as durable; `on_batch_result` reports the outcome.\n- Treat candidate-state serialization failure as a poll error. Do not send messages without the\n state needed to resume them safely.\n- Keep `State` small - rewritten every batch. No unbounded vecs.\n\nThe SDK allows one in-flight batch. Five consecutive NACKs stop a source using\nthe default policy and require a manual restart. Returning `Err` from\n`on_batch_result` is fatal regardless of the NACK limit, so\nretry transient backend failures inside the callback before returning an error.\nIf the runtime has not reported ACK or NACK after 30 seconds, the SDK warns\nand keeps waiting for that result without polling or replaying the batch.\nOnce the result is received, the SDK waits for `on_batch_result` to finish,\nso the callback must bound its own connection acquisition and retry budget.\n\n### Sleep first\n\n`poll()` must `sleep(self.poll_interval).await` before any work. Without it, an empty source spins a CPU.\n\n### Schema selection\n\nMatch `ProducedMessages.schema` to the bytes in `messages[i].payload`:\n\n- JSON-serialized rows → `Schema::Json`\n- Already-protobuf bytes → `Schema::Proto`\n- Already-avro bytes → `Schema::Avro`\n- Opaque → `Schema::Raw`\n\n### IDs and timestamps\n\n- `ProducedMessage.id: Option` - set when a natural ID exists (DB PK, document id). It rides the wire as the message id, which is what a **consumer** can dedupe an at-least-once duplicate on.\n- **Iggy itself does not dedupe on that id.** The server's only read of the field is `core/server/src/http/wire.rs`, which mints a fresh uuid when the incoming id is 0 and otherwise passes it through. Its dedup path is a per-client request-id watermark (`dedup_clients_max`) that never looks at this field.\n- `origin_timestamp: Option` - source-system event time in nanoseconds. Lets downstream sinks reason about lag.\n- `timestamp` and `checksum` are Iggy-side - leave `None`.\n\n### Concurrency\n\n- Runtime spawns ONE `poll()` task per source. No concurrent `poll()`.\n- Don't spawn your own long-running Tokio tasks: the runtime owns lifecycle. The exception is a source that listens rather than polls, which has to own its listener; `http_source` is the worked example, and it shuts its tasks down in the last `close()` rather than leaving them to outlive the connector.\n\n### Errors\n\n| Scenario | Variant |\n| ------------------------------------------- | ------------------------------------------------- |\n| Bad config in `new()`/`open()` | `Error::InitError` |\n| Cannot reach external system at startup | `Error::InitError` or `Error::Connection` |\n| Transient fetch failure (retry-worthy) | `Error::Connection` or `Error::HttpRequestFailed` |\n| Permanent fetch failure (auth, schema gone) | `Error::PermanentHttpError` |\n| Row failed to serialize | `Error::Serialization(...)` |\n| State serialization failed | `Error::Serialization(...)` |\n\nReturning `Err` from `poll()` is only logged by the SDK's FFI bridge\n(`sdk/src/source.rs::handle_messages`) - the loop continues, the next\n`poll()` runs. Connector status does NOT flip to `Error` from a poll\nfailure. Status `Error` is set by the runtime only on transform/encode\nfailure, Iggy send failure, or state save failure\n(`runtime/src/source.rs::source_forwarding_loop` calls to\n`context.sources.set_error`). To surface a poll failure as unhealth,\nraise it through the metric counter or escalate to `Error::InitError`\nfrom `open()`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: {}\", self.id, ...);\ninfo!(\"Restored state for ID: {id}, cursor: {:?}\", ...);\ndebug!(\"Polled {} rows for ID: {}\", rows.len(), self.id);\nwarn!(\"Transient fetch failure for ID: {}, will retry: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, total produced: {}\", self.id, ...);\n```\n\nIggy consumer-loop labels use literal API names (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. `async fn poll(&mut self)` - won't compile. Use `&self` + `Mutex`.\n2. Holding `state.lock()` across the fetch I/O - blocks `close()`, causes shutdown timeouts.\n3. Forgetting to sleep - 100% CPU on idle source.\n4. Committing a cursor or deleting source data in `poll()` - stage it and wait for ACK.\n5. Unbounded data in `State` - rewritten every batch. keep O(constant).\n6. `std::sync::Mutex` - blocks the executor. Use `tokio::sync::Mutex`.\n7. Not setting `ProducedMessage.id` when a stable ID exists - leaves a consumer nothing to dedupe a replayed duplicate on. It does not make the write idempotent server-side, because nothing there reads it.\n8. Spawning side tasks - the runtime owns the scheduler. The one exception is a source that listens rather than polls and must own its listener (see [Concurrency](#concurrency) and the STOP list); it owes an explicit shutdown in the last `close()` that awaits its tasks.\n\n## Tests\n\nMandatory four canonical source state tests (see [connector-testing](../connector-testing/SKILL.md) for the full pattern). Copy from `sources/random_source/src/lib.rs::tests`. Plus config defaults, payload building, schema selection.\n\nIntegration tests under `core/integration/tests/connectors//` for any source backed by external infra. Use `#[iggy_harness]` + a `TestFixture` backed by `testcontainers-modules`. Reference: `core/integration/tests/connectors/postgres/postgres_source.rs` (multi-mode tests) + `restart.rs` (state survives restart). Exercise both ACK and NACK paths when the source stages cursors or destructive work.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__source --all-targets -- -D warnings\ncargo test -p iggy_connector__source\n\n# Integration tests:\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sources/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-runtime/SKILL.md":"---\nname: connector-runtime\ndescription: Modify the Apache Iggy Connectors runtime (`core/connectors/runtime/`) - the host process that loads plugins, manages lifecycle, and bridges Apache Iggy streams to plugins via FFI. Use when changing plugin loading, the FFI dispatch, the sink/source manager, state storage, config providers, the HTTP control API, metrics, or transforms wiring. NOT for plugin code.\n---\n\n# Modifying the Apache Iggy Connectors Runtime\n\nRuntime is the host. Loads `.so`/`.dylib`/`.dll` plugins, drives their\nlifecycle, ferries messages between Apache Iggy and plugins, exposes\n`/stats`, `/metrics`, control HTTP API. Bugs here affect every plugin.\n\n> **Universal connector rules** (benchmark, SecretString, verbose, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers runtime internals only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Architecture invariants](#architecture-invariants)\n- [State storage (`state.rs`)](#state-storage-staters)\n- [Source forwarding loop](#source-forwarding-loop)\n- [Sink consumption loop](#sink-consumption-loop)\n- [Logging format](#logging-format)\n- [The `verbose` flag](#the-verbose-flag)\n- [Config provider rules](#config-provider-rules)\n- [Error categorization](#error-categorization)\n- [Metrics (`metrics.rs`)](#metrics-metricsrs)\n- [Drop accounting (wired sites)](#drop-accounting-wired-sites)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing FFI `#[repr(C)]` layouts (`SourceApi`, `SinkApi`, `RawMessages`, headers) - existing plugin `.so`s misalign.\n- Modifying `state.rs` save protocol (atomic tmp + rename + fsync ordering) - corruption risk.\n- Renaming the default consumer group `iggy-connect-sink-{key}` - operators hold offsets there.\n- Hot-reloading or unloading a `.so` - mid-FFI tasks would segfault. Architectural change required.\n- Changing the SerDe split (postcard for FFI payloads, serde_json for configs, rmp_serde for state) - downstream incompat.\n\n## File map\n\n```text\nruntime/src/\n├── main.rs Entry, plugin path resolution, SourceApi/SinkApi FFI structs\n├── sink.rs Sink lifecycle, Iggy consumer wiring, FFI consume calls\n├── source.rs Source lifecycle, flume forwarding, state save loop\n├── stream.rs Stream + consumer/producer setup\n├── transform.rs Loads transforms from config, applies them in chain\n├── state.rs FileStateProvider, atomic ConnectorState file I/O\n├── context.rs Per-instance context (Iggy client, metrics, config)\n├── error.rs RuntimeError with as_code() for HTTP responses\n├── log.rs init_logging (text/JSON x telemetry-on/off matrix) + LOG_CALLBACK\n├── benchmark.rs emit_sink_event / emit_source_event + as_micros\n├── metrics.rs Prometheus families + gauges + stage histograms + label caches\n├── stats.rs /stats endpoint payload assembly\n├── manager/{mod,sink,source}.rs SinkManager / SourceManager, status, restart_guard\n├── configs/\n│ ├── runtime.rs RuntimeConfig + LoggingConfig + LogFormat\n│ ├── connectors.rs ConnectorConfig, SinkConfig, SourceConfig (verbose + benchmark)\n│ └── connectors/{local,http}_provider.rs\n└── api/ HTTP control + observability endpoints\n```\n\n## Architecture invariants\n\nLoad-bearing assumptions. Breaking them silently corrupts data or hangs.\n\n### Plugin instance identity\n\n- Each instance gets a globally unique `plugin_id: u32` from `PLUGIN_ID: AtomicU32` in `main.rs`. Never reused.\n- Same `.so` may load many instances. each gets a fresh `plugin_id` but shares the `Container` (dlopen handle) via `Arc` (`manager/{sink,source}.rs`).\n- All FFI calls keyed by `plugin_id`. Mixing IDs = data to the wrong plugin.\n\n### FFI pointer lifetimes\n\n- Config and state are `(*const u8, usize)`. Valid for the call duration only. Plugin must copy if it needs the bytes past the call. SDK macros do this correctly. don't bypass.\n- `LogCallback` lifetime spans the whole process - kept static in `log.rs`. Don't make it dynamic.\n\n### Container ownership\n\n- `dlopen2::wrapper::Container<{Sink,Source}Api>` (defined in `main.rs`) is shared via `Arc` across all instances of the same `.so`.\n- Arc must outlive every spawned task that calls into the plugin. Manager holds the Arc in `{Sink,Source}Details` for the connector lifetime.\n- Unloading a `.so` while a task is mid-FFI-call would segfault. Runtime does not unload. Hot reload needs a rearchitect.\n\n### Plugin manager state\n\n- `SinkManager` / `SourceManager` use `DashMap>>`. Key = TOML `key` field.\n- Status transitions (`Starting → Running → Stopped/Error`) update running counters only on transitions to/from `Running`. See `SinkManager::update_status`.\n- `restart_guard: Arc>` prevents concurrent restart of the same connector. `restart_connector()` uses `try_lock()` - returns OK without restart if busy. Intentional anti-thundering-herd.\n\n### Serialization split\n\n- FFI message payloads (`TopicMetadata`, `MessagesMetadata`, `RawMessages`, headers): `postcard` (compact, stable wire).\n- Connector configs passed to plugin: `serde_json` (TOML in, JSON over FFI - human-editable).\n- `ConnectorState` bytes inside plugins: `rmp_serde` (MessagePack - compact, opaque to runtime).\n\nDon't mix.\n\n## State storage (`state.rs`)\n\n- File path: `{state_path}/source_{connector_key}.state`. Mode `0o600` on Unix.\n- `save()` is crash-atomic. `rename(2)` is atomic in the namespace (no observer sees a half-renamed file). the preceding `sync_data` on the tmp file plus the post-rename parent-dir `sync_all` are what make a crash leave either the old or new content - never truncated. The parent-dir fsync failure is propagated as `CannotWriteStateFile` (not swallowed) so a lost rename surfaces instead of silently restarting the source from scratch. A `Mutex<()>` serializes concurrent saves on the same provider.\n- Save after every successful Apache Iggy send. Save failure logs + continues. next batch retries.\n- `load()` returns `Ok(None)` for missing or empty files. After a `NotFound` read, re-stats the parent directory: missing parent -> `Err(CannotOpenStateFile)` so a broken state path fails at init rather than masquerading as \"fresh start\".\n- Sinks have no state - only sources.\n\n## Source forwarding loop\n\n1. `iggy_source_handle(id, send_callback)` - plugin registers itself.\n2. Plugin polls + invokes `send_callback(plugin_id, ptr, len)`.\n3. Callback runs in the SDK macro's spawned async task. Pushes postcard `ProducedMessages` into a `flume` channel keyed by `plugin_id` in `SOURCE_SENDERS: Lazy>` (`pub(crate)`). `SourceSenderEntry` wraps the sender + a pre-extracted owned `Counter` (the `errors` series, `Arc` inside). The FFI callback bumps errors on deserialize or channel-closed failure with one relaxed atomic - no `Family` lookup, no `Arc` handle.\n4. The runtime registers the optional source-stop callback, then `source_forwarding_loop` pulls from the channel, deserializes, applies transforms, encodes via `StreamEncoder`, and sends to the Iggy producer. The SDK warns after 30 seconds without a result but keeps the same batch pending; it does not poll or enqueue another copy.\n5. On success, save returned `ConnectorState` via `FileStateProvider`, then return the batch result to the plugin. A result NACK may cause replay with capped backoff.\n6. A stop-callback reason ends the forwarding loop and reports an unexpected stop. Closing the stop channel on normal shutdown disables that select arm, allowing already queued batches to receive results before the forwarding channel disconnects.\n7. At loop exit, clean up `SOURCE_SENDERS` and update status and metrics. A self-stop reports `Error` and removes the source from the running gauge; normal shutdown reports `Stopped`.\n\n**Shutdown ordering (`manager/source.rs::stop_connector`):**\n\n1. Call `iggy_source_close` FIRST. It blocks until the plugin's polling task stops, so no new send callbacks fire after it returns.\n2. `cleanup_sender(plugin_id)` NEXT - dropping the channel sender lets the forwarding task finish any queued batches before `recv_async()` resolves with `Disconnected` and exits cleanly, instead of blocking until the abort timeout.\n3. Finally await spawned handlers with `tokio::time::timeout`. On timeout, `handle.abort()` + drain - prevents leaked tasks colliding with the next `start_connector` (a late `file.save()` could otherwise race the new instance). The silent-drop branch in `handle_produced_messages` only covers the window between close and cleanup.\n\nGotchas:\n\n- `SOURCE_SENDERS` must be cleaned up on connector close or memory leaks (channel + task).\n- `spawn_source_handler` wraps the outer `iggy_source_handle(id, callback)` FFI call in `tokio::task::spawn_blocking()`. That call returns quickly - the SDK macro internally `runtime.spawn`s an async `handle_messages` task and returns. **`send_callback` invocations come from that async task, not from `spawn_blocking`.** A long-running synchronous poll inside the plugin would block one Tokio worker. async polls don't.\n- No timeout on the registration call - a plugin whose `iggy_source_handle` never returns stalls one blocking worker for the process lifetime.\n\n## Sink consumption loop\n\nPer `[[streams]]` entry per sink:\n\n1. Build / ensure consumer group `iggy-connect-sink-{key}` (`connect` matches the docker image name `apache/iggy-connect` - see `runtime/src/sink.rs::default_consumer_group`). Overridable via the stream config's `consumer_group`.\n2. Spawn one task per topic (`spawn_consume_tasks`).\n3. Poll Iggy → batch messages (default `batch_length = 1000`, `poll_interval = 5ms`). A `consumer.next()` `Err` (transport/decode at the Iggy client boundary) bumps `errors` and `continue`s - offset already auto-committed via `AutoCommit::When(AutoCommitWhen::PollingMessages)`, message effectively dropped.\n4. Decode via `Schema::decoder()`. Decode failure logs + bumps `errors`. `messages_filtered` is reserved for the transform filter contract (`Ok(None)`).\n5. Apply transforms (chain). A transform `Err` is logged, bumps `errors`, and drops only that message (drop-and-continue, mirroring the source) - one bad payload never kills the batch. `Ok(None)` filters the message - bumps `messages_filtered{connector_type=\"sink\"}`.\n6. Postcard-encode batch as `RawMessages`. Headers as a separate postcard blob. Per-message failures (missing field, payload conversion, header serialization) bump `errors` + skip. Batch-level failures (postcard serialization of metadata / `RawMessages`) propagate `Err` -> `spawn_consume_tasks` wrapper bumps `errors` + `set_error`. The total histogram and benchmark emit still fire for the failed batch before the `Err` propagates.\n7. Call `iggy_sink_consume(id, topic_meta, messages_meta, messages)`.\n8. Non-zero return → log + `errors` increment.\n\nPer-message counters (decode errors, transform errors, filters, field/serialize drops) are accumulated and flushed once per batch via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels`, not one `Family` lookup per message.\n\nStages (both labels snake_case): `decode` (schema decode loop), `prepare` (transform + serialize = total - decode - ffi), `ffi` (plugin consume call), `total`. Same `decode` / `prepare` meaning as the source side.\n\nGotchas:\n\n- Batch flushes on `current_offset != message_offset` (gap) OR batch full - higher latency for low-volume topics.\n- Auto-commit happens on poll - idempotency at the sink protects against re-delivery on crash before plugin write.\n\n## Logging format\n\n`LoggingConfig` in `configs/runtime.rs` exposes `format: LogFormat` (`Text` default, `Json`), env-addressable via `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `log::init_logging` matches on `(telemetry.enabled, format)` and installs the right `fmt::layer()`/`fmt::layer().json()` + OpenTelemetry layer. OTel pipeline untouched by `format` - only the stdout layer switches.\n\n`fmt::layer()` and `fmt::layer().json()` are different concrete types, so the stdout layer is built once as a `Box>` (via `.boxed()`) by matching on `format`, then a single 2-arm branch on `telemetry.enabled` attaches the OTel layers. When extending, keep the boxed-layer + 2-arm shape rather than re-expanding into a 4-way match.\n\n## The `verbose` flag\n\n`SinkConfig` and `SourceConfig` in `configs/connectors.rs` carry `verbose: bool` (default false). Runtime threads it to `spawn_consume_tasks` / source equivalent to gate per-batch log promotion:\n\n```rust\nif verbose {\n info!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n} else {\n debug!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n}\n```\n\nNew per-batch logging follows the same pattern. Default to `debug!`, upgrade to `info!` when `verbose`. No third level.\n\n## Config provider rules\n\n### Local (`configs/connectors/local_provider.rs`)\n\n- Discovery walk reads `*.toml` from `config_dir`. Skips hidden files (`.`) and `Cargo.toml`.\n- Filename: `{key}_{type}[_v{N}].toml` (e.g., `postgres_sink_v2.toml`).\n- One TOML deserialized into `SinkConfig` or `SourceConfig`. Grouped by `key`. Highest `version` wins unless `.active_versions.toml` overrides.\n- `.active_versions.toml` is NOT read via the discovery walk (hidden). Read via direct path through `LocalConnectorsConfigProvider::active_versions_file_path`.\n\n### HTTP (`configs/connectors/http_provider.rs`)\n\n- `reqwest_middleware` + `RetryTransientMiddleware`. Retries 5xx + connection errors. Never 4xx (`PermanentHttpError` convention).\n- Default URL templates documented in `runtime/README.md`. Configurable via TOML.\n\n### Env-var overrides via `ConfigEnv` derive\n\n`SinkConfig`, `SourceConfig`, and inner structs derive `ConfigEnv` (`configs_derive::ConfigEnv`). Generates env-var addressability as `IGGY_CONNECTORS___` for primitive fields. Used heavily by integration tests to inject testcontainer ports - see `core/integration/tests/connectors/fixtures/postgres/container.rs` for env-var constants. Mark new compound fields `#[config_env(skip)]`, leaf primitives `#[config_env(leaf)]`.\n\n### Versioning\n\n- Configs carry `version: u64`. Local provider auto-increments if absent. HTTP provider trusts the server.\n- Restart on version change uses `restart_guard`.\n\n## Error categorization\n\n`RuntimeError` (`error.rs`) carries `as_code()` for HTTP API responses.\n\n| Class | Fatal? | Example |\n| ------------------ | --------------------------- | -------------------------------- |\n| Config load | yes (process exit) | Bad TOML, missing required field |\n| Iggy client init | yes | Auth failure at startup |\n| State dir create | yes | Permission denied |\n| Plugin .so resolve | per-plugin (`FailedPlugin`) | Missing file |\n| Plugin open FFI | per-plugin (status = Error) | Plugin returned non-zero |\n| Message decode | per-message (skip) | Bad protobuf bytes |\n| Iggy send | per-batch (metric) | Network blip |\n| State save | per-batch (log) | Disk full |\n\nFatal errors propagate to `main` and exit. Per-connector / per-message errors are isolated.\n\n## Metrics (`metrics.rs`)\n\nAll families labeled by `connector_key` + `connector_type` (histogram adds `stage`):\n\n- **Counters**: `iggy_connector_messages_{produced,sent,consumed,processed,filtered}_total` and `iggy_connector_errors_total`. These are the *rendered* names; each is registered without the `_total`, which the OpenMetrics encoder appends.\n - `messages_filtered_total` - intentional drops via transform `Ok(None)`.\n - `errors_total` - unexpected drops (decode/encode/build failure, missing field, ...) + batch-level failures, one per failed run.\n - `sink_runs_total` - FFI `consume()` calls for sink batches, one per contiguous payload-variant run. Divide by `stage_duration_seconds_count{stage=\"total\"}` for runs per batch.\n- **Histograms**: `iggy_connector_stage_duration_seconds{stage}` (snake_case stage labels - `prepare`, `ffi`, `decode`, `iggy_send`, `state_save`, `total`). Buckets `STAGE_BUCKETS_SECONDS`. Always populated regardless of any flag. One sample per batch for every stage; `ffi` is summed over the batch's runs. Scraped at `/metrics` when `[http.metrics] enabled = true`.\n- **Gauges**: `iggy_connectors_{sources,sinks}_{total,running}`.\n\n`/stats` JSON surface mirrors counters per-connector via `ConnectorStats` (`sdk/api.rs`): `messages_filtered`, `errors`, kind-specific counters.\n\nWhen adding a metric:\n\n- Add family to `Metrics` struct + `init`, register with name + help text. Never end a `Counter` family's registered name in `_total`: the encoder appends it and the series renders `_total_total`. Gauges get no suffix, so a gauge name may end in `_total` literally.\n- New label sets define `EncodeLabelSet` struct + label enum (hand-impl `EncodeLabelValue` for snake_case values - the derive emits PascalCase).\n- Histograms: pass `fn() -> Histogram` to `Family::new_with_constructor`.\n- Add unit tests under `mod tests` with `given_*_when_*_should_*` BDD names.\n\n## Drop accounting (wired sites)\n\nPer-message drops in the batch loops are counted into a local `u64` and flushed once after the loop via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels` - one `Family` lookup per batch, not per message. One-shot drops outside a loop (e.g. `consumer.next()` Err) still call `inc_errors_with_labels` directly. Hot path uses pre-built `SinkLabels`/`SourceLabels`. `&str`-based wrappers are `#[cfg(test)]`-only.\n\n- `sink.rs::consume_messages` - `consumer.next()` Err\n- `sink.rs::process_messages` - decode, transform Err (drop-and-continue), missing required fields, payload conversion, header serialization (accumulated, flushed once per batch)\n- `sink.rs::spawn_consume_tasks` task wrapper - bumps once on `consume_messages` Err\n- `source.rs::source_forwarding_loop` - payload decode, prepare (transform/encode) failure, Iggy send Err, state save Err\n- `source.rs::process_messages` - transform Err (logs + bumps `errors` + continue. does NOT propagate, so one bad payload doesn't flip the connector to permanent ERROR), transform encode failure, `build_iggy_message` failure\n- `source.rs::handle_produced_messages` - postcard deserialize failure, `sender.send` channel-closed\n\nFilter case bumps `messages_filtered` via `inc_messages_filtered_with_labels`. Adding a new drop path: mirror this pattern.\n\n## Hard rules\n\n1. **Never block the executor.** All I/O async. `spawn_blocking` only for FFI registration (already in place).\n2. **Plugin ID counter is monotonic.** No reset, no reuse.\n3. **FFI return codes:** `0` success, non-zero failure.\n4. **Don't add static mutable state** beyond `LOG_CALLBACK`, `PLUGIN_ID`, `SOURCE_SENDERS`.\n5. **Pair `cleanup_sender(id)` with shutdown** for sources (avoid flume leak). Order: close FFI -> cleanup sender -> drain/abort tasks.\n6. **Restart uses `restart_guard.try_lock()`** - no thundering-herd regression.\n7. **No timeouts on plugin FFI calls without a kill-task strategy.** A timeout that returns from the runtime but leaves the plugin running has the worst of both worlds.\n\n## Common pitfalls\n\n1. Mutating `INSTANCES` (in SDK macro) from runtime side - impossible by design. only the macro touches it.\n2. Calling `iggy_*_open` with a duplicate ID - returns `-1`. Means runtime forgot to `iggy_*_close` first.\n3. Holding a `DashMap` shard guard across `.await` - lockups. Pattern: `.get().clone()`, drop guard, await.\n4. Adding fields to `#[repr(C)]` types without an SDK version bump - existing plugins misalign on `postcard::from_bytes`.\n5. Changing the default consumer group name without coordinating with operators - they have offsets stored there.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy-connectors --all-targets -- -D warnings\ncargo test -p iggy-connectors\n\n# Sanity-build in-tree plugins:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n\n# Runtime-focused integration tests:\ncargo test -p integration -- connectors::runtime::\n\n# Smoke-test with example config:\nIGGY_CONNECTORS_CONFIG_PATH=core/connectors/runtime/example_config/config.toml \\\n cargo run --bin iggy-connectors\n```\n\nUpdate `runtime/README.md` if endpoints, env vars, or config schema change.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connectors-overview/SKILL.md":"---\nname: connectors-overview\ndescription: Entry point and index for the Apache Iggy Connectors subsystem (`core/connectors/`). Load this first when working anywhere under `core/connectors/` - runtime, SDK, sinks, sources, or transforms. Routes to the focused per-area skills.\n---\n\n# Apache Iggy Connectors - Overview\n\nRepo-wide rules (Apache headers, fmt/sort/clippy order, idiomatic Rust traits, imports at top, no `cargo install`, LLM-slop tells) live in [AGENTS.md](../../../AGENTS.md). This file owns **connector-specific** rules and routes to the per-area skills.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [What sinks and sources are](#what-sinks-and-sources-are)\n- [Which skill to load](#which-skill-to-load)\n- [Stick to conventions](#stick-to-conventions)\n- [Connector-wide rules](#connector-wide-rules)\n- [Benchmark mode (per-batch timing observability)](#benchmark-mode-per-batch-timing-observability)\n- [Logging format](#logging-format)\n- [Exemplars](#exemplars)\n- [Concrete efficiency patterns](#concrete-efficiency-patterns)\n- [Drop accounting](#drop-accounting)\n- [Common review smells](#common-review-smells)\n- [File map](#file-map)\n- [Cite by symbol, not line number](#cite-by-symbol-not-line-number)\n\n## STOP and ask the user before\n\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI signature in `sdk/src/{sink,source}.rs` - breaks every pre-built plugin `.so`.\n- Changing the runtime's wire conventions (postcard FFI payload structs, default consumer group naming, plugin path resolution).\n- Modifying `runtime/src/state.rs` save protocol (atomic rename + fsync ordering) - corruption risk.\n- Renaming or repurposing a `Schema` variant - decoders/encoders pinned to wire bytes.\n- Promoting a transient `Error` variant to `Permanent*` or vice versa - affects retry behavior across every plugin.\n\n## What sinks and sources are\n\nA **sink** is a plugin that **consumes** messages from Apache Iggy streams and writes them to an external system (Postgres, Elasticsearch, Iceberg, HTTP endpoint, Mongo, object store, stdout, ...). Implements `iggy_connector_sdk::Sink`.\n\nA **source** is a plugin that **produces** messages from an external system into Apache Iggy streams (poll a DB table, scroll an ES index, generate random data, ...). Implements `iggy_connector_sdk::Source`.\n\nBoth compile as `cdylib` shared libraries (`.so`/`.dylib`/`.dll`) loaded by the **connectors runtime** at startup via `dlopen`. The runtime drives lifecycle (open/handle/consume/close), bridges Apache Iggy ⇄ plugin via a small FFI, applies optional transforms, and persists source state.\n\n```text\n ┌─ optional transforms ─┐\nExternal ──poll──▶ SOURCE ──FFI──▶ RUNTIME ──encode──▶ Apache Iggy stream\n system plugin ▲ ▲\n │ │\n state save (msgpack) │\n │\n ┌─ optional transforms ─┐ │\nApache Iggy stream ──decode──▶ RUNTIME ──FFI──▶ SINK ──write──▶ External\n plugin system\n```\n\nHeaders set on the source side ride through transforms (which may modify, drop, or pass them) and arrive at the sink with `BTreeMap` preserved deterministically.\n\n## Which skill to load\n\n| Task | Skill |\n| --------------------------------------------------- | --------------------- |\n| Write a new sink plugin | `connector-sink` |\n| Write a new source plugin | `connector-source` |\n| Add schema / decoder / encoder / SDK trait surface | `connector-sdk` |\n| Change runtime internals (FFI, manager, state, ...) | `connector-runtime` |\n| Add a transform (field-level or format conversion) | `connector-transform` |\n| Write unit / integration tests for any of the above | `connector-testing` |\n\n## Stick to conventions\n\nThe connectors codebase is intentionally repetitive across plugins. Cross-plugin consistency makes review, LLM-assisted contribution, and onboarding tractable.\n\n1. **Find the closest existing plugin by shape** (DB-write -> `postgres_sink`, HTTP -> `http_sink`, polling source -> `postgres_source`).\n2. **Copy its file layout, naming, log format, error mapping, test structure** verbatim. Only the backend-specific code (the client call) should differ.\n3. **Don't reinvent existing patterns.** The SDK's `retry.rs` covers HTTP retry + circuit breaker. Existing plugins cover batching, idempotency, header encoding, secret handling.\n4. **Don't refactor unrelated code in the same PR.** Match conventions. propose convention changes separately.\n\n## Connector-wide rules\n\n### Hot loops\n\n- **Zero clone.** Use `&self`, `&str`, `&[T]`, `&Payload`. For `Payload::Json`, `try_to_bytes(&self)` serializes single-pass without cloning the `OwnedValue` tree.\n- **Move-out without cloning** via `std::mem::replace` / `std::mem::take`.\n- **`Vec::with_capacity(n)`** when building per-batch buffers. reallocation in hot path is wasted work.\n\n### Async + concurrency\n\n- **`async fn` only.** Never `.block_on()` inside a `Sink`/`Source` impl - the runtime drives the executor.\n- **Hold locks briefly.** Pattern: lock, clone/read what's needed, drop guard, then do I/O. Never hold a `Mutex` across an external `.await`.\n- **Atomic counters** (`AtomicU64`) preferred over `Mutex` for hot-path metrics.\n- **No `tokio::spawn` from plugin code.** Runtime owns lifecycle. orphans survive `close()`.\n\n### Logging\n\n- `tracing` only. `info!` for lifecycle, `debug!` per-batch detail, `warn!` recoverable, `error!` failures. Include `connector ID: {self.id}` and the connector name on every line.\n- **`verbose: bool`** on `SinkConfig`/`SourceConfig` (TOML top-level). Runtime upgrades its own per-batch `debug!` to `info!` when set. Plugins should mirror with a `verbose_logging: Option` field inside `plugin_config` (convention: `postgres_sink::PostgresSinkConfig::verbose_logging`). Default both to `false`.\n- **Never log secrets** (connection strings, API keys, tokens). Redact URLs at the log site. Credential fields must use `SecretString` (see below).\n\n### Secrets\n\nAny credential-bearing field (connection strings, API keys, bearer tokens, AWS keys) must be `SecretString` from the `secrecy` crate. Plain `String` for a credential is a review-blocker: `SecretString` redacts on `Debug`, so it is what keeps a credential out of a log line that formats the whole config.\n\n**`serde_secret::serialize_secret` EXPOSES the secret. It does not redact.** It calls `expose_secret()` and writes the plaintext. `SecretString` deliberately has no `Serialize` impl, and that absence is the protection - so adding `serialize_with` is what *unblocks* the derive and turns a compile-time guarantee into plaintext output. Use it only where the plaintext is the point: a wire payload, a persisted config, an API response that exposes credentials by design.\n\nSo the default for a plugin config struct is **derive `Deserialize`, but not `Serialize`**. `Deserialize` is required: the SDK glue deserializes the config into the plugin's own struct (`sdk/src/{sink,source}.rs` call `serde_json::from_str::` under a `DeserializeOwned` bound).\n\nWhat never happens is the return trip. The runtime holds plugin configuration as a `serde_json::Value` - parsed from TOML, posted as JSON to the control API, or injected by env var - and hands that across the FFI, so nothing re-serializes the plugin's struct. Leaving `Serialize` off makes that compiler-enforced instead of convention-enforced (`sources/http_source/src/lib.rs::HttpSourceConfig` does this, and comments the omission so nobody adds it back).\n\nPattern:\n\n```rust\nuse secrecy::{ExposeSecret, SecretString};\n\n// `Deserialize` only. Nothing re-serializes a plugin config, and leaving\n// `Serialize` off is what makes the credential unserializable rather than\n// merely un-serialized.\n#[derive(Debug, Clone, Deserialize)]\npub struct MyConfig {\n pub connection_string: SecretString,\n}\n\n// At call site:\nlet pool = PgPoolOptions::new()\n .connect(self.config.connection_string.expose_secret())\n .await?;\n```\n\nIf a config struct genuinely needs `Serialize`, `serde_secret::serialize_redacted` (and `serialize_optional_redacted`) write `[REDACTED]` in place of the value. Reach for `serialize_secret` only when the caller must get the real thing back. The sinks and sources listed below predate that helper and use the exposing one; the annotation is inert today, but it is not the protection it looks like.\n\nNote that none of this protects the credential from the runtime's own control API, which returns plugin configuration verbatim - see #3802. Plugin-side annotations are inert there because the runtime never routes through them.\n\nPlugin-side uses of the exposing helpers: `sinks/{postgres,mongodb,elasticsearch,influxdb,s3,surrealdb}_sink`, `sources/{postgres,elasticsearch,influxdb}_source`.\n\nThat list is plugin-side only, not an inventory of every caller in the tree, and the others are not all mistakes: `runtime/src/api/config.rs` puts `serialize_secret` on `HttpConfig::api_key` (inert for the same reason), and several `core/common` wire-payload types (login, create-user, change-password, PAT) use these helpers by design, because there the credential *is* the payload.\n\n### Errors\n\n- **Use `Error::PermanentHttpError` / `Error::SchemaMismatch` / `Error::CatalogCommitError` for non-retryable failures.** Returning a transient variant for bad data trips circuit breakers and hammers the backend.\n- **No `unwrap()` / `expect()`** on Results from external I/O outside tests.\n\n### Forward-compat config\n\nNew TOML fields use `#[serde(default)]` or `Option`. Adding a field must not break existing configs.\n\n## Benchmark mode (per-batch timing observability)\n\nEvery sink and source supports an opt-in `benchmark: bool` flag (`SinkConfig`/`SourceConfig`, also env `IGGY_CONNECTORS___BENCHMARK=true`). Two independent surfaces:\n\n1. **Tracing events** (only when flag is on) - one structured `info!` per batch under target `iggy_connectors::benchmark` with per-stage microsecond timings:\n - Sink: `connector_type, connector_key, stream, topic, partition_id, current_offset, batch_size, processed_count, decode_us, prepare_us, ffi_us, total_us`\n - Source: `connector_type, connector_key, stream, topic, batch_size, sent_count, decode_us, prepare_us, iggy_send_us, state_saved, state_save_us, total_us` (`state_save_us` is 0 when `state_saved` is false - emitted as flat number + bool, not an Option, so the JSON layer keeps it numeric)\n - Filter live: `RUST_LOG=iggy_connectors::benchmark=info`\n - Also emitted on startup: a one-shot `\"Benchmark mode enabled for ...\"` info log per connector.\n\n2. **Prometheus histograms** (always on, regardless of flag):\n - Metric: `iggy_connector_stage_duration_seconds{connector_key, connector_type, stage}` (both labels snake_case)\n - `stage` (snake_case): `decode`, `prepare`, `total` on both sides; `ffi` (sink only); `iggy_send`, `state_save` (source only). `decode` and `prepare` mean the same thing on both sides - decode is schema decoding, prepare is transform + encode/serialize.\n - Buckets: see `runtime/src/metrics.rs::STAGE_BUCKETS_SECONDS`\n - Example: `histogram_quantile(0.95, sum(rate(iggy_connector_stage_duration_seconds_bucket{stage=\"ffi\"}[5m])) by (le, connector_key))`\n\nThe flag only gates the verbose text event. the histogram observation is always on. Per-stage call path: `Instant::now()` + SipHash over `StageLabels` + parking_lot `RwLock::read` + `HashMap` probe + linear scan inside `Histogram::observe`. Benchmark on target hardware if you suspect the overhead matters at your batch rate.\n\n`iggy_connector_messages_filtered_total{connector_key, connector_type}` counts intentional drops by transforms returning `Ok(None)`, distinct from `errors_total` for unexpected drops. Both also exposed in `/stats` JSON (`ConnectorStats.messages_filtered`, `.errors`).\n\nImplementation: `runtime/src/benchmark.rs` (text-event emitters +\n`as_micros`), `runtime/src/metrics.rs::{Stage, SinkLabels, SourceLabels,\nobserve_stage_with_labels}` (pre-built label cache avoids per-batch\n`String` clones. `&str`-based wrappers are `#[cfg(test)]` only). Tests:\n`runtime/src/metrics.rs::tests::given_stage_histogram_*` (unit),\n`core/integration/tests/connectors/runtime/benchmark.rs` (integration:\ntext emission, env override, disabled-flag, histograms via `/metrics`,\nJSON log format, parser unit tests).\n\n## Logging format\n\n`runtime/src/configs/runtime.rs::LoggingConfig` exposes `format = \"text\" | \"json\"` (text default), env `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `runtime/src/log.rs::init_logging` matches on `(telemetry.enabled, format)` to install the right `fmt::layer()`/`fmt::layer().json()` combined with the OpenTelemetry layer when telemetry is on. JSON applies to the stdout layer only - OTel export pipeline unaffected by this flag.\n\n## Exemplars\n\n| Purpose | Plugin |\n| --------------------------------------------------------- | --------------------------------------------------------------------------------------- |\n| Simplest sink (read first) | `sinks/stdout_sink/` |\n| Real-infra sink + integration tests | `sinks/postgres_sink/` + `integration/tests/connectors/postgres/postgres_sink.rs` |\n| Feature-rich sink config (validation, batch modes, retry) | `sinks/http_sink/` |\n| Atomic counters on hot path | `sinks/mongodb_sink/` |\n| Simplest source (4 canonical state tests) | `sources/random_source/` |\n| Real-infra source | `sources/postgres_source/` + `integration/tests/connectors/postgres/postgres_source.rs` |\n\nRead the relevant exemplar end-to-end before writing or modifying a connector.\n\n## Concrete efficiency patterns\n\nEach implemented in at least one in-tree plugin or runtime path.\n\n| Pattern | Where | When |\n| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- |\n| `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))` | `sinks/http_sink/src/lib.rs` (`send_individual`, `send_ndjson`, `send_json_array`) | Take payload from `&mut ConsumedMessage` no clone |\n| `std::mem::take(&mut batch)` | `runtime/src/sink.rs::consume_messages`, `manager/{sink,source}.rs` | Drain a `Vec` field with one move |\n| `std::mem::swap` | `sinks/http_sink` (retry_delay / max_retry_delay validation) | Fix config-field ordering in `new()` |\n| `Vec::with_capacity(messages.len())` | Every batching sink | Pre-size per-batch buffers |\n| `bytes::Bytes::from(body)` | `sinks/{http,influxdb}_sink` | Cheap ref-counted body sharing across retries |\n| `AtomicU64` counters | `sinks/mongodb_sink` | Lock-free per-message metric updates |\n| `Payload::try_to_bytes(&self)` | `sdk/src/lib.rs::Payload::try_to_bytes` | Serialize `Payload::Json` no-clone |\n| `simd_json::to_owned_value(&mut bytes)` | `sdk/src/decoders/json.rs::JsonStreamDecoder::decode` | In-place JSON parse |\n| Brief-lock fetch | `sources/postgres_source/src/lib.rs::poll_tables` | Lock read -> drop -> I/O -> lock write |\n| `is_transient_error(&e)` SQLSTATE mapping | `sinks/postgres_sink/src/lib.rs` | Driver errors -> retry vs `PermanentHttpError` |\n| Two-constructor pattern (`new` lenient + `try_new` strict + `Default`) | `sdk/src/decoders/avro.rs::AvroStreamDecoder` | Stateful decoders/encoders where schema can fail |\n| Duplicate-ID FFI guard | `sdk/src/sink.rs::sink_connector!`, `sdk/src/source.rs::source_connector!` | Prevents silent data loss on reopen-without-close |\n| `restart_guard.try_lock()` | `runtime/src/manager/{sink,source}.rs::restart_connector` | No thundering-herd restarts |\n| `tokio::time::timeout(..., handle).await` + `handle.abort()` on timeout | `runtime/src/manager/source.rs::stop_connector` | Bounded shutdown + leak prevention |\n| `flume::unbounded()` channel | `runtime/src/source.rs::spawn_source_handler` / `source_forwarding_loop` | MPSC handoff from SDK async task to runtime loop |\n| `tokio::sync::watch::channel(())` | `sdk/src/{sink,source}.rs`, `runtime/src/sink.rs`, `runtime/src/manager/*` | One-shot shutdown broadcast |\n| `dashmap::DashMap` | `runtime/src/manager/sink.rs`, `source.rs::SOURCE_SENDERS`, SDK `INSTANCES` | Lock-free concurrent keyed access |\n| `secrecy::SecretString` + `iggy_common::serde_secret::serialize_secret` | `sinks/postgres_sink::PostgresSinkConfig::connection_string` | `Debug` redacts; `serialize_secret` EXPOSES |\n\n## Drop accounting\n\n- **Filter drops** (transform returning `Ok(None)`) bump `messages_filtered`.\n- **Every non-filter drop** bumps `errors`. Inside the per-batch loops the count is accumulated in a local and flushed once after the loop via `inc_errors_by_with_labels` (filters likewise via `inc_messages_filtered_with_labels`) - one `Family` lookup per batch, not per message. One-shot drops outside a loop call `inc_errors_with_labels` directly. Hot-path calls always use the pre-built `SinkLabels`/`SourceLabels` cache. `&str`-based wrappers on `Metrics` are `#[cfg(test)]`-only.\n\n## Common review smells\n\n- `.clone()` on `Payload::Json` or `OwnedValue` - use `try_to_bytes(&self)`.\n- `&mut self` on `Sink::consume` or `Source::poll` impls - won't compile, flag any creative workaround.\n- `std::sync::Mutex` held across `.await` - swap for `tokio::sync::Mutex`.\n- Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` in plugin `Cargo.toml`.\n- Source plugin without the four canonical state tests (see `connector-testing`).\n- New silent message drop without a metric increment.\n- Wrapping `format!()` around args passed to `error!`/`warn!`/`info!`/`debug!` - eager `format!` allocates even when level filters the line out. Pass args directly: `error!(\"foo: {x}\")` or `error!(error = %x, \"foo\")`.\n- Logging a connection string, API key, or token.\n- Plain `String` for a credential field - use `SecretString`.\n- `tokio::spawn` inside plugin code - runtime owns lifecycle.\n- `std::time::SystemTime::now()` in transforms - non-deterministic, breaks tests.\n\n## File map\n\n```text\ncore/connectors/\n├── README.md\n├── runtime/\n│ ├── src/\n│ │ ├── main.rs FFI structs (SinkApi, SourceApi), plugin path resolution\n│ │ ├── manager/{sink,source}.rs Lifecycle, restart_guard, status transitions\n│ │ ├── configs/ Local + HTTP config providers, ConfigEnv derive\n│ │ ├── {sink,source,stream,transform,state,context}.rs\n│ │ ├── benchmark.rs, log.rs, metrics.rs\n│ │ └── api/ HTTP control plane (/sinks, /sources, /stats, /metrics)\n│ └── example_config/ Reference TOML\n├── sdk/\n│ └── src/\n│ ├── lib.rs Sink/Source traits, Payload, Schema, Error\n│ ├── {sink,source}.rs FFI containers + sink_connector!/source_connector! macros\n│ ├── decoders/encoders/ Per-schema (json/raw/text/proto/flatbuffer/avro)\n│ ├── transforms/ add/delete/update/filter fields, format conversions\n│ └── retry.rs CircuitBreaker, HttpRetryMiddleware\n├── sinks/_sink/\n└── sources/_source/\n\ncore/integration/tests/connectors/ # Real-infra integration tests\n├── fixtures/ # testcontainers-modules wrappers per backend\n├── postgres/, elasticsearch/, mongodb/, iceberg/, influxdb/, quickwit/, http/, delta/\n├── runtime/ # error_isolation, benchmark\n└── api/ # HTTP API endpoint tests\n```\n\n## Cite by symbol, not line number\n\nSkills cite files by path + symbol (`PostgresContainer::start` in `core/integration/tests/connectors/fixtures/postgres/container.rs`). Line numbers go stale on the first refactor. symbol names survive. Grep the symbol if you need the exact spot.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-testing/SKILL.md":"---\nname: connector-testing\ndescription: Write tests for Apache Iggy connectors - sinks, sources, transforms, SDK, runtime. Covers BDD unit-test naming, the canonical source state round-trip tests, and the real-infra integration test layout under `core/integration/tests/connectors/` with `testcontainers-modules` + `#[iggy_harness]`. Load when writing or reviewing tests in `core/connectors/` or `core/integration/tests/connectors/`. Use for connector test authoring. NOT for runtime internals or non-connector test patterns.\n---\n\n# Testing Apache Iggy Connectors\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). Repo-wide testing rules (`#[iggy_harness]`, BDD naming, harness layout) live in [AGENTS.md](../../../AGENTS.md#testing). This skill covers connector-specific test conventions.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Unit tests](#unit-tests)\n- [Integration tests (real infra via Docker)](#integration-tests-real-infra-via-docker)\n- [SDK + runtime tests](#sdk--runtime-tests)\n- [CI / verification flow](#ci--verification-flow)\n- [Common pitfalls](#common-pitfalls)\n- [Examples worth copying](#examples-worth-copying)\n\n## STOP and ask the user before\n\n- Mocking the database or backend in an integration test - we hit real infra via `testcontainers-modules` (mocks miss migration bugs).\n- Adding a unit test that requires Docker - that belongs under `core/integration/tests/connectors//`, not in `/src/lib.rs`.\n- Skipping the four canonical source state tests - they're mandatory for every source plugin.\n- Naming a test `test_foo` / `does_bar` - we use BDD `given_X_when_Y_should_Z`.\n\nThere are **two distinct test layers**. Use both - they catch different bugs.\n\n| Layer | Location | What it covers | When required |\n| --------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------- |\n| **Unit tests** | `#[cfg(test)] mod tests` at EOF of `/src/lib.rs` | Config parsing, defaults, fallbacks, payload conversion, query/document building, state ser/de | Every plugin |\n| **Integration tests** | `core/integration/tests/connectors//` (uses real backend via Docker) | End-to-end: runtime + plugin + real Postgres/ES/Iceberg/Mongo/etc. + Iggy server | Every plugin that integrates with an external system |\n\n## Unit tests\n\n### Naming - BDD `given_when_should`\n\nMatch the convention used in the codebase (`sources/random_source/src/lib.rs::tests`, `sinks/postgres_sink/src/lib.rs::tests`):\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_messages_produced() { ... }\n\n#[test]\nfn given_no_state_should_start_fresh() { ... }\n\n#[test]\nfn given_invalid_duration_should_fall_back_to_default() { ... }\n\n#[test]\nfn given_payload_json_should_serialize_correctly() { ... }\n```\n\nYou may **omit one part** when the test is small (`given_X_should_Y` instead of full `given_X_when_Y_should_Z`), but **stay consistent within a file**. Imperative `test_foo` / `does_bar` names are not the convention here.\n\nA test whose subject is a **type** rather than a scenario has no \"given\" to name, and drops the prefix: it reads `_should_`. The canonical state round trip below is `state_should_be_serializable_and_deserializable` in all three shipped sources (`random_source`, `postgres_source`, `http_source`). Leave those as they are rather than bending them into the scenario form.\n\n### Prove a test can fail, before trusting it\n\nA passing test says nothing until you have watched it fail. Break the behaviour it names, run the\nsuite, and put the code back:\n\n```bash\n# commit first: this reverts with `git checkout --`, which discards\n# every uncommitted change in the file, not only the mutation\ngit status --porcelain -- # must be empty\n\ncargo nextest run -p # expect a failure naming that test\ngit checkout -- \n```\n\nTwo failures this catches, both seen in review of `http_source`:\n\n- **A fixture that makes the interesting case unreachable.** A guard test for endpoint resurrection\n passed `&[]` as the static config, so there was nothing to resurrect and the assertion held\n whatever `restore` did.\n- **An assertion aimed at something self-healing.** A test for a teardown guard asserted on gauges,\n which `Metrics::encode` rebuilds from the live instances on every scrape, so it passed with the\n guard deleted. Delete such a test rather than keep it; a test that cannot fail is worse than none,\n because it reads as coverage.\n\nWatch for an arm hidden behind a catch-all: `StatusClass::from` ends in `_ => ServerError`, so the\nredirect arm could be removed with every existing assertion still green.\n\n### `test_config()` helper\n\nEvery plugin's tests start with a small helper returning a tuned-down version of the production config. New tests extend it. don't construct from scratch each time.\n\n```rust\nfn test_config() -> RandomSourceConfig {\n RandomSourceConfig {\n interval: Some(\"100ms\".to_string()),\n max_count: Some(100),\n messages_range: Some((5, 10)),\n payload_size: Some(50),\n }\n}\n```\n\n### Async testing pattern\n\nThe trait methods are `async`. For unit tests, build a runtime locally - don't add `#[tokio::test]` in `src/lib.rs` (it pulls test macros into prod build):\n\n```rust\n#[test]\nfn given_X_should_Y() {\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n // await calls here\n });\n}\n```\n\n### Mandatory: four canonical source state tests\n\nEvery source must have these. Copy from `sources/random_source/src/lib.rs::tests`:\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_() {\n let state = State { /* fields */ };\n let serialized = rmp_serde::to_vec(&state).expect(\"Failed to serialize state\");\n let connector_state = ConnectorState(serialized);\n\n let source = MySource::new(1, test_config(), Some(connector_state));\n\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n let restored = source.state.lock().await;\n assert_eq!(restored., /* expected */);\n });\n}\n\n#[test]\nfn given_no_state_should_start_fresh() {\n let source = MySource::new(1, test_config(), None);\n /* assert default state */\n}\n\n#[test]\nfn given_invalid_state_should_start_fresh() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n let source = MySource::new(1, test_config(), Some(invalid));\n /* assert default state */\n}\n\n#[test]\nfn state_should_be_serializable_and_deserializable() {\n let original = State { /* fields */ };\n let bytes = rmp_serde::to_vec(&original).unwrap();\n let restored: State = rmp_serde::from_slice(&bytes).unwrap();\n assert_eq!(original., restored.);\n}\n```\n\nState is the only thing that survives a plugin restart. Silent corruption here means lost data on the next deploy.\n\n**The third test has two correct shapes, and which one applies is a design decision rather than a style choice.**\n\nStarting fresh is right when the state is a cursor. The worst case is re-reading from the beginning, which the at-least-once contract already permits, so discarding an undecodable cursor costs duplicates and nothing else.\n\nIt is wrong when the state carries something whose *absence* is a downgrade rather than a repeat. `http_source` keeps its revocation tombstones in state, so starting fresh after a decode failure would serve every revoked endpoint again, with its secret. Its `restore` returns `Err` instead and the connector refuses to open. A source in that position asserts the refusal:\n\n```rust\n#[test]\nfn given_invalid_state_should_refuse_to_open() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n\n let restored = MyState::restore(&config_entries(), Some(invalid), 1);\n\n assert!(\n restored.is_err(),\n \"discarding this state silently would re-serve what it was revoking\"\n );\n}\n```\n\nName the test for whichever it is and say why in a comment. What is never right is starting fresh silently when the state that was discarded was load-bearing.\n\n### Sink unit tests\n\nCover pure per-message logic:\n\n- Config defaults + optional fields + invalid values → fallback.\n- Payload conversion for each `Payload` variant the sink claims to handle.\n- Header encoding (especially base64 for binary headers).\n- Query/document building (e.g., `postgres_sink::build_batch_insert_query`).\n- Transient vs permanent error classification (e.g., `postgres_sink::is_transient_error`).\n\n### Transform tests\n\nFour branches must be covered (see `connector-transform`):\n\n1. Happy path: matching input → expected output.\n2. Filter path: input that should be dropped returns `Ok(None)`.\n3. Pass-through path: non-JSON payload (or unmatched condition) returns `Ok(Some(unchanged))`.\n4. Invalid config: `new()` returns `Err(Error::InvalidConfigValue(_))`.\n\n### What NOT to mock\n\n- The underlying client lib (sqlx, reqwest, mongodb). Tests would exercise the mock, not the code.\n- The Iggy server.\n- The runtime.\n\nIf you need a fake backend, use `wiremock` (a real local HTTP server) - see `core/integration/tests/connectors/http/`.\n\n## Integration tests (real infra via Docker)\n\nThe canonical pattern lives in `core/integration/tests/connectors/`. Plugins that integrate with external services (Postgres, Elasticsearch, Iceberg, MongoDB, InfluxDB, Quickwit, HTTP/wiremock) have a paired fixture + test file there.\n\n### Layout\n\n```text\ncore/integration/tests/connectors/\n├── mod.rs Common helpers (TestMessage, create_test_messages)\n├── fixtures/ One subdir per backend\n│ ├── postgres/\n│ │ ├── container.rs PostgresContainer (testcontainers wrapper), env-var consts\n│ │ ├── sink.rs PostgresSinkFixture, PostgresSinkByteaFixture, PostgresSinkJsonFixture\n│ │ ├── source.rs PostgresSourceJsonFixture, PostgresSourceJsonbFixture, ...\n│ │ └── mod.rs re-exports\n│ ├── elasticsearch/ iceberg/ mongodb/ influxdb/ quickwit/ http/ delta/\n│ └── wiremock.rs HTTP mocking for http_sink tests\n├── postgres/\n│ ├── postgres_sink.rs The actual #[iggy_harness] tests\n│ ├── postgres_source.rs\n│ ├── restart.rs Restart + state-survival tests\n│ ├── sink.toml Runtime config used by sink tests\n│ ├── source.toml Runtime config used by source tests\n│ └── mod.rs Shared constants (TEST_MESSAGE_COUNT, POLL_ATTEMPTS, ...)\n├── delta/ elasticsearch/ iceberg/ mongodb/ influxdb/ random/ quickwit/ http/ stdout/\n├── http_config_provider/ HTTP config provider end-to-end\n├── runtime/ Runtime-only tests (error isolation, missing plugin, ...)\n└── api/ HTTP control plane tests\n```\n\n### Container naming + cleanup\n\nEvery test container MUST be named with the `iggy-test-` prefix so a single\n`docker ps -aqf 'name=^iggy-test-'` sweep reaps them (`just clean-test-containers`,\nalias `ctc`; the `test`/`nextest` recipes also reap on exit). New fixtures get\nthis for free by naming through `fixtures::unique_container_name(\"\")`, which\nappends a uuid for parallel-safe ephemeral containers. Reuse fixtures use a fixed\n`iggy-test-` literal instead, because the stable name is what lets later\nnextest processes attach to the same container via `ReuseDirective::Always`. Do\nnot hand-roll a different prefix - the sweep only finds `iggy-test-*`.\n\nAs of now only `elasticsearch` and `doris` share a container. Both have a slow\nboot (JVM / Doris FE+BE, tens of seconds), so reusing one across the whole\nregression beats spinning a fresh one per test. Every other fixture boots a\nfresh, ephemeral container per test - cheap enough that the isolation is worth\nmore than the reuse.\n\n### The `#[iggy_harness]` proc macro\n\nEach integration test is annotated with this macro from the `integration` crate. It boots an in-process Iggy server (and optionally the connectors runtime) for the duration of the test, runs the seeds, then injects a `&TestHarness` and your fixture.\n\n```rust\nuse integration::harness::seeds;\nuse integration::iggy_harness;\n\n#[iggy_harness(\n server(connectors_runtime(config_path = \"tests/connectors/postgres/sink.toml\")),\n seed = seeds::connector_stream\n)]\nasync fn json_messages_sink_stores_as_bytea(\n harness: &TestHarness,\n fixture: PostgresSinkFixture,\n) {\n let client = harness.root_client().await.unwrap();\n let pool = fixture.create_pool().await.expect(\"Failed to create pool\");\n fixture.wait_for_table(&pool, \"iggy_messages\").await;\n // ... send messages, assert on rows\n}\n```\n\nKey pieces:\n\n- `server(connectors_runtime(config_path = ...))` boots the connectors runtime against the given TOML.\n- `seed = seeds::connector_stream` runs the named seed to create the test stream/topic.\n- `fixture: PostgresSinkFixture` - the test harness calls `PostgresSinkFixture::setup()` (which spins up the testcontainer) and injects the result.\n- `harness: &TestHarness` (defined in `core/integration/src/harness/orchestrator/harness.rs`) exposes `.root_client().await`, `.connectors_runtime().expect(\"...\").http_url()`, etc. - `connectors_runtime()` returns `Option`, so `.expect()` first.\n\n### Fixture pattern (`TestFixture` trait)\n\nEach fixture implements `integration::harness::TestFixture`. Pattern:\n\n```rust\n// core/integration/tests/connectors/fixtures//container.rs\n\nuse testcontainers_modules::{\n ,\n testcontainers::{ContainerAsync, runners::AsyncRunner},\n};\n\npub struct MyContainer {\n container: ContainerAsync<::Image>,\n pub connection_string: String,\n}\n\nimpl MyContainer {\n pub async fn start() -> Result {\n let container = ::Image::default().start().await?;\n let port = container.get_host_port_ipv4(DEFAULT_PORT).await?;\n Ok(Self {\n container,\n connection_string: format!(\"...://localhost:{port}\"),\n })\n }\n}\n```\n\nThe fixture itself wraps the container and implements `TestFixture::setup()` to spin it up and inject env vars that the runtime picks up.\n\n### Env-var injection via `ConfigEnv`\n\nThe runtime's config structs derive `ConfigEnv` (`configs_derive::ConfigEnv`, applied in `runtime/src/configs/connectors.rs` to `SinkConfig`, `SourceConfig`, `StreamConsumerConfig`, `StreamProducerConfig`). Fields are addressable by env var with a path-built prefix. Three concrete forms observed in `core/integration/tests/connectors/fixtures/postgres/container.rs`:\n\n```rust\n// Plugin-config fields (your sink/source's own config struct):\n// IGGY_CONNECTORS___PLUGIN_CONFIG_\npub const ENV_SINK_CONNECTION_STRING: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PLUGIN_CONFIG_CONNECTION_STRING\";\n\n// Indexed nested fields (per-stream entries are a Vec):\n// IGGY_CONNECTORS___STREAMS__\npub const ENV_SINK_STREAMS_0_STREAM: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_STREAMS_0_STREAM\";\n\n// Top-level scalar fields on SinkConfig/SourceConfig itself:\n// IGGY_CONNECTORS___\npub const ENV_SINK_PATH: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PATH\";\n```\n\n`` is `SINK` or `SOURCE`. `` is the connector key (TOML key field, uppercased). `PLUGIN_CONFIG` is the literal segment used when reaching into your plugin's nested config. Fields marked `#[config_env(skip)]` (e.g., `transforms`, `plugin_config` as a whole) are NOT env-addressable. primitive leaves marked `#[config_env(leaf)]` are.\n\nThe fixture sets these in `setup()` before the runtime reads its config. This is how a dynamic container port reaches a static TOML.\n\n### Test naming for integration tests\n\nIntegration tests use a **descriptive declarative** style, not the unit-test `given_should` form. Match the codebase:\n\n```rust\nasync fn json_messages_sink_stores_as_bytea(...) { ... }\nasync fn binary_messages_sink_stores_as_bytea(...) { ... }\nasync fn json_messages_sink_stores_as_jsonb(...) { ... }\nasync fn delete_after_read_source_removes_rows_after_producing(...) { ... }\nasync fn processed_column_source_marks_rows_after_producing(...) { ... }\nasync fn state_persists_across_connector_restart(...) { ... }\nasync fn restart_sink_connector_continues_processing(...) { ... }\n```\n\nPattern: `__`. Reads like a sentence describing what the test proves.\n\n### Polling vs sleeping\n\nIntegration tests poll until an expected condition - never `sleep(big_duration)`. Constants `POLL_ATTEMPTS` and `POLL_INTERVAL_MS` defined per-backend `mod.rs`. Fixture helpers like `PostgresSinkFixture::fetch_rows_as` and `PostgresSinkFixture::wait_for_table` encapsulate the pattern.\n\n### When to add an integration test\n\n- New sink/source plugin → at minimum one happy-path test per supported `Schema` and per supported plugin mode. Concrete coverage in `core/integration/tests/connectors/postgres/`: 3 sink tests (`json_messages_sink_stores_as_bytea`, `binary_messages_sink_stores_as_bytea`, `json_messages_sink_stores_as_jsonb`) backed by 3 sink fixtures, plus 5 source fixtures (`Json`, `Jsonb`, `Bytea`, `Delete`, `Mark`) and a dedicated `restart.rs` for state-survival.\n- Bug fix in a plugin → regression test reproducing the bug.\n- Behavior that crosses the FFI boundary (state restart, transform chain, schema decode) → integration test, not unit test.\n\n### When NOT to add an integration test\n\n- Pure logic (query builder, payload converter) - that's a unit test.\n- Config validation - unit test.\n- Code paths already covered by an equivalent test for another backend, when the new backend has identical behavior.\n\n## SDK + runtime tests\n\n- `core/connectors/sdk/tests/` - cross-module SDK concerns. In tree: `protobuf_integration.rs`, `flatbuffer_integration.rs` (round-trip schema decoders/encoders).\n- Runtime tests live under `core/integration/tests/connectors/runtime/` and `core/integration/tests/connectors/api/`. They exercise the runtime alone (no plugin-specific backend) - error isolation, missing plugin, invalid config, HTTP API contract.\n\n## CI / verification flow\n\nBefore pushing:\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p --all-targets -- -D warnings\ncargo test -p \n```\n\nFor integration tests, the relevant package is `integration`. There is **no `--test connectors` target** - the `integration` crate has no `[[test]]` entries and `tests/` is structured as a single binary entry through `tests/mod.rs` with `connectors/`, `server/`, `cli/`, etc. as nested modules. Run via name filter:\n\n```text\ncargo test -p integration -- connectors::postgres::postgres_sink\ncargo test -p integration -- connectors::postgres:: # all postgres tests\ncargo test -p integration -- connectors:: # all connector tests\n```\n\nIntegration tests assume Docker is available locally and pull images from a public registry on first run.\n\n## Common pitfalls\n\n1. **Skipping integration tests for a real-infra plugin.** Unit tests don't catch FFI/lifecycle/encoding regressions.\n2. **Forgetting the four canonical source state tests.** This is the most common review comment.\n3. **Mocking the client library** instead of using `testcontainers` - test exercises the mock.\n4. **`#[tokio::test]` in `src/lib.rs`** - pulls tokio test macros into prod build. Use `Runtime::new()` instead. (`#[tokio::test]` is fine in `tests/` directories, which compile separately.)\n5. **`unwrap()` in test assertions** - prefer `assert_eq!` / `expect(\"descriptive\")` so failures tell you which step blew up.\n6. **Tests depending on system time** - inject the clock or use `tokio::time::pause()`.\n7. **Hidden coupling via shared globals** (static `AtomicU64`, env vars) - tests must be independent and parallelizable.\n8. **Asserting on `Display` of an error** - use `matches!(err, Error::Variant(_))`. `assert_eq!(err.to_string(), ...)` breaks on message tweaks.\n9. **Sleeping a fixed duration** instead of polling - causes flakes when CI is slow.\n\n## Examples worth copying\n\n| Concern | Where |\n| ----------------------------------- | -------------------------------------------------------------------- |\n| Source state round-trip | `sources/random_source/src/lib.rs::tests` |\n| Config + query-building unit tests | `sinks/postgres_sink/src/lib.rs::tests` |\n| Real-infra sink integration tests | `core/integration/tests/connectors/postgres/postgres_sink.rs` |\n| Real-infra source integration tests | `core/integration/tests/connectors/postgres/postgres_source.rs` |\n| Restart + state survival | `core/integration/tests/connectors/postgres/restart.rs` |\n| Wiremock-driven HTTP sink test | `core/integration/tests/connectors/http/http_sink.rs` |\n| Testcontainer wrapper | `core/integration/tests/connectors/fixtures/postgres/container.rs` |\n| Fixture impl `TestFixture` | `core/integration/tests/connectors/fixtures/postgres/sink.rs` |\n| Runtime-only error-isolation | `core/integration/tests/connectors/runtime/error_isolation.rs` |\n| Transform tests | `sdk/src/transforms/add_fields.rs::tests`, `filter_fields.rs::tests` |\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-source/TEMPLATE.md":"# Source plugin skeleton\n\nBoilerplate for a new `core/connectors/sources/_source/`. Adapt\nthe `MySource` / `Client` / row types to the backend driver.\n\n`Cargo.toml` is identical to a sink's (see\n[connector-sink/TEMPLATE.md](../connector-sink/TEMPLATE.md)) - only\nthe crate name suffix changes (`iggy_connector__source`) and the\nupstream client dep.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, `impl Source` after,\nhelpers below.\n\n```rust\n/* Apache 2.0 header */\n\nuse std::str::FromStr;\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,\n source::SourceBatchResult, source_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse tokio::sync::Mutex;\nuse tokio::time::sleep;\nuse tracing::{debug, error, info, warn};\n\nsource_connector!(MySource);\n\nconst CONNECTOR_NAME: &str = \"My source\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySourceConfig {\n pub endpoint: String,\n pub poll_interval: Option, // humantime, \"10s\" default\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option,\n pub verbose_logging: Option,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct State {\n cursor: Option, // WAL LSN, scroll id, timestamp, ...\n last_offset: u64,\n messages_produced: u64,\n}\n\n#[derive(Debug)]\npub struct MySource {\n id: u32,\n config: MySourceConfig,\n poll_interval: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n pending: Mutex>,\n}\n\nimpl MySource {\n pub fn new(id: u32, config: MySourceConfig, state: Option) -> Self {\n let raw_interval = config.poll_interval.clone().unwrap_or_else(|| \"10s\".into());\n let poll_interval = humantime::Duration::from_str(&raw_interval)\n .map(|d| *d)\n .unwrap_or_else(|_| {\n warn!(\"Invalid poll_interval for {CONNECTOR_NAME} ID: {id}, defaulting to 10s\");\n Duration::from_secs(10)\n });\n\n let verbose = config.verbose_logging.unwrap_or(false);\n\n let restored = state\n .and_then(|s| s.deserialize::(CONNECTOR_NAME, id))\n .inspect(|s| info!(\n \"Restored state for {CONNECTOR_NAME} ID: {id}, last_offset: {}, cursor: {:?}\",\n s.last_offset, s.cursor\n ));\n\n Self {\n id,\n config,\n poll_interval,\n verbose,\n client: None,\n state: Mutex::new(restored.unwrap_or(State {\n cursor: None,\n last_offset: 0,\n messages_produced: 0,\n })),\n pending: Mutex::new(None),\n }\n }\n}\n\n#[async_trait]\nimpl Source for MySource {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn poll(&self) -> Result {\n sleep(self.poll_interval).await; // sleep first - backpressure\n\n let mut candidate = self.state.lock().await.clone();\n\n let fetched = self.fetch_since(candidate.cursor.as_deref()).await?;\n\n let mut messages = Vec::with_capacity(fetched.len());\n let mut next_cursor = None;\n for row in fetched {\n let payload = simd_json::to_vec(&row).map_err(|e|\n Error::Serialization(format!(\"row serialize: {e}\"))\n )?;\n messages.push(ProducedMessage {\n id: Some(row.id as u128),\n checksum: None,\n timestamp: None,\n origin_timestamp: Some(row.created_at_ns),\n headers: None,\n payload,\n });\n next_cursor = Some(row.cursor_value);\n }\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} produced {} messages, next cursor: {:?}\",\n self.id, messages.len(), next_cursor\n );\n }\n\n if messages.is_empty() {\n return Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: None,\n });\n }\n\n candidate.messages_produced += messages.len() as u64;\n candidate.cursor = next_cursor;\n let persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n *self.pending.lock().await = Some(candidate);\n\n Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: Some(persisted),\n })\n }\n\n async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {\n let (SourceBatchResult::Ack, Some(candidate)) =\n (result, self.pending.lock().await.take())\n else {\n return Ok(());\n };\n *self.state.lock().await = candidate;\n Ok(())\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx pools\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, total produced: {}\",\n self.id, state.messages_produced\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySourceConfig) -> Result { /* ... */ }\n```\n",".claude/skills/connector-sink/TEMPLATE.md":"# Sink plugin skeleton\n\nBoilerplate for a new `core/connectors/sinks/_sink/`. Adapt the\n`MySink` / `Client` types to the backend driver you're integrating.\n\n## Cargo.toml\n\n```toml\n# Apache 2.0 header (copy verbatim from any existing sink Cargo.toml)\n[package]\nname = \"iggy_connector__sink\"\nversion = \"0.5.1-edge.1\" # match the version most other sinks use\nedition = \"2024\"\nlicense = \"Apache-2.0\"\npublish = false\n# ...keywords, description, repository, homepage all identical to existing sinks\n\n[package.metadata.cargo-machete]\nignored = [\"dashmap\", \"once_cell\"] # used by sink_connector! macro\n\n[lib]\ncrate-type = [\"cdylib\", \"lib\"] # cdylib = runtime-loadable; lib = unit tests\n\n[dependencies]\nasync-trait = { workspace = true }\ndashmap = { workspace = true }\niggy_connector_sdk = { workspace = true }\nonce_cell = { workspace = true }\nserde = { workspace = true }\ntokio = { workspace = true }\ntracing = { workspace = true }\n# + your client crate (reqwest, sqlx, mongodb, ...)\n```\n\nRun `cargo sort --no-format --workspace` after edits.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, then `impl`, then private\nhelpers.\n\n```rust\n/* Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements... */ // full Apache header\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata, sink_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse std::str::FromStr;\nuse std::time::Duration;\nuse tokio::sync::Mutex;\nuse tracing::{debug, error, info, warn};\n\nsink_connector!(MySink); // generates FFI symbols + version export\n\nconst CONNECTOR_NAME: &str = \"My sink\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySinkConfig {\n pub endpoint: String,\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option, // humantime, e.g. \"500ms\"\n pub verbose_logging: Option, // mirror runtime's `verbose` flag\n // Every runtime-tunable field is Option; defaults applied in new()\n}\n\n#[derive(Debug)]\npub struct MySink {\n id: u32,\n config: MySinkConfig,\n batch_size: usize,\n max_retries: u32,\n retry_delay: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n}\n\n#[derive(Debug)]\nstruct State {\n messages_processed: u64,\n errors: u64,\n}\n\nimpl MySink {\n pub fn new(id: u32, config: MySinkConfig) -> Self {\n let batch_size = config.batch_size.unwrap_or(100) as usize;\n let max_retries = config.max_retries.unwrap_or(3);\n let retry_delay = config\n .retry_delay\n .as_deref()\n .and_then(|raw| humantime::Duration::from_str(raw).ok().map(|d| *d))\n .unwrap_or_else(|| {\n warn!(\"Invalid retry_delay for {CONNECTOR_NAME} ID: {id}, defaulting to 500ms\");\n Duration::from_millis(500)\n });\n let verbose = config.verbose_logging.unwrap_or(false);\n Self {\n id,\n config,\n batch_size,\n max_retries,\n retry_delay,\n verbose,\n client: None,\n state: Mutex::new(State { messages_processed: 0, errors: 0 }),\n }\n }\n}\n\n#[async_trait]\nimpl Sink for MySink {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n client.ping()\n .await\n .map_err(|e| Error::InitError(format!(\"connectivity check failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn consume(\n &self,\n topic_metadata: &TopicMetadata,\n messages_metadata: MessagesMetadata,\n messages: Vec,\n ) -> Result<(), Error> {\n let Some(client) = self.client.as_ref() else {\n return Err(Error::InitError(\"client not initialized\".into()));\n };\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages from stream: {}, topic: {}, offset: {}, current_offset: {}\",\n self.id, messages.len(), topic_metadata.stream, topic_metadata.topic,\n messages_metadata.partition_id, messages_metadata.current_offset\n );\n } else {\n debug!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages\",\n self.id, messages.len()\n );\n }\n\n let mut last_err: Option = None;\n for batch in messages.chunks(self.batch_size) {\n match self.send_batch(client, batch).await {\n Ok(()) => { /* counter */ }\n Err(Error::PermanentHttpError(message)) => {\n error!(\n \"{CONNECTOR_NAME} ID: {} dropping batch (permanent): {message}\",\n self.id\n );\n }\n Err(error) => {\n last_err = Some(error);\n }\n }\n }\n match last_err {\n Some(error) => Err(error),\n None => Ok(()),\n }\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n // sqlx pools have `.close().await`; reqwest/mongodb/elasticsearch just drop.\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, processed: {}, errors: {}\",\n self.id, state.messages_processed, state.errors\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySinkConfig) -> Result { /* ... */ }\n```\n","AGENTS.md":"# Apache Iggy Agent Guidelines\n\n**Apache Iggy** is a persistent message streaming platform\nin Rust. Thread-per-core shared-nothing, `io_uring` + `compio`.\nTransports: QUIC, WebSocket, TCP (custom binary), HTTP (REST). SDKs:\nRust, .NET, Java, Python, Go, C++, Node.js. A connectors subsystem\ningests from / egresses to external systems via dlopened plugins.\n\n> Skills live under `.claude/skills/`, with per-skill links under\n> `.agents/skills/` for Codex. Load\n> [connectors-overview](.claude/skills/connectors-overview/SKILL.md)\n> first for any change under `core/connectors/`. Other subsystems\n> follow the repo-wide principles in this file.\n\n## Contents\n\n- [Apache Iggy Agent Guidelines](#apache-iggy-agent-guidelines)\n - [Contents](#contents)\n - [STOP and ask the user before](#stop-and-ask-the-user-before)\n - [Helping someone contribute](#helping-someone-contribute)\n - [Quick reference](#quick-reference)\n - [Structure](#structure)\n - [Where to look](#where-to-look)\n - [Skills](#skills)\n - [Repo-wide principles](#repo-wide-principles)\n - [Tooling](#tooling)\n - [Testing](#testing)\n - [Pitfalls](#pitfalls)\n - [Local state directories](#local-state-directories)\n - [Commits, PRs, releases](#commits-prs-releases)\n - [Discussion and support](#discussion-and-support)\n\n## STOP and ask the user before\n\n- Writing code for an issue not assigned to the user\n (`gh issue view --json assignees`). If it isn't, or you can't\n confirm it, they must ask on the issue and wait to be assigned.\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI\n signature in `core/connectors/sdk/src/{sink,source}.rs` (breaks\n every pre-built plugin .so).\n- Modifying `core/binary_protocol/` wire-format types tagged\n `#[repr(C)]` (breaks every running client).\n- Touching the state-save path (`core/connectors/runtime/src/state.rs`)\n in a way that changes the atomic-rename protocol (corruption risk).\n- Renaming default consumer groups, plugin path resolution rules, or\n any other operator-facing identifier.\n- Force-pushing to `master`, amending shipped commits, or skipping\n hooks with `--no-verify` / `--no-gpg-sign`.\n\n## Helping someone contribute\n\nMost PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.\n\n- The PR must link an issue the user was assigned to before any work started. See the STOP list above.\n- Fill in every section of the PR template. Under AI Usage, name the agent and say what it wrote. Never leave the section out or write \"None\" when an agent wrote any of the change.\n- Run `prek run` before the PR is opened. If prek isn't installed, ask the user to install it (`cargo install prek`, then `prek install`). Don't substitute the individual checks.\n- If a check can't run, tell the user which one and why, and treat the PR as not ready. Never write text in a PR asking a maintainer to run something.\n- The user should write the rationale in their own words and be able to explain every change without asking you.\n- New contributors keep one open PR at a time.\n- If you found a bug, the first PR holds a failing test. The fix waits until the approach is agreed on the issue.\n\n## Quick reference\n\n```bash\n# Verification order is enforced by CI (.github/actions/rust/pre-merge).\ncargo fmt --all # step 1\ncargo sort --no-format --workspace # step 2 (--no-format mandatory)\ncargo clippy --all-features --all-targets -- -D warnings # step 3\ncargo test -p # step 4\n\n# Other CI checks (mirror in pre-commit; pass --fix where supported)\n./scripts/ci/taplo.sh # TOML\n./scripts/ci/markdownlint.sh # Markdown\n./scripts/ci/shellcheck.sh # Shell\n./scripts/ci/license-headers.sh # Apache headers\n./scripts/ci/trailing-whitespace.sh\n./scripts/ci/trailing-newline.sh\n./scripts/ci/sync-rustc-version.sh\n./scripts/ci/python-sdk-version-sync.sh\n./scripts/ci/uv-lock-check.sh\n./scripts/ci/binary-artifacts.sh --check\n./scripts/ci/third-party-licenses.sh --validate --manifest \n\n# Run things\ncargo run --bin iggy-server -- --with-default-root-credentials\ncargo run --bin iggy -- \nIGGY_CONNECTORS_CONFIG_PATH=... cargo run --bin iggy-connectors\n./scripts/run-bdd-tests.sh\n./scripts/run-benches.sh\n```\n\nPre-commit hooks wire the same scripts. Install with `pre-commit\ninstall` (Python) or `prek install` (Rust drop-in) - both read\n`.pre-commit-config.yaml`.\n\n## Structure\n\n```text\niggy/\n├── core/\n│ ├── server/ Iggy server binary (Viewstamped Replication)\n│ ├── sdk/ Rust client SDK\n│ ├── cli/ iggy CLI\n│ ├── connectors/ Connectors runtime + SDK + sinks/sources\n│ ├── common/ Shared types (IggyDuration, IggyTimestamp, ...)\n│ ├── binary_protocol/ Wire format (stable)\n│ ├── configs/ Config plumbing + ConfigEnv derive\n│ ├── ai/mcp/ MCP server\n│ ├── bench/ Benchmark suite\n│ ├── integration/ Cross-crate integration tests\n│ ├── simulator/ Deterministic simulator\n│ ├── consensus/ Raft (Miri-checked unsafe iobuf)\n│ ├── journal/ WAL / journal\n│ ├── message_bus/ In-process bus\n│ ├── shard/ Shard runtime\n│ ├── harness_derive/ #[iggy_harness] proc macro\n│ └── metadata, partitions, server_common, configs_derive, clock, tools\n├── foreign/ Java / .NET / Python / Go / C++ / Node.js SDKs\n├── bdd/ Cross-language BDD (Gherkin)\n├── examples/ Per-SDK runnable examples\n├── helm/ Helm charts\n├── web/ Web UI\n└── scripts/ CI + release + dev scripts\n```\n\n## Where to look\n\n| Task | Location |\n| --------------------- | ---------------------------------------- |\n| Wire protocol | `core/binary_protocol/` |\n| Server | `core/server/src/` |\n| Rust client SDK | `core/sdk/src/` |\n| Connectors | `core/connectors/` -> connector-* skills |\n| Integration tests | `core/integration/tests/` |\n| Cross-SDK BDD | `bdd/` |\n| Benchmark suite | `core/bench/` |\n\n## Skills\n\nEach `SKILL.md` has YAML frontmatter (name, description). For\nconnectors work, load `connectors-overview` first as router.\nClaude Code uses `/skill-name`; Codex uses `$skill-name` or `/skills`.\nKeep one source under `.claude/skills/` and link its directory from\n`.agents/skills/`. Keep templates and references inside the source directory.\n\n- [connectors-overview](.claude/skills/connectors-overview/SKILL.md) - router + universal connector rules\n- [connector-runtime](.claude/skills/connector-runtime/SKILL.md) - FFI host, lifecycle, state, metrics\n- [connector-sdk](.claude/skills/connector-sdk/SKILL.md) - traits, decoders/encoders, retry\n- [connector-sink](.claude/skills/connector-sink/SKILL.md) - sink plugin authoring\n- [connector-source](.claude/skills/connector-source/SKILL.md) - source plugin authoring\n- [connector-transform](.claude/skills/connector-transform/SKILL.md) - transform authoring\n- [connector-testing](.claude/skills/connector-testing/SKILL.md) - unit + integration test patterns\n\nReview skills are user-invoked only. Keep `disable-model-invocation: true`\nfor Claude Code and `policy.allow_implicit_invocation: false` in\n`agents/openai.yaml` for Codex. Do not replicate their steps without an\nexplicit invocation. When a non-trivial change passes verification,\nsuggest `/team-review ` in Claude Code or `$team-review `\nin Codex.\n\n- [connector-review](.claude/skills/connector-review/SKILL.md) - adversarial connectors PR/branch review with validation\n- [team-review](.claude/skills/team-review/SKILL.md) - adversarial 4-expert PR/branch review, ~10 subagents per run\n- [team-review-slim](.claude/skills/team-review-slim/SKILL.md) - four-expert review without independent validation rounds\n\nReview skills share [execution instructions](.claude/skills/team-review/references/execution.md)\nfor target preparation and client-specific agent tools. Run\n`./scripts/ci/skills.sh` to check skill metadata, links, and invocation policies.\n\n## Repo-wide principles\n\n1. **Apache 2.0 header on every new source file:** Follow the comment style configured in `licenserc.toml`; common examples: `.rs` uses `// ...`, `Cargo.toml` and shell use `# ...`.\n2. **Verification order:** `fmt --all` -> `sort --no-format --workspace` -> `clippy --all-features --all-targets -- -D warnings` -> `test`. Plus `taplo.sh --check`. CI enforces all.\n3. **Always pass `--no-format` to `cargo sort`.** Without it multi-line feature arrays flatten and `taplo` re-expands them, churning the workspace.\n4. **Avoid LLM-slop tells: em dashes, gratuitous semicolons, hedging narrative, trailing summaries.** Defaults, not absolute bans. Same rule for code, comments, commits, PR descriptions.\n5. **Idiomatic Rust over free helpers.** `FromStr` not `parse_foo`, `Display` not `format_foo`, `From`/`TryFrom` not `to_foo_from_bar`. Structured `Err` enums, not `String`.\n6. **Imports at the top.** No inline `use` inside functions or blocks. Group: std, external crates, then `crate::`.\n7. **Precise names.** No `b`, `p`, `t`, `m`. Match literal API field names as log labels.\n8. **Comments explain WHY, not WHAT.** Default to no comment. Add one only when a hidden constraint or workaround would surprise a future reader. Never reference the current task / PR.\n9. **Never `cargo install`** without authorization. Toolchain pinned in `rust-toolchain.toml`. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check.\n10. **No `unwrap()` / `expect()` on Results from external I/O** outside tests.\n11. **`tokio::sync::Mutex` (not `std::sync::Mutex`)** wherever a lock is held across `.await`. Hold locks briefly: lock, clone, drop guard, then do I/O.\n12. **Code reads top to bottom.** Public consts and public fns first, helpers below, deeper helpers below those.\n13. **Zero clone in hot loops.** `&self`, `&str`, `&[T]`. `Vec::with_capacity(n)` when size is known.\n14. **Forward-compatible config.** New TOML fields use `#[serde(default)]` or `Option`.\n\n## Tooling\n\n| Tool | Where |\n| --------------------------------------------- | ------------------------------------------- |\n| `cargo fmt`, `cargo sort`, `cargo clippy` | pre-commit + CI |\n| `cargo machete` | CI `machete` task |\n| `cargo nextest` + `cargo-rail` | CI `test-*` DAG-scoped |\n| `cargo llvm-cov` | CI coverage |\n| `cargo http-registry` | CI `verify-publish` |\n| Miri | CI `miri` (`binary_protocol` + `consensus`) |\n| `taplo`, `markdownlint-cli`, `shellcheck` | pre-commit + CI |\n| `typos` | pre-commit |\n| `cargo about` + `license-checker-rseidelsohn` | `scripts/ci/third-party-licenses.sh` |\n| HawkEye / `license-headers.sh` | pre-commit + CI |\n| `uv`, `ruff` | pre-commit (Python) |\n| `golangci-lint` | pre-push (Go) |\n\n## Testing\n\n- **Unit:** `#[cfg(test)] mod tests` at EOF of `src/lib.rs` (or per-module).\n- **In-crate integration:** `/tests/`.\n- **Cross-crate (real infra):** `core/integration/tests/`. Harness in\n `core/integration/src/harness/` spawns real `iggy-server`\n (`ServerHandle`), connectors runtime (`ConnectorsRuntimeHandle`),\n MCP server (`McpHandle`), client (`ClientHandle`). `#[iggy_harness]`\n proc macro wires fixtures. Suites: `cli/`, `cluster/`,\n `config_provider/`, `connectors/`, `data_integrity/`, `mcp/`, `sdk/`,\n `server/`, `state/`, `storage/`. External backends inside\n `connectors/` use `testcontainers-modules`. Per-test logs land in\n `test_logs/`.\n- **Cross-SDK BDD:** `bdd/`. Gherkin features against every SDK. Driver: `scripts/run-bdd-tests.sh`.\n\nBDD test naming: `given_X_when_Y_should_Z` (3-part). Promote to 4-part `given_X_when_Y_then_Z_should_W` only with a distinct \"then\" intermediate. Be consistent inside a file.\n\nFilter a single integration test by path. Build the binaries the harness launches first: it runs\nwhatever is already in `target/`, so a stale one fails as though the change under test broke it.\n\n```bash\ncargo build --bin iggy-server --bin iggy-connectors\ncargo test -p integration -- connectors::runtime::benchmark::given_logging_format_json\n```\n\n## Pitfalls\n\n- **Docker is required for most integration tests.** `testcontainers-modules` spawns containers per test.\n- **`cargo sort` without `--no-format` will churn the workspace.** Use the pre-commit hook.\n- **Rust toolchain is pinned in `rust-toolchain.toml`.** Older system `cargo` fails on transitive deps. prefer rustup so the pin auto-resolves.\n- **`test_logs/` grows quickly.** Wipe between major refactors.\n- **Miri only covers `binary_protocol` + `consensus`.** Cannot emulate `io_uring` syscalls. do not try to expand Miri to crates that pull `compio`.\n- **Integration crate has no `--test` target.** Filter by test path inside the single `mod.rs` binary.\n- **The integration harness runs prebuilt binaries and never builds them.** `Command::cargo_bin` only resolves a path, so a stale `iggy-server`, `iggy-connectors` or `iggy-mcp` is used silently. CI restores revision-checked artifacts from the runtime and Rust test builds; a local run does not build them, and a stale binary fails in ways that look like the change under test. Rebuild after anything that touches a launched binary or its config.\n\n## Local state directories\n\nCreated by server, connectors runtime, or tests. Gitignored. Safe to delete to reset:\n\n- `local_data/` - server data\n- `local_state/` - connectors runtime state files\n- `test_logs/` - per-test logs + `local_data` snapshots\n- `performance_results/` - benchmark output\n- `target/` - cargo build\n\n## Commits, PRs, releases\n\n- Conventional commits with scope: `feat(connectors): ...`. Subject <= 72 chars. Allowed types/scopes enforced by `.github/workflows/pr-title.yml`.\n- Body: why before what. No bullet lists of files changed.\n- Co-authored-by trailer when applicable.\n- Workspace + SDK versions sync via `scripts/extract-version.sh` and `scripts/ci/{sync-rust,python-sdk}-version{,-sync}.sh`.\n- Crate publish chain verified by `scripts/verify-crates-publish.sh` against `cargo-http-registry`.\n- Third-party license enumeration: `scripts/ci/third-party-licenses.sh` (ASF policy).\n\n## Discussion and support\n\n- Design: \n- Chat: \n- Docs: \n"},"files":{".claude/skills/connector-sdk/SKILL.md":"---\nname: connector-sdk\ndescription: Extend or modify the Apache Iggy Connectors SDK (`core/connectors/sdk/`). Use when adding a new `Schema` variant, a new `StreamDecoder`/`StreamEncoder`, a new `Error` variant, modifying the `Sink`/`Source`/`Transform` trait surface, the FFI macros, or the `retry`/`api`/`convert` modules. NOT for writing plugins - use `connector-sink` or `connector-source` for those.\n---\n\n# Extending the Apache Iggy Connectors SDK\n\nThe SDK (`core/connectors/sdk/`) is the **stable contract** between the\nruntime and every plugin. Changes ripple to every sink/source in-tree\nand every third-party plugin. Treat the public surface as a versioned API.\n\n> **Universal connector rules** (benchmark, SecretString, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers SDK-level changes only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Cardinal rules](#cardinal-rules)\n- [Adding a new `Schema` variant](#adding-a-new-schema-variant)\n- [Adding a `StreamDecoder` / `StreamEncoder`](#adding-a-streamdecoder--streamencoder)\n- [Adding a new `Error` variant](#adding-a-new-error-variant)\n- [Modifying `Sink` / `Source` / `Transform` traits](#modifying-sink--source--transform-traits)\n- [FFI macros (`sink_connector!`, `source_connector!`)](#ffi-macros-sink_connector-source_connector)\n- [`Payload::try_to_bytes(&self)` is non-negotiable for JSON](#payloadtry_to_bytesself-is-non-negotiable-for-json)\n- [Retry helpers (`retry.rs`)](#retry-helpers-retryrs)\n- [`ConnectorState`](#connectorstate)\n- [Tests for SDK changes](#tests-for-sdk-changes)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Bumping the SDK MAJOR version or changing any `#[repr(C)]` layout - misaligns every pre-built plugin `.so`.\n- Changing `Sink` / `Source` / `Transform` / `StreamDecoder` / `StreamEncoder` trait signatures - cascades through every plugin.\n- Removing or renaming a `Schema` variant or `Error` variant - decoders/encoders pinned to it. downstream pattern matches break.\n- Changing wire serialization (postcard / rmp_serde / serde_json) for any FFI / state / config payload.\n- Changing FFI macro return codes or function names (`iggy_*_open` / `consume` / `handle` / `close`) - runtime + plugins both need lockstep update.\n\n## File map\n\n```text\nsdk/src/\n├── lib.rs Traits (Sink, Source, StreamDecoder, StreamEncoder),\n│ Payload, Schema, ConnectorState, Error enum, FFI message structs.\n├── sink.rs SinkContainer + sink_connector! macro (FFI plumbing).\n├── source.rs SourceContainer + source_connector! macro (FFI plumbing).\n├── api.rs ConnectorStatus, ConnectorStats (feature = \"api\").\n├── convert.rs owned_value_into_serde_json / owned_value_to_serde_json (simd_json ⇄ serde_json bridge).\n├── log.rs CallbackLayer for tracing across FFI.\n├── retry.rs retry_async + RetryPolicy, CircuitBreaker, HttpRetryMiddleware.\n├── decoders/ One per schema: json, raw, text, proto, flatbuffer, avro.\n├── encoders/ Mirror of decoders.\n└── transforms/ add_fields, delete_fields, update_fields, filter_fields,\n unwrap_envelope, proto_convert, flatbuffer_convert, avro_convert.\n```\n\n## Cardinal rules\n\n1. **Apache 2.0 license header** on every new file.\n2. **`Send + Sync`** on every public trait (FFI runs across thread pools).\n3. **`#[repr(C)]`** on every type that crosses FFI (`Schema`, `TopicMetadata`, `MessagesMetadata`, `RawMessage`, `ProducedMessages`, `ConsumedMessage`, `DecodedMessage`, etc. in `lib.rs`). Adding fields requires bumping the SDK version - existing plugins built against the old layout will misalign on `postcard::from_bytes`.\n4. **postcard** for FFI message serialization (handled by `SinkContainer::consume`, `SourceContainer`'s `handle_messages`). **MessagePack (`rmp_serde`)** for `ConnectorState`. **JSON** (`serde_json`) only for human-editable config that crosses FFI. Don't mix.\n5. **`simd_json::OwnedValue`** for JSON payloads, not `serde_json::Value`. Use `convert::owned_value_into_serde_json` as a bridge when you own the value (the common case - it moves strings instead of copying them); fall back to the borrowing `convert::owned_value_to_serde_json` only when you still need the `OwnedValue` afterward.\n6. **`BTreeMap`** for headers - deterministic ordering. Never `HashMap` on the wire.\n7. **No breaking changes** to `Sink`/`Source`/`StreamDecoder`/`StreamEncoder`/`Transform` trait signatures without coordinating with all in-tree plugins in the same PR.\n8. **A `Schema` variant that changes meaning is as breaking as a new variant.** The discriminant crosses FFI unchanged, so a plugin built against the old SDK keeps decoding it the old way and misreads every batch with no error anywhere. Same process as a trait change: bump the SDK minor version, update every in-tree plugin in the same PR, and state in the PR description that plugins built against the previous SDK must be rebuilt.\n\n## Adding a new `Schema` variant\n\nThe `Schema` enum (in `lib.rs`) is `#[repr(C)]` - it crosses FFI. Touch points to update **in one PR**:\n\n1. `Schema` enum: add variant with `#[strum(to_string = \"...\")]` matching the `serde(rename_all=\"snake_case\")` form.\n2. `Payload` enum: add a matching variant carrying the deserialized form.\n3. `Payload::try_into_vec` - consuming bytes-out path.\n4. `Payload::try_to_bytes` - **borrowing** bytes-out path. For non-trivial payloads (parsed trees), implement a no-clone serialization, not a `clone() + serialize`. See the `Payload::Json` arm for the canonical optimization.\n5. `Payload::schema()` - variant → `Schema`. The runtime tags every sink batch from this, so a missing arm will not compile but a wrong one silently mislabels the batch.\n6. `Payload::Display`.\n7. `Schema::try_into_payload` - bytes → `Payload`, for a tag naming a wire format (the source path).\n8. `Payload::try_from_schema` - bytes → `Payload`, for a tag that came from `Payload::schema()` (the sink path). The exact inverse of item 5.\n9. `Schema::decoder()` - factory returning `Arc`.\n10. `Schema::encoder()` - factory returning `Arc`.\n11. New `decoders/.rs` and `encoders/.rs`.\n12. Update the schema list in `sdk/README.md`.\n13. Tests: round-trip encode/decode, error paths. `Payload::schema()` must round-trip through `Payload::try_from_schema`, with no exceptions: every variant survives that trip and the test is table-driven over all of them.\n\nMiss any of these → silent failures at runtime (typically `Error::InvalidPayloadType` or surprising decoder behavior in plugin code).\n\n## Adding a `StreamDecoder` / `StreamEncoder`\n\nPattern from `decoders/json.rs::JsonStreamDecoder`:\n\n```rust\npub struct MyStreamDecoder; // unit struct if stateless\n\nimpl StreamDecoder for MyStreamDecoder {\n fn schema(&self) -> Schema { Schema::MyFormat }\n fn decode(&self, payload: Vec) -> Result {\n // Decode payload bytes into Payload::MyFormat(...).\n // Errors: Error::CannotDecode(Schema::MyFormat), Error::InvalidJsonPayload, etc.\n }\n}\n```\n\nFor **stateful** decoders (proto/avro/flatbuffer) holding schema descriptors:\n\n- Store the schema state directly in the struct (`config`, `message_descriptor`, `schema`). The instance is built once and wrapped in `Arc` for sharing - it must be `Send + Sync` but does not need additional locking if construction-time loading is final.\n- Provide **two constructors** when schema loading can fail: lenient `new(config) -> Self` (logs + degrades) AND strict `try_new(config) -> Result` (fail-fast). Plus a `Default` impl for the no-schema case. Canonical example: `sdk/src/decoders/avro.rs` exports all three (`AvroStreamDecoder::new`, `::try_new`, `Default`).\n- If you genuinely need to mutate the schema after construction (e.g., a `update_config` method), use `&mut self` plus `std::mem::replace(&mut self.config, new_config)` to swap without cloning the old value. Live pattern: `sdk/src/decoders/avro.rs::AvroStreamDecoder::update_config` and `sdk/src/encoders/avro.rs::AvroStreamEncoder::update_config`.\n- A fresh decoder instance is created via `Schema::decoder()` on each call (it returns `Arc`), so per-decoder caching of shared mutable state is the wrong abstraction.\n\n## Adding a new `Error` variant\n\nThe `Error` enum (in `lib.rs`) is `Clone + PartialEq + Eq + Hash`. New variants must preserve these. Guidelines:\n\n- **Use existing variants first.** Only add a new one if the failure mode is distinct in handling, not just description. `InvalidConfigValue(String)` covers most \"bad config\" cases.\n- **Carry context as `String`** - don't add a struct unless multiple discrete fields are needed by callers programmatically.\n- **Document retry semantics in the docstring.** Existing variants (`PermanentHttpError`, `CatalogCommitError`, `TransactionApplyError`) document retry behavior - mirror that style.\n- **Place near related variants** to keep the enum readable top-to-bottom.\n\n## Modifying `Sink` / `Source` / `Transform` traits\n\n**Breaking changes.** Required process:\n\n1. Document migration in the SDK changelog (and `connectors/sdk/README.md`).\n2. Update every in-tree plugin in the same PR. CI builds them all.\n3. Bump SDK minor version.\n4. Note in PR description: dynamic plugins compiled against the old SDK version will fail to load with mismatched FFI symbols.\n\nNon-breaking additions (new default method, new struct field with `#[serde(default)]`, new variant on a non-exhaustive enum) are preferable.\n\n## FFI macros (`sink_connector!`, `source_connector!`)\n\nThe macros (in `sink.rs` and `source.rs`) generate:\n\n- `iggy__open(id, config_ptr, config_len, [state_ptr, state_len,] log_callback) -> i32`\n- `iggy_sink_consume(id, ...)` / `iggy_source_handle(id, callback) -> i32`\n- `iggy__close(id) -> i32`\n- `iggy__version() -> *const c_char` (static lifetime, from `env!(\"CARGO_PKG_VERSION\")`)\n\nInvariants:\n\n- **Duplicate-ID guard** returns `-1` if the caller reopens an ID without closing first. Don't remove this - it prevents silent buffered-data loss.\n- `INSTANCES: Lazy>>` (and `SourceContainer<$type>` mirror) is the only mutable global the macro introduces - keep it that way.\n- Return codes: `0` success, `-1` invalid call, `1` open failure. Don't repurpose.\n\nChanges to the FFI signature must update `runtime/src/main.rs::{SourceApi, SinkApi}` in the same PR.\n\n## `Payload::try_to_bytes(&self)` is non-negotiable for JSON\n\nPlugin authors call this on every consumed message. The implementation in `lib.rs::Payload::try_to_bytes` documents *why* it skips the deep `OwnedValue` clone - replacing `O(n) clone + O(n) serialize` with `O(n) serialize`. If you add a new variant requiring a parsed tree, do the same: serialize in place, don't clone.\n\n## Retry helpers (`retry.rs`)\n\n- `CircuitBreaker`: threshold + cooldown, `try_lock()` on the success path to avoid hot-path contention.\n- `retry_async(policy, context, should_retry, op)`: the retry loop for anything failing as `Err`. Owns attempt counting, backoff and the per-retry log; returns `RetryFailure { error, attempts, exhausted }` so callers write their own terminal log.\n- `retry_backoff(base, retry, max)`: backoff only, for a loop that computes its own delay. `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath and applies no jitter, so call it directly only when the delay must be exact (`source.rs::nack_retry_delay`, whose tests assert exact values).\n- `HttpRetryMiddleware`: integrates with `reqwest-middleware`. Retries 429 + 5xx + network errors. Honors `Retry-After` on 429 and 5xx as a floor over the computed backoff, capped at a one-hour `MAX_RETRY_AFTER`, not `max_delay`.\n- `max_retries` = **total attempts** including the first try, not extra retries. Document if you change this convention. `meilisearch_sink` is the standing exception: its `max_retries` / `max_open_retries` count retries *after* the first, as its README states.\n- New helpers must take `Duration` (not `u64 millis`) on the public API. Internal computation uses `humantime` parsing of `String`.\n\n## `ConnectorState`\n\n- Bytes are opaque - the wrapping type does not impose schema.\n- Serialization is MessagePack via `rmp_serde`. Compact, deterministic, well-supported in serde.\n- `serialize`/`deserialize` helpers return `Option` and log on failure. **Failures are non-fatal** - design downstream code to tolerate fresh state.\n- Don't change the underlying format without coordinating with all sources.\n\n## Tests for SDK changes\n\n- Unit tests for any new pure function in the changed module.\n- Round-trip tests for new schemas/transforms (encode → decode == identity for JSON-equivalent payloads).\n- `sdk/tests/` integration tests when the change crosses module boundaries.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n\n# Rebuild all plugins to catch breaking-change leaks:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n# If FFI signatures changed, also build the runtime:\ncargo build -p iggy-connectors\n```\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-sink/SKILL.md":"---\nname: connector-sink\ndescription: Author a new Apache Iggy connector sink plugin under core/connectors/sinks/. Sinks consume messages from Apache Iggy streams and push them to an external system (DB, search engine, HTTP endpoint, object store). Load when creating, modifying, or reviewing a sink crate. Use for sink plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Sink\n\nA **sink** is a Rust `cdylib` that implements `iggy_connector_sdk::Sink`\nand exposes FFI symbols via the `sink_connector!` macro. The runtime\nloads it as `.so`/`.dylib`/`.dll`, polls Apache Iggy on its behalf,\nruns the transform chain, decodes via the configured `Schema`, and\nhands the sink batches of `ConsumedMessage`. The sink is responsible\nfor getting them to the external system reliably and efficiently.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's sink-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Sink::open` / `consume` / `close`) - that's an SDK change, not a plugin one.\n- Touching default `batch_size`, retry caps, or backoff defaults baked into existing in-tree sinks.\n- Promoting an `Err` variant to retryable when it previously was not - retrying bad data trips circuit breakers.\n- Adding a new sink that introduces a backend the runtime doesn't yet integration-test for.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand when authoring).\n- Exemplars: `stdout_sink` (minimal), `postgres_sink` (DB + transient detection), `http_sink` (validation, batch modes, retry middleware), `mongodb_sink` (atomic counters), `elasticsearch_sink` / `iceberg_sink` (backend-specific idioms).\n\n## Hard rules\n\n### Lifecycle\n\n- `Sink::consume` takes **`&self`** - never `&mut self`. Mutable state goes behind `tokio::sync::Mutex` / `AtomicU64`. Prefer atomics for hot counters - `mongodb_sink` uses `messages_processed: AtomicU64` + `insertion_errors: AtomicU64` and avoids a state lock on the hot path entirely.\n- `Sink::open` builds the client AND tests connectivity. Fail fast with `Error::InitError`.\n- `Sink::close` flushes + drops resources. Take ownership via `client: Option` + `.take()`. sqlx pools have `.close().await`. reqwest/mongodb/elasticsearch clients just drop. Log final stats either way.\n\n### Config\n\n- Every runtime-tunable field is `Option` with a default applied in `new()`. Forward-compat: adding a field doesn't break existing TOML.\n- Durations: `Option`, parsed via `humantime::Duration` in `new()`. Two equivalent in-tree idioms - inline `HumanDuration::from_str(&raw).map(|d| *d).unwrap_or_else(..)` (`random_source`, `postgres_source`) or a local `parse_duration` helper (`http_sink`, `sdk/src/retry.rs`). Fall back to a default on parse failure with `warn!` - never panic. We do **not** use `humantime_serde`.\n- **Conflict resolution**: if two config fields can be in invalid relative ordering, fix it in `new()` + `warn!` - don't error. `http_sink` swaps `retry_delay` and `max_retry_delay` when reversed to avoid an `ExponentialBackoff` panic downstream.\n- Validation order: structural in `new()`, connectivity in `open()`, **never in `consume()`**.\n\n### Payloads (efficiency-critical)\n\n- Dispatch on the `Payload` variant. Handle `Json`, `Raw`, `Text` and `Proto` at minimum. `Proto` holds text that is a JSON document when `proto_convert` had no descriptor or could not encode, and arbitrary text otherwise: call `Payload::json_document()` first and fall back to the text handling when it returns `None`. Unsupported -> `Error::InvalidPayloadType` or fall back to base64 (`elasticsearch_sink`).\n- For `Payload::Json`, call `try_to_bytes(&self)` - single-pass serialization without cloning the `OwnedValue` tree (`sdk/src/lib.rs::Payload::try_to_bytes`). Never `payload.clone().try_into_vec()`.\n- To take a payload out of `ConsumedMessage` without cloning: `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))`. Standard pattern across `http_sink::send_individual` / `send_ndjson` / `send_json_array`.\n- Pre-allocate per-batch buffers with `Vec::with_capacity(messages.len())` (or `* factor` for serialized bodies). Used across `postgres_sink`, `mongodb_sink`, `quickwit_sink`, `elasticsearch_sink`, `http_sink`.\n\n### Headers\n\n- `message.headers: Option>`. Check `.is_empty()` before iterating - runtime treats empty as `None`.\n- For binary values use `v.as_raw()` + base64. for text use `v.to_string_value()`. See `http_sink::EncodedHeader`.\n- `BTreeMap` for deterministic ordering. Don't collect into `HashMap`.\n\n### Errors\n\n| Scenario | SDK variant | Retry? |\n| ------------------------------------------ | --------------------------------------------------------------------------- | ---------------------------------------- |\n| Init / connectivity failure | `Error::InitError` | n/a (fails open) |\n| Bad config value | `Error::InvalidConfigValue(field)` | n/a |\n| Single-message validation failure | `Error::InvalidRecordValue(reason)` | no - skip + log |\n| Transient backend (5xx, conn reset, ...) | `Error::HttpRequestFailed` / `Error::Connection` / `Error::CannotStoreData` | yes |\n| Permanent 4xx, schema mismatch, constraint | `Error::PermanentHttpError` / `Error::SchemaMismatch` | **no** - retrying trips circuit breakers |\n| I/O failure mid-write (parquet/iceberg) | `Error::WriteFailure` | caller decides |\n| Catalog commit failure (Iceberg) | `Error::CatalogCommitError` | **no - not idempotent** |\n\n### Retry\n\n- SDK helpers cover the simple case: `iggy_connector_sdk::retry::check_connectivity_with_retry(...)` for `open()`, `HttpRetryMiddleware` for default 429/5xx/network policy on reqwest clients.\n- Custom strategies: `reqwest-middleware` + `reqwest_retry::RetryTransientMiddleware::new_with_policy_and_strategy`. `http_sink` defines its own `HttpSinkRetryStrategy` (honors `success_status_codes`, per-status decisions).\n- Non-HTTP clients: write `is_transient_error(&e)` mapping driver-specific codes. `postgres_sink::is_transient_error` maps SQLSTATEs `40001`, `40P01`, `57P01-03`, `08000/03/06`.\n- Retry loop: new connectors use `iggy_connector_sdk::retry::retry_async(policy, context, should_retry, op)` for anything that fails as `Err`. It owns attempt counting, backoff and the per-retry log, and returns `RetryFailure { error, attempts, exhausted }`; the caller logs the terminal failure.\n- Backoff only, for a loop that computes its own delay (it retries on an `Ok` response, carries a deadline, or reconnects between attempts): `retry_backoff(base, retry, max)`, where `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath, and it applies no jitter; call it directly only when the delay must be exact, as `sdk/src/source.rs::nack_retry_delay` needs for its tests.\n- Cap retries at 3 total attempts.\n\n### Idempotency\n\nRuntime retries on any `Err` - same batch can be redelivered. In-tree guarantees:\n\n- **Postgres** (`postgres_sink::build_batch_insert_query`): PK on `id`, but plain `INSERT` (no `ON CONFLICT`). Duplicate retry triggers `23505`, `is_transient_error` does NOT mark it transient -> error propagates. Adding `ON CONFLICT (id) DO NOTHING` is a known improvement.\n- **MongoDB**: composite `_id` from `stream:topic:partition:message_id`, via `insert_many(docs).ordered(false)`. Per-document duplicate-key errors don't abort the batch - effective idempotency under retry.\n- **Elasticsearch** (`bulk_index_documents`): bulk action specifies only `_index`, no `_id` -> ES auto-generates ids -> NOT idempotent. Extensions should add `\"_id\"`.\n- **HTTP**: endpoint-dependent. Document in the plugin README.\n- **Iceberg**: transactional commit. `Error::CatalogCommitError` documents that the txn is consumed on `commit()`. Rebuild-then-retry can duplicate data - callers must check the catalog first.\n\nWhen writing a new sink, prefer dedup-on-write using the message `id` field. Mongo's upsert-style is the cleanest reference.\n\n### Batching\n\n- Default `batch_size: 100`. Tune per-backend.\n- Chunk via `messages.chunks(self.batch_size)`. Do not buffer across `consume()` - runtime already batches at the consumer-poll boundary.\n- For HTTP backends support multiple batch modes when applicable (individual / ndjson / json_array / raw). See `http_sink::BatchMode`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: \", self.id);\ndebug!(\"Processing batch of {} messages, offset: {}\", batch.len(), offset);\nwarn!(\"Retry {attempt}/{max} for ID: {}, reason: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, processed: {}\", self.id, count);\n```\n\nUse literal API field names as labels (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` - runtime can't dlopen a regular rlib.\n2. Cloning the whole `Payload::Json` to serialize - use `try_to_bytes(&self)`.\n3. Holding `Mutex` across `.await` when the locked region does I/O.\n4. `&mut self` on `consume` - won't compile. you need interior mutability.\n5. Returning the first error and dropping subsequent batches - track `last_err`, process all, return.\n6. Forgetting to map `PermanentHttpError` - retries hammer the backend with bad data.\n7. Mutating config in `consume()` - config is a snapshot from `new()`. Reload requires runtime restart.\n8. Logging the connection string. Redact at construction: `http_sink::sanitize_url_for_log` populates `HttpSink.log_url`.\n\n## Tests\n\nUnit tests at EOF of `src/lib.rs`. See [connector-testing](../connector-testing/SKILL.md) for BDD naming + `test_config()` helper.\n\nReal-infra integration tests live under `core/integration/tests/connectors//` using `#[iggy_harness]` + a backend fixture that spins up Docker via `testcontainers-modules`. Plugins integrating with external services must have an integration test. Reference: `core/integration/tests/connectors/postgres/postgres_sink.rs` + `fixtures/postgres/`.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__sink --all-targets -- -D warnings\ncargo test -p iggy_connector__sink\n\n# Integration tests (filter by name; integration crate has no per-area target):\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sinks/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-transform/SKILL.md":"---\nname: connector-transform\ndescription: Add a new transform under `core/connectors/sdk/src/transforms/`. Transforms mutate, filter, or convert messages between source/encode and decode/sink. Use when adding field-level transforms (add/delete/update/filter), format conversions, envelope unwrapping, or similar config-driven message manipulations for Apache Iggy connectors. Use for transform authoring. NOT for runtime internals or sink/source loops.\n---\n\n# Adding a New Apache Iggy Connector Transform\n\nA transform implements `iggy_connector_sdk::transforms::Transform`. It\nruns after decode (for sinks) or before encode (for sources), in a chain\nconfigured per-connector. Config-driven, composable.\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers transform authoring only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [The trait](#the-trait)\n- [Steps to add a new transform](#steps-to-add-a-new-transform)\n- [Rules](#rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Adding a new transform that mutates payload SHAPE (renaming PK fields, dropping schema-required columns) - breaks downstream sinks silently.\n- Introducing non-determinism (system time, RNG without seed) - breaks tests + replay.\n- Changing the `Transform` trait signature (sync vs async, return type) - cascades to every transform.\n- Making transforms hold mutable runtime state - they're constructed once and `Arc`-cloned across tasks. mutation requires interior locking and is rarely what you actually want.\n\n## File map\n\n```text\nsdk/src/transforms/\n├── mod.rs Transform trait, TransformType enum, from_config() factory,\n│ FieldValue + ComputedValue helpers\n├── add_fields.rs AddFields - append static or computed fields to JSON\n├── delete_fields.rs DeleteFields - remove fields by key\n├── update_fields.rs UpdateFields - conditional field replacement\n├── filter_fields.rs FilterFields - keep/drop messages by key/value pattern\n├── unwrap_envelope.rs UnwrapEnvelope - promote nested field to top-level payload\n├── proto_convert.rs ProtoConvert - bidirectional Proto ⇄ Json/Text/Raw/etc.\n├── flatbuffer_convert.rs FlatBufferConvert\n├── avro_convert.rs AvroConvert\n└── json/ Shared JSON helpers (path traversal, ops)\n```\n\n## The trait\n\n```rust\n// sdk/src/transforms/mod.rs - Transform\npub trait Transform: Send + Sync {\n fn r#type(&self) -> TransformType;\n fn transform(\n &self,\n metadata: &TopicMetadata,\n message: DecodedMessage,\n ) -> Result, Error>;\n}\n```\n\nSemantics:\n\n- **`&self`** - transforms are immutable post-construction. Internal state (compiled regex, parsed paths, schema descriptors) is computed in `new()` and stored as plain struct fields. No locks needed - the whole instance is shared via `Arc`.\n- Takes `message` **by value** - the transform owns the `DecodedMessage` and may mutate freely without cloning.\n- Returns `Ok(Some(msg))` to pass through (possibly modified), `Ok(None)` to **filter** (drop silently), `Err(e)` for actual failures.\n- `&TopicMetadata` is for routing-aware logic. Most transforms ignore it.\n\n## Steps to add a new transform\n\n### 1. Create the file\n\n`sdk/src/transforms/my_transform.rs`. Apache 2.0 license header. Code reads top to bottom - config struct, then transform struct, then `impl`, then helpers.\n\n```rust\n/* Apache 2.0 header */\n\nuse crate::transforms::{Transform, TransformType};\nuse crate::{DecodedMessage, Error, Payload, TopicMetadata};\nuse serde::{Deserialize, Serialize};\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\npub struct MyTransformConfig {\n pub key: String,\n pub option: Option,\n}\n\npub struct MyTransform {\n config: MyTransformConfig,\n}\n\nimpl MyTransform {\n pub fn new(config: MyTransformConfig) -> Result {\n if config.key.is_empty() {\n return Err(Error::InvalidConfigValue(\n \"my_transform.key must not be empty\".into(),\n ));\n }\n Ok(Self { config })\n }\n}\n\nimpl Transform for MyTransform {\n fn r#type(&self) -> TransformType {\n TransformType::MyTransform\n }\n\n fn transform(\n &self,\n _metadata: &TopicMetadata,\n mut message: DecodedMessage,\n ) -> Result, Error> {\n // Most transforms operate only on Payload::Json - non-JSON is a no-op.\n let Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n };\n\n // Mutate `value` in place via simd_json::OwnedValue API.\n // Return Ok(None) to filter, Ok(Some(message)) to pass through.\n\n Ok(Some(message))\n }\n}\n```\n\n### 2. Wire into `transforms/mod.rs`\n\nThree edits, all required:\n\n(a) Module declaration + re-export at the top:\n\n```rust\nmod my_transform;\npub use my_transform::{MyTransform, MyTransformConfig};\n```\n\n(b) Discriminant in `TransformType`:\n\n```rust\npub enum TransformType {\n AddFields,\n // ...\n MyTransform, // snake_case form in TOML: \"my_transform\"\n}\n```\n\n(c) Branch in `from_config()`:\n\n```rust\nTransformType::MyTransform => {\n // For new transforms, prefer Error::InvalidConfigValue with formatted context -\n // it preserves the underlying serde_json error. Existing code uses bare\n // Error::InvalidConfig for most variants (UnwrapEnvelope is the exception).\n let cfg: MyTransformConfig = serde_json::from_value(raw.clone())\n .map_err(|error| Error::InvalidConfigValue(format!(\"my_transform: {error}\")))?;\n Ok(Arc::new(MyTransform::new(cfg)?))\n}\n```\n\n`from_config()` returns `Arc` so the same instance is shared across the pipeline.\n\n### 3. Document\n\nAdd a TOML usage example to `sdk/README.md`:\n\n```toml\n[transforms.my_transform]\nenabled = true\nkey = \"some_field\"\noption = true\n```\n\n### 4. Tests\n\nBottom of `my_transform.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn given_matching_field_should_transform() { ... }\n\n #[test]\n fn given_non_json_payload_should_passthrough() { ... }\n\n #[test]\n fn given_filter_condition_should_drop_message() { /* assert Ok(None) */ }\n\n #[test]\n fn given_invalid_config_should_error_in_new() {\n let cfg = MyTransformConfig { key: \"\".into(), option: None };\n assert!(matches!(MyTransform::new(cfg), Err(Error::InvalidConfigValue(_))));\n }\n\n fn topic_meta() -> TopicMetadata {\n TopicMetadata { stream: \"s\".into(), topic: \"t\".into() }\n }\n\n fn json_message(json: serde_json::Value) -> DecodedMessage { /* ... */ }\n}\n```\n\n## Rules\n\n### Operate on JSON only (mostly)\n\nMost field-level transforms only make sense for `Payload::Json`. Convention (see `add_fields.rs::AddFields::transform`): silently pass through non-JSON:\n\n```rust\nlet Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n};\n```\n\nFormat-conversion transforms (`proto_convert`, `avro_convert`, `flatbuffer_convert`) are the exception - they explicitly convert between variants.\n\n### Mutate `OwnedValue` in place, don't clone-then-replace\n\n`simd_json::OwnedValue` is mutable in place:\n\n```rust\nif let OwnedValue::Object(ref mut obj) = value {\n obj.insert(key.into(), new_value);\n}\n```\n\nDo NOT clone the value, mutate the clone, then `message.payload = Payload::Json(new)`. The whole point of `&mut value` is in-place mutation.\n\n### Filter semantics\n\n`Ok(None)` drops the message **silently** - no error log, no metric. By design (filters succeed when they decide to drop). For visibility, log at `debug!` level.\n\n### Config validation\n\nValidate in `new()` and return `Err(Error::InvalidConfigValue(...))`. Don't `panic!`. Don't validate on the hot path.\n\nFor transforms with regex or compiled patterns, compile once in `new()` and store the compiled form.\n\n### Stateless except for config\n\nTransforms are constructed once at startup and `Arc`-cloned across tasks. Must be `Send + Sync`. No mutable runtime state - if you need accumulator-like behavior, that's not a `Transform`, that's a sink.\n\n### Compiled state goes in the struct, not behind a lock\n\nThings derived from config (compiled regex, parsed paths, schema descriptors) should be computed in `new()` and stored as plain struct fields. The whole instance is shared via `Arc`, so per-field locking is unnecessary. See `sdk/src/transforms/filter_fields.rs::FilterFields::new` for the canonical pattern: regexes compile in `new()`, get stored as struct fields, are then read via `&self`.\n\n### `transform()` is sync\n\nThe trait method is sync (not `async`). Don't propose making it async - that would require pervasive change across the runtime. Schemas and other external resources should be loaded in `new()` (potentially via a fallible `try_new`), not lazily during `transform()`.\n\n### Errors\n\n| Scenario | Variant |\n| ---------------------------------- | -------------------------------------------------- |\n| Config validation in `new()` | `Error::InvalidConfigValue(\"transform_name: ...\")` |\n| Per-message validation (bad shape) | `Error::InvalidRecordValue(\"...\")` |\n| Schema resolution failure | `Error::InvalidTransformer` |\n| Format conversion failure | `Error::InvalidPayloadType` |\n\nReturning `Err` from `transform()` stops the message and is logged. For non-fatal field-level issues (field missing when transform expects it), prefer pass-through (`Ok(Some(message))`) unless config explicitly says to fail.\n\n## Common pitfalls\n\n1. **Forgetting one of the three `mod.rs` edits** - module declaration, `TransformType` variant, `from_config()` branch.\n2. **`#[serde(rename_all = \"snake_case\")]` mismatch** between enum variant name and TOML key.\n3. **Cloning `OwnedValue`** when in-place mutation would work.\n4. **Validating config inside `transform()`** instead of `new()`.\n5. **`std::sync::Mutex` for internal state** on the hot path - precompute in `new()` and store as plain struct fields. Transforms run on every message. locks are wasted work.\n6. **Returning `Err` for a filter** - filters use `Ok(None)`.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n```\n\nVerify by adding the transform to `core/connectors/runtime/example_config/connectors/` and running the runtime - the transform should apply to each message.\n\nIf the transform changes payload shape sufficiently to break downstream sinks (e.g., renaming primary keys), call it out in the PR description.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-source/SKILL.md":"---\nname: connector-source\ndescription: Author a new Apache Iggy connector source plugin under core/connectors/sources/. Sources poll an external system (DB, API, queue) and produce messages into Apache Iggy streams. Load when creating, modifying, or reviewing a source crate. Use for source plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Source\n\nA **source** is a Rust `cdylib` that implements\n`iggy_connector_sdk::Source` and exposes FFI symbols via the\n`source_connector!` macro. The runtime calls `poll()` in a loop,\napplies transforms, encodes via the configured `Schema`, sends to\nApache Iggy, and persists the returned `ConnectorState` after every\nsuccessful send.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's source-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Source::open` / `poll` / `on_batch_result` / `close`) - that's an SDK change.\n- Adding a long-running side task in the plugin - the runtime owns lifecycle, and orphans survive `close()`. Sanctioned only where the source is itself a server the runtime cannot drive, as in `http_source`'s listener, and then only with an explicit shutdown in the last `close()` that awaits its tasks before returning.\n- Persisting unbounded state - `State` is rewritten every batch.\n- Adding a source that requires authoritative offsets external to Apache Iggy without coordinating retention.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand).\n- Exemplars: `random_source` (minimal + canonical state tests), `postgres_source` (cursor / delete-after-read / processed-column modes, restart-survives-state tests), `elasticsearch_source` (scroll cursor), `influxdb_source` (time-series scan).\n\n## Hard rules\n\n### `poll()` signature is `&self`\n\nThe macro shares the source as `Arc` across the FFI callback and forwarding loop. Signature: `async fn poll(&self) -> ...` - any mutable state behind `tokio::sync::Mutex`. **Single most common new-contributor mistake.**\n\n### Lock discipline\n\nNever hold the state `Mutex` across upstream I/O. Build a candidate from committed\nstate, then stage it until the runtime reports the batch result:\n\n```rust\nlet mut candidate = self.state.lock().await.clone();\nlet rows = client.query(&sql, &[&candidate.cursor]).await?;\ncandidate.cursor = Some(new_cursor);\nlet persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n*self.pending.lock().await = Some(candidate);\n```\n\n### Delivery acknowledgment\n\n`on_batch_result` (added by #3855) is how a source learns what happened to the batch it just\nreturned. The SDK keeps exactly one batch in flight: it will not call `poll()` again until this\nreturns. By default it stops after `MAX_CONSECUTIVE_NACKS` (5) consecutive NACKs; a source can\noverride `batch_policy()` to disable the limit when its accepted input cannot be replayed after\nrestart. A stop is reported to the runtime only when the plugin exports the current SDK's stop\ncallback, so rebuild older source plugins.\n\n- `Ack` means the runtime sent the batch **and** persisted its state. `Nack` means it could not\n confirm both, which is **not** the same as neither happening: a batch that reached the topic but\n whose state save failed is NACKed. A missing runtime result leaves the batch pending; it does not\n produce a synthetic NACK. A source that replays on `Nack` is at-least-once, not exactly-once.\n- The trait has a **default no-op**, which suits only a source with no staged cursor and no\n destructive work. If `poll()` advances a cursor, deletes rows, or drains an in-memory buffer,\n omitting this loses data silently and nothing will tell you. `random_source` and `http_source`\n implement it; the other shipped sources take the default and skip rows on a NACK.\n- Stage in `poll()`, apply on `Ack`, discard or replay on `Nack`. A source whose input is pushed to\n it, rather than re-readable upstream, has to hold the batch itself: see `http_source`'s staging.\n- Returning `Err` from it stops the source immediately, so it is not a retry signal.\n\n### State persistence\n\n- `ConnectorState` is `Vec` via MessagePack (`rmp_serde`). Use\n `ConnectorState::serialize(&state, NAME, id)` and\n `ConnectorState::deserialize::(NAME, id)`.\n- `poll()` must not commit cursors or destructive work. Return messages with candidate state and\n keep the corresponding work staged.\n- The runtime sends the batch, saves its candidate state to\n `{state_path}/source_{key}.state`, then calls `on_batch_result(Ack)`. Commit staged in-memory\n state and external delete or mark operations only on ACK. A NACK discards the candidate so the\n same data can be polled again.\n- A crash between `poll()` returning and state persistence leaves the prior cursor for the next\n poll, so downstream must tolerate at-least-once delivery.\n- Cursor sources that stage state until `on_batch_result` can attach state to the corresponding\n message batch. Control-plane state that is independent of a publish can ride an empty batch\n instead.\n- Return `state: None` for an empty poll when no watermark changed. If an empty poll advances a\n watermark, stage and return the new state through the same ACK handshake.\n- The runtime can still NACK an empty batch when state storage is latched or a pending checkpoint\n cannot resolve. Never treat hand-off as durable; `on_batch_result` reports the outcome.\n- Treat candidate-state serialization failure as a poll error. Do not send messages without the\n state needed to resume them safely.\n- Keep `State` small - rewritten every batch. No unbounded vecs.\n\nThe SDK allows one in-flight batch. Five consecutive NACKs stop a source using\nthe default policy and require a manual restart. Returning `Err` from\n`on_batch_result` is fatal regardless of the NACK limit, so\nretry transient backend failures inside the callback before returning an error.\nIf the runtime has not reported ACK or NACK after 30 seconds, the SDK warns\nand keeps waiting for that result without polling or replaying the batch.\nOnce the result is received, the SDK waits for `on_batch_result` to finish,\nso the callback must bound its own connection acquisition and retry budget.\n\n### Sleep first\n\n`poll()` must `sleep(self.poll_interval).await` before any work. Without it, an empty source spins a CPU.\n\n### Schema selection\n\nMatch `ProducedMessages.schema` to the bytes in `messages[i].payload`:\n\n- JSON-serialized rows → `Schema::Json`\n- Already-protobuf bytes → `Schema::Proto`\n- Already-avro bytes → `Schema::Avro`\n- Opaque → `Schema::Raw`\n\n### IDs and timestamps\n\n- `ProducedMessage.id: Option` - set when a natural ID exists (DB PK, document id). It rides the wire as the message id, which is what a **consumer** can dedupe an at-least-once duplicate on.\n- **Iggy itself does not dedupe on that id.** The server's only read of the field is `core/server/src/http/wire.rs`, which mints a fresh uuid when the incoming id is 0 and otherwise passes it through. Its dedup path is a per-client request-id watermark (`dedup_clients_max`) that never looks at this field.\n- `origin_timestamp: Option` - source-system event time in nanoseconds. Lets downstream sinks reason about lag.\n- `timestamp` and `checksum` are Iggy-side - leave `None`.\n\n### Concurrency\n\n- Runtime spawns ONE `poll()` task per source. No concurrent `poll()`.\n- Don't spawn your own long-running Tokio tasks: the runtime owns lifecycle. The exception is a source that listens rather than polls, which has to own its listener; `http_source` is the worked example, and it shuts its tasks down in the last `close()` rather than leaving them to outlive the connector.\n\n### Errors\n\n| Scenario | Variant |\n| ------------------------------------------- | ------------------------------------------------- |\n| Bad config in `new()`/`open()` | `Error::InitError` |\n| Cannot reach external system at startup | `Error::InitError` or `Error::Connection` |\n| Transient fetch failure (retry-worthy) | `Error::Connection` or `Error::HttpRequestFailed` |\n| Permanent fetch failure (auth, schema gone) | `Error::PermanentHttpError` |\n| Row failed to serialize | `Error::Serialization(...)` |\n| State serialization failed | `Error::Serialization(...)` |\n\nReturning `Err` from `poll()` is only logged by the SDK's FFI bridge\n(`sdk/src/source.rs::handle_messages`) - the loop continues, the next\n`poll()` runs. Connector status does NOT flip to `Error` from a poll\nfailure. Status `Error` is set by the runtime only on transform/encode\nfailure, Iggy send failure, or state save failure\n(`runtime/src/source.rs::source_forwarding_loop` calls to\n`context.sources.set_error`). To surface a poll failure as unhealth,\nraise it through the metric counter or escalate to `Error::InitError`\nfrom `open()`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: {}\", self.id, ...);\ninfo!(\"Restored state for ID: {id}, cursor: {:?}\", ...);\ndebug!(\"Polled {} rows for ID: {}\", rows.len(), self.id);\nwarn!(\"Transient fetch failure for ID: {}, will retry: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, total produced: {}\", self.id, ...);\n```\n\nIggy consumer-loop labels use literal API names (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. `async fn poll(&mut self)` - won't compile. Use `&self` + `Mutex`.\n2. Holding `state.lock()` across the fetch I/O - blocks `close()`, causes shutdown timeouts.\n3. Forgetting to sleep - 100% CPU on idle source.\n4. Committing a cursor or deleting source data in `poll()` - stage it and wait for ACK.\n5. Unbounded data in `State` - rewritten every batch. keep O(constant).\n6. `std::sync::Mutex` - blocks the executor. Use `tokio::sync::Mutex`.\n7. Not setting `ProducedMessage.id` when a stable ID exists - leaves a consumer nothing to dedupe a replayed duplicate on. It does not make the write idempotent server-side, because nothing there reads it.\n8. Spawning side tasks - the runtime owns the scheduler. The one exception is a source that listens rather than polls and must own its listener (see [Concurrency](#concurrency) and the STOP list); it owes an explicit shutdown in the last `close()` that awaits its tasks.\n\n## Tests\n\nMandatory four canonical source state tests (see [connector-testing](../connector-testing/SKILL.md) for the full pattern). Copy from `sources/random_source/src/lib.rs::tests`. Plus config defaults, payload building, schema selection.\n\nIntegration tests under `core/integration/tests/connectors//` for any source backed by external infra. Use `#[iggy_harness]` + a `TestFixture` backed by `testcontainers-modules`. Reference: `core/integration/tests/connectors/postgres/postgres_source.rs` (multi-mode tests) + `restart.rs` (state survives restart). Exercise both ACK and NACK paths when the source stages cursors or destructive work.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__source --all-targets -- -D warnings\ncargo test -p iggy_connector__source\n\n# Integration tests:\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sources/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-runtime/SKILL.md":"---\nname: connector-runtime\ndescription: Modify the Apache Iggy Connectors runtime (`core/connectors/runtime/`) - the host process that loads plugins, manages lifecycle, and bridges Apache Iggy streams to plugins via FFI. Use when changing plugin loading, the FFI dispatch, the sink/source manager, state storage, config providers, the HTTP control API, metrics, or transforms wiring. NOT for plugin code.\n---\n\n# Modifying the Apache Iggy Connectors Runtime\n\nRuntime is the host. Loads `.so`/`.dylib`/`.dll` plugins, drives their\nlifecycle, ferries messages between Apache Iggy and plugins, exposes\n`/stats`, `/metrics`, control HTTP API. Bugs here affect every plugin.\n\n> **Universal connector rules** (benchmark, SecretString, verbose, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers runtime internals only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Architecture invariants](#architecture-invariants)\n- [State storage (`state.rs`)](#state-storage-staters)\n- [Source forwarding loop](#source-forwarding-loop)\n- [Sink consumption loop](#sink-consumption-loop)\n- [Logging format](#logging-format)\n- [The `verbose` flag](#the-verbose-flag)\n- [Config provider rules](#config-provider-rules)\n- [Error categorization](#error-categorization)\n- [Metrics (`metrics.rs`)](#metrics-metricsrs)\n- [Drop accounting (wired sites)](#drop-accounting-wired-sites)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing FFI `#[repr(C)]` layouts (`SourceApi`, `SinkApi`, `RawMessages`, headers) - existing plugin `.so`s misalign.\n- Modifying `state.rs` save protocol (atomic tmp + rename + fsync ordering) - corruption risk.\n- Renaming the default consumer group `iggy-connect-sink-{key}` - operators hold offsets there.\n- Hot-reloading or unloading a `.so` - mid-FFI tasks would segfault. Architectural change required.\n- Changing the SerDe split (postcard for FFI payloads, serde_json for configs, rmp_serde for state) - downstream incompat.\n\n## File map\n\n```text\nruntime/src/\n├── main.rs Entry, plugin path resolution, SourceApi/SinkApi FFI structs\n├── sink.rs Sink lifecycle, Iggy consumer wiring, FFI consume calls\n├── source.rs Source lifecycle, flume forwarding, state save loop\n├── stream.rs Stream + consumer/producer setup\n├── transform.rs Loads transforms from config, applies them in chain\n├── state.rs FileStateProvider, atomic ConnectorState file I/O\n├── context.rs Per-instance context (Iggy client, metrics, config)\n├── error.rs RuntimeError with as_code() for HTTP responses\n├── log.rs init_logging (text/JSON x telemetry-on/off matrix) + LOG_CALLBACK\n├── benchmark.rs emit_sink_event / emit_source_event + as_micros\n├── metrics.rs Prometheus families + gauges + stage histograms + label caches\n├── stats.rs /stats endpoint payload assembly\n├── manager/{mod,sink,source}.rs SinkManager / SourceManager, status, restart_guard\n├── configs/\n│ ├── runtime.rs RuntimeConfig + LoggingConfig + LogFormat\n│ ├── connectors.rs ConnectorConfig, SinkConfig, SourceConfig (verbose + benchmark)\n│ └── connectors/{local,http}_provider.rs\n└── api/ HTTP control + observability endpoints\n```\n\n## Architecture invariants\n\nLoad-bearing assumptions. Breaking them silently corrupts data or hangs.\n\n### Plugin instance identity\n\n- Each instance gets a globally unique `plugin_id: u32` from `PLUGIN_ID: AtomicU32` in `main.rs`. Never reused.\n- Same `.so` may load many instances. each gets a fresh `plugin_id` but shares the `Container` (dlopen handle) via `Arc` (`manager/{sink,source}.rs`).\n- All FFI calls keyed by `plugin_id`. Mixing IDs = data to the wrong plugin.\n\n### FFI pointer lifetimes\n\n- Config and state are `(*const u8, usize)`. Valid for the call duration only. Plugin must copy if it needs the bytes past the call. SDK macros do this correctly. don't bypass.\n- `LogCallback` lifetime spans the whole process - kept static in `log.rs`. Don't make it dynamic.\n\n### Container ownership\n\n- `dlopen2::wrapper::Container<{Sink,Source}Api>` (defined in `main.rs`) is shared via `Arc` across all instances of the same `.so`.\n- Arc must outlive every spawned task that calls into the plugin. Manager holds the Arc in `{Sink,Source}Details` for the connector lifetime.\n- Unloading a `.so` while a task is mid-FFI-call would segfault. Runtime does not unload. Hot reload needs a rearchitect.\n\n### Plugin manager state\n\n- `SinkManager` / `SourceManager` use `DashMap>>`. Key = TOML `key` field.\n- Status transitions (`Starting → Running → Stopped/Error`) update running counters only on transitions to/from `Running`. See `SinkManager::update_status`.\n- `restart_guard: Arc>` prevents concurrent restart of the same connector. `restart_connector()` uses `try_lock()` - returns OK without restart if busy. Intentional anti-thundering-herd.\n\n### Serialization split\n\n- FFI message payloads (`TopicMetadata`, `MessagesMetadata`, `RawMessages`, headers): `postcard` (compact, stable wire).\n- Connector configs passed to plugin: `serde_json` (TOML in, JSON over FFI - human-editable).\n- `ConnectorState` bytes inside plugins: `rmp_serde` (MessagePack - compact, opaque to runtime).\n\nDon't mix.\n\n## State storage (`state.rs`)\n\n- File path: `{state_path}/source_{connector_key}.state`. Mode `0o600` on Unix.\n- `save()` is crash-atomic. `rename(2)` is atomic in the namespace (no observer sees a half-renamed file). the preceding `sync_data` on the tmp file plus the post-rename parent-dir `sync_all` are what make a crash leave either the old or new content - never truncated. The parent-dir fsync failure is propagated as `CannotWriteStateFile` (not swallowed) so a lost rename surfaces instead of silently restarting the source from scratch. A `Mutex<()>` serializes concurrent saves on the same provider.\n- Save after every successful Apache Iggy send. Save failure logs + continues. next batch retries.\n- `load()` returns `Ok(None)` for missing or empty files. After a `NotFound` read, re-stats the parent directory: missing parent -> `Err(CannotOpenStateFile)` so a broken state path fails at init rather than masquerading as \"fresh start\".\n- Sinks have no state - only sources.\n\n## Source forwarding loop\n\n1. `iggy_source_handle(id, send_callback)` - plugin registers itself.\n2. Plugin polls + invokes `send_callback(plugin_id, ptr, len)`.\n3. Callback runs in the SDK macro's spawned async task. Pushes postcard `ProducedMessages` into a `flume` channel keyed by `plugin_id` in `SOURCE_SENDERS: Lazy>` (`pub(crate)`). `SourceSenderEntry` wraps the sender + a pre-extracted owned `Counter` (the `errors` series, `Arc` inside). The FFI callback bumps errors on deserialize or channel-closed failure with one relaxed atomic - no `Family` lookup, no `Arc` handle.\n4. The runtime registers the optional source-stop callback, then `source_forwarding_loop` pulls from the channel, deserializes, applies transforms, encodes via `StreamEncoder`, and sends to the Iggy producer. The SDK warns after 30 seconds without a result but keeps the same batch pending; it does not poll or enqueue another copy.\n5. On success, save returned `ConnectorState` via `FileStateProvider`, then return the batch result to the plugin. A result NACK may cause replay with capped backoff.\n6. A stop-callback reason ends the forwarding loop and reports an unexpected stop. Closing the stop channel on normal shutdown disables that select arm, allowing already queued batches to receive results before the forwarding channel disconnects.\n7. At loop exit, clean up `SOURCE_SENDERS` and update status and metrics. A self-stop reports `Error` and removes the source from the running gauge; normal shutdown reports `Stopped`.\n\n**Shutdown ordering (`manager/source.rs::stop_connector`):**\n\n1. Call `iggy_source_close` FIRST. It blocks until the plugin's polling task stops, so no new send callbacks fire after it returns.\n2. `cleanup_sender(plugin_id)` NEXT - dropping the channel sender lets the forwarding task finish any queued batches before `recv_async()` resolves with `Disconnected` and exits cleanly, instead of blocking until the abort timeout.\n3. Finally await spawned handlers with `tokio::time::timeout`. On timeout, `handle.abort()` + drain - prevents leaked tasks colliding with the next `start_connector` (a late `file.save()` could otherwise race the new instance). The silent-drop branch in `handle_produced_messages` only covers the window between close and cleanup.\n\nGotchas:\n\n- `SOURCE_SENDERS` must be cleaned up on connector close or memory leaks (channel + task).\n- `spawn_source_handler` wraps the outer `iggy_source_handle(id, callback)` FFI call in `tokio::task::spawn_blocking()`. That call returns quickly - the SDK macro internally `runtime.spawn`s an async `handle_messages` task and returns. **`send_callback` invocations come from that async task, not from `spawn_blocking`.** A long-running synchronous poll inside the plugin would block one Tokio worker. async polls don't.\n- No timeout on the registration call - a plugin whose `iggy_source_handle` never returns stalls one blocking worker for the process lifetime.\n\n## Sink consumption loop\n\nPer `[[streams]]` entry per sink:\n\n1. Build / ensure consumer group `iggy-connect-sink-{key}` (`connect` matches the docker image name `apache/iggy-connect` - see `runtime/src/sink.rs::default_consumer_group`). Overridable via the stream config's `consumer_group`.\n2. Spawn one task per topic (`spawn_consume_tasks`).\n3. Poll Iggy → batch messages (default `batch_length = 1000`, `poll_interval = 5ms`). A `consumer.next()` `Err` (transport/decode at the Iggy client boundary) bumps `errors` and `continue`s - offset already auto-committed via `AutoCommit::When(AutoCommitWhen::PollingMessages)`, message effectively dropped.\n4. Decode via `Schema::decoder()`. Decode failure logs + bumps `errors`. `messages_filtered` is reserved for the transform filter contract (`Ok(None)`).\n5. Apply transforms (chain). A transform `Err` is logged, bumps `errors`, and drops only that message (drop-and-continue, mirroring the source) - one bad payload never kills the batch. `Ok(None)` filters the message - bumps `messages_filtered{connector_type=\"sink\"}`.\n6. Postcard-encode batch as `RawMessages`. Headers as a separate postcard blob. Per-message failures (missing field, payload conversion, header serialization) bump `errors` + skip. Batch-level failures (postcard serialization of metadata / `RawMessages`) propagate `Err` -> `spawn_consume_tasks` wrapper bumps `errors` + `set_error`. The total histogram and benchmark emit still fire for the failed batch before the `Err` propagates.\n7. Call `iggy_sink_consume(id, topic_meta, messages_meta, messages)`.\n8. Non-zero return → log + `errors` increment.\n\nPer-message counters (decode errors, transform errors, filters, field/serialize drops) are accumulated and flushed once per batch via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels`, not one `Family` lookup per message.\n\nStages (both labels snake_case): `decode` (schema decode loop), `prepare` (transform + serialize = total - decode - ffi), `ffi` (plugin consume call), `total`. Same `decode` / `prepare` meaning as the source side.\n\nGotchas:\n\n- Batch flushes on `current_offset != message_offset` (gap) OR batch full - higher latency for low-volume topics.\n- Auto-commit happens on poll - idempotency at the sink protects against re-delivery on crash before plugin write.\n\n## Logging format\n\n`LoggingConfig` in `configs/runtime.rs` exposes `format: LogFormat` (`Text` default, `Json`), env-addressable via `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `log::init_logging` matches on `(telemetry.enabled, format)` and installs the right `fmt::layer()`/`fmt::layer().json()` + OpenTelemetry layer. OTel pipeline untouched by `format` - only the stdout layer switches.\n\n`fmt::layer()` and `fmt::layer().json()` are different concrete types, so the stdout layer is built once as a `Box>` (via `.boxed()`) by matching on `format`, then a single 2-arm branch on `telemetry.enabled` attaches the OTel layers. When extending, keep the boxed-layer + 2-arm shape rather than re-expanding into a 4-way match.\n\n## The `verbose` flag\n\n`SinkConfig` and `SourceConfig` in `configs/connectors.rs` carry `verbose: bool` (default false). Runtime threads it to `spawn_consume_tasks` / source equivalent to gate per-batch log promotion:\n\n```rust\nif verbose {\n info!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n} else {\n debug!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n}\n```\n\nNew per-batch logging follows the same pattern. Default to `debug!`, upgrade to `info!` when `verbose`. No third level.\n\n## Config provider rules\n\n### Local (`configs/connectors/local_provider.rs`)\n\n- Discovery walk reads `*.toml` from `config_dir`. Skips hidden files (`.`) and `Cargo.toml`.\n- Filename: `{key}_{type}[_v{N}].toml` (e.g., `postgres_sink_v2.toml`).\n- One TOML deserialized into `SinkConfig` or `SourceConfig`. Grouped by `key`. Highest `version` wins unless `.active_versions.toml` overrides.\n- `.active_versions.toml` is NOT read via the discovery walk (hidden). Read via direct path through `LocalConnectorsConfigProvider::active_versions_file_path`.\n\n### HTTP (`configs/connectors/http_provider.rs`)\n\n- `reqwest_middleware` + `RetryTransientMiddleware`. Retries 5xx + connection errors. Never 4xx (`PermanentHttpError` convention).\n- Default URL templates documented in `runtime/README.md`. Configurable via TOML.\n\n### Env-var overrides via `ConfigEnv` derive\n\n`SinkConfig`, `SourceConfig`, and inner structs derive `ConfigEnv` (`configs_derive::ConfigEnv`). Generates env-var addressability as `IGGY_CONNECTORS___` for primitive fields. Used heavily by integration tests to inject testcontainer ports - see `core/integration/tests/connectors/fixtures/postgres/container.rs` for env-var constants. Mark new compound fields `#[config_env(skip)]`, leaf primitives `#[config_env(leaf)]`.\n\n### Versioning\n\n- Configs carry `version: u64`. Local provider auto-increments if absent. HTTP provider trusts the server.\n- Restart on version change uses `restart_guard`.\n\n## Error categorization\n\n`RuntimeError` (`error.rs`) carries `as_code()` for HTTP API responses.\n\n| Class | Fatal? | Example |\n| ------------------ | --------------------------- | -------------------------------- |\n| Config load | yes (process exit) | Bad TOML, missing required field |\n| Iggy client init | yes | Auth failure at startup |\n| State dir create | yes | Permission denied |\n| Plugin .so resolve | per-plugin (`FailedPlugin`) | Missing file |\n| Plugin open FFI | per-plugin (status = Error) | Plugin returned non-zero |\n| Message decode | per-message (skip) | Bad protobuf bytes |\n| Iggy send | per-batch (metric) | Network blip |\n| State save | per-batch (log) | Disk full |\n\nFatal errors propagate to `main` and exit. Per-connector / per-message errors are isolated.\n\n## Metrics (`metrics.rs`)\n\nAll families labeled by `connector_key` + `connector_type` (histogram adds `stage`):\n\n- **Counters**: `iggy_connector_messages_{produced,sent,consumed,processed,filtered}_total` and `iggy_connector_errors_total`. These are the *rendered* names; each is registered without the `_total`, which the OpenMetrics encoder appends.\n - `messages_filtered_total` - intentional drops via transform `Ok(None)`.\n - `errors_total` - unexpected drops (decode/encode/build failure, missing field, ...) + batch-level failures, one per failed run.\n - `sink_runs_total` - FFI `consume()` calls for sink batches, one per contiguous payload-variant run. Divide by `stage_duration_seconds_count{stage=\"total\"}` for runs per batch.\n- **Histograms**: `iggy_connector_stage_duration_seconds{stage}` (snake_case stage labels - `prepare`, `ffi`, `decode`, `iggy_send`, `state_save`, `total`). Buckets `STAGE_BUCKETS_SECONDS`. Always populated regardless of any flag. One sample per batch for every stage; `ffi` is summed over the batch's runs. Scraped at `/metrics` when `[http.metrics] enabled = true`.\n- **Gauges**: `iggy_connectors_{sources,sinks}_{total,running}`.\n\n`/stats` JSON surface mirrors counters per-connector via `ConnectorStats` (`sdk/api.rs`): `messages_filtered`, `errors`, kind-specific counters.\n\nWhen adding a metric:\n\n- Add family to `Metrics` struct + `init`, register with name + help text. Never end a `Counter` family's registered name in `_total`: the encoder appends it and the series renders `_total_total`. Gauges get no suffix, so a gauge name may end in `_total` literally.\n- New label sets define `EncodeLabelSet` struct + label enum (hand-impl `EncodeLabelValue` for snake_case values - the derive emits PascalCase).\n- Histograms: pass `fn() -> Histogram` to `Family::new_with_constructor`.\n- Add unit tests under `mod tests` with `given_*_when_*_should_*` BDD names.\n\n## Drop accounting (wired sites)\n\nPer-message drops in the batch loops are counted into a local `u64` and flushed once after the loop via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels` - one `Family` lookup per batch, not per message. One-shot drops outside a loop (e.g. `consumer.next()` Err) still call `inc_errors_with_labels` directly. Hot path uses pre-built `SinkLabels`/`SourceLabels`. `&str`-based wrappers are `#[cfg(test)]`-only.\n\n- `sink.rs::consume_messages` - `consumer.next()` Err\n- `sink.rs::process_messages` - decode, transform Err (drop-and-continue), missing required fields, payload conversion, header serialization (accumulated, flushed once per batch)\n- `sink.rs::spawn_consume_tasks` task wrapper - bumps once on `consume_messages` Err\n- `source.rs::source_forwarding_loop` - payload decode, prepare (transform/encode) failure, Iggy send Err, state save Err\n- `source.rs::process_messages` - transform Err (logs + bumps `errors` + continue. does NOT propagate, so one bad payload doesn't flip the connector to permanent ERROR), transform encode failure, `build_iggy_message` failure\n- `source.rs::handle_produced_messages` - postcard deserialize failure, `sender.send` channel-closed\n\nFilter case bumps `messages_filtered` via `inc_messages_filtered_with_labels`. Adding a new drop path: mirror this pattern.\n\n## Hard rules\n\n1. **Never block the executor.** All I/O async. `spawn_blocking` only for FFI registration (already in place).\n2. **Plugin ID counter is monotonic.** No reset, no reuse.\n3. **FFI return codes:** `0` success, non-zero failure.\n4. **Don't add static mutable state** beyond `LOG_CALLBACK`, `PLUGIN_ID`, `SOURCE_SENDERS`.\n5. **Pair `cleanup_sender(id)` with shutdown** for sources (avoid flume leak). Order: close FFI -> cleanup sender -> drain/abort tasks.\n6. **Restart uses `restart_guard.try_lock()`** - no thundering-herd regression.\n7. **No timeouts on plugin FFI calls without a kill-task strategy.** A timeout that returns from the runtime but leaves the plugin running has the worst of both worlds.\n\n## Common pitfalls\n\n1. Mutating `INSTANCES` (in SDK macro) from runtime side - impossible by design. only the macro touches it.\n2. Calling `iggy_*_open` with a duplicate ID - returns `-1`. Means runtime forgot to `iggy_*_close` first.\n3. Holding a `DashMap` shard guard across `.await` - lockups. Pattern: `.get().clone()`, drop guard, await.\n4. Adding fields to `#[repr(C)]` types without an SDK version bump - existing plugins misalign on `postcard::from_bytes`.\n5. Changing the default consumer group name without coordinating with operators - they have offsets stored there.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy-connectors --all-targets -- -D warnings\ncargo test -p iggy-connectors\n\n# Sanity-build in-tree plugins:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n\n# Runtime-focused integration tests:\ncargo test -p integration -- connectors::runtime::\n\n# Smoke-test with example config:\nIGGY_CONNECTORS_CONFIG_PATH=core/connectors/runtime/example_config/config.toml \\\n cargo run --bin iggy-connectors\n```\n\nUpdate `runtime/README.md` if endpoints, env vars, or config schema change.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connectors-overview/SKILL.md":"---\nname: connectors-overview\ndescription: Entry point and index for the Apache Iggy Connectors subsystem (`core/connectors/`). Load this first when working anywhere under `core/connectors/` - runtime, SDK, sinks, sources, or transforms. Routes to the focused per-area skills.\n---\n\n# Apache Iggy Connectors - Overview\n\nRepo-wide rules (Apache headers, fmt/sort/clippy order, idiomatic Rust traits, imports at top, no `cargo install`, LLM-slop tells) live in [AGENTS.md](../../../AGENTS.md). This file owns **connector-specific** rules and routes to the per-area skills.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [What sinks and sources are](#what-sinks-and-sources-are)\n- [Which skill to load](#which-skill-to-load)\n- [Stick to conventions](#stick-to-conventions)\n- [Connector-wide rules](#connector-wide-rules)\n- [Benchmark mode (per-batch timing observability)](#benchmark-mode-per-batch-timing-observability)\n- [Logging format](#logging-format)\n- [Exemplars](#exemplars)\n- [Concrete efficiency patterns](#concrete-efficiency-patterns)\n- [Drop accounting](#drop-accounting)\n- [Common review smells](#common-review-smells)\n- [File map](#file-map)\n- [Cite by symbol, not line number](#cite-by-symbol-not-line-number)\n\n## STOP and ask the user before\n\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI signature in `sdk/src/{sink,source}.rs` - breaks every pre-built plugin `.so`.\n- Changing the runtime's wire conventions (postcard FFI payload structs, default consumer group naming, plugin path resolution).\n- Modifying `runtime/src/state.rs` save protocol (atomic rename + fsync ordering) - corruption risk.\n- Renaming or repurposing a `Schema` variant - decoders/encoders pinned to wire bytes.\n- Promoting a transient `Error` variant to `Permanent*` or vice versa - affects retry behavior across every plugin.\n\n## What sinks and sources are\n\nA **sink** is a plugin that **consumes** messages from Apache Iggy streams and writes them to an external system (Postgres, Elasticsearch, Iceberg, HTTP endpoint, Mongo, object store, stdout, ...). Implements `iggy_connector_sdk::Sink`.\n\nA **source** is a plugin that **produces** messages from an external system into Apache Iggy streams (poll a DB table, scroll an ES index, generate random data, ...). Implements `iggy_connector_sdk::Source`.\n\nBoth compile as `cdylib` shared libraries (`.so`/`.dylib`/`.dll`) loaded by the **connectors runtime** at startup via `dlopen`. The runtime drives lifecycle (open/handle/consume/close), bridges Apache Iggy ⇄ plugin via a small FFI, applies optional transforms, and persists source state.\n\n```text\n ┌─ optional transforms ─┐\nExternal ──poll──▶ SOURCE ──FFI──▶ RUNTIME ──encode──▶ Apache Iggy stream\n system plugin ▲ ▲\n │ │\n state save (msgpack) │\n │\n ┌─ optional transforms ─┐ │\nApache Iggy stream ──decode──▶ RUNTIME ──FFI──▶ SINK ──write──▶ External\n plugin system\n```\n\nHeaders set on the source side ride through transforms (which may modify, drop, or pass them) and arrive at the sink with `BTreeMap` preserved deterministically.\n\n## Which skill to load\n\n| Task | Skill |\n| --------------------------------------------------- | --------------------- |\n| Write a new sink plugin | `connector-sink` |\n| Write a new source plugin | `connector-source` |\n| Add schema / decoder / encoder / SDK trait surface | `connector-sdk` |\n| Change runtime internals (FFI, manager, state, ...) | `connector-runtime` |\n| Add a transform (field-level or format conversion) | `connector-transform` |\n| Write unit / integration tests for any of the above | `connector-testing` |\n\n## Stick to conventions\n\nThe connectors codebase is intentionally repetitive across plugins. Cross-plugin consistency makes review, LLM-assisted contribution, and onboarding tractable.\n\n1. **Find the closest existing plugin by shape** (DB-write -> `postgres_sink`, HTTP -> `http_sink`, polling source -> `postgres_source`).\n2. **Copy its file layout, naming, log format, error mapping, test structure** verbatim. Only the backend-specific code (the client call) should differ.\n3. **Don't reinvent existing patterns.** The SDK's `retry.rs` covers HTTP retry + circuit breaker. Existing plugins cover batching, idempotency, header encoding, secret handling.\n4. **Don't refactor unrelated code in the same PR.** Match conventions. propose convention changes separately.\n\n## Connector-wide rules\n\n### Hot loops\n\n- **Zero clone.** Use `&self`, `&str`, `&[T]`, `&Payload`. For `Payload::Json`, `try_to_bytes(&self)` serializes single-pass without cloning the `OwnedValue` tree.\n- **Move-out without cloning** via `std::mem::replace` / `std::mem::take`.\n- **`Vec::with_capacity(n)`** when building per-batch buffers. reallocation in hot path is wasted work.\n\n### Async + concurrency\n\n- **`async fn` only.** Never `.block_on()` inside a `Sink`/`Source` impl - the runtime drives the executor.\n- **Hold locks briefly.** Pattern: lock, clone/read what's needed, drop guard, then do I/O. Never hold a `Mutex` across an external `.await`.\n- **Atomic counters** (`AtomicU64`) preferred over `Mutex` for hot-path metrics.\n- **No `tokio::spawn` from plugin code.** Runtime owns lifecycle. orphans survive `close()`.\n\n### Logging\n\n- `tracing` only. `info!` for lifecycle, `debug!` per-batch detail, `warn!` recoverable, `error!` failures. Include `connector ID: {self.id}` and the connector name on every line.\n- **`verbose: bool`** on `SinkConfig`/`SourceConfig` (TOML top-level). Runtime upgrades its own per-batch `debug!` to `info!` when set. Plugins should mirror with a `verbose_logging: Option` field inside `plugin_config` (convention: `postgres_sink::PostgresSinkConfig::verbose_logging`). Default both to `false`.\n- **Never log secrets** (connection strings, API keys, tokens). Redact URLs at the log site. Credential fields must use `SecretString` (see below).\n\n### Secrets\n\nAny credential-bearing field (connection strings, API keys, bearer tokens, AWS keys) must be `SecretString` from the `secrecy` crate. Plain `String` for a credential is a review-blocker: `SecretString` redacts on `Debug`, so it is what keeps a credential out of a log line that formats the whole config.\n\n**`serde_secret::serialize_secret` EXPOSES the secret. It does not redact.** It calls `expose_secret()` and writes the plaintext. `SecretString` deliberately has no `Serialize` impl, and that absence is the protection - so adding `serialize_with` is what *unblocks* the derive and turns a compile-time guarantee into plaintext output. Use it only where the plaintext is the point: a wire payload, a persisted config, an API response that exposes credentials by design.\n\nSo the default for a plugin config struct is **derive `Deserialize`, but not `Serialize`**. `Deserialize` is required: the SDK glue deserializes the config into the plugin's own struct (`sdk/src/{sink,source}.rs` call `serde_json::from_str::` under a `DeserializeOwned` bound).\n\nWhat never happens is the return trip. The runtime holds plugin configuration as a `serde_json::Value` - parsed from TOML, posted as JSON to the control API, or injected by env var - and hands that across the FFI, so nothing re-serializes the plugin's struct. Leaving `Serialize` off makes that compiler-enforced instead of convention-enforced (`sources/http_source/src/lib.rs::HttpSourceConfig` does this, and comments the omission so nobody adds it back).\n\nPattern:\n\n```rust\nuse secrecy::{ExposeSecret, SecretString};\n\n// `Deserialize` only. Nothing re-serializes a plugin config, and leaving\n// `Serialize` off is what makes the credential unserializable rather than\n// merely un-serialized.\n#[derive(Debug, Clone, Deserialize)]\npub struct MyConfig {\n pub connection_string: SecretString,\n}\n\n// At call site:\nlet pool = PgPoolOptions::new()\n .connect(self.config.connection_string.expose_secret())\n .await?;\n```\n\nIf a config struct genuinely needs `Serialize`, `serde_secret::serialize_redacted` (and `serialize_optional_redacted`) write `[REDACTED]` in place of the value. Reach for `serialize_secret` only when the caller must get the real thing back. The sinks and sources listed below predate that helper and use the exposing one; the annotation is inert today, but it is not the protection it looks like.\n\nNote that none of this protects the credential from the runtime's own control API, which returns plugin configuration verbatim - see #3802. Plugin-side annotations are inert there because the runtime never routes through them.\n\nPlugin-side uses of the exposing helpers: `sinks/{postgres,mongodb,elasticsearch,influxdb,s3,surrealdb}_sink`, `sources/{postgres,elasticsearch,influxdb}_source`.\n\nThat list is plugin-side only, not an inventory of every caller in the tree, and the others are not all mistakes: `runtime/src/api/config.rs` puts `serialize_secret` on `HttpConfig::api_key` (inert for the same reason), and several `core/common` wire-payload types (login, create-user, change-password, PAT) use these helpers by design, because there the credential *is* the payload.\n\n### Errors\n\n- **Use `Error::PermanentHttpError` / `Error::SchemaMismatch` / `Error::CatalogCommitError` for non-retryable failures.** Returning a transient variant for bad data trips circuit breakers and hammers the backend.\n- **No `unwrap()` / `expect()`** on Results from external I/O outside tests.\n\n### Forward-compat config\n\nNew TOML fields use `#[serde(default)]` or `Option`. Adding a field must not break existing configs.\n\n## Benchmark mode (per-batch timing observability)\n\nEvery sink and source supports an opt-in `benchmark: bool` flag (`SinkConfig`/`SourceConfig`, also env `IGGY_CONNECTORS___BENCHMARK=true`). Two independent surfaces:\n\n1. **Tracing events** (only when flag is on) - one structured `info!` per batch under target `iggy_connectors::benchmark` with per-stage microsecond timings:\n - Sink: `connector_type, connector_key, stream, topic, partition_id, current_offset, batch_size, processed_count, decode_us, prepare_us, ffi_us, total_us`\n - Source: `connector_type, connector_key, stream, topic, batch_size, sent_count, decode_us, prepare_us, iggy_send_us, state_saved, state_save_us, total_us` (`state_save_us` is 0 when `state_saved` is false - emitted as flat number + bool, not an Option, so the JSON layer keeps it numeric)\n - Filter live: `RUST_LOG=iggy_connectors::benchmark=info`\n - Also emitted on startup: a one-shot `\"Benchmark mode enabled for ...\"` info log per connector.\n\n2. **Prometheus histograms** (always on, regardless of flag):\n - Metric: `iggy_connector_stage_duration_seconds{connector_key, connector_type, stage}` (both labels snake_case)\n - `stage` (snake_case): `decode`, `prepare`, `total` on both sides; `ffi` (sink only); `iggy_send`, `state_save` (source only). `decode` and `prepare` mean the same thing on both sides - decode is schema decoding, prepare is transform + encode/serialize.\n - Buckets: see `runtime/src/metrics.rs::STAGE_BUCKETS_SECONDS`\n - Example: `histogram_quantile(0.95, sum(rate(iggy_connector_stage_duration_seconds_bucket{stage=\"ffi\"}[5m])) by (le, connector_key))`\n\nThe flag only gates the verbose text event. the histogram observation is always on. Per-stage call path: `Instant::now()` + SipHash over `StageLabels` + parking_lot `RwLock::read` + `HashMap` probe + linear scan inside `Histogram::observe`. Benchmark on target hardware if you suspect the overhead matters at your batch rate.\n\n`iggy_connector_messages_filtered_total{connector_key, connector_type}` counts intentional drops by transforms returning `Ok(None)`, distinct from `errors_total` for unexpected drops. Both also exposed in `/stats` JSON (`ConnectorStats.messages_filtered`, `.errors`).\n\nImplementation: `runtime/src/benchmark.rs` (text-event emitters +\n`as_micros`), `runtime/src/metrics.rs::{Stage, SinkLabels, SourceLabels,\nobserve_stage_with_labels}` (pre-built label cache avoids per-batch\n`String` clones. `&str`-based wrappers are `#[cfg(test)]` only). Tests:\n`runtime/src/metrics.rs::tests::given_stage_histogram_*` (unit),\n`core/integration/tests/connectors/runtime/benchmark.rs` (integration:\ntext emission, env override, disabled-flag, histograms via `/metrics`,\nJSON log format, parser unit tests).\n\n## Logging format\n\n`runtime/src/configs/runtime.rs::LoggingConfig` exposes `format = \"text\" | \"json\"` (text default), env `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `runtime/src/log.rs::init_logging` matches on `(telemetry.enabled, format)` to install the right `fmt::layer()`/`fmt::layer().json()` combined with the OpenTelemetry layer when telemetry is on. JSON applies to the stdout layer only - OTel export pipeline unaffected by this flag.\n\n## Exemplars\n\n| Purpose | Plugin |\n| --------------------------------------------------------- | --------------------------------------------------------------------------------------- |\n| Simplest sink (read first) | `sinks/stdout_sink/` |\n| Real-infra sink + integration tests | `sinks/postgres_sink/` + `integration/tests/connectors/postgres/postgres_sink.rs` |\n| Feature-rich sink config (validation, batch modes, retry) | `sinks/http_sink/` |\n| Atomic counters on hot path | `sinks/mongodb_sink/` |\n| Simplest source (4 canonical state tests) | `sources/random_source/` |\n| Real-infra source | `sources/postgres_source/` + `integration/tests/connectors/postgres/postgres_source.rs` |\n\nRead the relevant exemplar end-to-end before writing or modifying a connector.\n\n## Concrete efficiency patterns\n\nEach implemented in at least one in-tree plugin or runtime path.\n\n| Pattern | Where | When |\n| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- |\n| `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))` | `sinks/http_sink/src/lib.rs` (`send_individual`, `send_ndjson`, `send_json_array`) | Take payload from `&mut ConsumedMessage` no clone |\n| `std::mem::take(&mut batch)` | `runtime/src/sink.rs::consume_messages`, `manager/{sink,source}.rs` | Drain a `Vec` field with one move |\n| `std::mem::swap` | `sinks/http_sink` (retry_delay / max_retry_delay validation) | Fix config-field ordering in `new()` |\n| `Vec::with_capacity(messages.len())` | Every batching sink | Pre-size per-batch buffers |\n| `bytes::Bytes::from(body)` | `sinks/{http,influxdb}_sink` | Cheap ref-counted body sharing across retries |\n| `AtomicU64` counters | `sinks/mongodb_sink` | Lock-free per-message metric updates |\n| `Payload::try_to_bytes(&self)` | `sdk/src/lib.rs::Payload::try_to_bytes` | Serialize `Payload::Json` no-clone |\n| `simd_json::to_owned_value(&mut bytes)` | `sdk/src/decoders/json.rs::JsonStreamDecoder::decode` | In-place JSON parse |\n| Brief-lock fetch | `sources/postgres_source/src/lib.rs::poll_tables` | Lock read -> drop -> I/O -> lock write |\n| `is_transient_error(&e)` SQLSTATE mapping | `sinks/postgres_sink/src/lib.rs` | Driver errors -> retry vs `PermanentHttpError` |\n| Two-constructor pattern (`new` lenient + `try_new` strict + `Default`) | `sdk/src/decoders/avro.rs::AvroStreamDecoder` | Stateful decoders/encoders where schema can fail |\n| Duplicate-ID FFI guard | `sdk/src/sink.rs::sink_connector!`, `sdk/src/source.rs::source_connector!` | Prevents silent data loss on reopen-without-close |\n| `restart_guard.try_lock()` | `runtime/src/manager/{sink,source}.rs::restart_connector` | No thundering-herd restarts |\n| `tokio::time::timeout(..., handle).await` + `handle.abort()` on timeout | `runtime/src/manager/source.rs::stop_connector` | Bounded shutdown + leak prevention |\n| `flume::unbounded()` channel | `runtime/src/source.rs::spawn_source_handler` / `source_forwarding_loop` | MPSC handoff from SDK async task to runtime loop |\n| `tokio::sync::watch::channel(())` | `sdk/src/{sink,source}.rs`, `runtime/src/sink.rs`, `runtime/src/manager/*` | One-shot shutdown broadcast |\n| `dashmap::DashMap` | `runtime/src/manager/sink.rs`, `source.rs::SOURCE_SENDERS`, SDK `INSTANCES` | Lock-free concurrent keyed access |\n| `secrecy::SecretString` + `iggy_common::serde_secret::serialize_secret` | `sinks/postgres_sink::PostgresSinkConfig::connection_string` | `Debug` redacts; `serialize_secret` EXPOSES |\n\n## Drop accounting\n\n- **Filter drops** (transform returning `Ok(None)`) bump `messages_filtered`.\n- **Every non-filter drop** bumps `errors`. Inside the per-batch loops the count is accumulated in a local and flushed once after the loop via `inc_errors_by_with_labels` (filters likewise via `inc_messages_filtered_with_labels`) - one `Family` lookup per batch, not per message. One-shot drops outside a loop call `inc_errors_with_labels` directly. Hot-path calls always use the pre-built `SinkLabels`/`SourceLabels` cache. `&str`-based wrappers on `Metrics` are `#[cfg(test)]`-only.\n\n## Common review smells\n\n- `.clone()` on `Payload::Json` or `OwnedValue` - use `try_to_bytes(&self)`.\n- `&mut self` on `Sink::consume` or `Source::poll` impls - won't compile, flag any creative workaround.\n- `std::sync::Mutex` held across `.await` - swap for `tokio::sync::Mutex`.\n- Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` in plugin `Cargo.toml`.\n- Source plugin without the four canonical state tests (see `connector-testing`).\n- New silent message drop without a metric increment.\n- Wrapping `format!()` around args passed to `error!`/`warn!`/`info!`/`debug!` - eager `format!` allocates even when level filters the line out. Pass args directly: `error!(\"foo: {x}\")` or `error!(error = %x, \"foo\")`.\n- Logging a connection string, API key, or token.\n- Plain `String` for a credential field - use `SecretString`.\n- `tokio::spawn` inside plugin code - runtime owns lifecycle.\n- `std::time::SystemTime::now()` in transforms - non-deterministic, breaks tests.\n\n## File map\n\n```text\ncore/connectors/\n├── README.md\n├── runtime/\n│ ├── src/\n│ │ ├── main.rs FFI structs (SinkApi, SourceApi), plugin path resolution\n│ │ ├── manager/{sink,source}.rs Lifecycle, restart_guard, status transitions\n│ │ ├── configs/ Local + HTTP config providers, ConfigEnv derive\n│ │ ├── {sink,source,stream,transform,state,context}.rs\n│ │ ├── benchmark.rs, log.rs, metrics.rs\n│ │ └── api/ HTTP control plane (/sinks, /sources, /stats, /metrics)\n│ └── example_config/ Reference TOML\n├── sdk/\n│ └── src/\n│ ├── lib.rs Sink/Source traits, Payload, Schema, Error\n│ ├── {sink,source}.rs FFI containers + sink_connector!/source_connector! macros\n│ ├── decoders/encoders/ Per-schema (json/raw/text/proto/flatbuffer/avro)\n│ ├── transforms/ add/delete/update/filter fields, format conversions\n│ └── retry.rs CircuitBreaker, HttpRetryMiddleware\n├── sinks/_sink/\n└── sources/_source/\n\ncore/integration/tests/connectors/ # Real-infra integration tests\n├── fixtures/ # testcontainers-modules wrappers per backend\n├── postgres/, elasticsearch/, mongodb/, iceberg/, influxdb/, quickwit/, http/, delta/\n├── runtime/ # error_isolation, benchmark\n└── api/ # HTTP API endpoint tests\n```\n\n## Cite by symbol, not line number\n\nSkills cite files by path + symbol (`PostgresContainer::start` in `core/integration/tests/connectors/fixtures/postgres/container.rs`). Line numbers go stale on the first refactor. symbol names survive. Grep the symbol if you need the exact spot.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-testing/SKILL.md":"---\nname: connector-testing\ndescription: Write tests for Apache Iggy connectors - sinks, sources, transforms, SDK, runtime. Covers BDD unit-test naming, the canonical source state round-trip tests, and the real-infra integration test layout under `core/integration/tests/connectors/` with `testcontainers-modules` + `#[iggy_harness]`. Load when writing or reviewing tests in `core/connectors/` or `core/integration/tests/connectors/`. Use for connector test authoring. NOT for runtime internals or non-connector test patterns.\n---\n\n# Testing Apache Iggy Connectors\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). Repo-wide testing rules (`#[iggy_harness]`, BDD naming, harness layout) live in [AGENTS.md](../../../AGENTS.md#testing). This skill covers connector-specific test conventions.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Unit tests](#unit-tests)\n- [Integration tests (real infra via Docker)](#integration-tests-real-infra-via-docker)\n- [SDK + runtime tests](#sdk--runtime-tests)\n- [CI / verification flow](#ci--verification-flow)\n- [Common pitfalls](#common-pitfalls)\n- [Examples worth copying](#examples-worth-copying)\n\n## STOP and ask the user before\n\n- Mocking the database or backend in an integration test - we hit real infra via `testcontainers-modules` (mocks miss migration bugs).\n- Adding a unit test that requires Docker - that belongs under `core/integration/tests/connectors//`, not in `/src/lib.rs`.\n- Skipping the four canonical source state tests - they're mandatory for every source plugin.\n- Naming a test `test_foo` / `does_bar` - we use BDD `given_X_when_Y_should_Z`.\n\nThere are **two distinct test layers**. Use both - they catch different bugs.\n\n| Layer | Location | What it covers | When required |\n| --------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------- |\n| **Unit tests** | `#[cfg(test)] mod tests` at EOF of `/src/lib.rs` | Config parsing, defaults, fallbacks, payload conversion, query/document building, state ser/de | Every plugin |\n| **Integration tests** | `core/integration/tests/connectors//` (uses real backend via Docker) | End-to-end: runtime + plugin + real Postgres/ES/Iceberg/Mongo/etc. + Iggy server | Every plugin that integrates with an external system |\n\n## Unit tests\n\n### Naming - BDD `given_when_should`\n\nMatch the convention used in the codebase (`sources/random_source/src/lib.rs::tests`, `sinks/postgres_sink/src/lib.rs::tests`):\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_messages_produced() { ... }\n\n#[test]\nfn given_no_state_should_start_fresh() { ... }\n\n#[test]\nfn given_invalid_duration_should_fall_back_to_default() { ... }\n\n#[test]\nfn given_payload_json_should_serialize_correctly() { ... }\n```\n\nYou may **omit one part** when the test is small (`given_X_should_Y` instead of full `given_X_when_Y_should_Z`), but **stay consistent within a file**. Imperative `test_foo` / `does_bar` names are not the convention here.\n\nA test whose subject is a **type** rather than a scenario has no \"given\" to name, and drops the prefix: it reads `_should_`. The canonical state round trip below is `state_should_be_serializable_and_deserializable` in all three shipped sources (`random_source`, `postgres_source`, `http_source`). Leave those as they are rather than bending them into the scenario form.\n\n### Prove a test can fail, before trusting it\n\nA passing test says nothing until you have watched it fail. Break the behaviour it names, run the\nsuite, and put the code back:\n\n```bash\n# commit first: this reverts with `git checkout --`, which discards\n# every uncommitted change in the file, not only the mutation\ngit status --porcelain -- # must be empty\n\ncargo nextest run -p # expect a failure naming that test\ngit checkout -- \n```\n\nTwo failures this catches, both seen in review of `http_source`:\n\n- **A fixture that makes the interesting case unreachable.** A guard test for endpoint resurrection\n passed `&[]` as the static config, so there was nothing to resurrect and the assertion held\n whatever `restore` did.\n- **An assertion aimed at something self-healing.** A test for a teardown guard asserted on gauges,\n which `Metrics::encode` rebuilds from the live instances on every scrape, so it passed with the\n guard deleted. Delete such a test rather than keep it; a test that cannot fail is worse than none,\n because it reads as coverage.\n\nWatch for an arm hidden behind a catch-all: `StatusClass::from` ends in `_ => ServerError`, so the\nredirect arm could be removed with every existing assertion still green.\n\n### `test_config()` helper\n\nEvery plugin's tests start with a small helper returning a tuned-down version of the production config. New tests extend it. don't construct from scratch each time.\n\n```rust\nfn test_config() -> RandomSourceConfig {\n RandomSourceConfig {\n interval: Some(\"100ms\".to_string()),\n max_count: Some(100),\n messages_range: Some((5, 10)),\n payload_size: Some(50),\n }\n}\n```\n\n### Async testing pattern\n\nThe trait methods are `async`. For unit tests, build a runtime locally - don't add `#[tokio::test]` in `src/lib.rs` (it pulls test macros into prod build):\n\n```rust\n#[test]\nfn given_X_should_Y() {\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n // await calls here\n });\n}\n```\n\n### Mandatory: four canonical source state tests\n\nEvery source must have these. Copy from `sources/random_source/src/lib.rs::tests`:\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_() {\n let state = State { /* fields */ };\n let serialized = rmp_serde::to_vec(&state).expect(\"Failed to serialize state\");\n let connector_state = ConnectorState(serialized);\n\n let source = MySource::new(1, test_config(), Some(connector_state));\n\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n let restored = source.state.lock().await;\n assert_eq!(restored., /* expected */);\n });\n}\n\n#[test]\nfn given_no_state_should_start_fresh() {\n let source = MySource::new(1, test_config(), None);\n /* assert default state */\n}\n\n#[test]\nfn given_invalid_state_should_start_fresh() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n let source = MySource::new(1, test_config(), Some(invalid));\n /* assert default state */\n}\n\n#[test]\nfn state_should_be_serializable_and_deserializable() {\n let original = State { /* fields */ };\n let bytes = rmp_serde::to_vec(&original).unwrap();\n let restored: State = rmp_serde::from_slice(&bytes).unwrap();\n assert_eq!(original., restored.);\n}\n```\n\nState is the only thing that survives a plugin restart. Silent corruption here means lost data on the next deploy.\n\n**The third test has two correct shapes, and which one applies is a design decision rather than a style choice.**\n\nStarting fresh is right when the state is a cursor. The worst case is re-reading from the beginning, which the at-least-once contract already permits, so discarding an undecodable cursor costs duplicates and nothing else.\n\nIt is wrong when the state carries something whose *absence* is a downgrade rather than a repeat. `http_source` keeps its revocation tombstones in state, so starting fresh after a decode failure would serve every revoked endpoint again, with its secret. Its `restore` returns `Err` instead and the connector refuses to open. A source in that position asserts the refusal:\n\n```rust\n#[test]\nfn given_invalid_state_should_refuse_to_open() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n\n let restored = MyState::restore(&config_entries(), Some(invalid), 1);\n\n assert!(\n restored.is_err(),\n \"discarding this state silently would re-serve what it was revoking\"\n );\n}\n```\n\nName the test for whichever it is and say why in a comment. What is never right is starting fresh silently when the state that was discarded was load-bearing.\n\n### Sink unit tests\n\nCover pure per-message logic:\n\n- Config defaults + optional fields + invalid values → fallback.\n- Payload conversion for each `Payload` variant the sink claims to handle.\n- Header encoding (especially base64 for binary headers).\n- Query/document building (e.g., `postgres_sink::build_batch_insert_query`).\n- Transient vs permanent error classification (e.g., `postgres_sink::is_transient_error`).\n\n### Transform tests\n\nFour branches must be covered (see `connector-transform`):\n\n1. Happy path: matching input → expected output.\n2. Filter path: input that should be dropped returns `Ok(None)`.\n3. Pass-through path: non-JSON payload (or unmatched condition) returns `Ok(Some(unchanged))`.\n4. Invalid config: `new()` returns `Err(Error::InvalidConfigValue(_))`.\n\n### What NOT to mock\n\n- The underlying client lib (sqlx, reqwest, mongodb). Tests would exercise the mock, not the code.\n- The Iggy server.\n- The runtime.\n\nIf you need a fake backend, use `wiremock` (a real local HTTP server) - see `core/integration/tests/connectors/http/`.\n\n## Integration tests (real infra via Docker)\n\nThe canonical pattern lives in `core/integration/tests/connectors/`. Plugins that integrate with external services (Postgres, Elasticsearch, Iceberg, MongoDB, InfluxDB, Quickwit, HTTP/wiremock) have a paired fixture + test file there.\n\n### Layout\n\n```text\ncore/integration/tests/connectors/\n├── mod.rs Common helpers (TestMessage, create_test_messages)\n├── fixtures/ One subdir per backend\n│ ├── postgres/\n│ │ ├── container.rs PostgresContainer (testcontainers wrapper), env-var consts\n│ │ ├── sink.rs PostgresSinkFixture, PostgresSinkByteaFixture, PostgresSinkJsonFixture\n│ │ ├── source.rs PostgresSourceJsonFixture, PostgresSourceJsonbFixture, ...\n│ │ └── mod.rs re-exports\n│ ├── elasticsearch/ iceberg/ mongodb/ influxdb/ quickwit/ http/ delta/\n│ └── wiremock.rs HTTP mocking for http_sink tests\n├── postgres/\n│ ├── postgres_sink.rs The actual #[iggy_harness] tests\n│ ├── postgres_source.rs\n│ ├── restart.rs Restart + state-survival tests\n│ ├── sink.toml Runtime config used by sink tests\n│ ├── source.toml Runtime config used by source tests\n│ └── mod.rs Shared constants (TEST_MESSAGE_COUNT, POLL_ATTEMPTS, ...)\n├── delta/ elasticsearch/ iceberg/ mongodb/ influxdb/ random/ quickwit/ http/ stdout/\n├── http_config_provider/ HTTP config provider end-to-end\n├── runtime/ Runtime-only tests (error isolation, missing plugin, ...)\n└── api/ HTTP control plane tests\n```\n\n### Container naming + cleanup\n\nEvery test container MUST be named with the `iggy-test-` prefix so a single\n`docker ps -aqf 'name=^iggy-test-'` sweep reaps them (`just clean-test-containers`,\nalias `ctc`; the `test`/`nextest` recipes also reap on exit). New fixtures get\nthis for free by naming through `fixtures::unique_container_name(\"\")`, which\nappends a uuid for parallel-safe ephemeral containers. Reuse fixtures use a fixed\n`iggy-test-` literal instead, because the stable name is what lets later\nnextest processes attach to the same container via `ReuseDirective::Always`. Do\nnot hand-roll a different prefix - the sweep only finds `iggy-test-*`.\n\nAs of now only `elasticsearch` and `doris` share a container. Both have a slow\nboot (JVM / Doris FE+BE, tens of seconds), so reusing one across the whole\nregression beats spinning a fresh one per test. Every other fixture boots a\nfresh, ephemeral container per test - cheap enough that the isolation is worth\nmore than the reuse.\n\n### The `#[iggy_harness]` proc macro\n\nEach integration test is annotated with this macro from the `integration` crate. It boots an in-process Iggy server (and optionally the connectors runtime) for the duration of the test, runs the seeds, then injects a `&TestHarness` and your fixture.\n\n```rust\nuse integration::harness::seeds;\nuse integration::iggy_harness;\n\n#[iggy_harness(\n server(connectors_runtime(config_path = \"tests/connectors/postgres/sink.toml\")),\n seed = seeds::connector_stream\n)]\nasync fn json_messages_sink_stores_as_bytea(\n harness: &TestHarness,\n fixture: PostgresSinkFixture,\n) {\n let client = harness.root_client().await.unwrap();\n let pool = fixture.create_pool().await.expect(\"Failed to create pool\");\n fixture.wait_for_table(&pool, \"iggy_messages\").await;\n // ... send messages, assert on rows\n}\n```\n\nKey pieces:\n\n- `server(connectors_runtime(config_path = ...))` boots the connectors runtime against the given TOML.\n- `seed = seeds::connector_stream` runs the named seed to create the test stream/topic.\n- `fixture: PostgresSinkFixture` - the test harness calls `PostgresSinkFixture::setup()` (which spins up the testcontainer) and injects the result.\n- `harness: &TestHarness` (defined in `core/integration/src/harness/orchestrator/harness.rs`) exposes `.root_client().await`, `.connectors_runtime().expect(\"...\").http_url()`, etc. - `connectors_runtime()` returns `Option`, so `.expect()` first.\n\n### Fixture pattern (`TestFixture` trait)\n\nEach fixture implements `integration::harness::TestFixture`. Pattern:\n\n```rust\n// core/integration/tests/connectors/fixtures//container.rs\n\nuse testcontainers_modules::{\n ,\n testcontainers::{ContainerAsync, runners::AsyncRunner},\n};\n\npub struct MyContainer {\n container: ContainerAsync<::Image>,\n pub connection_string: String,\n}\n\nimpl MyContainer {\n pub async fn start() -> Result {\n let container = ::Image::default().start().await?;\n let port = container.get_host_port_ipv4(DEFAULT_PORT).await?;\n Ok(Self {\n container,\n connection_string: format!(\"...://localhost:{port}\"),\n })\n }\n}\n```\n\nThe fixture itself wraps the container and implements `TestFixture::setup()` to spin it up and inject env vars that the runtime picks up.\n\n### Env-var injection via `ConfigEnv`\n\nThe runtime's config structs derive `ConfigEnv` (`configs_derive::ConfigEnv`, applied in `runtime/src/configs/connectors.rs` to `SinkConfig`, `SourceConfig`, `StreamConsumerConfig`, `StreamProducerConfig`). Fields are addressable by env var with a path-built prefix. Three concrete forms observed in `core/integration/tests/connectors/fixtures/postgres/container.rs`:\n\n```rust\n// Plugin-config fields (your sink/source's own config struct):\n// IGGY_CONNECTORS___PLUGIN_CONFIG_\npub const ENV_SINK_CONNECTION_STRING: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PLUGIN_CONFIG_CONNECTION_STRING\";\n\n// Indexed nested fields (per-stream entries are a Vec):\n// IGGY_CONNECTORS___STREAMS__\npub const ENV_SINK_STREAMS_0_STREAM: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_STREAMS_0_STREAM\";\n\n// Top-level scalar fields on SinkConfig/SourceConfig itself:\n// IGGY_CONNECTORS___\npub const ENV_SINK_PATH: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PATH\";\n```\n\n`` is `SINK` or `SOURCE`. `` is the connector key (TOML key field, uppercased). `PLUGIN_CONFIG` is the literal segment used when reaching into your plugin's nested config. Fields marked `#[config_env(skip)]` (e.g., `transforms`, `plugin_config` as a whole) are NOT env-addressable. primitive leaves marked `#[config_env(leaf)]` are.\n\nThe fixture sets these in `setup()` before the runtime reads its config. This is how a dynamic container port reaches a static TOML.\n\n### Test naming for integration tests\n\nIntegration tests use a **descriptive declarative** style, not the unit-test `given_should` form. Match the codebase:\n\n```rust\nasync fn json_messages_sink_stores_as_bytea(...) { ... }\nasync fn binary_messages_sink_stores_as_bytea(...) { ... }\nasync fn json_messages_sink_stores_as_jsonb(...) { ... }\nasync fn delete_after_read_source_removes_rows_after_producing(...) { ... }\nasync fn processed_column_source_marks_rows_after_producing(...) { ... }\nasync fn state_persists_across_connector_restart(...) { ... }\nasync fn restart_sink_connector_continues_processing(...) { ... }\n```\n\nPattern: `__`. Reads like a sentence describing what the test proves.\n\n### Polling vs sleeping\n\nIntegration tests poll until an expected condition - never `sleep(big_duration)`. Constants `POLL_ATTEMPTS` and `POLL_INTERVAL_MS` defined per-backend `mod.rs`. Fixture helpers like `PostgresSinkFixture::fetch_rows_as` and `PostgresSinkFixture::wait_for_table` encapsulate the pattern.\n\n### When to add an integration test\n\n- New sink/source plugin → at minimum one happy-path test per supported `Schema` and per supported plugin mode. Concrete coverage in `core/integration/tests/connectors/postgres/`: 3 sink tests (`json_messages_sink_stores_as_bytea`, `binary_messages_sink_stores_as_bytea`, `json_messages_sink_stores_as_jsonb`) backed by 3 sink fixtures, plus 5 source fixtures (`Json`, `Jsonb`, `Bytea`, `Delete`, `Mark`) and a dedicated `restart.rs` for state-survival.\n- Bug fix in a plugin → regression test reproducing the bug.\n- Behavior that crosses the FFI boundary (state restart, transform chain, schema decode) → integration test, not unit test.\n\n### When NOT to add an integration test\n\n- Pure logic (query builder, payload converter) - that's a unit test.\n- Config validation - unit test.\n- Code paths already covered by an equivalent test for another backend, when the new backend has identical behavior.\n\n## SDK + runtime tests\n\n- `core/connectors/sdk/tests/` - cross-module SDK concerns. In tree: `protobuf_integration.rs`, `flatbuffer_integration.rs` (round-trip schema decoders/encoders).\n- Runtime tests live under `core/integration/tests/connectors/runtime/` and `core/integration/tests/connectors/api/`. They exercise the runtime alone (no plugin-specific backend) - error isolation, missing plugin, invalid config, HTTP API contract.\n\n## CI / verification flow\n\nBefore pushing:\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p --all-targets -- -D warnings\ncargo test -p \n```\n\nFor integration tests, the relevant package is `integration`. There is **no `--test connectors` target** - the `integration` crate has no `[[test]]` entries and `tests/` is structured as a single binary entry through `tests/mod.rs` with `connectors/`, `server/`, `cli/`, etc. as nested modules. Run via name filter:\n\n```text\ncargo test -p integration -- connectors::postgres::postgres_sink\ncargo test -p integration -- connectors::postgres:: # all postgres tests\ncargo test -p integration -- connectors:: # all connector tests\n```\n\nIntegration tests assume Docker is available locally and pull images from a public registry on first run.\n\n## Common pitfalls\n\n1. **Skipping integration tests for a real-infra plugin.** Unit tests don't catch FFI/lifecycle/encoding regressions.\n2. **Forgetting the four canonical source state tests.** This is the most common review comment.\n3. **Mocking the client library** instead of using `testcontainers` - test exercises the mock.\n4. **`#[tokio::test]` in `src/lib.rs`** - pulls tokio test macros into prod build. Use `Runtime::new()` instead. (`#[tokio::test]` is fine in `tests/` directories, which compile separately.)\n5. **`unwrap()` in test assertions** - prefer `assert_eq!` / `expect(\"descriptive\")` so failures tell you which step blew up.\n6. **Tests depending on system time** - inject the clock or use `tokio::time::pause()`.\n7. **Hidden coupling via shared globals** (static `AtomicU64`, env vars) - tests must be independent and parallelizable.\n8. **Asserting on `Display` of an error** - use `matches!(err, Error::Variant(_))`. `assert_eq!(err.to_string(), ...)` breaks on message tweaks.\n9. **Sleeping a fixed duration** instead of polling - causes flakes when CI is slow.\n\n## Examples worth copying\n\n| Concern | Where |\n| ----------------------------------- | -------------------------------------------------------------------- |\n| Source state round-trip | `sources/random_source/src/lib.rs::tests` |\n| Config + query-building unit tests | `sinks/postgres_sink/src/lib.rs::tests` |\n| Real-infra sink integration tests | `core/integration/tests/connectors/postgres/postgres_sink.rs` |\n| Real-infra source integration tests | `core/integration/tests/connectors/postgres/postgres_source.rs` |\n| Restart + state survival | `core/integration/tests/connectors/postgres/restart.rs` |\n| Wiremock-driven HTTP sink test | `core/integration/tests/connectors/http/http_sink.rs` |\n| Testcontainer wrapper | `core/integration/tests/connectors/fixtures/postgres/container.rs` |\n| Fixture impl `TestFixture` | `core/integration/tests/connectors/fixtures/postgres/sink.rs` |\n| Runtime-only error-isolation | `core/integration/tests/connectors/runtime/error_isolation.rs` |\n| Transform tests | `sdk/src/transforms/add_fields.rs::tests`, `filter_fields.rs::tests` |\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n",".claude/skills/connector-source/TEMPLATE.md":"# Source plugin skeleton\n\nBoilerplate for a new `core/connectors/sources/_source/`. Adapt\nthe `MySource` / `Client` / row types to the backend driver.\n\n`Cargo.toml` is identical to a sink's (see\n[connector-sink/TEMPLATE.md](../connector-sink/TEMPLATE.md)) - only\nthe crate name suffix changes (`iggy_connector__source`) and the\nupstream client dep.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, `impl Source` after,\nhelpers below.\n\n```rust\n/* Apache 2.0 header */\n\nuse std::str::FromStr;\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,\n source::SourceBatchResult, source_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse tokio::sync::Mutex;\nuse tokio::time::sleep;\nuse tracing::{debug, error, info, warn};\n\nsource_connector!(MySource);\n\nconst CONNECTOR_NAME: &str = \"My source\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySourceConfig {\n pub endpoint: String,\n pub poll_interval: Option, // humantime, \"10s\" default\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option,\n pub verbose_logging: Option,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct State {\n cursor: Option, // WAL LSN, scroll id, timestamp, ...\n last_offset: u64,\n messages_produced: u64,\n}\n\n#[derive(Debug)]\npub struct MySource {\n id: u32,\n config: MySourceConfig,\n poll_interval: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n pending: Mutex>,\n}\n\nimpl MySource {\n pub fn new(id: u32, config: MySourceConfig, state: Option) -> Self {\n let raw_interval = config.poll_interval.clone().unwrap_or_else(|| \"10s\".into());\n let poll_interval = humantime::Duration::from_str(&raw_interval)\n .map(|d| *d)\n .unwrap_or_else(|_| {\n warn!(\"Invalid poll_interval for {CONNECTOR_NAME} ID: {id}, defaulting to 10s\");\n Duration::from_secs(10)\n });\n\n let verbose = config.verbose_logging.unwrap_or(false);\n\n let restored = state\n .and_then(|s| s.deserialize::(CONNECTOR_NAME, id))\n .inspect(|s| info!(\n \"Restored state for {CONNECTOR_NAME} ID: {id}, last_offset: {}, cursor: {:?}\",\n s.last_offset, s.cursor\n ));\n\n Self {\n id,\n config,\n poll_interval,\n verbose,\n client: None,\n state: Mutex::new(restored.unwrap_or(State {\n cursor: None,\n last_offset: 0,\n messages_produced: 0,\n })),\n pending: Mutex::new(None),\n }\n }\n}\n\n#[async_trait]\nimpl Source for MySource {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn poll(&self) -> Result {\n sleep(self.poll_interval).await; // sleep first - backpressure\n\n let mut candidate = self.state.lock().await.clone();\n\n let fetched = self.fetch_since(candidate.cursor.as_deref()).await?;\n\n let mut messages = Vec::with_capacity(fetched.len());\n let mut next_cursor = None;\n for row in fetched {\n let payload = simd_json::to_vec(&row).map_err(|e|\n Error::Serialization(format!(\"row serialize: {e}\"))\n )?;\n messages.push(ProducedMessage {\n id: Some(row.id as u128),\n checksum: None,\n timestamp: None,\n origin_timestamp: Some(row.created_at_ns),\n headers: None,\n payload,\n });\n next_cursor = Some(row.cursor_value);\n }\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} produced {} messages, next cursor: {:?}\",\n self.id, messages.len(), next_cursor\n );\n }\n\n if messages.is_empty() {\n return Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: None,\n });\n }\n\n candidate.messages_produced += messages.len() as u64;\n candidate.cursor = next_cursor;\n let persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n *self.pending.lock().await = Some(candidate);\n\n Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: Some(persisted),\n })\n }\n\n async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {\n let (SourceBatchResult::Ack, Some(candidate)) =\n (result, self.pending.lock().await.take())\n else {\n return Ok(());\n };\n *self.state.lock().await = candidate;\n Ok(())\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx pools\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, total produced: {}\",\n self.id, state.messages_produced\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySourceConfig) -> Result { /* ... */ }\n```\n",".claude/skills/connector-sink/TEMPLATE.md":"# Sink plugin skeleton\n\nBoilerplate for a new `core/connectors/sinks/_sink/`. Adapt the\n`MySink` / `Client` types to the backend driver you're integrating.\n\n## Cargo.toml\n\n```toml\n# Apache 2.0 header (copy verbatim from any existing sink Cargo.toml)\n[package]\nname = \"iggy_connector__sink\"\nversion = \"0.5.1-edge.1\" # match the version most other sinks use\nedition = \"2024\"\nlicense = \"Apache-2.0\"\npublish = false\n# ...keywords, description, repository, homepage all identical to existing sinks\n\n[package.metadata.cargo-machete]\nignored = [\"dashmap\", \"once_cell\"] # used by sink_connector! macro\n\n[lib]\ncrate-type = [\"cdylib\", \"lib\"] # cdylib = runtime-loadable; lib = unit tests\n\n[dependencies]\nasync-trait = { workspace = true }\ndashmap = { workspace = true }\niggy_connector_sdk = { workspace = true }\nonce_cell = { workspace = true }\nserde = { workspace = true }\ntokio = { workspace = true }\ntracing = { workspace = true }\n# + your client crate (reqwest, sqlx, mongodb, ...)\n```\n\nRun `cargo sort --no-format --workspace` after edits.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, then `impl`, then private\nhelpers.\n\n```rust\n/* Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements... */ // full Apache header\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata, sink_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse std::str::FromStr;\nuse std::time::Duration;\nuse tokio::sync::Mutex;\nuse tracing::{debug, error, info, warn};\n\nsink_connector!(MySink); // generates FFI symbols + version export\n\nconst CONNECTOR_NAME: &str = \"My sink\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySinkConfig {\n pub endpoint: String,\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option, // humantime, e.g. \"500ms\"\n pub verbose_logging: Option, // mirror runtime's `verbose` flag\n // Every runtime-tunable field is Option; defaults applied in new()\n}\n\n#[derive(Debug)]\npub struct MySink {\n id: u32,\n config: MySinkConfig,\n batch_size: usize,\n max_retries: u32,\n retry_delay: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n}\n\n#[derive(Debug)]\nstruct State {\n messages_processed: u64,\n errors: u64,\n}\n\nimpl MySink {\n pub fn new(id: u32, config: MySinkConfig) -> Self {\n let batch_size = config.batch_size.unwrap_or(100) as usize;\n let max_retries = config.max_retries.unwrap_or(3);\n let retry_delay = config\n .retry_delay\n .as_deref()\n .and_then(|raw| humantime::Duration::from_str(raw).ok().map(|d| *d))\n .unwrap_or_else(|| {\n warn!(\"Invalid retry_delay for {CONNECTOR_NAME} ID: {id}, defaulting to 500ms\");\n Duration::from_millis(500)\n });\n let verbose = config.verbose_logging.unwrap_or(false);\n Self {\n id,\n config,\n batch_size,\n max_retries,\n retry_delay,\n verbose,\n client: None,\n state: Mutex::new(State { messages_processed: 0, errors: 0 }),\n }\n }\n}\n\n#[async_trait]\nimpl Sink for MySink {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n client.ping()\n .await\n .map_err(|e| Error::InitError(format!(\"connectivity check failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn consume(\n &self,\n topic_metadata: &TopicMetadata,\n messages_metadata: MessagesMetadata,\n messages: Vec,\n ) -> Result<(), Error> {\n let Some(client) = self.client.as_ref() else {\n return Err(Error::InitError(\"client not initialized\".into()));\n };\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages from stream: {}, topic: {}, offset: {}, current_offset: {}\",\n self.id, messages.len(), topic_metadata.stream, topic_metadata.topic,\n messages_metadata.partition_id, messages_metadata.current_offset\n );\n } else {\n debug!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages\",\n self.id, messages.len()\n );\n }\n\n let mut last_err: Option = None;\n for batch in messages.chunks(self.batch_size) {\n match self.send_batch(client, batch).await {\n Ok(()) => { /* counter */ }\n Err(Error::PermanentHttpError(message)) => {\n error!(\n \"{CONNECTOR_NAME} ID: {} dropping batch (permanent): {message}\",\n self.id\n );\n }\n Err(error) => {\n last_err = Some(error);\n }\n }\n }\n match last_err {\n Some(error) => Err(error),\n None => Ok(()),\n }\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n // sqlx pools have `.close().await`; reqwest/mongodb/elasticsearch just drop.\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, processed: {}, errors: {}\",\n self.id, state.messages_processed, state.errors\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySinkConfig) -> Result { /* ... */ }\n```\n","AGENTS.md":"# Apache Iggy Agent Guidelines\n\n**Apache Iggy** is a persistent message streaming platform\nin Rust. Thread-per-core shared-nothing, `io_uring` + `compio`.\nTransports: QUIC, WebSocket, TCP (custom binary), HTTP (REST). SDKs:\nRust, .NET, Java, Python, Go, C++, Node.js. A connectors subsystem\ningests from / egresses to external systems via dlopened plugins.\n\n> Skills live under `.claude/skills/`, with per-skill links under\n> `.agents/skills/` for Codex. Load\n> [connectors-overview](.claude/skills/connectors-overview/SKILL.md)\n> first for any change under `core/connectors/`. Other subsystems\n> follow the repo-wide principles in this file.\n\n## Contents\n\n- [Apache Iggy Agent Guidelines](#apache-iggy-agent-guidelines)\n - [Contents](#contents)\n - [STOP and ask the user before](#stop-and-ask-the-user-before)\n - [Helping someone contribute](#helping-someone-contribute)\n - [Quick reference](#quick-reference)\n - [Structure](#structure)\n - [Where to look](#where-to-look)\n - [Skills](#skills)\n - [Repo-wide principles](#repo-wide-principles)\n - [Tooling](#tooling)\n - [Testing](#testing)\n - [Pitfalls](#pitfalls)\n - [Local state directories](#local-state-directories)\n - [Commits, PRs, releases](#commits-prs-releases)\n - [Discussion and support](#discussion-and-support)\n\n## STOP and ask the user before\n\n- Writing code for an issue not assigned to the user\n (`gh issue view --json assignees`). If it isn't, or you can't\n confirm it, they must ask on the issue and wait to be assigned.\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI\n signature in `core/connectors/sdk/src/{sink,source}.rs` (breaks\n every pre-built plugin .so).\n- Modifying `core/binary_protocol/` wire-format types tagged\n `#[repr(C)]` (breaks every running client).\n- Touching the state-save path (`core/connectors/runtime/src/state.rs`)\n in a way that changes the atomic-rename protocol (corruption risk).\n- Renaming default consumer groups, plugin path resolution rules, or\n any other operator-facing identifier.\n- Force-pushing to `master`, amending shipped commits, or skipping\n hooks with `--no-verify` / `--no-gpg-sign`.\n\n## Helping someone contribute\n\nMost PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.\n\n- The PR must link an issue the user was assigned to before any work started. See the STOP list above.\n- Fill in every section of the PR template. Under AI Usage, name the agent and say what it wrote. Never leave the section out or write \"None\" when an agent wrote any of the change.\n- Run `prek run` before the PR is opened. If prek isn't installed, ask the user to install it (`cargo install prek`, then `prek install`). Don't substitute the individual checks.\n- If a check can't run, tell the user which one and why, and treat the PR as not ready. Never write text in a PR asking a maintainer to run something.\n- The user should write the rationale in their own words and be able to explain every change without asking you.\n- New contributors keep one open PR at a time.\n- If you found a bug, the first PR holds a failing test. The fix waits until the approach is agreed on the issue.\n\n## Quick reference\n\n```bash\n# Verification order is enforced by CI (.github/actions/rust/pre-merge).\ncargo fmt --all # step 1\ncargo sort --no-format --workspace # step 2 (--no-format mandatory)\ncargo clippy --all-features --all-targets -- -D warnings # step 3\ncargo test -p # step 4\n\n# Other CI checks (mirror in pre-commit; pass --fix where supported)\n./scripts/ci/taplo.sh # TOML\n./scripts/ci/markdownlint.sh # Markdown\n./scripts/ci/shellcheck.sh # Shell\n./scripts/ci/license-headers.sh # Apache headers\n./scripts/ci/trailing-whitespace.sh\n./scripts/ci/trailing-newline.sh\n./scripts/ci/sync-rustc-version.sh\n./scripts/ci/python-sdk-version-sync.sh\n./scripts/ci/uv-lock-check.sh\n./scripts/ci/binary-artifacts.sh --check\n./scripts/ci/third-party-licenses.sh --validate --manifest \n\n# Run things\ncargo run --bin iggy-server -- --with-default-root-credentials\ncargo run --bin iggy -- \nIGGY_CONNECTORS_CONFIG_PATH=... cargo run --bin iggy-connectors\n./scripts/run-bdd-tests.sh\n./scripts/run-benches.sh\n```\n\nPre-commit hooks wire the same scripts. Install with `pre-commit\ninstall` (Python) or `prek install` (Rust drop-in) - both read\n`.pre-commit-config.yaml`.\n\n## Structure\n\n```text\niggy/\n├── core/\n│ ├── server/ Iggy server binary (Viewstamped Replication)\n│ ├── sdk/ Rust client SDK\n│ ├── cli/ iggy CLI\n│ ├── connectors/ Connectors runtime + SDK + sinks/sources\n│ ├── common/ Shared types (IggyDuration, IggyTimestamp, ...)\n│ ├── binary_protocol/ Wire format (stable)\n│ ├── configs/ Config plumbing + ConfigEnv derive\n│ ├── ai/mcp/ MCP server\n│ ├── bench/ Benchmark suite\n│ ├── integration/ Cross-crate integration tests\n│ ├── simulator/ Deterministic simulator\n│ ├── consensus/ Raft (Miri-checked unsafe iobuf)\n│ ├── journal/ WAL / journal\n│ ├── message_bus/ In-process bus\n│ ├── shard/ Shard runtime\n│ ├── harness_derive/ #[iggy_harness] proc macro\n│ └── metadata, partitions, server_common, configs_derive, clock, tools\n├── foreign/ Java / .NET / Python / Go / C++ / Node.js SDKs\n├── bdd/ Cross-language BDD (Gherkin)\n├── examples/ Per-SDK runnable examples\n├── helm/ Helm charts\n├── web/ Web UI\n└── scripts/ CI + release + dev scripts\n```\n\n## Where to look\n\n| Task | Location |\n| --------------------- | ---------------------------------------- |\n| Wire protocol | `core/binary_protocol/` |\n| Server | `core/server/src/` |\n| Rust client SDK | `core/sdk/src/` |\n| Connectors | `core/connectors/` -> connector-* skills |\n| Integration tests | `core/integration/tests/` |\n| Cross-SDK BDD | `bdd/` |\n| Benchmark suite | `core/bench/` |\n\n## Skills\n\nEach `SKILL.md` has YAML frontmatter (name, description). For\nconnectors work, load `connectors-overview` first as router.\nClaude Code uses `/skill-name`; Codex uses `$skill-name` or `/skills`.\nKeep one source under `.claude/skills/` and link its directory from\n`.agents/skills/`. Keep templates and references inside the source directory.\n\n- [connectors-overview](.claude/skills/connectors-overview/SKILL.md) - router + universal connector rules\n- [connector-runtime](.claude/skills/connector-runtime/SKILL.md) - FFI host, lifecycle, state, metrics\n- [connector-sdk](.claude/skills/connector-sdk/SKILL.md) - traits, decoders/encoders, retry\n- [connector-sink](.claude/skills/connector-sink/SKILL.md) - sink plugin authoring\n- [connector-source](.claude/skills/connector-source/SKILL.md) - source plugin authoring\n- [connector-transform](.claude/skills/connector-transform/SKILL.md) - transform authoring\n- [connector-testing](.claude/skills/connector-testing/SKILL.md) - unit + integration test patterns\n\nReview skills are user-invoked only. Keep `disable-model-invocation: true`\nfor Claude Code and `policy.allow_implicit_invocation: false` in\n`agents/openai.yaml` for Codex. Do not replicate their steps without an\nexplicit invocation. When a non-trivial change passes verification,\nsuggest `/team-review ` in Claude Code or `$team-review `\nin Codex.\n\n- [connector-review](.claude/skills/connector-review/SKILL.md) - adversarial connectors PR/branch review with validation\n- [team-review](.claude/skills/team-review/SKILL.md) - adversarial 4-expert PR/branch review, ~10 subagents per run\n- [team-review-slim](.claude/skills/team-review-slim/SKILL.md) - four-expert review without independent validation rounds\n\nReview skills share [execution instructions](.claude/skills/team-review/references/execution.md)\nfor target preparation and client-specific agent tools. Run\n`./scripts/ci/skills.sh` to check skill metadata, links, and invocation policies.\n\n## Repo-wide principles\n\n1. **Apache 2.0 header on every new source file:** Follow the comment style configured in `licenserc.toml`; common examples: `.rs` uses `// ...`, `Cargo.toml` and shell use `# ...`.\n2. **Verification order:** `fmt --all` -> `sort --no-format --workspace` -> `clippy --all-features --all-targets -- -D warnings` -> `test`. Plus `taplo.sh --check`. CI enforces all.\n3. **Always pass `--no-format` to `cargo sort`.** Without it multi-line feature arrays flatten and `taplo` re-expands them, churning the workspace.\n4. **Avoid LLM-slop tells: em dashes, gratuitous semicolons, hedging narrative, trailing summaries.** Defaults, not absolute bans. Same rule for code, comments, commits, PR descriptions.\n5. **Idiomatic Rust over free helpers.** `FromStr` not `parse_foo`, `Display` not `format_foo`, `From`/`TryFrom` not `to_foo_from_bar`. Structured `Err` enums, not `String`.\n6. **Imports at the top.** No inline `use` inside functions or blocks. Group: std, external crates, then `crate::`.\n7. **Precise names.** No `b`, `p`, `t`, `m`. Match literal API field names as log labels.\n8. **Comments explain WHY, not WHAT.** Default to no comment. Add one only when a hidden constraint or workaround would surprise a future reader. Never reference the current task / PR.\n9. **Never `cargo install`** without authorization. Toolchain pinned in `rust-toolchain.toml`. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check.\n10. **No `unwrap()` / `expect()` on Results from external I/O** outside tests.\n11. **`tokio::sync::Mutex` (not `std::sync::Mutex`)** wherever a lock is held across `.await`. Hold locks briefly: lock, clone, drop guard, then do I/O.\n12. **Code reads top to bottom.** Public consts and public fns first, helpers below, deeper helpers below those.\n13. **Zero clone in hot loops.** `&self`, `&str`, `&[T]`. `Vec::with_capacity(n)` when size is known.\n14. **Forward-compatible config.** New TOML fields use `#[serde(default)]` or `Option`.\n\n## Tooling\n\n| Tool | Where |\n| --------------------------------------------- | ------------------------------------------- |\n| `cargo fmt`, `cargo sort`, `cargo clippy` | pre-commit + CI |\n| `cargo machete` | CI `machete` task |\n| `cargo nextest` + `cargo-rail` | CI `test-*` DAG-scoped |\n| `cargo llvm-cov` | CI coverage |\n| `cargo http-registry` | CI `verify-publish` |\n| Miri | CI `miri` (`binary_protocol` + `consensus`) |\n| `taplo`, `markdownlint-cli`, `shellcheck` | pre-commit + CI |\n| `typos` | pre-commit |\n| `cargo about` + `license-checker-rseidelsohn` | `scripts/ci/third-party-licenses.sh` |\n| HawkEye / `license-headers.sh` | pre-commit + CI |\n| `uv`, `ruff` | pre-commit (Python) |\n| `golangci-lint` | pre-push (Go) |\n\n## Testing\n\n- **Unit:** `#[cfg(test)] mod tests` at EOF of `src/lib.rs` (or per-module).\n- **In-crate integration:** `/tests/`.\n- **Cross-crate (real infra):** `core/integration/tests/`. Harness in\n `core/integration/src/harness/` spawns real `iggy-server`\n (`ServerHandle`), connectors runtime (`ConnectorsRuntimeHandle`),\n MCP server (`McpHandle`), client (`ClientHandle`). `#[iggy_harness]`\n proc macro wires fixtures. Suites: `cli/`, `cluster/`,\n `config_provider/`, `connectors/`, `data_integrity/`, `mcp/`, `sdk/`,\n `server/`, `state/`, `storage/`. External backends inside\n `connectors/` use `testcontainers-modules`. Per-test logs land in\n `test_logs/`.\n- **Cross-SDK BDD:** `bdd/`. Gherkin features against every SDK. Driver: `scripts/run-bdd-tests.sh`.\n\nBDD test naming: `given_X_when_Y_should_Z` (3-part). Promote to 4-part `given_X_when_Y_then_Z_should_W` only with a distinct \"then\" intermediate. Be consistent inside a file.\n\nFilter a single integration test by path. Build the binaries the harness launches first: it runs\nwhatever is already in `target/`, so a stale one fails as though the change under test broke it.\n\n```bash\ncargo build --bin iggy-server --bin iggy-connectors\ncargo test -p integration -- connectors::runtime::benchmark::given_logging_format_json\n```\n\n## Pitfalls\n\n- **Docker is required for most integration tests.** `testcontainers-modules` spawns containers per test.\n- **`cargo sort` without `--no-format` will churn the workspace.** Use the pre-commit hook.\n- **Rust toolchain is pinned in `rust-toolchain.toml`.** Older system `cargo` fails on transitive deps. prefer rustup so the pin auto-resolves.\n- **`test_logs/` grows quickly.** Wipe between major refactors.\n- **Miri only covers `binary_protocol` + `consensus`.** Cannot emulate `io_uring` syscalls. do not try to expand Miri to crates that pull `compio`.\n- **Integration crate has no `--test` target.** Filter by test path inside the single `mod.rs` binary.\n- **The integration harness runs prebuilt binaries and never builds them.** `Command::cargo_bin` only resolves a path, so a stale `iggy-server`, `iggy-connectors` or `iggy-mcp` is used silently. CI restores revision-checked artifacts from the runtime and Rust test builds; a local run does not build them, and a stale binary fails in ways that look like the change under test. Rebuild after anything that touches a launched binary or its config.\n\n## Local state directories\n\nCreated by server, connectors runtime, or tests. Gitignored. Safe to delete to reset:\n\n- `local_data/` - server data\n- `local_state/` - connectors runtime state files\n- `test_logs/` - per-test logs + `local_data` snapshots\n- `performance_results/` - benchmark output\n- `target/` - cargo build\n\n## Commits, PRs, releases\n\n- Conventional commits with scope: `feat(connectors): ...`. Subject <= 72 chars. Allowed types/scopes enforced by `.github/workflows/pr-title.yml`.\n- Body: why before what. No bullet lists of files changed.\n- Co-authored-by trailer when applicable.\n- Workspace + SDK versions sync via `scripts/extract-version.sh` and `scripts/ci/{sync-rust,python-sdk}-version{,-sync}.sh`.\n- Crate publish chain verified by `scripts/verify-crates-publish.sh` against `cargo-http-registry`.\n- Third-party license enumeration: `scripts/ci/third-party-licenses.sh` (ASF policy).\n\n## Discussion and support\n\n- Design: \n- Chat: \n- Docs: \n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Apache Iggy Agent Guidelines\n\n**Apache Iggy** is a persistent message streaming platform\nin Rust. Thread-per-core shared-nothing, `io_uring` + `compio`.\nTransports: QUIC, WebSocket, TCP (custom binary), HTTP (REST). SDKs:\nRust, .NET, Java, Python, Go, C++, Node.js. A connectors subsystem\ningests from / egresses to external systems via dlopened plugins.\n\n> Skills live under `.claude/skills/`, with per-skill links under\n> `.agents/skills/` for Codex. Load\n> [connectors-overview](.claude/skills/connectors-overview/SKILL.md)\n> first for any change under `core/connectors/`. Other subsystems\n> follow the repo-wide principles in this file.\n\n## Contents\n\n- [Apache Iggy Agent Guidelines](#apache-iggy-agent-guidelines)\n - [Contents](#contents)\n - [STOP and ask the user before](#stop-and-ask-the-user-before)\n - [Helping someone contribute](#helping-someone-contribute)\n - [Quick reference](#quick-reference)\n - [Structure](#structure)\n - [Where to look](#where-to-look)\n - [Skills](#skills)\n - [Repo-wide principles](#repo-wide-principles)\n - [Tooling](#tooling)\n - [Testing](#testing)\n - [Pitfalls](#pitfalls)\n - [Local state directories](#local-state-directories)\n - [Commits, PRs, releases](#commits-prs-releases)\n - [Discussion and support](#discussion-and-support)\n\n## STOP and ask the user before\n\n- Writing code for an issue not assigned to the user\n (`gh issue view --json assignees`). If it isn't, or you can't\n confirm it, they must ask on the issue and wait to be assigned.\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI\n signature in `core/connectors/sdk/src/{sink,source}.rs` (breaks\n every pre-built plugin .so).\n- Modifying `core/binary_protocol/` wire-format types tagged\n `#[repr(C)]` (breaks every running client).\n- Touching the state-save path (`core/connectors/runtime/src/state.rs`)\n in a way that changes the atomic-rename protocol (corruption risk).\n- Renaming default consumer groups, plugin path resolution rules, or\n any other operator-facing identifier.\n- Force-pushing to `master`, amending shipped commits, or skipping\n hooks with `--no-verify` / `--no-gpg-sign`.\n\n## Helping someone contribute\n\nMost PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.\n\n- The PR must link an issue the user was assigned to before any work started. See the STOP list above.\n- Fill in every section of the PR template. Under AI Usage, name the agent and say what it wrote. Never leave the section out or write \"None\" when an agent wrote any of the change.\n- Run `prek run` before the PR is opened. If prek isn't installed, ask the user to install it (`cargo install prek`, then `prek install`). Don't substitute the individual checks.\n- If a check can't run, tell the user which one and why, and treat the PR as not ready. Never write text in a PR asking a maintainer to run something.\n- The user should write the rationale in their own words and be able to explain every change without asking you.\n- New contributors keep one open PR at a time.\n- If you found a bug, the first PR holds a failing test. The fix waits until the approach is agreed on the issue.\n\n## Quick reference\n\n```bash\n# Verification order is enforced by CI (.github/actions/rust/pre-merge).\ncargo fmt --all # step 1\ncargo sort --no-format --workspace # step 2 (--no-format mandatory)\ncargo clippy --all-features --all-targets -- -D warnings # step 3\ncargo test -p # step 4\n\n# Other CI checks (mirror in pre-commit; pass --fix where supported)\n./scripts/ci/taplo.sh # TOML\n./scripts/ci/markdownlint.sh # Markdown\n./scripts/ci/shellcheck.sh # Shell\n./scripts/ci/license-headers.sh # Apache headers\n./scripts/ci/trailing-whitespace.sh\n./scripts/ci/trailing-newline.sh\n./scripts/ci/sync-rustc-version.sh\n./scripts/ci/python-sdk-version-sync.sh\n./scripts/ci/uv-lock-check.sh\n./scripts/ci/binary-artifacts.sh --check\n./scripts/ci/third-party-licenses.sh --validate --manifest \n\n# Run things\ncargo run --bin iggy-server -- --with-default-root-credentials\ncargo run --bin iggy -- \nIGGY_CONNECTORS_CONFIG_PATH=... cargo run --bin iggy-connectors\n./scripts/run-bdd-tests.sh\n./scripts/run-benches.sh\n```\n\nPre-commit hooks wire the same scripts. Install with `pre-commit\ninstall` (Python) or `prek install` (Rust drop-in) - both read\n`.pre-commit-config.yaml`.\n\n## Structure\n\n```text\niggy/\n├── core/\n│ ├── server/ Iggy server binary (Viewstamped Replication)\n│ ├── sdk/ Rust client SDK\n│ ├── cli/ iggy CLI\n│ ├── connectors/ Connectors runtime + SDK + sinks/sources\n│ ├── common/ Shared types (IggyDuration, IggyTimestamp, ...)\n│ ├── binary_protocol/ Wire format (stable)\n│ ├── configs/ Config plumbing + ConfigEnv derive\n│ ├── ai/mcp/ MCP server\n│ ├── bench/ Benchmark suite\n│ ├── integration/ Cross-crate integration tests\n│ ├── simulator/ Deterministic simulator\n│ ├── consensus/ Raft (Miri-checked unsafe iobuf)\n│ ├── journal/ WAL / journal\n│ ├── message_bus/ In-process bus\n│ ├── shard/ Shard runtime\n│ ├── harness_derive/ #[iggy_harness] proc macro\n│ └── metadata, partitions, server_common, configs_derive, clock, tools\n├── foreign/ Java / .NET / Python / Go / C++ / Node.js SDKs\n├── bdd/ Cross-language BDD (Gherkin)\n├── examples/ Per-SDK runnable examples\n├── helm/ Helm charts\n├── web/ Web UI\n└── scripts/ CI + release + dev scripts\n```\n\n## Where to look\n\n| Task | Location |\n| --------------------- | ---------------------------------------- |\n| Wire protocol | `core/binary_protocol/` |\n| Server | `core/server/src/` |\n| Rust client SDK | `core/sdk/src/` |\n| Connectors | `core/connectors/` -> connector-* skills |\n| Integration tests | `core/integration/tests/` |\n| Cross-SDK BDD | `bdd/` |\n| Benchmark suite | `core/bench/` |\n\n## Skills\n\nEach `SKILL.md` has YAML frontmatter (name, description). For\nconnectors work, load `connectors-overview` first as router.\nClaude Code uses `/skill-name`; Codex uses `$skill-name` or `/skills`.\nKeep one source under `.claude/skills/` and link its directory from\n`.agents/skills/`. Keep templates and references inside the source directory.\n\n- [connectors-overview](.claude/skills/connectors-overview/SKILL.md) - router + universal connector rules\n- [connector-runtime](.claude/skills/connector-runtime/SKILL.md) - FFI host, lifecycle, state, metrics\n- [connector-sdk](.claude/skills/connector-sdk/SKILL.md) - traits, decoders/encoders, retry\n- [connector-sink](.claude/skills/connector-sink/SKILL.md) - sink plugin authoring\n- [connector-source](.claude/skills/connector-source/SKILL.md) - source plugin authoring\n- [connector-transform](.claude/skills/connector-transform/SKILL.md) - transform authoring\n- [connector-testing](.claude/skills/connector-testing/SKILL.md) - unit + integration test patterns\n\nReview skills are user-invoked only. Keep `disable-model-invocation: true`\nfor Claude Code and `policy.allow_implicit_invocation: false` in\n`agents/openai.yaml` for Codex. Do not replicate their steps without an\nexplicit invocation. When a non-trivial change passes verification,\nsuggest `/team-review ` in Claude Code or `$team-review `\nin Codex.\n\n- [connector-review](.claude/skills/connector-review/SKILL.md) - adversarial connectors PR/branch review with validation\n- [team-review](.claude/skills/team-review/SKILL.md) - adversarial 4-expert PR/branch review, ~10 subagents per run\n- [team-review-slim](.claude/skills/team-review-slim/SKILL.md) - four-expert review without independent validation rounds\n\nReview skills share [execution instructions](.claude/skills/team-review/references/execution.md)\nfor target preparation and client-specific agent tools. Run\n`./scripts/ci/skills.sh` to check skill metadata, links, and invocation policies.\n\n## Repo-wide principles\n\n1. **Apache 2.0 header on every new source file:** Follow the comment style configured in `licenserc.toml`; common examples: `.rs` uses `// ...`, `Cargo.toml` and shell use `# ...`.\n2. **Verification order:** `fmt --all` -> `sort --no-format --workspace` -> `clippy --all-features --all-targets -- -D warnings` -> `test`. Plus `taplo.sh --check`. CI enforces all.\n3. **Always pass `--no-format` to `cargo sort`.** Without it multi-line feature arrays flatten and `taplo` re-expands them, churning the workspace.\n4. **Avoid LLM-slop tells: em dashes, gratuitous semicolons, hedging narrative, trailing summaries.** Defaults, not absolute bans. Same rule for code, comments, commits, PR descriptions.\n5. **Idiomatic Rust over free helpers.** `FromStr` not `parse_foo`, `Display` not `format_foo`, `From`/`TryFrom` not `to_foo_from_bar`. Structured `Err` enums, not `String`.\n6. **Imports at the top.** No inline `use` inside functions or blocks. Group: std, external crates, then `crate::`.\n7. **Precise names.** No `b`, `p`, `t`, `m`. Match literal API field names as log labels.\n8. **Comments explain WHY, not WHAT.** Default to no comment. Add one only when a hidden constraint or workaround would surprise a future reader. Never reference the current task / PR.\n9. **Never `cargo install`** without authorization. Toolchain pinned in `rust-toolchain.toml`. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check.\n10. **No `unwrap()` / `expect()` on Results from external I/O** outside tests.\n11. **`tokio::sync::Mutex` (not `std::sync::Mutex`)** wherever a lock is held across `.await`. Hold locks briefly: lock, clone, drop guard, then do I/O.\n12. **Code reads top to bottom.** Public consts and public fns first, helpers below, deeper helpers below those.\n13. **Zero clone in hot loops.** `&self`, `&str`, `&[T]`. `Vec::with_capacity(n)` when size is known.\n14. **Forward-compatible config.** New TOML fields use `#[serde(default)]` or `Option`.\n\n## Tooling\n\n| Tool | Where |\n| --------------------------------------------- | ------------------------------------------- |\n| `cargo fmt`, `cargo sort`, `cargo clippy` | pre-commit + CI |\n| `cargo machete` | CI `machete` task |\n| `cargo nextest` + `cargo-rail` | CI `test-*` DAG-scoped |\n| `cargo llvm-cov` | CI coverage |\n| `cargo http-registry` | CI `verify-publish` |\n| Miri | CI `miri` (`binary_protocol` + `consensus`) |\n| `taplo`, `markdownlint-cli`, `shellcheck` | pre-commit + CI |\n| `typos` | pre-commit |\n| `cargo about` + `license-checker-rseidelsohn` | `scripts/ci/third-party-licenses.sh` |\n| HawkEye / `license-headers.sh` | pre-commit + CI |\n| `uv`, `ruff` | pre-commit (Python) |\n| `golangci-lint` | pre-push (Go) |\n\n## Testing\n\n- **Unit:** `#[cfg(test)] mod tests` at EOF of `src/lib.rs` (or per-module).\n- **In-crate integration:** `/tests/`.\n- **Cross-crate (real infra):** `core/integration/tests/`. Harness in\n `core/integration/src/harness/` spawns real `iggy-server`\n (`ServerHandle`), connectors runtime (`ConnectorsRuntimeHandle`),\n MCP server (`McpHandle`), client (`ClientHandle`). `#[iggy_harness]`\n proc macro wires fixtures. Suites: `cli/`, `cluster/`,\n `config_provider/`, `connectors/`, `data_integrity/`, `mcp/`, `sdk/`,\n `server/`, `state/`, `storage/`. External backends inside\n `connectors/` use `testcontainers-modules`. Per-test logs land in\n `test_logs/`.\n- **Cross-SDK BDD:** `bdd/`. Gherkin features against every SDK. Driver: `scripts/run-bdd-tests.sh`.\n\nBDD test naming: `given_X_when_Y_should_Z` (3-part). Promote to 4-part `given_X_when_Y_then_Z_should_W` only with a distinct \"then\" intermediate. Be consistent inside a file.\n\nFilter a single integration test by path. Build the binaries the harness launches first: it runs\nwhatever is already in `target/`, so a stale one fails as though the change under test broke it.\n\n```bash\ncargo build --bin iggy-server --bin iggy-connectors\ncargo test -p integration -- connectors::runtime::benchmark::given_logging_format_json\n```\n\n## Pitfalls\n\n- **Docker is required for most integration tests.** `testcontainers-modules` spawns containers per test.\n- **`cargo sort` without `--no-format` will churn the workspace.** Use the pre-commit hook.\n- **Rust toolchain is pinned in `rust-toolchain.toml`.** Older system `cargo` fails on transitive deps. prefer rustup so the pin auto-resolves.\n- **`test_logs/` grows quickly.** Wipe between major refactors.\n- **Miri only covers `binary_protocol` + `consensus`.** Cannot emulate `io_uring` syscalls. do not try to expand Miri to crates that pull `compio`.\n- **Integration crate has no `--test` target.** Filter by test path inside the single `mod.rs` binary.\n- **The integration harness runs prebuilt binaries and never builds them.** `Command::cargo_bin` only resolves a path, so a stale `iggy-server`, `iggy-connectors` or `iggy-mcp` is used silently. CI restores revision-checked artifacts from the runtime and Rust test builds; a local run does not build them, and a stale binary fails in ways that look like the change under test. Rebuild after anything that touches a launched binary or its config.\n\n## Local state directories\n\nCreated by server, connectors runtime, or tests. Gitignored. Safe to delete to reset:\n\n- `local_data/` - server data\n- `local_state/` - connectors runtime state files\n- `test_logs/` - per-test logs + `local_data` snapshots\n- `performance_results/` - benchmark output\n- `target/` - cargo build\n\n## Commits, PRs, releases\n\n- Conventional commits with scope: `feat(connectors): ...`. Subject <= 72 chars. Allowed types/scopes enforced by `.github/workflows/pr-title.yml`.\n- Body: why before what. No bullet lists of files changed.\n- Co-authored-by trailer when applicable.\n- Workspace + SDK versions sync via `scripts/extract-version.sh` and `scripts/ci/{sync-rust,python-sdk}-version{,-sync}.sh`.\n- Crate publish chain verified by `scripts/verify-crates-publish.sh` against `cargo-http-registry`.\n- Third-party license enumeration: `scripts/ci/third-party-licenses.sh` (ASF policy).\n\n## Discussion and support\n\n- Design: \n- Chat: \n- Docs: \n","isInternal":false,"tokens":3833,"sizeBytes":15640},{"name":"SKILL.md","path":".claude/skills/connector-runtime/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-runtime/SKILL.md","title":"connector-runtime","category":"anthropic-skill","format":"markdown","content":"---\nname: connector-runtime\ndescription: Modify the Apache Iggy Connectors runtime (`core/connectors/runtime/`) - the host process that loads plugins, manages lifecycle, and bridges Apache Iggy streams to plugins via FFI. Use when changing plugin loading, the FFI dispatch, the sink/source manager, state storage, config providers, the HTTP control API, metrics, or transforms wiring. NOT for plugin code.\n---\n\n# Modifying the Apache Iggy Connectors Runtime\n\nRuntime is the host. Loads `.so`/`.dylib`/`.dll` plugins, drives their\nlifecycle, ferries messages between Apache Iggy and plugins, exposes\n`/stats`, `/metrics`, control HTTP API. Bugs here affect every plugin.\n\n> **Universal connector rules** (benchmark, SecretString, verbose, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers runtime internals only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Architecture invariants](#architecture-invariants)\n- [State storage (`state.rs`)](#state-storage-staters)\n- [Source forwarding loop](#source-forwarding-loop)\n- [Sink consumption loop](#sink-consumption-loop)\n- [Logging format](#logging-format)\n- [The `verbose` flag](#the-verbose-flag)\n- [Config provider rules](#config-provider-rules)\n- [Error categorization](#error-categorization)\n- [Metrics (`metrics.rs`)](#metrics-metricsrs)\n- [Drop accounting (wired sites)](#drop-accounting-wired-sites)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing FFI `#[repr(C)]` layouts (`SourceApi`, `SinkApi`, `RawMessages`, headers) - existing plugin `.so`s misalign.\n- Modifying `state.rs` save protocol (atomic tmp + rename + fsync ordering) - corruption risk.\n- Renaming the default consumer group `iggy-connect-sink-{key}` - operators hold offsets there.\n- Hot-reloading or unloading a `.so` - mid-FFI tasks would segfault. Architectural change required.\n- Changing the SerDe split (postcard for FFI payloads, serde_json for configs, rmp_serde for state) - downstream incompat.\n\n## File map\n\n```text\nruntime/src/\n├── main.rs Entry, plugin path resolution, SourceApi/SinkApi FFI structs\n├── sink.rs Sink lifecycle, Iggy consumer wiring, FFI consume calls\n├── source.rs Source lifecycle, flume forwarding, state save loop\n├── stream.rs Stream + consumer/producer setup\n├── transform.rs Loads transforms from config, applies them in chain\n├── state.rs FileStateProvider, atomic ConnectorState file I/O\n├── context.rs Per-instance context (Iggy client, metrics, config)\n├── error.rs RuntimeError with as_code() for HTTP responses\n├── log.rs init_logging (text/JSON x telemetry-on/off matrix) + LOG_CALLBACK\n├── benchmark.rs emit_sink_event / emit_source_event + as_micros\n├── metrics.rs Prometheus families + gauges + stage histograms + label caches\n├── stats.rs /stats endpoint payload assembly\n├── manager/{mod,sink,source}.rs SinkManager / SourceManager, status, restart_guard\n├── configs/\n│ ├── runtime.rs RuntimeConfig + LoggingConfig + LogFormat\n│ ├── connectors.rs ConnectorConfig, SinkConfig, SourceConfig (verbose + benchmark)\n│ └── connectors/{local,http}_provider.rs\n└── api/ HTTP control + observability endpoints\n```\n\n## Architecture invariants\n\nLoad-bearing assumptions. Breaking them silently corrupts data or hangs.\n\n### Plugin instance identity\n\n- Each instance gets a globally unique `plugin_id: u32` from `PLUGIN_ID: AtomicU32` in `main.rs`. Never reused.\n- Same `.so` may load many instances. each gets a fresh `plugin_id` but shares the `Container` (dlopen handle) via `Arc` (`manager/{sink,source}.rs`).\n- All FFI calls keyed by `plugin_id`. Mixing IDs = data to the wrong plugin.\n\n### FFI pointer lifetimes\n\n- Config and state are `(*const u8, usize)`. Valid for the call duration only. Plugin must copy if it needs the bytes past the call. SDK macros do this correctly. don't bypass.\n- `LogCallback` lifetime spans the whole process - kept static in `log.rs`. Don't make it dynamic.\n\n### Container ownership\n\n- `dlopen2::wrapper::Container<{Sink,Source}Api>` (defined in `main.rs`) is shared via `Arc` across all instances of the same `.so`.\n- Arc must outlive every spawned task that calls into the plugin. Manager holds the Arc in `{Sink,Source}Details` for the connector lifetime.\n- Unloading a `.so` while a task is mid-FFI-call would segfault. Runtime does not unload. Hot reload needs a rearchitect.\n\n### Plugin manager state\n\n- `SinkManager` / `SourceManager` use `DashMap>>`. Key = TOML `key` field.\n- Status transitions (`Starting → Running → Stopped/Error`) update running counters only on transitions to/from `Running`. See `SinkManager::update_status`.\n- `restart_guard: Arc>` prevents concurrent restart of the same connector. `restart_connector()` uses `try_lock()` - returns OK without restart if busy. Intentional anti-thundering-herd.\n\n### Serialization split\n\n- FFI message payloads (`TopicMetadata`, `MessagesMetadata`, `RawMessages`, headers): `postcard` (compact, stable wire).\n- Connector configs passed to plugin: `serde_json` (TOML in, JSON over FFI - human-editable).\n- `ConnectorState` bytes inside plugins: `rmp_serde` (MessagePack - compact, opaque to runtime).\n\nDon't mix.\n\n## State storage (`state.rs`)\n\n- File path: `{state_path}/source_{connector_key}.state`. Mode `0o600` on Unix.\n- `save()` is crash-atomic. `rename(2)` is atomic in the namespace (no observer sees a half-renamed file). the preceding `sync_data` on the tmp file plus the post-rename parent-dir `sync_all` are what make a crash leave either the old or new content - never truncated. The parent-dir fsync failure is propagated as `CannotWriteStateFile` (not swallowed) so a lost rename surfaces instead of silently restarting the source from scratch. A `Mutex<()>` serializes concurrent saves on the same provider.\n- Save after every successful Apache Iggy send. Save failure logs + continues. next batch retries.\n- `load()` returns `Ok(None)` for missing or empty files. After a `NotFound` read, re-stats the parent directory: missing parent -> `Err(CannotOpenStateFile)` so a broken state path fails at init rather than masquerading as \"fresh start\".\n- Sinks have no state - only sources.\n\n## Source forwarding loop\n\n1. `iggy_source_handle(id, send_callback)` - plugin registers itself.\n2. Plugin polls + invokes `send_callback(plugin_id, ptr, len)`.\n3. Callback runs in the SDK macro's spawned async task. Pushes postcard `ProducedMessages` into a `flume` channel keyed by `plugin_id` in `SOURCE_SENDERS: Lazy>` (`pub(crate)`). `SourceSenderEntry` wraps the sender + a pre-extracted owned `Counter` (the `errors` series, `Arc` inside). The FFI callback bumps errors on deserialize or channel-closed failure with one relaxed atomic - no `Family` lookup, no `Arc` handle.\n4. The runtime registers the optional source-stop callback, then `source_forwarding_loop` pulls from the channel, deserializes, applies transforms, encodes via `StreamEncoder`, and sends to the Iggy producer. The SDK warns after 30 seconds without a result but keeps the same batch pending; it does not poll or enqueue another copy.\n5. On success, save returned `ConnectorState` via `FileStateProvider`, then return the batch result to the plugin. A result NACK may cause replay with capped backoff.\n6. A stop-callback reason ends the forwarding loop and reports an unexpected stop. Closing the stop channel on normal shutdown disables that select arm, allowing already queued batches to receive results before the forwarding channel disconnects.\n7. At loop exit, clean up `SOURCE_SENDERS` and update status and metrics. A self-stop reports `Error` and removes the source from the running gauge; normal shutdown reports `Stopped`.\n\n**Shutdown ordering (`manager/source.rs::stop_connector`):**\n\n1. Call `iggy_source_close` FIRST. It blocks until the plugin's polling task stops, so no new send callbacks fire after it returns.\n2. `cleanup_sender(plugin_id)` NEXT - dropping the channel sender lets the forwarding task finish any queued batches before `recv_async()` resolves with `Disconnected` and exits cleanly, instead of blocking until the abort timeout.\n3. Finally await spawned handlers with `tokio::time::timeout`. On timeout, `handle.abort()` + drain - prevents leaked tasks colliding with the next `start_connector` (a late `file.save()` could otherwise race the new instance). The silent-drop branch in `handle_produced_messages` only covers the window between close and cleanup.\n\nGotchas:\n\n- `SOURCE_SENDERS` must be cleaned up on connector close or memory leaks (channel + task).\n- `spawn_source_handler` wraps the outer `iggy_source_handle(id, callback)` FFI call in `tokio::task::spawn_blocking()`. That call returns quickly - the SDK macro internally `runtime.spawn`s an async `handle_messages` task and returns. **`send_callback` invocations come from that async task, not from `spawn_blocking`.** A long-running synchronous poll inside the plugin would block one Tokio worker. async polls don't.\n- No timeout on the registration call - a plugin whose `iggy_source_handle` never returns stalls one blocking worker for the process lifetime.\n\n## Sink consumption loop\n\nPer `[[streams]]` entry per sink:\n\n1. Build / ensure consumer group `iggy-connect-sink-{key}` (`connect` matches the docker image name `apache/iggy-connect` - see `runtime/src/sink.rs::default_consumer_group`). Overridable via the stream config's `consumer_group`.\n2. Spawn one task per topic (`spawn_consume_tasks`).\n3. Poll Iggy → batch messages (default `batch_length = 1000`, `poll_interval = 5ms`). A `consumer.next()` `Err` (transport/decode at the Iggy client boundary) bumps `errors` and `continue`s - offset already auto-committed via `AutoCommit::When(AutoCommitWhen::PollingMessages)`, message effectively dropped.\n4. Decode via `Schema::decoder()`. Decode failure logs + bumps `errors`. `messages_filtered` is reserved for the transform filter contract (`Ok(None)`).\n5. Apply transforms (chain). A transform `Err` is logged, bumps `errors`, and drops only that message (drop-and-continue, mirroring the source) - one bad payload never kills the batch. `Ok(None)` filters the message - bumps `messages_filtered{connector_type=\"sink\"}`.\n6. Postcard-encode batch as `RawMessages`. Headers as a separate postcard blob. Per-message failures (missing field, payload conversion, header serialization) bump `errors` + skip. Batch-level failures (postcard serialization of metadata / `RawMessages`) propagate `Err` -> `spawn_consume_tasks` wrapper bumps `errors` + `set_error`. The total histogram and benchmark emit still fire for the failed batch before the `Err` propagates.\n7. Call `iggy_sink_consume(id, topic_meta, messages_meta, messages)`.\n8. Non-zero return → log + `errors` increment.\n\nPer-message counters (decode errors, transform errors, filters, field/serialize drops) are accumulated and flushed once per batch via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels`, not one `Family` lookup per message.\n\nStages (both labels snake_case): `decode` (schema decode loop), `prepare` (transform + serialize = total - decode - ffi), `ffi` (plugin consume call), `total`. Same `decode` / `prepare` meaning as the source side.\n\nGotchas:\n\n- Batch flushes on `current_offset != message_offset` (gap) OR batch full - higher latency for low-volume topics.\n- Auto-commit happens on poll - idempotency at the sink protects against re-delivery on crash before plugin write.\n\n## Logging format\n\n`LoggingConfig` in `configs/runtime.rs` exposes `format: LogFormat` (`Text` default, `Json`), env-addressable via `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `log::init_logging` matches on `(telemetry.enabled, format)` and installs the right `fmt::layer()`/`fmt::layer().json()` + OpenTelemetry layer. OTel pipeline untouched by `format` - only the stdout layer switches.\n\n`fmt::layer()` and `fmt::layer().json()` are different concrete types, so the stdout layer is built once as a `Box>` (via `.boxed()`) by matching on `format`, then a single 2-arm branch on `telemetry.enabled` attaches the OTel layers. When extending, keep the boxed-layer + 2-arm shape rather than re-expanding into a 4-way match.\n\n## The `verbose` flag\n\n`SinkConfig` and `SourceConfig` in `configs/connectors.rs` carry `verbose: bool` (default false). Runtime threads it to `spawn_consume_tasks` / source equivalent to gate per-batch log promotion:\n\n```rust\nif verbose {\n info!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n} else {\n debug!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n}\n```\n\nNew per-batch logging follows the same pattern. Default to `debug!`, upgrade to `info!` when `verbose`. No third level.\n\n## Config provider rules\n\n### Local (`configs/connectors/local_provider.rs`)\n\n- Discovery walk reads `*.toml` from `config_dir`. Skips hidden files (`.`) and `Cargo.toml`.\n- Filename: `{key}_{type}[_v{N}].toml` (e.g., `postgres_sink_v2.toml`).\n- One TOML deserialized into `SinkConfig` or `SourceConfig`. Grouped by `key`. Highest `version` wins unless `.active_versions.toml` overrides.\n- `.active_versions.toml` is NOT read via the discovery walk (hidden). Read via direct path through `LocalConnectorsConfigProvider::active_versions_file_path`.\n\n### HTTP (`configs/connectors/http_provider.rs`)\n\n- `reqwest_middleware` + `RetryTransientMiddleware`. Retries 5xx + connection errors. Never 4xx (`PermanentHttpError` convention).\n- Default URL templates documented in `runtime/README.md`. Configurable via TOML.\n\n### Env-var overrides via `ConfigEnv` derive\n\n`SinkConfig`, `SourceConfig`, and inner structs derive `ConfigEnv` (`configs_derive::ConfigEnv`). Generates env-var addressability as `IGGY_CONNECTORS___` for primitive fields. Used heavily by integration tests to inject testcontainer ports - see `core/integration/tests/connectors/fixtures/postgres/container.rs` for env-var constants. Mark new compound fields `#[config_env(skip)]`, leaf primitives `#[config_env(leaf)]`.\n\n### Versioning\n\n- Configs carry `version: u64`. Local provider auto-increments if absent. HTTP provider trusts the server.\n- Restart on version change uses `restart_guard`.\n\n## Error categorization\n\n`RuntimeError` (`error.rs`) carries `as_code()` for HTTP API responses.\n\n| Class | Fatal? | Example |\n| ------------------ | --------------------------- | -------------------------------- |\n| Config load | yes (process exit) | Bad TOML, missing required field |\n| Iggy client init | yes | Auth failure at startup |\n| State dir create | yes | Permission denied |\n| Plugin .so resolve | per-plugin (`FailedPlugin`) | Missing file |\n| Plugin open FFI | per-plugin (status = Error) | Plugin returned non-zero |\n| Message decode | per-message (skip) | Bad protobuf bytes |\n| Iggy send | per-batch (metric) | Network blip |\n| State save | per-batch (log) | Disk full |\n\nFatal errors propagate to `main` and exit. Per-connector / per-message errors are isolated.\n\n## Metrics (`metrics.rs`)\n\nAll families labeled by `connector_key` + `connector_type` (histogram adds `stage`):\n\n- **Counters**: `iggy_connector_messages_{produced,sent,consumed,processed,filtered}_total` and `iggy_connector_errors_total`. These are the *rendered* names; each is registered without the `_total`, which the OpenMetrics encoder appends.\n - `messages_filtered_total` - intentional drops via transform `Ok(None)`.\n - `errors_total` - unexpected drops (decode/encode/build failure, missing field, ...) + batch-level failures, one per failed run.\n - `sink_runs_total` - FFI `consume()` calls for sink batches, one per contiguous payload-variant run. Divide by `stage_duration_seconds_count{stage=\"total\"}` for runs per batch.\n- **Histograms**: `iggy_connector_stage_duration_seconds{stage}` (snake_case stage labels - `prepare`, `ffi`, `decode`, `iggy_send`, `state_save`, `total`). Buckets `STAGE_BUCKETS_SECONDS`. Always populated regardless of any flag. One sample per batch for every stage; `ffi` is summed over the batch's runs. Scraped at `/metrics` when `[http.metrics] enabled = true`.\n- **Gauges**: `iggy_connectors_{sources,sinks}_{total,running}`.\n\n`/stats` JSON surface mirrors counters per-connector via `ConnectorStats` (`sdk/api.rs`): `messages_filtered`, `errors`, kind-specific counters.\n\nWhen adding a metric:\n\n- Add family to `Metrics` struct + `init`, register with name + help text. Never end a `Counter` family's registered name in `_total`: the encoder appends it and the series renders `_total_total`. Gauges get no suffix, so a gauge name may end in `_total` literally.\n- New label sets define `EncodeLabelSet` struct + label enum (hand-impl `EncodeLabelValue` for snake_case values - the derive emits PascalCase).\n- Histograms: pass `fn() -> Histogram` to `Family::new_with_constructor`.\n- Add unit tests under `mod tests` with `given_*_when_*_should_*` BDD names.\n\n## Drop accounting (wired sites)\n\nPer-message drops in the batch loops are counted into a local `u64` and flushed once after the loop via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels` - one `Family` lookup per batch, not per message. One-shot drops outside a loop (e.g. `consumer.next()` Err) still call `inc_errors_with_labels` directly. Hot path uses pre-built `SinkLabels`/`SourceLabels`. `&str`-based wrappers are `#[cfg(test)]`-only.\n\n- `sink.rs::consume_messages` - `consumer.next()` Err\n- `sink.rs::process_messages` - decode, transform Err (drop-and-continue), missing required fields, payload conversion, header serialization (accumulated, flushed once per batch)\n- `sink.rs::spawn_consume_tasks` task wrapper - bumps once on `consume_messages` Err\n- `source.rs::source_forwarding_loop` - payload decode, prepare (transform/encode) failure, Iggy send Err, state save Err\n- `source.rs::process_messages` - transform Err (logs + bumps `errors` + continue. does NOT propagate, so one bad payload doesn't flip the connector to permanent ERROR), transform encode failure, `build_iggy_message` failure\n- `source.rs::handle_produced_messages` - postcard deserialize failure, `sender.send` channel-closed\n\nFilter case bumps `messages_filtered` via `inc_messages_filtered_with_labels`. Adding a new drop path: mirror this pattern.\n\n## Hard rules\n\n1. **Never block the executor.** All I/O async. `spawn_blocking` only for FFI registration (already in place).\n2. **Plugin ID counter is monotonic.** No reset, no reuse.\n3. **FFI return codes:** `0` success, non-zero failure.\n4. **Don't add static mutable state** beyond `LOG_CALLBACK`, `PLUGIN_ID`, `SOURCE_SENDERS`.\n5. **Pair `cleanup_sender(id)` with shutdown** for sources (avoid flume leak). Order: close FFI -> cleanup sender -> drain/abort tasks.\n6. **Restart uses `restart_guard.try_lock()`** - no thundering-herd regression.\n7. **No timeouts on plugin FFI calls without a kill-task strategy.** A timeout that returns from the runtime but leaves the plugin running has the worst of both worlds.\n\n## Common pitfalls\n\n1. Mutating `INSTANCES` (in SDK macro) from runtime side - impossible by design. only the macro touches it.\n2. Calling `iggy_*_open` with a duplicate ID - returns `-1`. Means runtime forgot to `iggy_*_close` first.\n3. Holding a `DashMap` shard guard across `.await` - lockups. Pattern: `.get().clone()`, drop guard, await.\n4. Adding fields to `#[repr(C)]` types without an SDK version bump - existing plugins misalign on `postcard::from_bytes`.\n5. Changing the default consumer group name without coordinating with operators - they have offsets stored there.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy-connectors --all-targets -- -D warnings\ncargo test -p iggy-connectors\n\n# Sanity-build in-tree plugins:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n\n# Runtime-focused integration tests:\ncargo test -p integration -- connectors::runtime::\n\n# Smoke-test with example config:\nIGGY_CONNECTORS_CONFIG_PATH=core/connectors/runtime/example_config/config.toml \\\n cargo run --bin iggy-connectors\n```\n\nUpdate `runtime/README.md` if endpoints, env vars, or config schema change.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connector-runtime","description":"Modify the Apache Iggy Connectors runtime (`core/connectors/runtime/`) - the host process that loads plugins, manages lifecycle, and bridges Apache Iggy streams to plugins via FFI. Use when changing plugin loading, the FFI dispatch, the sink/source manager, state storage, config providers, the HTTP control API, metrics, or transforms wiring. NOT for plugin code."},"isInternal":false,"tokens":4942,"sizeBytes":21078},{"name":"SKILL.md","path":".claude/skills/connector-sdk/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-sdk/SKILL.md","title":"connector-sdk","category":"anthropic-skill","format":"markdown","content":"---\nname: connector-sdk\ndescription: Extend or modify the Apache Iggy Connectors SDK (`core/connectors/sdk/`). Use when adding a new `Schema` variant, a new `StreamDecoder`/`StreamEncoder`, a new `Error` variant, modifying the `Sink`/`Source`/`Transform` trait surface, the FFI macros, or the `retry`/`api`/`convert` modules. NOT for writing plugins - use `connector-sink` or `connector-source` for those.\n---\n\n# Extending the Apache Iggy Connectors SDK\n\nThe SDK (`core/connectors/sdk/`) is the **stable contract** between the\nruntime and every plugin. Changes ripple to every sink/source in-tree\nand every third-party plugin. Treat the public surface as a versioned API.\n\n> **Universal connector rules** (benchmark, SecretString, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers SDK-level changes only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Cardinal rules](#cardinal-rules)\n- [Adding a new `Schema` variant](#adding-a-new-schema-variant)\n- [Adding a `StreamDecoder` / `StreamEncoder`](#adding-a-streamdecoder--streamencoder)\n- [Adding a new `Error` variant](#adding-a-new-error-variant)\n- [Modifying `Sink` / `Source` / `Transform` traits](#modifying-sink--source--transform-traits)\n- [FFI macros (`sink_connector!`, `source_connector!`)](#ffi-macros-sink_connector-source_connector)\n- [`Payload::try_to_bytes(&self)` is non-negotiable for JSON](#payloadtry_to_bytesself-is-non-negotiable-for-json)\n- [Retry helpers (`retry.rs`)](#retry-helpers-retryrs)\n- [`ConnectorState`](#connectorstate)\n- [Tests for SDK changes](#tests-for-sdk-changes)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Bumping the SDK MAJOR version or changing any `#[repr(C)]` layout - misaligns every pre-built plugin `.so`.\n- Changing `Sink` / `Source` / `Transform` / `StreamDecoder` / `StreamEncoder` trait signatures - cascades through every plugin.\n- Removing or renaming a `Schema` variant or `Error` variant - decoders/encoders pinned to it. downstream pattern matches break.\n- Changing wire serialization (postcard / rmp_serde / serde_json) for any FFI / state / config payload.\n- Changing FFI macro return codes or function names (`iggy_*_open` / `consume` / `handle` / `close`) - runtime + plugins both need lockstep update.\n\n## File map\n\n```text\nsdk/src/\n├── lib.rs Traits (Sink, Source, StreamDecoder, StreamEncoder),\n│ Payload, Schema, ConnectorState, Error enum, FFI message structs.\n├── sink.rs SinkContainer + sink_connector! macro (FFI plumbing).\n├── source.rs SourceContainer + source_connector! macro (FFI plumbing).\n├── api.rs ConnectorStatus, ConnectorStats (feature = \"api\").\n├── convert.rs owned_value_into_serde_json / owned_value_to_serde_json (simd_json ⇄ serde_json bridge).\n├── log.rs CallbackLayer for tracing across FFI.\n├── retry.rs retry_async + RetryPolicy, CircuitBreaker, HttpRetryMiddleware.\n├── decoders/ One per schema: json, raw, text, proto, flatbuffer, avro.\n├── encoders/ Mirror of decoders.\n└── transforms/ add_fields, delete_fields, update_fields, filter_fields,\n unwrap_envelope, proto_convert, flatbuffer_convert, avro_convert.\n```\n\n## Cardinal rules\n\n1. **Apache 2.0 license header** on every new file.\n2. **`Send + Sync`** on every public trait (FFI runs across thread pools).\n3. **`#[repr(C)]`** on every type that crosses FFI (`Schema`, `TopicMetadata`, `MessagesMetadata`, `RawMessage`, `ProducedMessages`, `ConsumedMessage`, `DecodedMessage`, etc. in `lib.rs`). Adding fields requires bumping the SDK version - existing plugins built against the old layout will misalign on `postcard::from_bytes`.\n4. **postcard** for FFI message serialization (handled by `SinkContainer::consume`, `SourceContainer`'s `handle_messages`). **MessagePack (`rmp_serde`)** for `ConnectorState`. **JSON** (`serde_json`) only for human-editable config that crosses FFI. Don't mix.\n5. **`simd_json::OwnedValue`** for JSON payloads, not `serde_json::Value`. Use `convert::owned_value_into_serde_json` as a bridge when you own the value (the common case - it moves strings instead of copying them); fall back to the borrowing `convert::owned_value_to_serde_json` only when you still need the `OwnedValue` afterward.\n6. **`BTreeMap`** for headers - deterministic ordering. Never `HashMap` on the wire.\n7. **No breaking changes** to `Sink`/`Source`/`StreamDecoder`/`StreamEncoder`/`Transform` trait signatures without coordinating with all in-tree plugins in the same PR.\n8. **A `Schema` variant that changes meaning is as breaking as a new variant.** The discriminant crosses FFI unchanged, so a plugin built against the old SDK keeps decoding it the old way and misreads every batch with no error anywhere. Same process as a trait change: bump the SDK minor version, update every in-tree plugin in the same PR, and state in the PR description that plugins built against the previous SDK must be rebuilt.\n\n## Adding a new `Schema` variant\n\nThe `Schema` enum (in `lib.rs`) is `#[repr(C)]` - it crosses FFI. Touch points to update **in one PR**:\n\n1. `Schema` enum: add variant with `#[strum(to_string = \"...\")]` matching the `serde(rename_all=\"snake_case\")` form.\n2. `Payload` enum: add a matching variant carrying the deserialized form.\n3. `Payload::try_into_vec` - consuming bytes-out path.\n4. `Payload::try_to_bytes` - **borrowing** bytes-out path. For non-trivial payloads (parsed trees), implement a no-clone serialization, not a `clone() + serialize`. See the `Payload::Json` arm for the canonical optimization.\n5. `Payload::schema()` - variant → `Schema`. The runtime tags every sink batch from this, so a missing arm will not compile but a wrong one silently mislabels the batch.\n6. `Payload::Display`.\n7. `Schema::try_into_payload` - bytes → `Payload`, for a tag naming a wire format (the source path).\n8. `Payload::try_from_schema` - bytes → `Payload`, for a tag that came from `Payload::schema()` (the sink path). The exact inverse of item 5.\n9. `Schema::decoder()` - factory returning `Arc`.\n10. `Schema::encoder()` - factory returning `Arc`.\n11. New `decoders/.rs` and `encoders/.rs`.\n12. Update the schema list in `sdk/README.md`.\n13. Tests: round-trip encode/decode, error paths. `Payload::schema()` must round-trip through `Payload::try_from_schema`, with no exceptions: every variant survives that trip and the test is table-driven over all of them.\n\nMiss any of these → silent failures at runtime (typically `Error::InvalidPayloadType` or surprising decoder behavior in plugin code).\n\n## Adding a `StreamDecoder` / `StreamEncoder`\n\nPattern from `decoders/json.rs::JsonStreamDecoder`:\n\n```rust\npub struct MyStreamDecoder; // unit struct if stateless\n\nimpl StreamDecoder for MyStreamDecoder {\n fn schema(&self) -> Schema { Schema::MyFormat }\n fn decode(&self, payload: Vec) -> Result {\n // Decode payload bytes into Payload::MyFormat(...).\n // Errors: Error::CannotDecode(Schema::MyFormat), Error::InvalidJsonPayload, etc.\n }\n}\n```\n\nFor **stateful** decoders (proto/avro/flatbuffer) holding schema descriptors:\n\n- Store the schema state directly in the struct (`config`, `message_descriptor`, `schema`). The instance is built once and wrapped in `Arc` for sharing - it must be `Send + Sync` but does not need additional locking if construction-time loading is final.\n- Provide **two constructors** when schema loading can fail: lenient `new(config) -> Self` (logs + degrades) AND strict `try_new(config) -> Result` (fail-fast). Plus a `Default` impl for the no-schema case. Canonical example: `sdk/src/decoders/avro.rs` exports all three (`AvroStreamDecoder::new`, `::try_new`, `Default`).\n- If you genuinely need to mutate the schema after construction (e.g., a `update_config` method), use `&mut self` plus `std::mem::replace(&mut self.config, new_config)` to swap without cloning the old value. Live pattern: `sdk/src/decoders/avro.rs::AvroStreamDecoder::update_config` and `sdk/src/encoders/avro.rs::AvroStreamEncoder::update_config`.\n- A fresh decoder instance is created via `Schema::decoder()` on each call (it returns `Arc`), so per-decoder caching of shared mutable state is the wrong abstraction.\n\n## Adding a new `Error` variant\n\nThe `Error` enum (in `lib.rs`) is `Clone + PartialEq + Eq + Hash`. New variants must preserve these. Guidelines:\n\n- **Use existing variants first.** Only add a new one if the failure mode is distinct in handling, not just description. `InvalidConfigValue(String)` covers most \"bad config\" cases.\n- **Carry context as `String`** - don't add a struct unless multiple discrete fields are needed by callers programmatically.\n- **Document retry semantics in the docstring.** Existing variants (`PermanentHttpError`, `CatalogCommitError`, `TransactionApplyError`) document retry behavior - mirror that style.\n- **Place near related variants** to keep the enum readable top-to-bottom.\n\n## Modifying `Sink` / `Source` / `Transform` traits\n\n**Breaking changes.** Required process:\n\n1. Document migration in the SDK changelog (and `connectors/sdk/README.md`).\n2. Update every in-tree plugin in the same PR. CI builds them all.\n3. Bump SDK minor version.\n4. Note in PR description: dynamic plugins compiled against the old SDK version will fail to load with mismatched FFI symbols.\n\nNon-breaking additions (new default method, new struct field with `#[serde(default)]`, new variant on a non-exhaustive enum) are preferable.\n\n## FFI macros (`sink_connector!`, `source_connector!`)\n\nThe macros (in `sink.rs` and `source.rs`) generate:\n\n- `iggy__open(id, config_ptr, config_len, [state_ptr, state_len,] log_callback) -> i32`\n- `iggy_sink_consume(id, ...)` / `iggy_source_handle(id, callback) -> i32`\n- `iggy__close(id) -> i32`\n- `iggy__version() -> *const c_char` (static lifetime, from `env!(\"CARGO_PKG_VERSION\")`)\n\nInvariants:\n\n- **Duplicate-ID guard** returns `-1` if the caller reopens an ID without closing first. Don't remove this - it prevents silent buffered-data loss.\n- `INSTANCES: Lazy>>` (and `SourceContainer<$type>` mirror) is the only mutable global the macro introduces - keep it that way.\n- Return codes: `0` success, `-1` invalid call, `1` open failure. Don't repurpose.\n\nChanges to the FFI signature must update `runtime/src/main.rs::{SourceApi, SinkApi}` in the same PR.\n\n## `Payload::try_to_bytes(&self)` is non-negotiable for JSON\n\nPlugin authors call this on every consumed message. The implementation in `lib.rs::Payload::try_to_bytes` documents *why* it skips the deep `OwnedValue` clone - replacing `O(n) clone + O(n) serialize` with `O(n) serialize`. If you add a new variant requiring a parsed tree, do the same: serialize in place, don't clone.\n\n## Retry helpers (`retry.rs`)\n\n- `CircuitBreaker`: threshold + cooldown, `try_lock()` on the success path to avoid hot-path contention.\n- `retry_async(policy, context, should_retry, op)`: the retry loop for anything failing as `Err`. Owns attempt counting, backoff and the per-retry log; returns `RetryFailure { error, attempts, exhausted }` so callers write their own terminal log.\n- `retry_backoff(base, retry, max)`: backoff only, for a loop that computes its own delay. `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath and applies no jitter, so call it directly only when the delay must be exact (`source.rs::nack_retry_delay`, whose tests assert exact values).\n- `HttpRetryMiddleware`: integrates with `reqwest-middleware`. Retries 429 + 5xx + network errors. Honors `Retry-After` on 429 and 5xx as a floor over the computed backoff, capped at a one-hour `MAX_RETRY_AFTER`, not `max_delay`.\n- `max_retries` = **total attempts** including the first try, not extra retries. Document if you change this convention. `meilisearch_sink` is the standing exception: its `max_retries` / `max_open_retries` count retries *after* the first, as its README states.\n- New helpers must take `Duration` (not `u64 millis`) on the public API. Internal computation uses `humantime` parsing of `String`.\n\n## `ConnectorState`\n\n- Bytes are opaque - the wrapping type does not impose schema.\n- Serialization is MessagePack via `rmp_serde`. Compact, deterministic, well-supported in serde.\n- `serialize`/`deserialize` helpers return `Option` and log on failure. **Failures are non-fatal** - design downstream code to tolerate fresh state.\n- Don't change the underlying format without coordinating with all sources.\n\n## Tests for SDK changes\n\n- Unit tests for any new pure function in the changed module.\n- Round-trip tests for new schemas/transforms (encode → decode == identity for JSON-equivalent payloads).\n- `sdk/tests/` integration tests when the change crosses module boundaries.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n\n# Rebuild all plugins to catch breaking-change leaks:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n# If FFI signatures changed, also build the runtime:\ncargo build -p iggy-connectors\n```\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connector-sdk","description":"Extend or modify the Apache Iggy Connectors SDK (`core/connectors/sdk/`). Use when adding a new `Schema` variant, a new `StreamDecoder`/`StreamEncoder`, a new `Error` variant, modifying the `Sink`/`Source`/`Transform` trait surface, the FFI macros, or the `retry`/`api`/`convert` modules. NOT for writing plugins - use `connector-sink` or `connector-source` for those."},"isInternal":false,"tokens":3362,"sizeBytes":13622},{"name":"SKILL.md","path":".claude/skills/connector-sink/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-sink/SKILL.md","title":"connector-sink","category":"anthropic-skill","format":"markdown","content":"---\nname: connector-sink\ndescription: Author a new Apache Iggy connector sink plugin under core/connectors/sinks/. Sinks consume messages from Apache Iggy streams and push them to an external system (DB, search engine, HTTP endpoint, object store). Load when creating, modifying, or reviewing a sink crate. Use for sink plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Sink\n\nA **sink** is a Rust `cdylib` that implements `iggy_connector_sdk::Sink`\nand exposes FFI symbols via the `sink_connector!` macro. The runtime\nloads it as `.so`/`.dylib`/`.dll`, polls Apache Iggy on its behalf,\nruns the transform chain, decodes via the configured `Schema`, and\nhands the sink batches of `ConsumedMessage`. The sink is responsible\nfor getting them to the external system reliably and efficiently.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's sink-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Sink::open` / `consume` / `close`) - that's an SDK change, not a plugin one.\n- Touching default `batch_size`, retry caps, or backoff defaults baked into existing in-tree sinks.\n- Promoting an `Err` variant to retryable when it previously was not - retrying bad data trips circuit breakers.\n- Adding a new sink that introduces a backend the runtime doesn't yet integration-test for.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand when authoring).\n- Exemplars: `stdout_sink` (minimal), `postgres_sink` (DB + transient detection), `http_sink` (validation, batch modes, retry middleware), `mongodb_sink` (atomic counters), `elasticsearch_sink` / `iceberg_sink` (backend-specific idioms).\n\n## Hard rules\n\n### Lifecycle\n\n- `Sink::consume` takes **`&self`** - never `&mut self`. Mutable state goes behind `tokio::sync::Mutex` / `AtomicU64`. Prefer atomics for hot counters - `mongodb_sink` uses `messages_processed: AtomicU64` + `insertion_errors: AtomicU64` and avoids a state lock on the hot path entirely.\n- `Sink::open` builds the client AND tests connectivity. Fail fast with `Error::InitError`.\n- `Sink::close` flushes + drops resources. Take ownership via `client: Option` + `.take()`. sqlx pools have `.close().await`. reqwest/mongodb/elasticsearch clients just drop. Log final stats either way.\n\n### Config\n\n- Every runtime-tunable field is `Option` with a default applied in `new()`. Forward-compat: adding a field doesn't break existing TOML.\n- Durations: `Option`, parsed via `humantime::Duration` in `new()`. Two equivalent in-tree idioms - inline `HumanDuration::from_str(&raw).map(|d| *d).unwrap_or_else(..)` (`random_source`, `postgres_source`) or a local `parse_duration` helper (`http_sink`, `sdk/src/retry.rs`). Fall back to a default on parse failure with `warn!` - never panic. We do **not** use `humantime_serde`.\n- **Conflict resolution**: if two config fields can be in invalid relative ordering, fix it in `new()` + `warn!` - don't error. `http_sink` swaps `retry_delay` and `max_retry_delay` when reversed to avoid an `ExponentialBackoff` panic downstream.\n- Validation order: structural in `new()`, connectivity in `open()`, **never in `consume()`**.\n\n### Payloads (efficiency-critical)\n\n- Dispatch on the `Payload` variant. Handle `Json`, `Raw`, `Text` and `Proto` at minimum. `Proto` holds text that is a JSON document when `proto_convert` had no descriptor or could not encode, and arbitrary text otherwise: call `Payload::json_document()` first and fall back to the text handling when it returns `None`. Unsupported -> `Error::InvalidPayloadType` or fall back to base64 (`elasticsearch_sink`).\n- For `Payload::Json`, call `try_to_bytes(&self)` - single-pass serialization without cloning the `OwnedValue` tree (`sdk/src/lib.rs::Payload::try_to_bytes`). Never `payload.clone().try_into_vec()`.\n- To take a payload out of `ConsumedMessage` without cloning: `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))`. Standard pattern across `http_sink::send_individual` / `send_ndjson` / `send_json_array`.\n- Pre-allocate per-batch buffers with `Vec::with_capacity(messages.len())` (or `* factor` for serialized bodies). Used across `postgres_sink`, `mongodb_sink`, `quickwit_sink`, `elasticsearch_sink`, `http_sink`.\n\n### Headers\n\n- `message.headers: Option>`. Check `.is_empty()` before iterating - runtime treats empty as `None`.\n- For binary values use `v.as_raw()` + base64. for text use `v.to_string_value()`. See `http_sink::EncodedHeader`.\n- `BTreeMap` for deterministic ordering. Don't collect into `HashMap`.\n\n### Errors\n\n| Scenario | SDK variant | Retry? |\n| ------------------------------------------ | --------------------------------------------------------------------------- | ---------------------------------------- |\n| Init / connectivity failure | `Error::InitError` | n/a (fails open) |\n| Bad config value | `Error::InvalidConfigValue(field)` | n/a |\n| Single-message validation failure | `Error::InvalidRecordValue(reason)` | no - skip + log |\n| Transient backend (5xx, conn reset, ...) | `Error::HttpRequestFailed` / `Error::Connection` / `Error::CannotStoreData` | yes |\n| Permanent 4xx, schema mismatch, constraint | `Error::PermanentHttpError` / `Error::SchemaMismatch` | **no** - retrying trips circuit breakers |\n| I/O failure mid-write (parquet/iceberg) | `Error::WriteFailure` | caller decides |\n| Catalog commit failure (Iceberg) | `Error::CatalogCommitError` | **no - not idempotent** |\n\n### Retry\n\n- SDK helpers cover the simple case: `iggy_connector_sdk::retry::check_connectivity_with_retry(...)` for `open()`, `HttpRetryMiddleware` for default 429/5xx/network policy on reqwest clients.\n- Custom strategies: `reqwest-middleware` + `reqwest_retry::RetryTransientMiddleware::new_with_policy_and_strategy`. `http_sink` defines its own `HttpSinkRetryStrategy` (honors `success_status_codes`, per-status decisions).\n- Non-HTTP clients: write `is_transient_error(&e)` mapping driver-specific codes. `postgres_sink::is_transient_error` maps SQLSTATEs `40001`, `40P01`, `57P01-03`, `08000/03/06`.\n- Retry loop: new connectors use `iggy_connector_sdk::retry::retry_async(policy, context, should_retry, op)` for anything that fails as `Err`. It owns attempt counting, backoff and the per-retry log, and returns `RetryFailure { error, attempts, exhausted }`; the caller logs the terminal failure.\n- Backoff only, for a loop that computes its own delay (it retries on an `Ok` response, carries a deadline, or reconnects between attempts): `retry_backoff(base, retry, max)`, where `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath, and it applies no jitter; call it directly only when the delay must be exact, as `sdk/src/source.rs::nack_retry_delay` needs for its tests.\n- Cap retries at 3 total attempts.\n\n### Idempotency\n\nRuntime retries on any `Err` - same batch can be redelivered. In-tree guarantees:\n\n- **Postgres** (`postgres_sink::build_batch_insert_query`): PK on `id`, but plain `INSERT` (no `ON CONFLICT`). Duplicate retry triggers `23505`, `is_transient_error` does NOT mark it transient -> error propagates. Adding `ON CONFLICT (id) DO NOTHING` is a known improvement.\n- **MongoDB**: composite `_id` from `stream:topic:partition:message_id`, via `insert_many(docs).ordered(false)`. Per-document duplicate-key errors don't abort the batch - effective idempotency under retry.\n- **Elasticsearch** (`bulk_index_documents`): bulk action specifies only `_index`, no `_id` -> ES auto-generates ids -> NOT idempotent. Extensions should add `\"_id\"`.\n- **HTTP**: endpoint-dependent. Document in the plugin README.\n- **Iceberg**: transactional commit. `Error::CatalogCommitError` documents that the txn is consumed on `commit()`. Rebuild-then-retry can duplicate data - callers must check the catalog first.\n\nWhen writing a new sink, prefer dedup-on-write using the message `id` field. Mongo's upsert-style is the cleanest reference.\n\n### Batching\n\n- Default `batch_size: 100`. Tune per-backend.\n- Chunk via `messages.chunks(self.batch_size)`. Do not buffer across `consume()` - runtime already batches at the consumer-poll boundary.\n- For HTTP backends support multiple batch modes when applicable (individual / ndjson / json_array / raw). See `http_sink::BatchMode`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: \", self.id);\ndebug!(\"Processing batch of {} messages, offset: {}\", batch.len(), offset);\nwarn!(\"Retry {attempt}/{max} for ID: {}, reason: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, processed: {}\", self.id, count);\n```\n\nUse literal API field names as labels (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` - runtime can't dlopen a regular rlib.\n2. Cloning the whole `Payload::Json` to serialize - use `try_to_bytes(&self)`.\n3. Holding `Mutex` across `.await` when the locked region does I/O.\n4. `&mut self` on `consume` - won't compile. you need interior mutability.\n5. Returning the first error and dropping subsequent batches - track `last_err`, process all, return.\n6. Forgetting to map `PermanentHttpError` - retries hammer the backend with bad data.\n7. Mutating config in `consume()` - config is a snapshot from `new()`. Reload requires runtime restart.\n8. Logging the connection string. Redact at construction: `http_sink::sanitize_url_for_log` populates `HttpSink.log_url`.\n\n## Tests\n\nUnit tests at EOF of `src/lib.rs`. See [connector-testing](../connector-testing/SKILL.md) for BDD naming + `test_config()` helper.\n\nReal-infra integration tests live under `core/integration/tests/connectors//` using `#[iggy_harness]` + a backend fixture that spins up Docker via `testcontainers-modules`. Plugins integrating with external services must have an integration test. Reference: `core/integration/tests/connectors/postgres/postgres_sink.rs` + `fixtures/postgres/`.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__sink --all-targets -- -D warnings\ncargo test -p iggy_connector__sink\n\n# Integration tests (filter by name; integration crate has no per-area target):\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sinks/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connector-sink","description":"Author a new Apache Iggy connector sink plugin under core/connectors/sinks/. Sinks consume messages from Apache Iggy streams and push them to an external system (DB, search engine, HTTP endpoint, object store). Load when creating, modifying, or reviewing a sink crate. Use for sink plugin authoring. NOT for runtime internals (see connector-runtime)."},"isInternal":false,"tokens":2713,"sizeBytes":11643},{"name":"TEMPLATE.md","path":".claude/skills/connector-sink/TEMPLATE.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-sink/TEMPLATE.md","title":"Connector-sink Skill","category":"anthropic-skill","format":"markdown","content":"# Sink plugin skeleton\n\nBoilerplate for a new `core/connectors/sinks/_sink/`. Adapt the\n`MySink` / `Client` types to the backend driver you're integrating.\n\n## Cargo.toml\n\n```toml\n# Apache 2.0 header (copy verbatim from any existing sink Cargo.toml)\n[package]\nname = \"iggy_connector__sink\"\nversion = \"0.5.1-edge.1\" # match the version most other sinks use\nedition = \"2024\"\nlicense = \"Apache-2.0\"\npublish = false\n# ...keywords, description, repository, homepage all identical to existing sinks\n\n[package.metadata.cargo-machete]\nignored = [\"dashmap\", \"once_cell\"] # used by sink_connector! macro\n\n[lib]\ncrate-type = [\"cdylib\", \"lib\"] # cdylib = runtime-loadable; lib = unit tests\n\n[dependencies]\nasync-trait = { workspace = true }\ndashmap = { workspace = true }\niggy_connector_sdk = { workspace = true }\nonce_cell = { workspace = true }\nserde = { workspace = true }\ntokio = { workspace = true }\ntracing = { workspace = true }\n# + your client crate (reqwest, sqlx, mongodb, ...)\n```\n\nRun `cargo sort --no-format --workspace` after edits.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, then `impl`, then private\nhelpers.\n\n```rust\n/* Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements... */ // full Apache header\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata, sink_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse std::str::FromStr;\nuse std::time::Duration;\nuse tokio::sync::Mutex;\nuse tracing::{debug, error, info, warn};\n\nsink_connector!(MySink); // generates FFI symbols + version export\n\nconst CONNECTOR_NAME: &str = \"My sink\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySinkConfig {\n pub endpoint: String,\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option, // humantime, e.g. \"500ms\"\n pub verbose_logging: Option, // mirror runtime's `verbose` flag\n // Every runtime-tunable field is Option; defaults applied in new()\n}\n\n#[derive(Debug)]\npub struct MySink {\n id: u32,\n config: MySinkConfig,\n batch_size: usize,\n max_retries: u32,\n retry_delay: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n}\n\n#[derive(Debug)]\nstruct State {\n messages_processed: u64,\n errors: u64,\n}\n\nimpl MySink {\n pub fn new(id: u32, config: MySinkConfig) -> Self {\n let batch_size = config.batch_size.unwrap_or(100) as usize;\n let max_retries = config.max_retries.unwrap_or(3);\n let retry_delay = config\n .retry_delay\n .as_deref()\n .and_then(|raw| humantime::Duration::from_str(raw).ok().map(|d| *d))\n .unwrap_or_else(|| {\n warn!(\"Invalid retry_delay for {CONNECTOR_NAME} ID: {id}, defaulting to 500ms\");\n Duration::from_millis(500)\n });\n let verbose = config.verbose_logging.unwrap_or(false);\n Self {\n id,\n config,\n batch_size,\n max_retries,\n retry_delay,\n verbose,\n client: None,\n state: Mutex::new(State { messages_processed: 0, errors: 0 }),\n }\n }\n}\n\n#[async_trait]\nimpl Sink for MySink {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n client.ping()\n .await\n .map_err(|e| Error::InitError(format!(\"connectivity check failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn consume(\n &self,\n topic_metadata: &TopicMetadata,\n messages_metadata: MessagesMetadata,\n messages: Vec,\n ) -> Result<(), Error> {\n let Some(client) = self.client.as_ref() else {\n return Err(Error::InitError(\"client not initialized\".into()));\n };\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages from stream: {}, topic: {}, offset: {}, current_offset: {}\",\n self.id, messages.len(), topic_metadata.stream, topic_metadata.topic,\n messages_metadata.partition_id, messages_metadata.current_offset\n );\n } else {\n debug!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages\",\n self.id, messages.len()\n );\n }\n\n let mut last_err: Option = None;\n for batch in messages.chunks(self.batch_size) {\n match self.send_batch(client, batch).await {\n Ok(()) => { /* counter */ }\n Err(Error::PermanentHttpError(message)) => {\n error!(\n \"{CONNECTOR_NAME} ID: {} dropping batch (permanent): {message}\",\n self.id\n );\n }\n Err(error) => {\n last_err = Some(error);\n }\n }\n }\n match last_err {\n Some(error) => Err(error),\n None => Ok(()),\n }\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n // sqlx pools have `.close().await`; reqwest/mongodb/elasticsearch just drop.\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, processed: {}, errors: {}\",\n self.id, state.messages_processed, state.errors\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySinkConfig) -> Result { /* ... */ }\n```\n","isInternal":false,"tokens":1396,"sizeBytes":6034},{"name":"SKILL.md","path":".claude/skills/connector-source/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-source/SKILL.md","title":"connector-source","category":"anthropic-skill","format":"markdown","content":"---\nname: connector-source\ndescription: Author a new Apache Iggy connector source plugin under core/connectors/sources/. Sources poll an external system (DB, API, queue) and produce messages into Apache Iggy streams. Load when creating, modifying, or reviewing a source crate. Use for source plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Source\n\nA **source** is a Rust `cdylib` that implements\n`iggy_connector_sdk::Source` and exposes FFI symbols via the\n`source_connector!` macro. The runtime calls `poll()` in a loop,\napplies transforms, encodes via the configured `Schema`, sends to\nApache Iggy, and persists the returned `ConnectorState` after every\nsuccessful send.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's source-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Source::open` / `poll` / `on_batch_result` / `close`) - that's an SDK change.\n- Adding a long-running side task in the plugin - the runtime owns lifecycle, and orphans survive `close()`. Sanctioned only where the source is itself a server the runtime cannot drive, as in `http_source`'s listener, and then only with an explicit shutdown in the last `close()` that awaits its tasks before returning.\n- Persisting unbounded state - `State` is rewritten every batch.\n- Adding a source that requires authoritative offsets external to Apache Iggy without coordinating retention.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand).\n- Exemplars: `random_source` (minimal + canonical state tests), `postgres_source` (cursor / delete-after-read / processed-column modes, restart-survives-state tests), `elasticsearch_source` (scroll cursor), `influxdb_source` (time-series scan).\n\n## Hard rules\n\n### `poll()` signature is `&self`\n\nThe macro shares the source as `Arc` across the FFI callback and forwarding loop. Signature: `async fn poll(&self) -> ...` - any mutable state behind `tokio::sync::Mutex`. **Single most common new-contributor mistake.**\n\n### Lock discipline\n\nNever hold the state `Mutex` across upstream I/O. Build a candidate from committed\nstate, then stage it until the runtime reports the batch result:\n\n```rust\nlet mut candidate = self.state.lock().await.clone();\nlet rows = client.query(&sql, &[&candidate.cursor]).await?;\ncandidate.cursor = Some(new_cursor);\nlet persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n*self.pending.lock().await = Some(candidate);\n```\n\n### Delivery acknowledgment\n\n`on_batch_result` (added by #3855) is how a source learns what happened to the batch it just\nreturned. The SDK keeps exactly one batch in flight: it will not call `poll()` again until this\nreturns. By default it stops after `MAX_CONSECUTIVE_NACKS` (5) consecutive NACKs; a source can\noverride `batch_policy()` to disable the limit when its accepted input cannot be replayed after\nrestart. A stop is reported to the runtime only when the plugin exports the current SDK's stop\ncallback, so rebuild older source plugins.\n\n- `Ack` means the runtime sent the batch **and** persisted its state. `Nack` means it could not\n confirm both, which is **not** the same as neither happening: a batch that reached the topic but\n whose state save failed is NACKed. A missing runtime result leaves the batch pending; it does not\n produce a synthetic NACK. A source that replays on `Nack` is at-least-once, not exactly-once.\n- The trait has a **default no-op**, which suits only a source with no staged cursor and no\n destructive work. If `poll()` advances a cursor, deletes rows, or drains an in-memory buffer,\n omitting this loses data silently and nothing will tell you. `random_source` and `http_source`\n implement it; the other shipped sources take the default and skip rows on a NACK.\n- Stage in `poll()`, apply on `Ack`, discard or replay on `Nack`. A source whose input is pushed to\n it, rather than re-readable upstream, has to hold the batch itself: see `http_source`'s staging.\n- Returning `Err` from it stops the source immediately, so it is not a retry signal.\n\n### State persistence\n\n- `ConnectorState` is `Vec` via MessagePack (`rmp_serde`). Use\n `ConnectorState::serialize(&state, NAME, id)` and\n `ConnectorState::deserialize::(NAME, id)`.\n- `poll()` must not commit cursors or destructive work. Return messages with candidate state and\n keep the corresponding work staged.\n- The runtime sends the batch, saves its candidate state to\n `{state_path}/source_{key}.state`, then calls `on_batch_result(Ack)`. Commit staged in-memory\n state and external delete or mark operations only on ACK. A NACK discards the candidate so the\n same data can be polled again.\n- A crash between `poll()` returning and state persistence leaves the prior cursor for the next\n poll, so downstream must tolerate at-least-once delivery.\n- Cursor sources that stage state until `on_batch_result` can attach state to the corresponding\n message batch. Control-plane state that is independent of a publish can ride an empty batch\n instead.\n- Return `state: None` for an empty poll when no watermark changed. If an empty poll advances a\n watermark, stage and return the new state through the same ACK handshake.\n- The runtime can still NACK an empty batch when state storage is latched or a pending checkpoint\n cannot resolve. Never treat hand-off as durable; `on_batch_result` reports the outcome.\n- Treat candidate-state serialization failure as a poll error. Do not send messages without the\n state needed to resume them safely.\n- Keep `State` small - rewritten every batch. No unbounded vecs.\n\nThe SDK allows one in-flight batch. Five consecutive NACKs stop a source using\nthe default policy and require a manual restart. Returning `Err` from\n`on_batch_result` is fatal regardless of the NACK limit, so\nretry transient backend failures inside the callback before returning an error.\nIf the runtime has not reported ACK or NACK after 30 seconds, the SDK warns\nand keeps waiting for that result without polling or replaying the batch.\nOnce the result is received, the SDK waits for `on_batch_result` to finish,\nso the callback must bound its own connection acquisition and retry budget.\n\n### Sleep first\n\n`poll()` must `sleep(self.poll_interval).await` before any work. Without it, an empty source spins a CPU.\n\n### Schema selection\n\nMatch `ProducedMessages.schema` to the bytes in `messages[i].payload`:\n\n- JSON-serialized rows → `Schema::Json`\n- Already-protobuf bytes → `Schema::Proto`\n- Already-avro bytes → `Schema::Avro`\n- Opaque → `Schema::Raw`\n\n### IDs and timestamps\n\n- `ProducedMessage.id: Option` - set when a natural ID exists (DB PK, document id). It rides the wire as the message id, which is what a **consumer** can dedupe an at-least-once duplicate on.\n- **Iggy itself does not dedupe on that id.** The server's only read of the field is `core/server/src/http/wire.rs`, which mints a fresh uuid when the incoming id is 0 and otherwise passes it through. Its dedup path is a per-client request-id watermark (`dedup_clients_max`) that never looks at this field.\n- `origin_timestamp: Option` - source-system event time in nanoseconds. Lets downstream sinks reason about lag.\n- `timestamp` and `checksum` are Iggy-side - leave `None`.\n\n### Concurrency\n\n- Runtime spawns ONE `poll()` task per source. No concurrent `poll()`.\n- Don't spawn your own long-running Tokio tasks: the runtime owns lifecycle. The exception is a source that listens rather than polls, which has to own its listener; `http_source` is the worked example, and it shuts its tasks down in the last `close()` rather than leaving them to outlive the connector.\n\n### Errors\n\n| Scenario | Variant |\n| ------------------------------------------- | ------------------------------------------------- |\n| Bad config in `new()`/`open()` | `Error::InitError` |\n| Cannot reach external system at startup | `Error::InitError` or `Error::Connection` |\n| Transient fetch failure (retry-worthy) | `Error::Connection` or `Error::HttpRequestFailed` |\n| Permanent fetch failure (auth, schema gone) | `Error::PermanentHttpError` |\n| Row failed to serialize | `Error::Serialization(...)` |\n| State serialization failed | `Error::Serialization(...)` |\n\nReturning `Err` from `poll()` is only logged by the SDK's FFI bridge\n(`sdk/src/source.rs::handle_messages`) - the loop continues, the next\n`poll()` runs. Connector status does NOT flip to `Error` from a poll\nfailure. Status `Error` is set by the runtime only on transform/encode\nfailure, Iggy send failure, or state save failure\n(`runtime/src/source.rs::source_forwarding_loop` calls to\n`context.sources.set_error`). To surface a poll failure as unhealth,\nraise it through the metric counter or escalate to `Error::InitError`\nfrom `open()`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: {}\", self.id, ...);\ninfo!(\"Restored state for ID: {id}, cursor: {:?}\", ...);\ndebug!(\"Polled {} rows for ID: {}\", rows.len(), self.id);\nwarn!(\"Transient fetch failure for ID: {}, will retry: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, total produced: {}\", self.id, ...);\n```\n\nIggy consumer-loop labels use literal API names (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. `async fn poll(&mut self)` - won't compile. Use `&self` + `Mutex`.\n2. Holding `state.lock()` across the fetch I/O - blocks `close()`, causes shutdown timeouts.\n3. Forgetting to sleep - 100% CPU on idle source.\n4. Committing a cursor or deleting source data in `poll()` - stage it and wait for ACK.\n5. Unbounded data in `State` - rewritten every batch. keep O(constant).\n6. `std::sync::Mutex` - blocks the executor. Use `tokio::sync::Mutex`.\n7. Not setting `ProducedMessage.id` when a stable ID exists - leaves a consumer nothing to dedupe a replayed duplicate on. It does not make the write idempotent server-side, because nothing there reads it.\n8. Spawning side tasks - the runtime owns the scheduler. The one exception is a source that listens rather than polls and must own its listener (see [Concurrency](#concurrency) and the STOP list); it owes an explicit shutdown in the last `close()` that awaits its tasks.\n\n## Tests\n\nMandatory four canonical source state tests (see [connector-testing](../connector-testing/SKILL.md) for the full pattern). Copy from `sources/random_source/src/lib.rs::tests`. Plus config defaults, payload building, schema selection.\n\nIntegration tests under `core/integration/tests/connectors//` for any source backed by external infra. Use `#[iggy_harness]` + a `TestFixture` backed by `testcontainers-modules`. Reference: `core/integration/tests/connectors/postgres/postgres_source.rs` (multi-mode tests) + `restart.rs` (state survives restart). Exercise both ACK and NACK paths when the source stages cursors or destructive work.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__source --all-targets -- -D warnings\ncargo test -p iggy_connector__source\n\n# Integration tests:\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sources/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connector-source","description":"Author a new Apache Iggy connector source plugin under core/connectors/sources/. Sources poll an external system (DB, API, queue) and produce messages into Apache Iggy streams. Load when creating, modifying, or reviewing a source crate. Use for source plugin authoring. NOT for runtime internals (see connector-runtime)."},"isInternal":false,"tokens":2824,"sizeBytes":12221},{"name":"TEMPLATE.md","path":".claude/skills/connector-source/TEMPLATE.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-source/TEMPLATE.md","title":"Connector-source Skill","category":"anthropic-skill","format":"markdown","content":"# Source plugin skeleton\n\nBoilerplate for a new `core/connectors/sources/_source/`. Adapt\nthe `MySource` / `Client` / row types to the backend driver.\n\n`Cargo.toml` is identical to a sink's (see\n[connector-sink/TEMPLATE.md](../connector-sink/TEMPLATE.md)) - only\nthe crate name suffix changes (`iggy_connector__source`) and the\nupstream client dep.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, `impl Source` after,\nhelpers below.\n\n```rust\n/* Apache 2.0 header */\n\nuse std::str::FromStr;\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,\n source::SourceBatchResult, source_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse tokio::sync::Mutex;\nuse tokio::time::sleep;\nuse tracing::{debug, error, info, warn};\n\nsource_connector!(MySource);\n\nconst CONNECTOR_NAME: &str = \"My source\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySourceConfig {\n pub endpoint: String,\n pub poll_interval: Option, // humantime, \"10s\" default\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option,\n pub verbose_logging: Option,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct State {\n cursor: Option, // WAL LSN, scroll id, timestamp, ...\n last_offset: u64,\n messages_produced: u64,\n}\n\n#[derive(Debug)]\npub struct MySource {\n id: u32,\n config: MySourceConfig,\n poll_interval: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n pending: Mutex>,\n}\n\nimpl MySource {\n pub fn new(id: u32, config: MySourceConfig, state: Option) -> Self {\n let raw_interval = config.poll_interval.clone().unwrap_or_else(|| \"10s\".into());\n let poll_interval = humantime::Duration::from_str(&raw_interval)\n .map(|d| *d)\n .unwrap_or_else(|_| {\n warn!(\"Invalid poll_interval for {CONNECTOR_NAME} ID: {id}, defaulting to 10s\");\n Duration::from_secs(10)\n });\n\n let verbose = config.verbose_logging.unwrap_or(false);\n\n let restored = state\n .and_then(|s| s.deserialize::(CONNECTOR_NAME, id))\n .inspect(|s| info!(\n \"Restored state for {CONNECTOR_NAME} ID: {id}, last_offset: {}, cursor: {:?}\",\n s.last_offset, s.cursor\n ));\n\n Self {\n id,\n config,\n poll_interval,\n verbose,\n client: None,\n state: Mutex::new(restored.unwrap_or(State {\n cursor: None,\n last_offset: 0,\n messages_produced: 0,\n })),\n pending: Mutex::new(None),\n }\n }\n}\n\n#[async_trait]\nimpl Source for MySource {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn poll(&self) -> Result {\n sleep(self.poll_interval).await; // sleep first - backpressure\n\n let mut candidate = self.state.lock().await.clone();\n\n let fetched = self.fetch_since(candidate.cursor.as_deref()).await?;\n\n let mut messages = Vec::with_capacity(fetched.len());\n let mut next_cursor = None;\n for row in fetched {\n let payload = simd_json::to_vec(&row).map_err(|e|\n Error::Serialization(format!(\"row serialize: {e}\"))\n )?;\n messages.push(ProducedMessage {\n id: Some(row.id as u128),\n checksum: None,\n timestamp: None,\n origin_timestamp: Some(row.created_at_ns),\n headers: None,\n payload,\n });\n next_cursor = Some(row.cursor_value);\n }\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} produced {} messages, next cursor: {:?}\",\n self.id, messages.len(), next_cursor\n );\n }\n\n if messages.is_empty() {\n return Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: None,\n });\n }\n\n candidate.messages_produced += messages.len() as u64;\n candidate.cursor = next_cursor;\n let persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n *self.pending.lock().await = Some(candidate);\n\n Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: Some(persisted),\n })\n }\n\n async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {\n let (SourceBatchResult::Ack, Some(candidate)) =\n (result, self.pending.lock().await.take())\n else {\n return Ok(());\n };\n *self.state.lock().await = candidate;\n Ok(())\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx pools\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, total produced: {}\",\n self.id, state.messages_produced\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySourceConfig) -> Result { /* ... */ }\n```\n","isInternal":false,"tokens":1319,"sizeBytes":5877},{"name":"SKILL.md","path":".claude/skills/connector-testing/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-testing/SKILL.md","title":"connector-testing","category":"anthropic-skill","format":"markdown","content":"---\nname: connector-testing\ndescription: Write tests for Apache Iggy connectors - sinks, sources, transforms, SDK, runtime. Covers BDD unit-test naming, the canonical source state round-trip tests, and the real-infra integration test layout under `core/integration/tests/connectors/` with `testcontainers-modules` + `#[iggy_harness]`. Load when writing or reviewing tests in `core/connectors/` or `core/integration/tests/connectors/`. Use for connector test authoring. NOT for runtime internals or non-connector test patterns.\n---\n\n# Testing Apache Iggy Connectors\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). Repo-wide testing rules (`#[iggy_harness]`, BDD naming, harness layout) live in [AGENTS.md](../../../AGENTS.md#testing). This skill covers connector-specific test conventions.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Unit tests](#unit-tests)\n- [Integration tests (real infra via Docker)](#integration-tests-real-infra-via-docker)\n- [SDK + runtime tests](#sdk--runtime-tests)\n- [CI / verification flow](#ci--verification-flow)\n- [Common pitfalls](#common-pitfalls)\n- [Examples worth copying](#examples-worth-copying)\n\n## STOP and ask the user before\n\n- Mocking the database or backend in an integration test - we hit real infra via `testcontainers-modules` (mocks miss migration bugs).\n- Adding a unit test that requires Docker - that belongs under `core/integration/tests/connectors//`, not in `/src/lib.rs`.\n- Skipping the four canonical source state tests - they're mandatory for every source plugin.\n- Naming a test `test_foo` / `does_bar` - we use BDD `given_X_when_Y_should_Z`.\n\nThere are **two distinct test layers**. Use both - they catch different bugs.\n\n| Layer | Location | What it covers | When required |\n| --------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------- |\n| **Unit tests** | `#[cfg(test)] mod tests` at EOF of `/src/lib.rs` | Config parsing, defaults, fallbacks, payload conversion, query/document building, state ser/de | Every plugin |\n| **Integration tests** | `core/integration/tests/connectors//` (uses real backend via Docker) | End-to-end: runtime + plugin + real Postgres/ES/Iceberg/Mongo/etc. + Iggy server | Every plugin that integrates with an external system |\n\n## Unit tests\n\n### Naming - BDD `given_when_should`\n\nMatch the convention used in the codebase (`sources/random_source/src/lib.rs::tests`, `sinks/postgres_sink/src/lib.rs::tests`):\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_messages_produced() { ... }\n\n#[test]\nfn given_no_state_should_start_fresh() { ... }\n\n#[test]\nfn given_invalid_duration_should_fall_back_to_default() { ... }\n\n#[test]\nfn given_payload_json_should_serialize_correctly() { ... }\n```\n\nYou may **omit one part** when the test is small (`given_X_should_Y` instead of full `given_X_when_Y_should_Z`), but **stay consistent within a file**. Imperative `test_foo` / `does_bar` names are not the convention here.\n\nA test whose subject is a **type** rather than a scenario has no \"given\" to name, and drops the prefix: it reads `_should_`. The canonical state round trip below is `state_should_be_serializable_and_deserializable` in all three shipped sources (`random_source`, `postgres_source`, `http_source`). Leave those as they are rather than bending them into the scenario form.\n\n### Prove a test can fail, before trusting it\n\nA passing test says nothing until you have watched it fail. Break the behaviour it names, run the\nsuite, and put the code back:\n\n```bash\n# commit first: this reverts with `git checkout --`, which discards\n# every uncommitted change in the file, not only the mutation\ngit status --porcelain -- # must be empty\n\ncargo nextest run -p # expect a failure naming that test\ngit checkout -- \n```\n\nTwo failures this catches, both seen in review of `http_source`:\n\n- **A fixture that makes the interesting case unreachable.** A guard test for endpoint resurrection\n passed `&[]` as the static config, so there was nothing to resurrect and the assertion held\n whatever `restore` did.\n- **An assertion aimed at something self-healing.** A test for a teardown guard asserted on gauges,\n which `Metrics::encode` rebuilds from the live instances on every scrape, so it passed with the\n guard deleted. Delete such a test rather than keep it; a test that cannot fail is worse than none,\n because it reads as coverage.\n\nWatch for an arm hidden behind a catch-all: `StatusClass::from` ends in `_ => ServerError`, so the\nredirect arm could be removed with every existing assertion still green.\n\n### `test_config()` helper\n\nEvery plugin's tests start with a small helper returning a tuned-down version of the production config. New tests extend it. don't construct from scratch each time.\n\n```rust\nfn test_config() -> RandomSourceConfig {\n RandomSourceConfig {\n interval: Some(\"100ms\".to_string()),\n max_count: Some(100),\n messages_range: Some((5, 10)),\n payload_size: Some(50),\n }\n}\n```\n\n### Async testing pattern\n\nThe trait methods are `async`. For unit tests, build a runtime locally - don't add `#[tokio::test]` in `src/lib.rs` (it pulls test macros into prod build):\n\n```rust\n#[test]\nfn given_X_should_Y() {\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n // await calls here\n });\n}\n```\n\n### Mandatory: four canonical source state tests\n\nEvery source must have these. Copy from `sources/random_source/src/lib.rs::tests`:\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_() {\n let state = State { /* fields */ };\n let serialized = rmp_serde::to_vec(&state).expect(\"Failed to serialize state\");\n let connector_state = ConnectorState(serialized);\n\n let source = MySource::new(1, test_config(), Some(connector_state));\n\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n let restored = source.state.lock().await;\n assert_eq!(restored., /* expected */);\n });\n}\n\n#[test]\nfn given_no_state_should_start_fresh() {\n let source = MySource::new(1, test_config(), None);\n /* assert default state */\n}\n\n#[test]\nfn given_invalid_state_should_start_fresh() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n let source = MySource::new(1, test_config(), Some(invalid));\n /* assert default state */\n}\n\n#[test]\nfn state_should_be_serializable_and_deserializable() {\n let original = State { /* fields */ };\n let bytes = rmp_serde::to_vec(&original).unwrap();\n let restored: State = rmp_serde::from_slice(&bytes).unwrap();\n assert_eq!(original., restored.);\n}\n```\n\nState is the only thing that survives a plugin restart. Silent corruption here means lost data on the next deploy.\n\n**The third test has two correct shapes, and which one applies is a design decision rather than a style choice.**\n\nStarting fresh is right when the state is a cursor. The worst case is re-reading from the beginning, which the at-least-once contract already permits, so discarding an undecodable cursor costs duplicates and nothing else.\n\nIt is wrong when the state carries something whose *absence* is a downgrade rather than a repeat. `http_source` keeps its revocation tombstones in state, so starting fresh after a decode failure would serve every revoked endpoint again, with its secret. Its `restore` returns `Err` instead and the connector refuses to open. A source in that position asserts the refusal:\n\n```rust\n#[test]\nfn given_invalid_state_should_refuse_to_open() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n\n let restored = MyState::restore(&config_entries(), Some(invalid), 1);\n\n assert!(\n restored.is_err(),\n \"discarding this state silently would re-serve what it was revoking\"\n );\n}\n```\n\nName the test for whichever it is and say why in a comment. What is never right is starting fresh silently when the state that was discarded was load-bearing.\n\n### Sink unit tests\n\nCover pure per-message logic:\n\n- Config defaults + optional fields + invalid values → fallback.\n- Payload conversion for each `Payload` variant the sink claims to handle.\n- Header encoding (especially base64 for binary headers).\n- Query/document building (e.g., `postgres_sink::build_batch_insert_query`).\n- Transient vs permanent error classification (e.g., `postgres_sink::is_transient_error`).\n\n### Transform tests\n\nFour branches must be covered (see `connector-transform`):\n\n1. Happy path: matching input → expected output.\n2. Filter path: input that should be dropped returns `Ok(None)`.\n3. Pass-through path: non-JSON payload (or unmatched condition) returns `Ok(Some(unchanged))`.\n4. Invalid config: `new()` returns `Err(Error::InvalidConfigValue(_))`.\n\n### What NOT to mock\n\n- The underlying client lib (sqlx, reqwest, mongodb). Tests would exercise the mock, not the code.\n- The Iggy server.\n- The runtime.\n\nIf you need a fake backend, use `wiremock` (a real local HTTP server) - see `core/integration/tests/connectors/http/`.\n\n## Integration tests (real infra via Docker)\n\nThe canonical pattern lives in `core/integration/tests/connectors/`. Plugins that integrate with external services (Postgres, Elasticsearch, Iceberg, MongoDB, InfluxDB, Quickwit, HTTP/wiremock) have a paired fixture + test file there.\n\n### Layout\n\n```text\ncore/integration/tests/connectors/\n├── mod.rs Common helpers (TestMessage, create_test_messages)\n├── fixtures/ One subdir per backend\n│ ├── postgres/\n│ │ ├── container.rs PostgresContainer (testcontainers wrapper), env-var consts\n│ │ ├── sink.rs PostgresSinkFixture, PostgresSinkByteaFixture, PostgresSinkJsonFixture\n│ │ ├── source.rs PostgresSourceJsonFixture, PostgresSourceJsonbFixture, ...\n│ │ └── mod.rs re-exports\n│ ├── elasticsearch/ iceberg/ mongodb/ influxdb/ quickwit/ http/ delta/\n│ └── wiremock.rs HTTP mocking for http_sink tests\n├── postgres/\n│ ├── postgres_sink.rs The actual #[iggy_harness] tests\n│ ├── postgres_source.rs\n│ ├── restart.rs Restart + state-survival tests\n│ ├── sink.toml Runtime config used by sink tests\n│ ├── source.toml Runtime config used by source tests\n│ └── mod.rs Shared constants (TEST_MESSAGE_COUNT, POLL_ATTEMPTS, ...)\n├── delta/ elasticsearch/ iceberg/ mongodb/ influxdb/ random/ quickwit/ http/ stdout/\n├── http_config_provider/ HTTP config provider end-to-end\n├── runtime/ Runtime-only tests (error isolation, missing plugin, ...)\n└── api/ HTTP control plane tests\n```\n\n### Container naming + cleanup\n\nEvery test container MUST be named with the `iggy-test-` prefix so a single\n`docker ps -aqf 'name=^iggy-test-'` sweep reaps them (`just clean-test-containers`,\nalias `ctc`; the `test`/`nextest` recipes also reap on exit). New fixtures get\nthis for free by naming through `fixtures::unique_container_name(\"\")`, which\nappends a uuid for parallel-safe ephemeral containers. Reuse fixtures use a fixed\n`iggy-test-` literal instead, because the stable name is what lets later\nnextest processes attach to the same container via `ReuseDirective::Always`. Do\nnot hand-roll a different prefix - the sweep only finds `iggy-test-*`.\n\nAs of now only `elasticsearch` and `doris` share a container. Both have a slow\nboot (JVM / Doris FE+BE, tens of seconds), so reusing one across the whole\nregression beats spinning a fresh one per test. Every other fixture boots a\nfresh, ephemeral container per test - cheap enough that the isolation is worth\nmore than the reuse.\n\n### The `#[iggy_harness]` proc macro\n\nEach integration test is annotated with this macro from the `integration` crate. It boots an in-process Iggy server (and optionally the connectors runtime) for the duration of the test, runs the seeds, then injects a `&TestHarness` and your fixture.\n\n```rust\nuse integration::harness::seeds;\nuse integration::iggy_harness;\n\n#[iggy_harness(\n server(connectors_runtime(config_path = \"tests/connectors/postgres/sink.toml\")),\n seed = seeds::connector_stream\n)]\nasync fn json_messages_sink_stores_as_bytea(\n harness: &TestHarness,\n fixture: PostgresSinkFixture,\n) {\n let client = harness.root_client().await.unwrap();\n let pool = fixture.create_pool().await.expect(\"Failed to create pool\");\n fixture.wait_for_table(&pool, \"iggy_messages\").await;\n // ... send messages, assert on rows\n}\n```\n\nKey pieces:\n\n- `server(connectors_runtime(config_path = ...))` boots the connectors runtime against the given TOML.\n- `seed = seeds::connector_stream` runs the named seed to create the test stream/topic.\n- `fixture: PostgresSinkFixture` - the test harness calls `PostgresSinkFixture::setup()` (which spins up the testcontainer) and injects the result.\n- `harness: &TestHarness` (defined in `core/integration/src/harness/orchestrator/harness.rs`) exposes `.root_client().await`, `.connectors_runtime().expect(\"...\").http_url()`, etc. - `connectors_runtime()` returns `Option`, so `.expect()` first.\n\n### Fixture pattern (`TestFixture` trait)\n\nEach fixture implements `integration::harness::TestFixture`. Pattern:\n\n```rust\n// core/integration/tests/connectors/fixtures//container.rs\n\nuse testcontainers_modules::{\n ,\n testcontainers::{ContainerAsync, runners::AsyncRunner},\n};\n\npub struct MyContainer {\n container: ContainerAsync<::Image>,\n pub connection_string: String,\n}\n\nimpl MyContainer {\n pub async fn start() -> Result {\n let container = ::Image::default().start().await?;\n let port = container.get_host_port_ipv4(DEFAULT_PORT).await?;\n Ok(Self {\n container,\n connection_string: format!(\"...://localhost:{port}\"),\n })\n }\n}\n```\n\nThe fixture itself wraps the container and implements `TestFixture::setup()` to spin it up and inject env vars that the runtime picks up.\n\n### Env-var injection via `ConfigEnv`\n\nThe runtime's config structs derive `ConfigEnv` (`configs_derive::ConfigEnv`, applied in `runtime/src/configs/connectors.rs` to `SinkConfig`, `SourceConfig`, `StreamConsumerConfig`, `StreamProducerConfig`). Fields are addressable by env var with a path-built prefix. Three concrete forms observed in `core/integration/tests/connectors/fixtures/postgres/container.rs`:\n\n```rust\n// Plugin-config fields (your sink/source's own config struct):\n// IGGY_CONNECTORS___PLUGIN_CONFIG_\npub const ENV_SINK_CONNECTION_STRING: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PLUGIN_CONFIG_CONNECTION_STRING\";\n\n// Indexed nested fields (per-stream entries are a Vec):\n// IGGY_CONNECTORS___STREAMS__\npub const ENV_SINK_STREAMS_0_STREAM: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_STREAMS_0_STREAM\";\n\n// Top-level scalar fields on SinkConfig/SourceConfig itself:\n// IGGY_CONNECTORS___\npub const ENV_SINK_PATH: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PATH\";\n```\n\n`` is `SINK` or `SOURCE`. `` is the connector key (TOML key field, uppercased). `PLUGIN_CONFIG` is the literal segment used when reaching into your plugin's nested config. Fields marked `#[config_env(skip)]` (e.g., `transforms`, `plugin_config` as a whole) are NOT env-addressable. primitive leaves marked `#[config_env(leaf)]` are.\n\nThe fixture sets these in `setup()` before the runtime reads its config. This is how a dynamic container port reaches a static TOML.\n\n### Test naming for integration tests\n\nIntegration tests use a **descriptive declarative** style, not the unit-test `given_should` form. Match the codebase:\n\n```rust\nasync fn json_messages_sink_stores_as_bytea(...) { ... }\nasync fn binary_messages_sink_stores_as_bytea(...) { ... }\nasync fn json_messages_sink_stores_as_jsonb(...) { ... }\nasync fn delete_after_read_source_removes_rows_after_producing(...) { ... }\nasync fn processed_column_source_marks_rows_after_producing(...) { ... }\nasync fn state_persists_across_connector_restart(...) { ... }\nasync fn restart_sink_connector_continues_processing(...) { ... }\n```\n\nPattern: `__`. Reads like a sentence describing what the test proves.\n\n### Polling vs sleeping\n\nIntegration tests poll until an expected condition - never `sleep(big_duration)`. Constants `POLL_ATTEMPTS` and `POLL_INTERVAL_MS` defined per-backend `mod.rs`. Fixture helpers like `PostgresSinkFixture::fetch_rows_as` and `PostgresSinkFixture::wait_for_table` encapsulate the pattern.\n\n### When to add an integration test\n\n- New sink/source plugin → at minimum one happy-path test per supported `Schema` and per supported plugin mode. Concrete coverage in `core/integration/tests/connectors/postgres/`: 3 sink tests (`json_messages_sink_stores_as_bytea`, `binary_messages_sink_stores_as_bytea`, `json_messages_sink_stores_as_jsonb`) backed by 3 sink fixtures, plus 5 source fixtures (`Json`, `Jsonb`, `Bytea`, `Delete`, `Mark`) and a dedicated `restart.rs` for state-survival.\n- Bug fix in a plugin → regression test reproducing the bug.\n- Behavior that crosses the FFI boundary (state restart, transform chain, schema decode) → integration test, not unit test.\n\n### When NOT to add an integration test\n\n- Pure logic (query builder, payload converter) - that's a unit test.\n- Config validation - unit test.\n- Code paths already covered by an equivalent test for another backend, when the new backend has identical behavior.\n\n## SDK + runtime tests\n\n- `core/connectors/sdk/tests/` - cross-module SDK concerns. In tree: `protobuf_integration.rs`, `flatbuffer_integration.rs` (round-trip schema decoders/encoders).\n- Runtime tests live under `core/integration/tests/connectors/runtime/` and `core/integration/tests/connectors/api/`. They exercise the runtime alone (no plugin-specific backend) - error isolation, missing plugin, invalid config, HTTP API contract.\n\n## CI / verification flow\n\nBefore pushing:\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p --all-targets -- -D warnings\ncargo test -p \n```\n\nFor integration tests, the relevant package is `integration`. There is **no `--test connectors` target** - the `integration` crate has no `[[test]]` entries and `tests/` is structured as a single binary entry through `tests/mod.rs` with `connectors/`, `server/`, `cli/`, etc. as nested modules. Run via name filter:\n\n```text\ncargo test -p integration -- connectors::postgres::postgres_sink\ncargo test -p integration -- connectors::postgres:: # all postgres tests\ncargo test -p integration -- connectors:: # all connector tests\n```\n\nIntegration tests assume Docker is available locally and pull images from a public registry on first run.\n\n## Common pitfalls\n\n1. **Skipping integration tests for a real-infra plugin.** Unit tests don't catch FFI/lifecycle/encoding regressions.\n2. **Forgetting the four canonical source state tests.** This is the most common review comment.\n3. **Mocking the client library** instead of using `testcontainers` - test exercises the mock.\n4. **`#[tokio::test]` in `src/lib.rs`** - pulls tokio test macros into prod build. Use `Runtime::new()` instead. (`#[tokio::test]` is fine in `tests/` directories, which compile separately.)\n5. **`unwrap()` in test assertions** - prefer `assert_eq!` / `expect(\"descriptive\")` so failures tell you which step blew up.\n6. **Tests depending on system time** - inject the clock or use `tokio::time::pause()`.\n7. **Hidden coupling via shared globals** (static `AtomicU64`, env vars) - tests must be independent and parallelizable.\n8. **Asserting on `Display` of an error** - use `matches!(err, Error::Variant(_))`. `assert_eq!(err.to_string(), ...)` breaks on message tweaks.\n9. **Sleeping a fixed duration** instead of polling - causes flakes when CI is slow.\n\n## Examples worth copying\n\n| Concern | Where |\n| ----------------------------------- | -------------------------------------------------------------------- |\n| Source state round-trip | `sources/random_source/src/lib.rs::tests` |\n| Config + query-building unit tests | `sinks/postgres_sink/src/lib.rs::tests` |\n| Real-infra sink integration tests | `core/integration/tests/connectors/postgres/postgres_sink.rs` |\n| Real-infra source integration tests | `core/integration/tests/connectors/postgres/postgres_source.rs` |\n| Restart + state survival | `core/integration/tests/connectors/postgres/restart.rs` |\n| Wiremock-driven HTTP sink test | `core/integration/tests/connectors/http/http_sink.rs` |\n| Testcontainer wrapper | `core/integration/tests/connectors/fixtures/postgres/container.rs` |\n| Fixture impl `TestFixture` | `core/integration/tests/connectors/fixtures/postgres/sink.rs` |\n| Runtime-only error-isolation | `core/integration/tests/connectors/runtime/error_isolation.rs` |\n| Transform tests | `sdk/src/transforms/add_fields.rs::tests`, `filter_fields.rs::tests` |\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connector-testing","description":"Write tests for Apache Iggy connectors - sinks, sources, transforms, SDK, runtime. Covers BDD unit-test naming, the canonical source state round-trip tests, and the real-infra integration test layout under `core/integration/tests/connectors/` with `testcontainers-modules` + `#[iggy_harness]`. Load when writing or reviewing tests in `core/connectors/` or `core/integration/tests/connectors/`. Use for connector test authoring. NOT for runtime internals or non-connector test patterns."},"isInternal":false,"tokens":5004,"sizeBytes":22205},{"name":"SKILL.md","path":".claude/skills/connector-transform/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connector-transform/SKILL.md","title":"connector-transform","category":"anthropic-skill","format":"markdown","content":"---\nname: connector-transform\ndescription: Add a new transform under `core/connectors/sdk/src/transforms/`. Transforms mutate, filter, or convert messages between source/encode and decode/sink. Use when adding field-level transforms (add/delete/update/filter), format conversions, envelope unwrapping, or similar config-driven message manipulations for Apache Iggy connectors. Use for transform authoring. NOT for runtime internals or sink/source loops.\n---\n\n# Adding a New Apache Iggy Connector Transform\n\nA transform implements `iggy_connector_sdk::transforms::Transform`. It\nruns after decode (for sinks) or before encode (for sources), in a chain\nconfigured per-connector. Config-driven, composable.\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers transform authoring only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [The trait](#the-trait)\n- [Steps to add a new transform](#steps-to-add-a-new-transform)\n- [Rules](#rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Adding a new transform that mutates payload SHAPE (renaming PK fields, dropping schema-required columns) - breaks downstream sinks silently.\n- Introducing non-determinism (system time, RNG without seed) - breaks tests + replay.\n- Changing the `Transform` trait signature (sync vs async, return type) - cascades to every transform.\n- Making transforms hold mutable runtime state - they're constructed once and `Arc`-cloned across tasks. mutation requires interior locking and is rarely what you actually want.\n\n## File map\n\n```text\nsdk/src/transforms/\n├── mod.rs Transform trait, TransformType enum, from_config() factory,\n│ FieldValue + ComputedValue helpers\n├── add_fields.rs AddFields - append static or computed fields to JSON\n├── delete_fields.rs DeleteFields - remove fields by key\n├── update_fields.rs UpdateFields - conditional field replacement\n├── filter_fields.rs FilterFields - keep/drop messages by key/value pattern\n├── unwrap_envelope.rs UnwrapEnvelope - promote nested field to top-level payload\n├── proto_convert.rs ProtoConvert - bidirectional Proto ⇄ Json/Text/Raw/etc.\n├── flatbuffer_convert.rs FlatBufferConvert\n├── avro_convert.rs AvroConvert\n└── json/ Shared JSON helpers (path traversal, ops)\n```\n\n## The trait\n\n```rust\n// sdk/src/transforms/mod.rs - Transform\npub trait Transform: Send + Sync {\n fn r#type(&self) -> TransformType;\n fn transform(\n &self,\n metadata: &TopicMetadata,\n message: DecodedMessage,\n ) -> Result, Error>;\n}\n```\n\nSemantics:\n\n- **`&self`** - transforms are immutable post-construction. Internal state (compiled regex, parsed paths, schema descriptors) is computed in `new()` and stored as plain struct fields. No locks needed - the whole instance is shared via `Arc`.\n- Takes `message` **by value** - the transform owns the `DecodedMessage` and may mutate freely without cloning.\n- Returns `Ok(Some(msg))` to pass through (possibly modified), `Ok(None)` to **filter** (drop silently), `Err(e)` for actual failures.\n- `&TopicMetadata` is for routing-aware logic. Most transforms ignore it.\n\n## Steps to add a new transform\n\n### 1. Create the file\n\n`sdk/src/transforms/my_transform.rs`. Apache 2.0 license header. Code reads top to bottom - config struct, then transform struct, then `impl`, then helpers.\n\n```rust\n/* Apache 2.0 header */\n\nuse crate::transforms::{Transform, TransformType};\nuse crate::{DecodedMessage, Error, Payload, TopicMetadata};\nuse serde::{Deserialize, Serialize};\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\npub struct MyTransformConfig {\n pub key: String,\n pub option: Option,\n}\n\npub struct MyTransform {\n config: MyTransformConfig,\n}\n\nimpl MyTransform {\n pub fn new(config: MyTransformConfig) -> Result {\n if config.key.is_empty() {\n return Err(Error::InvalidConfigValue(\n \"my_transform.key must not be empty\".into(),\n ));\n }\n Ok(Self { config })\n }\n}\n\nimpl Transform for MyTransform {\n fn r#type(&self) -> TransformType {\n TransformType::MyTransform\n }\n\n fn transform(\n &self,\n _metadata: &TopicMetadata,\n mut message: DecodedMessage,\n ) -> Result, Error> {\n // Most transforms operate only on Payload::Json - non-JSON is a no-op.\n let Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n };\n\n // Mutate `value` in place via simd_json::OwnedValue API.\n // Return Ok(None) to filter, Ok(Some(message)) to pass through.\n\n Ok(Some(message))\n }\n}\n```\n\n### 2. Wire into `transforms/mod.rs`\n\nThree edits, all required:\n\n(a) Module declaration + re-export at the top:\n\n```rust\nmod my_transform;\npub use my_transform::{MyTransform, MyTransformConfig};\n```\n\n(b) Discriminant in `TransformType`:\n\n```rust\npub enum TransformType {\n AddFields,\n // ...\n MyTransform, // snake_case form in TOML: \"my_transform\"\n}\n```\n\n(c) Branch in `from_config()`:\n\n```rust\nTransformType::MyTransform => {\n // For new transforms, prefer Error::InvalidConfigValue with formatted context -\n // it preserves the underlying serde_json error. Existing code uses bare\n // Error::InvalidConfig for most variants (UnwrapEnvelope is the exception).\n let cfg: MyTransformConfig = serde_json::from_value(raw.clone())\n .map_err(|error| Error::InvalidConfigValue(format!(\"my_transform: {error}\")))?;\n Ok(Arc::new(MyTransform::new(cfg)?))\n}\n```\n\n`from_config()` returns `Arc` so the same instance is shared across the pipeline.\n\n### 3. Document\n\nAdd a TOML usage example to `sdk/README.md`:\n\n```toml\n[transforms.my_transform]\nenabled = true\nkey = \"some_field\"\noption = true\n```\n\n### 4. Tests\n\nBottom of `my_transform.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn given_matching_field_should_transform() { ... }\n\n #[test]\n fn given_non_json_payload_should_passthrough() { ... }\n\n #[test]\n fn given_filter_condition_should_drop_message() { /* assert Ok(None) */ }\n\n #[test]\n fn given_invalid_config_should_error_in_new() {\n let cfg = MyTransformConfig { key: \"\".into(), option: None };\n assert!(matches!(MyTransform::new(cfg), Err(Error::InvalidConfigValue(_))));\n }\n\n fn topic_meta() -> TopicMetadata {\n TopicMetadata { stream: \"s\".into(), topic: \"t\".into() }\n }\n\n fn json_message(json: serde_json::Value) -> DecodedMessage { /* ... */ }\n}\n```\n\n## Rules\n\n### Operate on JSON only (mostly)\n\nMost field-level transforms only make sense for `Payload::Json`. Convention (see `add_fields.rs::AddFields::transform`): silently pass through non-JSON:\n\n```rust\nlet Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n};\n```\n\nFormat-conversion transforms (`proto_convert`, `avro_convert`, `flatbuffer_convert`) are the exception - they explicitly convert between variants.\n\n### Mutate `OwnedValue` in place, don't clone-then-replace\n\n`simd_json::OwnedValue` is mutable in place:\n\n```rust\nif let OwnedValue::Object(ref mut obj) = value {\n obj.insert(key.into(), new_value);\n}\n```\n\nDo NOT clone the value, mutate the clone, then `message.payload = Payload::Json(new)`. The whole point of `&mut value` is in-place mutation.\n\n### Filter semantics\n\n`Ok(None)` drops the message **silently** - no error log, no metric. By design (filters succeed when they decide to drop). For visibility, log at `debug!` level.\n\n### Config validation\n\nValidate in `new()` and return `Err(Error::InvalidConfigValue(...))`. Don't `panic!`. Don't validate on the hot path.\n\nFor transforms with regex or compiled patterns, compile once in `new()` and store the compiled form.\n\n### Stateless except for config\n\nTransforms are constructed once at startup and `Arc`-cloned across tasks. Must be `Send + Sync`. No mutable runtime state - if you need accumulator-like behavior, that's not a `Transform`, that's a sink.\n\n### Compiled state goes in the struct, not behind a lock\n\nThings derived from config (compiled regex, parsed paths, schema descriptors) should be computed in `new()` and stored as plain struct fields. The whole instance is shared via `Arc`, so per-field locking is unnecessary. See `sdk/src/transforms/filter_fields.rs::FilterFields::new` for the canonical pattern: regexes compile in `new()`, get stored as struct fields, are then read via `&self`.\n\n### `transform()` is sync\n\nThe trait method is sync (not `async`). Don't propose making it async - that would require pervasive change across the runtime. Schemas and other external resources should be loaded in `new()` (potentially via a fallible `try_new`), not lazily during `transform()`.\n\n### Errors\n\n| Scenario | Variant |\n| ---------------------------------- | -------------------------------------------------- |\n| Config validation in `new()` | `Error::InvalidConfigValue(\"transform_name: ...\")` |\n| Per-message validation (bad shape) | `Error::InvalidRecordValue(\"...\")` |\n| Schema resolution failure | `Error::InvalidTransformer` |\n| Format conversion failure | `Error::InvalidPayloadType` |\n\nReturning `Err` from `transform()` stops the message and is logged. For non-fatal field-level issues (field missing when transform expects it), prefer pass-through (`Ok(Some(message))`) unless config explicitly says to fail.\n\n## Common pitfalls\n\n1. **Forgetting one of the three `mod.rs` edits** - module declaration, `TransformType` variant, `from_config()` branch.\n2. **`#[serde(rename_all = \"snake_case\")]` mismatch** between enum variant name and TOML key.\n3. **Cloning `OwnedValue`** when in-place mutation would work.\n4. **Validating config inside `transform()`** instead of `new()`.\n5. **`std::sync::Mutex` for internal state** on the hot path - precompute in `new()` and store as plain struct fields. Transforms run on every message. locks are wasted work.\n6. **Returning `Err` for a filter** - filters use `Ok(None)`.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n```\n\nVerify by adding the transform to `core/connectors/runtime/example_config/connectors/` and running the runtime - the transform should apply to each message.\n\nIf the transform changes payload shape sufficiently to break downstream sinks (e.g., renaming primary keys), call it out in the PR description.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connector-transform","description":"Add a new transform under `core/connectors/sdk/src/transforms/`. Transforms mutate, filter, or convert messages between source/encode and decode/sink. Use when adding field-level transforms (add/delete/update/filter), format conversions, envelope unwrapping, or similar config-driven message manipulations for Apache Iggy connectors. Use for transform authoring. NOT for runtime internals or sink/source loops."},"isInternal":false,"tokens":2539,"sizeBytes":11088},{"name":"SKILL.md","path":".claude/skills/connectors-overview/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/iggy/HEAD/.claude/skills/connectors-overview/SKILL.md","title":"connectors-overview","category":"anthropic-skill","format":"markdown","content":"---\nname: connectors-overview\ndescription: Entry point and index for the Apache Iggy Connectors subsystem (`core/connectors/`). Load this first when working anywhere under `core/connectors/` - runtime, SDK, sinks, sources, or transforms. Routes to the focused per-area skills.\n---\n\n# Apache Iggy Connectors - Overview\n\nRepo-wide rules (Apache headers, fmt/sort/clippy order, idiomatic Rust traits, imports at top, no `cargo install`, LLM-slop tells) live in [AGENTS.md](../../../AGENTS.md). This file owns **connector-specific** rules and routes to the per-area skills.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [What sinks and sources are](#what-sinks-and-sources-are)\n- [Which skill to load](#which-skill-to-load)\n- [Stick to conventions](#stick-to-conventions)\n- [Connector-wide rules](#connector-wide-rules)\n- [Benchmark mode (per-batch timing observability)](#benchmark-mode-per-batch-timing-observability)\n- [Logging format](#logging-format)\n- [Exemplars](#exemplars)\n- [Concrete efficiency patterns](#concrete-efficiency-patterns)\n- [Drop accounting](#drop-accounting)\n- [Common review smells](#common-review-smells)\n- [File map](#file-map)\n- [Cite by symbol, not line number](#cite-by-symbol-not-line-number)\n\n## STOP and ask the user before\n\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI signature in `sdk/src/{sink,source}.rs` - breaks every pre-built plugin `.so`.\n- Changing the runtime's wire conventions (postcard FFI payload structs, default consumer group naming, plugin path resolution).\n- Modifying `runtime/src/state.rs` save protocol (atomic rename + fsync ordering) - corruption risk.\n- Renaming or repurposing a `Schema` variant - decoders/encoders pinned to wire bytes.\n- Promoting a transient `Error` variant to `Permanent*` or vice versa - affects retry behavior across every plugin.\n\n## What sinks and sources are\n\nA **sink** is a plugin that **consumes** messages from Apache Iggy streams and writes them to an external system (Postgres, Elasticsearch, Iceberg, HTTP endpoint, Mongo, object store, stdout, ...). Implements `iggy_connector_sdk::Sink`.\n\nA **source** is a plugin that **produces** messages from an external system into Apache Iggy streams (poll a DB table, scroll an ES index, generate random data, ...). Implements `iggy_connector_sdk::Source`.\n\nBoth compile as `cdylib` shared libraries (`.so`/`.dylib`/`.dll`) loaded by the **connectors runtime** at startup via `dlopen`. The runtime drives lifecycle (open/handle/consume/close), bridges Apache Iggy ⇄ plugin via a small FFI, applies optional transforms, and persists source state.\n\n```text\n ┌─ optional transforms ─┐\nExternal ──poll──▶ SOURCE ──FFI──▶ RUNTIME ──encode──▶ Apache Iggy stream\n system plugin ▲ ▲\n │ │\n state save (msgpack) │\n │\n ┌─ optional transforms ─┐ │\nApache Iggy stream ──decode──▶ RUNTIME ──FFI──▶ SINK ──write──▶ External\n plugin system\n```\n\nHeaders set on the source side ride through transforms (which may modify, drop, or pass them) and arrive at the sink with `BTreeMap` preserved deterministically.\n\n## Which skill to load\n\n| Task | Skill |\n| --------------------------------------------------- | --------------------- |\n| Write a new sink plugin | `connector-sink` |\n| Write a new source plugin | `connector-source` |\n| Add schema / decoder / encoder / SDK trait surface | `connector-sdk` |\n| Change runtime internals (FFI, manager, state, ...) | `connector-runtime` |\n| Add a transform (field-level or format conversion) | `connector-transform` |\n| Write unit / integration tests for any of the above | `connector-testing` |\n\n## Stick to conventions\n\nThe connectors codebase is intentionally repetitive across plugins. Cross-plugin consistency makes review, LLM-assisted contribution, and onboarding tractable.\n\n1. **Find the closest existing plugin by shape** (DB-write -> `postgres_sink`, HTTP -> `http_sink`, polling source -> `postgres_source`).\n2. **Copy its file layout, naming, log format, error mapping, test structure** verbatim. Only the backend-specific code (the client call) should differ.\n3. **Don't reinvent existing patterns.** The SDK's `retry.rs` covers HTTP retry + circuit breaker. Existing plugins cover batching, idempotency, header encoding, secret handling.\n4. **Don't refactor unrelated code in the same PR.** Match conventions. propose convention changes separately.\n\n## Connector-wide rules\n\n### Hot loops\n\n- **Zero clone.** Use `&self`, `&str`, `&[T]`, `&Payload`. For `Payload::Json`, `try_to_bytes(&self)` serializes single-pass without cloning the `OwnedValue` tree.\n- **Move-out without cloning** via `std::mem::replace` / `std::mem::take`.\n- **`Vec::with_capacity(n)`** when building per-batch buffers. reallocation in hot path is wasted work.\n\n### Async + concurrency\n\n- **`async fn` only.** Never `.block_on()` inside a `Sink`/`Source` impl - the runtime drives the executor.\n- **Hold locks briefly.** Pattern: lock, clone/read what's needed, drop guard, then do I/O. Never hold a `Mutex` across an external `.await`.\n- **Atomic counters** (`AtomicU64`) preferred over `Mutex` for hot-path metrics.\n- **No `tokio::spawn` from plugin code.** Runtime owns lifecycle. orphans survive `close()`.\n\n### Logging\n\n- `tracing` only. `info!` for lifecycle, `debug!` per-batch detail, `warn!` recoverable, `error!` failures. Include `connector ID: {self.id}` and the connector name on every line.\n- **`verbose: bool`** on `SinkConfig`/`SourceConfig` (TOML top-level). Runtime upgrades its own per-batch `debug!` to `info!` when set. Plugins should mirror with a `verbose_logging: Option` field inside `plugin_config` (convention: `postgres_sink::PostgresSinkConfig::verbose_logging`). Default both to `false`.\n- **Never log secrets** (connection strings, API keys, tokens). Redact URLs at the log site. Credential fields must use `SecretString` (see below).\n\n### Secrets\n\nAny credential-bearing field (connection strings, API keys, bearer tokens, AWS keys) must be `SecretString` from the `secrecy` crate. Plain `String` for a credential is a review-blocker: `SecretString` redacts on `Debug`, so it is what keeps a credential out of a log line that formats the whole config.\n\n**`serde_secret::serialize_secret` EXPOSES the secret. It does not redact.** It calls `expose_secret()` and writes the plaintext. `SecretString` deliberately has no `Serialize` impl, and that absence is the protection - so adding `serialize_with` is what *unblocks* the derive and turns a compile-time guarantee into plaintext output. Use it only where the plaintext is the point: a wire payload, a persisted config, an API response that exposes credentials by design.\n\nSo the default for a plugin config struct is **derive `Deserialize`, but not `Serialize`**. `Deserialize` is required: the SDK glue deserializes the config into the plugin's own struct (`sdk/src/{sink,source}.rs` call `serde_json::from_str::` under a `DeserializeOwned` bound).\n\nWhat never happens is the return trip. The runtime holds plugin configuration as a `serde_json::Value` - parsed from TOML, posted as JSON to the control API, or injected by env var - and hands that across the FFI, so nothing re-serializes the plugin's struct. Leaving `Serialize` off makes that compiler-enforced instead of convention-enforced (`sources/http_source/src/lib.rs::HttpSourceConfig` does this, and comments the omission so nobody adds it back).\n\nPattern:\n\n```rust\nuse secrecy::{ExposeSecret, SecretString};\n\n// `Deserialize` only. Nothing re-serializes a plugin config, and leaving\n// `Serialize` off is what makes the credential unserializable rather than\n// merely un-serialized.\n#[derive(Debug, Clone, Deserialize)]\npub struct MyConfig {\n pub connection_string: SecretString,\n}\n\n// At call site:\nlet pool = PgPoolOptions::new()\n .connect(self.config.connection_string.expose_secret())\n .await?;\n```\n\nIf a config struct genuinely needs `Serialize`, `serde_secret::serialize_redacted` (and `serialize_optional_redacted`) write `[REDACTED]` in place of the value. Reach for `serialize_secret` only when the caller must get the real thing back. The sinks and sources listed below predate that helper and use the exposing one; the annotation is inert today, but it is not the protection it looks like.\n\nNote that none of this protects the credential from the runtime's own control API, which returns plugin configuration verbatim - see #3802. Plugin-side annotations are inert there because the runtime never routes through them.\n\nPlugin-side uses of the exposing helpers: `sinks/{postgres,mongodb,elasticsearch,influxdb,s3,surrealdb}_sink`, `sources/{postgres,elasticsearch,influxdb}_source`.\n\nThat list is plugin-side only, not an inventory of every caller in the tree, and the others are not all mistakes: `runtime/src/api/config.rs` puts `serialize_secret` on `HttpConfig::api_key` (inert for the same reason), and several `core/common` wire-payload types (login, create-user, change-password, PAT) use these helpers by design, because there the credential *is* the payload.\n\n### Errors\n\n- **Use `Error::PermanentHttpError` / `Error::SchemaMismatch` / `Error::CatalogCommitError` for non-retryable failures.** Returning a transient variant for bad data trips circuit breakers and hammers the backend.\n- **No `unwrap()` / `expect()`** on Results from external I/O outside tests.\n\n### Forward-compat config\n\nNew TOML fields use `#[serde(default)]` or `Option`. Adding a field must not break existing configs.\n\n## Benchmark mode (per-batch timing observability)\n\nEvery sink and source supports an opt-in `benchmark: bool` flag (`SinkConfig`/`SourceConfig`, also env `IGGY_CONNECTORS___BENCHMARK=true`). Two independent surfaces:\n\n1. **Tracing events** (only when flag is on) - one structured `info!` per batch under target `iggy_connectors::benchmark` with per-stage microsecond timings:\n - Sink: `connector_type, connector_key, stream, topic, partition_id, current_offset, batch_size, processed_count, decode_us, prepare_us, ffi_us, total_us`\n - Source: `connector_type, connector_key, stream, topic, batch_size, sent_count, decode_us, prepare_us, iggy_send_us, state_saved, state_save_us, total_us` (`state_save_us` is 0 when `state_saved` is false - emitted as flat number + bool, not an Option, so the JSON layer keeps it numeric)\n - Filter live: `RUST_LOG=iggy_connectors::benchmark=info`\n - Also emitted on startup: a one-shot `\"Benchmark mode enabled for ...\"` info log per connector.\n\n2. **Prometheus histograms** (always on, regardless of flag):\n - Metric: `iggy_connector_stage_duration_seconds{connector_key, connector_type, stage}` (both labels snake_case)\n - `stage` (snake_case): `decode`, `prepare`, `total` on both sides; `ffi` (sink only); `iggy_send`, `state_save` (source only). `decode` and `prepare` mean the same thing on both sides - decode is schema decoding, prepare is transform + encode/serialize.\n - Buckets: see `runtime/src/metrics.rs::STAGE_BUCKETS_SECONDS`\n - Example: `histogram_quantile(0.95, sum(rate(iggy_connector_stage_duration_seconds_bucket{stage=\"ffi\"}[5m])) by (le, connector_key))`\n\nThe flag only gates the verbose text event. the histogram observation is always on. Per-stage call path: `Instant::now()` + SipHash over `StageLabels` + parking_lot `RwLock::read` + `HashMap` probe + linear scan inside `Histogram::observe`. Benchmark on target hardware if you suspect the overhead matters at your batch rate.\n\n`iggy_connector_messages_filtered_total{connector_key, connector_type}` counts intentional drops by transforms returning `Ok(None)`, distinct from `errors_total` for unexpected drops. Both also exposed in `/stats` JSON (`ConnectorStats.messages_filtered`, `.errors`).\n\nImplementation: `runtime/src/benchmark.rs` (text-event emitters +\n`as_micros`), `runtime/src/metrics.rs::{Stage, SinkLabels, SourceLabels,\nobserve_stage_with_labels}` (pre-built label cache avoids per-batch\n`String` clones. `&str`-based wrappers are `#[cfg(test)]` only). Tests:\n`runtime/src/metrics.rs::tests::given_stage_histogram_*` (unit),\n`core/integration/tests/connectors/runtime/benchmark.rs` (integration:\ntext emission, env override, disabled-flag, histograms via `/metrics`,\nJSON log format, parser unit tests).\n\n## Logging format\n\n`runtime/src/configs/runtime.rs::LoggingConfig` exposes `format = \"text\" | \"json\"` (text default), env `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `runtime/src/log.rs::init_logging` matches on `(telemetry.enabled, format)` to install the right `fmt::layer()`/`fmt::layer().json()` combined with the OpenTelemetry layer when telemetry is on. JSON applies to the stdout layer only - OTel export pipeline unaffected by this flag.\n\n## Exemplars\n\n| Purpose | Plugin |\n| --------------------------------------------------------- | --------------------------------------------------------------------------------------- |\n| Simplest sink (read first) | `sinks/stdout_sink/` |\n| Real-infra sink + integration tests | `sinks/postgres_sink/` + `integration/tests/connectors/postgres/postgres_sink.rs` |\n| Feature-rich sink config (validation, batch modes, retry) | `sinks/http_sink/` |\n| Atomic counters on hot path | `sinks/mongodb_sink/` |\n| Simplest source (4 canonical state tests) | `sources/random_source/` |\n| Real-infra source | `sources/postgres_source/` + `integration/tests/connectors/postgres/postgres_source.rs` |\n\nRead the relevant exemplar end-to-end before writing or modifying a connector.\n\n## Concrete efficiency patterns\n\nEach implemented in at least one in-tree plugin or runtime path.\n\n| Pattern | Where | When |\n| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- |\n| `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))` | `sinks/http_sink/src/lib.rs` (`send_individual`, `send_ndjson`, `send_json_array`) | Take payload from `&mut ConsumedMessage` no clone |\n| `std::mem::take(&mut batch)` | `runtime/src/sink.rs::consume_messages`, `manager/{sink,source}.rs` | Drain a `Vec` field with one move |\n| `std::mem::swap` | `sinks/http_sink` (retry_delay / max_retry_delay validation) | Fix config-field ordering in `new()` |\n| `Vec::with_capacity(messages.len())` | Every batching sink | Pre-size per-batch buffers |\n| `bytes::Bytes::from(body)` | `sinks/{http,influxdb}_sink` | Cheap ref-counted body sharing across retries |\n| `AtomicU64` counters | `sinks/mongodb_sink` | Lock-free per-message metric updates |\n| `Payload::try_to_bytes(&self)` | `sdk/src/lib.rs::Payload::try_to_bytes` | Serialize `Payload::Json` no-clone |\n| `simd_json::to_owned_value(&mut bytes)` | `sdk/src/decoders/json.rs::JsonStreamDecoder::decode` | In-place JSON parse |\n| Brief-lock fetch | `sources/postgres_source/src/lib.rs::poll_tables` | Lock read -> drop -> I/O -> lock write |\n| `is_transient_error(&e)` SQLSTATE mapping | `sinks/postgres_sink/src/lib.rs` | Driver errors -> retry vs `PermanentHttpError` |\n| Two-constructor pattern (`new` lenient + `try_new` strict + `Default`) | `sdk/src/decoders/avro.rs::AvroStreamDecoder` | Stateful decoders/encoders where schema can fail |\n| Duplicate-ID FFI guard | `sdk/src/sink.rs::sink_connector!`, `sdk/src/source.rs::source_connector!` | Prevents silent data loss on reopen-without-close |\n| `restart_guard.try_lock()` | `runtime/src/manager/{sink,source}.rs::restart_connector` | No thundering-herd restarts |\n| `tokio::time::timeout(..., handle).await` + `handle.abort()` on timeout | `runtime/src/manager/source.rs::stop_connector` | Bounded shutdown + leak prevention |\n| `flume::unbounded()` channel | `runtime/src/source.rs::spawn_source_handler` / `source_forwarding_loop` | MPSC handoff from SDK async task to runtime loop |\n| `tokio::sync::watch::channel(())` | `sdk/src/{sink,source}.rs`, `runtime/src/sink.rs`, `runtime/src/manager/*` | One-shot shutdown broadcast |\n| `dashmap::DashMap` | `runtime/src/manager/sink.rs`, `source.rs::SOURCE_SENDERS`, SDK `INSTANCES` | Lock-free concurrent keyed access |\n| `secrecy::SecretString` + `iggy_common::serde_secret::serialize_secret` | `sinks/postgres_sink::PostgresSinkConfig::connection_string` | `Debug` redacts; `serialize_secret` EXPOSES |\n\n## Drop accounting\n\n- **Filter drops** (transform returning `Ok(None)`) bump `messages_filtered`.\n- **Every non-filter drop** bumps `errors`. Inside the per-batch loops the count is accumulated in a local and flushed once after the loop via `inc_errors_by_with_labels` (filters likewise via `inc_messages_filtered_with_labels`) - one `Family` lookup per batch, not per message. One-shot drops outside a loop call `inc_errors_with_labels` directly. Hot-path calls always use the pre-built `SinkLabels`/`SourceLabels` cache. `&str`-based wrappers on `Metrics` are `#[cfg(test)]`-only.\n\n## Common review smells\n\n- `.clone()` on `Payload::Json` or `OwnedValue` - use `try_to_bytes(&self)`.\n- `&mut self` on `Sink::consume` or `Source::poll` impls - won't compile, flag any creative workaround.\n- `std::sync::Mutex` held across `.await` - swap for `tokio::sync::Mutex`.\n- Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` in plugin `Cargo.toml`.\n- Source plugin without the four canonical state tests (see `connector-testing`).\n- New silent message drop without a metric increment.\n- Wrapping `format!()` around args passed to `error!`/`warn!`/`info!`/`debug!` - eager `format!` allocates even when level filters the line out. Pass args directly: `error!(\"foo: {x}\")` or `error!(error = %x, \"foo\")`.\n- Logging a connection string, API key, or token.\n- Plain `String` for a credential field - use `SecretString`.\n- `tokio::spawn` inside plugin code - runtime owns lifecycle.\n- `std::time::SystemTime::now()` in transforms - non-deterministic, breaks tests.\n\n## File map\n\n```text\ncore/connectors/\n├── README.md\n├── runtime/\n│ ├── src/\n│ │ ├── main.rs FFI structs (SinkApi, SourceApi), plugin path resolution\n│ │ ├── manager/{sink,source}.rs Lifecycle, restart_guard, status transitions\n│ │ ├── configs/ Local + HTTP config providers, ConfigEnv derive\n│ │ ├── {sink,source,stream,transform,state,context}.rs\n│ │ ├── benchmark.rs, log.rs, metrics.rs\n│ │ └── api/ HTTP control plane (/sinks, /sources, /stats, /metrics)\n│ └── example_config/ Reference TOML\n├── sdk/\n│ └── src/\n│ ├── lib.rs Sink/Source traits, Payload, Schema, Error\n│ ├── {sink,source}.rs FFI containers + sink_connector!/source_connector! macros\n│ ├── decoders/encoders/ Per-schema (json/raw/text/proto/flatbuffer/avro)\n│ ├── transforms/ add/delete/update/filter fields, format conversions\n│ └── retry.rs CircuitBreaker, HttpRetryMiddleware\n├── sinks/_sink/\n└── sources/_source/\n\ncore/integration/tests/connectors/ # Real-infra integration tests\n├── fixtures/ # testcontainers-modules wrappers per backend\n├── postgres/, elasticsearch/, mongodb/, iceberg/, influxdb/, quickwit/, http/, delta/\n├── runtime/ # error_isolation, benchmark\n└── api/ # HTTP API endpoint tests\n```\n\n## Cite by symbol, not line number\n\nSkills cite files by path + symbol (`PostgresContainer::start` in `core/integration/tests/connectors/fixtures/postgres/container.rs`). Line numbers go stale on the first refactor. symbol names survive. Grep the symbol if you need the exact spot.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n","frontmatter":{"name":"connectors-overview","description":"Entry point and index for the Apache Iggy Connectors subsystem (`core/connectors/`). Load this first when working anywhere under `core/connectors/` - runtime, SDK, sinks, sources, or transforms. Routes to the focused per-area skills."},"isInternal":false,"tokens":4912,"sizeBytes":22416}],"systemPromptSnippet":"\n\n# Apache Iggy Agent Guidelines\n\n**Apache Iggy** is a persistent message streaming platform\nin Rust. Thread-per-core shared-nothing, `io_uring` + `compio`.\nTransports: QUIC, WebSocket, TCP (custom binary), HTTP (REST). SDKs:\nRust, .NET, Java, Python, Go, C++, Node.js. A connectors subsystem\ningests from / egresses to external systems via dlopened plugins.\n\n> Skills live under `.claude/skills/`, with per-skill links under\n> `.agents/skills/` for Codex. Load\n> [connectors-overview](.claude/skills/connectors-overview/SKILL.md)\n> first for any change under `core/connectors/`. Other subsystems\n> follow the repo-wide principles in this file.\n\n## Contents\n\n- [Apache Iggy Agent Guidelines](#apache-iggy-agent-guidelines)\n - [Contents](#contents)\n - [STOP and ask the user before](#stop-and-ask-the-user-before)\n - [Helping someone contribute](#helping-someone-contribute)\n - [Quick reference](#quick-reference)\n - [Structure](#structure)\n - [Where to look](#where-to-look)\n - [Skills](#skills)\n - [Repo-wide principles](#repo-wide-principles)\n - [Tooling](#tooling)\n - [Testing](#testing)\n - [Pitfalls](#pitfalls)\n - [Local state directories](#local-state-directories)\n - [Commits, PRs, releases](#commits-prs-releases)\n - [Discussion and support](#discussion-and-support)\n\n## STOP and ask the user before\n\n- Writing code for an issue not assigned to the user\n (`gh issue view --json assignees`). If it isn't, or you can't\n confirm it, they must ask on the issue and wait to be assigned.\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI\n signature in `core/connectors/sdk/src/{sink,source}.rs` (breaks\n every pre-built plugin .so).\n- Modifying `core/binary_protocol/` wire-format types tagged\n `#[repr(C)]` (breaks every running client).\n- Touching the state-save path (`core/connectors/runtime/src/state.rs`)\n in a way that changes the atomic-rename protocol (corruption risk).\n- Renaming default consumer groups, plugin path resolution rules, or\n any other operator-facing identifier.\n- Force-pushing to `master`, amending shipped commits, or skipping\n hooks with `--no-verify` / `--no-gpg-sign`.\n\n## Helping someone contribute\n\nMost PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.\n\n- The PR must link an issue the user was assigned to before any work started. See the STOP list above.\n- Fill in every section of the PR template. Under AI Usage, name the agent and say what it wrote. Never leave the section out or write \"None\" when an agent wrote any of the change.\n- Run `prek run` before the PR is opened. If prek isn't installed, ask the user to install it (`cargo install prek`, then `prek install`). Don't substitute the individual checks.\n- If a check can't run, tell the user which one and why, and treat the PR as not ready. Never write text in a PR asking a maintainer to run something.\n- The user should write the rationale in their own words and be able to explain every change without asking you.\n- New contributors keep one open PR at a time.\n- If you found a bug, the first PR holds a failing test. The fix waits until the approach is agreed on the issue.\n\n## Quick reference\n\n```bash\n# Verification order is enforced by CI (.github/actions/rust/pre-merge).\ncargo fmt --all # step 1\ncargo sort --no-format --workspace # step 2 (--no-format mandatory)\ncargo clippy --all-features --all-targets -- -D warnings # step 3\ncargo test -p # step 4\n\n# Other CI checks (mirror in pre-commit; pass --fix where supported)\n./scripts/ci/taplo.sh # TOML\n./scripts/ci/markdownlint.sh # Markdown\n./scripts/ci/shellcheck.sh # Shell\n./scripts/ci/license-headers.sh # Apache headers\n./scripts/ci/trailing-whitespace.sh\n./scripts/ci/trailing-newline.sh\n./scripts/ci/sync-rustc-version.sh\n./scripts/ci/python-sdk-version-sync.sh\n./scripts/ci/uv-lock-check.sh\n./scripts/ci/binary-artifacts.sh --check\n./scripts/ci/third-party-licenses.sh --validate --manifest \n\n# Run things\ncargo run --bin iggy-server -- --with-default-root-credentials\ncargo run --bin iggy -- \nIGGY_CONNECTORS_CONFIG_PATH=... cargo run --bin iggy-connectors\n./scripts/run-bdd-tests.sh\n./scripts/run-benches.sh\n```\n\nPre-commit hooks wire the same scripts. Install with `pre-commit\ninstall` (Python) or `prek install` (Rust drop-in) - both read\n`.pre-commit-config.yaml`.\n\n## Structure\n\n```text\niggy/\n├── core/\n│ ├── server/ Iggy server binary (Viewstamped Replication)\n│ ├── sdk/ Rust client SDK\n│ ├── cli/ iggy CLI\n│ ├── connectors/ Connectors runtime + SDK + sinks/sources\n│ ├── common/ Shared types (IggyDuration, IggyTimestamp, ...)\n│ ├── binary_protocol/ Wire format (stable)\n│ ├── configs/ Config plumbing + ConfigEnv derive\n│ ├── ai/mcp/ MCP server\n│ ├── bench/ Benchmark suite\n│ ├── integration/ Cross-crate integration tests\n│ ├── simulator/ Deterministic simulator\n│ ├── consensus/ Raft (Miri-checked unsafe iobuf)\n│ ├── journal/ WAL / journal\n│ ├── message_bus/ In-process bus\n│ ├── shard/ Shard runtime\n│ ├── harness_derive/ #[iggy_harness] proc macro\n│ └── metadata, partitions, server_common, configs_derive, clock, tools\n├── foreign/ Java / .NET / Python / Go / C++ / Node.js SDKs\n├── bdd/ Cross-language BDD (Gherkin)\n├── examples/ Per-SDK runnable examples\n├── helm/ Helm charts\n├── web/ Web UI\n└── scripts/ CI + release + dev scripts\n```\n\n## Where to look\n\n| Task | Location |\n| --------------------- | ---------------------------------------- |\n| Wire protocol | `core/binary_protocol/` |\n| Server | `core/server/src/` |\n| Rust client SDK | `core/sdk/src/` |\n| Connectors | `core/connectors/` -> connector-* skills |\n| Integration tests | `core/integration/tests/` |\n| Cross-SDK BDD | `bdd/` |\n| Benchmark suite | `core/bench/` |\n\n## Skills\n\nEach `SKILL.md` has YAML frontmatter (name, description). For\nconnectors work, load `connectors-overview` first as router.\nClaude Code uses `/skill-name`; Codex uses `$skill-name` or `/skills`.\nKeep one source under `.claude/skills/` and link its directory from\n`.agents/skills/`. Keep templates and references inside the source directory.\n\n- [connectors-overview](.claude/skills/connectors-overview/SKILL.md) - router + universal connector rules\n- [connector-runtime](.claude/skills/connector-runtime/SKILL.md) - FFI host, lifecycle, state, metrics\n- [connector-sdk](.claude/skills/connector-sdk/SKILL.md) - traits, decoders/encoders, retry\n- [connector-sink](.claude/skills/connector-sink/SKILL.md) - sink plugin authoring\n- [connector-source](.claude/skills/connector-source/SKILL.md) - source plugin authoring\n- [connector-transform](.claude/skills/connector-transform/SKILL.md) - transform authoring\n- [connector-testing](.claude/skills/connector-testing/SKILL.md) - unit + integration test patterns\n\nReview skills are user-invoked only. Keep `disable-model-invocation: true`\nfor Claude Code and `policy.allow_implicit_invocation: false` in\n`agents/openai.yaml` for Codex. Do not replicate their steps without an\nexplicit invocation. When a non-trivial change passes verification,\nsuggest `/team-review ` in Claude Code or `$team-review `\nin Codex.\n\n- [connector-review](.claude/skills/connector-review/SKILL.md) - adversarial connectors PR/branch review with validation\n- [team-review](.claude/skills/team-review/SKILL.md) - adversarial 4-expert PR/branch review, ~10 subagents per run\n- [team-review-slim](.claude/skills/team-review-slim/SKILL.md) - four-expert review without independent validation rounds\n\nReview skills share [execution instructions](.claude/skills/team-review/references/execution.md)\nfor target preparation and client-specific agent tools. Run\n`./scripts/ci/skills.sh` to check skill metadata, links, and invocation policies.\n\n## Repo-wide principles\n\n1. **Apache 2.0 header on every new source file:** Follow the comment style configured in `licenserc.toml`; common examples: `.rs` uses `// ...`, `Cargo.toml` and shell use `# ...`.\n2. **Verification order:** `fmt --all` -> `sort --no-format --workspace` -> `clippy --all-features --all-targets -- -D warnings` -> `test`. Plus `taplo.sh --check`. CI enforces all.\n3. **Always pass `--no-format` to `cargo sort`.** Without it multi-line feature arrays flatten and `taplo` re-expands them, churning the workspace.\n4. **Avoid LLM-slop tells: em dashes, gratuitous semicolons, hedging narrative, trailing summaries.** Defaults, not absolute bans. Same rule for code, comments, commits, PR descriptions.\n5. **Idiomatic Rust over free helpers.** `FromStr` not `parse_foo`, `Display` not `format_foo`, `From`/`TryFrom` not `to_foo_from_bar`. Structured `Err` enums, not `String`.\n6. **Imports at the top.** No inline `use` inside functions or blocks. Group: std, external crates, then `crate::`.\n7. **Precise names.** No `b`, `p`, `t`, `m`. Match literal API field names as log labels.\n8. **Comments explain WHY, not WHAT.** Default to no comment. Add one only when a hidden constraint or workaround would surprise a future reader. Never reference the current task / PR.\n9. **Never `cargo install`** without authorization. Toolchain pinned in `rust-toolchain.toml`. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check.\n10. **No `unwrap()` / `expect()` on Results from external I/O** outside tests.\n11. **`tokio::sync::Mutex` (not `std::sync::Mutex`)** wherever a lock is held across `.await`. Hold locks briefly: lock, clone, drop guard, then do I/O.\n12. **Code reads top to bottom.** Public consts and public fns first, helpers below, deeper helpers below those.\n13. **Zero clone in hot loops.** `&self`, `&str`, `&[T]`. `Vec::with_capacity(n)` when size is known.\n14. **Forward-compatible config.** New TOML fields use `#[serde(default)]` or `Option`.\n\n## Tooling\n\n| Tool | Where |\n| --------------------------------------------- | ------------------------------------------- |\n| `cargo fmt`, `cargo sort`, `cargo clippy` | pre-commit + CI |\n| `cargo machete` | CI `machete` task |\n| `cargo nextest` + `cargo-rail` | CI `test-*` DAG-scoped |\n| `cargo llvm-cov` | CI coverage |\n| `cargo http-registry` | CI `verify-publish` |\n| Miri | CI `miri` (`binary_protocol` + `consensus`) |\n| `taplo`, `markdownlint-cli`, `shellcheck` | pre-commit + CI |\n| `typos` | pre-commit |\n| `cargo about` + `license-checker-rseidelsohn` | `scripts/ci/third-party-licenses.sh` |\n| HawkEye / `license-headers.sh` | pre-commit + CI |\n| `uv`, `ruff` | pre-commit (Python) |\n| `golangci-lint` | pre-push (Go) |\n\n## Testing\n\n- **Unit:** `#[cfg(test)] mod tests` at EOF of `src/lib.rs` (or per-module).\n- **In-crate integration:** `/tests/`.\n- **Cross-crate (real infra):** `core/integration/tests/`. Harness in\n `core/integration/src/harness/` spawns real `iggy-server`\n (`ServerHandle`), connectors runtime (`ConnectorsRuntimeHandle`),\n MCP server (`McpHandle`), client (`ClientHandle`). `#[iggy_harness]`\n proc macro wires fixtures. Suites: `cli/`, `cluster/`,\n `config_provider/`, `connectors/`, `data_integrity/`, `mcp/`, `sdk/`,\n `server/`, `state/`, `storage/`. External backends inside\n `connectors/` use `testcontainers-modules`. Per-test logs land in\n `test_logs/`.\n- **Cross-SDK BDD:** `bdd/`. Gherkin features against every SDK. Driver: `scripts/run-bdd-tests.sh`.\n\nBDD test naming: `given_X_when_Y_should_Z` (3-part). Promote to 4-part `given_X_when_Y_then_Z_should_W` only with a distinct \"then\" intermediate. Be consistent inside a file.\n\nFilter a single integration test by path. Build the binaries the harness launches first: it runs\nwhatever is already in `target/`, so a stale one fails as though the change under test broke it.\n\n```bash\ncargo build --bin iggy-server --bin iggy-connectors\ncargo test -p integration -- connectors::runtime::benchmark::given_logging_format_json\n```\n\n## Pitfalls\n\n- **Docker is required for most integration tests.** `testcontainers-modules` spawns containers per test.\n- **`cargo sort` without `--no-format` will churn the workspace.** Use the pre-commit hook.\n- **Rust toolchain is pinned in `rust-toolchain.toml`.** Older system `cargo` fails on transitive deps. prefer rustup so the pin auto-resolves.\n- **`test_logs/` grows quickly.** Wipe between major refactors.\n- **Miri only covers `binary_protocol` + `consensus`.** Cannot emulate `io_uring` syscalls. do not try to expand Miri to crates that pull `compio`.\n- **Integration crate has no `--test` target.** Filter by test path inside the single `mod.rs` binary.\n- **The integration harness runs prebuilt binaries and never builds them.** `Command::cargo_bin` only resolves a path, so a stale `iggy-server`, `iggy-connectors` or `iggy-mcp` is used silently. CI restores revision-checked artifacts from the runtime and Rust test builds; a local run does not build them, and a stale binary fails in ways that look like the change under test. Rebuild after anything that touches a launched binary or its config.\n\n## Local state directories\n\nCreated by server, connectors runtime, or tests. Gitignored. Safe to delete to reset:\n\n- `local_data/` - server data\n- `local_state/` - connectors runtime state files\n- `test_logs/` - per-test logs + `local_data` snapshots\n- `performance_results/` - benchmark output\n- `target/` - cargo build\n\n## Commits, PRs, releases\n\n- Conventional commits with scope: `feat(connectors): ...`. Subject <= 72 chars. Allowed types/scopes enforced by `.github/workflows/pr-title.yml`.\n- Body: why before what. No bullet lists of files changed.\n- Co-authored-by trailer when applicable.\n- Workspace + SDK versions sync via `scripts/extract-version.sh` and `scripts/ci/{sync-rust,python-sdk}-version{,-sync}.sh`.\n- Crate publish chain verified by `scripts/verify-crates-publish.sh` against `cargo-http-registry`.\n- Third-party license enumeration: `scripts/ci/third-party-licenses.sh` (ASF policy).\n\n## Discussion and support\n\n- Design: \n- Chat: \n- Docs: \n\n\n---\nname: connector-runtime\ndescription: Modify the Apache Iggy Connectors runtime (`core/connectors/runtime/`) - the host process that loads plugins, manages lifecycle, and bridges Apache Iggy streams to plugins via FFI. Use when changing plugin loading, the FFI dispatch, the sink/source manager, state storage, config providers, the HTTP control API, metrics, or transforms wiring. NOT for plugin code.\n---\n\n# Modifying the Apache Iggy Connectors Runtime\n\nRuntime is the host. Loads `.so`/`.dylib`/`.dll` plugins, drives their\nlifecycle, ferries messages between Apache Iggy and plugins, exposes\n`/stats`, `/metrics`, control HTTP API. Bugs here affect every plugin.\n\n> **Universal connector rules** (benchmark, SecretString, verbose, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers runtime internals only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Architecture invariants](#architecture-invariants)\n- [State storage (`state.rs`)](#state-storage-staters)\n- [Source forwarding loop](#source-forwarding-loop)\n- [Sink consumption loop](#sink-consumption-loop)\n- [Logging format](#logging-format)\n- [The `verbose` flag](#the-verbose-flag)\n- [Config provider rules](#config-provider-rules)\n- [Error categorization](#error-categorization)\n- [Metrics (`metrics.rs`)](#metrics-metricsrs)\n- [Drop accounting (wired sites)](#drop-accounting-wired-sites)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing FFI `#[repr(C)]` layouts (`SourceApi`, `SinkApi`, `RawMessages`, headers) - existing plugin `.so`s misalign.\n- Modifying `state.rs` save protocol (atomic tmp + rename + fsync ordering) - corruption risk.\n- Renaming the default consumer group `iggy-connect-sink-{key}` - operators hold offsets there.\n- Hot-reloading or unloading a `.so` - mid-FFI tasks would segfault. Architectural change required.\n- Changing the SerDe split (postcard for FFI payloads, serde_json for configs, rmp_serde for state) - downstream incompat.\n\n## File map\n\n```text\nruntime/src/\n├── main.rs Entry, plugin path resolution, SourceApi/SinkApi FFI structs\n├── sink.rs Sink lifecycle, Iggy consumer wiring, FFI consume calls\n├── source.rs Source lifecycle, flume forwarding, state save loop\n├── stream.rs Stream + consumer/producer setup\n├── transform.rs Loads transforms from config, applies them in chain\n├── state.rs FileStateProvider, atomic ConnectorState file I/O\n├── context.rs Per-instance context (Iggy client, metrics, config)\n├── error.rs RuntimeError with as_code() for HTTP responses\n├── log.rs init_logging (text/JSON x telemetry-on/off matrix) + LOG_CALLBACK\n├── benchmark.rs emit_sink_event / emit_source_event + as_micros\n├── metrics.rs Prometheus families + gauges + stage histograms + label caches\n├── stats.rs /stats endpoint payload assembly\n├── manager/{mod,sink,source}.rs SinkManager / SourceManager, status, restart_guard\n├── configs/\n│ ├── runtime.rs RuntimeConfig + LoggingConfig + LogFormat\n│ ├── connectors.rs ConnectorConfig, SinkConfig, SourceConfig (verbose + benchmark)\n│ └── connectors/{local,http}_provider.rs\n└── api/ HTTP control + observability endpoints\n```\n\n## Architecture invariants\n\nLoad-bearing assumptions. Breaking them silently corrupts data or hangs.\n\n### Plugin instance identity\n\n- Each instance gets a globally unique `plugin_id: u32` from `PLUGIN_ID: AtomicU32` in `main.rs`. Never reused.\n- Same `.so` may load many instances. each gets a fresh `plugin_id` but shares the `Container` (dlopen handle) via `Arc` (`manager/{sink,source}.rs`).\n- All FFI calls keyed by `plugin_id`. Mixing IDs = data to the wrong plugin.\n\n### FFI pointer lifetimes\n\n- Config and state are `(*const u8, usize)`. Valid for the call duration only. Plugin must copy if it needs the bytes past the call. SDK macros do this correctly. don't bypass.\n- `LogCallback` lifetime spans the whole process - kept static in `log.rs`. Don't make it dynamic.\n\n### Container ownership\n\n- `dlopen2::wrapper::Container<{Sink,Source}Api>` (defined in `main.rs`) is shared via `Arc` across all instances of the same `.so`.\n- Arc must outlive every spawned task that calls into the plugin. Manager holds the Arc in `{Sink,Source}Details` for the connector lifetime.\n- Unloading a `.so` while a task is mid-FFI-call would segfault. Runtime does not unload. Hot reload needs a rearchitect.\n\n### Plugin manager state\n\n- `SinkManager` / `SourceManager` use `DashMap>>`. Key = TOML `key` field.\n- Status transitions (`Starting → Running → Stopped/Error`) update running counters only on transitions to/from `Running`. See `SinkManager::update_status`.\n- `restart_guard: Arc>` prevents concurrent restart of the same connector. `restart_connector()` uses `try_lock()` - returns OK without restart if busy. Intentional anti-thundering-herd.\n\n### Serialization split\n\n- FFI message payloads (`TopicMetadata`, `MessagesMetadata`, `RawMessages`, headers): `postcard` (compact, stable wire).\n- Connector configs passed to plugin: `serde_json` (TOML in, JSON over FFI - human-editable).\n- `ConnectorState` bytes inside plugins: `rmp_serde` (MessagePack - compact, opaque to runtime).\n\nDon't mix.\n\n## State storage (`state.rs`)\n\n- File path: `{state_path}/source_{connector_key}.state`. Mode `0o600` on Unix.\n- `save()` is crash-atomic. `rename(2)` is atomic in the namespace (no observer sees a half-renamed file). the preceding `sync_data` on the tmp file plus the post-rename parent-dir `sync_all` are what make a crash leave either the old or new content - never truncated. The parent-dir fsync failure is propagated as `CannotWriteStateFile` (not swallowed) so a lost rename surfaces instead of silently restarting the source from scratch. A `Mutex<()>` serializes concurrent saves on the same provider.\n- Save after every successful Apache Iggy send. Save failure logs + continues. next batch retries.\n- `load()` returns `Ok(None)` for missing or empty files. After a `NotFound` read, re-stats the parent directory: missing parent -> `Err(CannotOpenStateFile)` so a broken state path fails at init rather than masquerading as \"fresh start\".\n- Sinks have no state - only sources.\n\n## Source forwarding loop\n\n1. `iggy_source_handle(id, send_callback)` - plugin registers itself.\n2. Plugin polls + invokes `send_callback(plugin_id, ptr, len)`.\n3. Callback runs in the SDK macro's spawned async task. Pushes postcard `ProducedMessages` into a `flume` channel keyed by `plugin_id` in `SOURCE_SENDERS: Lazy>` (`pub(crate)`). `SourceSenderEntry` wraps the sender + a pre-extracted owned `Counter` (the `errors` series, `Arc` inside). The FFI callback bumps errors on deserialize or channel-closed failure with one relaxed atomic - no `Family` lookup, no `Arc` handle.\n4. The runtime registers the optional source-stop callback, then `source_forwarding_loop` pulls from the channel, deserializes, applies transforms, encodes via `StreamEncoder`, and sends to the Iggy producer. The SDK warns after 30 seconds without a result but keeps the same batch pending; it does not poll or enqueue another copy.\n5. On success, save returned `ConnectorState` via `FileStateProvider`, then return the batch result to the plugin. A result NACK may cause replay with capped backoff.\n6. A stop-callback reason ends the forwarding loop and reports an unexpected stop. Closing the stop channel on normal shutdown disables that select arm, allowing already queued batches to receive results before the forwarding channel disconnects.\n7. At loop exit, clean up `SOURCE_SENDERS` and update status and metrics. A self-stop reports `Error` and removes the source from the running gauge; normal shutdown reports `Stopped`.\n\n**Shutdown ordering (`manager/source.rs::stop_connector`):**\n\n1. Call `iggy_source_close` FIRST. It blocks until the plugin's polling task stops, so no new send callbacks fire after it returns.\n2. `cleanup_sender(plugin_id)` NEXT - dropping the channel sender lets the forwarding task finish any queued batches before `recv_async()` resolves with `Disconnected` and exits cleanly, instead of blocking until the abort timeout.\n3. Finally await spawned handlers with `tokio::time::timeout`. On timeout, `handle.abort()` + drain - prevents leaked tasks colliding with the next `start_connector` (a late `file.save()` could otherwise race the new instance). The silent-drop branch in `handle_produced_messages` only covers the window between close and cleanup.\n\nGotchas:\n\n- `SOURCE_SENDERS` must be cleaned up on connector close or memory leaks (channel + task).\n- `spawn_source_handler` wraps the outer `iggy_source_handle(id, callback)` FFI call in `tokio::task::spawn_blocking()`. That call returns quickly - the SDK macro internally `runtime.spawn`s an async `handle_messages` task and returns. **`send_callback` invocations come from that async task, not from `spawn_blocking`.** A long-running synchronous poll inside the plugin would block one Tokio worker. async polls don't.\n- No timeout on the registration call - a plugin whose `iggy_source_handle` never returns stalls one blocking worker for the process lifetime.\n\n## Sink consumption loop\n\nPer `[[streams]]` entry per sink:\n\n1. Build / ensure consumer group `iggy-connect-sink-{key}` (`connect` matches the docker image name `apache/iggy-connect` - see `runtime/src/sink.rs::default_consumer_group`). Overridable via the stream config's `consumer_group`.\n2. Spawn one task per topic (`spawn_consume_tasks`).\n3. Poll Iggy → batch messages (default `batch_length = 1000`, `poll_interval = 5ms`). A `consumer.next()` `Err` (transport/decode at the Iggy client boundary) bumps `errors` and `continue`s - offset already auto-committed via `AutoCommit::When(AutoCommitWhen::PollingMessages)`, message effectively dropped.\n4. Decode via `Schema::decoder()`. Decode failure logs + bumps `errors`. `messages_filtered` is reserved for the transform filter contract (`Ok(None)`).\n5. Apply transforms (chain). A transform `Err` is logged, bumps `errors`, and drops only that message (drop-and-continue, mirroring the source) - one bad payload never kills the batch. `Ok(None)` filters the message - bumps `messages_filtered{connector_type=\"sink\"}`.\n6. Postcard-encode batch as `RawMessages`. Headers as a separate postcard blob. Per-message failures (missing field, payload conversion, header serialization) bump `errors` + skip. Batch-level failures (postcard serialization of metadata / `RawMessages`) propagate `Err` -> `spawn_consume_tasks` wrapper bumps `errors` + `set_error`. The total histogram and benchmark emit still fire for the failed batch before the `Err` propagates.\n7. Call `iggy_sink_consume(id, topic_meta, messages_meta, messages)`.\n8. Non-zero return → log + `errors` increment.\n\nPer-message counters (decode errors, transform errors, filters, field/serialize drops) are accumulated and flushed once per batch via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels`, not one `Family` lookup per message.\n\nStages (both labels snake_case): `decode` (schema decode loop), `prepare` (transform + serialize = total - decode - ffi), `ffi` (plugin consume call), `total`. Same `decode` / `prepare` meaning as the source side.\n\nGotchas:\n\n- Batch flushes on `current_offset != message_offset` (gap) OR batch full - higher latency for low-volume topics.\n- Auto-commit happens on poll - idempotency at the sink protects against re-delivery on crash before plugin write.\n\n## Logging format\n\n`LoggingConfig` in `configs/runtime.rs` exposes `format: LogFormat` (`Text` default, `Json`), env-addressable via `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `log::init_logging` matches on `(telemetry.enabled, format)` and installs the right `fmt::layer()`/`fmt::layer().json()` + OpenTelemetry layer. OTel pipeline untouched by `format` - only the stdout layer switches.\n\n`fmt::layer()` and `fmt::layer().json()` are different concrete types, so the stdout layer is built once as a `Box>` (via `.boxed()`) by matching on `format`, then a single 2-arm branch on `telemetry.enabled` attaches the OTel layers. When extending, keep the boxed-layer + 2-arm shape rather than re-expanding into a 4-way match.\n\n## The `verbose` flag\n\n`SinkConfig` and `SourceConfig` in `configs/connectors.rs` carry `verbose: bool` (default false). Runtime threads it to `spawn_consume_tasks` / source equivalent to gate per-batch log promotion:\n\n```rust\nif verbose {\n info!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n} else {\n debug!(\"Processing {messages_count} messages for sink connector with ID: {plugin_id}\");\n}\n```\n\nNew per-batch logging follows the same pattern. Default to `debug!`, upgrade to `info!` when `verbose`. No third level.\n\n## Config provider rules\n\n### Local (`configs/connectors/local_provider.rs`)\n\n- Discovery walk reads `*.toml` from `config_dir`. Skips hidden files (`.`) and `Cargo.toml`.\n- Filename: `{key}_{type}[_v{N}].toml` (e.g., `postgres_sink_v2.toml`).\n- One TOML deserialized into `SinkConfig` or `SourceConfig`. Grouped by `key`. Highest `version` wins unless `.active_versions.toml` overrides.\n- `.active_versions.toml` is NOT read via the discovery walk (hidden). Read via direct path through `LocalConnectorsConfigProvider::active_versions_file_path`.\n\n### HTTP (`configs/connectors/http_provider.rs`)\n\n- `reqwest_middleware` + `RetryTransientMiddleware`. Retries 5xx + connection errors. Never 4xx (`PermanentHttpError` convention).\n- Default URL templates documented in `runtime/README.md`. Configurable via TOML.\n\n### Env-var overrides via `ConfigEnv` derive\n\n`SinkConfig`, `SourceConfig`, and inner structs derive `ConfigEnv` (`configs_derive::ConfigEnv`). Generates env-var addressability as `IGGY_CONNECTORS___` for primitive fields. Used heavily by integration tests to inject testcontainer ports - see `core/integration/tests/connectors/fixtures/postgres/container.rs` for env-var constants. Mark new compound fields `#[config_env(skip)]`, leaf primitives `#[config_env(leaf)]`.\n\n### Versioning\n\n- Configs carry `version: u64`. Local provider auto-increments if absent. HTTP provider trusts the server.\n- Restart on version change uses `restart_guard`.\n\n## Error categorization\n\n`RuntimeError` (`error.rs`) carries `as_code()` for HTTP API responses.\n\n| Class | Fatal? | Example |\n| ------------------ | --------------------------- | -------------------------------- |\n| Config load | yes (process exit) | Bad TOML, missing required field |\n| Iggy client init | yes | Auth failure at startup |\n| State dir create | yes | Permission denied |\n| Plugin .so resolve | per-plugin (`FailedPlugin`) | Missing file |\n| Plugin open FFI | per-plugin (status = Error) | Plugin returned non-zero |\n| Message decode | per-message (skip) | Bad protobuf bytes |\n| Iggy send | per-batch (metric) | Network blip |\n| State save | per-batch (log) | Disk full |\n\nFatal errors propagate to `main` and exit. Per-connector / per-message errors are isolated.\n\n## Metrics (`metrics.rs`)\n\nAll families labeled by `connector_key` + `connector_type` (histogram adds `stage`):\n\n- **Counters**: `iggy_connector_messages_{produced,sent,consumed,processed,filtered}_total` and `iggy_connector_errors_total`. These are the *rendered* names; each is registered without the `_total`, which the OpenMetrics encoder appends.\n - `messages_filtered_total` - intentional drops via transform `Ok(None)`.\n - `errors_total` - unexpected drops (decode/encode/build failure, missing field, ...) + batch-level failures, one per failed run.\n - `sink_runs_total` - FFI `consume()` calls for sink batches, one per contiguous payload-variant run. Divide by `stage_duration_seconds_count{stage=\"total\"}` for runs per batch.\n- **Histograms**: `iggy_connector_stage_duration_seconds{stage}` (snake_case stage labels - `prepare`, `ffi`, `decode`, `iggy_send`, `state_save`, `total`). Buckets `STAGE_BUCKETS_SECONDS`. Always populated regardless of any flag. One sample per batch for every stage; `ffi` is summed over the batch's runs. Scraped at `/metrics` when `[http.metrics] enabled = true`.\n- **Gauges**: `iggy_connectors_{sources,sinks}_{total,running}`.\n\n`/stats` JSON surface mirrors counters per-connector via `ConnectorStats` (`sdk/api.rs`): `messages_filtered`, `errors`, kind-specific counters.\n\nWhen adding a metric:\n\n- Add family to `Metrics` struct + `init`, register with name + help text. Never end a `Counter` family's registered name in `_total`: the encoder appends it and the series renders `_total_total`. Gauges get no suffix, so a gauge name may end in `_total` literally.\n- New label sets define `EncodeLabelSet` struct + label enum (hand-impl `EncodeLabelValue` for snake_case values - the derive emits PascalCase).\n- Histograms: pass `fn() -> Histogram` to `Family::new_with_constructor`.\n- Add unit tests under `mod tests` with `given_*_when_*_should_*` BDD names.\n\n## Drop accounting (wired sites)\n\nPer-message drops in the batch loops are counted into a local `u64` and flushed once after the loop via `inc_errors_by_with_labels` / `inc_messages_filtered_with_labels` - one `Family` lookup per batch, not per message. One-shot drops outside a loop (e.g. `consumer.next()` Err) still call `inc_errors_with_labels` directly. Hot path uses pre-built `SinkLabels`/`SourceLabels`. `&str`-based wrappers are `#[cfg(test)]`-only.\n\n- `sink.rs::consume_messages` - `consumer.next()` Err\n- `sink.rs::process_messages` - decode, transform Err (drop-and-continue), missing required fields, payload conversion, header serialization (accumulated, flushed once per batch)\n- `sink.rs::spawn_consume_tasks` task wrapper - bumps once on `consume_messages` Err\n- `source.rs::source_forwarding_loop` - payload decode, prepare (transform/encode) failure, Iggy send Err, state save Err\n- `source.rs::process_messages` - transform Err (logs + bumps `errors` + continue. does NOT propagate, so one bad payload doesn't flip the connector to permanent ERROR), transform encode failure, `build_iggy_message` failure\n- `source.rs::handle_produced_messages` - postcard deserialize failure, `sender.send` channel-closed\n\nFilter case bumps `messages_filtered` via `inc_messages_filtered_with_labels`. Adding a new drop path: mirror this pattern.\n\n## Hard rules\n\n1. **Never block the executor.** All I/O async. `spawn_blocking` only for FFI registration (already in place).\n2. **Plugin ID counter is monotonic.** No reset, no reuse.\n3. **FFI return codes:** `0` success, non-zero failure.\n4. **Don't add static mutable state** beyond `LOG_CALLBACK`, `PLUGIN_ID`, `SOURCE_SENDERS`.\n5. **Pair `cleanup_sender(id)` with shutdown** for sources (avoid flume leak). Order: close FFI -> cleanup sender -> drain/abort tasks.\n6. **Restart uses `restart_guard.try_lock()`** - no thundering-herd regression.\n7. **No timeouts on plugin FFI calls without a kill-task strategy.** A timeout that returns from the runtime but leaves the plugin running has the worst of both worlds.\n\n## Common pitfalls\n\n1. Mutating `INSTANCES` (in SDK macro) from runtime side - impossible by design. only the macro touches it.\n2. Calling `iggy_*_open` with a duplicate ID - returns `-1`. Means runtime forgot to `iggy_*_close` first.\n3. Holding a `DashMap` shard guard across `.await` - lockups. Pattern: `.get().clone()`, drop guard, await.\n4. Adding fields to `#[repr(C)]` types without an SDK version bump - existing plugins misalign on `postcard::from_bytes`.\n5. Changing the default consumer group name without coordinating with operators - they have offsets stored there.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy-connectors --all-targets -- -D warnings\ncargo test -p iggy-connectors\n\n# Sanity-build in-tree plugins:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n\n# Runtime-focused integration tests:\ncargo test -p integration -- connectors::runtime::\n\n# Smoke-test with example config:\nIGGY_CONNECTORS_CONFIG_PATH=core/connectors/runtime/example_config/config.toml \\\n cargo run --bin iggy-connectors\n```\n\nUpdate `runtime/README.md` if endpoints, env vars, or config schema change.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n\n\n---\nname: connector-sdk\ndescription: Extend or modify the Apache Iggy Connectors SDK (`core/connectors/sdk/`). Use when adding a new `Schema` variant, a new `StreamDecoder`/`StreamEncoder`, a new `Error` variant, modifying the `Sink`/`Source`/`Transform` trait surface, the FFI macros, or the `retry`/`api`/`convert` modules. NOT for writing plugins - use `connector-sink` or `connector-source` for those.\n---\n\n# Extending the Apache Iggy Connectors SDK\n\nThe SDK (`core/connectors/sdk/`) is the **stable contract** between the\nruntime and every plugin. Changes ripple to every sink/source in-tree\nand every third-party plugin. Treat the public surface as a versioned API.\n\n> **Universal connector rules** (benchmark, SecretString, drop accounting, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers SDK-level changes only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [Cardinal rules](#cardinal-rules)\n- [Adding a new `Schema` variant](#adding-a-new-schema-variant)\n- [Adding a `StreamDecoder` / `StreamEncoder`](#adding-a-streamdecoder--streamencoder)\n- [Adding a new `Error` variant](#adding-a-new-error-variant)\n- [Modifying `Sink` / `Source` / `Transform` traits](#modifying-sink--source--transform-traits)\n- [FFI macros (`sink_connector!`, `source_connector!`)](#ffi-macros-sink_connector-source_connector)\n- [`Payload::try_to_bytes(&self)` is non-negotiable for JSON](#payloadtry_to_bytesself-is-non-negotiable-for-json)\n- [Retry helpers (`retry.rs`)](#retry-helpers-retryrs)\n- [`ConnectorState`](#connectorstate)\n- [Tests for SDK changes](#tests-for-sdk-changes)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Bumping the SDK MAJOR version or changing any `#[repr(C)]` layout - misaligns every pre-built plugin `.so`.\n- Changing `Sink` / `Source` / `Transform` / `StreamDecoder` / `StreamEncoder` trait signatures - cascades through every plugin.\n- Removing or renaming a `Schema` variant or `Error` variant - decoders/encoders pinned to it. downstream pattern matches break.\n- Changing wire serialization (postcard / rmp_serde / serde_json) for any FFI / state / config payload.\n- Changing FFI macro return codes or function names (`iggy_*_open` / `consume` / `handle` / `close`) - runtime + plugins both need lockstep update.\n\n## File map\n\n```text\nsdk/src/\n├── lib.rs Traits (Sink, Source, StreamDecoder, StreamEncoder),\n│ Payload, Schema, ConnectorState, Error enum, FFI message structs.\n├── sink.rs SinkContainer + sink_connector! macro (FFI plumbing).\n├── source.rs SourceContainer + source_connector! macro (FFI plumbing).\n├── api.rs ConnectorStatus, ConnectorStats (feature = \"api\").\n├── convert.rs owned_value_into_serde_json / owned_value_to_serde_json (simd_json ⇄ serde_json bridge).\n├── log.rs CallbackLayer for tracing across FFI.\n├── retry.rs retry_async + RetryPolicy, CircuitBreaker, HttpRetryMiddleware.\n├── decoders/ One per schema: json, raw, text, proto, flatbuffer, avro.\n├── encoders/ Mirror of decoders.\n└── transforms/ add_fields, delete_fields, update_fields, filter_fields,\n unwrap_envelope, proto_convert, flatbuffer_convert, avro_convert.\n```\n\n## Cardinal rules\n\n1. **Apache 2.0 license header** on every new file.\n2. **`Send + Sync`** on every public trait (FFI runs across thread pools).\n3. **`#[repr(C)]`** on every type that crosses FFI (`Schema`, `TopicMetadata`, `MessagesMetadata`, `RawMessage`, `ProducedMessages`, `ConsumedMessage`, `DecodedMessage`, etc. in `lib.rs`). Adding fields requires bumping the SDK version - existing plugins built against the old layout will misalign on `postcard::from_bytes`.\n4. **postcard** for FFI message serialization (handled by `SinkContainer::consume`, `SourceContainer`'s `handle_messages`). **MessagePack (`rmp_serde`)** for `ConnectorState`. **JSON** (`serde_json`) only for human-editable config that crosses FFI. Don't mix.\n5. **`simd_json::OwnedValue`** for JSON payloads, not `serde_json::Value`. Use `convert::owned_value_into_serde_json` as a bridge when you own the value (the common case - it moves strings instead of copying them); fall back to the borrowing `convert::owned_value_to_serde_json` only when you still need the `OwnedValue` afterward.\n6. **`BTreeMap`** for headers - deterministic ordering. Never `HashMap` on the wire.\n7. **No breaking changes** to `Sink`/`Source`/`StreamDecoder`/`StreamEncoder`/`Transform` trait signatures without coordinating with all in-tree plugins in the same PR.\n8. **A `Schema` variant that changes meaning is as breaking as a new variant.** The discriminant crosses FFI unchanged, so a plugin built against the old SDK keeps decoding it the old way and misreads every batch with no error anywhere. Same process as a trait change: bump the SDK minor version, update every in-tree plugin in the same PR, and state in the PR description that plugins built against the previous SDK must be rebuilt.\n\n## Adding a new `Schema` variant\n\nThe `Schema` enum (in `lib.rs`) is `#[repr(C)]` - it crosses FFI. Touch points to update **in one PR**:\n\n1. `Schema` enum: add variant with `#[strum(to_string = \"...\")]` matching the `serde(rename_all=\"snake_case\")` form.\n2. `Payload` enum: add a matching variant carrying the deserialized form.\n3. `Payload::try_into_vec` - consuming bytes-out path.\n4. `Payload::try_to_bytes` - **borrowing** bytes-out path. For non-trivial payloads (parsed trees), implement a no-clone serialization, not a `clone() + serialize`. See the `Payload::Json` arm for the canonical optimization.\n5. `Payload::schema()` - variant → `Schema`. The runtime tags every sink batch from this, so a missing arm will not compile but a wrong one silently mislabels the batch.\n6. `Payload::Display`.\n7. `Schema::try_into_payload` - bytes → `Payload`, for a tag naming a wire format (the source path).\n8. `Payload::try_from_schema` - bytes → `Payload`, for a tag that came from `Payload::schema()` (the sink path). The exact inverse of item 5.\n9. `Schema::decoder()` - factory returning `Arc`.\n10. `Schema::encoder()` - factory returning `Arc`.\n11. New `decoders/.rs` and `encoders/.rs`.\n12. Update the schema list in `sdk/README.md`.\n13. Tests: round-trip encode/decode, error paths. `Payload::schema()` must round-trip through `Payload::try_from_schema`, with no exceptions: every variant survives that trip and the test is table-driven over all of them.\n\nMiss any of these → silent failures at runtime (typically `Error::InvalidPayloadType` or surprising decoder behavior in plugin code).\n\n## Adding a `StreamDecoder` / `StreamEncoder`\n\nPattern from `decoders/json.rs::JsonStreamDecoder`:\n\n```rust\npub struct MyStreamDecoder; // unit struct if stateless\n\nimpl StreamDecoder for MyStreamDecoder {\n fn schema(&self) -> Schema { Schema::MyFormat }\n fn decode(&self, payload: Vec) -> Result {\n // Decode payload bytes into Payload::MyFormat(...).\n // Errors: Error::CannotDecode(Schema::MyFormat), Error::InvalidJsonPayload, etc.\n }\n}\n```\n\nFor **stateful** decoders (proto/avro/flatbuffer) holding schema descriptors:\n\n- Store the schema state directly in the struct (`config`, `message_descriptor`, `schema`). The instance is built once and wrapped in `Arc` for sharing - it must be `Send + Sync` but does not need additional locking if construction-time loading is final.\n- Provide **two constructors** when schema loading can fail: lenient `new(config) -> Self` (logs + degrades) AND strict `try_new(config) -> Result` (fail-fast). Plus a `Default` impl for the no-schema case. Canonical example: `sdk/src/decoders/avro.rs` exports all three (`AvroStreamDecoder::new`, `::try_new`, `Default`).\n- If you genuinely need to mutate the schema after construction (e.g., a `update_config` method), use `&mut self` plus `std::mem::replace(&mut self.config, new_config)` to swap without cloning the old value. Live pattern: `sdk/src/decoders/avro.rs::AvroStreamDecoder::update_config` and `sdk/src/encoders/avro.rs::AvroStreamEncoder::update_config`.\n- A fresh decoder instance is created via `Schema::decoder()` on each call (it returns `Arc`), so per-decoder caching of shared mutable state is the wrong abstraction.\n\n## Adding a new `Error` variant\n\nThe `Error` enum (in `lib.rs`) is `Clone + PartialEq + Eq + Hash`. New variants must preserve these. Guidelines:\n\n- **Use existing variants first.** Only add a new one if the failure mode is distinct in handling, not just description. `InvalidConfigValue(String)` covers most \"bad config\" cases.\n- **Carry context as `String`** - don't add a struct unless multiple discrete fields are needed by callers programmatically.\n- **Document retry semantics in the docstring.** Existing variants (`PermanentHttpError`, `CatalogCommitError`, `TransactionApplyError`) document retry behavior - mirror that style.\n- **Place near related variants** to keep the enum readable top-to-bottom.\n\n## Modifying `Sink` / `Source` / `Transform` traits\n\n**Breaking changes.** Required process:\n\n1. Document migration in the SDK changelog (and `connectors/sdk/README.md`).\n2. Update every in-tree plugin in the same PR. CI builds them all.\n3. Bump SDK minor version.\n4. Note in PR description: dynamic plugins compiled against the old SDK version will fail to load with mismatched FFI symbols.\n\nNon-breaking additions (new default method, new struct field with `#[serde(default)]`, new variant on a non-exhaustive enum) are preferable.\n\n## FFI macros (`sink_connector!`, `source_connector!`)\n\nThe macros (in `sink.rs` and `source.rs`) generate:\n\n- `iggy__open(id, config_ptr, config_len, [state_ptr, state_len,] log_callback) -> i32`\n- `iggy_sink_consume(id, ...)` / `iggy_source_handle(id, callback) -> i32`\n- `iggy__close(id) -> i32`\n- `iggy__version() -> *const c_char` (static lifetime, from `env!(\"CARGO_PKG_VERSION\")`)\n\nInvariants:\n\n- **Duplicate-ID guard** returns `-1` if the caller reopens an ID without closing first. Don't remove this - it prevents silent buffered-data loss.\n- `INSTANCES: Lazy>>` (and `SourceContainer<$type>` mirror) is the only mutable global the macro introduces - keep it that way.\n- Return codes: `0` success, `-1` invalid call, `1` open failure. Don't repurpose.\n\nChanges to the FFI signature must update `runtime/src/main.rs::{SourceApi, SinkApi}` in the same PR.\n\n## `Payload::try_to_bytes(&self)` is non-negotiable for JSON\n\nPlugin authors call this on every consumed message. The implementation in `lib.rs::Payload::try_to_bytes` documents *why* it skips the deep `OwnedValue` clone - replacing `O(n) clone + O(n) serialize` with `O(n) serialize`. If you add a new variant requiring a parsed tree, do the same: serialize in place, don't clone.\n\n## Retry helpers (`retry.rs`)\n\n- `CircuitBreaker`: threshold + cooldown, `try_lock()` on the success path to avoid hot-path contention.\n- `retry_async(policy, context, should_retry, op)`: the retry loop for anything failing as `Err`. Owns attempt counting, backoff and the per-retry log; returns `RetryFailure { error, attempts, exhausted }` so callers write their own terminal log.\n- `retry_backoff(base, retry, max)`: backoff only, for a loop that computes its own delay. `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath and applies no jitter, so call it directly only when the delay must be exact (`source.rs::nack_retry_delay`, whose tests assert exact values).\n- `HttpRetryMiddleware`: integrates with `reqwest-middleware`. Retries 429 + 5xx + network errors. Honors `Retry-After` on 429 and 5xx as a floor over the computed backoff, capped at a one-hour `MAX_RETRY_AFTER`, not `max_delay`.\n- `max_retries` = **total attempts** including the first try, not extra retries. Document if you change this convention. `meilisearch_sink` is the standing exception: its `max_retries` / `max_open_retries` count retries *after* the first, as its README states.\n- New helpers must take `Duration` (not `u64 millis`) on the public API. Internal computation uses `humantime` parsing of `String`.\n\n## `ConnectorState`\n\n- Bytes are opaque - the wrapping type does not impose schema.\n- Serialization is MessagePack via `rmp_serde`. Compact, deterministic, well-supported in serde.\n- `serialize`/`deserialize` helpers return `Option` and log on failure. **Failures are non-fatal** - design downstream code to tolerate fresh state.\n- Don't change the underlying format without coordinating with all sources.\n\n## Tests for SDK changes\n\n- Unit tests for any new pure function in the changed module.\n- Round-trip tests for new schemas/transforms (encode → decode == identity for JSON-equivalent payloads).\n- `sdk/tests/` integration tests when the change crosses module boundaries.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n\n# Rebuild all plugins to catch breaking-change leaks:\ncargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source\n# If FFI signatures changed, also build the runtime:\ncargo build -p iggy-connectors\n```\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n\n\n---\nname: connector-sink\ndescription: Author a new Apache Iggy connector sink plugin under core/connectors/sinks/. Sinks consume messages from Apache Iggy streams and push them to an external system (DB, search engine, HTTP endpoint, object store). Load when creating, modifying, or reviewing a sink crate. Use for sink plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Sink\n\nA **sink** is a Rust `cdylib` that implements `iggy_connector_sdk::Sink`\nand exposes FFI symbols via the `sink_connector!` macro. The runtime\nloads it as `.so`/`.dylib`/`.dll`, polls Apache Iggy on its behalf,\nruns the transform chain, decodes via the configured `Schema`, and\nhands the sink batches of `ConsumedMessage`. The sink is responsible\nfor getting them to the external system reliably and efficiently.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's sink-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Sink::open` / `consume` / `close`) - that's an SDK change, not a plugin one.\n- Touching default `batch_size`, retry caps, or backoff defaults baked into existing in-tree sinks.\n- Promoting an `Err` variant to retryable when it previously was not - retrying bad data trips circuit breakers.\n- Adding a new sink that introduces a backend the runtime doesn't yet integration-test for.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand when authoring).\n- Exemplars: `stdout_sink` (minimal), `postgres_sink` (DB + transient detection), `http_sink` (validation, batch modes, retry middleware), `mongodb_sink` (atomic counters), `elasticsearch_sink` / `iceberg_sink` (backend-specific idioms).\n\n## Hard rules\n\n### Lifecycle\n\n- `Sink::consume` takes **`&self`** - never `&mut self`. Mutable state goes behind `tokio::sync::Mutex` / `AtomicU64`. Prefer atomics for hot counters - `mongodb_sink` uses `messages_processed: AtomicU64` + `insertion_errors: AtomicU64` and avoids a state lock on the hot path entirely.\n- `Sink::open` builds the client AND tests connectivity. Fail fast with `Error::InitError`.\n- `Sink::close` flushes + drops resources. Take ownership via `client: Option` + `.take()`. sqlx pools have `.close().await`. reqwest/mongodb/elasticsearch clients just drop. Log final stats either way.\n\n### Config\n\n- Every runtime-tunable field is `Option` with a default applied in `new()`. Forward-compat: adding a field doesn't break existing TOML.\n- Durations: `Option`, parsed via `humantime::Duration` in `new()`. Two equivalent in-tree idioms - inline `HumanDuration::from_str(&raw).map(|d| *d).unwrap_or_else(..)` (`random_source`, `postgres_source`) or a local `parse_duration` helper (`http_sink`, `sdk/src/retry.rs`). Fall back to a default on parse failure with `warn!` - never panic. We do **not** use `humantime_serde`.\n- **Conflict resolution**: if two config fields can be in invalid relative ordering, fix it in `new()` + `warn!` - don't error. `http_sink` swaps `retry_delay` and `max_retry_delay` when reversed to avoid an `ExponentialBackoff` panic downstream.\n- Validation order: structural in `new()`, connectivity in `open()`, **never in `consume()`**.\n\n### Payloads (efficiency-critical)\n\n- Dispatch on the `Payload` variant. Handle `Json`, `Raw`, `Text` and `Proto` at minimum. `Proto` holds text that is a JSON document when `proto_convert` had no descriptor or could not encode, and arbitrary text otherwise: call `Payload::json_document()` first and fall back to the text handling when it returns `None`. Unsupported -> `Error::InvalidPayloadType` or fall back to base64 (`elasticsearch_sink`).\n- For `Payload::Json`, call `try_to_bytes(&self)` - single-pass serialization without cloning the `OwnedValue` tree (`sdk/src/lib.rs::Payload::try_to_bytes`). Never `payload.clone().try_into_vec()`.\n- To take a payload out of `ConsumedMessage` without cloning: `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))`. Standard pattern across `http_sink::send_individual` / `send_ndjson` / `send_json_array`.\n- Pre-allocate per-batch buffers with `Vec::with_capacity(messages.len())` (or `* factor` for serialized bodies). Used across `postgres_sink`, `mongodb_sink`, `quickwit_sink`, `elasticsearch_sink`, `http_sink`.\n\n### Headers\n\n- `message.headers: Option>`. Check `.is_empty()` before iterating - runtime treats empty as `None`.\n- For binary values use `v.as_raw()` + base64. for text use `v.to_string_value()`. See `http_sink::EncodedHeader`.\n- `BTreeMap` for deterministic ordering. Don't collect into `HashMap`.\n\n### Errors\n\n| Scenario | SDK variant | Retry? |\n| ------------------------------------------ | --------------------------------------------------------------------------- | ---------------------------------------- |\n| Init / connectivity failure | `Error::InitError` | n/a (fails open) |\n| Bad config value | `Error::InvalidConfigValue(field)` | n/a |\n| Single-message validation failure | `Error::InvalidRecordValue(reason)` | no - skip + log |\n| Transient backend (5xx, conn reset, ...) | `Error::HttpRequestFailed` / `Error::Connection` / `Error::CannotStoreData` | yes |\n| Permanent 4xx, schema mismatch, constraint | `Error::PermanentHttpError` / `Error::SchemaMismatch` | **no** - retrying trips circuit breakers |\n| I/O failure mid-write (parquet/iceberg) | `Error::WriteFailure` | caller decides |\n| Catalog commit failure (Iceberg) | `Error::CatalogCommitError` | **no - not idempotent** |\n\n### Retry\n\n- SDK helpers cover the simple case: `iggy_connector_sdk::retry::check_connectivity_with_retry(...)` for `open()`, `HttpRetryMiddleware` for default 429/5xx/network policy on reqwest clients.\n- Custom strategies: `reqwest-middleware` + `reqwest_retry::RetryTransientMiddleware::new_with_policy_and_strategy`. `http_sink` defines its own `HttpSinkRetryStrategy` (honors `success_status_codes`, per-status decisions).\n- Non-HTTP clients: write `is_transient_error(&e)` mapping driver-specific codes. `postgres_sink::is_transient_error` maps SQLSTATEs `40001`, `40P01`, `57P01-03`, `08000/03/06`.\n- Retry loop: new connectors use `iggy_connector_sdk::retry::retry_async(policy, context, should_retry, op)` for anything that fails as `Err`. It owns attempt counting, backoff and the per-retry log, and returns `RetryFailure { error, attempts, exhausted }`; the caller logs the terminal failure.\n- Backoff only, for a loop that computes its own delay (it retries on an `Ok` response, carries a deadline, or reconnects between attempts): `retry_backoff(base, retry, max)`, where `retry` is 1-based. `exponential_backoff` is the 0-based primitive underneath, and it applies no jitter; call it directly only when the delay must be exact, as `sdk/src/source.rs::nack_retry_delay` needs for its tests.\n- Cap retries at 3 total attempts.\n\n### Idempotency\n\nRuntime retries on any `Err` - same batch can be redelivered. In-tree guarantees:\n\n- **Postgres** (`postgres_sink::build_batch_insert_query`): PK on `id`, but plain `INSERT` (no `ON CONFLICT`). Duplicate retry triggers `23505`, `is_transient_error` does NOT mark it transient -> error propagates. Adding `ON CONFLICT (id) DO NOTHING` is a known improvement.\n- **MongoDB**: composite `_id` from `stream:topic:partition:message_id`, via `insert_many(docs).ordered(false)`. Per-document duplicate-key errors don't abort the batch - effective idempotency under retry.\n- **Elasticsearch** (`bulk_index_documents`): bulk action specifies only `_index`, no `_id` -> ES auto-generates ids -> NOT idempotent. Extensions should add `\"_id\"`.\n- **HTTP**: endpoint-dependent. Document in the plugin README.\n- **Iceberg**: transactional commit. `Error::CatalogCommitError` documents that the txn is consumed on `commit()`. Rebuild-then-retry can duplicate data - callers must check the catalog first.\n\nWhen writing a new sink, prefer dedup-on-write using the message `id` field. Mongo's upsert-style is the cleanest reference.\n\n### Batching\n\n- Default `batch_size: 100`. Tune per-backend.\n- Chunk via `messages.chunks(self.batch_size)`. Do not buffer across `consume()` - runtime already batches at the consumer-poll boundary.\n- For HTTP backends support multiple batch modes when applicable (individual / ndjson / json_array / raw). See `http_sink::BatchMode`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: \", self.id);\ndebug!(\"Processing batch of {} messages, offset: {}\", batch.len(), offset);\nwarn!(\"Retry {attempt}/{max} for ID: {}, reason: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, processed: {}\", self.id, count);\n```\n\nUse literal API field names as labels (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` - runtime can't dlopen a regular rlib.\n2. Cloning the whole `Payload::Json` to serialize - use `try_to_bytes(&self)`.\n3. Holding `Mutex` across `.await` when the locked region does I/O.\n4. `&mut self` on `consume` - won't compile. you need interior mutability.\n5. Returning the first error and dropping subsequent batches - track `last_err`, process all, return.\n6. Forgetting to map `PermanentHttpError` - retries hammer the backend with bad data.\n7. Mutating config in `consume()` - config is a snapshot from `new()`. Reload requires runtime restart.\n8. Logging the connection string. Redact at construction: `http_sink::sanitize_url_for_log` populates `HttpSink.log_url`.\n\n## Tests\n\nUnit tests at EOF of `src/lib.rs`. See [connector-testing](../connector-testing/SKILL.md) for BDD naming + `test_config()` helper.\n\nReal-infra integration tests live under `core/integration/tests/connectors//` using `#[iggy_harness]` + a backend fixture that spins up Docker via `testcontainers-modules`. Plugins integrating with external services must have an integration test. Reference: `core/integration/tests/connectors/postgres/postgres_sink.rs` + `fixtures/postgres/`.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__sink --all-targets -- -D warnings\ncargo test -p iggy_connector__sink\n\n# Integration tests (filter by name; integration crate has no per-area target):\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sinks/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n\n\n# Sink plugin skeleton\n\nBoilerplate for a new `core/connectors/sinks/_sink/`. Adapt the\n`MySink` / `Client` types to the backend driver you're integrating.\n\n## Cargo.toml\n\n```toml\n# Apache 2.0 header (copy verbatim from any existing sink Cargo.toml)\n[package]\nname = \"iggy_connector__sink\"\nversion = \"0.5.1-edge.1\" # match the version most other sinks use\nedition = \"2024\"\nlicense = \"Apache-2.0\"\npublish = false\n# ...keywords, description, repository, homepage all identical to existing sinks\n\n[package.metadata.cargo-machete]\nignored = [\"dashmap\", \"once_cell\"] # used by sink_connector! macro\n\n[lib]\ncrate-type = [\"cdylib\", \"lib\"] # cdylib = runtime-loadable; lib = unit tests\n\n[dependencies]\nasync-trait = { workspace = true }\ndashmap = { workspace = true }\niggy_connector_sdk = { workspace = true }\nonce_cell = { workspace = true }\nserde = { workspace = true }\ntokio = { workspace = true }\ntracing = { workspace = true }\n# + your client crate (reqwest, sqlx, mongodb, ...)\n```\n\nRun `cargo sort --no-format --workspace` after edits.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, then `impl`, then private\nhelpers.\n\n```rust\n/* Licensed to the Apache Software Foundation (ASF) under one\n * or more contributor license agreements... */ // full Apache header\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata, sink_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse std::str::FromStr;\nuse std::time::Duration;\nuse tokio::sync::Mutex;\nuse tracing::{debug, error, info, warn};\n\nsink_connector!(MySink); // generates FFI symbols + version export\n\nconst CONNECTOR_NAME: &str = \"My sink\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySinkConfig {\n pub endpoint: String,\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option, // humantime, e.g. \"500ms\"\n pub verbose_logging: Option, // mirror runtime's `verbose` flag\n // Every runtime-tunable field is Option; defaults applied in new()\n}\n\n#[derive(Debug)]\npub struct MySink {\n id: u32,\n config: MySinkConfig,\n batch_size: usize,\n max_retries: u32,\n retry_delay: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n}\n\n#[derive(Debug)]\nstruct State {\n messages_processed: u64,\n errors: u64,\n}\n\nimpl MySink {\n pub fn new(id: u32, config: MySinkConfig) -> Self {\n let batch_size = config.batch_size.unwrap_or(100) as usize;\n let max_retries = config.max_retries.unwrap_or(3);\n let retry_delay = config\n .retry_delay\n .as_deref()\n .and_then(|raw| humantime::Duration::from_str(raw).ok().map(|d| *d))\n .unwrap_or_else(|| {\n warn!(\"Invalid retry_delay for {CONNECTOR_NAME} ID: {id}, defaulting to 500ms\");\n Duration::from_millis(500)\n });\n let verbose = config.verbose_logging.unwrap_or(false);\n Self {\n id,\n config,\n batch_size,\n max_retries,\n retry_delay,\n verbose,\n client: None,\n state: Mutex::new(State { messages_processed: 0, errors: 0 }),\n }\n }\n}\n\n#[async_trait]\nimpl Sink for MySink {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n client.ping()\n .await\n .map_err(|e| Error::InitError(format!(\"connectivity check failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn consume(\n &self,\n topic_metadata: &TopicMetadata,\n messages_metadata: MessagesMetadata,\n messages: Vec,\n ) -> Result<(), Error> {\n let Some(client) = self.client.as_ref() else {\n return Err(Error::InitError(\"client not initialized\".into()));\n };\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages from stream: {}, topic: {}, offset: {}, current_offset: {}\",\n self.id, messages.len(), topic_metadata.stream, topic_metadata.topic,\n messages_metadata.partition_id, messages_metadata.current_offset\n );\n } else {\n debug!(\n \"{CONNECTOR_NAME} ID: {} consuming {} messages\",\n self.id, messages.len()\n );\n }\n\n let mut last_err: Option = None;\n for batch in messages.chunks(self.batch_size) {\n match self.send_batch(client, batch).await {\n Ok(()) => { /* counter */ }\n Err(Error::PermanentHttpError(message)) => {\n error!(\n \"{CONNECTOR_NAME} ID: {} dropping batch (permanent): {message}\",\n self.id\n );\n }\n Err(error) => {\n last_err = Some(error);\n }\n }\n }\n match last_err {\n Some(error) => Err(error),\n None => Ok(()),\n }\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n // sqlx pools have `.close().await`; reqwest/mongodb/elasticsearch just drop.\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, processed: {}, errors: {}\",\n self.id, state.messages_processed, state.errors\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySinkConfig) -> Result { /* ... */ }\n```\n\n\n---\nname: connector-source\ndescription: Author a new Apache Iggy connector source plugin under core/connectors/sources/. Sources poll an external system (DB, API, queue) and produce messages into Apache Iggy streams. Load when creating, modifying, or reviewing a source crate. Use for source plugin authoring. NOT for runtime internals (see connector-runtime).\n---\n\n# Writing an Apache Iggy Connector Source\n\nA **source** is a Rust `cdylib` that implements\n`iggy_connector_sdk::Source` and exposes FFI symbols via the\n`source_connector!` macro. The runtime calls `poll()` in a loop,\napplies transforms, encodes via the configured `Schema`, sends to\nApache Iggy, and persists the returned `ConnectorState` after every\nsuccessful send.\n\n> **Universal connector rules** (SecretString, benchmark, verbose flag, drop accounting, filter contract, exemplar patterns) live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill\n> covers only what's source-specific.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Quick reference](#quick-reference)\n- [Hard rules](#hard-rules)\n- [Common pitfalls](#common-pitfalls)\n- [Tests](#tests)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Changing the SDK trait surface (`Source::open` / `poll` / `on_batch_result` / `close`) - that's an SDK change.\n- Adding a long-running side task in the plugin - the runtime owns lifecycle, and orphans survive `close()`. Sanctioned only where the source is itself a server the runtime cannot drive, as in `http_source`'s listener, and then only with an explicit shutdown in the last `close()` that awaits its tasks before returning.\n- Persisting unbounded state - `State` is rewritten every batch.\n- Adding a source that requires authoritative offsets external to Apache Iggy without coordinating retention.\n\n## Quick reference\n\n- Skeleton: [TEMPLATE.md](TEMPLATE.md) (load on demand).\n- Exemplars: `random_source` (minimal + canonical state tests), `postgres_source` (cursor / delete-after-read / processed-column modes, restart-survives-state tests), `elasticsearch_source` (scroll cursor), `influxdb_source` (time-series scan).\n\n## Hard rules\n\n### `poll()` signature is `&self`\n\nThe macro shares the source as `Arc` across the FFI callback and forwarding loop. Signature: `async fn poll(&self) -> ...` - any mutable state behind `tokio::sync::Mutex`. **Single most common new-contributor mistake.**\n\n### Lock discipline\n\nNever hold the state `Mutex` across upstream I/O. Build a candidate from committed\nstate, then stage it until the runtime reports the batch result:\n\n```rust\nlet mut candidate = self.state.lock().await.clone();\nlet rows = client.query(&sql, &[&candidate.cursor]).await?;\ncandidate.cursor = Some(new_cursor);\nlet persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n*self.pending.lock().await = Some(candidate);\n```\n\n### Delivery acknowledgment\n\n`on_batch_result` (added by #3855) is how a source learns what happened to the batch it just\nreturned. The SDK keeps exactly one batch in flight: it will not call `poll()` again until this\nreturns. By default it stops after `MAX_CONSECUTIVE_NACKS` (5) consecutive NACKs; a source can\noverride `batch_policy()` to disable the limit when its accepted input cannot be replayed after\nrestart. A stop is reported to the runtime only when the plugin exports the current SDK's stop\ncallback, so rebuild older source plugins.\n\n- `Ack` means the runtime sent the batch **and** persisted its state. `Nack` means it could not\n confirm both, which is **not** the same as neither happening: a batch that reached the topic but\n whose state save failed is NACKed. A missing runtime result leaves the batch pending; it does not\n produce a synthetic NACK. A source that replays on `Nack` is at-least-once, not exactly-once.\n- The trait has a **default no-op**, which suits only a source with no staged cursor and no\n destructive work. If `poll()` advances a cursor, deletes rows, or drains an in-memory buffer,\n omitting this loses data silently and nothing will tell you. `random_source` and `http_source`\n implement it; the other shipped sources take the default and skip rows on a NACK.\n- Stage in `poll()`, apply on `Ack`, discard or replay on `Nack`. A source whose input is pushed to\n it, rather than re-readable upstream, has to hold the batch itself: see `http_source`'s staging.\n- Returning `Err` from it stops the source immediately, so it is not a retry signal.\n\n### State persistence\n\n- `ConnectorState` is `Vec` via MessagePack (`rmp_serde`). Use\n `ConnectorState::serialize(&state, NAME, id)` and\n `ConnectorState::deserialize::(NAME, id)`.\n- `poll()` must not commit cursors or destructive work. Return messages with candidate state and\n keep the corresponding work staged.\n- The runtime sends the batch, saves its candidate state to\n `{state_path}/source_{key}.state`, then calls `on_batch_result(Ack)`. Commit staged in-memory\n state and external delete or mark operations only on ACK. A NACK discards the candidate so the\n same data can be polled again.\n- A crash between `poll()` returning and state persistence leaves the prior cursor for the next\n poll, so downstream must tolerate at-least-once delivery.\n- Cursor sources that stage state until `on_batch_result` can attach state to the corresponding\n message batch. Control-plane state that is independent of a publish can ride an empty batch\n instead.\n- Return `state: None` for an empty poll when no watermark changed. If an empty poll advances a\n watermark, stage and return the new state through the same ACK handshake.\n- The runtime can still NACK an empty batch when state storage is latched or a pending checkpoint\n cannot resolve. Never treat hand-off as durable; `on_batch_result` reports the outcome.\n- Treat candidate-state serialization failure as a poll error. Do not send messages without the\n state needed to resume them safely.\n- Keep `State` small - rewritten every batch. No unbounded vecs.\n\nThe SDK allows one in-flight batch. Five consecutive NACKs stop a source using\nthe default policy and require a manual restart. Returning `Err` from\n`on_batch_result` is fatal regardless of the NACK limit, so\nretry transient backend failures inside the callback before returning an error.\nIf the runtime has not reported ACK or NACK after 30 seconds, the SDK warns\nand keeps waiting for that result without polling or replaying the batch.\nOnce the result is received, the SDK waits for `on_batch_result` to finish,\nso the callback must bound its own connection acquisition and retry budget.\n\n### Sleep first\n\n`poll()` must `sleep(self.poll_interval).await` before any work. Without it, an empty source spins a CPU.\n\n### Schema selection\n\nMatch `ProducedMessages.schema` to the bytes in `messages[i].payload`:\n\n- JSON-serialized rows → `Schema::Json`\n- Already-protobuf bytes → `Schema::Proto`\n- Already-avro bytes → `Schema::Avro`\n- Opaque → `Schema::Raw`\n\n### IDs and timestamps\n\n- `ProducedMessage.id: Option` - set when a natural ID exists (DB PK, document id). It rides the wire as the message id, which is what a **consumer** can dedupe an at-least-once duplicate on.\n- **Iggy itself does not dedupe on that id.** The server's only read of the field is `core/server/src/http/wire.rs`, which mints a fresh uuid when the incoming id is 0 and otherwise passes it through. Its dedup path is a per-client request-id watermark (`dedup_clients_max`) that never looks at this field.\n- `origin_timestamp: Option` - source-system event time in nanoseconds. Lets downstream sinks reason about lag.\n- `timestamp` and `checksum` are Iggy-side - leave `None`.\n\n### Concurrency\n\n- Runtime spawns ONE `poll()` task per source. No concurrent `poll()`.\n- Don't spawn your own long-running Tokio tasks: the runtime owns lifecycle. The exception is a source that listens rather than polls, which has to own its listener; `http_source` is the worked example, and it shuts its tasks down in the last `close()` rather than leaving them to outlive the connector.\n\n### Errors\n\n| Scenario | Variant |\n| ------------------------------------------- | ------------------------------------------------- |\n| Bad config in `new()`/`open()` | `Error::InitError` |\n| Cannot reach external system at startup | `Error::InitError` or `Error::Connection` |\n| Transient fetch failure (retry-worthy) | `Error::Connection` or `Error::HttpRequestFailed` |\n| Permanent fetch failure (auth, schema gone) | `Error::PermanentHttpError` |\n| Row failed to serialize | `Error::Serialization(...)` |\n| State serialization failed | `Error::Serialization(...)` |\n\nReturning `Err` from `poll()` is only logged by the SDK's FFI bridge\n(`sdk/src/source.rs::handle_messages`) - the loop continues, the next\n`poll()` runs. Connector status does NOT flip to `Error` from a poll\nfailure. Status `Error` is set by the runtime only on transform/encode\nfailure, Iggy send failure, or state save failure\n(`runtime/src/source.rs::source_forwarding_loop` calls to\n`context.sources.set_error`). To surface a poll failure as unhealth,\nraise it through the metric counter or escalate to `Error::InitError`\nfrom `open()`.\n\n### Logging\n\n```rust\ninfo!(\"Opened connector ID: {}, endpoint: {}\", self.id, ...);\ninfo!(\"Restored state for ID: {id}, cursor: {:?}\", ...);\ndebug!(\"Polled {} rows for ID: {}\", rows.len(), self.id);\nwarn!(\"Transient fetch failure for ID: {}, will retry: {error}\", self.id);\nerror!(\"Failed to for ID: {}, error: {error}\", self.id);\ninfo!(\"Closed connector ID: {}, total produced: {}\", self.id, ...);\n```\n\nIggy consumer-loop labels use literal API names (`offset=`, `current_offset=`).\n\n## Common pitfalls\n\n1. `async fn poll(&mut self)` - won't compile. Use `&self` + `Mutex`.\n2. Holding `state.lock()` across the fetch I/O - blocks `close()`, causes shutdown timeouts.\n3. Forgetting to sleep - 100% CPU on idle source.\n4. Committing a cursor or deleting source data in `poll()` - stage it and wait for ACK.\n5. Unbounded data in `State` - rewritten every batch. keep O(constant).\n6. `std::sync::Mutex` - blocks the executor. Use `tokio::sync::Mutex`.\n7. Not setting `ProducedMessage.id` when a stable ID exists - leaves a consumer nothing to dedupe a replayed duplicate on. It does not make the write idempotent server-side, because nothing there reads it.\n8. Spawning side tasks - the runtime owns the scheduler. The one exception is a source that listens rather than polls and must own its listener (see [Concurrency](#concurrency) and the STOP list); it owes an explicit shutdown in the last `close()` that awaits its tasks.\n\n## Tests\n\nMandatory four canonical source state tests (see [connector-testing](../connector-testing/SKILL.md) for the full pattern). Copy from `sources/random_source/src/lib.rs::tests`. Plus config defaults, payload building, schema selection.\n\nIntegration tests under `core/integration/tests/connectors//` for any source backed by external infra. Use `#[iggy_harness]` + a `TestFixture` backed by `testcontainers-modules`. Reference: `core/integration/tests/connectors/postgres/postgres_source.rs` (multi-mode tests) + `restart.rs` (state survives restart). Exercise both ACK and NACK paths when the source stages cursors or destructive work.\n\n## Before declaring done\n\n```bash\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector__source --all-targets -- -D warnings\ncargo test -p iggy_connector__source\n\n# Integration tests:\ncargo test -p integration -- connectors::::\n```\n\nUpdate `core/connectors/sources/README.md` and add a sample TOML under `core/connectors/runtime/example_config/connectors/`.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n\n\n# Source plugin skeleton\n\nBoilerplate for a new `core/connectors/sources/_source/`. Adapt\nthe `MySource` / `Client` / row types to the backend driver.\n\n`Cargo.toml` is identical to a sink's (see\n[connector-sink/TEMPLATE.md](../connector-sink/TEMPLATE.md)) - only\nthe crate name suffix changes (`iggy_connector__source`) and the\nupstream client dep.\n\n## src/lib.rs\n\nCode reads top to bottom. Public types first, `impl Source` after,\nhelpers below.\n\n```rust\n/* Apache 2.0 header */\n\nuse std::str::FromStr;\nuse std::time::Duration;\n\nuse async_trait::async_trait;\nuse iggy_connector_sdk::{\n ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,\n source::SourceBatchResult, source_connector,\n};\nuse serde::{Deserialize, Serialize};\nuse tokio::sync::Mutex;\nuse tokio::time::sleep;\nuse tracing::{debug, error, info, warn};\n\nsource_connector!(MySource);\n\nconst CONNECTOR_NAME: &str = \"My source\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct MySourceConfig {\n pub endpoint: String,\n pub poll_interval: Option, // humantime, \"10s\" default\n pub batch_size: Option,\n pub max_retries: Option,\n pub retry_delay: Option,\n pub verbose_logging: Option,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct State {\n cursor: Option, // WAL LSN, scroll id, timestamp, ...\n last_offset: u64,\n messages_produced: u64,\n}\n\n#[derive(Debug)]\npub struct MySource {\n id: u32,\n config: MySourceConfig,\n poll_interval: Duration,\n verbose: bool,\n client: Option,\n state: Mutex,\n pending: Mutex>,\n}\n\nimpl MySource {\n pub fn new(id: u32, config: MySourceConfig, state: Option) -> Self {\n let raw_interval = config.poll_interval.clone().unwrap_or_else(|| \"10s\".into());\n let poll_interval = humantime::Duration::from_str(&raw_interval)\n .map(|d| *d)\n .unwrap_or_else(|_| {\n warn!(\"Invalid poll_interval for {CONNECTOR_NAME} ID: {id}, defaulting to 10s\");\n Duration::from_secs(10)\n });\n\n let verbose = config.verbose_logging.unwrap_or(false);\n\n let restored = state\n .and_then(|s| s.deserialize::(CONNECTOR_NAME, id))\n .inspect(|s| info!(\n \"Restored state for {CONNECTOR_NAME} ID: {id}, last_offset: {}, cursor: {:?}\",\n s.last_offset, s.cursor\n ));\n\n Self {\n id,\n config,\n poll_interval,\n verbose,\n client: None,\n state: Mutex::new(restored.unwrap_or(State {\n cursor: None,\n last_offset: 0,\n messages_produced: 0,\n })),\n pending: Mutex::new(None),\n }\n }\n}\n\n#[async_trait]\nimpl Source for MySource {\n async fn open(&mut self) -> Result<(), Error> {\n let client = build_client(&self.config)\n .await\n .map_err(|e| Error::InitError(format!(\"client build failed: {e}\")))?;\n self.client = Some(client);\n info!(\n \"Opened {CONNECTOR_NAME} connector ID: {}, endpoint: {}\",\n self.id, self.config.endpoint\n );\n Ok(())\n }\n\n async fn poll(&self) -> Result {\n sleep(self.poll_interval).await; // sleep first - backpressure\n\n let mut candidate = self.state.lock().await.clone();\n\n let fetched = self.fetch_since(candidate.cursor.as_deref()).await?;\n\n let mut messages = Vec::with_capacity(fetched.len());\n let mut next_cursor = None;\n for row in fetched {\n let payload = simd_json::to_vec(&row).map_err(|e|\n Error::Serialization(format!(\"row serialize: {e}\"))\n )?;\n messages.push(ProducedMessage {\n id: Some(row.id as u128),\n checksum: None,\n timestamp: None,\n origin_timestamp: Some(row.created_at_ns),\n headers: None,\n payload,\n });\n next_cursor = Some(row.cursor_value);\n }\n\n if self.verbose {\n info!(\n \"{CONNECTOR_NAME} ID: {} produced {} messages, next cursor: {:?}\",\n self.id, messages.len(), next_cursor\n );\n }\n\n if messages.is_empty() {\n return Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: None,\n });\n }\n\n candidate.messages_produced += messages.len() as u64;\n candidate.cursor = next_cursor;\n let persisted = ConnectorState::serialize(&candidate, CONNECTOR_NAME, self.id)\n .ok_or_else(|| Error::Serialization(\"failed to serialize source state\".into()))?;\n *self.pending.lock().await = Some(candidate);\n\n Ok(ProducedMessages {\n schema: Schema::Json,\n messages,\n state: Some(persisted),\n })\n }\n\n async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {\n let (SourceBatchResult::Ack, Some(candidate)) =\n (result, self.pending.lock().await.take())\n else {\n return Ok(());\n };\n *self.state.lock().await = candidate;\n Ok(())\n }\n\n async fn close(&mut self) -> Result<(), Error> {\n if let Some(client) = self.client.take() {\n let _ = client; // or `client.close().await;` for sqlx pools\n }\n let state = self.state.lock().await;\n info!(\n \"Closed {CONNECTOR_NAME} connector ID: {}, total produced: {}\",\n self.id, state.messages_produced\n );\n Ok(())\n }\n}\n\nasync fn build_client(config: &MySourceConfig) -> Result { /* ... */ }\n```\n\n\n---\nname: connector-testing\ndescription: Write tests for Apache Iggy connectors - sinks, sources, transforms, SDK, runtime. Covers BDD unit-test naming, the canonical source state round-trip tests, and the real-infra integration test layout under `core/integration/tests/connectors/` with `testcontainers-modules` + `#[iggy_harness]`. Load when writing or reviewing tests in `core/connectors/` or `core/integration/tests/connectors/`. Use for connector test authoring. NOT for runtime internals or non-connector test patterns.\n---\n\n# Testing Apache Iggy Connectors\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). Repo-wide testing rules (`#[iggy_harness]`, BDD naming, harness layout) live in [AGENTS.md](../../../AGENTS.md#testing). This skill covers connector-specific test conventions.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [Unit tests](#unit-tests)\n- [Integration tests (real infra via Docker)](#integration-tests-real-infra-via-docker)\n- [SDK + runtime tests](#sdk--runtime-tests)\n- [CI / verification flow](#ci--verification-flow)\n- [Common pitfalls](#common-pitfalls)\n- [Examples worth copying](#examples-worth-copying)\n\n## STOP and ask the user before\n\n- Mocking the database or backend in an integration test - we hit real infra via `testcontainers-modules` (mocks miss migration bugs).\n- Adding a unit test that requires Docker - that belongs under `core/integration/tests/connectors//`, not in `/src/lib.rs`.\n- Skipping the four canonical source state tests - they're mandatory for every source plugin.\n- Naming a test `test_foo` / `does_bar` - we use BDD `given_X_when_Y_should_Z`.\n\nThere are **two distinct test layers**. Use both - they catch different bugs.\n\n| Layer | Location | What it covers | When required |\n| --------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------- |\n| **Unit tests** | `#[cfg(test)] mod tests` at EOF of `/src/lib.rs` | Config parsing, defaults, fallbacks, payload conversion, query/document building, state ser/de | Every plugin |\n| **Integration tests** | `core/integration/tests/connectors//` (uses real backend via Docker) | End-to-end: runtime + plugin + real Postgres/ES/Iceberg/Mongo/etc. + Iggy server | Every plugin that integrates with an external system |\n\n## Unit tests\n\n### Naming - BDD `given_when_should`\n\nMatch the convention used in the codebase (`sources/random_source/src/lib.rs::tests`, `sinks/postgres_sink/src/lib.rs::tests`):\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_messages_produced() { ... }\n\n#[test]\nfn given_no_state_should_start_fresh() { ... }\n\n#[test]\nfn given_invalid_duration_should_fall_back_to_default() { ... }\n\n#[test]\nfn given_payload_json_should_serialize_correctly() { ... }\n```\n\nYou may **omit one part** when the test is small (`given_X_should_Y` instead of full `given_X_when_Y_should_Z`), but **stay consistent within a file**. Imperative `test_foo` / `does_bar` names are not the convention here.\n\nA test whose subject is a **type** rather than a scenario has no \"given\" to name, and drops the prefix: it reads `_should_`. The canonical state round trip below is `state_should_be_serializable_and_deserializable` in all three shipped sources (`random_source`, `postgres_source`, `http_source`). Leave those as they are rather than bending them into the scenario form.\n\n### Prove a test can fail, before trusting it\n\nA passing test says nothing until you have watched it fail. Break the behaviour it names, run the\nsuite, and put the code back:\n\n```bash\n# commit first: this reverts with `git checkout --`, which discards\n# every uncommitted change in the file, not only the mutation\ngit status --porcelain -- # must be empty\n\ncargo nextest run -p # expect a failure naming that test\ngit checkout -- \n```\n\nTwo failures this catches, both seen in review of `http_source`:\n\n- **A fixture that makes the interesting case unreachable.** A guard test for endpoint resurrection\n passed `&[]` as the static config, so there was nothing to resurrect and the assertion held\n whatever `restore` did.\n- **An assertion aimed at something self-healing.** A test for a teardown guard asserted on gauges,\n which `Metrics::encode` rebuilds from the live instances on every scrape, so it passed with the\n guard deleted. Delete such a test rather than keep it; a test that cannot fail is worse than none,\n because it reads as coverage.\n\nWatch for an arm hidden behind a catch-all: `StatusClass::from` ends in `_ => ServerError`, so the\nredirect arm could be removed with every existing assertion still green.\n\n### `test_config()` helper\n\nEvery plugin's tests start with a small helper returning a tuned-down version of the production config. New tests extend it. don't construct from scratch each time.\n\n```rust\nfn test_config() -> RandomSourceConfig {\n RandomSourceConfig {\n interval: Some(\"100ms\".to_string()),\n max_count: Some(100),\n messages_range: Some((5, 10)),\n payload_size: Some(50),\n }\n}\n```\n\n### Async testing pattern\n\nThe trait methods are `async`. For unit tests, build a runtime locally - don't add `#[tokio::test]` in `src/lib.rs` (it pulls test macros into prod build):\n\n```rust\n#[test]\nfn given_X_should_Y() {\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n // await calls here\n });\n}\n```\n\n### Mandatory: four canonical source state tests\n\nEvery source must have these. Copy from `sources/random_source/src/lib.rs::tests`:\n\n```rust\n#[test]\nfn given_persisted_state_should_restore_() {\n let state = State { /* fields */ };\n let serialized = rmp_serde::to_vec(&state).expect(\"Failed to serialize state\");\n let connector_state = ConnectorState(serialized);\n\n let source = MySource::new(1, test_config(), Some(connector_state));\n\n let runtime = tokio::runtime::Runtime::new().unwrap();\n runtime.block_on(async {\n let restored = source.state.lock().await;\n assert_eq!(restored., /* expected */);\n });\n}\n\n#[test]\nfn given_no_state_should_start_fresh() {\n let source = MySource::new(1, test_config(), None);\n /* assert default state */\n}\n\n#[test]\nfn given_invalid_state_should_start_fresh() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n let source = MySource::new(1, test_config(), Some(invalid));\n /* assert default state */\n}\n\n#[test]\nfn state_should_be_serializable_and_deserializable() {\n let original = State { /* fields */ };\n let bytes = rmp_serde::to_vec(&original).unwrap();\n let restored: State = rmp_serde::from_slice(&bytes).unwrap();\n assert_eq!(original., restored.);\n}\n```\n\nState is the only thing that survives a plugin restart. Silent corruption here means lost data on the next deploy.\n\n**The third test has two correct shapes, and which one applies is a design decision rather than a style choice.**\n\nStarting fresh is right when the state is a cursor. The worst case is re-reading from the beginning, which the at-least-once contract already permits, so discarding an undecodable cursor costs duplicates and nothing else.\n\nIt is wrong when the state carries something whose *absence* is a downgrade rather than a repeat. `http_source` keeps its revocation tombstones in state, so starting fresh after a decode failure would serve every revoked endpoint again, with its secret. Its `restore` returns `Err` instead and the connector refuses to open. A source in that position asserts the refusal:\n\n```rust\n#[test]\nfn given_invalid_state_should_refuse_to_open() {\n let invalid = ConnectorState(b\"not valid msgpack\".to_vec());\n\n let restored = MyState::restore(&config_entries(), Some(invalid), 1);\n\n assert!(\n restored.is_err(),\n \"discarding this state silently would re-serve what it was revoking\"\n );\n}\n```\n\nName the test for whichever it is and say why in a comment. What is never right is starting fresh silently when the state that was discarded was load-bearing.\n\n### Sink unit tests\n\nCover pure per-message logic:\n\n- Config defaults + optional fields + invalid values → fallback.\n- Payload conversion for each `Payload` variant the sink claims to handle.\n- Header encoding (especially base64 for binary headers).\n- Query/document building (e.g., `postgres_sink::build_batch_insert_query`).\n- Transient vs permanent error classification (e.g., `postgres_sink::is_transient_error`).\n\n### Transform tests\n\nFour branches must be covered (see `connector-transform`):\n\n1. Happy path: matching input → expected output.\n2. Filter path: input that should be dropped returns `Ok(None)`.\n3. Pass-through path: non-JSON payload (or unmatched condition) returns `Ok(Some(unchanged))`.\n4. Invalid config: `new()` returns `Err(Error::InvalidConfigValue(_))`.\n\n### What NOT to mock\n\n- The underlying client lib (sqlx, reqwest, mongodb). Tests would exercise the mock, not the code.\n- The Iggy server.\n- The runtime.\n\nIf you need a fake backend, use `wiremock` (a real local HTTP server) - see `core/integration/tests/connectors/http/`.\n\n## Integration tests (real infra via Docker)\n\nThe canonical pattern lives in `core/integration/tests/connectors/`. Plugins that integrate with external services (Postgres, Elasticsearch, Iceberg, MongoDB, InfluxDB, Quickwit, HTTP/wiremock) have a paired fixture + test file there.\n\n### Layout\n\n```text\ncore/integration/tests/connectors/\n├── mod.rs Common helpers (TestMessage, create_test_messages)\n├── fixtures/ One subdir per backend\n│ ├── postgres/\n│ │ ├── container.rs PostgresContainer (testcontainers wrapper), env-var consts\n│ │ ├── sink.rs PostgresSinkFixture, PostgresSinkByteaFixture, PostgresSinkJsonFixture\n│ │ ├── source.rs PostgresSourceJsonFixture, PostgresSourceJsonbFixture, ...\n│ │ └── mod.rs re-exports\n│ ├── elasticsearch/ iceberg/ mongodb/ influxdb/ quickwit/ http/ delta/\n│ └── wiremock.rs HTTP mocking for http_sink tests\n├── postgres/\n│ ├── postgres_sink.rs The actual #[iggy_harness] tests\n│ ├── postgres_source.rs\n│ ├── restart.rs Restart + state-survival tests\n│ ├── sink.toml Runtime config used by sink tests\n│ ├── source.toml Runtime config used by source tests\n│ └── mod.rs Shared constants (TEST_MESSAGE_COUNT, POLL_ATTEMPTS, ...)\n├── delta/ elasticsearch/ iceberg/ mongodb/ influxdb/ random/ quickwit/ http/ stdout/\n├── http_config_provider/ HTTP config provider end-to-end\n├── runtime/ Runtime-only tests (error isolation, missing plugin, ...)\n└── api/ HTTP control plane tests\n```\n\n### Container naming + cleanup\n\nEvery test container MUST be named with the `iggy-test-` prefix so a single\n`docker ps -aqf 'name=^iggy-test-'` sweep reaps them (`just clean-test-containers`,\nalias `ctc`; the `test`/`nextest` recipes also reap on exit). New fixtures get\nthis for free by naming through `fixtures::unique_container_name(\"\")`, which\nappends a uuid for parallel-safe ephemeral containers. Reuse fixtures use a fixed\n`iggy-test-` literal instead, because the stable name is what lets later\nnextest processes attach to the same container via `ReuseDirective::Always`. Do\nnot hand-roll a different prefix - the sweep only finds `iggy-test-*`.\n\nAs of now only `elasticsearch` and `doris` share a container. Both have a slow\nboot (JVM / Doris FE+BE, tens of seconds), so reusing one across the whole\nregression beats spinning a fresh one per test. Every other fixture boots a\nfresh, ephemeral container per test - cheap enough that the isolation is worth\nmore than the reuse.\n\n### The `#[iggy_harness]` proc macro\n\nEach integration test is annotated with this macro from the `integration` crate. It boots an in-process Iggy server (and optionally the connectors runtime) for the duration of the test, runs the seeds, then injects a `&TestHarness` and your fixture.\n\n```rust\nuse integration::harness::seeds;\nuse integration::iggy_harness;\n\n#[iggy_harness(\n server(connectors_runtime(config_path = \"tests/connectors/postgres/sink.toml\")),\n seed = seeds::connector_stream\n)]\nasync fn json_messages_sink_stores_as_bytea(\n harness: &TestHarness,\n fixture: PostgresSinkFixture,\n) {\n let client = harness.root_client().await.unwrap();\n let pool = fixture.create_pool().await.expect(\"Failed to create pool\");\n fixture.wait_for_table(&pool, \"iggy_messages\").await;\n // ... send messages, assert on rows\n}\n```\n\nKey pieces:\n\n- `server(connectors_runtime(config_path = ...))` boots the connectors runtime against the given TOML.\n- `seed = seeds::connector_stream` runs the named seed to create the test stream/topic.\n- `fixture: PostgresSinkFixture` - the test harness calls `PostgresSinkFixture::setup()` (which spins up the testcontainer) and injects the result.\n- `harness: &TestHarness` (defined in `core/integration/src/harness/orchestrator/harness.rs`) exposes `.root_client().await`, `.connectors_runtime().expect(\"...\").http_url()`, etc. - `connectors_runtime()` returns `Option`, so `.expect()` first.\n\n### Fixture pattern (`TestFixture` trait)\n\nEach fixture implements `integration::harness::TestFixture`. Pattern:\n\n```rust\n// core/integration/tests/connectors/fixtures//container.rs\n\nuse testcontainers_modules::{\n ,\n testcontainers::{ContainerAsync, runners::AsyncRunner},\n};\n\npub struct MyContainer {\n container: ContainerAsync<::Image>,\n pub connection_string: String,\n}\n\nimpl MyContainer {\n pub async fn start() -> Result {\n let container = ::Image::default().start().await?;\n let port = container.get_host_port_ipv4(DEFAULT_PORT).await?;\n Ok(Self {\n container,\n connection_string: format!(\"...://localhost:{port}\"),\n })\n }\n}\n```\n\nThe fixture itself wraps the container and implements `TestFixture::setup()` to spin it up and inject env vars that the runtime picks up.\n\n### Env-var injection via `ConfigEnv`\n\nThe runtime's config structs derive `ConfigEnv` (`configs_derive::ConfigEnv`, applied in `runtime/src/configs/connectors.rs` to `SinkConfig`, `SourceConfig`, `StreamConsumerConfig`, `StreamProducerConfig`). Fields are addressable by env var with a path-built prefix. Three concrete forms observed in `core/integration/tests/connectors/fixtures/postgres/container.rs`:\n\n```rust\n// Plugin-config fields (your sink/source's own config struct):\n// IGGY_CONNECTORS___PLUGIN_CONFIG_\npub const ENV_SINK_CONNECTION_STRING: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PLUGIN_CONFIG_CONNECTION_STRING\";\n\n// Indexed nested fields (per-stream entries are a Vec):\n// IGGY_CONNECTORS___STREAMS__\npub const ENV_SINK_STREAMS_0_STREAM: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_STREAMS_0_STREAM\";\n\n// Top-level scalar fields on SinkConfig/SourceConfig itself:\n// IGGY_CONNECTORS___\npub const ENV_SINK_PATH: &str =\n \"IGGY_CONNECTORS_SINK_POSTGRES_PATH\";\n```\n\n`` is `SINK` or `SOURCE`. `` is the connector key (TOML key field, uppercased). `PLUGIN_CONFIG` is the literal segment used when reaching into your plugin's nested config. Fields marked `#[config_env(skip)]` (e.g., `transforms`, `plugin_config` as a whole) are NOT env-addressable. primitive leaves marked `#[config_env(leaf)]` are.\n\nThe fixture sets these in `setup()` before the runtime reads its config. This is how a dynamic container port reaches a static TOML.\n\n### Test naming for integration tests\n\nIntegration tests use a **descriptive declarative** style, not the unit-test `given_should` form. Match the codebase:\n\n```rust\nasync fn json_messages_sink_stores_as_bytea(...) { ... }\nasync fn binary_messages_sink_stores_as_bytea(...) { ... }\nasync fn json_messages_sink_stores_as_jsonb(...) { ... }\nasync fn delete_after_read_source_removes_rows_after_producing(...) { ... }\nasync fn processed_column_source_marks_rows_after_producing(...) { ... }\nasync fn state_persists_across_connector_restart(...) { ... }\nasync fn restart_sink_connector_continues_processing(...) { ... }\n```\n\nPattern: `__`. Reads like a sentence describing what the test proves.\n\n### Polling vs sleeping\n\nIntegration tests poll until an expected condition - never `sleep(big_duration)`. Constants `POLL_ATTEMPTS` and `POLL_INTERVAL_MS` defined per-backend `mod.rs`. Fixture helpers like `PostgresSinkFixture::fetch_rows_as` and `PostgresSinkFixture::wait_for_table` encapsulate the pattern.\n\n### When to add an integration test\n\n- New sink/source plugin → at minimum one happy-path test per supported `Schema` and per supported plugin mode. Concrete coverage in `core/integration/tests/connectors/postgres/`: 3 sink tests (`json_messages_sink_stores_as_bytea`, `binary_messages_sink_stores_as_bytea`, `json_messages_sink_stores_as_jsonb`) backed by 3 sink fixtures, plus 5 source fixtures (`Json`, `Jsonb`, `Bytea`, `Delete`, `Mark`) and a dedicated `restart.rs` for state-survival.\n- Bug fix in a plugin → regression test reproducing the bug.\n- Behavior that crosses the FFI boundary (state restart, transform chain, schema decode) → integration test, not unit test.\n\n### When NOT to add an integration test\n\n- Pure logic (query builder, payload converter) - that's a unit test.\n- Config validation - unit test.\n- Code paths already covered by an equivalent test for another backend, when the new backend has identical behavior.\n\n## SDK + runtime tests\n\n- `core/connectors/sdk/tests/` - cross-module SDK concerns. In tree: `protobuf_integration.rs`, `flatbuffer_integration.rs` (round-trip schema decoders/encoders).\n- Runtime tests live under `core/integration/tests/connectors/runtime/` and `core/integration/tests/connectors/api/`. They exercise the runtime alone (no plugin-specific backend) - error isolation, missing plugin, invalid config, HTTP API contract.\n\n## CI / verification flow\n\nBefore pushing:\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p --all-targets -- -D warnings\ncargo test -p \n```\n\nFor integration tests, the relevant package is `integration`. There is **no `--test connectors` target** - the `integration` crate has no `[[test]]` entries and `tests/` is structured as a single binary entry through `tests/mod.rs` with `connectors/`, `server/`, `cli/`, etc. as nested modules. Run via name filter:\n\n```text\ncargo test -p integration -- connectors::postgres::postgres_sink\ncargo test -p integration -- connectors::postgres:: # all postgres tests\ncargo test -p integration -- connectors:: # all connector tests\n```\n\nIntegration tests assume Docker is available locally and pull images from a public registry on first run.\n\n## Common pitfalls\n\n1. **Skipping integration tests for a real-infra plugin.** Unit tests don't catch FFI/lifecycle/encoding regressions.\n2. **Forgetting the four canonical source state tests.** This is the most common review comment.\n3. **Mocking the client library** instead of using `testcontainers` - test exercises the mock.\n4. **`#[tokio::test]` in `src/lib.rs`** - pulls tokio test macros into prod build. Use `Runtime::new()` instead. (`#[tokio::test]` is fine in `tests/` directories, which compile separately.)\n5. **`unwrap()` in test assertions** - prefer `assert_eq!` / `expect(\"descriptive\")` so failures tell you which step blew up.\n6. **Tests depending on system time** - inject the clock or use `tokio::time::pause()`.\n7. **Hidden coupling via shared globals** (static `AtomicU64`, env vars) - tests must be independent and parallelizable.\n8. **Asserting on `Display` of an error** - use `matches!(err, Error::Variant(_))`. `assert_eq!(err.to_string(), ...)` breaks on message tweaks.\n9. **Sleeping a fixed duration** instead of polling - causes flakes when CI is slow.\n\n## Examples worth copying\n\n| Concern | Where |\n| ----------------------------------- | -------------------------------------------------------------------- |\n| Source state round-trip | `sources/random_source/src/lib.rs::tests` |\n| Config + query-building unit tests | `sinks/postgres_sink/src/lib.rs::tests` |\n| Real-infra sink integration tests | `core/integration/tests/connectors/postgres/postgres_sink.rs` |\n| Real-infra source integration tests | `core/integration/tests/connectors/postgres/postgres_source.rs` |\n| Restart + state survival | `core/integration/tests/connectors/postgres/restart.rs` |\n| Wiremock-driven HTTP sink test | `core/integration/tests/connectors/http/http_sink.rs` |\n| Testcontainer wrapper | `core/integration/tests/connectors/fixtures/postgres/container.rs` |\n| Fixture impl `TestFixture` | `core/integration/tests/connectors/fixtures/postgres/sink.rs` |\n| Runtime-only error-isolation | `core/integration/tests/connectors/runtime/error_isolation.rs` |\n| Transform tests | `sdk/src/transforms/add_fields.rs::tests`, `filter_fields.rs::tests` |\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n\n\n---\nname: connector-transform\ndescription: Add a new transform under `core/connectors/sdk/src/transforms/`. Transforms mutate, filter, or convert messages between source/encode and decode/sink. Use when adding field-level transforms (add/delete/update/filter), format conversions, envelope unwrapping, or similar config-driven message manipulations for Apache Iggy connectors. Use for transform authoring. NOT for runtime internals or sink/source loops.\n---\n\n# Adding a New Apache Iggy Connector Transform\n\nA transform implements `iggy_connector_sdk::transforms::Transform`. It\nruns after decode (for sinks) or before encode (for sources), in a chain\nconfigured per-connector. Config-driven, composable.\n\n> **Universal connector rules** live in\n> [connectors-overview](../connectors-overview/SKILL.md). This skill covers transform authoring only.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [File map](#file-map)\n- [The trait](#the-trait)\n- [Steps to add a new transform](#steps-to-add-a-new-transform)\n- [Rules](#rules)\n- [Common pitfalls](#common-pitfalls)\n- [Before declaring done](#before-declaring-done)\n\n## STOP and ask the user before\n\n- Adding a new transform that mutates payload SHAPE (renaming PK fields, dropping schema-required columns) - breaks downstream sinks silently.\n- Introducing non-determinism (system time, RNG without seed) - breaks tests + replay.\n- Changing the `Transform` trait signature (sync vs async, return type) - cascades to every transform.\n- Making transforms hold mutable runtime state - they're constructed once and `Arc`-cloned across tasks. mutation requires interior locking and is rarely what you actually want.\n\n## File map\n\n```text\nsdk/src/transforms/\n├── mod.rs Transform trait, TransformType enum, from_config() factory,\n│ FieldValue + ComputedValue helpers\n├── add_fields.rs AddFields - append static or computed fields to JSON\n├── delete_fields.rs DeleteFields - remove fields by key\n├── update_fields.rs UpdateFields - conditional field replacement\n├── filter_fields.rs FilterFields - keep/drop messages by key/value pattern\n├── unwrap_envelope.rs UnwrapEnvelope - promote nested field to top-level payload\n├── proto_convert.rs ProtoConvert - bidirectional Proto ⇄ Json/Text/Raw/etc.\n├── flatbuffer_convert.rs FlatBufferConvert\n├── avro_convert.rs AvroConvert\n└── json/ Shared JSON helpers (path traversal, ops)\n```\n\n## The trait\n\n```rust\n// sdk/src/transforms/mod.rs - Transform\npub trait Transform: Send + Sync {\n fn r#type(&self) -> TransformType;\n fn transform(\n &self,\n metadata: &TopicMetadata,\n message: DecodedMessage,\n ) -> Result, Error>;\n}\n```\n\nSemantics:\n\n- **`&self`** - transforms are immutable post-construction. Internal state (compiled regex, parsed paths, schema descriptors) is computed in `new()` and stored as plain struct fields. No locks needed - the whole instance is shared via `Arc`.\n- Takes `message` **by value** - the transform owns the `DecodedMessage` and may mutate freely without cloning.\n- Returns `Ok(Some(msg))` to pass through (possibly modified), `Ok(None)` to **filter** (drop silently), `Err(e)` for actual failures.\n- `&TopicMetadata` is for routing-aware logic. Most transforms ignore it.\n\n## Steps to add a new transform\n\n### 1. Create the file\n\n`sdk/src/transforms/my_transform.rs`. Apache 2.0 license header. Code reads top to bottom - config struct, then transform struct, then `impl`, then helpers.\n\n```rust\n/* Apache 2.0 header */\n\nuse crate::transforms::{Transform, TransformType};\nuse crate::{DecodedMessage, Error, Payload, TopicMetadata};\nuse serde::{Deserialize, Serialize};\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\npub struct MyTransformConfig {\n pub key: String,\n pub option: Option,\n}\n\npub struct MyTransform {\n config: MyTransformConfig,\n}\n\nimpl MyTransform {\n pub fn new(config: MyTransformConfig) -> Result {\n if config.key.is_empty() {\n return Err(Error::InvalidConfigValue(\n \"my_transform.key must not be empty\".into(),\n ));\n }\n Ok(Self { config })\n }\n}\n\nimpl Transform for MyTransform {\n fn r#type(&self) -> TransformType {\n TransformType::MyTransform\n }\n\n fn transform(\n &self,\n _metadata: &TopicMetadata,\n mut message: DecodedMessage,\n ) -> Result, Error> {\n // Most transforms operate only on Payload::Json - non-JSON is a no-op.\n let Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n };\n\n // Mutate `value` in place via simd_json::OwnedValue API.\n // Return Ok(None) to filter, Ok(Some(message)) to pass through.\n\n Ok(Some(message))\n }\n}\n```\n\n### 2. Wire into `transforms/mod.rs`\n\nThree edits, all required:\n\n(a) Module declaration + re-export at the top:\n\n```rust\nmod my_transform;\npub use my_transform::{MyTransform, MyTransformConfig};\n```\n\n(b) Discriminant in `TransformType`:\n\n```rust\npub enum TransformType {\n AddFields,\n // ...\n MyTransform, // snake_case form in TOML: \"my_transform\"\n}\n```\n\n(c) Branch in `from_config()`:\n\n```rust\nTransformType::MyTransform => {\n // For new transforms, prefer Error::InvalidConfigValue with formatted context -\n // it preserves the underlying serde_json error. Existing code uses bare\n // Error::InvalidConfig for most variants (UnwrapEnvelope is the exception).\n let cfg: MyTransformConfig = serde_json::from_value(raw.clone())\n .map_err(|error| Error::InvalidConfigValue(format!(\"my_transform: {error}\")))?;\n Ok(Arc::new(MyTransform::new(cfg)?))\n}\n```\n\n`from_config()` returns `Arc` so the same instance is shared across the pipeline.\n\n### 3. Document\n\nAdd a TOML usage example to `sdk/README.md`:\n\n```toml\n[transforms.my_transform]\nenabled = true\nkey = \"some_field\"\noption = true\n```\n\n### 4. Tests\n\nBottom of `my_transform.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn given_matching_field_should_transform() { ... }\n\n #[test]\n fn given_non_json_payload_should_passthrough() { ... }\n\n #[test]\n fn given_filter_condition_should_drop_message() { /* assert Ok(None) */ }\n\n #[test]\n fn given_invalid_config_should_error_in_new() {\n let cfg = MyTransformConfig { key: \"\".into(), option: None };\n assert!(matches!(MyTransform::new(cfg), Err(Error::InvalidConfigValue(_))));\n }\n\n fn topic_meta() -> TopicMetadata {\n TopicMetadata { stream: \"s\".into(), topic: \"t\".into() }\n }\n\n fn json_message(json: serde_json::Value) -> DecodedMessage { /* ... */ }\n}\n```\n\n## Rules\n\n### Operate on JSON only (mostly)\n\nMost field-level transforms only make sense for `Payload::Json`. Convention (see `add_fields.rs::AddFields::transform`): silently pass through non-JSON:\n\n```rust\nlet Payload::Json(ref mut value) = message.payload else {\n return Ok(Some(message));\n};\n```\n\nFormat-conversion transforms (`proto_convert`, `avro_convert`, `flatbuffer_convert`) are the exception - they explicitly convert between variants.\n\n### Mutate `OwnedValue` in place, don't clone-then-replace\n\n`simd_json::OwnedValue` is mutable in place:\n\n```rust\nif let OwnedValue::Object(ref mut obj) = value {\n obj.insert(key.into(), new_value);\n}\n```\n\nDo NOT clone the value, mutate the clone, then `message.payload = Payload::Json(new)`. The whole point of `&mut value` is in-place mutation.\n\n### Filter semantics\n\n`Ok(None)` drops the message **silently** - no error log, no metric. By design (filters succeed when they decide to drop). For visibility, log at `debug!` level.\n\n### Config validation\n\nValidate in `new()` and return `Err(Error::InvalidConfigValue(...))`. Don't `panic!`. Don't validate on the hot path.\n\nFor transforms with regex or compiled patterns, compile once in `new()` and store the compiled form.\n\n### Stateless except for config\n\nTransforms are constructed once at startup and `Arc`-cloned across tasks. Must be `Send + Sync`. No mutable runtime state - if you need accumulator-like behavior, that's not a `Transform`, that's a sink.\n\n### Compiled state goes in the struct, not behind a lock\n\nThings derived from config (compiled regex, parsed paths, schema descriptors) should be computed in `new()` and stored as plain struct fields. The whole instance is shared via `Arc`, so per-field locking is unnecessary. See `sdk/src/transforms/filter_fields.rs::FilterFields::new` for the canonical pattern: regexes compile in `new()`, get stored as struct fields, are then read via `&self`.\n\n### `transform()` is sync\n\nThe trait method is sync (not `async`). Don't propose making it async - that would require pervasive change across the runtime. Schemas and other external resources should be loaded in `new()` (potentially via a fallible `try_new`), not lazily during `transform()`.\n\n### Errors\n\n| Scenario | Variant |\n| ---------------------------------- | -------------------------------------------------- |\n| Config validation in `new()` | `Error::InvalidConfigValue(\"transform_name: ...\")` |\n| Per-message validation (bad shape) | `Error::InvalidRecordValue(\"...\")` |\n| Schema resolution failure | `Error::InvalidTransformer` |\n| Format conversion failure | `Error::InvalidPayloadType` |\n\nReturning `Err` from `transform()` stops the message and is logged. For non-fatal field-level issues (field missing when transform expects it), prefer pass-through (`Ok(Some(message))`) unless config explicitly says to fail.\n\n## Common pitfalls\n\n1. **Forgetting one of the three `mod.rs` edits** - module declaration, `TransformType` variant, `from_config()` branch.\n2. **`#[serde(rename_all = \"snake_case\")]` mismatch** between enum variant name and TOML key.\n3. **Cloning `OwnedValue`** when in-place mutation would work.\n4. **Validating config inside `transform()`** instead of `new()`.\n5. **`std::sync::Mutex` for internal state** on the hot path - precompute in `new()` and store as plain struct fields. Transforms run on every message. locks are wasted work.\n6. **Returning `Err` for a filter** - filters use `Ok(None)`.\n\n## Before declaring done\n\n```text\ncargo fmt --all\ncargo sort --no-format --workspace\ncargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings\ncargo test -p iggy_connector_sdk --all-features\n```\n\nVerify by adding the transform to `core/connectors/runtime/example_config/connectors/` and running the runtime - the transform should apply to each message.\n\nIf the transform changes payload shape sufficiently to break downstream sinks (e.g., renaming primary keys), call it out in the PR description.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n\n\n---\nname: connectors-overview\ndescription: Entry point and index for the Apache Iggy Connectors subsystem (`core/connectors/`). Load this first when working anywhere under `core/connectors/` - runtime, SDK, sinks, sources, or transforms. Routes to the focused per-area skills.\n---\n\n# Apache Iggy Connectors - Overview\n\nRepo-wide rules (Apache headers, fmt/sort/clippy order, idiomatic Rust traits, imports at top, no `cargo install`, LLM-slop tells) live in [AGENTS.md](../../../AGENTS.md). This file owns **connector-specific** rules and routes to the per-area skills.\n\n## Contents\n\n- [STOP and ask the user before](#stop-and-ask-the-user-before)\n- [What sinks and sources are](#what-sinks-and-sources-are)\n- [Which skill to load](#which-skill-to-load)\n- [Stick to conventions](#stick-to-conventions)\n- [Connector-wide rules](#connector-wide-rules)\n- [Benchmark mode (per-batch timing observability)](#benchmark-mode-per-batch-timing-observability)\n- [Logging format](#logging-format)\n- [Exemplars](#exemplars)\n- [Concrete efficiency patterns](#concrete-efficiency-patterns)\n- [Drop accounting](#drop-accounting)\n- [Common review smells](#common-review-smells)\n- [File map](#file-map)\n- [Cite by symbol, not line number](#cite-by-symbol-not-line-number)\n\n## STOP and ask the user before\n\n- Bumping `iggy_connector_sdk` MAJOR version or changing any FFI signature in `sdk/src/{sink,source}.rs` - breaks every pre-built plugin `.so`.\n- Changing the runtime's wire conventions (postcard FFI payload structs, default consumer group naming, plugin path resolution).\n- Modifying `runtime/src/state.rs` save protocol (atomic rename + fsync ordering) - corruption risk.\n- Renaming or repurposing a `Schema` variant - decoders/encoders pinned to wire bytes.\n- Promoting a transient `Error` variant to `Permanent*` or vice versa - affects retry behavior across every plugin.\n\n## What sinks and sources are\n\nA **sink** is a plugin that **consumes** messages from Apache Iggy streams and writes them to an external system (Postgres, Elasticsearch, Iceberg, HTTP endpoint, Mongo, object store, stdout, ...). Implements `iggy_connector_sdk::Sink`.\n\nA **source** is a plugin that **produces** messages from an external system into Apache Iggy streams (poll a DB table, scroll an ES index, generate random data, ...). Implements `iggy_connector_sdk::Source`.\n\nBoth compile as `cdylib` shared libraries (`.so`/`.dylib`/`.dll`) loaded by the **connectors runtime** at startup via `dlopen`. The runtime drives lifecycle (open/handle/consume/close), bridges Apache Iggy ⇄ plugin via a small FFI, applies optional transforms, and persists source state.\n\n```text\n ┌─ optional transforms ─┐\nExternal ──poll──▶ SOURCE ──FFI──▶ RUNTIME ──encode──▶ Apache Iggy stream\n system plugin ▲ ▲\n │ │\n state save (msgpack) │\n │\n ┌─ optional transforms ─┐ │\nApache Iggy stream ──decode──▶ RUNTIME ──FFI──▶ SINK ──write──▶ External\n plugin system\n```\n\nHeaders set on the source side ride through transforms (which may modify, drop, or pass them) and arrive at the sink with `BTreeMap` preserved deterministically.\n\n## Which skill to load\n\n| Task | Skill |\n| --------------------------------------------------- | --------------------- |\n| Write a new sink plugin | `connector-sink` |\n| Write a new source plugin | `connector-source` |\n| Add schema / decoder / encoder / SDK trait surface | `connector-sdk` |\n| Change runtime internals (FFI, manager, state, ...) | `connector-runtime` |\n| Add a transform (field-level or format conversion) | `connector-transform` |\n| Write unit / integration tests for any of the above | `connector-testing` |\n\n## Stick to conventions\n\nThe connectors codebase is intentionally repetitive across plugins. Cross-plugin consistency makes review, LLM-assisted contribution, and onboarding tractable.\n\n1. **Find the closest existing plugin by shape** (DB-write -> `postgres_sink`, HTTP -> `http_sink`, polling source -> `postgres_source`).\n2. **Copy its file layout, naming, log format, error mapping, test structure** verbatim. Only the backend-specific code (the client call) should differ.\n3. **Don't reinvent existing patterns.** The SDK's `retry.rs` covers HTTP retry + circuit breaker. Existing plugins cover batching, idempotency, header encoding, secret handling.\n4. **Don't refactor unrelated code in the same PR.** Match conventions. propose convention changes separately.\n\n## Connector-wide rules\n\n### Hot loops\n\n- **Zero clone.** Use `&self`, `&str`, `&[T]`, `&Payload`. For `Payload::Json`, `try_to_bytes(&self)` serializes single-pass without cloning the `OwnedValue` tree.\n- **Move-out without cloning** via `std::mem::replace` / `std::mem::take`.\n- **`Vec::with_capacity(n)`** when building per-batch buffers. reallocation in hot path is wasted work.\n\n### Async + concurrency\n\n- **`async fn` only.** Never `.block_on()` inside a `Sink`/`Source` impl - the runtime drives the executor.\n- **Hold locks briefly.** Pattern: lock, clone/read what's needed, drop guard, then do I/O. Never hold a `Mutex` across an external `.await`.\n- **Atomic counters** (`AtomicU64`) preferred over `Mutex` for hot-path metrics.\n- **No `tokio::spawn` from plugin code.** Runtime owns lifecycle. orphans survive `close()`.\n\n### Logging\n\n- `tracing` only. `info!` for lifecycle, `debug!` per-batch detail, `warn!` recoverable, `error!` failures. Include `connector ID: {self.id}` and the connector name on every line.\n- **`verbose: bool`** on `SinkConfig`/`SourceConfig` (TOML top-level). Runtime upgrades its own per-batch `debug!` to `info!` when set. Plugins should mirror with a `verbose_logging: Option` field inside `plugin_config` (convention: `postgres_sink::PostgresSinkConfig::verbose_logging`). Default both to `false`.\n- **Never log secrets** (connection strings, API keys, tokens). Redact URLs at the log site. Credential fields must use `SecretString` (see below).\n\n### Secrets\n\nAny credential-bearing field (connection strings, API keys, bearer tokens, AWS keys) must be `SecretString` from the `secrecy` crate. Plain `String` for a credential is a review-blocker: `SecretString` redacts on `Debug`, so it is what keeps a credential out of a log line that formats the whole config.\n\n**`serde_secret::serialize_secret` EXPOSES the secret. It does not redact.** It calls `expose_secret()` and writes the plaintext. `SecretString` deliberately has no `Serialize` impl, and that absence is the protection - so adding `serialize_with` is what *unblocks* the derive and turns a compile-time guarantee into plaintext output. Use it only where the plaintext is the point: a wire payload, a persisted config, an API response that exposes credentials by design.\n\nSo the default for a plugin config struct is **derive `Deserialize`, but not `Serialize`**. `Deserialize` is required: the SDK glue deserializes the config into the plugin's own struct (`sdk/src/{sink,source}.rs` call `serde_json::from_str::` under a `DeserializeOwned` bound).\n\nWhat never happens is the return trip. The runtime holds plugin configuration as a `serde_json::Value` - parsed from TOML, posted as JSON to the control API, or injected by env var - and hands that across the FFI, so nothing re-serializes the plugin's struct. Leaving `Serialize` off makes that compiler-enforced instead of convention-enforced (`sources/http_source/src/lib.rs::HttpSourceConfig` does this, and comments the omission so nobody adds it back).\n\nPattern:\n\n```rust\nuse secrecy::{ExposeSecret, SecretString};\n\n// `Deserialize` only. Nothing re-serializes a plugin config, and leaving\n// `Serialize` off is what makes the credential unserializable rather than\n// merely un-serialized.\n#[derive(Debug, Clone, Deserialize)]\npub struct MyConfig {\n pub connection_string: SecretString,\n}\n\n// At call site:\nlet pool = PgPoolOptions::new()\n .connect(self.config.connection_string.expose_secret())\n .await?;\n```\n\nIf a config struct genuinely needs `Serialize`, `serde_secret::serialize_redacted` (and `serialize_optional_redacted`) write `[REDACTED]` in place of the value. Reach for `serialize_secret` only when the caller must get the real thing back. The sinks and sources listed below predate that helper and use the exposing one; the annotation is inert today, but it is not the protection it looks like.\n\nNote that none of this protects the credential from the runtime's own control API, which returns plugin configuration verbatim - see #3802. Plugin-side annotations are inert there because the runtime never routes through them.\n\nPlugin-side uses of the exposing helpers: `sinks/{postgres,mongodb,elasticsearch,influxdb,s3,surrealdb}_sink`, `sources/{postgres,elasticsearch,influxdb}_source`.\n\nThat list is plugin-side only, not an inventory of every caller in the tree, and the others are not all mistakes: `runtime/src/api/config.rs` puts `serialize_secret` on `HttpConfig::api_key` (inert for the same reason), and several `core/common` wire-payload types (login, create-user, change-password, PAT) use these helpers by design, because there the credential *is* the payload.\n\n### Errors\n\n- **Use `Error::PermanentHttpError` / `Error::SchemaMismatch` / `Error::CatalogCommitError` for non-retryable failures.** Returning a transient variant for bad data trips circuit breakers and hammers the backend.\n- **No `unwrap()` / `expect()`** on Results from external I/O outside tests.\n\n### Forward-compat config\n\nNew TOML fields use `#[serde(default)]` or `Option`. Adding a field must not break existing configs.\n\n## Benchmark mode (per-batch timing observability)\n\nEvery sink and source supports an opt-in `benchmark: bool` flag (`SinkConfig`/`SourceConfig`, also env `IGGY_CONNECTORS___BENCHMARK=true`). Two independent surfaces:\n\n1. **Tracing events** (only when flag is on) - one structured `info!` per batch under target `iggy_connectors::benchmark` with per-stage microsecond timings:\n - Sink: `connector_type, connector_key, stream, topic, partition_id, current_offset, batch_size, processed_count, decode_us, prepare_us, ffi_us, total_us`\n - Source: `connector_type, connector_key, stream, topic, batch_size, sent_count, decode_us, prepare_us, iggy_send_us, state_saved, state_save_us, total_us` (`state_save_us` is 0 when `state_saved` is false - emitted as flat number + bool, not an Option, so the JSON layer keeps it numeric)\n - Filter live: `RUST_LOG=iggy_connectors::benchmark=info`\n - Also emitted on startup: a one-shot `\"Benchmark mode enabled for ...\"` info log per connector.\n\n2. **Prometheus histograms** (always on, regardless of flag):\n - Metric: `iggy_connector_stage_duration_seconds{connector_key, connector_type, stage}` (both labels snake_case)\n - `stage` (snake_case): `decode`, `prepare`, `total` on both sides; `ffi` (sink only); `iggy_send`, `state_save` (source only). `decode` and `prepare` mean the same thing on both sides - decode is schema decoding, prepare is transform + encode/serialize.\n - Buckets: see `runtime/src/metrics.rs::STAGE_BUCKETS_SECONDS`\n - Example: `histogram_quantile(0.95, sum(rate(iggy_connector_stage_duration_seconds_bucket{stage=\"ffi\"}[5m])) by (le, connector_key))`\n\nThe flag only gates the verbose text event. the histogram observation is always on. Per-stage call path: `Instant::now()` + SipHash over `StageLabels` + parking_lot `RwLock::read` + `HashMap` probe + linear scan inside `Histogram::observe`. Benchmark on target hardware if you suspect the overhead matters at your batch rate.\n\n`iggy_connector_messages_filtered_total{connector_key, connector_type}` counts intentional drops by transforms returning `Ok(None)`, distinct from `errors_total` for unexpected drops. Both also exposed in `/stats` JSON (`ConnectorStats.messages_filtered`, `.errors`).\n\nImplementation: `runtime/src/benchmark.rs` (text-event emitters +\n`as_micros`), `runtime/src/metrics.rs::{Stage, SinkLabels, SourceLabels,\nobserve_stage_with_labels}` (pre-built label cache avoids per-batch\n`String` clones. `&str`-based wrappers are `#[cfg(test)]` only). Tests:\n`runtime/src/metrics.rs::tests::given_stage_histogram_*` (unit),\n`core/integration/tests/connectors/runtime/benchmark.rs` (integration:\ntext emission, env override, disabled-flag, histograms via `/metrics`,\nJSON log format, parser unit tests).\n\n## Logging format\n\n`runtime/src/configs/runtime.rs::LoggingConfig` exposes `format = \"text\" | \"json\"` (text default), env `IGGY_CONNECTORS_LOGGING_FORMAT=json`. `runtime/src/log.rs::init_logging` matches on `(telemetry.enabled, format)` to install the right `fmt::layer()`/`fmt::layer().json()` combined with the OpenTelemetry layer when telemetry is on. JSON applies to the stdout layer only - OTel export pipeline unaffected by this flag.\n\n## Exemplars\n\n| Purpose | Plugin |\n| --------------------------------------------------------- | --------------------------------------------------------------------------------------- |\n| Simplest sink (read first) | `sinks/stdout_sink/` |\n| Real-infra sink + integration tests | `sinks/postgres_sink/` + `integration/tests/connectors/postgres/postgres_sink.rs` |\n| Feature-rich sink config (validation, batch modes, retry) | `sinks/http_sink/` |\n| Atomic counters on hot path | `sinks/mongodb_sink/` |\n| Simplest source (4 canonical state tests) | `sources/random_source/` |\n| Real-infra source | `sources/postgres_source/` + `integration/tests/connectors/postgres/postgres_source.rs` |\n\nRead the relevant exemplar end-to-end before writing or modifying a connector.\n\n## Concrete efficiency patterns\n\nEach implemented in at least one in-tree plugin or runtime path.\n\n| Pattern | Where | When |\n| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- |\n| `std::mem::replace(&mut message.payload, Payload::Raw(vec![]))` | `sinks/http_sink/src/lib.rs` (`send_individual`, `send_ndjson`, `send_json_array`) | Take payload from `&mut ConsumedMessage` no clone |\n| `std::mem::take(&mut batch)` | `runtime/src/sink.rs::consume_messages`, `manager/{sink,source}.rs` | Drain a `Vec` field with one move |\n| `std::mem::swap` | `sinks/http_sink` (retry_delay / max_retry_delay validation) | Fix config-field ordering in `new()` |\n| `Vec::with_capacity(messages.len())` | Every batching sink | Pre-size per-batch buffers |\n| `bytes::Bytes::from(body)` | `sinks/{http,influxdb}_sink` | Cheap ref-counted body sharing across retries |\n| `AtomicU64` counters | `sinks/mongodb_sink` | Lock-free per-message metric updates |\n| `Payload::try_to_bytes(&self)` | `sdk/src/lib.rs::Payload::try_to_bytes` | Serialize `Payload::Json` no-clone |\n| `simd_json::to_owned_value(&mut bytes)` | `sdk/src/decoders/json.rs::JsonStreamDecoder::decode` | In-place JSON parse |\n| Brief-lock fetch | `sources/postgres_source/src/lib.rs::poll_tables` | Lock read -> drop -> I/O -> lock write |\n| `is_transient_error(&e)` SQLSTATE mapping | `sinks/postgres_sink/src/lib.rs` | Driver errors -> retry vs `PermanentHttpError` |\n| Two-constructor pattern (`new` lenient + `try_new` strict + `Default`) | `sdk/src/decoders/avro.rs::AvroStreamDecoder` | Stateful decoders/encoders where schema can fail |\n| Duplicate-ID FFI guard | `sdk/src/sink.rs::sink_connector!`, `sdk/src/source.rs::source_connector!` | Prevents silent data loss on reopen-without-close |\n| `restart_guard.try_lock()` | `runtime/src/manager/{sink,source}.rs::restart_connector` | No thundering-herd restarts |\n| `tokio::time::timeout(..., handle).await` + `handle.abort()` on timeout | `runtime/src/manager/source.rs::stop_connector` | Bounded shutdown + leak prevention |\n| `flume::unbounded()` channel | `runtime/src/source.rs::spawn_source_handler` / `source_forwarding_loop` | MPSC handoff from SDK async task to runtime loop |\n| `tokio::sync::watch::channel(())` | `sdk/src/{sink,source}.rs`, `runtime/src/sink.rs`, `runtime/src/manager/*` | One-shot shutdown broadcast |\n| `dashmap::DashMap` | `runtime/src/manager/sink.rs`, `source.rs::SOURCE_SENDERS`, SDK `INSTANCES` | Lock-free concurrent keyed access |\n| `secrecy::SecretString` + `iggy_common::serde_secret::serialize_secret` | `sinks/postgres_sink::PostgresSinkConfig::connection_string` | `Debug` redacts; `serialize_secret` EXPOSES |\n\n## Drop accounting\n\n- **Filter drops** (transform returning `Ok(None)`) bump `messages_filtered`.\n- **Every non-filter drop** bumps `errors`. Inside the per-batch loops the count is accumulated in a local and flushed once after the loop via `inc_errors_by_with_labels` (filters likewise via `inc_messages_filtered_with_labels`) - one `Family` lookup per batch, not per message. One-shot drops outside a loop call `inc_errors_with_labels` directly. Hot-path calls always use the pre-built `SinkLabels`/`SourceLabels` cache. `&str`-based wrappers on `Metrics` are `#[cfg(test)]`-only.\n\n## Common review smells\n\n- `.clone()` on `Payload::Json` or `OwnedValue` - use `try_to_bytes(&self)`.\n- `&mut self` on `Sink::consume` or `Source::poll` impls - won't compile, flag any creative workaround.\n- `std::sync::Mutex` held across `.await` - swap for `tokio::sync::Mutex`.\n- Missing `[lib] crate-type = [\"cdylib\", \"lib\"]` in plugin `Cargo.toml`.\n- Source plugin without the four canonical state tests (see `connector-testing`).\n- New silent message drop without a metric increment.\n- Wrapping `format!()` around args passed to `error!`/`warn!`/`info!`/`debug!` - eager `format!` allocates even when level filters the line out. Pass args directly: `error!(\"foo: {x}\")` or `error!(error = %x, \"foo\")`.\n- Logging a connection string, API key, or token.\n- Plain `String` for a credential field - use `SecretString`.\n- `tokio::spawn` inside plugin code - runtime owns lifecycle.\n- `std::time::SystemTime::now()` in transforms - non-deterministic, breaks tests.\n\n## File map\n\n```text\ncore/connectors/\n├── README.md\n├── runtime/\n│ ├── src/\n│ │ ├── main.rs FFI structs (SinkApi, SourceApi), plugin path resolution\n│ │ ├── manager/{sink,source}.rs Lifecycle, restart_guard, status transitions\n│ │ ├── configs/ Local + HTTP config providers, ConfigEnv derive\n│ │ ├── {sink,source,stream,transform,state,context}.rs\n│ │ ├── benchmark.rs, log.rs, metrics.rs\n│ │ └── api/ HTTP control plane (/sinks, /sources, /stats, /metrics)\n│ └── example_config/ Reference TOML\n├── sdk/\n│ └── src/\n│ ├── lib.rs Sink/Source traits, Payload, Schema, Error\n│ ├── {sink,source}.rs FFI containers + sink_connector!/source_connector! macros\n│ ├── decoders/encoders/ Per-schema (json/raw/text/proto/flatbuffer/avro)\n│ ├── transforms/ add/delete/update/filter fields, format conversions\n│ └── retry.rs CircuitBreaker, HttpRetryMiddleware\n├── sinks/_sink/\n└── sources/_source/\n\ncore/integration/tests/connectors/ # Real-infra integration tests\n├── fixtures/ # testcontainers-modules wrappers per backend\n├── postgres/, elasticsearch/, mongodb/, iceberg/, influxdb/, quickwit/, http/, delta/\n├── runtime/ # error_isolation, benchmark\n└── api/ # HTTP API endpoint tests\n```\n\n## Cite by symbol, not line number\n\nSkills cite files by path + symbol (`PostgresContainer::start` in `core/integration/tests/connectors/fixtures/postgres/container.rs`). Line numbers go stale on the first refactor. symbol names survive. Grep the symbol if you need the exact spot.\n\n---\n\nDiscussion / help: see [AGENTS.md](../../../AGENTS.md#discussion-and-support).\n"}