{"owner":"Giskard-AI","repo":"giskard-oss","hasSkills":true,"totalSkillsCount":19,"totalTokensCount":10679,"categories":["claude-rule","anthropic-skill","cursor-rule","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":[".claude/CLAUDE.md",".claude/skills/check/SKILL.md",".claude/skills/fix-cve/SKILL.md",".claude/skills/giskard-oss-quality-gate/SKILL.md",".claude/skills/setup-env/SKILL.md",".cursor/rules/documentation.mdc",".cursor/rules/guidelines.mdc",".cursor/rules/python.mdc",".cursor/skills/giskard-checks-new-check/SKILL.md",".cursor/skills/giskard-checks-new-check/reference.md",".cursor/skills/giskard-oss-pr-ready/SKILL.md","AGENTS.md","libs/giskard-agents/.cursor/rules/generator-adapter.mdc","libs/giskard-checks/.cursor/rules/02-project.mdc","libs/giskard-checks/.cursor/rules/03-development.mdc","libs/giskard-llm/.cursor/rules/01-architecture.mdc","libs/giskard-llm/.cursor/rules/02-testing.mdc","libs/giskard-llm/.cursor/rules/03-development.mdc","libs/giskard-llm/.cursor/rules/04-provider-docs.mdc"],"skills":{".claude/CLAUDE.md":"@AGENTS.md\n",".claude/skills/check/SKILL.md":"---\nname: check\ndescription: Run the giskard-oss verification gate (format, check, unit tests) for a package\ndisable-model-invocation: true\n---\n\n# check\n\nRun the giskard-oss verification gate that `AGENTS.md` mandates before any task is \"done\":\n\n```\nmake format && make check && make test-unit PACKAGE=\n```\n\n## Usage\n\n`/check ` — run the full gate scoped to one lib's unit tests.\n\n`/check` — run the gate; unit tests cover all libs (slower).\n\n`` is a directory under `libs/`: `giskard-agents`, `giskard-checks`, `giskard-core`, `giskard-llm`, `giskard-scan`.\n\n## What to run\n\nWith a package:\n\n```bash\nmake format && make check && make test-unit PACKAGE=\n```\n\nWithout a package:\n\n```bash\nmake format && make check && make test-unit\n```\n\n`format` and `check` are repo-wide and take no `PACKAGE`. Only `test-unit` is scoped.\n\n## Notes\n\n- Run the three steps as one `&&` chain so a failure stops the gate early.\n- Report failures with the command's output — do not claim the gate passed unless every step exited 0.\n- `make format` mutates files (ruff format + check --fix); show the resulting diff if anything changed.\n",".claude/skills/fix-cve/SKILL.md":"---\nname: fix-cve\ndescription: Use when pip-audit (or a CVE/GHSA advisory, Dependabot, or security scan) flags a vulnerable Python dependency in giskard-oss and it needs upgrading to a fixed version in the uv lockfile.\n---\n\n# fix-cve\n\nRemediate a vulnerable dependency flagged by `pip-audit` (or any CVE/GHSA advisory) by upgrading it in the **uv lockfile**, then verify with the repo gate.\n\n`pip-audit` runs as part of `make check` (the `security` target). The advisory looks like:\n\n```\nName Version ID Fix Versions\n------- ------- ------------------- ------------\nmsgpack 1.1.2 GHSA-6v7p-g79w-8964 1.2.1\n```\n\nYou need: the **package name** and a **fix version**.\n\n## Procedure\n\n```bash\n# 1. Upgrade the package in the lockfile (works for direct AND transitive deps).\nuv lock --upgrade-package \n\n# 2. Apply to the environment.\nuv sync\n\n# 3. Verify the advisory is gone — re-run the exact audit command.\nuv run pip-audit --skip-editable\n```\n\nThen run the full repo gate (it re-runs `pip-audit` via the `security` target, plus format/lint/types/tests):\n\n```bash\nmake format && make check && make test-unit\n```\n\nOr invoke `/check` (the project verification-gate skill).\n\n## Key facts\n\n- **Always upgrade in the lockfile, not the environment.** Use `uv lock --upgrade-package `, NOT `pip install` / `uv pip install`. The lockfile is the source of truth; a `pip install` fix is lost on the next `uv sync`.\n- **Transitive deps are fixed the same way.** Most flagged packages (e.g. `msgpack`, pulled in by `cachecontrol`) are not in any `pyproject.toml`. You still upgrade them by name with `uv lock --upgrade-package ` — no `pyproject.toml` edit needed. Do NOT add a direct dependency just to pin it.\n- **`uv lock --upgrade-package ` takes the newest version the constraints allow.** Check the resulting `uv.lock` entry to confirm the version actually moved to the fix version — a cap may have held it below.\n\n### When a constraint caps the version below the fix\n\nIf another dependency caps the package below the fix version, the bare upgrade silently stays on the old version, and forcing it with `uv lock --upgrade-package \">=\"` **fails with a resolution error** (`No solution found...`) — uv constraints can only *narrow* the version set, never expand past a cap. Escalate in this order:\n\n1. **Upgrade the capping parent.** Find who imposes the cap (`uv tree --invert --package `), then `uv lock --upgrade-package `. A newer parent often relaxes the cap. Preferred — keeps the dependency graph honest.\n2. **Override the transitive dep** if no parent upgrade resolves it. Add to `pyproject.toml`:\n ```toml\n [tool.uv]\n override-dependencies = [\">=\"]\n ```\n Overrides *expand* the allowed set (the only escape hatch a cap can't block), then re-run `uv lock`. Use sparingly — it ignores the parent's declared constraint, so re-verify nothing breaks with the full gate.\n- **Lockfile-only changes have no \"affected lib\".** Run the full `make test-unit` (all packages), not a single `PACKAGE=` scope — the dep could affect any lib.\n\n## Verifying the fix\n\nDo not assert \"fixed\" — show evidence:\n\n1. `uv run pip-audit --skip-editable` prints **\"No known vulnerabilities found\"** (editable `giskard-*` libs in the skip list are expected and harmless).\n2. The `uv.lock` diff shows the package moved to the fix version (`Updated vX -> vY` from `uv lock`).\n3. `make check` passes (its `security` target re-runs the audit).\n\n## Common mistakes\n\n| Mistake | Fix |\n|---------|-----|\n| `pip install ==` to fix it | Use `uv lock --upgrade-package `; pip changes don't persist in the lockfile |\n| Editing `pyproject.toml` for a transitive dep | Not needed — `uv lock --upgrade-package` handles transitive deps by name |\n| Skipping re-audit, assuming the upgrade worked | Re-run `uv run pip-audit --skip-editable`; a constraint may have silently capped the version below the fix |\n| Force-pinning past a cap with `\">=\"` and hitting `No solution found` | Constraints can't expand past a cap — upgrade the capping parent, or use `[tool.uv] override-dependencies` |\n| Guessing the verification command | Run `/check` or `make format && make check && make test-unit`; `make check` includes `pip-audit` |\n| Scoping tests to one `PACKAGE=` | A lockfile change has no single affected lib; run all unit tests |\n",".claude/skills/giskard-oss-quality-gate/SKILL.md":"---\nname: giskard-oss-quality-gate\ndescription: Run the full code quality gate for the giskard-oss monorepo—format, lint, typecheck, and unit tests across all packages. Use when finishing a feature, before opening a PR, or when the user asks to check quality or run the gate.\ndisable-model-invocation: true\nallowed-tools: Bash(make *)\n---\n\n# giskard-oss — quality gate\n\n## Results\n\n```!\necho \"=== make format ===\" && make format 2>&1 | tail -5\necho \"=== make check ===\" && make check 2>&1 | tail -10\necho \"=== make test-unit PACKAGE=giskard-core ===\" && make test-unit PACKAGE=giskard-core 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-checks ===\" && make test-unit PACKAGE=giskard-checks 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-agents ===\" && make test-unit PACKAGE=giskard-agents 2>&1 | tail -5\n```\n\nReport pass/fail for each step. Fix root causes on failure — no `# type: ignore`, no skips. Re-run after fixing.\n",".claude/skills/setup-env/SKILL.md":"---\ndescription: Set up the full workspace dev environment. Use when the user wants to install dependencies, sync libs, or prepare the environment.\ndisable-model-invocation: true\nallowed-tools: Bash(make *) Bash(uv *)\n---\n\n## Setup dev env\n\n!`make setup 2>&1 | tail -20`\n\n## Validate installation and notify user\n\nCheck the output above for errors. If any errors occurred, try to fix them and re-run the failed step. Report whether installation was successful.\n",".cursor/rules/documentation.mdc":"---\ndescription: Documentation and commenting guidelines for an OSS, beginner-friendly codebase.\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Documentation Guidelines\n\nThis is an open-source project. Code should be understandable by newcomers who may not have prior context on the architecture.\n\n## Public API Docstrings\n\nEvery public class, method, and function **must** have a NumPy-style docstring that includes:\n- A one-line summary of what it does\n- Parameters with types and descriptions\n- Return value description\n- A short usage example when the behavior is non-obvious\n\n```python\nasync def run(self, max_steps: int | None = None) -> Chat:\n \"\"\"Execute the workflow and return the resulting chat.\n\n Parameters\n ----------\n max_steps : int, optional\n Maximum number of steps to run. If not provided, the workflow\n runs until completion.\n\n Returns\n -------\n Chat\n The completed chat with all messages and tool call results.\n\n Example\n -------\n >>> chat = await generator.chat(\"Hello!\").run()\n >>> print(chat.last.content)\n \"\"\"\n```\n\n## Inline Comments\n\n- **Do** comment non-obvious control flow, especially async orchestration, retry logic, and tool dispatch.\n- **Do** explain *why* something is done when the reason isn't obvious from the code.\n- **Don't** narrate what the code does line-by-line (\"increment counter\", \"return result\").\n- **Don't** leave TODO comments without a linked issue or clear owner.\n\n```python\n# Good — explains the why\n# Re-parse tool_calls from the raw response because LiteLLM may\n# return partial function arguments when streaming.\ntool_calls = _reparse_tool_calls(raw)\n\n# Bad — restates the code\n# Parse the tool calls\ntool_calls = _reparse_tool_calls(raw)\n```\n\n## Module-Level Docstrings\n\nEach module (`.py` file) should have a brief docstring explaining its role in the system. This helps newcomers navigate the codebase.\n\n## README and Examples\n\n- Keep `README.md` examples accurate and runnable.\n- When adding a new feature, add a corresponding example to the README.\n- Use progressive disclosure: simple usage first, advanced patterns later.\n",".cursor/rules/guidelines.mdc":"---\ndescription: Team-wide guardrails for AI-assisted development.\nalwaysApply: true\n---\n# Guidelines\n\n## No Inventions\n\nOnly make changes that are explicitly requested. Do not add unrequested features, refactors, or improvements.\n\n## Preserve Existing Code\n\nDo not remove unrelated code, functionalities, or comments without explicit reason. Pay attention to preserving existing structures.\n\n## No CODEMAP\n\nNever generate architecture maps, codemap files, or similar high-maintenance documentation that duplicates what is already in cursor rules, docstrings, and README. These files go stale fast and create maintenance burden.\n\n## Root Causes, No Temporary Fixes\n\nTrace failures to their source before making a change. Do not:\n- Add `# type: ignore` / `# pyright: ignore` without first confirming no structural fix is possible.\n- Modify test assertions to force a pass — if a test reveals a real bug, fix the bug.\n- Add `try/except` to suppress exceptions that should be fixed.\n\nIf the root cause is in a different library than where the symptom appears, say so before proposing a fix.\n\n## Minimal Scope\n\nOnly touch files directly required by the task. Do not reformat untouched lines, rename unrelated variables, or add type annotations to code you didn't otherwise change. Propose nearby improvements separately and wait for approval.\n",".cursor/rules/python.mdc":"---\ndescription: Python conventions for the giskard workspace (3.12+, Pydantic v2, async, testing).\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Python Conventions\n\n## Python Version\n\n- Python >= 3.12 required across all projects.\n- **Never use `from __future__ import annotations`** — it breaks Pydantic v2 model resolution at runtime and is unnecessary with 3.12+.\n\n## Modern Python 3.12+ Features\n\nPrefer modern syntax over legacy `typing` equivalents.\n\n### PEP 695 — Type Parameter Syntax\n\n```python\n# Good — PEP 695 inline type parameters\nclass Box[T]:\n def __init__(self, value: T): ...\n\ndef first[T](items: list[T]) -> T: ...\n\n# Bad — legacy TypeVar / Generic\nfrom typing import TypeVar, Generic\nT = TypeVar('T')\nclass Box(Generic[T]):\n def __init__(self, value: T): ...\n```\n\n### PEP 695 — Bounded and Constrained Generics\n\n```python\n# Good\nclass NumberBox[T: (int, float)]: ...\nclass Comparable[T: int]: ...\n\n# Bad\nT = TypeVar('T', int, float)\nT = TypeVar('T', bound=int)\n```\n\n### PEP 695 — `type` Statement for Type Aliases\n\n```python\n# Good — unambiguous, supports forward references and recursive types\ntype Vector = list[float]\ntype Tree[T] = T | list[Tree[T]]\n\n# Bad — legacy TypeAlias\nfrom typing import TypeAlias\nVector: TypeAlias = list[float]\n```\n\n### PEP 695 — ParamSpec in Class Syntax\n\nNiche — document with a docstring when used.\n\n```python\n# Good\nclass Wrapper[**P, R]:\n \"\"\"Generic wrapper preserving the signature of the wrapped callable.\n\n Type Parameters\n ---------------\n **P\n Parameter specification of the wrapped callable.\n R\n Return type of the wrapped callable.\n \"\"\"\n ...\n\n# Bad\nfrom typing import ParamSpec\nP = ParamSpec('P')\n```\n\n### Union Syntax (PEP 604)\n\n```python\n# Good\ndef process(value: str | int) -> str | None: ...\n\n# Bad\nfrom typing import Optional, Union\ndef process(value: Union[str, int]) -> Optional[str]: ...\n```\n\n### Built-in Generics\n\nUse built-in types directly — do not import from `typing`.\n\n```python\n# Good\ndef transform(items: list[str]) -> dict[str, int]: ...\ncoords: tuple[float, float, float]\n\n# Bad\nfrom typing import List, Dict, Tuple\ndef transform(items: List[str]) -> Dict[str, int]: ...\n```\n\n### `@override` Decorator\n\nOptional but recommended for methods overriding a parent, especially in large class hierarchies. Catches silent breakage when the parent method is renamed.\n\n```python\nfrom typing import override\n\nclass Child(Base):\n @override\n def process(self) -> None: ...\n```\n\n## Pydantic v2\n\n- Use `BaseModel` for structured data that needs serialization.\n- Prefer immutable models with `ConfigDict(frozen=True)`.\n- Discriminated unions via `@discriminated_base` / `@Cls.register(\"kind\")` from `giskard-core`.\n- Validate external inputs at boundaries; keep internal code operating on typed objects.\n\n## Imports and Naming\n\n- Absolute imports within each `giskard.*` package.\n- Import order enforced by Ruff (`isort`).\n- PEP 8 naming: `snake_case` functions/variables, `PascalCase` classes, `UPPER_CASE` constants.\n\n## Async Patterns\n\n- All workflow execution is async. Never use blocking calls inside async code.\n- Use `asyncio.gather` for parallel execution.\n- Checks and generators may be sync or async — design APIs to support both.\n\n## Error Handling\n\n- Prefer result objects (e.g. `CheckResult`) over exceptions for control flow.\n- Raise exceptions only for truly exceptional states (invalid config, programmer errors).\n- Let runners catch and wrap exceptions when appropriate.\n\n## Testing\n\n- pytest with `asyncio_mode = \"auto\"` — no `@pytest.mark.asyncio` needed.\n- Parametrize tests with descriptive IDs.\n- `@pytest.mark.functional` for tests hitting external APIs.\n- Co-locate tests under `tests/` mirroring package structure.\n- Test success, failure, error, and edge-case paths.\n",".cursor/skills/giskard-checks-new-check/SKILL.md":"---\nname: giskard-checks-new-check\ndescription: Guides implementation of new or changed eval checks in the giskard-checks package—Check subclasses, discriminated kind registration, public exports, and tests. Use when adding or modifying built-in checks, LLM-based judges, @Check.register kinds, serialization of TestCase or Scenario, or when editing files under libs/giskard-checks/src/giskard/checks related to checks.\n---\n\n# giskard-checks — new check\n\n## Scope\n\nLibrary code lives under `libs/giskard-checks/`. Follow existing patterns; use absolute imports inside `giskard.checks`. Python 3.12+, Pydantic v2.\n\nRead project rules when relevant: `libs/giskard-checks/.cursor/rules/02-project.mdc`, `03-development.mdc`.\n\n## Classify the check\n\n| Kind | Module | Base class |\n|------|--------|------------|\n| Deterministic (string, numeric, custom logic on trace) | `src/giskard/checks/builtin/` | `Check` (often shared helpers in existing builtins) |\n| Uses LLM to score or classify | `src/giskard/checks/judges/` | `BaseLLMCheck` |\n\nFor one-off or experimental logic, users can use `from_fn` / `FnCheck` without a new class—only add a new type when it is a reusable, serializable primitive.\n\n## JSONPath fields (`key` and `*_key`)\n\n### Naming and typing (required)\n\nAny `Check` field named `key` or whose name ends with `_key` is a JSONPath into the trace. It **must** be annotated with `JSONPathStr`, or `JSONPathStr | None`, or `JSONPathStr | MISSING` (import `MISSING` from `pydantic.experimental.missing_sentinel`). Otherwise `tests/core/test_jsonpath_enforcement.py` fails.\n\nInside the library, import from `..core.extraction` (module `giskard.checks.core.extraction`).\n\n### Path syntax\n\n- Every path must start with `trace.` (validated at model construction via `jsonpath_ng`).\n- Resolution runs against `{\"trace\": trace.model_dump()}`. Prefer `trace.last.outputs`, `trace.last.inputs`, `trace.last.metadata.*` over brittle indices; `trace.last` is the usual default for “current turn” data.\n\n### Reading values\n\n- **`resolve(trace, key)`** — Use when the value always comes from the trace (e.g. the subject `target_key`).\n- **`provided_or_resolve(trace, key=..., value=...)`** — Use when the user may pass an inline value **or** fall back to a path (e.g. `keyword` + `keyword_key`). If `value is not MISSING`, that wins; otherwise the path is evaluated.\n\n### Naming JSONPath fields\n\n- The field naming the **value under test** is always `target_key` (matches the Giskard Hub). It is the only accepted name for that selector and serializes as `target_key`.\n- Every **other** JSONPath field is `{static}_key`, named after the sibling static field holding the literal it would otherwise resolve (`keyword`/`keyword_key`, `context`/`context_key`).\n- Both rules are enforced by `tests/core/test_key_naming_convention.py`.\n\nOptional inline-or-path fields: type as `T | MISSING = MISSING` (and `JSONPathStr | MISSING = MISSING` for optional `*_key` fields). Pass fields directly to `provided_or_resolve`; validate combinations with a `@model_validator` if only one of (inline, `*_key`) may be set (see `StringMatching` / `ComparisonCheck`). Do not use `None` to mean \"extract from trace\"—omit the field so it stays `MISSING`. Explicit `None` is only valid when comparing against or supplying `None` as a real value (e.g. `Equals(expected_value=None)`).\n\n### Failures and types\n\n- Missing matches become **`NoMatch`**. Check with `isinstance(x, NoMatch)` and return `CheckResult.error` with a message that names the field/key (structural — the assertion could not be evaluated). The same applies to wrong type for the configured mode, unsupported comparison, and similar preconditions.\n- When the assertion runs and does not hold, return `CheckResult.failure`.\n- Some paths return a **list** (multiple matches or list-producing JSONPath). See `resolve` in `core/extraction.py` if the check must treat collections differently.\n\n### Defaults\n\nUse `Field(default=\"trace.last.outputs\", ...)` (or another sensible default) so typical scenarios need no extra configuration.\n\nMore detail: [reference.md](reference.md).\n\n## Implementation checklist\n\n1. **Kind string** — Pick a unique snake_case discriminator. Search the repo for `@Check.register(\"` to avoid duplicates.\n\n2. **Class** — Subclass `Check[...]` or `BaseLLMCheck[...]` from `..core.check` / `..judges.base`. Use Pydantic `Field` for config. For non-LLM checks, implement `async def run(self, trace: TraceType) -> CheckResult` (use `CheckResult.success` / `CheckResult.failure` / `CheckResult.error`; put extra data in `details=` when useful).\n\n3. **Registration** — Decorate with `@Check.register(\"your_kind\")` on the class definition.\n\n4. **Wire imports so the kind is registered** — Discriminated unions only know subclasses that were imported. Add an import of the new module to the appropriate package `__init__.py` (e.g. `builtin/__init__.py` re-exports; `judges/__init__.py` for judges). Update `src/giskard/checks/__init__.py` exports and `__all__` if the type is public API.\n\n5. **JSONPath fields** — Follow the section above for every `key` / `*_key`; run tests so `test_jsonpath_enforcement` passes.\n\n6. **Tests** — Add `tests/builtin/test_.py` or extend an existing file (e.g. `test_judge.py`). Mirror package layout. Use `pytest` and async tests where `run` is async. Cover pass, fail, and edge cases (missing values, wrong types).\n\n7. **Validate** — From the **repository root** (this monorepo uses the root `Makefile`):\n\n - `make test-unit PACKAGE=giskard-checks` — unit tests for that lib\n - `make check` — lint, format, compat, typecheck, etc.\n\n Use `make test PACKAGE=giskard-checks` if changes must coexist with functional tests.\n\n `make test-unit PACKAGE=giskard-checks` includes `tests/core/test_jsonpath_enforcement.py`.\n\n## Serialization note\n\nAfter `model_dump()`, `model_validate()` needs every custom `Check` subclass already imported. If deserialization tests fail with an unknown kind, ensure the defining module is imported on the test (or via package `__init__`) before validation.\n\n## LLM checks\n\nFor `BaseLLMCheck`, implement `get_prompt` and respect `output_type` / structured parsing as in existing judges. Details: [reference.md](reference.md).\n",".cursor/skills/giskard-checks-new-check/reference.md":"# Reference — check variants in giskard-checks\n\n## JSONPath enforcement and helpers\n\n- **Source**: `libs/giskard-checks/src/giskard/checks/core/extraction.py` — `JSONPathStr`, `resolve`, `provided_or_resolve`, `NoMatch`.\n- **Enforcement**: `libs/giskard-checks/tests/core/test_jsonpath_enforcement.py` — field names `key` or `*_key` must carry the `JSONPathStr` marker (including inside unions with `None` or `MISSING`).\n- **`MISSING` inline-or-path** (`ComparisonCheck`): `expected_value: ExpectedType | MISSING = MISSING` and `expected_value_key: JSONPathStr | MISSING = MISSING`; validate with `is MISSING` checks; in `run`, call `provided_or_resolve(trace, key=self.expected_value_key, value=self.expected_value)` so an inline `expected_value` (including explicit `None`) skips the path.\n- **`MISSING` optional path** (`StringMatching.keyword_key`): `JSONPathStr | MISSING = MISSING` with a validator that requires exactly one of inline value or key—do not use a bare `str` annotation for any `*_key` field.\n- **Multi-match**: `resolve` returns a list when the expression yields multiple matches or the parsed path is list-like (`Slice`, `Union`, etc.); single-match paths return one value or `NoMatch`.\n\n## Non-LLM `Check`\n\n- See `libs/giskard-checks/src/giskard/checks/builtin/` for patterns: comparison checks, `FnCheck`, `StringMatching` / `RegexMatching`, `SemanticSimilarity`.\n- Subclass `Check[InputType, OutputType, TraceType]` when generics matter; many builtins use `Trace[dict[str, str], dict[str, str]]` or similar.\n- Prefer returning `CheckResult` with clear `message` and structured `details` for debugging.\n\n## `BaseLLMCheck` (judges)\n\n- Defined in `libs/giskard-checks/src/giskard/checks/judges/base.py`.\n- Implement `get_prompt()` returning `str`, `Message`, `MessageTemplate`, or `TemplateReference`.\n- Default structured output model is `LLMCheckResult` (`passed`, `reason`); override `output_type` if using a different Pydantic model for the LLM response.\n- Uses `WithGeneratorMixin` — generator defaults come from library settings (`set_default_generator` / `get_default_generator` in public API).\n- Examples: `Groundedness`, `Conformity`, `LLMJudge` in the same `judges/` package.\n\n## Export surfaces\n\nTypical touch points for a **public** check:\n\n1. `builtin/__init__.py` or `judges/__init__.py` — import and re-export; update `__all__`.\n2. `src/giskard/checks/__init__.py` — import from submodule and add to `__all__` alongside other checks.\n\nInternal-only types are rare; if you add one, skip top-level exports but still register the kind and import the module where deserialization must work.\n",".cursor/skills/giskard-oss-pr-ready/SKILL.md":"---\nname: giskard-oss-pr-ready\ndescription: Describes the merge-ready workflow for the giskard-oss monorepo using the repository root Makefile and the same steps as GitHub Actions—format, check, unit tests, optional pre-commit. Use when preparing or updating a pull request, confirming CI will pass, running pre-push validation, or when the user asks for a branch to be merge-ready.\n---\n\n# giskard-oss — PR ready\n\n## Context\n\nWork from the **repository root** (`giskard-oss/`). The monorepo `Makefile` lives there.\n\nMain CI (`.github/workflows/ci.yml`) runs: `make install install-tools`, then `make check`, then `make test-unit PACKAGE=` on Python 3.12–3.14.\n\n## Merge-ready checklist\n\nRun in order:\n\n1. **`make format`** — Ruff format + fix where applicable.\n\n2. **`make check`** — Lint, format check, Python 3.12 compat (vermin), basedpyright, pip-audit, license check. This matches the `lint` job in CI (after `make install install-tools`).\n\n3. **`make test-unit`** — All packages, or narrow scope:\n - `make test-unit PACKAGE=giskard-checks` (same pattern for `giskard-core`, `giskard-agents`)\n\n4. **Optional — pre-commit** — If hooks are installed: `pre-commit run`. First-time setup: `make pre-commit-install` (part of `make setup`).\n\n## Dependencies\n\nIf checks fail from missing tools, run **`make install`** and **`make install-tools`** (CI runs both before `make check`).\n\n## Functional tests\n\n`make test-functional` (and full `make test`) hit live APIs and are not the default PR gate in `ci.yml` (they run in `.github/workflows/integration-tests.yml` with repo secrets). Run them only when the change requires `@pytest.mark.functional` coverage.\n\n### `.env` setup (local)\n\n1. At the **repository root**, create a `.env` file (already listed in `.gitignore`—do not commit secrets).\n2. Define credentials for the models your tests use. The integration workflow sets:\n - `GEMINI_API_KEY` — required for the default Gemini routes used by functional tests.\n - `TEST_MODEL` — optional; default `gemini/gemini-3.5-flash` (see `libs/giskard-agents/tests/conftest.py`).\n - `TEST_EMBEDDING_MODEL` — optional; default `gemini/gemini-embedding-001`.\n3. The repo does not auto-load `.env` in pytest—**export variables into the shell** before running tests, for example:\n\n ```bash\n set -a && source .env && set +a\n make test-functional PACKAGE=giskard-agents\n ```\n\n Use valid shell assignments in `.env` (e.g. `GEMINI_API_KEY=...`). If you point `TEST_MODEL` / `TEST_EMBEDDING_MODEL` at another provider, add whatever API keys that provider expects (LiteLLM / the underlying SDK).\n\n## Verification gate\n\nBefore declaring a branch merge-ready, include the following evidence in your response:\n\n1. Last line(s) of `make check` output — must show no errors.\n2. Pytest summary from `make test-unit` — e.g. `52 passed, 0 failed`.\n3. If any step was skipped or failed, state it explicitly.\n\n\"It should pass\" is not evidence. Show actual output.\n\n## Library-specific rules\n\nFor package conventions (API patterns, tests layout), read `libs//.cursor/rules/`. Makefile targets are always invoked from the repository root as listed above.\n","AGENTS.md":"# AGENTS.md\n\ngiskard-oss — behavioral config for interactive coding assistants with a human in the loop.\n\n## Workflow Orchestration\n\n### 1. Plan Mode Default\n– Enter plan mode for ANY non-trivial task (3+ steps or touches multiple libs)\n– Brainstorm first (design → spec), then write an implementation plan\n– Save specs and plans in the repo's established planning docs. Prefer `docs/specs/` and `docs/plans/`; if the repo already uses another planning directory, follow that convention.\n– If something goes sideways, STOP and re-plan immediately\n\n### 2. Subagent Strategy\n– Use subagents or equivalent isolated work sessions when the assistant supports them\n– Prefer a fresh subagent/session per task; controller curates exactly the context each one needs\n– Two-stage review after each task: spec compliance first, then code quality\n\n### 3. Self-Improvement Loop\n– After ANY correction: self-document the rule (naming, file headers, examples); only add to AGENTS.md if it cannot be self-documented\n– When a correction changes what was implemented vs. what the plan says: update the plan file immediately so reviewers never see a mismatch\n\n### 4. Verification Before Done\n– Never mark a task complete without proving it works\n– Run: `make format && make check && make test-unit PACKAGE=`\n– Show actual output. Never assume tests pass.\n– Ask yourself: \"Would a staff engineer approve this?\"\n\n### 5. Demand Elegance\n– Pause and ask \"is there a more elegant solution?\"\n– Skip for simple fixes — don't over-engineer\n\n### 6. Autonomous Bug Fixing\n– When given a bug report with clear scope: just fix it\n– No `# type: ignore`, no patched test assertions — fix the root cause\n\n### 7. Commit Hygiene\n– Use Conventional Commits for every commit message (for example, `fix(llm): update google interactions translator`)\n\n## Task Management\n1. Brainstorm → Spec → Plan — save to the repo's established planning docs, preferably `docs/specs/` and `docs/plans/`\n2. Verify Plan — check in before starting\n3. Track Progress — mark items complete as you go\n4. Explain Changes — high-level summary at each step\n5. Document Results — add review section to the plan file\n6. Capture Lessons — update AGENTS.md directly after corrections\n\n## Core Principles\n– Simplicity First: make every change as simple as possible; prefer deleting lines over adding them\n– No Laziness: find root causes; no band-aids, no temporary fixes; senior developer standards\n– Minimal Impact: only touch what's necessary; no side effects; no reformatting untouched lines\n\n## Cursor Cloud specific instructions\n\nNon-obvious context for cloud agents (startup refresh is `make install` then `make install-tools`; `uv` is on `PATH` for login shells via `~/.bashrc`).\n\n– This repo is a `uv` workspace monorepo of pure Python libraries (`libs/giskard-*`). There is no server, daemon, or CLI to start — \"running\" the product means importing the libraries from Python or running `pytest`. Prefer Makefile targets over raw `uv`/`pytest` (`make help`).\n– Import namespace is `giskard.` (e.g. `import giskard.checks`), not `giskard_checks`.\n– Do not run full `make setup` in this VM: its `pre-commit-install` step fails with `Cowardly refusing to install hooks with core.hooksPath set`. That is expected (global `core.hooksPath`) and harmless — use `make install` + `make install-tools` instead; git hooks are not needed to validate work.\n– Lint/test/build/run all work without secrets: `make lint`, `make test-unit [PACKAGE=]`, and importing the libs. Unit tests exclude anything marked `functional`.\n– Functional/integration tests are excluded by default and require live LLM provider API keys (e.g. `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`); `giskard-scan` may also reach Hugging Face Hub. Do not expect these to run without those secrets and network access.\n– `make check` runs `security` (`pip-audit`) and license checks (`check-licenses`/`check-notices`) that reach the network (PyPI / license metadata); if offline these steps can fail even when the code is fine.\n","libs/giskard-agents/.cursor/rules/generator-adapter.mdc":"---\ndescription: Generator as protocol adapter — architectural rules for provider translation in giskard-agents.\nalwaysApply: true\n---\n\n# Generator as Protocol Adapter\n\n## Core Principles\n\n1. **Message is the canonical internal format.** `role=\"tool\"`, `tool_call_id`, `tool_calls` are the project's standard fields. Never reshape `Message` to accommodate a specific provider.\n\n2. **Tool.run() returns `str`.** The tool serializes its own output. The workflow wraps the string into a `Message`. The workflow never calls `json.dumps` on tool results.\n\n3. **The generator owns all provider translation.** Each generator subclass is responsible for converting between internal `Message` objects and whatever wire format the provider expects. Serialization is internal to the subclass — `BaseGenerator` does not impose any wire format.\n\n## Rules\n\n- **Do not call provider APIs (litellm, anthropic, openai, etc.) from workflow, tool, or chat code.** Provider calls belong exclusively in generator subclasses.\n- **Do not import provider libraries outside of generator modules.** The `generators/` package is the only place that should depend on provider SDKs.\n- **`_complete` is the template method.** It merges params via `GenerationParams.merge()` and delegates to the abstract `_call_model`. Do not override `_complete` in production subclasses — override `_call_model` instead.\n- **`_call_model` receives internal types and returns `Response`.** Signature: `_call_model(messages: list[Message], params: GenerationParams, metadata: dict[str, Any] | None = None) -> Response`. The base class never touches wire formats. The `metadata` parameter carries caller-provided context (e.g. trace IDs, tags) and the returned `Response.metadata` carries provider-side information back to the caller.\n- **Serialization is the subclass's concern.** Each subclass handles translating `Message`/`Tool` to its provider's wire format inside `_call_model` (e.g. `GiskardLLMGenerator._serialize_messages`). Do not add serialization methods to `BaseGenerator`.\n- **Param merging lives on `GenerationParams.merge()`.** Scalar fields from overrides replace the base; tools are concatenated. Do not reimplement merge logic elsewhere.\n- **Adding a new provider?** Subclass `BaseGenerator` and implement `_call_model`. Handle all serialization/deserialization inside the subclass. Do not modify `Message`, `Tool`, or workflow code.\n- **Cross-cutting concerns use middleware.** Retries, rate limiting, logging, and similar concerns are implemented as `CompletionMiddleware` subclasses, not generator mixins. See `generators/middleware.py`.\n","libs/giskard-checks/.cursor/rules/02-project.mdc":"---\ndescription: High level overview of the project. Always read this before starting a new task.\nalwaysApply: false\n---\n# Project Rules and Conventions\n\n## Project Overview\n\nThis is the `giskard-checks` library - lightweight primitives to define and run checks against model interactions.\n\n## Key Files\n\n- Monorepo root `Makefile` — development workflow (run `make help` from repository root)\n- `pyproject.toml` — build configuration, dependencies, and tooling\n- `README.md` — end-user documentation and quickstart\n\n## Development Workflow\n\nAlways use Makefile commands for development tasks. See [03-development.mdc](03-development.mdc) for the complete workflow.\n\n## Code Quality\n\n- Python >= 3.12 required\n- Use Pydantic v2 models for type safety\n- Async-friendly: checks can be sync or async\n- Follow the existing patterns in the codebase\n- Use absolute imports within the `giskard.checks` package\n\n## Testing\n\n- From the repository root: `make test-unit PACKAGE=giskard-checks` for unit tests (or `make test` for all tests including functional)\n- Tests are in `tests/` mirroring the package structure\n- Static checks aligned with CI: `make check` (after `make install install-tools` if needed)\n","libs/giskard-checks/.cursor/rules/03-development.mdc":"---\ndescription: All devs and agent must read this before creating a PR\nalwaysApply: false\n---\n# Development Workflow\n\nThe monorepo **repository root** defines the `Makefile`. Run all targets from that directory (`giskard-oss/`). Prefer `make` over ad hoc `uv` or tool invocations.\n\n## Quick Start\n\n```bash\nmake setup # Complete development setup (install deps + tools)\nmake format && make check && make test-unit PACKAGE=giskard-checks\n```\n\n## Common Commands\n\n- `make help` - Show all available commands with descriptions\n- `make setup` - Complete development setup (install deps + tools)\n- `make install` - Install project dependencies\n- `make install-tools` - Install development tools (ruff, vermin, pre-commit)\n- `make test` - Run all tests (unit + functional) for all libs or `PACKAGE=`\n- `make test-unit` - Unit tests only; use `PACKAGE=giskard-checks` to scope this library\n- `make lint` - Run linting checks\n- `make format` - Format code with ruff\n- `make check-format` - Check if code is formatted correctly\n- `make check-compat` - Check Python 3.12 compatibility\n- `make check` - Run all static checks (lint, format, compat, typecheck, security, licenses)—matches the CI `lint` job (with tools installed)\n- `make clean` - Clean up build artifacts and caches\n\n## Pre-commit\n\n- `make pre-commit-install` - Setup pre-commit hooks\n- `make pre-commit-run` - Run pre-commit on all files\n\n## Guidelines\n\n1. Always use `make` commands instead of direct `uv` or tool invocations\n2. Before pushing, run `make format`, `make check`, and `make test-unit PACKAGE=giskard-checks` (or omit `PACKAGE` to run unit tests for every lib)—this aligns with `.github/workflows/ci.yml`\n3. Use `make setup` for initial project setup\n4. Use `make clean` to clean up build artifacts when needed\n\n## See Also\n\n- [Makefile](../../../../Makefile) at repository root — full command definitions and help\n- [README.md](../../README.md) — package overview and quickstart\n","libs/giskard-llm/.cursor/rules/01-architecture.mdc":"---\ndescription: Architecture and design principles for giskard-llm. Always read before modifying providers, routing, or error handling.\nalwaysApply: true\n---\n\n# giskard-llm Architecture\n\n## Overview\n\n`giskard-llm` is a lightweight routing layer that dispatches `\"provider/model\"` strings to native LLM SDKs. It replaces litellm with direct SDK calls while presenting a unified response shape to consumers.\n\n## Architecture\n\nThe public API (`acompletion`, `aembedding`, `aresponse`) routes `\"provider/model\"` strings through a lazy-loading registry to provider implementations. Providers are self-contained modules under `providers/`, each conforming to one or more `Protocol` classes (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`) and responsible for: role mapping, message validation, tool conversion, error mapping, and response normalization.\n\nShared contracts live at the package root: Pydantic v2 response types (OpenAI-shaped), a unified error hierarchy with `status_code` for consumer retry logic, and a `should_retry` helper.\n\n## Type Conventions\n\nInput types (messages, tool definitions, tool results) use `TypedDict` — lightweight, constructed with dict literals. Output types (responses, tool calls, embeddings) use Pydantic `_BaseModel` — attribute access, `.model_dump()`, validation on construction. See `docs/design.md` for rationale.\n\n## Core Principles\n\n1. **Lazy imports.** Importing `giskard.llm` must never require any provider SDK. SDKs are imported inside provider modules only.\n2. **Strict message validation.** Providers validate messages before calling the SDK and raise `BadRequestError` with a clear message. Invalid input must not silently pass through to opaque SDK errors. Opt-in relaxation (e.g., `merge_system=True`) is explicit.\n3. **Unified error boundary.** Raw SDK exceptions never escape a provider. Every provider maps its SDK errors to the `errors.py` hierarchy.\n4. **Response normalization.** All providers convert native responses to the shared `types.py` models. Consumers never see provider-specific shapes.\n5. **Provider config via env vars or `**params`.** The public API has no provider-specific kwargs. Configuration flows through environment variables or pass-through params.\n6. **Provider behavior is self-documented.** Each provider class must have a comprehensive docstring covering: env vars, role mapping, error mapping, supported features, and provider-specific kwargs. This is the source of truth for provider behavior.\n\n## Adding a New Provider\n\n1. Create `providers/.py` implementing the relevant protocols (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`).\n2. Guard SDK imports with `try/except ImportError` and raise `ProviderNotAvailableError`.\n3. Register it in the provider registry in `routing.py`.\n4. Add the SDK as an optional dependency extra in `pyproject.toml`.\n5. Implement role mapping, message validation, tool conversion, error mapping, and response conversion.\n6. Write the class docstring following the provider documentation template (see `04-provider-docs.mdc`).\n7. Add unit tests (mocked SDK) and functional test scenarios.\n","libs/giskard-llm/.cursor/rules/02-testing.mdc":"---\ndescription: Testing conventions for giskard-llm. Read before writing or modifying tests.\nglobs: \"**/test*.py,**/conftest.py\"\n---\n\n# giskard-llm Testing Conventions\n\n## Test Structure\n\n- **Unit tests** (`tests/`): Mocked SDK calls, no API keys needed. Cover routing, message conversion, error mapping, response conversion, validation.\n- **Functional tests** (`tests/functional/`): Real API calls. Cover end-to-end scenarios across all providers.\n\n## Provider Marks and Auto-Skip\n\nEvery functional test is marked with its provider (`@pytest.mark.google`, `@pytest.mark.openai`, etc.). Auto-skip logic in `conftest.py` skips tests whose provider SDK is not installed, so unit test runs never fail due to missing optional dependencies.\n\n## Scenario Design\n\n- **Assert on structure, not content.** Tests must pass with even the weakest model. Assert non-empty responses, correct roles, correct types, parseable JSON — never assert on specific wording.\n- **Meaningful inputs and outputs.** Test scenarios should exercise real behavior: system prompts that produce verifiable effects, tool calls with checkable arguments, structured output that validates against a schema. The goal is a test that fails for the right reasons.\n","libs/giskard-llm/.cursor/rules/03-development.mdc":"---\ndescription: Development workflow for giskard-llm. Read before creating a PR.\nalwaysApply: false\n---\n\n# giskard-llm Development Workflow\n\n## Dependencies\n\n- **Core**: `pydantic>=2.0` only. No other runtime dependencies.\n- **Provider SDKs**: Optional extras in `pyproject.toml` (`openai`, `google`, `anthropic`, `azure`). Never add a provider SDK to core.\n- **Package manager**: `uv`.\n\n## Format and Lint\n\n`ruff format` and `ruff check` via pre-commit hooks. If `basedpyright` fails on provider files due to uninstalled SDKs, add `# pyright: reportMissingImports=false` at the top of the provider file.\n\n## CI\n\n- **Unit tests**: run on every PR, no SDK required — all provider interactions are mocked.\n- **Functional tests**: run per-provider in a matrix. Each matrix entry installs only its SDK and injects only its env vars at the step level.\n- **`workflow_dispatch`**: allows manual triggering with org-membership check.\n\n## Commit Messages\n\nSemantic format scoped to the lib: `feat(giskard-llm): add azure provider`, `fix(giskard-llm): google system message extraction`.\n","libs/giskard-llm/.cursor/rules/04-provider-docs.mdc":"---\ndescription: Provider documentation standards. Read when creating or modifying a provider.\nglobs: \"**/providers/*.py\"\n---\n\n# Provider Documentation\n\nEach provider class must have a comprehensive docstring that serves as the single source of truth for that provider's behavior. This replaces external documentation that would go stale.\n\n## Required Docstring Sections\n\nEvery provider class docstring must cover:\n\n1. **Overview** — SDK used, what model prefix routes here (e.g., `\"google/\"`, `\"azure/\"`).\n2. **Authentication** — Required env vars (e.g., `GOOGLE_API_KEY`) and alternative kwargs.\n3. **Role mapping** — How canonical roles (`system`, `user`, `assistant`, `tool`) map to the SDK format.\n4. **Message constraints** — Provider-specific validation rules (alternation, system message handling, etc.).\n5. **Tool call format** — How tool definitions and tool results are converted.\n6. **Error mapping** — Which SDK exceptions map to which `LLMError` subclasses.\n7. **Supported features** — Completion, embeddings, structured output, and any limitations.\n8. **Provider-specific kwargs** — Any pass-through params unique to this provider (e.g., `merge_system` for Anthropic).\n\n## Example\n\n```python\nclass GoogleProvider:\n \"\"\"Google Gemini provider using the ``google-genai`` SDK.\n\n Routing prefix: ``google/``\n\n Authentication:\n - Env: ``GOOGLE_API_KEY`` (or ``GEMINI_API_KEY``)\n - Kwargs: ``api_key``\n\n Role mapping:\n - ``system`` -> extracted to ``system_instruction`` config (accepts a list)\n - ``assistant`` -> ``model``\n - ``tool`` -> ``function_response`` part\n - ``user`` -> ``user``\n\n Message constraints:\n - Multiple system messages: supported natively (passed as list)\n - System-only messages: raises ``BadRequestError``\n - No strict alternation required\n\n Tool call format:\n - Tool definitions: converted to ``FunctionDeclaration``\n - Tool results: converted to ``function_response`` parts\n - Tool call IDs: synthetic (``call_``) since Gemini doesn't provide them\n\n Error mapping:\n - ``google.genai.errors.ClientError`` (400) -> ``BadRequestError``\n - ``google.genai.errors.ClientError`` (401/403) -> ``AuthenticationError``\n - ``google.genai.errors.ClientError`` (429) -> ``RateLimitError``\n - ``google.genai.errors.ServerError`` -> ``ServerError``\n\n Supported features:\n - Completion: yes\n - Embeddings: yes\n - Structured output (response_format): yes, via ``response_schema``\n\n Provider-specific kwargs:\n - ``safety_settings``: override default safety settings\n \"\"\"\n```\n\n## Rules\n\n- **The docstring is the spec.** When behavior changes, update the docstring in the same commit.\n- **No external provider docs.** Don't maintain separate markdown files per provider. The class docstring is authoritative.\n- **README references providers briefly.** The top-level README has a summary table; details link to the source.\n"},"files":{".claude/CLAUDE.md":"@AGENTS.md\n",".claude/skills/check/SKILL.md":"---\nname: check\ndescription: Run the giskard-oss verification gate (format, check, unit tests) for a package\ndisable-model-invocation: true\n---\n\n# check\n\nRun the giskard-oss verification gate that `AGENTS.md` mandates before any task is \"done\":\n\n```\nmake format && make check && make test-unit PACKAGE=\n```\n\n## Usage\n\n`/check ` — run the full gate scoped to one lib's unit tests.\n\n`/check` — run the gate; unit tests cover all libs (slower).\n\n`` is a directory under `libs/`: `giskard-agents`, `giskard-checks`, `giskard-core`, `giskard-llm`, `giskard-scan`.\n\n## What to run\n\nWith a package:\n\n```bash\nmake format && make check && make test-unit PACKAGE=\n```\n\nWithout a package:\n\n```bash\nmake format && make check && make test-unit\n```\n\n`format` and `check` are repo-wide and take no `PACKAGE`. Only `test-unit` is scoped.\n\n## Notes\n\n- Run the three steps as one `&&` chain so a failure stops the gate early.\n- Report failures with the command's output — do not claim the gate passed unless every step exited 0.\n- `make format` mutates files (ruff format + check --fix); show the resulting diff if anything changed.\n",".claude/skills/fix-cve/SKILL.md":"---\nname: fix-cve\ndescription: Use when pip-audit (or a CVE/GHSA advisory, Dependabot, or security scan) flags a vulnerable Python dependency in giskard-oss and it needs upgrading to a fixed version in the uv lockfile.\n---\n\n# fix-cve\n\nRemediate a vulnerable dependency flagged by `pip-audit` (or any CVE/GHSA advisory) by upgrading it in the **uv lockfile**, then verify with the repo gate.\n\n`pip-audit` runs as part of `make check` (the `security` target). The advisory looks like:\n\n```\nName Version ID Fix Versions\n------- ------- ------------------- ------------\nmsgpack 1.1.2 GHSA-6v7p-g79w-8964 1.2.1\n```\n\nYou need: the **package name** and a **fix version**.\n\n## Procedure\n\n```bash\n# 1. Upgrade the package in the lockfile (works for direct AND transitive deps).\nuv lock --upgrade-package \n\n# 2. Apply to the environment.\nuv sync\n\n# 3. Verify the advisory is gone — re-run the exact audit command.\nuv run pip-audit --skip-editable\n```\n\nThen run the full repo gate (it re-runs `pip-audit` via the `security` target, plus format/lint/types/tests):\n\n```bash\nmake format && make check && make test-unit\n```\n\nOr invoke `/check` (the project verification-gate skill).\n\n## Key facts\n\n- **Always upgrade in the lockfile, not the environment.** Use `uv lock --upgrade-package `, NOT `pip install` / `uv pip install`. The lockfile is the source of truth; a `pip install` fix is lost on the next `uv sync`.\n- **Transitive deps are fixed the same way.** Most flagged packages (e.g. `msgpack`, pulled in by `cachecontrol`) are not in any `pyproject.toml`. You still upgrade them by name with `uv lock --upgrade-package ` — no `pyproject.toml` edit needed. Do NOT add a direct dependency just to pin it.\n- **`uv lock --upgrade-package ` takes the newest version the constraints allow.** Check the resulting `uv.lock` entry to confirm the version actually moved to the fix version — a cap may have held it below.\n\n### When a constraint caps the version below the fix\n\nIf another dependency caps the package below the fix version, the bare upgrade silently stays on the old version, and forcing it with `uv lock --upgrade-package \">=\"` **fails with a resolution error** (`No solution found...`) — uv constraints can only *narrow* the version set, never expand past a cap. Escalate in this order:\n\n1. **Upgrade the capping parent.** Find who imposes the cap (`uv tree --invert --package `), then `uv lock --upgrade-package `. A newer parent often relaxes the cap. Preferred — keeps the dependency graph honest.\n2. **Override the transitive dep** if no parent upgrade resolves it. Add to `pyproject.toml`:\n ```toml\n [tool.uv]\n override-dependencies = [\">=\"]\n ```\n Overrides *expand* the allowed set (the only escape hatch a cap can't block), then re-run `uv lock`. Use sparingly — it ignores the parent's declared constraint, so re-verify nothing breaks with the full gate.\n- **Lockfile-only changes have no \"affected lib\".** Run the full `make test-unit` (all packages), not a single `PACKAGE=` scope — the dep could affect any lib.\n\n## Verifying the fix\n\nDo not assert \"fixed\" — show evidence:\n\n1. `uv run pip-audit --skip-editable` prints **\"No known vulnerabilities found\"** (editable `giskard-*` libs in the skip list are expected and harmless).\n2. The `uv.lock` diff shows the package moved to the fix version (`Updated vX -> vY` from `uv lock`).\n3. `make check` passes (its `security` target re-runs the audit).\n\n## Common mistakes\n\n| Mistake | Fix |\n|---------|-----|\n| `pip install ==` to fix it | Use `uv lock --upgrade-package `; pip changes don't persist in the lockfile |\n| Editing `pyproject.toml` for a transitive dep | Not needed — `uv lock --upgrade-package` handles transitive deps by name |\n| Skipping re-audit, assuming the upgrade worked | Re-run `uv run pip-audit --skip-editable`; a constraint may have silently capped the version below the fix |\n| Force-pinning past a cap with `\">=\"` and hitting `No solution found` | Constraints can't expand past a cap — upgrade the capping parent, or use `[tool.uv] override-dependencies` |\n| Guessing the verification command | Run `/check` or `make format && make check && make test-unit`; `make check` includes `pip-audit` |\n| Scoping tests to one `PACKAGE=` | A lockfile change has no single affected lib; run all unit tests |\n",".claude/skills/giskard-oss-quality-gate/SKILL.md":"---\nname: giskard-oss-quality-gate\ndescription: Run the full code quality gate for the giskard-oss monorepo—format, lint, typecheck, and unit tests across all packages. Use when finishing a feature, before opening a PR, or when the user asks to check quality or run the gate.\ndisable-model-invocation: true\nallowed-tools: Bash(make *)\n---\n\n# giskard-oss — quality gate\n\n## Results\n\n```!\necho \"=== make format ===\" && make format 2>&1 | tail -5\necho \"=== make check ===\" && make check 2>&1 | tail -10\necho \"=== make test-unit PACKAGE=giskard-core ===\" && make test-unit PACKAGE=giskard-core 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-checks ===\" && make test-unit PACKAGE=giskard-checks 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-agents ===\" && make test-unit PACKAGE=giskard-agents 2>&1 | tail -5\n```\n\nReport pass/fail for each step. Fix root causes on failure — no `# type: ignore`, no skips. Re-run after fixing.\n",".claude/skills/setup-env/SKILL.md":"---\ndescription: Set up the full workspace dev environment. Use when the user wants to install dependencies, sync libs, or prepare the environment.\ndisable-model-invocation: true\nallowed-tools: Bash(make *) Bash(uv *)\n---\n\n## Setup dev env\n\n!`make setup 2>&1 | tail -20`\n\n## Validate installation and notify user\n\nCheck the output above for errors. If any errors occurred, try to fix them and re-run the failed step. Report whether installation was successful.\n",".cursor/rules/documentation.mdc":"---\ndescription: Documentation and commenting guidelines for an OSS, beginner-friendly codebase.\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Documentation Guidelines\n\nThis is an open-source project. Code should be understandable by newcomers who may not have prior context on the architecture.\n\n## Public API Docstrings\n\nEvery public class, method, and function **must** have a NumPy-style docstring that includes:\n- A one-line summary of what it does\n- Parameters with types and descriptions\n- Return value description\n- A short usage example when the behavior is non-obvious\n\n```python\nasync def run(self, max_steps: int | None = None) -> Chat:\n \"\"\"Execute the workflow and return the resulting chat.\n\n Parameters\n ----------\n max_steps : int, optional\n Maximum number of steps to run. If not provided, the workflow\n runs until completion.\n\n Returns\n -------\n Chat\n The completed chat with all messages and tool call results.\n\n Example\n -------\n >>> chat = await generator.chat(\"Hello!\").run()\n >>> print(chat.last.content)\n \"\"\"\n```\n\n## Inline Comments\n\n- **Do** comment non-obvious control flow, especially async orchestration, retry logic, and tool dispatch.\n- **Do** explain *why* something is done when the reason isn't obvious from the code.\n- **Don't** narrate what the code does line-by-line (\"increment counter\", \"return result\").\n- **Don't** leave TODO comments without a linked issue or clear owner.\n\n```python\n# Good — explains the why\n# Re-parse tool_calls from the raw response because LiteLLM may\n# return partial function arguments when streaming.\ntool_calls = _reparse_tool_calls(raw)\n\n# Bad — restates the code\n# Parse the tool calls\ntool_calls = _reparse_tool_calls(raw)\n```\n\n## Module-Level Docstrings\n\nEach module (`.py` file) should have a brief docstring explaining its role in the system. This helps newcomers navigate the codebase.\n\n## README and Examples\n\n- Keep `README.md` examples accurate and runnable.\n- When adding a new feature, add a corresponding example to the README.\n- Use progressive disclosure: simple usage first, advanced patterns later.\n",".cursor/rules/guidelines.mdc":"---\ndescription: Team-wide guardrails for AI-assisted development.\nalwaysApply: true\n---\n# Guidelines\n\n## No Inventions\n\nOnly make changes that are explicitly requested. Do not add unrequested features, refactors, or improvements.\n\n## Preserve Existing Code\n\nDo not remove unrelated code, functionalities, or comments without explicit reason. Pay attention to preserving existing structures.\n\n## No CODEMAP\n\nNever generate architecture maps, codemap files, or similar high-maintenance documentation that duplicates what is already in cursor rules, docstrings, and README. These files go stale fast and create maintenance burden.\n\n## Root Causes, No Temporary Fixes\n\nTrace failures to their source before making a change. Do not:\n- Add `# type: ignore` / `# pyright: ignore` without first confirming no structural fix is possible.\n- Modify test assertions to force a pass — if a test reveals a real bug, fix the bug.\n- Add `try/except` to suppress exceptions that should be fixed.\n\nIf the root cause is in a different library than where the symptom appears, say so before proposing a fix.\n\n## Minimal Scope\n\nOnly touch files directly required by the task. Do not reformat untouched lines, rename unrelated variables, or add type annotations to code you didn't otherwise change. Propose nearby improvements separately and wait for approval.\n",".cursor/rules/python.mdc":"---\ndescription: Python conventions for the giskard workspace (3.12+, Pydantic v2, async, testing).\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Python Conventions\n\n## Python Version\n\n- Python >= 3.12 required across all projects.\n- **Never use `from __future__ import annotations`** — it breaks Pydantic v2 model resolution at runtime and is unnecessary with 3.12+.\n\n## Modern Python 3.12+ Features\n\nPrefer modern syntax over legacy `typing` equivalents.\n\n### PEP 695 — Type Parameter Syntax\n\n```python\n# Good — PEP 695 inline type parameters\nclass Box[T]:\n def __init__(self, value: T): ...\n\ndef first[T](items: list[T]) -> T: ...\n\n# Bad — legacy TypeVar / Generic\nfrom typing import TypeVar, Generic\nT = TypeVar('T')\nclass Box(Generic[T]):\n def __init__(self, value: T): ...\n```\n\n### PEP 695 — Bounded and Constrained Generics\n\n```python\n# Good\nclass NumberBox[T: (int, float)]: ...\nclass Comparable[T: int]: ...\n\n# Bad\nT = TypeVar('T', int, float)\nT = TypeVar('T', bound=int)\n```\n\n### PEP 695 — `type` Statement for Type Aliases\n\n```python\n# Good — unambiguous, supports forward references and recursive types\ntype Vector = list[float]\ntype Tree[T] = T | list[Tree[T]]\n\n# Bad — legacy TypeAlias\nfrom typing import TypeAlias\nVector: TypeAlias = list[float]\n```\n\n### PEP 695 — ParamSpec in Class Syntax\n\nNiche — document with a docstring when used.\n\n```python\n# Good\nclass Wrapper[**P, R]:\n \"\"\"Generic wrapper preserving the signature of the wrapped callable.\n\n Type Parameters\n ---------------\n **P\n Parameter specification of the wrapped callable.\n R\n Return type of the wrapped callable.\n \"\"\"\n ...\n\n# Bad\nfrom typing import ParamSpec\nP = ParamSpec('P')\n```\n\n### Union Syntax (PEP 604)\n\n```python\n# Good\ndef process(value: str | int) -> str | None: ...\n\n# Bad\nfrom typing import Optional, Union\ndef process(value: Union[str, int]) -> Optional[str]: ...\n```\n\n### Built-in Generics\n\nUse built-in types directly — do not import from `typing`.\n\n```python\n# Good\ndef transform(items: list[str]) -> dict[str, int]: ...\ncoords: tuple[float, float, float]\n\n# Bad\nfrom typing import List, Dict, Tuple\ndef transform(items: List[str]) -> Dict[str, int]: ...\n```\n\n### `@override` Decorator\n\nOptional but recommended for methods overriding a parent, especially in large class hierarchies. Catches silent breakage when the parent method is renamed.\n\n```python\nfrom typing import override\n\nclass Child(Base):\n @override\n def process(self) -> None: ...\n```\n\n## Pydantic v2\n\n- Use `BaseModel` for structured data that needs serialization.\n- Prefer immutable models with `ConfigDict(frozen=True)`.\n- Discriminated unions via `@discriminated_base` / `@Cls.register(\"kind\")` from `giskard-core`.\n- Validate external inputs at boundaries; keep internal code operating on typed objects.\n\n## Imports and Naming\n\n- Absolute imports within each `giskard.*` package.\n- Import order enforced by Ruff (`isort`).\n- PEP 8 naming: `snake_case` functions/variables, `PascalCase` classes, `UPPER_CASE` constants.\n\n## Async Patterns\n\n- All workflow execution is async. Never use blocking calls inside async code.\n- Use `asyncio.gather` for parallel execution.\n- Checks and generators may be sync or async — design APIs to support both.\n\n## Error Handling\n\n- Prefer result objects (e.g. `CheckResult`) over exceptions for control flow.\n- Raise exceptions only for truly exceptional states (invalid config, programmer errors).\n- Let runners catch and wrap exceptions when appropriate.\n\n## Testing\n\n- pytest with `asyncio_mode = \"auto\"` — no `@pytest.mark.asyncio` needed.\n- Parametrize tests with descriptive IDs.\n- `@pytest.mark.functional` for tests hitting external APIs.\n- Co-locate tests under `tests/` mirroring package structure.\n- Test success, failure, error, and edge-case paths.\n",".cursor/skills/giskard-checks-new-check/SKILL.md":"---\nname: giskard-checks-new-check\ndescription: Guides implementation of new or changed eval checks in the giskard-checks package—Check subclasses, discriminated kind registration, public exports, and tests. Use when adding or modifying built-in checks, LLM-based judges, @Check.register kinds, serialization of TestCase or Scenario, or when editing files under libs/giskard-checks/src/giskard/checks related to checks.\n---\n\n# giskard-checks — new check\n\n## Scope\n\nLibrary code lives under `libs/giskard-checks/`. Follow existing patterns; use absolute imports inside `giskard.checks`. Python 3.12+, Pydantic v2.\n\nRead project rules when relevant: `libs/giskard-checks/.cursor/rules/02-project.mdc`, `03-development.mdc`.\n\n## Classify the check\n\n| Kind | Module | Base class |\n|------|--------|------------|\n| Deterministic (string, numeric, custom logic on trace) | `src/giskard/checks/builtin/` | `Check` (often shared helpers in existing builtins) |\n| Uses LLM to score or classify | `src/giskard/checks/judges/` | `BaseLLMCheck` |\n\nFor one-off or experimental logic, users can use `from_fn` / `FnCheck` without a new class—only add a new type when it is a reusable, serializable primitive.\n\n## JSONPath fields (`key` and `*_key`)\n\n### Naming and typing (required)\n\nAny `Check` field named `key` or whose name ends with `_key` is a JSONPath into the trace. It **must** be annotated with `JSONPathStr`, or `JSONPathStr | None`, or `JSONPathStr | MISSING` (import `MISSING` from `pydantic.experimental.missing_sentinel`). Otherwise `tests/core/test_jsonpath_enforcement.py` fails.\n\nInside the library, import from `..core.extraction` (module `giskard.checks.core.extraction`).\n\n### Path syntax\n\n- Every path must start with `trace.` (validated at model construction via `jsonpath_ng`).\n- Resolution runs against `{\"trace\": trace.model_dump()}`. Prefer `trace.last.outputs`, `trace.last.inputs`, `trace.last.metadata.*` over brittle indices; `trace.last` is the usual default for “current turn” data.\n\n### Reading values\n\n- **`resolve(trace, key)`** — Use when the value always comes from the trace (e.g. the subject `target_key`).\n- **`provided_or_resolve(trace, key=..., value=...)`** — Use when the user may pass an inline value **or** fall back to a path (e.g. `keyword` + `keyword_key`). If `value is not MISSING`, that wins; otherwise the path is evaluated.\n\n### Naming JSONPath fields\n\n- The field naming the **value under test** is always `target_key` (matches the Giskard Hub). It is the only accepted name for that selector and serializes as `target_key`.\n- Every **other** JSONPath field is `{static}_key`, named after the sibling static field holding the literal it would otherwise resolve (`keyword`/`keyword_key`, `context`/`context_key`).\n- Both rules are enforced by `tests/core/test_key_naming_convention.py`.\n\nOptional inline-or-path fields: type as `T | MISSING = MISSING` (and `JSONPathStr | MISSING = MISSING` for optional `*_key` fields). Pass fields directly to `provided_or_resolve`; validate combinations with a `@model_validator` if only one of (inline, `*_key`) may be set (see `StringMatching` / `ComparisonCheck`). Do not use `None` to mean \"extract from trace\"—omit the field so it stays `MISSING`. Explicit `None` is only valid when comparing against or supplying `None` as a real value (e.g. `Equals(expected_value=None)`).\n\n### Failures and types\n\n- Missing matches become **`NoMatch`**. Check with `isinstance(x, NoMatch)` and return `CheckResult.error` with a message that names the field/key (structural — the assertion could not be evaluated). The same applies to wrong type for the configured mode, unsupported comparison, and similar preconditions.\n- When the assertion runs and does not hold, return `CheckResult.failure`.\n- Some paths return a **list** (multiple matches or list-producing JSONPath). See `resolve` in `core/extraction.py` if the check must treat collections differently.\n\n### Defaults\n\nUse `Field(default=\"trace.last.outputs\", ...)` (or another sensible default) so typical scenarios need no extra configuration.\n\nMore detail: [reference.md](reference.md).\n\n## Implementation checklist\n\n1. **Kind string** — Pick a unique snake_case discriminator. Search the repo for `@Check.register(\"` to avoid duplicates.\n\n2. **Class** — Subclass `Check[...]` or `BaseLLMCheck[...]` from `..core.check` / `..judges.base`. Use Pydantic `Field` for config. For non-LLM checks, implement `async def run(self, trace: TraceType) -> CheckResult` (use `CheckResult.success` / `CheckResult.failure` / `CheckResult.error`; put extra data in `details=` when useful).\n\n3. **Registration** — Decorate with `@Check.register(\"your_kind\")` on the class definition.\n\n4. **Wire imports so the kind is registered** — Discriminated unions only know subclasses that were imported. Add an import of the new module to the appropriate package `__init__.py` (e.g. `builtin/__init__.py` re-exports; `judges/__init__.py` for judges). Update `src/giskard/checks/__init__.py` exports and `__all__` if the type is public API.\n\n5. **JSONPath fields** — Follow the section above for every `key` / `*_key`; run tests so `test_jsonpath_enforcement` passes.\n\n6. **Tests** — Add `tests/builtin/test_.py` or extend an existing file (e.g. `test_judge.py`). Mirror package layout. Use `pytest` and async tests where `run` is async. Cover pass, fail, and edge cases (missing values, wrong types).\n\n7. **Validate** — From the **repository root** (this monorepo uses the root `Makefile`):\n\n - `make test-unit PACKAGE=giskard-checks` — unit tests for that lib\n - `make check` — lint, format, compat, typecheck, etc.\n\n Use `make test PACKAGE=giskard-checks` if changes must coexist with functional tests.\n\n `make test-unit PACKAGE=giskard-checks` includes `tests/core/test_jsonpath_enforcement.py`.\n\n## Serialization note\n\nAfter `model_dump()`, `model_validate()` needs every custom `Check` subclass already imported. If deserialization tests fail with an unknown kind, ensure the defining module is imported on the test (or via package `__init__`) before validation.\n\n## LLM checks\n\nFor `BaseLLMCheck`, implement `get_prompt` and respect `output_type` / structured parsing as in existing judges. Details: [reference.md](reference.md).\n",".cursor/skills/giskard-checks-new-check/reference.md":"# Reference — check variants in giskard-checks\n\n## JSONPath enforcement and helpers\n\n- **Source**: `libs/giskard-checks/src/giskard/checks/core/extraction.py` — `JSONPathStr`, `resolve`, `provided_or_resolve`, `NoMatch`.\n- **Enforcement**: `libs/giskard-checks/tests/core/test_jsonpath_enforcement.py` — field names `key` or `*_key` must carry the `JSONPathStr` marker (including inside unions with `None` or `MISSING`).\n- **`MISSING` inline-or-path** (`ComparisonCheck`): `expected_value: ExpectedType | MISSING = MISSING` and `expected_value_key: JSONPathStr | MISSING = MISSING`; validate with `is MISSING` checks; in `run`, call `provided_or_resolve(trace, key=self.expected_value_key, value=self.expected_value)` so an inline `expected_value` (including explicit `None`) skips the path.\n- **`MISSING` optional path** (`StringMatching.keyword_key`): `JSONPathStr | MISSING = MISSING` with a validator that requires exactly one of inline value or key—do not use a bare `str` annotation for any `*_key` field.\n- **Multi-match**: `resolve` returns a list when the expression yields multiple matches or the parsed path is list-like (`Slice`, `Union`, etc.); single-match paths return one value or `NoMatch`.\n\n## Non-LLM `Check`\n\n- See `libs/giskard-checks/src/giskard/checks/builtin/` for patterns: comparison checks, `FnCheck`, `StringMatching` / `RegexMatching`, `SemanticSimilarity`.\n- Subclass `Check[InputType, OutputType, TraceType]` when generics matter; many builtins use `Trace[dict[str, str], dict[str, str]]` or similar.\n- Prefer returning `CheckResult` with clear `message` and structured `details` for debugging.\n\n## `BaseLLMCheck` (judges)\n\n- Defined in `libs/giskard-checks/src/giskard/checks/judges/base.py`.\n- Implement `get_prompt()` returning `str`, `Message`, `MessageTemplate`, or `TemplateReference`.\n- Default structured output model is `LLMCheckResult` (`passed`, `reason`); override `output_type` if using a different Pydantic model for the LLM response.\n- Uses `WithGeneratorMixin` — generator defaults come from library settings (`set_default_generator` / `get_default_generator` in public API).\n- Examples: `Groundedness`, `Conformity`, `LLMJudge` in the same `judges/` package.\n\n## Export surfaces\n\nTypical touch points for a **public** check:\n\n1. `builtin/__init__.py` or `judges/__init__.py` — import and re-export; update `__all__`.\n2. `src/giskard/checks/__init__.py` — import from submodule and add to `__all__` alongside other checks.\n\nInternal-only types are rare; if you add one, skip top-level exports but still register the kind and import the module where deserialization must work.\n",".cursor/skills/giskard-oss-pr-ready/SKILL.md":"---\nname: giskard-oss-pr-ready\ndescription: Describes the merge-ready workflow for the giskard-oss monorepo using the repository root Makefile and the same steps as GitHub Actions—format, check, unit tests, optional pre-commit. Use when preparing or updating a pull request, confirming CI will pass, running pre-push validation, or when the user asks for a branch to be merge-ready.\n---\n\n# giskard-oss — PR ready\n\n## Context\n\nWork from the **repository root** (`giskard-oss/`). The monorepo `Makefile` lives there.\n\nMain CI (`.github/workflows/ci.yml`) runs: `make install install-tools`, then `make check`, then `make test-unit PACKAGE=` on Python 3.12–3.14.\n\n## Merge-ready checklist\n\nRun in order:\n\n1. **`make format`** — Ruff format + fix where applicable.\n\n2. **`make check`** — Lint, format check, Python 3.12 compat (vermin), basedpyright, pip-audit, license check. This matches the `lint` job in CI (after `make install install-tools`).\n\n3. **`make test-unit`** — All packages, or narrow scope:\n - `make test-unit PACKAGE=giskard-checks` (same pattern for `giskard-core`, `giskard-agents`)\n\n4. **Optional — pre-commit** — If hooks are installed: `pre-commit run`. First-time setup: `make pre-commit-install` (part of `make setup`).\n\n## Dependencies\n\nIf checks fail from missing tools, run **`make install`** and **`make install-tools`** (CI runs both before `make check`).\n\n## Functional tests\n\n`make test-functional` (and full `make test`) hit live APIs and are not the default PR gate in `ci.yml` (they run in `.github/workflows/integration-tests.yml` with repo secrets). Run them only when the change requires `@pytest.mark.functional` coverage.\n\n### `.env` setup (local)\n\n1. At the **repository root**, create a `.env` file (already listed in `.gitignore`—do not commit secrets).\n2. Define credentials for the models your tests use. The integration workflow sets:\n - `GEMINI_API_KEY` — required for the default Gemini routes used by functional tests.\n - `TEST_MODEL` — optional; default `gemini/gemini-3.5-flash` (see `libs/giskard-agents/tests/conftest.py`).\n - `TEST_EMBEDDING_MODEL` — optional; default `gemini/gemini-embedding-001`.\n3. The repo does not auto-load `.env` in pytest—**export variables into the shell** before running tests, for example:\n\n ```bash\n set -a && source .env && set +a\n make test-functional PACKAGE=giskard-agents\n ```\n\n Use valid shell assignments in `.env` (e.g. `GEMINI_API_KEY=...`). If you point `TEST_MODEL` / `TEST_EMBEDDING_MODEL` at another provider, add whatever API keys that provider expects (LiteLLM / the underlying SDK).\n\n## Verification gate\n\nBefore declaring a branch merge-ready, include the following evidence in your response:\n\n1. Last line(s) of `make check` output — must show no errors.\n2. Pytest summary from `make test-unit` — e.g. `52 passed, 0 failed`.\n3. If any step was skipped or failed, state it explicitly.\n\n\"It should pass\" is not evidence. Show actual output.\n\n## Library-specific rules\n\nFor package conventions (API patterns, tests layout), read `libs//.cursor/rules/`. Makefile targets are always invoked from the repository root as listed above.\n","AGENTS.md":"# AGENTS.md\n\ngiskard-oss — behavioral config for interactive coding assistants with a human in the loop.\n\n## Workflow Orchestration\n\n### 1. Plan Mode Default\n– Enter plan mode for ANY non-trivial task (3+ steps or touches multiple libs)\n– Brainstorm first (design → spec), then write an implementation plan\n– Save specs and plans in the repo's established planning docs. Prefer `docs/specs/` and `docs/plans/`; if the repo already uses another planning directory, follow that convention.\n– If something goes sideways, STOP and re-plan immediately\n\n### 2. Subagent Strategy\n– Use subagents or equivalent isolated work sessions when the assistant supports them\n– Prefer a fresh subagent/session per task; controller curates exactly the context each one needs\n– Two-stage review after each task: spec compliance first, then code quality\n\n### 3. Self-Improvement Loop\n– After ANY correction: self-document the rule (naming, file headers, examples); only add to AGENTS.md if it cannot be self-documented\n– When a correction changes what was implemented vs. what the plan says: update the plan file immediately so reviewers never see a mismatch\n\n### 4. Verification Before Done\n– Never mark a task complete without proving it works\n– Run: `make format && make check && make test-unit PACKAGE=`\n– Show actual output. Never assume tests pass.\n– Ask yourself: \"Would a staff engineer approve this?\"\n\n### 5. Demand Elegance\n– Pause and ask \"is there a more elegant solution?\"\n– Skip for simple fixes — don't over-engineer\n\n### 6. Autonomous Bug Fixing\n– When given a bug report with clear scope: just fix it\n– No `# type: ignore`, no patched test assertions — fix the root cause\n\n### 7. Commit Hygiene\n– Use Conventional Commits for every commit message (for example, `fix(llm): update google interactions translator`)\n\n## Task Management\n1. Brainstorm → Spec → Plan — save to the repo's established planning docs, preferably `docs/specs/` and `docs/plans/`\n2. Verify Plan — check in before starting\n3. Track Progress — mark items complete as you go\n4. Explain Changes — high-level summary at each step\n5. Document Results — add review section to the plan file\n6. Capture Lessons — update AGENTS.md directly after corrections\n\n## Core Principles\n– Simplicity First: make every change as simple as possible; prefer deleting lines over adding them\n– No Laziness: find root causes; no band-aids, no temporary fixes; senior developer standards\n– Minimal Impact: only touch what's necessary; no side effects; no reformatting untouched lines\n\n## Cursor Cloud specific instructions\n\nNon-obvious context for cloud agents (startup refresh is `make install` then `make install-tools`; `uv` is on `PATH` for login shells via `~/.bashrc`).\n\n– This repo is a `uv` workspace monorepo of pure Python libraries (`libs/giskard-*`). There is no server, daemon, or CLI to start — \"running\" the product means importing the libraries from Python or running `pytest`. Prefer Makefile targets over raw `uv`/`pytest` (`make help`).\n– Import namespace is `giskard.` (e.g. `import giskard.checks`), not `giskard_checks`.\n– Do not run full `make setup` in this VM: its `pre-commit-install` step fails with `Cowardly refusing to install hooks with core.hooksPath set`. That is expected (global `core.hooksPath`) and harmless — use `make install` + `make install-tools` instead; git hooks are not needed to validate work.\n– Lint/test/build/run all work without secrets: `make lint`, `make test-unit [PACKAGE=]`, and importing the libs. Unit tests exclude anything marked `functional`.\n– Functional/integration tests are excluded by default and require live LLM provider API keys (e.g. `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`); `giskard-scan` may also reach Hugging Face Hub. Do not expect these to run without those secrets and network access.\n– `make check` runs `security` (`pip-audit`) and license checks (`check-licenses`/`check-notices`) that reach the network (PyPI / license metadata); if offline these steps can fail even when the code is fine.\n","libs/giskard-agents/.cursor/rules/generator-adapter.mdc":"---\ndescription: Generator as protocol adapter — architectural rules for provider translation in giskard-agents.\nalwaysApply: true\n---\n\n# Generator as Protocol Adapter\n\n## Core Principles\n\n1. **Message is the canonical internal format.** `role=\"tool\"`, `tool_call_id`, `tool_calls` are the project's standard fields. Never reshape `Message` to accommodate a specific provider.\n\n2. **Tool.run() returns `str`.** The tool serializes its own output. The workflow wraps the string into a `Message`. The workflow never calls `json.dumps` on tool results.\n\n3. **The generator owns all provider translation.** Each generator subclass is responsible for converting between internal `Message` objects and whatever wire format the provider expects. Serialization is internal to the subclass — `BaseGenerator` does not impose any wire format.\n\n## Rules\n\n- **Do not call provider APIs (litellm, anthropic, openai, etc.) from workflow, tool, or chat code.** Provider calls belong exclusively in generator subclasses.\n- **Do not import provider libraries outside of generator modules.** The `generators/` package is the only place that should depend on provider SDKs.\n- **`_complete` is the template method.** It merges params via `GenerationParams.merge()` and delegates to the abstract `_call_model`. Do not override `_complete` in production subclasses — override `_call_model` instead.\n- **`_call_model` receives internal types and returns `Response`.** Signature: `_call_model(messages: list[Message], params: GenerationParams, metadata: dict[str, Any] | None = None) -> Response`. The base class never touches wire formats. The `metadata` parameter carries caller-provided context (e.g. trace IDs, tags) and the returned `Response.metadata` carries provider-side information back to the caller.\n- **Serialization is the subclass's concern.** Each subclass handles translating `Message`/`Tool` to its provider's wire format inside `_call_model` (e.g. `GiskardLLMGenerator._serialize_messages`). Do not add serialization methods to `BaseGenerator`.\n- **Param merging lives on `GenerationParams.merge()`.** Scalar fields from overrides replace the base; tools are concatenated. Do not reimplement merge logic elsewhere.\n- **Adding a new provider?** Subclass `BaseGenerator` and implement `_call_model`. Handle all serialization/deserialization inside the subclass. Do not modify `Message`, `Tool`, or workflow code.\n- **Cross-cutting concerns use middleware.** Retries, rate limiting, logging, and similar concerns are implemented as `CompletionMiddleware` subclasses, not generator mixins. See `generators/middleware.py`.\n","libs/giskard-checks/.cursor/rules/02-project.mdc":"---\ndescription: High level overview of the project. Always read this before starting a new task.\nalwaysApply: false\n---\n# Project Rules and Conventions\n\n## Project Overview\n\nThis is the `giskard-checks` library - lightweight primitives to define and run checks against model interactions.\n\n## Key Files\n\n- Monorepo root `Makefile` — development workflow (run `make help` from repository root)\n- `pyproject.toml` — build configuration, dependencies, and tooling\n- `README.md` — end-user documentation and quickstart\n\n## Development Workflow\n\nAlways use Makefile commands for development tasks. See [03-development.mdc](03-development.mdc) for the complete workflow.\n\n## Code Quality\n\n- Python >= 3.12 required\n- Use Pydantic v2 models for type safety\n- Async-friendly: checks can be sync or async\n- Follow the existing patterns in the codebase\n- Use absolute imports within the `giskard.checks` package\n\n## Testing\n\n- From the repository root: `make test-unit PACKAGE=giskard-checks` for unit tests (or `make test` for all tests including functional)\n- Tests are in `tests/` mirroring the package structure\n- Static checks aligned with CI: `make check` (after `make install install-tools` if needed)\n","libs/giskard-checks/.cursor/rules/03-development.mdc":"---\ndescription: All devs and agent must read this before creating a PR\nalwaysApply: false\n---\n# Development Workflow\n\nThe monorepo **repository root** defines the `Makefile`. Run all targets from that directory (`giskard-oss/`). Prefer `make` over ad hoc `uv` or tool invocations.\n\n## Quick Start\n\n```bash\nmake setup # Complete development setup (install deps + tools)\nmake format && make check && make test-unit PACKAGE=giskard-checks\n```\n\n## Common Commands\n\n- `make help` - Show all available commands with descriptions\n- `make setup` - Complete development setup (install deps + tools)\n- `make install` - Install project dependencies\n- `make install-tools` - Install development tools (ruff, vermin, pre-commit)\n- `make test` - Run all tests (unit + functional) for all libs or `PACKAGE=`\n- `make test-unit` - Unit tests only; use `PACKAGE=giskard-checks` to scope this library\n- `make lint` - Run linting checks\n- `make format` - Format code with ruff\n- `make check-format` - Check if code is formatted correctly\n- `make check-compat` - Check Python 3.12 compatibility\n- `make check` - Run all static checks (lint, format, compat, typecheck, security, licenses)—matches the CI `lint` job (with tools installed)\n- `make clean` - Clean up build artifacts and caches\n\n## Pre-commit\n\n- `make pre-commit-install` - Setup pre-commit hooks\n- `make pre-commit-run` - Run pre-commit on all files\n\n## Guidelines\n\n1. Always use `make` commands instead of direct `uv` or tool invocations\n2. Before pushing, run `make format`, `make check`, and `make test-unit PACKAGE=giskard-checks` (or omit `PACKAGE` to run unit tests for every lib)—this aligns with `.github/workflows/ci.yml`\n3. Use `make setup` for initial project setup\n4. Use `make clean` to clean up build artifacts when needed\n\n## See Also\n\n- [Makefile](../../../../Makefile) at repository root — full command definitions and help\n- [README.md](../../README.md) — package overview and quickstart\n","libs/giskard-llm/.cursor/rules/01-architecture.mdc":"---\ndescription: Architecture and design principles for giskard-llm. Always read before modifying providers, routing, or error handling.\nalwaysApply: true\n---\n\n# giskard-llm Architecture\n\n## Overview\n\n`giskard-llm` is a lightweight routing layer that dispatches `\"provider/model\"` strings to native LLM SDKs. It replaces litellm with direct SDK calls while presenting a unified response shape to consumers.\n\n## Architecture\n\nThe public API (`acompletion`, `aembedding`, `aresponse`) routes `\"provider/model\"` strings through a lazy-loading registry to provider implementations. Providers are self-contained modules under `providers/`, each conforming to one or more `Protocol` classes (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`) and responsible for: role mapping, message validation, tool conversion, error mapping, and response normalization.\n\nShared contracts live at the package root: Pydantic v2 response types (OpenAI-shaped), a unified error hierarchy with `status_code` for consumer retry logic, and a `should_retry` helper.\n\n## Type Conventions\n\nInput types (messages, tool definitions, tool results) use `TypedDict` — lightweight, constructed with dict literals. Output types (responses, tool calls, embeddings) use Pydantic `_BaseModel` — attribute access, `.model_dump()`, validation on construction. See `docs/design.md` for rationale.\n\n## Core Principles\n\n1. **Lazy imports.** Importing `giskard.llm` must never require any provider SDK. SDKs are imported inside provider modules only.\n2. **Strict message validation.** Providers validate messages before calling the SDK and raise `BadRequestError` with a clear message. Invalid input must not silently pass through to opaque SDK errors. Opt-in relaxation (e.g., `merge_system=True`) is explicit.\n3. **Unified error boundary.** Raw SDK exceptions never escape a provider. Every provider maps its SDK errors to the `errors.py` hierarchy.\n4. **Response normalization.** All providers convert native responses to the shared `types.py` models. Consumers never see provider-specific shapes.\n5. **Provider config via env vars or `**params`.** The public API has no provider-specific kwargs. Configuration flows through environment variables or pass-through params.\n6. **Provider behavior is self-documented.** Each provider class must have a comprehensive docstring covering: env vars, role mapping, error mapping, supported features, and provider-specific kwargs. This is the source of truth for provider behavior.\n\n## Adding a New Provider\n\n1. Create `providers/.py` implementing the relevant protocols (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`).\n2. Guard SDK imports with `try/except ImportError` and raise `ProviderNotAvailableError`.\n3. Register it in the provider registry in `routing.py`.\n4. Add the SDK as an optional dependency extra in `pyproject.toml`.\n5. Implement role mapping, message validation, tool conversion, error mapping, and response conversion.\n6. Write the class docstring following the provider documentation template (see `04-provider-docs.mdc`).\n7. Add unit tests (mocked SDK) and functional test scenarios.\n","libs/giskard-llm/.cursor/rules/02-testing.mdc":"---\ndescription: Testing conventions for giskard-llm. Read before writing or modifying tests.\nglobs: \"**/test*.py,**/conftest.py\"\n---\n\n# giskard-llm Testing Conventions\n\n## Test Structure\n\n- **Unit tests** (`tests/`): Mocked SDK calls, no API keys needed. Cover routing, message conversion, error mapping, response conversion, validation.\n- **Functional tests** (`tests/functional/`): Real API calls. Cover end-to-end scenarios across all providers.\n\n## Provider Marks and Auto-Skip\n\nEvery functional test is marked with its provider (`@pytest.mark.google`, `@pytest.mark.openai`, etc.). Auto-skip logic in `conftest.py` skips tests whose provider SDK is not installed, so unit test runs never fail due to missing optional dependencies.\n\n## Scenario Design\n\n- **Assert on structure, not content.** Tests must pass with even the weakest model. Assert non-empty responses, correct roles, correct types, parseable JSON — never assert on specific wording.\n- **Meaningful inputs and outputs.** Test scenarios should exercise real behavior: system prompts that produce verifiable effects, tool calls with checkable arguments, structured output that validates against a schema. The goal is a test that fails for the right reasons.\n","libs/giskard-llm/.cursor/rules/03-development.mdc":"---\ndescription: Development workflow for giskard-llm. Read before creating a PR.\nalwaysApply: false\n---\n\n# giskard-llm Development Workflow\n\n## Dependencies\n\n- **Core**: `pydantic>=2.0` only. No other runtime dependencies.\n- **Provider SDKs**: Optional extras in `pyproject.toml` (`openai`, `google`, `anthropic`, `azure`). Never add a provider SDK to core.\n- **Package manager**: `uv`.\n\n## Format and Lint\n\n`ruff format` and `ruff check` via pre-commit hooks. If `basedpyright` fails on provider files due to uninstalled SDKs, add `# pyright: reportMissingImports=false` at the top of the provider file.\n\n## CI\n\n- **Unit tests**: run on every PR, no SDK required — all provider interactions are mocked.\n- **Functional tests**: run per-provider in a matrix. Each matrix entry installs only its SDK and injects only its env vars at the step level.\n- **`workflow_dispatch`**: allows manual triggering with org-membership check.\n\n## Commit Messages\n\nSemantic format scoped to the lib: `feat(giskard-llm): add azure provider`, `fix(giskard-llm): google system message extraction`.\n","libs/giskard-llm/.cursor/rules/04-provider-docs.mdc":"---\ndescription: Provider documentation standards. Read when creating or modifying a provider.\nglobs: \"**/providers/*.py\"\n---\n\n# Provider Documentation\n\nEach provider class must have a comprehensive docstring that serves as the single source of truth for that provider's behavior. This replaces external documentation that would go stale.\n\n## Required Docstring Sections\n\nEvery provider class docstring must cover:\n\n1. **Overview** — SDK used, what model prefix routes here (e.g., `\"google/\"`, `\"azure/\"`).\n2. **Authentication** — Required env vars (e.g., `GOOGLE_API_KEY`) and alternative kwargs.\n3. **Role mapping** — How canonical roles (`system`, `user`, `assistant`, `tool`) map to the SDK format.\n4. **Message constraints** — Provider-specific validation rules (alternation, system message handling, etc.).\n5. **Tool call format** — How tool definitions and tool results are converted.\n6. **Error mapping** — Which SDK exceptions map to which `LLMError` subclasses.\n7. **Supported features** — Completion, embeddings, structured output, and any limitations.\n8. **Provider-specific kwargs** — Any pass-through params unique to this provider (e.g., `merge_system` for Anthropic).\n\n## Example\n\n```python\nclass GoogleProvider:\n \"\"\"Google Gemini provider using the ``google-genai`` SDK.\n\n Routing prefix: ``google/``\n\n Authentication:\n - Env: ``GOOGLE_API_KEY`` (or ``GEMINI_API_KEY``)\n - Kwargs: ``api_key``\n\n Role mapping:\n - ``system`` -> extracted to ``system_instruction`` config (accepts a list)\n - ``assistant`` -> ``model``\n - ``tool`` -> ``function_response`` part\n - ``user`` -> ``user``\n\n Message constraints:\n - Multiple system messages: supported natively (passed as list)\n - System-only messages: raises ``BadRequestError``\n - No strict alternation required\n\n Tool call format:\n - Tool definitions: converted to ``FunctionDeclaration``\n - Tool results: converted to ``function_response`` parts\n - Tool call IDs: synthetic (``call_``) since Gemini doesn't provide them\n\n Error mapping:\n - ``google.genai.errors.ClientError`` (400) -> ``BadRequestError``\n - ``google.genai.errors.ClientError`` (401/403) -> ``AuthenticationError``\n - ``google.genai.errors.ClientError`` (429) -> ``RateLimitError``\n - ``google.genai.errors.ServerError`` -> ``ServerError``\n\n Supported features:\n - Completion: yes\n - Embeddings: yes\n - Structured output (response_format): yes, via ``response_schema``\n\n Provider-specific kwargs:\n - ``safety_settings``: override default safety settings\n \"\"\"\n```\n\n## Rules\n\n- **The docstring is the spec.** When behavior changes, update the docstring in the same commit.\n- **No external provider docs.** Don't maintain separate markdown files per provider. The class docstring is authoritative.\n- **README references providers briefly.** The top-level README has a summary table; details link to the source.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\ngiskard-oss — behavioral config for interactive coding assistants with a human in the loop.\n\n## Workflow Orchestration\n\n### 1. Plan Mode Default\n– Enter plan mode for ANY non-trivial task (3+ steps or touches multiple libs)\n– Brainstorm first (design → spec), then write an implementation plan\n– Save specs and plans in the repo's established planning docs. Prefer `docs/specs/` and `docs/plans/`; if the repo already uses another planning directory, follow that convention.\n– If something goes sideways, STOP and re-plan immediately\n\n### 2. Subagent Strategy\n– Use subagents or equivalent isolated work sessions when the assistant supports them\n– Prefer a fresh subagent/session per task; controller curates exactly the context each one needs\n– Two-stage review after each task: spec compliance first, then code quality\n\n### 3. Self-Improvement Loop\n– After ANY correction: self-document the rule (naming, file headers, examples); only add to AGENTS.md if it cannot be self-documented\n– When a correction changes what was implemented vs. what the plan says: update the plan file immediately so reviewers never see a mismatch\n\n### 4. Verification Before Done\n– Never mark a task complete without proving it works\n– Run: `make format && make check && make test-unit PACKAGE=`\n– Show actual output. Never assume tests pass.\n– Ask yourself: \"Would a staff engineer approve this?\"\n\n### 5. Demand Elegance\n– Pause and ask \"is there a more elegant solution?\"\n– Skip for simple fixes — don't over-engineer\n\n### 6. Autonomous Bug Fixing\n– When given a bug report with clear scope: just fix it\n– No `# type: ignore`, no patched test assertions — fix the root cause\n\n### 7. Commit Hygiene\n– Use Conventional Commits for every commit message (for example, `fix(llm): update google interactions translator`)\n\n## Task Management\n1. Brainstorm → Spec → Plan — save to the repo's established planning docs, preferably `docs/specs/` and `docs/plans/`\n2. Verify Plan — check in before starting\n3. Track Progress — mark items complete as you go\n4. Explain Changes — high-level summary at each step\n5. Document Results — add review section to the plan file\n6. Capture Lessons — update AGENTS.md directly after corrections\n\n## Core Principles\n– Simplicity First: make every change as simple as possible; prefer deleting lines over adding them\n– No Laziness: find root causes; no band-aids, no temporary fixes; senior developer standards\n– Minimal Impact: only touch what's necessary; no side effects; no reformatting untouched lines\n\n## Cursor Cloud specific instructions\n\nNon-obvious context for cloud agents (startup refresh is `make install` then `make install-tools`; `uv` is on `PATH` for login shells via `~/.bashrc`).\n\n– This repo is a `uv` workspace monorepo of pure Python libraries (`libs/giskard-*`). There is no server, daemon, or CLI to start — \"running\" the product means importing the libraries from Python or running `pytest`. Prefer Makefile targets over raw `uv`/`pytest` (`make help`).\n– Import namespace is `giskard.` (e.g. `import giskard.checks`), not `giskard_checks`.\n– Do not run full `make setup` in this VM: its `pre-commit-install` step fails with `Cowardly refusing to install hooks with core.hooksPath set`. That is expected (global `core.hooksPath`) and harmless — use `make install` + `make install-tools` instead; git hooks are not needed to validate work.\n– Lint/test/build/run all work without secrets: `make lint`, `make test-unit [PACKAGE=]`, and importing the libs. Unit tests exclude anything marked `functional`.\n– Functional/integration tests are excluded by default and require live LLM provider API keys (e.g. `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`); `giskard-scan` may also reach Hugging Face Hub. Do not expect these to run without those secrets and network access.\n– `make check` runs `security` (`pip-audit`) and license checks (`check-licenses`/`check-notices`) that reach the network (PyPI / license metadata); if offline these steps can fail even when the code is fine.\n","isInternal":false,"tokens":962,"sizeBytes":4139},{"name":"CLAUDE.md","path":".claude/CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.claude/CLAUDE.md","title":".claude - Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"@AGENTS.md\n","isInternal":false,"tokens":5,"sizeBytes":11},{"name":"documentation.mdc","path":".cursor/rules/documentation.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.cursor/rules/documentation.mdc","title":"Cursor Rule: documentation","category":"cursor-rule","format":"mdc","content":"---\ndescription: Documentation and commenting guidelines for an OSS, beginner-friendly codebase.\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Documentation Guidelines\n\nThis is an open-source project. Code should be understandable by newcomers who may not have prior context on the architecture.\n\n## Public API Docstrings\n\nEvery public class, method, and function **must** have a NumPy-style docstring that includes:\n- A one-line summary of what it does\n- Parameters with types and descriptions\n- Return value description\n- A short usage example when the behavior is non-obvious\n\n```python\nasync def run(self, max_steps: int | None = None) -> Chat:\n \"\"\"Execute the workflow and return the resulting chat.\n\n Parameters\n ----------\n max_steps : int, optional\n Maximum number of steps to run. If not provided, the workflow\n runs until completion.\n\n Returns\n -------\n Chat\n The completed chat with all messages and tool call results.\n\n Example\n -------\n >>> chat = await generator.chat(\"Hello!\").run()\n >>> print(chat.last.content)\n \"\"\"\n```\n\n## Inline Comments\n\n- **Do** comment non-obvious control flow, especially async orchestration, retry logic, and tool dispatch.\n- **Do** explain *why* something is done when the reason isn't obvious from the code.\n- **Don't** narrate what the code does line-by-line (\"increment counter\", \"return result\").\n- **Don't** leave TODO comments without a linked issue or clear owner.\n\n```python\n# Good — explains the why\n# Re-parse tool_calls from the raw response because LiteLLM may\n# return partial function arguments when streaming.\ntool_calls = _reparse_tool_calls(raw)\n\n# Bad — restates the code\n# Parse the tool calls\ntool_calls = _reparse_tool_calls(raw)\n```\n\n## Module-Level Docstrings\n\nEach module (`.py` file) should have a brief docstring explaining its role in the system. This helps newcomers navigate the codebase.\n\n## README and Examples\n\n- Keep `README.md` examples accurate and runnable.\n- When adding a new feature, add a corresponding example to the README.\n- Use progressive disclosure: simple usage first, advanced patterns later.\n","frontmatter":{"description":"Documentation and commenting guidelines for an OSS, beginner-friendly codebase.","globs":"**/*.py","alwaysApply":false},"isInternal":false,"tokens":465,"sizeBytes":2140},{"name":"guidelines.mdc","path":".cursor/rules/guidelines.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.cursor/rules/guidelines.mdc","title":"Cursor Rule: guidelines","category":"cursor-rule","format":"mdc","content":"---\ndescription: Team-wide guardrails for AI-assisted development.\nalwaysApply: true\n---\n# Guidelines\n\n## No Inventions\n\nOnly make changes that are explicitly requested. Do not add unrequested features, refactors, or improvements.\n\n## Preserve Existing Code\n\nDo not remove unrelated code, functionalities, or comments without explicit reason. Pay attention to preserving existing structures.\n\n## No CODEMAP\n\nNever generate architecture maps, codemap files, or similar high-maintenance documentation that duplicates what is already in cursor rules, docstrings, and README. These files go stale fast and create maintenance burden.\n\n## Root Causes, No Temporary Fixes\n\nTrace failures to their source before making a change. Do not:\n- Add `# type: ignore` / `# pyright: ignore` without first confirming no structural fix is possible.\n- Modify test assertions to force a pass — if a test reveals a real bug, fix the bug.\n- Add `try/except` to suppress exceptions that should be fixed.\n\nIf the root cause is in a different library than where the symptom appears, say so before proposing a fix.\n\n## Minimal Scope\n\nOnly touch files directly required by the task. Do not reformat untouched lines, rename unrelated variables, or add type annotations to code you didn't otherwise change. Propose nearby improvements separately and wait for approval.\n","frontmatter":{"description":"Team-wide guardrails for AI-assisted development.","alwaysApply":true},"isInternal":false,"tokens":269,"sizeBytes":1341},{"name":"python.mdc","path":".cursor/rules/python.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.cursor/rules/python.mdc","title":"Cursor Rule: python","category":"cursor-rule","format":"mdc","content":"---\ndescription: Python conventions for the giskard workspace (3.12+, Pydantic v2, async, testing).\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Python Conventions\n\n## Python Version\n\n- Python >= 3.12 required across all projects.\n- **Never use `from __future__ import annotations`** — it breaks Pydantic v2 model resolution at runtime and is unnecessary with 3.12+.\n\n## Modern Python 3.12+ Features\n\nPrefer modern syntax over legacy `typing` equivalents.\n\n### PEP 695 — Type Parameter Syntax\n\n```python\n# Good — PEP 695 inline type parameters\nclass Box[T]:\n def __init__(self, value: T): ...\n\ndef first[T](items: list[T]) -> T: ...\n\n# Bad — legacy TypeVar / Generic\nfrom typing import TypeVar, Generic\nT = TypeVar('T')\nclass Box(Generic[T]):\n def __init__(self, value: T): ...\n```\n\n### PEP 695 — Bounded and Constrained Generics\n\n```python\n# Good\nclass NumberBox[T: (int, float)]: ...\nclass Comparable[T: int]: ...\n\n# Bad\nT = TypeVar('T', int, float)\nT = TypeVar('T', bound=int)\n```\n\n### PEP 695 — `type` Statement for Type Aliases\n\n```python\n# Good — unambiguous, supports forward references and recursive types\ntype Vector = list[float]\ntype Tree[T] = T | list[Tree[T]]\n\n# Bad — legacy TypeAlias\nfrom typing import TypeAlias\nVector: TypeAlias = list[float]\n```\n\n### PEP 695 — ParamSpec in Class Syntax\n\nNiche — document with a docstring when used.\n\n```python\n# Good\nclass Wrapper[**P, R]:\n \"\"\"Generic wrapper preserving the signature of the wrapped callable.\n\n Type Parameters\n ---------------\n **P\n Parameter specification of the wrapped callable.\n R\n Return type of the wrapped callable.\n \"\"\"\n ...\n\n# Bad\nfrom typing import ParamSpec\nP = ParamSpec('P')\n```\n\n### Union Syntax (PEP 604)\n\n```python\n# Good\ndef process(value: str | int) -> str | None: ...\n\n# Bad\nfrom typing import Optional, Union\ndef process(value: Union[str, int]) -> Optional[str]: ...\n```\n\n### Built-in Generics\n\nUse built-in types directly — do not import from `typing`.\n\n```python\n# Good\ndef transform(items: list[str]) -> dict[str, int]: ...\ncoords: tuple[float, float, float]\n\n# Bad\nfrom typing import List, Dict, Tuple\ndef transform(items: List[str]) -> Dict[str, int]: ...\n```\n\n### `@override` Decorator\n\nOptional but recommended for methods overriding a parent, especially in large class hierarchies. Catches silent breakage when the parent method is renamed.\n\n```python\nfrom typing import override\n\nclass Child(Base):\n @override\n def process(self) -> None: ...\n```\n\n## Pydantic v2\n\n- Use `BaseModel` for structured data that needs serialization.\n- Prefer immutable models with `ConfigDict(frozen=True)`.\n- Discriminated unions via `@discriminated_base` / `@Cls.register(\"kind\")` from `giskard-core`.\n- Validate external inputs at boundaries; keep internal code operating on typed objects.\n\n## Imports and Naming\n\n- Absolute imports within each `giskard.*` package.\n- Import order enforced by Ruff (`isort`).\n- PEP 8 naming: `snake_case` functions/variables, `PascalCase` classes, `UPPER_CASE` constants.\n\n## Async Patterns\n\n- All workflow execution is async. Never use blocking calls inside async code.\n- Use `asyncio.gather` for parallel execution.\n- Checks and generators may be sync or async — design APIs to support both.\n\n## Error Handling\n\n- Prefer result objects (e.g. `CheckResult`) over exceptions for control flow.\n- Raise exceptions only for truly exceptional states (invalid config, programmer errors).\n- Let runners catch and wrap exceptions when appropriate.\n\n## Testing\n\n- pytest with `asyncio_mode = \"auto\"` — no `@pytest.mark.asyncio` needed.\n- Parametrize tests with descriptive IDs.\n- `@pytest.mark.functional` for tests hitting external APIs.\n- Co-locate tests under `tests/` mirroring package structure.\n- Test success, failure, error, and edge-case paths.\n","frontmatter":{"description":"Python conventions for the giskard workspace (3.12+, Pydantic v2, async, testing).","globs":"**/*.py","alwaysApply":false},"isInternal":false,"tokens":949,"sizeBytes":3828},{"name":"generator-adapter.mdc","path":"libs/giskard-agents/.cursor/rules/generator-adapter.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-agents/.cursor/rules/generator-adapter.mdc","title":"Cursor Rule: generator-adapter","category":"cursor-rule","format":"mdc","content":"---\ndescription: Generator as protocol adapter — architectural rules for provider translation in giskard-agents.\nalwaysApply: true\n---\n\n# Generator as Protocol Adapter\n\n## Core Principles\n\n1. **Message is the canonical internal format.** `role=\"tool\"`, `tool_call_id`, `tool_calls` are the project's standard fields. Never reshape `Message` to accommodate a specific provider.\n\n2. **Tool.run() returns `str`.** The tool serializes its own output. The workflow wraps the string into a `Message`. The workflow never calls `json.dumps` on tool results.\n\n3. **The generator owns all provider translation.** Each generator subclass is responsible for converting between internal `Message` objects and whatever wire format the provider expects. Serialization is internal to the subclass — `BaseGenerator` does not impose any wire format.\n\n## Rules\n\n- **Do not call provider APIs (litellm, anthropic, openai, etc.) from workflow, tool, or chat code.** Provider calls belong exclusively in generator subclasses.\n- **Do not import provider libraries outside of generator modules.** The `generators/` package is the only place that should depend on provider SDKs.\n- **`_complete` is the template method.** It merges params via `GenerationParams.merge()` and delegates to the abstract `_call_model`. Do not override `_complete` in production subclasses — override `_call_model` instead.\n- **`_call_model` receives internal types and returns `Response`.** Signature: `_call_model(messages: list[Message], params: GenerationParams, metadata: dict[str, Any] | None = None) -> Response`. The base class never touches wire formats. The `metadata` parameter carries caller-provided context (e.g. trace IDs, tags) and the returned `Response.metadata` carries provider-side information back to the caller.\n- **Serialization is the subclass's concern.** Each subclass handles translating `Message`/`Tool` to its provider's wire format inside `_call_model` (e.g. `GiskardLLMGenerator._serialize_messages`). Do not add serialization methods to `BaseGenerator`.\n- **Param merging lives on `GenerationParams.merge()`.** Scalar fields from overrides replace the base; tools are concatenated. Do not reimplement merge logic elsewhere.\n- **Adding a new provider?** Subclass `BaseGenerator` and implement `_call_model`. Handle all serialization/deserialization inside the subclass. Do not modify `Message`, `Tool`, or workflow code.\n- **Cross-cutting concerns use middleware.** Retries, rate limiting, logging, and similar concerns are implemented as `CompletionMiddleware` subclasses, not generator mixins. See `generators/middleware.py`.\n","frontmatter":{"description":"Generator as protocol adapter — architectural rules for provider translation in giskard-agents.","alwaysApply":true},"isInternal":false,"tokens":556,"sizeBytes":2618},{"name":"02-project.mdc","path":"libs/giskard-checks/.cursor/rules/02-project.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-checks/.cursor/rules/02-project.mdc","title":"Cursor Rule: 02-project","category":"cursor-rule","format":"mdc","content":"---\ndescription: High level overview of the project. Always read this before starting a new task.\nalwaysApply: false\n---\n# Project Rules and Conventions\n\n## Project Overview\n\nThis is the `giskard-checks` library - lightweight primitives to define and run checks against model interactions.\n\n## Key Files\n\n- Monorepo root `Makefile` — development workflow (run `make help` from repository root)\n- `pyproject.toml` — build configuration, dependencies, and tooling\n- `README.md` — end-user documentation and quickstart\n\n## Development Workflow\n\nAlways use Makefile commands for development tasks. See [03-development.mdc](03-development.mdc) for the complete workflow.\n\n## Code Quality\n\n- Python >= 3.12 required\n- Use Pydantic v2 models for type safety\n- Async-friendly: checks can be sync or async\n- Follow the existing patterns in the codebase\n- Use absolute imports within the `giskard.checks` package\n\n## Testing\n\n- From the repository root: `make test-unit PACKAGE=giskard-checks` for unit tests (or `make test` for all tests including functional)\n- Tests are in `tests/` mirroring the package structure\n- Static checks aligned with CI: `make check` (after `make install install-tools` if needed)\n","frontmatter":{"description":"High level overview of the project. Always read this before starting a new task.","alwaysApply":false},"isInternal":false,"tokens":277,"sizeBytes":1206},{"name":"03-development.mdc","path":"libs/giskard-checks/.cursor/rules/03-development.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-checks/.cursor/rules/03-development.mdc","title":"Cursor Rule: 03-development","category":"cursor-rule","format":"mdc","content":"---\ndescription: All devs and agent must read this before creating a PR\nalwaysApply: false\n---\n# Development Workflow\n\nThe monorepo **repository root** defines the `Makefile`. Run all targets from that directory (`giskard-oss/`). Prefer `make` over ad hoc `uv` or tool invocations.\n\n## Quick Start\n\n```bash\nmake setup # Complete development setup (install deps + tools)\nmake format && make check && make test-unit PACKAGE=giskard-checks\n```\n\n## Common Commands\n\n- `make help` - Show all available commands with descriptions\n- `make setup` - Complete development setup (install deps + tools)\n- `make install` - Install project dependencies\n- `make install-tools` - Install development tools (ruff, vermin, pre-commit)\n- `make test` - Run all tests (unit + functional) for all libs or `PACKAGE=`\n- `make test-unit` - Unit tests only; use `PACKAGE=giskard-checks` to scope this library\n- `make lint` - Run linting checks\n- `make format` - Format code with ruff\n- `make check-format` - Check if code is formatted correctly\n- `make check-compat` - Check Python 3.12 compatibility\n- `make check` - Run all static checks (lint, format, compat, typecheck, security, licenses)—matches the CI `lint` job (with tools installed)\n- `make clean` - Clean up build artifacts and caches\n\n## Pre-commit\n\n- `make pre-commit-install` - Setup pre-commit hooks\n- `make pre-commit-run` - Run pre-commit on all files\n\n## Guidelines\n\n1. Always use `make` commands instead of direct `uv` or tool invocations\n2. Before pushing, run `make format`, `make check`, and `make test-unit PACKAGE=giskard-checks` (or omit `PACKAGE` to run unit tests for every lib)—this aligns with `.github/workflows/ci.yml`\n3. Use `make setup` for initial project setup\n4. Use `make clean` to clean up build artifacts when needed\n\n## See Also\n\n- [Makefile](../../../../Makefile) at repository root — full command definitions and help\n- [README.md](../../README.md) — package overview and quickstart\n","frontmatter":{"description":"All devs and agent must read this before creating a PR","alwaysApply":false},"isInternal":false,"tokens":490,"sizeBytes":1966},{"name":"01-architecture.mdc","path":"libs/giskard-llm/.cursor/rules/01-architecture.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-llm/.cursor/rules/01-architecture.mdc","title":"Cursor Rule: 01-architecture","category":"cursor-rule","format":"mdc","content":"---\ndescription: Architecture and design principles for giskard-llm. Always read before modifying providers, routing, or error handling.\nalwaysApply: true\n---\n\n# giskard-llm Architecture\n\n## Overview\n\n`giskard-llm` is a lightweight routing layer that dispatches `\"provider/model\"` strings to native LLM SDKs. It replaces litellm with direct SDK calls while presenting a unified response shape to consumers.\n\n## Architecture\n\nThe public API (`acompletion`, `aembedding`, `aresponse`) routes `\"provider/model\"` strings through a lazy-loading registry to provider implementations. Providers are self-contained modules under `providers/`, each conforming to one or more `Protocol` classes (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`) and responsible for: role mapping, message validation, tool conversion, error mapping, and response normalization.\n\nShared contracts live at the package root: Pydantic v2 response types (OpenAI-shaped), a unified error hierarchy with `status_code` for consumer retry logic, and a `should_retry` helper.\n\n## Type Conventions\n\nInput types (messages, tool definitions, tool results) use `TypedDict` — lightweight, constructed with dict literals. Output types (responses, tool calls, embeddings) use Pydantic `_BaseModel` — attribute access, `.model_dump()`, validation on construction. See `docs/design.md` for rationale.\n\n## Core Principles\n\n1. **Lazy imports.** Importing `giskard.llm` must never require any provider SDK. SDKs are imported inside provider modules only.\n2. **Strict message validation.** Providers validate messages before calling the SDK and raise `BadRequestError` with a clear message. Invalid input must not silently pass through to opaque SDK errors. Opt-in relaxation (e.g., `merge_system=True`) is explicit.\n3. **Unified error boundary.** Raw SDK exceptions never escape a provider. Every provider maps its SDK errors to the `errors.py` hierarchy.\n4. **Response normalization.** All providers convert native responses to the shared `types.py` models. Consumers never see provider-specific shapes.\n5. **Provider config via env vars or `**params`.** The public API has no provider-specific kwargs. Configuration flows through environment variables or pass-through params.\n6. **Provider behavior is self-documented.** Each provider class must have a comprehensive docstring covering: env vars, role mapping, error mapping, supported features, and provider-specific kwargs. This is the source of truth for provider behavior.\n\n## Adding a New Provider\n\n1. Create `providers/.py` implementing the relevant protocols (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`).\n2. Guard SDK imports with `try/except ImportError` and raise `ProviderNotAvailableError`.\n3. Register it in the provider registry in `routing.py`.\n4. Add the SDK as an optional dependency extra in `pyproject.toml`.\n5. Implement role mapping, message validation, tool conversion, error mapping, and response conversion.\n6. Write the class docstring following the provider documentation template (see `04-provider-docs.mdc`).\n7. Add unit tests (mocked SDK) and functional test scenarios.\n","frontmatter":{"description":"Architecture and design principles for giskard-llm. Always read before modifying providers, routing, or error handling.","alwaysApply":true},"isInternal":false,"tokens":659,"sizeBytes":3143},{"name":"02-testing.mdc","path":"libs/giskard-llm/.cursor/rules/02-testing.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-llm/.cursor/rules/02-testing.mdc","title":"Cursor Rule: 02-testing","category":"cursor-rule","format":"mdc","content":"---\ndescription: Testing conventions for giskard-llm. Read before writing or modifying tests.\nglobs: \"**/test*.py,**/conftest.py\"\n---\n\n# giskard-llm Testing Conventions\n\n## Test Structure\n\n- **Unit tests** (`tests/`): Mocked SDK calls, no API keys needed. Cover routing, message conversion, error mapping, response conversion, validation.\n- **Functional tests** (`tests/functional/`): Real API calls. Cover end-to-end scenarios across all providers.\n\n## Provider Marks and Auto-Skip\n\nEvery functional test is marked with its provider (`@pytest.mark.google`, `@pytest.mark.openai`, etc.). Auto-skip logic in `conftest.py` skips tests whose provider SDK is not installed, so unit test runs never fail due to missing optional dependencies.\n\n## Scenario Design\n\n- **Assert on structure, not content.** Tests must pass with even the weakest model. Assert non-empty responses, correct roles, correct types, parseable JSON — never assert on specific wording.\n- **Meaningful inputs and outputs.** Test scenarios should exercise real behavior: system prompts that produce verifiable effects, tool calls with checkable arguments, structured output that validates against a schema. The goal is a test that fails for the right reasons.\n","frontmatter":{"description":"Testing conventions for giskard-llm. Read before writing or modifying tests.","globs":"**/test*.py,**/conftest.py"},"isInternal":false,"tokens":269,"sizeBytes":1226},{"name":"03-development.mdc","path":"libs/giskard-llm/.cursor/rules/03-development.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-llm/.cursor/rules/03-development.mdc","title":"Cursor Rule: 03-development","category":"cursor-rule","format":"mdc","content":"---\ndescription: Development workflow for giskard-llm. Read before creating a PR.\nalwaysApply: false\n---\n\n# giskard-llm Development Workflow\n\n## Dependencies\n\n- **Core**: `pydantic>=2.0` only. No other runtime dependencies.\n- **Provider SDKs**: Optional extras in `pyproject.toml` (`openai`, `google`, `anthropic`, `azure`). Never add a provider SDK to core.\n- **Package manager**: `uv`.\n\n## Format and Lint\n\n`ruff format` and `ruff check` via pre-commit hooks. If `basedpyright` fails on provider files due to uninstalled SDKs, add `# pyright: reportMissingImports=false` at the top of the provider file.\n\n## CI\n\n- **Unit tests**: run on every PR, no SDK required — all provider interactions are mocked.\n- **Functional tests**: run per-provider in a matrix. Each matrix entry installs only its SDK and injects only its env vars at the step level.\n- **`workflow_dispatch`**: allows manual triggering with org-membership check.\n\n## Commit Messages\n\nSemantic format scoped to the lib: `feat(giskard-llm): add azure provider`, `fix(giskard-llm): google system message extraction`.\n","frontmatter":{"description":"Development workflow for giskard-llm. Read before creating a PR.","alwaysApply":false},"isInternal":false,"tokens":269,"sizeBytes":1080},{"name":"04-provider-docs.mdc","path":"libs/giskard-llm/.cursor/rules/04-provider-docs.mdc","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/libs/giskard-llm/.cursor/rules/04-provider-docs.mdc","title":"Cursor Rule: 04-provider-docs","category":"cursor-rule","format":"mdc","content":"---\ndescription: Provider documentation standards. Read when creating or modifying a provider.\nglobs: \"**/providers/*.py\"\n---\n\n# Provider Documentation\n\nEach provider class must have a comprehensive docstring that serves as the single source of truth for that provider's behavior. This replaces external documentation that would go stale.\n\n## Required Docstring Sections\n\nEvery provider class docstring must cover:\n\n1. **Overview** — SDK used, what model prefix routes here (e.g., `\"google/\"`, `\"azure/\"`).\n2. **Authentication** — Required env vars (e.g., `GOOGLE_API_KEY`) and alternative kwargs.\n3. **Role mapping** — How canonical roles (`system`, `user`, `assistant`, `tool`) map to the SDK format.\n4. **Message constraints** — Provider-specific validation rules (alternation, system message handling, etc.).\n5. **Tool call format** — How tool definitions and tool results are converted.\n6. **Error mapping** — Which SDK exceptions map to which `LLMError` subclasses.\n7. **Supported features** — Completion, embeddings, structured output, and any limitations.\n8. **Provider-specific kwargs** — Any pass-through params unique to this provider (e.g., `merge_system` for Anthropic).\n\n## Example\n\n```python\nclass GoogleProvider:\n \"\"\"Google Gemini provider using the ``google-genai`` SDK.\n\n Routing prefix: ``google/``\n\n Authentication:\n - Env: ``GOOGLE_API_KEY`` (or ``GEMINI_API_KEY``)\n - Kwargs: ``api_key``\n\n Role mapping:\n - ``system`` -> extracted to ``system_instruction`` config (accepts a list)\n - ``assistant`` -> ``model``\n - ``tool`` -> ``function_response`` part\n - ``user`` -> ``user``\n\n Message constraints:\n - Multiple system messages: supported natively (passed as list)\n - System-only messages: raises ``BadRequestError``\n - No strict alternation required\n\n Tool call format:\n - Tool definitions: converted to ``FunctionDeclaration``\n - Tool results: converted to ``function_response`` parts\n - Tool call IDs: synthetic (``call_``) since Gemini doesn't provide them\n\n Error mapping:\n - ``google.genai.errors.ClientError`` (400) -> ``BadRequestError``\n - ``google.genai.errors.ClientError`` (401/403) -> ``AuthenticationError``\n - ``google.genai.errors.ClientError`` (429) -> ``RateLimitError``\n - ``google.genai.errors.ServerError`` -> ``ServerError``\n\n Supported features:\n - Completion: yes\n - Embeddings: yes\n - Structured output (response_format): yes, via ``response_schema``\n\n Provider-specific kwargs:\n - ``safety_settings``: override default safety settings\n \"\"\"\n```\n\n## Rules\n\n- **The docstring is the spec.** When behavior changes, update the docstring in the same commit.\n- **No external provider docs.** Don't maintain separate markdown files per provider. The class docstring is authoritative.\n- **README references providers briefly.** The top-level README has a summary table; details link to the source.\n","frontmatter":{"description":"Provider documentation standards. Read when creating or modifying a provider.","globs":"**/providers/*.py"},"isInternal":false,"tokens":671,"sizeBytes":3030},{"name":"SKILL.md","path":".claude/skills/check/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.claude/skills/check/SKILL.md","title":"check","category":"anthropic-skill","format":"markdown","content":"---\nname: check\ndescription: Run the giskard-oss verification gate (format, check, unit tests) for a package\ndisable-model-invocation: true\n---\n\n# check\n\nRun the giskard-oss verification gate that `AGENTS.md` mandates before any task is \"done\":\n\n```\nmake format && make check && make test-unit PACKAGE=\n```\n\n## Usage\n\n`/check ` — run the full gate scoped to one lib's unit tests.\n\n`/check` — run the gate; unit tests cover all libs (slower).\n\n`` is a directory under `libs/`: `giskard-agents`, `giskard-checks`, `giskard-core`, `giskard-llm`, `giskard-scan`.\n\n## What to run\n\nWith a package:\n\n```bash\nmake format && make check && make test-unit PACKAGE=\n```\n\nWithout a package:\n\n```bash\nmake format && make check && make test-unit\n```\n\n`format` and `check` are repo-wide and take no `PACKAGE`. Only `test-unit` is scoped.\n\n## Notes\n\n- Run the three steps as one `&&` chain so a failure stops the gate early.\n- Report failures with the command's output — do not claim the gate passed unless every step exited 0.\n- `make format` mutates files (ruff format + check --fix); show the resulting diff if anything changed.\n","frontmatter":{"name":"check","description":"Run the giskard-oss verification gate (format, check, unit tests) for a package","disable-model-invocation":true,"disableModelInvocation":true},"isInternal":false,"tokens":306,"sizeBytes":1162},{"name":"SKILL.md","path":".claude/skills/fix-cve/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.claude/skills/fix-cve/SKILL.md","title":"fix-cve","category":"anthropic-skill","format":"markdown","content":"---\nname: fix-cve\ndescription: Use when pip-audit (or a CVE/GHSA advisory, Dependabot, or security scan) flags a vulnerable Python dependency in giskard-oss and it needs upgrading to a fixed version in the uv lockfile.\n---\n\n# fix-cve\n\nRemediate a vulnerable dependency flagged by `pip-audit` (or any CVE/GHSA advisory) by upgrading it in the **uv lockfile**, then verify with the repo gate.\n\n`pip-audit` runs as part of `make check` (the `security` target). The advisory looks like:\n\n```\nName Version ID Fix Versions\n------- ------- ------------------- ------------\nmsgpack 1.1.2 GHSA-6v7p-g79w-8964 1.2.1\n```\n\nYou need: the **package name** and a **fix version**.\n\n## Procedure\n\n```bash\n# 1. Upgrade the package in the lockfile (works for direct AND transitive deps).\nuv lock --upgrade-package \n\n# 2. Apply to the environment.\nuv sync\n\n# 3. Verify the advisory is gone — re-run the exact audit command.\nuv run pip-audit --skip-editable\n```\n\nThen run the full repo gate (it re-runs `pip-audit` via the `security` target, plus format/lint/types/tests):\n\n```bash\nmake format && make check && make test-unit\n```\n\nOr invoke `/check` (the project verification-gate skill).\n\n## Key facts\n\n- **Always upgrade in the lockfile, not the environment.** Use `uv lock --upgrade-package `, NOT `pip install` / `uv pip install`. The lockfile is the source of truth; a `pip install` fix is lost on the next `uv sync`.\n- **Transitive deps are fixed the same way.** Most flagged packages (e.g. `msgpack`, pulled in by `cachecontrol`) are not in any `pyproject.toml`. You still upgrade them by name with `uv lock --upgrade-package ` — no `pyproject.toml` edit needed. Do NOT add a direct dependency just to pin it.\n- **`uv lock --upgrade-package ` takes the newest version the constraints allow.** Check the resulting `uv.lock` entry to confirm the version actually moved to the fix version — a cap may have held it below.\n\n### When a constraint caps the version below the fix\n\nIf another dependency caps the package below the fix version, the bare upgrade silently stays on the old version, and forcing it with `uv lock --upgrade-package \">=\"` **fails with a resolution error** (`No solution found...`) — uv constraints can only *narrow* the version set, never expand past a cap. Escalate in this order:\n\n1. **Upgrade the capping parent.** Find who imposes the cap (`uv tree --invert --package `), then `uv lock --upgrade-package `. A newer parent often relaxes the cap. Preferred — keeps the dependency graph honest.\n2. **Override the transitive dep** if no parent upgrade resolves it. Add to `pyproject.toml`:\n ```toml\n [tool.uv]\n override-dependencies = [\">=\"]\n ```\n Overrides *expand* the allowed set (the only escape hatch a cap can't block), then re-run `uv lock`. Use sparingly — it ignores the parent's declared constraint, so re-verify nothing breaks with the full gate.\n- **Lockfile-only changes have no \"affected lib\".** Run the full `make test-unit` (all packages), not a single `PACKAGE=` scope — the dep could affect any lib.\n\n## Verifying the fix\n\nDo not assert \"fixed\" — show evidence:\n\n1. `uv run pip-audit --skip-editable` prints **\"No known vulnerabilities found\"** (editable `giskard-*` libs in the skip list are expected and harmless).\n2. The `uv.lock` diff shows the package moved to the fix version (`Updated vX -> vY` from `uv lock`).\n3. `make check` passes (its `security` target re-runs the audit).\n\n## Common mistakes\n\n| Mistake | Fix |\n|---------|-----|\n| `pip install ==` to fix it | Use `uv lock --upgrade-package `; pip changes don't persist in the lockfile |\n| Editing `pyproject.toml` for a transitive dep | Not needed — `uv lock --upgrade-package` handles transitive deps by name |\n| Skipping re-audit, assuming the upgrade worked | Re-run `uv run pip-audit --skip-editable`; a constraint may have silently capped the version below the fix |\n| Force-pinning past a cap with `\">=\"` and hitting `No solution found` | Constraints can't expand past a cap — upgrade the capping parent, or use `[tool.uv] override-dependencies` |\n| Guessing the verification command | Run `/check` or `make format && make check && make test-unit`; `make check` includes `pip-audit` |\n| Scoping tests to one `PACKAGE=` | A lockfile change has no single affected lib; run all unit tests |\n","frontmatter":{"name":"fix-cve","description":"Use when pip-audit (or a CVE/GHSA advisory, Dependabot, or security scan) flags a vulnerable Python dependency in giskard-oss and it needs upgrading to a fixed version in the uv lockfile."},"isInternal":false,"tokens":1130,"sizeBytes":4465},{"name":"SKILL.md","path":".claude/skills/giskard-oss-quality-gate/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.claude/skills/giskard-oss-quality-gate/SKILL.md","title":"giskard-oss-quality-gate","category":"anthropic-skill","format":"markdown","content":"---\nname: giskard-oss-quality-gate\ndescription: Run the full code quality gate for the giskard-oss monorepo—format, lint, typecheck, and unit tests across all packages. Use when finishing a feature, before opening a PR, or when the user asks to check quality or run the gate.\ndisable-model-invocation: true\nallowed-tools: Bash(make *)\n---\n\n# giskard-oss — quality gate\n\n## Results\n\n```!\necho \"=== make format ===\" && make format 2>&1 | tail -5\necho \"=== make check ===\" && make check 2>&1 | tail -10\necho \"=== make test-unit PACKAGE=giskard-core ===\" && make test-unit PACKAGE=giskard-core 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-checks ===\" && make test-unit PACKAGE=giskard-checks 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-agents ===\" && make test-unit PACKAGE=giskard-agents 2>&1 | tail -5\n```\n\nReport pass/fail for each step. Fix root causes on failure — no `# type: ignore`, no skips. Re-run after fixing.\n","frontmatter":{"name":"giskard-oss-quality-gate","description":"Run the full code quality gate for the giskard-oss monorepo—format, lint, typecheck, and unit tests across all packages. Use when finishing a feature, before opening a PR, or when the user asks to check quality or run the gate.","disable-model-invocation":true,"allowed-tools":"Bash(make *)","allowedTools":["Bash(make","*)"],"disableModelInvocation":true},"allowedTools":["Bash(make","*)"],"isInternal":false,"tokens":264,"sizeBytes":947},{"name":"SKILL.md","path":".claude/skills/setup-env/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.claude/skills/setup-env/SKILL.md","title":"setup-env","category":"anthropic-skill","format":"markdown","content":"---\ndescription: Set up the full workspace dev environment. Use when the user wants to install dependencies, sync libs, or prepare the environment.\ndisable-model-invocation: true\nallowed-tools: Bash(make *) Bash(uv *)\n---\n\n## Setup dev env\n\n!`make setup 2>&1 | tail -20`\n\n## Validate installation and notify user\n\nCheck the output above for errors. If any errors occurred, try to fix them and re-run the failed step. Report whether installation was successful.\n","frontmatter":{"description":"Set up the full workspace dev environment. Use when the user wants to install dependencies, sync libs, or prepare the environment.","disable-model-invocation":true,"allowed-tools":"Bash(make *) Bash(uv *)","allowedTools":["Bash(make","*)","Bash(uv","*)"],"disableModelInvocation":true},"allowedTools":["Bash(make","*)","Bash(uv","*)"],"isInternal":false,"tokens":100,"sizeBytes":461},{"name":"reference.md","path":".cursor/skills/giskard-checks-new-check/reference.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.cursor/skills/giskard-checks-new-check/reference.md","title":"Giskard-checks-new-check Skill","category":"anthropic-skill","format":"markdown","content":"# Reference — check variants in giskard-checks\n\n## JSONPath enforcement and helpers\n\n- **Source**: `libs/giskard-checks/src/giskard/checks/core/extraction.py` — `JSONPathStr`, `resolve`, `provided_or_resolve`, `NoMatch`.\n- **Enforcement**: `libs/giskard-checks/tests/core/test_jsonpath_enforcement.py` — field names `key` or `*_key` must carry the `JSONPathStr` marker (including inside unions with `None` or `MISSING`).\n- **`MISSING` inline-or-path** (`ComparisonCheck`): `expected_value: ExpectedType | MISSING = MISSING` and `expected_value_key: JSONPathStr | MISSING = MISSING`; validate with `is MISSING` checks; in `run`, call `provided_or_resolve(trace, key=self.expected_value_key, value=self.expected_value)` so an inline `expected_value` (including explicit `None`) skips the path.\n- **`MISSING` optional path** (`StringMatching.keyword_key`): `JSONPathStr | MISSING = MISSING` with a validator that requires exactly one of inline value or key—do not use a bare `str` annotation for any `*_key` field.\n- **Multi-match**: `resolve` returns a list when the expression yields multiple matches or the parsed path is list-like (`Slice`, `Union`, etc.); single-match paths return one value or `NoMatch`.\n\n## Non-LLM `Check`\n\n- See `libs/giskard-checks/src/giskard/checks/builtin/` for patterns: comparison checks, `FnCheck`, `StringMatching` / `RegexMatching`, `SemanticSimilarity`.\n- Subclass `Check[InputType, OutputType, TraceType]` when generics matter; many builtins use `Trace[dict[str, str], dict[str, str]]` or similar.\n- Prefer returning `CheckResult` with clear `message` and structured `details` for debugging.\n\n## `BaseLLMCheck` (judges)\n\n- Defined in `libs/giskard-checks/src/giskard/checks/judges/base.py`.\n- Implement `get_prompt()` returning `str`, `Message`, `MessageTemplate`, or `TemplateReference`.\n- Default structured output model is `LLMCheckResult` (`passed`, `reason`); override `output_type` if using a different Pydantic model for the LLM response.\n- Uses `WithGeneratorMixin` — generator defaults come from library settings (`set_default_generator` / `get_default_generator` in public API).\n- Examples: `Groundedness`, `Conformity`, `LLMJudge` in the same `judges/` package.\n\n## Export surfaces\n\nTypical touch points for a **public** check:\n\n1. `builtin/__init__.py` or `judges/__init__.py` — import and re-export; update `__all__`.\n2. `src/giskard/checks/__init__.py` — import from submodule and add to `__all__` alongside other checks.\n\nInternal-only types are rare; if you add one, skip top-level exports but still register the kind and import the module where deserialization must work.\n","isInternal":false,"tokens":667,"sizeBytes":2637},{"name":"SKILL.md","path":".cursor/skills/giskard-checks-new-check/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.cursor/skills/giskard-checks-new-check/SKILL.md","title":"giskard-checks-new-check","category":"anthropic-skill","format":"markdown","content":"---\nname: giskard-checks-new-check\ndescription: Guides implementation of new or changed eval checks in the giskard-checks package—Check subclasses, discriminated kind registration, public exports, and tests. Use when adding or modifying built-in checks, LLM-based judges, @Check.register kinds, serialization of TestCase or Scenario, or when editing files under libs/giskard-checks/src/giskard/checks related to checks.\n---\n\n# giskard-checks — new check\n\n## Scope\n\nLibrary code lives under `libs/giskard-checks/`. Follow existing patterns; use absolute imports inside `giskard.checks`. Python 3.12+, Pydantic v2.\n\nRead project rules when relevant: `libs/giskard-checks/.cursor/rules/02-project.mdc`, `03-development.mdc`.\n\n## Classify the check\n\n| Kind | Module | Base class |\n|------|--------|------------|\n| Deterministic (string, numeric, custom logic on trace) | `src/giskard/checks/builtin/` | `Check` (often shared helpers in existing builtins) |\n| Uses LLM to score or classify | `src/giskard/checks/judges/` | `BaseLLMCheck` |\n\nFor one-off or experimental logic, users can use `from_fn` / `FnCheck` without a new class—only add a new type when it is a reusable, serializable primitive.\n\n## JSONPath fields (`key` and `*_key`)\n\n### Naming and typing (required)\n\nAny `Check` field named `key` or whose name ends with `_key` is a JSONPath into the trace. It **must** be annotated with `JSONPathStr`, or `JSONPathStr | None`, or `JSONPathStr | MISSING` (import `MISSING` from `pydantic.experimental.missing_sentinel`). Otherwise `tests/core/test_jsonpath_enforcement.py` fails.\n\nInside the library, import from `..core.extraction` (module `giskard.checks.core.extraction`).\n\n### Path syntax\n\n- Every path must start with `trace.` (validated at model construction via `jsonpath_ng`).\n- Resolution runs against `{\"trace\": trace.model_dump()}`. Prefer `trace.last.outputs`, `trace.last.inputs`, `trace.last.metadata.*` over brittle indices; `trace.last` is the usual default for “current turn” data.\n\n### Reading values\n\n- **`resolve(trace, key)`** — Use when the value always comes from the trace (e.g. the subject `target_key`).\n- **`provided_or_resolve(trace, key=..., value=...)`** — Use when the user may pass an inline value **or** fall back to a path (e.g. `keyword` + `keyword_key`). If `value is not MISSING`, that wins; otherwise the path is evaluated.\n\n### Naming JSONPath fields\n\n- The field naming the **value under test** is always `target_key` (matches the Giskard Hub). It is the only accepted name for that selector and serializes as `target_key`.\n- Every **other** JSONPath field is `{static}_key`, named after the sibling static field holding the literal it would otherwise resolve (`keyword`/`keyword_key`, `context`/`context_key`).\n- Both rules are enforced by `tests/core/test_key_naming_convention.py`.\n\nOptional inline-or-path fields: type as `T | MISSING = MISSING` (and `JSONPathStr | MISSING = MISSING` for optional `*_key` fields). Pass fields directly to `provided_or_resolve`; validate combinations with a `@model_validator` if only one of (inline, `*_key`) may be set (see `StringMatching` / `ComparisonCheck`). Do not use `None` to mean \"extract from trace\"—omit the field so it stays `MISSING`. Explicit `None` is only valid when comparing against or supplying `None` as a real value (e.g. `Equals(expected_value=None)`).\n\n### Failures and types\n\n- Missing matches become **`NoMatch`**. Check with `isinstance(x, NoMatch)` and return `CheckResult.error` with a message that names the field/key (structural — the assertion could not be evaluated). The same applies to wrong type for the configured mode, unsupported comparison, and similar preconditions.\n- When the assertion runs and does not hold, return `CheckResult.failure`.\n- Some paths return a **list** (multiple matches or list-producing JSONPath). See `resolve` in `core/extraction.py` if the check must treat collections differently.\n\n### Defaults\n\nUse `Field(default=\"trace.last.outputs\", ...)` (or another sensible default) so typical scenarios need no extra configuration.\n\nMore detail: [reference.md](reference.md).\n\n## Implementation checklist\n\n1. **Kind string** — Pick a unique snake_case discriminator. Search the repo for `@Check.register(\"` to avoid duplicates.\n\n2. **Class** — Subclass `Check[...]` or `BaseLLMCheck[...]` from `..core.check` / `..judges.base`. Use Pydantic `Field` for config. For non-LLM checks, implement `async def run(self, trace: TraceType) -> CheckResult` (use `CheckResult.success` / `CheckResult.failure` / `CheckResult.error`; put extra data in `details=` when useful).\n\n3. **Registration** — Decorate with `@Check.register(\"your_kind\")` on the class definition.\n\n4. **Wire imports so the kind is registered** — Discriminated unions only know subclasses that were imported. Add an import of the new module to the appropriate package `__init__.py` (e.g. `builtin/__init__.py` re-exports; `judges/__init__.py` for judges). Update `src/giskard/checks/__init__.py` exports and `__all__` if the type is public API.\n\n5. **JSONPath fields** — Follow the section above for every `key` / `*_key`; run tests so `test_jsonpath_enforcement` passes.\n\n6. **Tests** — Add `tests/builtin/test_.py` or extend an existing file (e.g. `test_judge.py`). Mirror package layout. Use `pytest` and async tests where `run` is async. Cover pass, fail, and edge cases (missing values, wrong types).\n\n7. **Validate** — From the **repository root** (this monorepo uses the root `Makefile`):\n\n - `make test-unit PACKAGE=giskard-checks` — unit tests for that lib\n - `make check` — lint, format, compat, typecheck, etc.\n\n Use `make test PACKAGE=giskard-checks` if changes must coexist with functional tests.\n\n `make test-unit PACKAGE=giskard-checks` includes `tests/core/test_jsonpath_enforcement.py`.\n\n## Serialization note\n\nAfter `model_dump()`, `model_validate()` needs every custom `Check` subclass already imported. If deserialization tests fail with an unknown kind, ensure the defining module is imported on the test (or via package `__init__`) before validation.\n\n## LLM checks\n\nFor `BaseLLMCheck`, implement `get_prompt` and respect `output_type` / structured parsing as in existing judges. Details: [reference.md](reference.md).\n","frontmatter":{"name":"giskard-checks-new-check","description":"Guides implementation of new or changed eval checks in the giskard-checks package—Check subclasses, discriminated kind registration, public exports, and tests. Use when adding or modifying built-in checks, LLM-based judges, @Check.register kinds, serialization of TestCase or Scenario, or when editing files under libs/giskard-checks/src/giskard/checks related to checks."},"isInternal":false,"tokens":1542,"sizeBytes":6290},{"name":"SKILL.md","path":".cursor/skills/giskard-oss-pr-ready/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Giskard-AI/giskard-oss/HEAD/.cursor/skills/giskard-oss-pr-ready/SKILL.md","title":"giskard-oss-pr-ready","category":"anthropic-skill","format":"markdown","content":"---\nname: giskard-oss-pr-ready\ndescription: Describes the merge-ready workflow for the giskard-oss monorepo using the repository root Makefile and the same steps as GitHub Actions—format, check, unit tests, optional pre-commit. Use when preparing or updating a pull request, confirming CI will pass, running pre-push validation, or when the user asks for a branch to be merge-ready.\n---\n\n# giskard-oss — PR ready\n\n## Context\n\nWork from the **repository root** (`giskard-oss/`). The monorepo `Makefile` lives there.\n\nMain CI (`.github/workflows/ci.yml`) runs: `make install install-tools`, then `make check`, then `make test-unit PACKAGE=` on Python 3.12–3.14.\n\n## Merge-ready checklist\n\nRun in order:\n\n1. **`make format`** — Ruff format + fix where applicable.\n\n2. **`make check`** — Lint, format check, Python 3.12 compat (vermin), basedpyright, pip-audit, license check. This matches the `lint` job in CI (after `make install install-tools`).\n\n3. **`make test-unit`** — All packages, or narrow scope:\n - `make test-unit PACKAGE=giskard-checks` (same pattern for `giskard-core`, `giskard-agents`)\n\n4. **Optional — pre-commit** — If hooks are installed: `pre-commit run`. First-time setup: `make pre-commit-install` (part of `make setup`).\n\n## Dependencies\n\nIf checks fail from missing tools, run **`make install`** and **`make install-tools`** (CI runs both before `make check`).\n\n## Functional tests\n\n`make test-functional` (and full `make test`) hit live APIs and are not the default PR gate in `ci.yml` (they run in `.github/workflows/integration-tests.yml` with repo secrets). Run them only when the change requires `@pytest.mark.functional` coverage.\n\n### `.env` setup (local)\n\n1. At the **repository root**, create a `.env` file (already listed in `.gitignore`—do not commit secrets).\n2. Define credentials for the models your tests use. The integration workflow sets:\n - `GEMINI_API_KEY` — required for the default Gemini routes used by functional tests.\n - `TEST_MODEL` — optional; default `gemini/gemini-3.5-flash` (see `libs/giskard-agents/tests/conftest.py`).\n - `TEST_EMBEDDING_MODEL` — optional; default `gemini/gemini-embedding-001`.\n3. The repo does not auto-load `.env` in pytest—**export variables into the shell** before running tests, for example:\n\n ```bash\n set -a && source .env && set +a\n make test-functional PACKAGE=giskard-agents\n ```\n\n Use valid shell assignments in `.env` (e.g. `GEMINI_API_KEY=...`). If you point `TEST_MODEL` / `TEST_EMBEDDING_MODEL` at another provider, add whatever API keys that provider expects (LiteLLM / the underlying SDK).\n\n## Verification gate\n\nBefore declaring a branch merge-ready, include the following evidence in your response:\n\n1. Last line(s) of `make check` output — must show no errors.\n2. Pytest summary from `make test-unit` — e.g. `52 passed, 0 failed`.\n3. If any step was skipped or failed, state it explicitly.\n\n\"It should pass\" is not evidence. Show actual output.\n\n## Library-specific rules\n\nFor package conventions (API patterns, tests layout), read `libs//.cursor/rules/`. Makefile targets are always invoked from the repository root as listed above.\n","frontmatter":{"name":"giskard-oss-pr-ready","description":"Describes the merge-ready workflow for the giskard-oss monorepo using the repository root Makefile and the same steps as GitHub Actions—format, check, unit tests, optional pre-commit. Use when preparing or updating a pull request, confirming CI will pass, running pre-push validation, or when the user asks for a branch to be merge-ready."},"isInternal":false,"tokens":829,"sizeBytes":3236}],"systemPromptSnippet":"\n\n# AGENTS.md\n\ngiskard-oss — behavioral config for interactive coding assistants with a human in the loop.\n\n## Workflow Orchestration\n\n### 1. Plan Mode Default\n– Enter plan mode for ANY non-trivial task (3+ steps or touches multiple libs)\n– Brainstorm first (design → spec), then write an implementation plan\n– Save specs and plans in the repo's established planning docs. Prefer `docs/specs/` and `docs/plans/`; if the repo already uses another planning directory, follow that convention.\n– If something goes sideways, STOP and re-plan immediately\n\n### 2. Subagent Strategy\n– Use subagents or equivalent isolated work sessions when the assistant supports them\n– Prefer a fresh subagent/session per task; controller curates exactly the context each one needs\n– Two-stage review after each task: spec compliance first, then code quality\n\n### 3. Self-Improvement Loop\n– After ANY correction: self-document the rule (naming, file headers, examples); only add to AGENTS.md if it cannot be self-documented\n– When a correction changes what was implemented vs. what the plan says: update the plan file immediately so reviewers never see a mismatch\n\n### 4. Verification Before Done\n– Never mark a task complete without proving it works\n– Run: `make format && make check && make test-unit PACKAGE=`\n– Show actual output. Never assume tests pass.\n– Ask yourself: \"Would a staff engineer approve this?\"\n\n### 5. Demand Elegance\n– Pause and ask \"is there a more elegant solution?\"\n– Skip for simple fixes — don't over-engineer\n\n### 6. Autonomous Bug Fixing\n– When given a bug report with clear scope: just fix it\n– No `# type: ignore`, no patched test assertions — fix the root cause\n\n### 7. Commit Hygiene\n– Use Conventional Commits for every commit message (for example, `fix(llm): update google interactions translator`)\n\n## Task Management\n1. Brainstorm → Spec → Plan — save to the repo's established planning docs, preferably `docs/specs/` and `docs/plans/`\n2. Verify Plan — check in before starting\n3. Track Progress — mark items complete as you go\n4. Explain Changes — high-level summary at each step\n5. Document Results — add review section to the plan file\n6. Capture Lessons — update AGENTS.md directly after corrections\n\n## Core Principles\n– Simplicity First: make every change as simple as possible; prefer deleting lines over adding them\n– No Laziness: find root causes; no band-aids, no temporary fixes; senior developer standards\n– Minimal Impact: only touch what's necessary; no side effects; no reformatting untouched lines\n\n## Cursor Cloud specific instructions\n\nNon-obvious context for cloud agents (startup refresh is `make install` then `make install-tools`; `uv` is on `PATH` for login shells via `~/.bashrc`).\n\n– This repo is a `uv` workspace monorepo of pure Python libraries (`libs/giskard-*`). There is no server, daemon, or CLI to start — \"running\" the product means importing the libraries from Python or running `pytest`. Prefer Makefile targets over raw `uv`/`pytest` (`make help`).\n– Import namespace is `giskard.` (e.g. `import giskard.checks`), not `giskard_checks`.\n– Do not run full `make setup` in this VM: its `pre-commit-install` step fails with `Cowardly refusing to install hooks with core.hooksPath set`. That is expected (global `core.hooksPath`) and harmless — use `make install` + `make install-tools` instead; git hooks are not needed to validate work.\n– Lint/test/build/run all work without secrets: `make lint`, `make test-unit [PACKAGE=]`, and importing the libs. Unit tests exclude anything marked `functional`.\n– Functional/integration tests are excluded by default and require live LLM provider API keys (e.g. `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`); `giskard-scan` may also reach Hugging Face Hub. Do not expect these to run without those secrets and network access.\n– `make check` runs `security` (`pip-audit`) and license checks (`check-licenses`/`check-notices`) that reach the network (PyPI / license metadata); if offline these steps can fail even when the code is fine.\n\n\n@AGENTS.md\n\n\n---\ndescription: Documentation and commenting guidelines for an OSS, beginner-friendly codebase.\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Documentation Guidelines\n\nThis is an open-source project. Code should be understandable by newcomers who may not have prior context on the architecture.\n\n## Public API Docstrings\n\nEvery public class, method, and function **must** have a NumPy-style docstring that includes:\n- A one-line summary of what it does\n- Parameters with types and descriptions\n- Return value description\n- A short usage example when the behavior is non-obvious\n\n```python\nasync def run(self, max_steps: int | None = None) -> Chat:\n \"\"\"Execute the workflow and return the resulting chat.\n\n Parameters\n ----------\n max_steps : int, optional\n Maximum number of steps to run. If not provided, the workflow\n runs until completion.\n\n Returns\n -------\n Chat\n The completed chat with all messages and tool call results.\n\n Example\n -------\n >>> chat = await generator.chat(\"Hello!\").run()\n >>> print(chat.last.content)\n \"\"\"\n```\n\n## Inline Comments\n\n- **Do** comment non-obvious control flow, especially async orchestration, retry logic, and tool dispatch.\n- **Do** explain *why* something is done when the reason isn't obvious from the code.\n- **Don't** narrate what the code does line-by-line (\"increment counter\", \"return result\").\n- **Don't** leave TODO comments without a linked issue or clear owner.\n\n```python\n# Good — explains the why\n# Re-parse tool_calls from the raw response because LiteLLM may\n# return partial function arguments when streaming.\ntool_calls = _reparse_tool_calls(raw)\n\n# Bad — restates the code\n# Parse the tool calls\ntool_calls = _reparse_tool_calls(raw)\n```\n\n## Module-Level Docstrings\n\nEach module (`.py` file) should have a brief docstring explaining its role in the system. This helps newcomers navigate the codebase.\n\n## README and Examples\n\n- Keep `README.md` examples accurate and runnable.\n- When adding a new feature, add a corresponding example to the README.\n- Use progressive disclosure: simple usage first, advanced patterns later.\n\n\n---\ndescription: Team-wide guardrails for AI-assisted development.\nalwaysApply: true\n---\n# Guidelines\n\n## No Inventions\n\nOnly make changes that are explicitly requested. Do not add unrequested features, refactors, or improvements.\n\n## Preserve Existing Code\n\nDo not remove unrelated code, functionalities, or comments without explicit reason. Pay attention to preserving existing structures.\n\n## No CODEMAP\n\nNever generate architecture maps, codemap files, or similar high-maintenance documentation that duplicates what is already in cursor rules, docstrings, and README. These files go stale fast and create maintenance burden.\n\n## Root Causes, No Temporary Fixes\n\nTrace failures to their source before making a change. Do not:\n- Add `# type: ignore` / `# pyright: ignore` without first confirming no structural fix is possible.\n- Modify test assertions to force a pass — if a test reveals a real bug, fix the bug.\n- Add `try/except` to suppress exceptions that should be fixed.\n\nIf the root cause is in a different library than where the symptom appears, say so before proposing a fix.\n\n## Minimal Scope\n\nOnly touch files directly required by the task. Do not reformat untouched lines, rename unrelated variables, or add type annotations to code you didn't otherwise change. Propose nearby improvements separately and wait for approval.\n\n\n---\ndescription: Python conventions for the giskard workspace (3.12+, Pydantic v2, async, testing).\nglobs: \"**/*.py\"\nalwaysApply: false\n---\n# Python Conventions\n\n## Python Version\n\n- Python >= 3.12 required across all projects.\n- **Never use `from __future__ import annotations`** — it breaks Pydantic v2 model resolution at runtime and is unnecessary with 3.12+.\n\n## Modern Python 3.12+ Features\n\nPrefer modern syntax over legacy `typing` equivalents.\n\n### PEP 695 — Type Parameter Syntax\n\n```python\n# Good — PEP 695 inline type parameters\nclass Box[T]:\n def __init__(self, value: T): ...\n\ndef first[T](items: list[T]) -> T: ...\n\n# Bad — legacy TypeVar / Generic\nfrom typing import TypeVar, Generic\nT = TypeVar('T')\nclass Box(Generic[T]):\n def __init__(self, value: T): ...\n```\n\n### PEP 695 — Bounded and Constrained Generics\n\n```python\n# Good\nclass NumberBox[T: (int, float)]: ...\nclass Comparable[T: int]: ...\n\n# Bad\nT = TypeVar('T', int, float)\nT = TypeVar('T', bound=int)\n```\n\n### PEP 695 — `type` Statement for Type Aliases\n\n```python\n# Good — unambiguous, supports forward references and recursive types\ntype Vector = list[float]\ntype Tree[T] = T | list[Tree[T]]\n\n# Bad — legacy TypeAlias\nfrom typing import TypeAlias\nVector: TypeAlias = list[float]\n```\n\n### PEP 695 — ParamSpec in Class Syntax\n\nNiche — document with a docstring when used.\n\n```python\n# Good\nclass Wrapper[**P, R]:\n \"\"\"Generic wrapper preserving the signature of the wrapped callable.\n\n Type Parameters\n ---------------\n **P\n Parameter specification of the wrapped callable.\n R\n Return type of the wrapped callable.\n \"\"\"\n ...\n\n# Bad\nfrom typing import ParamSpec\nP = ParamSpec('P')\n```\n\n### Union Syntax (PEP 604)\n\n```python\n# Good\ndef process(value: str | int) -> str | None: ...\n\n# Bad\nfrom typing import Optional, Union\ndef process(value: Union[str, int]) -> Optional[str]: ...\n```\n\n### Built-in Generics\n\nUse built-in types directly — do not import from `typing`.\n\n```python\n# Good\ndef transform(items: list[str]) -> dict[str, int]: ...\ncoords: tuple[float, float, float]\n\n# Bad\nfrom typing import List, Dict, Tuple\ndef transform(items: List[str]) -> Dict[str, int]: ...\n```\n\n### `@override` Decorator\n\nOptional but recommended for methods overriding a parent, especially in large class hierarchies. Catches silent breakage when the parent method is renamed.\n\n```python\nfrom typing import override\n\nclass Child(Base):\n @override\n def process(self) -> None: ...\n```\n\n## Pydantic v2\n\n- Use `BaseModel` for structured data that needs serialization.\n- Prefer immutable models with `ConfigDict(frozen=True)`.\n- Discriminated unions via `@discriminated_base` / `@Cls.register(\"kind\")` from `giskard-core`.\n- Validate external inputs at boundaries; keep internal code operating on typed objects.\n\n## Imports and Naming\n\n- Absolute imports within each `giskard.*` package.\n- Import order enforced by Ruff (`isort`).\n- PEP 8 naming: `snake_case` functions/variables, `PascalCase` classes, `UPPER_CASE` constants.\n\n## Async Patterns\n\n- All workflow execution is async. Never use blocking calls inside async code.\n- Use `asyncio.gather` for parallel execution.\n- Checks and generators may be sync or async — design APIs to support both.\n\n## Error Handling\n\n- Prefer result objects (e.g. `CheckResult`) over exceptions for control flow.\n- Raise exceptions only for truly exceptional states (invalid config, programmer errors).\n- Let runners catch and wrap exceptions when appropriate.\n\n## Testing\n\n- pytest with `asyncio_mode = \"auto\"` — no `@pytest.mark.asyncio` needed.\n- Parametrize tests with descriptive IDs.\n- `@pytest.mark.functional` for tests hitting external APIs.\n- Co-locate tests under `tests/` mirroring package structure.\n- Test success, failure, error, and edge-case paths.\n\n\n---\ndescription: Generator as protocol adapter — architectural rules for provider translation in giskard-agents.\nalwaysApply: true\n---\n\n# Generator as Protocol Adapter\n\n## Core Principles\n\n1. **Message is the canonical internal format.** `role=\"tool\"`, `tool_call_id`, `tool_calls` are the project's standard fields. Never reshape `Message` to accommodate a specific provider.\n\n2. **Tool.run() returns `str`.** The tool serializes its own output. The workflow wraps the string into a `Message`. The workflow never calls `json.dumps` on tool results.\n\n3. **The generator owns all provider translation.** Each generator subclass is responsible for converting between internal `Message` objects and whatever wire format the provider expects. Serialization is internal to the subclass — `BaseGenerator` does not impose any wire format.\n\n## Rules\n\n- **Do not call provider APIs (litellm, anthropic, openai, etc.) from workflow, tool, or chat code.** Provider calls belong exclusively in generator subclasses.\n- **Do not import provider libraries outside of generator modules.** The `generators/` package is the only place that should depend on provider SDKs.\n- **`_complete` is the template method.** It merges params via `GenerationParams.merge()` and delegates to the abstract `_call_model`. Do not override `_complete` in production subclasses — override `_call_model` instead.\n- **`_call_model` receives internal types and returns `Response`.** Signature: `_call_model(messages: list[Message], params: GenerationParams, metadata: dict[str, Any] | None = None) -> Response`. The base class never touches wire formats. The `metadata` parameter carries caller-provided context (e.g. trace IDs, tags) and the returned `Response.metadata` carries provider-side information back to the caller.\n- **Serialization is the subclass's concern.** Each subclass handles translating `Message`/`Tool` to its provider's wire format inside `_call_model` (e.g. `GiskardLLMGenerator._serialize_messages`). Do not add serialization methods to `BaseGenerator`.\n- **Param merging lives on `GenerationParams.merge()`.** Scalar fields from overrides replace the base; tools are concatenated. Do not reimplement merge logic elsewhere.\n- **Adding a new provider?** Subclass `BaseGenerator` and implement `_call_model`. Handle all serialization/deserialization inside the subclass. Do not modify `Message`, `Tool`, or workflow code.\n- **Cross-cutting concerns use middleware.** Retries, rate limiting, logging, and similar concerns are implemented as `CompletionMiddleware` subclasses, not generator mixins. See `generators/middleware.py`.\n\n\n---\ndescription: High level overview of the project. Always read this before starting a new task.\nalwaysApply: false\n---\n# Project Rules and Conventions\n\n## Project Overview\n\nThis is the `giskard-checks` library - lightweight primitives to define and run checks against model interactions.\n\n## Key Files\n\n- Monorepo root `Makefile` — development workflow (run `make help` from repository root)\n- `pyproject.toml` — build configuration, dependencies, and tooling\n- `README.md` — end-user documentation and quickstart\n\n## Development Workflow\n\nAlways use Makefile commands for development tasks. See [03-development.mdc](03-development.mdc) for the complete workflow.\n\n## Code Quality\n\n- Python >= 3.12 required\n- Use Pydantic v2 models for type safety\n- Async-friendly: checks can be sync or async\n- Follow the existing patterns in the codebase\n- Use absolute imports within the `giskard.checks` package\n\n## Testing\n\n- From the repository root: `make test-unit PACKAGE=giskard-checks` for unit tests (or `make test` for all tests including functional)\n- Tests are in `tests/` mirroring the package structure\n- Static checks aligned with CI: `make check` (after `make install install-tools` if needed)\n\n\n---\ndescription: All devs and agent must read this before creating a PR\nalwaysApply: false\n---\n# Development Workflow\n\nThe monorepo **repository root** defines the `Makefile`. Run all targets from that directory (`giskard-oss/`). Prefer `make` over ad hoc `uv` or tool invocations.\n\n## Quick Start\n\n```bash\nmake setup # Complete development setup (install deps + tools)\nmake format && make check && make test-unit PACKAGE=giskard-checks\n```\n\n## Common Commands\n\n- `make help` - Show all available commands with descriptions\n- `make setup` - Complete development setup (install deps + tools)\n- `make install` - Install project dependencies\n- `make install-tools` - Install development tools (ruff, vermin, pre-commit)\n- `make test` - Run all tests (unit + functional) for all libs or `PACKAGE=`\n- `make test-unit` - Unit tests only; use `PACKAGE=giskard-checks` to scope this library\n- `make lint` - Run linting checks\n- `make format` - Format code with ruff\n- `make check-format` - Check if code is formatted correctly\n- `make check-compat` - Check Python 3.12 compatibility\n- `make check` - Run all static checks (lint, format, compat, typecheck, security, licenses)—matches the CI `lint` job (with tools installed)\n- `make clean` - Clean up build artifacts and caches\n\n## Pre-commit\n\n- `make pre-commit-install` - Setup pre-commit hooks\n- `make pre-commit-run` - Run pre-commit on all files\n\n## Guidelines\n\n1. Always use `make` commands instead of direct `uv` or tool invocations\n2. Before pushing, run `make format`, `make check`, and `make test-unit PACKAGE=giskard-checks` (or omit `PACKAGE` to run unit tests for every lib)—this aligns with `.github/workflows/ci.yml`\n3. Use `make setup` for initial project setup\n4. Use `make clean` to clean up build artifacts when needed\n\n## See Also\n\n- [Makefile](../../../../Makefile) at repository root — full command definitions and help\n- [README.md](../../README.md) — package overview and quickstart\n\n\n---\ndescription: Architecture and design principles for giskard-llm. Always read before modifying providers, routing, or error handling.\nalwaysApply: true\n---\n\n# giskard-llm Architecture\n\n## Overview\n\n`giskard-llm` is a lightweight routing layer that dispatches `\"provider/model\"` strings to native LLM SDKs. It replaces litellm with direct SDK calls while presenting a unified response shape to consumers.\n\n## Architecture\n\nThe public API (`acompletion`, `aembedding`, `aresponse`) routes `\"provider/model\"` strings through a lazy-loading registry to provider implementations. Providers are self-contained modules under `providers/`, each conforming to one or more `Protocol` classes (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`) and responsible for: role mapping, message validation, tool conversion, error mapping, and response normalization.\n\nShared contracts live at the package root: Pydantic v2 response types (OpenAI-shaped), a unified error hierarchy with `status_code` for consumer retry logic, and a `should_retry` helper.\n\n## Type Conventions\n\nInput types (messages, tool definitions, tool results) use `TypedDict` — lightweight, constructed with dict literals. Output types (responses, tool calls, embeddings) use Pydantic `_BaseModel` — attribute access, `.model_dump()`, validation on construction. See `docs/design.md` for rationale.\n\n## Core Principles\n\n1. **Lazy imports.** Importing `giskard.llm` must never require any provider SDK. SDKs are imported inside provider modules only.\n2. **Strict message validation.** Providers validate messages before calling the SDK and raise `BadRequestError` with a clear message. Invalid input must not silently pass through to opaque SDK errors. Opt-in relaxation (e.g., `merge_system=True`) is explicit.\n3. **Unified error boundary.** Raw SDK exceptions never escape a provider. Every provider maps its SDK errors to the `errors.py` hierarchy.\n4. **Response normalization.** All providers convert native responses to the shared `types.py` models. Consumers never see provider-specific shapes.\n5. **Provider config via env vars or `**params`.** The public API has no provider-specific kwargs. Configuration flows through environment variables or pass-through params.\n6. **Provider behavior is self-documented.** Each provider class must have a comprehensive docstring covering: env vars, role mapping, error mapping, supported features, and provider-specific kwargs. This is the source of truth for provider behavior.\n\n## Adding a New Provider\n\n1. Create `providers/.py` implementing the relevant protocols (`CompletionProvider`, `EmbeddingProvider`, `ResponseProvider`).\n2. Guard SDK imports with `try/except ImportError` and raise `ProviderNotAvailableError`.\n3. Register it in the provider registry in `routing.py`.\n4. Add the SDK as an optional dependency extra in `pyproject.toml`.\n5. Implement role mapping, message validation, tool conversion, error mapping, and response conversion.\n6. Write the class docstring following the provider documentation template (see `04-provider-docs.mdc`).\n7. Add unit tests (mocked SDK) and functional test scenarios.\n\n\n---\ndescription: Testing conventions for giskard-llm. Read before writing or modifying tests.\nglobs: \"**/test*.py,**/conftest.py\"\n---\n\n# giskard-llm Testing Conventions\n\n## Test Structure\n\n- **Unit tests** (`tests/`): Mocked SDK calls, no API keys needed. Cover routing, message conversion, error mapping, response conversion, validation.\n- **Functional tests** (`tests/functional/`): Real API calls. Cover end-to-end scenarios across all providers.\n\n## Provider Marks and Auto-Skip\n\nEvery functional test is marked with its provider (`@pytest.mark.google`, `@pytest.mark.openai`, etc.). Auto-skip logic in `conftest.py` skips tests whose provider SDK is not installed, so unit test runs never fail due to missing optional dependencies.\n\n## Scenario Design\n\n- **Assert on structure, not content.** Tests must pass with even the weakest model. Assert non-empty responses, correct roles, correct types, parseable JSON — never assert on specific wording.\n- **Meaningful inputs and outputs.** Test scenarios should exercise real behavior: system prompts that produce verifiable effects, tool calls with checkable arguments, structured output that validates against a schema. The goal is a test that fails for the right reasons.\n\n\n---\ndescription: Development workflow for giskard-llm. Read before creating a PR.\nalwaysApply: false\n---\n\n# giskard-llm Development Workflow\n\n## Dependencies\n\n- **Core**: `pydantic>=2.0` only. No other runtime dependencies.\n- **Provider SDKs**: Optional extras in `pyproject.toml` (`openai`, `google`, `anthropic`, `azure`). Never add a provider SDK to core.\n- **Package manager**: `uv`.\n\n## Format and Lint\n\n`ruff format` and `ruff check` via pre-commit hooks. If `basedpyright` fails on provider files due to uninstalled SDKs, add `# pyright: reportMissingImports=false` at the top of the provider file.\n\n## CI\n\n- **Unit tests**: run on every PR, no SDK required — all provider interactions are mocked.\n- **Functional tests**: run per-provider in a matrix. Each matrix entry installs only its SDK and injects only its env vars at the step level.\n- **`workflow_dispatch`**: allows manual triggering with org-membership check.\n\n## Commit Messages\n\nSemantic format scoped to the lib: `feat(giskard-llm): add azure provider`, `fix(giskard-llm): google system message extraction`.\n\n\n---\ndescription: Provider documentation standards. Read when creating or modifying a provider.\nglobs: \"**/providers/*.py\"\n---\n\n# Provider Documentation\n\nEach provider class must have a comprehensive docstring that serves as the single source of truth for that provider's behavior. This replaces external documentation that would go stale.\n\n## Required Docstring Sections\n\nEvery provider class docstring must cover:\n\n1. **Overview** — SDK used, what model prefix routes here (e.g., `\"google/\"`, `\"azure/\"`).\n2. **Authentication** — Required env vars (e.g., `GOOGLE_API_KEY`) and alternative kwargs.\n3. **Role mapping** — How canonical roles (`system`, `user`, `assistant`, `tool`) map to the SDK format.\n4. **Message constraints** — Provider-specific validation rules (alternation, system message handling, etc.).\n5. **Tool call format** — How tool definitions and tool results are converted.\n6. **Error mapping** — Which SDK exceptions map to which `LLMError` subclasses.\n7. **Supported features** — Completion, embeddings, structured output, and any limitations.\n8. **Provider-specific kwargs** — Any pass-through params unique to this provider (e.g., `merge_system` for Anthropic).\n\n## Example\n\n```python\nclass GoogleProvider:\n \"\"\"Google Gemini provider using the ``google-genai`` SDK.\n\n Routing prefix: ``google/``\n\n Authentication:\n - Env: ``GOOGLE_API_KEY`` (or ``GEMINI_API_KEY``)\n - Kwargs: ``api_key``\n\n Role mapping:\n - ``system`` -> extracted to ``system_instruction`` config (accepts a list)\n - ``assistant`` -> ``model``\n - ``tool`` -> ``function_response`` part\n - ``user`` -> ``user``\n\n Message constraints:\n - Multiple system messages: supported natively (passed as list)\n - System-only messages: raises ``BadRequestError``\n - No strict alternation required\n\n Tool call format:\n - Tool definitions: converted to ``FunctionDeclaration``\n - Tool results: converted to ``function_response`` parts\n - Tool call IDs: synthetic (``call_``) since Gemini doesn't provide them\n\n Error mapping:\n - ``google.genai.errors.ClientError`` (400) -> ``BadRequestError``\n - ``google.genai.errors.ClientError`` (401/403) -> ``AuthenticationError``\n - ``google.genai.errors.ClientError`` (429) -> ``RateLimitError``\n - ``google.genai.errors.ServerError`` -> ``ServerError``\n\n Supported features:\n - Completion: yes\n - Embeddings: yes\n - Structured output (response_format): yes, via ``response_schema``\n\n Provider-specific kwargs:\n - ``safety_settings``: override default safety settings\n \"\"\"\n```\n\n## Rules\n\n- **The docstring is the spec.** When behavior changes, update the docstring in the same commit.\n- **No external provider docs.** Don't maintain separate markdown files per provider. The class docstring is authoritative.\n- **README references providers briefly.** The top-level README has a summary table; details link to the source.\n\n\n---\nname: check\ndescription: Run the giskard-oss verification gate (format, check, unit tests) for a package\ndisable-model-invocation: true\n---\n\n# check\n\nRun the giskard-oss verification gate that `AGENTS.md` mandates before any task is \"done\":\n\n```\nmake format && make check && make test-unit PACKAGE=\n```\n\n## Usage\n\n`/check ` — run the full gate scoped to one lib's unit tests.\n\n`/check` — run the gate; unit tests cover all libs (slower).\n\n`` is a directory under `libs/`: `giskard-agents`, `giskard-checks`, `giskard-core`, `giskard-llm`, `giskard-scan`.\n\n## What to run\n\nWith a package:\n\n```bash\nmake format && make check && make test-unit PACKAGE=\n```\n\nWithout a package:\n\n```bash\nmake format && make check && make test-unit\n```\n\n`format` and `check` are repo-wide and take no `PACKAGE`. Only `test-unit` is scoped.\n\n## Notes\n\n- Run the three steps as one `&&` chain so a failure stops the gate early.\n- Report failures with the command's output — do not claim the gate passed unless every step exited 0.\n- `make format` mutates files (ruff format + check --fix); show the resulting diff if anything changed.\n\n\n---\nname: fix-cve\ndescription: Use when pip-audit (or a CVE/GHSA advisory, Dependabot, or security scan) flags a vulnerable Python dependency in giskard-oss and it needs upgrading to a fixed version in the uv lockfile.\n---\n\n# fix-cve\n\nRemediate a vulnerable dependency flagged by `pip-audit` (or any CVE/GHSA advisory) by upgrading it in the **uv lockfile**, then verify with the repo gate.\n\n`pip-audit` runs as part of `make check` (the `security` target). The advisory looks like:\n\n```\nName Version ID Fix Versions\n------- ------- ------------------- ------------\nmsgpack 1.1.2 GHSA-6v7p-g79w-8964 1.2.1\n```\n\nYou need: the **package name** and a **fix version**.\n\n## Procedure\n\n```bash\n# 1. Upgrade the package in the lockfile (works for direct AND transitive deps).\nuv lock --upgrade-package \n\n# 2. Apply to the environment.\nuv sync\n\n# 3. Verify the advisory is gone — re-run the exact audit command.\nuv run pip-audit --skip-editable\n```\n\nThen run the full repo gate (it re-runs `pip-audit` via the `security` target, plus format/lint/types/tests):\n\n```bash\nmake format && make check && make test-unit\n```\n\nOr invoke `/check` (the project verification-gate skill).\n\n## Key facts\n\n- **Always upgrade in the lockfile, not the environment.** Use `uv lock --upgrade-package `, NOT `pip install` / `uv pip install`. The lockfile is the source of truth; a `pip install` fix is lost on the next `uv sync`.\n- **Transitive deps are fixed the same way.** Most flagged packages (e.g. `msgpack`, pulled in by `cachecontrol`) are not in any `pyproject.toml`. You still upgrade them by name with `uv lock --upgrade-package ` — no `pyproject.toml` edit needed. Do NOT add a direct dependency just to pin it.\n- **`uv lock --upgrade-package ` takes the newest version the constraints allow.** Check the resulting `uv.lock` entry to confirm the version actually moved to the fix version — a cap may have held it below.\n\n### When a constraint caps the version below the fix\n\nIf another dependency caps the package below the fix version, the bare upgrade silently stays on the old version, and forcing it with `uv lock --upgrade-package \">=\"` **fails with a resolution error** (`No solution found...`) — uv constraints can only *narrow* the version set, never expand past a cap. Escalate in this order:\n\n1. **Upgrade the capping parent.** Find who imposes the cap (`uv tree --invert --package `), then `uv lock --upgrade-package `. A newer parent often relaxes the cap. Preferred — keeps the dependency graph honest.\n2. **Override the transitive dep** if no parent upgrade resolves it. Add to `pyproject.toml`:\n ```toml\n [tool.uv]\n override-dependencies = [\">=\"]\n ```\n Overrides *expand* the allowed set (the only escape hatch a cap can't block), then re-run `uv lock`. Use sparingly — it ignores the parent's declared constraint, so re-verify nothing breaks with the full gate.\n- **Lockfile-only changes have no \"affected lib\".** Run the full `make test-unit` (all packages), not a single `PACKAGE=` scope — the dep could affect any lib.\n\n## Verifying the fix\n\nDo not assert \"fixed\" — show evidence:\n\n1. `uv run pip-audit --skip-editable` prints **\"No known vulnerabilities found\"** (editable `giskard-*` libs in the skip list are expected and harmless).\n2. The `uv.lock` diff shows the package moved to the fix version (`Updated vX -> vY` from `uv lock`).\n3. `make check` passes (its `security` target re-runs the audit).\n\n## Common mistakes\n\n| Mistake | Fix |\n|---------|-----|\n| `pip install ==` to fix it | Use `uv lock --upgrade-package `; pip changes don't persist in the lockfile |\n| Editing `pyproject.toml` for a transitive dep | Not needed — `uv lock --upgrade-package` handles transitive deps by name |\n| Skipping re-audit, assuming the upgrade worked | Re-run `uv run pip-audit --skip-editable`; a constraint may have silently capped the version below the fix |\n| Force-pinning past a cap with `\">=\"` and hitting `No solution found` | Constraints can't expand past a cap — upgrade the capping parent, or use `[tool.uv] override-dependencies` |\n| Guessing the verification command | Run `/check` or `make format && make check && make test-unit`; `make check` includes `pip-audit` |\n| Scoping tests to one `PACKAGE=` | A lockfile change has no single affected lib; run all unit tests |\n\n\n---\nname: giskard-oss-quality-gate\ndescription: Run the full code quality gate for the giskard-oss monorepo—format, lint, typecheck, and unit tests across all packages. Use when finishing a feature, before opening a PR, or when the user asks to check quality or run the gate.\ndisable-model-invocation: true\nallowed-tools: Bash(make *)\n---\n\n# giskard-oss — quality gate\n\n## Results\n\n```!\necho \"=== make format ===\" && make format 2>&1 | tail -5\necho \"=== make check ===\" && make check 2>&1 | tail -10\necho \"=== make test-unit PACKAGE=giskard-core ===\" && make test-unit PACKAGE=giskard-core 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-checks ===\" && make test-unit PACKAGE=giskard-checks 2>&1 | tail -5\necho \"=== make test-unit PACKAGE=giskard-agents ===\" && make test-unit PACKAGE=giskard-agents 2>&1 | tail -5\n```\n\nReport pass/fail for each step. Fix root causes on failure — no `# type: ignore`, no skips. Re-run after fixing.\n\n\n---\ndescription: Set up the full workspace dev environment. Use when the user wants to install dependencies, sync libs, or prepare the environment.\ndisable-model-invocation: true\nallowed-tools: Bash(make *) Bash(uv *)\n---\n\n## Setup dev env\n\n!`make setup 2>&1 | tail -20`\n\n## Validate installation and notify user\n\nCheck the output above for errors. If any errors occurred, try to fix them and re-run the failed step. Report whether installation was successful.\n\n\n# Reference — check variants in giskard-checks\n\n## JSONPath enforcement and helpers\n\n- **Source**: `libs/giskard-checks/src/giskard/checks/core/extraction.py` — `JSONPathStr`, `resolve`, `provided_or_resolve`, `NoMatch`.\n- **Enforcement**: `libs/giskard-checks/tests/core/test_jsonpath_enforcement.py` — field names `key` or `*_key` must carry the `JSONPathStr` marker (including inside unions with `None` or `MISSING`).\n- **`MISSING` inline-or-path** (`ComparisonCheck`): `expected_value: ExpectedType | MISSING = MISSING` and `expected_value_key: JSONPathStr | MISSING = MISSING`; validate with `is MISSING` checks; in `run`, call `provided_or_resolve(trace, key=self.expected_value_key, value=self.expected_value)` so an inline `expected_value` (including explicit `None`) skips the path.\n- **`MISSING` optional path** (`StringMatching.keyword_key`): `JSONPathStr | MISSING = MISSING` with a validator that requires exactly one of inline value or key—do not use a bare `str` annotation for any `*_key` field.\n- **Multi-match**: `resolve` returns a list when the expression yields multiple matches or the parsed path is list-like (`Slice`, `Union`, etc.); single-match paths return one value or `NoMatch`.\n\n## Non-LLM `Check`\n\n- See `libs/giskard-checks/src/giskard/checks/builtin/` for patterns: comparison checks, `FnCheck`, `StringMatching` / `RegexMatching`, `SemanticSimilarity`.\n- Subclass `Check[InputType, OutputType, TraceType]` when generics matter; many builtins use `Trace[dict[str, str], dict[str, str]]` or similar.\n- Prefer returning `CheckResult` with clear `message` and structured `details` for debugging.\n\n## `BaseLLMCheck` (judges)\n\n- Defined in `libs/giskard-checks/src/giskard/checks/judges/base.py`.\n- Implement `get_prompt()` returning `str`, `Message`, `MessageTemplate`, or `TemplateReference`.\n- Default structured output model is `LLMCheckResult` (`passed`, `reason`); override `output_type` if using a different Pydantic model for the LLM response.\n- Uses `WithGeneratorMixin` — generator defaults come from library settings (`set_default_generator` / `get_default_generator` in public API).\n- Examples: `Groundedness`, `Conformity`, `LLMJudge` in the same `judges/` package.\n\n## Export surfaces\n\nTypical touch points for a **public** check:\n\n1. `builtin/__init__.py` or `judges/__init__.py` — import and re-export; update `__all__`.\n2. `src/giskard/checks/__init__.py` — import from submodule and add to `__all__` alongside other checks.\n\nInternal-only types are rare; if you add one, skip top-level exports but still register the kind and import the module where deserialization must work.\n\n\n---\nname: giskard-checks-new-check\ndescription: Guides implementation of new or changed eval checks in the giskard-checks package—Check subclasses, discriminated kind registration, public exports, and tests. Use when adding or modifying built-in checks, LLM-based judges, @Check.register kinds, serialization of TestCase or Scenario, or when editing files under libs/giskard-checks/src/giskard/checks related to checks.\n---\n\n# giskard-checks — new check\n\n## Scope\n\nLibrary code lives under `libs/giskard-checks/`. Follow existing patterns; use absolute imports inside `giskard.checks`. Python 3.12+, Pydantic v2.\n\nRead project rules when relevant: `libs/giskard-checks/.cursor/rules/02-project.mdc`, `03-development.mdc`.\n\n## Classify the check\n\n| Kind | Module | Base class |\n|------|--------|------------|\n| Deterministic (string, numeric, custom logic on trace) | `src/giskard/checks/builtin/` | `Check` (often shared helpers in existing builtins) |\n| Uses LLM to score or classify | `src/giskard/checks/judges/` | `BaseLLMCheck` |\n\nFor one-off or experimental logic, users can use `from_fn` / `FnCheck` without a new class—only add a new type when it is a reusable, serializable primitive.\n\n## JSONPath fields (`key` and `*_key`)\n\n### Naming and typing (required)\n\nAny `Check` field named `key` or whose name ends with `_key` is a JSONPath into the trace. It **must** be annotated with `JSONPathStr`, or `JSONPathStr | None`, or `JSONPathStr | MISSING` (import `MISSING` from `pydantic.experimental.missing_sentinel`). Otherwise `tests/core/test_jsonpath_enforcement.py` fails.\n\nInside the library, import from `..core.extraction` (module `giskard.checks.core.extraction`).\n\n### Path syntax\n\n- Every path must start with `trace.` (validated at model construction via `jsonpath_ng`).\n- Resolution runs against `{\"trace\": trace.model_dump()}`. Prefer `trace.last.outputs`, `trace.last.inputs`, `trace.last.metadata.*` over brittle indices; `trace.last` is the usual default for “current turn” data.\n\n### Reading values\n\n- **`resolve(trace, key)`** — Use when the value always comes from the trace (e.g. the subject `target_key`).\n- **`provided_or_resolve(trace, key=..., value=...)`** — Use when the user may pass an inline value **or** fall back to a path (e.g. `keyword` + `keyword_key`). If `value is not MISSING`, that wins; otherwise the path is evaluated.\n\n### Naming JSONPath fields\n\n- The field naming the **value under test** is always `target_key` (matches the Giskard Hub). It is the only accepted name for that selector and serializes as `target_key`.\n- Every **other** JSONPath field is `{static}_key`, named after the sibling static field holding the literal it would otherwise resolve (`keyword`/`keyword_key`, `context`/`context_key`).\n- Both rules are enforced by `tests/core/test_key_naming_convention.py`.\n\nOptional inline-or-path fields: type as `T | MISSING = MISSING` (and `JSONPathStr | MISSING = MISSING` for optional `*_key` fields). Pass fields directly to `provided_or_resolve`; validate combinations with a `@model_validator` if only one of (inline, `*_key`) may be set (see `StringMatching` / `ComparisonCheck`). Do not use `None` to mean \"extract from trace\"—omit the field so it stays `MISSING`. Explicit `None` is only valid when comparing against or supplying `None` as a real value (e.g. `Equals(expected_value=None)`).\n\n### Failures and types\n\n- Missing matches become **`NoMatch`**. Check with `isinstance(x, NoMatch)` and return `CheckResult.error` with a message that names the field/key (structural — the assertion could not be evaluated). The same applies to wrong type for the configured mode, unsupported comparison, and similar preconditions.\n- When the assertion runs and does not hold, return `CheckResult.failure`.\n- Some paths return a **list** (multiple matches or list-producing JSONPath). See `resolve` in `core/extraction.py` if the check must treat collections differently.\n\n### Defaults\n\nUse `Field(default=\"trace.last.outputs\", ...)` (or another sensible default) so typical scenarios need no extra configuration.\n\nMore detail: [reference.md](reference.md).\n\n## Implementation checklist\n\n1. **Kind string** — Pick a unique snake_case discriminator. Search the repo for `@Check.register(\"` to avoid duplicates.\n\n2. **Class** — Subclass `Check[...]` or `BaseLLMCheck[...]` from `..core.check` / `..judges.base`. Use Pydantic `Field` for config. For non-LLM checks, implement `async def run(self, trace: TraceType) -> CheckResult` (use `CheckResult.success` / `CheckResult.failure` / `CheckResult.error`; put extra data in `details=` when useful).\n\n3. **Registration** — Decorate with `@Check.register(\"your_kind\")` on the class definition.\n\n4. **Wire imports so the kind is registered** — Discriminated unions only know subclasses that were imported. Add an import of the new module to the appropriate package `__init__.py` (e.g. `builtin/__init__.py` re-exports; `judges/__init__.py` for judges). Update `src/giskard/checks/__init__.py` exports and `__all__` if the type is public API.\n\n5. **JSONPath fields** — Follow the section above for every `key` / `*_key`; run tests so `test_jsonpath_enforcement` passes.\n\n6. **Tests** — Add `tests/builtin/test_.py` or extend an existing file (e.g. `test_judge.py`). Mirror package layout. Use `pytest` and async tests where `run` is async. Cover pass, fail, and edge cases (missing values, wrong types).\n\n7. **Validate** — From the **repository root** (this monorepo uses the root `Makefile`):\n\n - `make test-unit PACKAGE=giskard-checks` — unit tests for that lib\n - `make check` — lint, format, compat, typecheck, etc.\n\n Use `make test PACKAGE=giskard-checks` if changes must coexist with functional tests.\n\n `make test-unit PACKAGE=giskard-checks` includes `tests/core/test_jsonpath_enforcement.py`.\n\n## Serialization note\n\nAfter `model_dump()`, `model_validate()` needs every custom `Check` subclass already imported. If deserialization tests fail with an unknown kind, ensure the defining module is imported on the test (or via package `__init__`) before validation.\n\n## LLM checks\n\nFor `BaseLLMCheck`, implement `get_prompt` and respect `output_type` / structured parsing as in existing judges. Details: [reference.md](reference.md).\n\n\n---\nname: giskard-oss-pr-ready\ndescription: Describes the merge-ready workflow for the giskard-oss monorepo using the repository root Makefile and the same steps as GitHub Actions—format, check, unit tests, optional pre-commit. Use when preparing or updating a pull request, confirming CI will pass, running pre-push validation, or when the user asks for a branch to be merge-ready.\n---\n\n# giskard-oss — PR ready\n\n## Context\n\nWork from the **repository root** (`giskard-oss/`). The monorepo `Makefile` lives there.\n\nMain CI (`.github/workflows/ci.yml`) runs: `make install install-tools`, then `make check`, then `make test-unit PACKAGE=` on Python 3.12–3.14.\n\n## Merge-ready checklist\n\nRun in order:\n\n1. **`make format`** — Ruff format + fix where applicable.\n\n2. **`make check`** — Lint, format check, Python 3.12 compat (vermin), basedpyright, pip-audit, license check. This matches the `lint` job in CI (after `make install install-tools`).\n\n3. **`make test-unit`** — All packages, or narrow scope:\n - `make test-unit PACKAGE=giskard-checks` (same pattern for `giskard-core`, `giskard-agents`)\n\n4. **Optional — pre-commit** — If hooks are installed: `pre-commit run`. First-time setup: `make pre-commit-install` (part of `make setup`).\n\n## Dependencies\n\nIf checks fail from missing tools, run **`make install`** and **`make install-tools`** (CI runs both before `make check`).\n\n## Functional tests\n\n`make test-functional` (and full `make test`) hit live APIs and are not the default PR gate in `ci.yml` (they run in `.github/workflows/integration-tests.yml` with repo secrets). Run them only when the change requires `@pytest.mark.functional` coverage.\n\n### `.env` setup (local)\n\n1. At the **repository root**, create a `.env` file (already listed in `.gitignore`—do not commit secrets).\n2. Define credentials for the models your tests use. The integration workflow sets:\n - `GEMINI_API_KEY` — required for the default Gemini routes used by functional tests.\n - `TEST_MODEL` — optional; default `gemini/gemini-3.5-flash` (see `libs/giskard-agents/tests/conftest.py`).\n - `TEST_EMBEDDING_MODEL` — optional; default `gemini/gemini-embedding-001`.\n3. The repo does not auto-load `.env` in pytest—**export variables into the shell** before running tests, for example:\n\n ```bash\n set -a && source .env && set +a\n make test-functional PACKAGE=giskard-agents\n ```\n\n Use valid shell assignments in `.env` (e.g. `GEMINI_API_KEY=...`). If you point `TEST_MODEL` / `TEST_EMBEDDING_MODEL` at another provider, add whatever API keys that provider expects (LiteLLM / the underlying SDK).\n\n## Verification gate\n\nBefore declaring a branch merge-ready, include the following evidence in your response:\n\n1. Last line(s) of `make check` output — must show no errors.\n2. Pytest summary from `make test-unit` — e.g. `52 passed, 0 failed`.\n3. If any step was skipped or failed, state it explicitly.\n\n\"It should pass\" is not evidence. Show actual output.\n\n## Library-specific rules\n\nFor package conventions (API patterns, tests layout), read `libs//.cursor/rules/`. Makefile targets are always invoked from the repository root as listed above.\n"}