{"owner":"microsoft","repo":"data-formulator","hasSkills":true,"totalSkillsCount":27,"totalTokensCount":34006,"categories":["cursor-rule","anthropic-skill","copilot-instructions","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":[".cursor/rules/backend-test-conventions.mdc",".cursor/rules/dataframe-serialization.mdc",".cursor/rules/dev-guides-first.mdc",".cursor/rules/error-response-safety.mdc",".cursor/rules/frontend-test-conventions.mdc",".cursor/rules/i18n-no-hardcoded-strings.mdc",".cursor/rules/implementation-review-checklist.mdc",".cursor/rules/incremental-development-cadence.mdc",".cursor/rules/language-injection-conventions.mdc",".cursor/rules/log-sanitization.mdc",".cursor/rules/package-manager-conventions.mdc",".cursor/rules/path-safety.mdc",".cursor/rules/test-driven-workflow.mdc",".cursor/rules/unified-error-protocol.mdc",".cursor/skills/error-handling/SKILL.md",".cursor/skills/language-injection/SKILL.md",".cursor/skills/path-safety/SKILL.md",".github/copilot-instructions.md","examples/plugins/README.md","py-src/data_formulator/analyst/skills/core/SKILL.md","py-src/data_formulator/analyst/skills/core/tools.json","py-src/data_formulator/analyst/skills/data-loading/SKILL.md","py-src/data_formulator/analyst/skills/data-loading/tools.json","py-src/data_formulator/analyst/skills/data_loading/SKILL.md","py-src/data_formulator/analyst/skills/data_loading/tools.json","py-src/data_formulator/analyst/skills/report/SKILL.md","py-src/data_formulator/analyst/skills/report/tools.json"],"skills":{".cursor/rules/backend-test-conventions.mdc":"---\r\ndescription: Backend Python test conventions (pytest)\r\nglobs: tests/backend/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# Backend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/backend/unit/`, `tests/backend/integration/`, `tests/backend/contract/`, or by domain: `tests/backend/routes/`, `tests/backend/errors/`, `tests/backend/agents/`, `tests/backend/data/`, `tests/backend/security/`, `tests/backend/auth/`, `tests/backend/knowledge/`.\r\n- Name files `test_.py` with descriptive snake_case names.\r\n- Shared test data goes in `tests/backend/fixtures/`.\r\n\r\n## File Structure\r\n\r\n```python\r\n\"\"\"One-paragraph summary of what is being tested and why.\r\n\r\nBackground\r\n----------\r\nBrief context about the feature or bug fix these tests validate.\r\n\"\"\"\r\nfrom __future__ import annotations\r\n\r\nimport pytest\r\n\r\npytestmark = [pytest.mark.backend]\r\n```\r\n\r\n## Conventions\r\n\r\n- Always add `pytestmark = [pytest.mark.backend]` at module level.\r\n- Group related tests in classes prefixed with `Test` (e.g. `TestStripImageBlocks`).\r\n- Use `pytest.mark.parametrize` for data-driven tests instead of writing repetitive cases.\r\n- Use `pytest.fixture()` for shared setup; keep fixtures close to where they are used.\r\n- Unit tests must **not** depend on Flask, network, or external services.\r\n- Mock external calls with `unittest.mock.patch` / `MagicMock`; never make real API calls.\r\n- Each test function should verify **one** behavior and have a clear name: `test__`.\r\n\r\n## Running Tests\r\n\r\n- When running pytest, use `-q` (quiet mode) instead of `-v` (verbose mode) for cleaner output.\r\n- **Preferred:** `python -m pytest tests/backend/ -q`\r\n- **Avoid:** `python -m pytest tests/backend/ -v 2>&1`\r\n\r\n## Error Handling Test Requirements\r\n\r\nApplication errors return **HTTP 200** with `status: \"error\"` in the body.\r\nOnly auth errors (`AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403) and\r\nuncontrolled transport failures (404/413/500) use non-200.\r\n\r\nWhen adding a new route or modifying error handling, include tests for:\r\n\r\n1. **AppError scenarios** — verify HTTP 200 + correct `ErrorCode` in body:\r\n ```python\r\n def test_missing_table_returns_error():\r\n with app.test_client() as client:\r\n resp = client.get('/api/tables/nonexistent')\r\n assert resp.status_code == 200 # application error → always 200\r\n data = resp.get_json()\r\n assert data[\"status\"] == \"error\"\r\n assert data[\"error\"][\"code\"] == \"TABLE_NOT_FOUND\"\r\n ```\r\n\r\n2. **Auth errors** — verify non-200 for authentication/authorization failures:\r\n ```python\r\n def test_access_denied_returns_403():\r\n resp = client.get('/api/protected-resource')\r\n assert resp.status_code == 403\r\n assert resp.get_json()[\"error\"][\"code\"] == \"ACCESS_DENIED\"\r\n ```\r\n\r\n3. **Unexpected exceptions** — verify the global handler catches them (unhandled = 500):\r\n ```python\r\n def test_unexpected_error_returns_500():\r\n with patch('module.function', side_effect=RuntimeError(\"boom\")):\r\n resp = client.get('/api/endpoint')\r\n assert resp.status_code == 500\r\n assert resp.get_json()[\"error\"][\"code\"] == \"INTERNAL_ERROR\"\r\n ```\r\n\r\n4. **Streaming endpoints** — verify error events in the NDJSON stream:\r\n ```python\r\n def test_stream_error_yields_ndjson_event():\r\n resp = client.post('/api/my-stream', ...)\r\n lines = [json.loads(l) for l in resp.data.decode().strip().split('\\n') if l]\r\n error_lines = [l for l in lines if l.get('type') == 'error']\r\n assert len(error_lines) >= 1\r\n assert error_lines[0]['error']['code'] == 'LLM_TIMEOUT'\r\n ```\r\n\r\n5. **Test fixture** — ensure `register_error_handlers(app)` is called in the test app fixture:\r\n ```python\r\n @pytest.fixture\r\n def app():\r\n from data_formulator.error_handler import register_error_handlers\r\n _app = flask.Flask(__name__)\r\n _app.config[\"TESTING\"] = True\r\n _app.register_blueprint(my_bp)\r\n register_error_handlers(_app)\r\n return _app\r\n ```\r\n\r\nSee `tests/backend/errors/test_errors.py` and `test_error_handler.py` for reference implementations.\r\n\r\n## Example\r\n\r\n```python\r\n# ❌ BAD – vague name, no parametrize, no marker\r\ndef test_it_works():\r\n assert sanitize(\"hello world\") == \"hello_world\"\r\n\r\n# ✅ GOOD\r\npytestmark = [pytest.mark.backend]\r\n\r\n@pytest.mark.parametrize(\"raw,expected\", [\r\n (\"hello world\", \"hello_world\"),\r\n (\"订单明细\", \"订单明细\"),\r\n])\r\ndef test_sanitize_preserves_unicode(raw: str, expected: str) -> None:\r\n assert sanitize(raw) == expected\r\n```\r\n",".cursor/rules/dataframe-serialization.mdc":"# DataFrame Serialization\n\nAll DataFrame-to-records conversion for API responses, streaming events, or\nfrontend-visible data MUST use the centralized helpers in\n`data_formulator.datalake.parquet_utils`:\n\n| Source type | Helper |\n|---|---|\n| `pd.DataFrame` | `df_to_safe_records(df)` |\n| `pa.Table` (Arrow) | `get_sample_rows_from_arrow(table)` |\n\n## Why\n\n`pandas.DataFrame.to_json(orient='records')` defaults to `date_format='epoch'`,\nwhich serializes datetime columns as **epoch milliseconds** (e.g. `1773532800000`).\nThe frontend interprets these as plain numbers and renders them with commas\n(`1,773,532,800,000`) instead of formatted dates.\n\n`df_to_safe_records` enforces `date_format='iso'` and `default_handler=str`,\nensuring datetimes become ISO-8601 strings and exotic types degrade gracefully.\n\n## Banned Patterns\n\n```python\n# BAD — missing date_format, datetimes become epoch numbers\njson.loads(df.to_json(orient='records'))\n\n# BAD — to_dict returns Timestamp objects, not JSON-safe values\ndf.to_dict(orient='records')\n\n# ACCEPTABLE but should be unified for consistency\njson.loads(df.to_json(orient='records', date_format='iso'))\n```\n\n## Correct Pattern\n\n```python\nfrom data_formulator.datalake.parquet_utils import df_to_safe_records\n\nrows = df_to_safe_records(df)\npreview = df_to_safe_records(df.head(5))\n```\n\n## Exceptions\n\nInternal data processing that never reaches the frontend or JSON serialization\n(e.g. Kusto SDK metadata parsing, Vega-Lite spec construction) may use\n`to_dict(orient='records')` directly.\n",".cursor/rules/dev-guides-first.mdc":"---\ndescription: Require reading dev-guides before development and keeping docs in sync\nglobs:\nalwaysApply: true\n---\n\n# Dev Guides First\n\n## Before Starting Any Development\n\nBefore implementing or designing a new feature, module, or significant change, **always\nread the relevant documents in `docs/dev-guides/`** to understand existing conventions:\n\n| Guide | When to Read |\n|-------|-------------|\n| `docs/dev-guides/1-streaming-protocol.md` | Any work on streaming endpoints or NDJSON protocol |\n| `docs/dev-guides/2-log-sanitization.md` | Any work involving logging, credentials, external services, or DataLoaders |\n| `docs/dev-guides/3-data-loader-development.md` | Any work on ExternalDataLoader, DataConnector, or connector routes |\n| `docs/dev-guides/4-authentication-oidc-tokenstore.md` | Any work on OIDC, TokenStore, AUTH_MODE, or SSO flows |\n| `docs/dev-guides/6-i18n-language-injection.md` | Any work on Agent prompts, Agent routes, backend user-visible messages, or frontend i18n |\n| `docs/dev-guides/7-unified-error-handling.md` | Any work on API errors, frontend API calls, streaming error events, or error tests |\n| `docs/dev-guides/8-path-safety.md` | Any work on backend file access, downloads, Workspace paths, Agent tools, DataLoaders, or sandbox config |\n| `docs/dev-guides/10-agent-knowledge-reasoning-log.md` | Any work on Agent knowledge injection, KnowledgeStore, reasoning logs, or experience distillation |\n| `docs/dev-guides/11-catalog-metadata-sync.md` | Any work on catalog sync, catalog_cache, catalog_annotations, metadata merge, Agent catalog tools, or frontend catalog browsing |\n| `docs/dev-guides/12-sandbox-session.md` | Any work on sandbox execution, Agent tool-calling loops, explore/execute_python code execution, or namespace management |\n| `docs/dev-guides/13-unified-row-limits.md` | Any work on row limits, data loading size caps, frontendRowLimit, MAX_IMPORT_ROWS, or DataLoader size parameter |\n| `docs/dev-guides/14-model-capability-runtime-degradation.md` | Any work on LLM Client calls, Agent LLM invocations, model capability checks, reasoning_effort, or image/vision degradation |\n| `docs/dev-guides/15-dataframe-serialization.md` | Any work on DataFrame→JSON serialization, Agent result rows, DataLoader sample_rows, or table API responses |\n| `.cursor/rules/i18n-no-hardcoded-strings.mdc` | Any work adding or changing user-visible strings in `src/` |\n\nAlso check `.cursor/rules/` and `.cursor/skills/` for related coding conventions.\n\n## After Introducing New Conventions\n\nWhen your work establishes a new development pattern, convention, or architectural\ndecision, you MUST update documentation before considering the task complete:\n\n1. **Update existing dev-guide** — if your change extends an existing convention\n (e.g. new sensitive key type → update `2-log-sanitization.md`)\n\n2. **Create new dev-guide** — if your change introduces a new cross-cutting convention\n that future developers must follow. Place in `docs/dev-guides/` with the next number prefix.\n\n3. **Update existing SKILL** — if your change affects how an existing skill works\n (e.g. new error handling pattern → update `error-handling/SKILL.md`)\n\n4. **Create new SKILL** — if your change introduces a reusable development workflow\n that AI agents should follow. Place in `.cursor/skills//SKILL.md`.\n\n5. **Update existing Rule** — if your change adds constraints to an existing rule\n (e.g. new banned pattern → update the relevant `.mdc` file)\n\n6. **Create new Rule** — if your change introduces file-scoped coding conventions\n that should be enforced automatically. Place in `.cursor/rules/.mdc`.\n\n## Documentation Sync Checklist\n\nFor every PR that introduces new patterns:\n\n- [ ] Searched `docs/dev-guides/` for related existing docs\n- [ ] Updated or created dev-guide if introducing cross-cutting conventions\n- [ ] Updated or created `.cursor/skills/` if introducing reusable workflows\n- [ ] Updated or created `.cursor/rules/` if introducing file-scoped constraints\n- [ ] New dev-guide includes a \"New Module Checklist\" section (if applicable)\n\n## File Naming Conventions\n\n- **dev-guides**: `-.md` (e.g. `3-data-loader-development.md`)\n- **skills**: `.cursor/skills//SKILL.md`\n- **rules**: `.cursor/rules/.mdc`\n",".cursor/rules/error-response-safety.mdc":"---\ndescription: Prevent information exposure through exception messages in HTTP responses\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Error Response Safety\n\nNever return raw exception text (`str(e)`, `f\"...{e}\"`) directly in HTTP responses.\nPython exceptions may contain stack traces, file paths, database connection strings,\nAPI keys, or internal IP addresses — all of which are security risks (CWE-209).\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full error handling contract.\n\n## Unified Error System\n\nNew error paths should use the `AppError` / `error_handler` infrastructure from\n`errors.py` and `error_handler.py`. Legacy `message` / `error_message` bodies are\nhistorical formats; do not add compatibility branches for them in new code.\nAll application-controlled errors return **HTTP 200** with `status: \"error\"` in the body.\nOnly auth errors (401/403) and uncontrolled transport errors (404/413/500) use non-200.\n\n### Non-streaming routes\n\nRaise `AppError` for new code or when structured error codes are useful. The\nglobal error handler returns HTTP 200 + JSON error envelope:\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\n\n# Business error — raise with appropriate code and safe message\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# LLM / external API error — use classify_and_wrap_llm_error\nfrom data_formulator.error_handler import classify_and_wrap_llm_error\n\nexcept Exception as e:\n raise classify_and_wrap_llm_error(e) from e\n```\n\n### Streaming routes\n\nUse `stream_error_event()` to yield a unified NDJSON error line.\nValidation before the stream starts returns `200 application/json` (not NDJSON):\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\nexcept AppError as e:\n yield stream_error_event(e)\nexcept Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n### Real examples from migrated endpoints\n\n```python\n# /data-agent-streaming — standard NDJSON error event\nexcept Exception as e:\n logger.error(\"Error in data-agent-streaming\", exc_info=e)\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# /generate-report-chat — same pattern\nexcept Exception as e:\n logger.exception(\"generate-report-chat failed\")\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# Validation failure before stream starts — use 200 JSON, not NDJSON\nif not request.is_json:\n return stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\"))\n```\n\n### Database / workspace errors (tables.py)\n\nUse `classify_and_raise_db_error()` instead of the legacy `sanitize_db_error_message()`:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\nexcept Exception as e:\n classify_and_raise_db_error(e) # raises AppError with safe message\n```\n\n### Connector errors (data_connector.py)\n\nUse `classify_and_raise_connector_error()` instead of the legacy `_sanitize_error()`:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e) # raises AppError with safe message\n```\n\n### Legacy functions (deprecated)\n\nThese functions still exist as wrappers but should NOT be used in new code:\n- `sanitize_db_error_message()` → use `classify_and_raise_db_error()`\n- `_sanitize_error()` → use `classify_and_raise_connector_error()`\n- `safe_error_response()` / `sanitize_error_message()` → use `raise AppError(...)`\n- `classify_llm_error()` is used internally by `classify_and_wrap_llm_error()`\n\n## Rules\n\n1. **Application errors** — HTTP 200 + `status: \"error\"`; prefer `error: {...}` for new code and never expose exception details.\n2. **Auth errors** — `AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403 (transport layer interception).\n3. **Transport errors** (uncontrolled) — only `404` (no route), `413` (body limit), `500` (unhandled crash) may be non-200.\n4. **Streaming errors** — always use `stream_error_event()` for NDJSON error lines.\n5. **Logging** — always log the full exception server-side (`logger.warning` / `logger.error`\n with `exc_info=True` when needed). The client never needs the stack trace.\n6. **Debug detail** — `AppError.detail` is only included in responses when `app.debug is True`.\n\n## Bare except Policy\n\n- **Never** use naked `except:` — always specify at least `except Exception:`\n- Use specific exception types when possible (`except (ValueError, TypeError):`)\n- Always log caught exceptions: `logger.warning(...)` for degraded operations, `logger.debug(...)` for expected failures\n- Process cleanup code (`terminate`, `join`, `close`) is exempt from logging requirements\n\n## Common Mistakes\n\n```python\n# ❌ BAD — raw exception leaks internal details\nreturn jsonify({\"message\": str(e)}), 500\nreturn jsonify({\"error\": f\"Failed: {e}\"}), 502\nyield json.dumps({\"type\": \"error\", \"error\": str(e)}) + \"\\n\"\n\n# ❌ BAD — non-200 HTTP status code for application errors\nreturn jsonify({\"status\": \"error\", \"error_message\": \"...\"}), 400\n\n# ❌ BAD — naked except catches SystemExit/KeyboardInterrupt\nexcept:\n pass\n\n# ❌ BAD — silent swallow without logging\nexcept Exception:\n pass\n\n# ✅ GOOD — non-streaming (global handler returns HTTP 200)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# ✅ GOOD — streaming\nyield stream_error_event(AppError(ErrorCode.LLM_TIMEOUT, \"Timeout\", retry=True))\n\n# ✅ GOOD — graceful degradation with logging\nexcept Exception as e:\n logger.warning(\"Could not read schema for %s\", table_name, exc_info=e)\n\n# ✅ GOOD — specific exception types\nexcept (ValueError, TypeError):\n pass # safe fallback\n```\n",".cursor/rules/frontend-test-conventions.mdc":"---\r\ndescription: Frontend TypeScript test conventions (Vitest)\r\nglobs: tests/frontend/**/*.test.{ts,tsx}\r\nalwaysApply: false\r\n---\r\n\r\n# Frontend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/frontend/unit/` mirroring the `src/` structure:\r\n - `tests/frontend/unit/data/` → tests for `src/data/`\r\n - `tests/frontend/unit/app/` → tests for `src/app/`\r\n - `tests/frontend/unit/views/` → tests for `src/views/`\r\n- Name files `.test.ts` (or `.test.tsx` for React rendering tests).\r\n\r\n## File Structure\r\n\r\n```typescript\r\nimport { describe, it, expect } from 'vitest';\r\n// For React rendering tests:\r\n// import { render } from '@testing-library/react';\r\n\r\nimport { myFunction } from '../../../../src/';\r\n\r\ndescribe('myFunction', () => {\r\n it('should handle ', () => {\r\n expect(myFunction(input)).toBe(expected);\r\n });\r\n});\r\n```\r\n\r\n## Conventions\r\n\r\n- Import `describe`, `it`, `expect` explicitly from `vitest` (globals are enabled but explicit imports improve readability).\r\n- Use `@testing-library/react` and `@testing-library/jest-dom` for component rendering tests.\r\n- Prefer testing **exported pure functions** over testing internal component state.\r\n- When component logic is complex, extract it into an exported helper and test that directly.\r\n- Group tests with `describe` blocks; use section comments (`// --- Null cases ---`) for clarity.\r\n- One assertion per `it` block when possible; name tests as `should `.\r\n- Do **not** import from `node_modules` internals; only use public API.\r\n- Keep tests independent — no shared mutable state between `it` blocks.\r\n\r\n## Example\r\n\r\n```typescript\r\n// ❌ BAD – no describe, vague test name\r\nimport { expect, test } from 'vitest';\r\ntest('works', () => { expect(fn(1)).toBe(2); });\r\n\r\n// ✅ GOOD\r\nimport { describe, it, expect } from 'vitest';\r\nimport { checkIsLikelyTextOnlyModel } from '../../../../src/views/DataLoadingThread';\r\n\r\ndescribe('checkIsLikelyTextOnlyModel', () => {\r\n it('returns true for deepseek-chat', () => {\r\n expect(checkIsLikelyTextOnlyModel('deepseek-chat')).toBe(true);\r\n });\r\n\r\n it('returns false for undefined', () => {\r\n expect(checkIsLikelyTextOnlyModel(undefined)).toBe(false);\r\n });\r\n});\r\n```\r\n",".cursor/rules/i18n-no-hardcoded-strings.mdc":"---\r\ndescription: No hardcoded UI strings — use i18n translation keys\r\nglobs: src/**/*.{ts,tsx}\r\nalwaysApply: true\r\n---\r\n\r\n# i18n: No Hardcoded UI Strings\r\n\r\nAll user-visible text in the frontend MUST go through the i18n system. Never hardcode Chinese, English, or any other language string directly in components.\r\n\r\n## How to Use\r\n\r\n```tsx\r\nimport { useTranslation } from 'react-i18next';\r\n\r\nconst { t } = useTranslation();\r\n\r\n// ✅ GOOD\r\n\r\n\r\n\r\n// ❌ BAD\r\n\r\n\r\n\r\n```\r\n\r\n## Non-Component Contexts (thunks, utilities, helpers)\r\n\r\nWhen you cannot use hooks (Redux thunks, plain `.ts` utilities, store helpers), import the i18n instance directly:\r\n\r\n```ts\r\nimport i18n from '../i18n';\r\n\r\n// ✅ GOOD — works outside React components\r\ni18n.t('messages.rowLimitReached', { count: 20000 })\r\n\r\n// ❌ BAD — hooks only work inside React function components\r\nconst { t } = useTranslation(); // will crash in a thunk\r\n```\r\n\r\nThe same rule applies: every user-visible string must use `i18n.t()` with a translation key, never a hardcoded literal.\r\n\r\n## Translation Files\r\n\r\n- English: `src/i18n/locales/en/.json`\r\n- Chinese: `src/i18n/locales/zh/.json`\r\n- Namespaces: `common`, `upload`, `chart`, `model`, `encoding`, `messages`, `navigation`, `dataLoading`, `errors`\r\n\r\nWhen adding a new key, add it to **both** `en` and `zh` locale files. Pick the namespace that fits; create a new namespace only if none applies.\r\n\r\n## What Counts as User-Visible\r\n\r\nMust use `t()`: button labels, tooltips, placeholders, error messages, dialog titles, tab names, toast notifications, table headers, empty-state text.\r\n\r\nMay stay hardcoded: log messages (`console.log`), error messages thrown but never displayed, internal constants, CSS class names, test IDs.\r\n\r\n**Cross-stack sentinel values** — strings shared between frontend and backend as identity markers (e.g. `UNTITLED_SESSION` in `dfSlice.ts` used by both Redux state and backend workspace API) must NOT be translated. They are internal constants, not display text. Translate them only at the UI rendering layer by mapping the sentinel to `t('...')`.\r\n\r\n## Backend Messages with message_code\r\n\r\nWhen the backend returns a user-visible message with a `message_code` (or `content_code`, `error_code`), use `translateBackend()` from `src/app/utils.tsx` to translate it:\r\n\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\n\r\n// Translate a backend error message\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n\r\n// Translate options with parallel code arrays\r\nconst options = translateBackendOptions(rawOptions, event.option_codes);\r\n```\r\n\r\nThe translation keys live in `messages.json` under the `agent` section. Always add **both** `en` and `zh` entries.\r\n",".cursor/rules/implementation-review-checklist.mdc":"---\ndescription: Require structural code review after implementation before final response\nglobs:\nalwaysApply: true\n---\n\n# Implementation Review Checklist\n\nAfter any substantive implementation, before the final response, review the\nuncommitted diff from a maintainer/code-review perspective. Do not only verify\nthat the feature works.\n\n## Required Checks\n\n- Inspect `git diff --stat` and relevant diffs to understand the actual blast radius.\n- Check responsibility boundaries: frontend UI, API route, service/helper, storage/session, and tests should each own the right part of the behavior.\n- Check API semantics: endpoint names, HTTP methods, response shapes, and side effects must match the product action. Similar UI actions should call the same backend API.\n- Check frontend/backend state consistency: after mutations, make sure Redux/local state, backend session/token/vault state, and subsequent list/status APIs agree.\n- Check duplication: if two paths clear the same state or perform the same API flow, extract a small helper or route instead of copying logic.\n- Check regression surfaces: refresh/reload/list/status paths must respect new flags or state, not just the primary click path.\n- Check tests: add or update focused tests for the changed contract; include at least one test for the bug/regression path when practical.\n- Check docs/rules/dev-guides when the change introduces or changes a cross-cutting convention.\n\n## Reporting\n\nIf issues are found during this review, fix clear issues before finalizing. If a\ntrade-off remains, call it out explicitly in the final response with the risk and\nwhy it was left as-is.\n",".cursor/rules/incremental-development-cadence.mdc":"---\ndescription: Enforce step-by-step implementation with design-doc checks, tests, and i18n after each step\nglobs:\nalwaysApply: true\n---\n\n# Incremental Development Cadence\n\nMulti-step tasks MUST be implemented **one step at a time**. After each logical unit (function, endpoint, component), complete all cross-cutting concerns before moving on.\n\n## After Each Step\n\n1. **Verify against design doc** — compare with `design-docs/` or issue spec; raise deviations before continuing\n2. **Add tests** — write/update tests for the just-implemented code, run and confirm green\n3. **Add i18n keys** — if user-visible text was introduced, add `en` + `zh` keys immediately\n4. **Update docs** — if a new cross-cutting convention was introduced, update dev-guide/Rule immediately\n\n## Banned Patterns\n\n- ❌ Implementing 3+ functions/endpoints before writing any tests\n- ❌ Sweeping all i18n keys in a final pass after all code is done\n- ❌ Checking design docs only after the entire implementation is complete\n- ❌ Mentioning \"tests and i18n still needed\" only in the final response\n",".cursor/rules/language-injection-conventions.mdc":"---\r\ndescription: Language injection conventions for LLM Agent prompts\r\nglobs: py-src/data_formulator/agents/**/*.py,py-src/data_formulator/routes/agents.py\r\nalwaysApply: false\r\n---\r\n\r\n# Language Injection Conventions\r\n\r\nLanguage flows per-request: `Frontend i18n → Accept-Language header → get_language_instruction() → system prompt`.\r\n\r\n## LLM Prompt Language Injection\r\n\r\n1. **User-facing LLM output** MUST inject language via `get_language_instruction(mode=...)` in the route handler.\r\n2. **Mode selection:** `\"full\"` for text-heavy agents, `\"compact\"` for code-generation agents and short-text endpoints.\r\n3. **Inject into system prompt only** — use `inject_language_instruction()` from `agent_language.py`. Never inject into user messages.\r\n4. **Do NOT inject** for non-user-facing calls (health checks, internal tool calls).\r\n5. **Do NOT duplicate** — if upstream messages already contain language instruction, skip.\r\n6. **Do NOT** use env vars, global interceptors, or hardcoded language strings (e.g. `\"回答请使用中文\"`) — route handlers should use `get_language_instruction(mode=...)`.\r\n7. **New language?** Add to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py` and add locale files in `src/i18n/locales//`.\r\n\r\n## Python-Side User-Visible Messages (message_code pattern)\r\n\r\nFor **fixed strings in Python** that are shown in the UI (error messages, clarify options, completion summaries), do NOT translate in Python. Instead:\r\n\r\n1. Keep the English string as the default value.\r\n2. Add a `message_code` (or `content_code`, `error_code`, `summary_code`) field with a key like `\"agent.someKey\"`.\r\n3. Optionally add `message_params` for interpolation.\r\n4. The frontend translates using `translateBackend(fallback, code, params)` from `src/app/utils.tsx`.\r\n5. Add the translation key to `src/i18n/locales/{en,zh}/messages.json` under the `agent` section.\r\n\r\n```python\r\n# ✅ GOOD — backend returns code, frontend translates\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\",\r\n \"message_code\": \"agent.emptyDataframe\",\r\n}\r\n\r\n# ❌ BAD — backend-side translation dict\r\nyield {\"message\": translate_in_python(\"empty_df\", lang)}\r\n```\r\n\r\nFor detailed architecture and anti-pattern explanations, see `docs/dev-guides/6-i18n-language-injection.md`.\r\n",".cursor/rules/log-sanitization.mdc":"---\ndescription: Prevent sensitive data leakage in server-side log messages\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Log Sanitization\n\nServer-side logs must never contain passwords, tokens, API keys, or other credentials\nin plain text. This rule complements `error-response-safety.mdc` (which covers client\nresponses) by addressing the **logging** side.\n\n## Architecture: Defense in Depth\n\nTwo layers protect against sensitive data in logs:\n\n1. **Layer 1 — Explicit utilities** (call-site): developers use `sanitize_url()`,\n `sanitize_params()`, `redact_token()` from `security/log_sanitizer.py`.\n2. **Layer 2 — SensitiveDataFilter** (global): a `logging.Filter` registered on all\n handlers in `configure_logging()` auto-redacts patterns that slip through.\n\nLayer 1 is the primary defense; Layer 2 is a safety net. Both are required.\n\n## When to Use Each Utility\n\n| Data Type | Utility | Example |\n|-----------|---------|---------|\n| `dict` with password/secret/token keys | `sanitize_params(params)` | `log.info(\"Init: %s\", sanitize_params(params))` |\n| URL (may have `user:pass@host` or sensitive query params) | `sanitize_url(url)` | `logger.info(\"Connecting to %s\", sanitize_url(url))` |\n| Bearer / access token | `redact_token(token)` | `logger.debug(\"Token: %s\", redact_token(token))` |\n| Normal string (no secrets) | Nothing needed | `logger.info(\"Processing table %s\", name)` |\n\n## Rules\n\n1. **Never log raw `params` / `config` dicts** that may contain `password`, `secret`,\n `api_key`, `token`, or `connection_string` keys. Always use `sanitize_params()`.\n\n2. **Never log full tokens or API keys.** Use `redact_token()` if you need to identify\n a token in logs.\n\n3. **Prefer `type(exc).__name__`** over `str(exc)` in warning/info logs when the\n exception may contain connection strings or upstream response bodies. Use\n `exc_info=True` for full tracebacks at ERROR level (the filter sanitizes `exc_text`).\n\n4. **URL logging** — use `sanitize_url()` for any URL from configuration or\n environment variables (issuer URLs, JWKS URLs, database URLs, API base URLs).\n It masks embedded credentials and sensitive query parameters such as\n `password`, `client_secret`, `token`, and `api_key`.\n\n5. **Use `%s`-style formatting** instead of f-strings for log messages. This allows\n the filter to intercept arguments before formatting and provides better\n compatibility with log aggregation tools.\n\n## Common Mistakes\n\n```python\n# BAD — dict may contain password\nlog.info(f\"Connecting with {params}\")\nlog.info(\"Connecting with %s\", params)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_params\nlog.info(\"Connecting with %s\", sanitize_params(params))\n\n# BAD — URL may embed credentials\nlogger.warning(\"Failed for %s\", discovery_url)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_url\nlogger.warning(\"Failed for %s\", sanitize_url(discovery_url))\n\n# BAD — exception may leak connection string or upstream body\nlogger.warning(\"Error: %s\", exc)\n\n# GOOD — log type name at warning level, full traceback at error level\nlogger.warning(\"Error: %s\", type(exc).__name__)\nlogger.error(\"Unexpected failure\", exc_info=True)\n\n# BAD — full token in logs\nlogger.debug(\"Using token %s\", token)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import redact_token\nlogger.debug(\"Using token %s\", redact_token(token))\n```\n\n## Disabling for Local Debug\n\nSet `LOG_SANITIZE=false` to disable the global filter during local debugging.\nThis should **never** be used in production or CI.\n\n## Adding New Sensitive Keys\n\nIf your module introduces a new credential key name (e.g. `custom_auth_token`),\nadd it to `SENSITIVE_KEYS` in `security/log_sanitizer.py` and add a corresponding\nregex pattern to `_SENSITIVE_KEY_NAMES` if needed.\n\n## New Module Checklist\n\nWhen creating a new module or feature that handles credentials or external services:\n\n- [ ] Audit all `logger.*()` calls — do any log connection params, URLs, or tokens?\n- [ ] Use `sanitize_params()` for any dict that may contain credentials\n- [ ] Use `sanitize_url()` for any URL from config/env\n- [ ] Use `redact_token()` for any token/key value\n- [ ] If introducing new credential key names, update `SENSITIVE_KEYS`\n- [ ] Run `python -m pytest tests/backend/security/test_log_sanitizer.py` to verify\n",".cursor/rules/package-manager-conventions.mdc":"---\r\ndescription: Use Yarn only, never npm/pnpm\r\nglobs: package.json, yarn.lock\r\nalwaysApply: true\r\n---\r\n\r\n# Package Manager Rules\r\n\r\n- Use Yarn v1.22.22 only - never use npm or pnpm\r\n- Never manually edit yarn.lock\r\n- Keep yarn.lock changes minimal when adding deps\r\n- Registry must be https://registry.yarnpkg.com\r\n",".cursor/rules/path-safety.mdc":"---\r\ndescription: 路径安全编码规范 — 编辑路由/Agent/Loader/Workspace/知识库代码时自动提醒\r\nglobs: py-src/data_formulator/{routes,agents,data_loader,datalake,security,knowledge}/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# 路径安全编码规范\r\n\r\n编辑此目录下的文件时，请遵守以下安全规则（详见 `docs/dev-guides/8-path-safety.md` 和 `.cursor/skills/path-safety/SKILL.md`）：\r\n\r\n## ConfinedDir 是唯一的路径约束原语\r\n\r\n所有接受用户可控路径的代码，**必须**通过 `ConfinedDir` 实例访问文件系统。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\njail = ConfinedDir(root_dir, mkdir=False)\r\ntry:\r\n target = jail.resolve(user_input)\r\nexcept ValueError:\r\n return {\"error\": \"Access denied\"}\r\n```\r\n\r\n## 禁止的模式\r\n\r\n```python\r\n# ❌ BAD — 裸路径拼接\r\ntarget = Path(root) / user_input\r\n\r\n# ❌ BAD — 手写 resolve + relative_to（已弃用，必须用 ConfinedDir）\r\ntarget = (root / user_input).resolve()\r\ntry:\r\n target.relative_to(root)\r\nexcept ValueError: ...\r\n\r\n# ❌ BAD — 手写 is_relative_to（已弃用，必须用 ConfinedDir）\r\nresolved = (root / user_input).resolve()\r\nif not resolved.is_relative_to(root.resolve()): ...\r\n\r\n# ❌ BAD — 路径比较用 str.startswith\r\nif str(target).startswith(str(root)): ...\r\n```\r\n\r\n## 正确的模式\r\n\r\n```python\r\n# ✅ GOOD — Agent 工具\r\nworkspace_jail = self.workspace.confined_root\r\nscratch_jail = self.workspace.confined_scratch\r\ntarget = workspace_jail.resolve(rel_path)\r\n\r\n# ✅ GOOD — 文件下载 route\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target)\r\n\r\n# ✅ GOOD — 文件上传（两层防御）\r\nsafe_name = secure_filename(raw_filename)\r\ntarget = scratch_jail.resolve(safe_name)\r\n\r\n# ✅ GOOD — Workspace data 文件\r\npath = workspace.get_file_path(filename) # 内部使用 ConfinedDir\r\n```\r\n\r\n## 部署守卫\r\n\r\n- 新增读取宿主文件系统的 Loader 必须在 `_enforce_deployment_restrictions()` 注册多用户禁用\r\n- 多用户模式（`WORKSPACE_BACKEND != \"local\"`）必须启用沙箱\r\n",".cursor/rules/test-driven-workflow.mdc":"---\r\ndescription: 测试优先开发与测试保护规则。确保新功能/Bug 修复必须有测试覆盖，测试失败时先诊断后修改。\r\nglobs: \r\nalwaysApply: true\r\n---\r\n\r\n# Test-Driven Workflow\r\n\r\n## 1. 测试优先：新功能 / Bug 修复必须有测试\r\n\r\n开发新功能或修复 Bug 时，**必须同步编写测试**，不能只提交业务代码：\r\n\r\n- **新功能** → 先写测试（或至少同步写），描述期望行为，再实现代码使测试通过\r\n- **Bug 修复** → 先写复现测试，确认测试失败，再修复代码使测试通过\r\n- **重构** → 确保重构前已有测试覆盖，重构后测试仍然全部通过\r\n\r\n完成实现后，主动运行相关测试验证：\r\n- 后端：`python -m pytest tests/backend/ -q`\r\n- 前端：`yarn test`\r\n\r\n## 2. 测试保护：不要轻易修改测试代码\r\n\r\n测试失败时，**禁止直接修改测试代码来让测试通过**。必须执行以下流程：\r\n\r\n1. **复现并定位**：运行失败的测试，确认失败现象\r\n2. **诊断分类**：\r\n - A：测试本身有问题（断言错误、setup 不正确）\r\n - B：业务实现有 Bug\r\n - C：需求/规格变更导致测试过时\r\n3. **给出方案**：至少提供两个可选方案（改测试 / 改实现 / 补规格），说明各自影响和风险\r\n4. **等待确认**：由用户决定采用哪个方案，再执行修改\r\n\r\n**绝对禁止**：\r\n- 测试一红就直接改测试代码\r\n- 删除或跳过（`@pytest.mark.skip`）失败的测试来\"解决\"问题\r\n- 修改测试的断言值来匹配错误的实现\r\n\r\n## 3. 精准修改：不动无关测试\r\n\r\n与 Karpathy 编码准则的\"精准修改\"一致：\r\n\r\n- 不\"顺手改进\"与当前任务无关的测试代码\r\n- 不重构正在通过的测试\r\n- 不改变已有测试的断言逻辑，除非被明确要求\r\n",".cursor/rules/unified-error-protocol.mdc":"---\ndescription: Unified error handling protocol for backend and frontend\nglobs: py-src/**/*.py,src/**/*.{ts,tsx}\nalwaysApply: false\n---\n\n# Unified Error Protocol\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full developer guide.\n\n## HTTP Status Code Policy\n\n**All application-controlled errors return HTTP 200** with `status: \"error\"` in the body.\nThis applies to both non-streaming JSON APIs and streaming pre-flight errors.\n\nOnly these scenarios use non-200:\n- `401` — `AUTH_REQUIRED` / `AUTH_EXPIRED` (so transport layer can intercept)\n- `403` — `ACCESS_DENIED` (so transport layer can intercept)\n- `404` — Flask routing: no matching route handler\n- `413` — WSGI: request body exceeds `MAX_CONTENT_LENGTH`\n- `500` — Unhandled exception that escaped all error handling\n\n**Streaming NDJSON APIs:**\n- Pre-stream validation failure: HTTP `200` + `application/json` (use `stream_preflight_error()`)\n- In-stream error: NDJSON `{\"type\": \"error\", \"error\": {...}}`\n- In-stream warning: NDJSON `{\"type\": \"warning\", \"warning\": {...}}`\n\n## Backend: Non-streaming JSON Format\n\n```jsonc\n// Success (HTTP 200)\n{\"status\": \"success\", \"data\": {...}}\n\n// Error (HTTP 200 for business errors, 401/403 for auth errors)\n{\n \"status\": \"error\",\n \"error\": {\n \"code\": \"TABLE_NOT_FOUND\",\n \"message\": \"Table not found\",\n \"retry\": false,\n \"detail\": \"...\" // Only in debug mode\n }\n}\n```\n\n## Backend: Streaming NDJSON Format\n\nAll streaming endpoints use `application/x-ndjson`. Each line is one JSON object:\n\n```jsonc\n{\"type\": \"text_delta\", \"data\": {...}}\n{\"type\": \"error\", \"error\": {\"code\": \"LLM_TIMEOUT\", \"message\": \"...\", \"retry\": true}}\n{\"type\": \"done\", \"data\": {...}}\n```\n\n## Backend: How to Handle Errors\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok, stream_preflight_error\nfrom data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n\n# Non-streaming: success\nreturn json_ok(data)\n\n# Non-streaming: error (global handler returns HTTP 200, auth errors get 401/403)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# Streaming: pre-flight error (always HTTP 200)\nreturn stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Bad input\"))\n\n# Streaming: in-stream error\nyield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n## Frontend: How to Consume APIs\n\n```typescript\nimport { apiRequest, streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\n// Non-streaming — apiRequest checks body.status (main path) and HTTP status (auth/infra)\ntry {\n const { data } = await apiRequest(url, options);\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n\n// Streaming — streamRequest handles pre-flight JSON errors and NDJSON events\ntry {\n for await (const event of streamRequest(url, options, signal)) {\n if (event.type === 'error') { /* handle inline */ }\n // ...process events...\n }\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n```\n\nAPI consumers MUST use `apiRequest()` / `streamRequest()` plus `handleApiError()`.\nDirect `fetchWithIdentity()` is reserved for lower-level clients and explicit protocol\nexceptions such as file downloads, blob/CSV responses, SPA/OIDC redirects, or third-party URLs.\n\n## Prohibited Patterns\n\n### Backend\n- ❌ `return jsonify({\"status\": \"ok\", \"data\": data})` — use `json_ok(data)` instead\n- ❌ `return jsonify({\"error_message\": str(e)})` — use `raise AppError(...)` instead\n- ❌ `yield json.dumps({\"type\": \"error\", \"error\": str(e)})` — use `stream_error_event()`\n- ❌ Bare `except:` or `except Exception: pass` — always log or propagate\n- ❌ `yield 'error: ' + json.dumps(...)` — non-standard prefix\n\n### Frontend\n- ❌ `.catch(() => {})` — either add a comment explaining why (best-effort), or use `handleApiError`\n- ❌ `.catch(console.error)` without user feedback — dispatch `addMessages` or use `handleApiError`\n- ❌ Raw `fetch()` for `/api/` URLs — use `fetchWithIdentity` (identity headers + 401 retry)\n- ❌ RTK thunk without `.rejected` handler — always add `.addCase(thunk.rejected, ...)` with `addMessages`\n- ❌ New code using `fetchWithIdentity().json()` — use `apiRequest()` instead\n\n### RTK Thunk Rejected Handler Pattern\n\n```typescript\n.addCase(myThunk.rejected, (state, action) => {\n if (action.error?.name !== 'AbortError') {\n state.messages.push({\n timestamp: Date.now(), type: 'warning',\n component: 'my-component',\n value: 'Human-readable error description',\n });\n }\n})\n```\n\n## Migrated Streaming Endpoints\n\nAll streaming endpoints now use `application/x-ndjson` and `stream_error_event()`:\n\n| Endpoint | Old format | New format | Status |\n|----|-----|---|-----|\n| `/data-agent-streaming` | `{status:\"error\", error_message}` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/get-recommendation-questions` | `error: {json}` prefix + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/generate-report-chat` | `data: {json}` SSE prefix + `text/event-stream` | Pure NDJSON + `application/x-ndjson` | ✅ Done |\n| `/data-loading-chat` | `str(e)` in error field | `stream_error_event()` safe message | ✅ Done |\n| `/clean-data-stream` | `\\n{json}\\n` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n\n## Complete ErrorCode Reference\n\n```\nAUTH_REQUIRED, AUTH_EXPIRED, ACCESS_DENIED\nINVALID_REQUEST, TABLE_NOT_FOUND, FILE_PARSE_ERROR, FILE_TOO_LARGE, VALIDATION_ERROR\nLLM_AUTH_FAILED, LLM_RATE_LIMIT, LLM_CONTEXT_TOO_LONG, LLM_MODEL_NOT_FOUND\nLLM_TIMEOUT, LLM_SERVICE_ERROR, LLM_CONTENT_FILTERED, LLM_UNKNOWN_ERROR\nDB_CONNECTION_FAILED, DB_QUERY_ERROR, DATA_LOAD_ERROR, CONNECTOR_ERROR\nCODE_EXECUTION_ERROR, AGENT_ERROR\nCATALOG_SYNC_TIMEOUT, CATALOG_NOT_FOUND, ANNOTATION_CONFLICT, ANNOTATION_INVALID_PATCH\nINTERNAL_ERROR, SERVICE_UNAVAILABLE\n```\n\n## Adding a New Error Code\n\n1. Add constant to `py-src/data_formulator/errors.py` `ErrorCode` class (no HTTP mapping needed, defaults to 200)\n2. Add i18n mapping to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`\n3. Add translations to `src/i18n/locales/en/errors.json` AND `zh/errors.json`\n\n## New Endpoint Checklist\n\nWhen adding a new API endpoint, verify:\n- [ ] Success: `json_ok(data)` → `{\"status\": \"success\", \"data\": ...}`, HTTP 200\n- [ ] Error: `raise AppError(...)` → HTTP 200 + error body (auth errors get 401/403)\n- [ ] Streaming pre-flight: `stream_preflight_error(...)` → HTTP 200 + `application/json`\n- [ ] Streaming in-stream: `stream_error_event(...)` via NDJSON\n- [ ] No `str(e)` in any response body\n- [ ] No naked `except:` — use specific exception types or `except Exception as e:` with logging\n- [ ] Frontend uses `apiRequest()` (non-streaming) or `streamRequest()` (streaming)\n- [ ] Error test cases assert HTTP 200 + `body.status == \"error\"` (auth: 401/403)\n",".cursor/skills/error-handling/SKILL.md":"---\nname: error-handling\ndescription: 统一错误处理系统。在添加 API 端点、修改错误处理、添加前端 API 调用、编写错误相关测试时使用。\n---\n\n# Error Handling Skill\n\nUnified error handling system for DF. Use when adding API endpoints, modifying error handling, or adding frontend API calls.\n\n> **Prerequisites**: Read `docs/dev-guides/7-unified-error-handling.md` before changing API error behavior.\n> Read `docs/dev-guides/2-log-sanitization.md` when the work involves logging, credentials, external services, or DataLoaders.\n> If your work introduces new error handling patterns or conventions, update this file and related dev-guides accordingly.\n\n## Architecture Overview\n\n```\nFrontend Backend\n──────── ───────\napiClient.ts errors.py\n├── apiRequest() ←── JSON ──── ├── ErrorCode (enum)\n├── streamRequest() ←── NDJSON ── └── AppError (exception)\n└── parseStreamLine()\n error_handler.py\nerrorCodes.ts ├── register_error_handlers(app)\n└── getErrorMessage() ├── classify_and_wrap_llm_error()\n └── stream_error_event()\nerrorHandler.ts\n└── handleApiError() security/sanitize.py\n └── classify_llm_error() (internal)\nMessageSnackbar ← dfSlice.messages\n```\n\n## Protocol Snapshot\n\nUse this contract for all new or reworked DF APIs:\n\n| Scenario | HTTP | Shape |\n|----------|------|-------|\n| Non-streaming success | `200` | `{\"status\": \"success\", \"data\": ...}` |\n| Non-streaming business/validation error | `200` | `{\"status\": \"error\", \"error\": {\"code\", \"message\", \"retry\", \"request_id\"}}` |\n| Non-streaming auth/authorization error | `401` / `403` | same structured error body |\n| Streaming preflight error | `200` | `application/json` + `{\"status\": \"error\", \"error\": ...}` |\n| Streaming in-flight fatal error | `200` | NDJSON line: `{\"type\": \"error\", \"error\": ...}` |\n| No Flask route / too large / unhandled crash | `404` / `413` / `500` | transport-level error |\n\nDo not use HTTP `400`/`422` for application validation errors in new code.\nDo not convert in-flight NDJSON errors to `status: \"error\"`; once the stream has\nstarted, event `type` is the protocol discriminator.\n\n## Backend: Adding a New API Endpoint\n\n### HTTP Status Code Policy\n\n**Application-controlled business and validation errors return HTTP 200** with\n`status: \"error\"` in the body. Only these use non-200:\n- `401`/`403` — auth errors (`AUTH_REQUIRED`, `AUTH_EXPIRED`, `ACCESS_DENIED`)\n- `404` — no matching Flask route\n- `413` — WSGI body limit exceeded\n- `500` — unhandled exception (program bug)\n\n### Non-streaming endpoint\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok\n\n@bp.route('/my-endpoint', methods=['POST'])\ndef my_endpoint():\n content = request.get_json()\n if not content.get('required_field'):\n raise AppError(ErrorCode.INVALID_REQUEST, \"Missing required_field\")\n\n try:\n result = do_work(content)\n except SomeBusinessError as e:\n raise AppError(ErrorCode.DATA_LOAD_ERROR, \"Failed to load data\") from e\n except Exception as e:\n from data_formulator.error_handler import classify_and_wrap_llm_error\n raise classify_and_wrap_llm_error(e) from e\n\n return json_ok(result)\n# Global handler returns: HTTP 200 + {\"status\": \"error\", \"error\": {code, message, retry}}\n# Auth errors (AUTH_REQUIRED/AUTH_EXPIRED/ACCESS_DENIED) return 401/403\n```\n\nLegacy `{\"status\": \"error\", \"message\": \"...\"}`, `error_message`, bare `{error}`,\nand `status: \"ok\"` responses are historical formats. Do not add new compatibility\nbranches for them; migrate the route to `json_ok()` / `AppError` before using\n`apiRequest()`.\n\n### Streaming endpoint\n\nValidation MUST be outside the generator. Failures return 200 JSON (not NDJSON).\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\n@bp.route('/my-stream', methods=['POST'])\ndef my_stream():\n if not request.is_json:\n return stream_preflight_error(\n AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\")\n )\n\n content = request.get_json()\n client = get_client(content['model'])\n\n def generate():\n try:\n for event in agent.run(...):\n yield json.dumps(event, ensure_ascii=False) + \"\\n\"\n except Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n return Response(stream_with_context(generate()), mimetype='application/x-ndjson')\n```\n\nStreaming runtime errors intentionally use `{\"type\": \"error\", \"error\": ...}`.\nThey cannot use a top-level `status` envelope because the HTTP response and NDJSON\nevent stream have already started.\n\n## Frontend: Consuming an API\n\n### Non-streaming\n\n```typescript\nimport { apiRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n const { data } = await apiRequest(getUrls().MY_ENDPOINT, {\n method: 'POST',\n body: JSON.stringify(payload),\n headers: { 'Content-Type': 'application/json' },\n });\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\nFor UI loading state, model the request lifecycle explicitly with\n`LoadableState` from `src/app/loadableState.ts`. Do not infer loading from\nmissing data (`!data`), because failed requests may legitimately leave data\nempty while loading has ended.\n\n### Streaming\n\n```typescript\nimport { streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n for await (const event of streamRequest(url, options, abortController.signal)) {\n switch (event.type) {\n case 'text_delta':\n break;\n case 'error':\n // Error arrived mid-stream — show inline in component.\n break;\n case 'done':\n break;\n }\n }\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\n### With callbacks\n\n```typescript\nhandleApiError(e, 'MyComponent', {\n onAuth: () => redirectToLogin(), // AUTH_REQUIRED / AUTH_EXPIRED\n onRetryable: () => retryOperation(), // LLM_RATE_LIMIT / LLM_TIMEOUT\n silent: true, // don't show Snackbar (component handles display)\n});\n```\n\n### Migration and special cases\n\nDF API consumers should use `apiRequest()` / `streamRequest()` and\n`handleApiError()`. Direct `fetchWithIdentity()` is for lower-level client helpers\nand explicit protocol exceptions such as file downloads, blob/CSV responses,\nOIDC redirects, SPA fallback, or third-party URLs.\n\nDo not apply the normal JSON API protocol mechanically to file downloads / CSV\nstreaming, SPA fallback, OIDC redirect flows, frontend fetches to third-party\nURLs, or errors after a streaming response has already started. Check the route's\nprotocol first, then preserve safe error bodies and avoid `str(exc)` exposure.\n\n## Adding a New Error Code\n\n1. **Backend** — Add to `py-src/data_formulator/errors.py` `ErrorCode`:\n ```python\n MY_NEW_ERROR = \"MY_NEW_ERROR\"\n ```\n No HTTP mapping needed — defaults to HTTP 200. Only add to `ERROR_CODE_HTTP_STATUS` if it's an auth code.\n\n2. **Frontend mapping** — Add to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`:\n ```typescript\n MY_NEW_ERROR: 'errors.myNewError',\n ```\n\n3. **Translations** — Add to both locale files:\n - `src/i18n/locales/en/errors.json`: `\"myNewError\": \"English message\"`\n - `src/i18n/locales/zh/errors.json`: `\"myNewError\": \"中文消息\"`\n\n## Migrated Endpoints Reference\n\nAll streaming endpoints are now on the unified protocol:\n\n| Endpoint | Format | Notes |\n|----------|--------|-------|\n| `/data-agent-streaming` | NDJSON + `stream_error_event()` | Emits top-level `type` events; errors use `{type:\"error\", error:{...}}` |\n| `/get-recommendation-questions` | NDJSON + `stream_error_event()` | Was `error: {json}` prefix |\n| `/generate-report-chat` | Pure NDJSON + `stream_error_event()` | Was SSE `data: {json}` prefix |\n| `/data-loading-chat` | NDJSON + `stream_error_event()` | `str(e)` removed |\n| `/clean-data-stream` | NDJSON + `stream_error_event()` | Was `\\n{json}\\n` format |\n\nNon-streaming endpoints:\n\n| Endpoint | Error Format | Notes |\n|----------|-------------|-------|\n| `/chart-insight` | `AppError` → HTTP 200 + `{status:\"error\", error:{code,message,retry}}` | Fully migrated. Frontend uses `fetchChartInsight` rejected reducer. |\n| All migrated endpoints | `AppError` → HTTP 200 + unified error body | credentials, knowledge, sessions, tables, agents |\n| `/derive-data`, `/refine-data`, `/sort-data`, `/process-data-on-load`, `/test-model` | `json_ok()` / `AppError` | Migrated to new format |\n\n## Empty Catch Policy\n\nNot all `.catch(() => {})` are bugs. Use this decision tree:\n\n1. **User-initiated action** (delete, refresh, submit) → must notify with `addMessages` or `handleApiError()`\n2. **Background/best-effort fetch** (connector list on mount, session list) → OK to swallow, but add a comment\n3. **RTK thunks** → always add `.rejected` handler with `addMessages`\n4. **AbortError** → filter out with `if (action.error?.name !== 'AbortError')`\n\n## Frontend Stream Parsing Pattern\n\nWhen consuming a migrated streaming endpoint, handle the current NDJSON event\nformat directly:\n\n```typescript\nconst data = JSON.parse(line);\nif (data.type === 'error') {\n const errMsg = data.error?.message || 'Unknown error';\n // show to user...\n}\n```\n\n## Backend: Database/Workspace Errors (tables.py)\n\nFor table CRUD endpoints, use the specialized classifier:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\n@tables_bp.route('/my-table-op', methods=['POST'])\ndef my_table_op():\n try:\n result = workspace.do_something()\n return jsonify({\"status\": \"success\", \"data\": result})\n except Exception as e:\n classify_and_raise_db_error(e)\n```\n\n`classify_and_raise_db_error` maps common DB errors to appropriate `AppError` codes\n(returned as HTTP 200 by the global handler, except ACCESS_DENIED → 403):\n- \"Table does not exist\" → `TABLE_NOT_FOUND` (HTTP 200)\n- \"Table already exists\" → `INVALID_REQUEST` (HTTP 200)\n- \"Permission denied\" → `ACCESS_DENIED` (HTTP 403)\n- Other → `CONNECTOR_ERROR` (HTTP 200)\n\n## Backend: Connector Errors (data_connector.py)\n\nFor connector endpoints, use:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e, operation=\"preview\")\n```\n\nConnector/DataLoader classification is intentionally simple and lives in\n`data_formulator.data_loader.connector_errors`. It maps common failures to a\nsmall stable set: `INVALID_REQUEST`, `CONNECTOR_AUTH_FAILED`, `AUTH_EXPIRED`,\n`ACCESS_DENIED`, `DB_CONNECTION_FAILED`, `DB_QUERY_ERROR`, `DATA_LOAD_ERROR`,\nor `CONNECTOR_ERROR`. Do not add endpoint-local string matching unless the\nclassifier cannot reasonably cover the category.\n\nAll JSON errors include `error.request_id` and an `X-Request-Id` response\nheader. Show/copy this ID for users when reporting backend failures; do not\nshow raw exception text in production. Unhandled 500 responses must never\ninclude raw tracebacks, even in debug mode; return a safe category plus\n`request_id` and keep full stack traces in server logs only.\n\n## Debugging Error Propagation\n\nWhen an error isn't reaching the frontend:\n\n1. **Check backend logs** — is the error logged?\n2. **Check response format**:\n - Non-streaming: `{\"status\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n - Streaming: one line `{\"type\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n3. **Check Content-Type** — streaming must be `application/x-ndjson`, not `application/json` or `text/event-stream`\n4. **Check frontend parser** — is the consumer looking for `data.type === 'error'`?\n5. **Check global handler** — verify `register_error_handlers(app)` is called in `app.py`\n6. **Check blueprint handlers** — blueprint-level `errorhandler(Exception)` takes priority over global handlers\n\nLegacy `message` / `error_message` bodies are protocol violations on migrated API paths.\n\n## Log Sanitization (Sensitive Data in Server Logs)\n\nServer-side logs must never leak passwords, tokens, API keys, or connection strings.\nThe project uses a defense-in-depth approach with two layers.\n\n### Layer 1: Explicit Utilities (call-site)\n\n```python\nfrom data_formulator.security.log_sanitizer import (\n sanitize_url, sanitize_params, redact_token,\n)\n\n# Dict with credentials → sanitize_params()\nlog.info(\"Connecting with: %s\", sanitize_params(params))\n\n# URL that may embed credentials → sanitize_url()\nlogger.info(\"Issuer: %s\", sanitize_url(issuer_url))\n\n# Token/API key → redact_token()\nlogger.debug(\"Token: %s\", redact_token(token))\n```\n\n### Layer 2: SensitiveDataFilter (global safety net)\n\nRegistered in `app.py:configure_logging()`. Automatically redacts:\n- URL credentials (`://user:pass@host`)\n- `Bearer` tokens\n- `password=xxx`, `api_key=xxx`, `secret=xxx` patterns\n- JWT-like base64 strings\n- Python dict repr with sensitive keys\n\nDisable with `LOG_SANITIZE=false` for local debugging only.\n\n### When to Use What\n\n| Data | Utility | Why not just filter? |\n|------|---------|---------------------|\n| `dict` with password keys | `sanitize_params()` | Filter can't identify arbitrary password values in dict repr |\n| URL from config/env | `sanitize_url()` | Explicit is clearer; filter is backup |\n| Token/key value | `redact_token()` | Explicit is clearer; filter is backup |\n| Normal text | Nothing | Filter handles edge cases |\n\n### New Module Checklist\n\nWhen adding a module that handles credentials or external services:\n\n1. Audit all `logger.*()` calls for credential/URL/token logging\n2. Use `sanitize_params()` for dicts, `sanitize_url()` for URLs, `redact_token()` for tokens\n3. Prefer `type(exc).__name__` over `str(exc)` in warning-level logs\n4. If introducing new credential key names, add to `SENSITIVE_KEYS` in `log_sanitizer.py`\n\n## Key Files\n\n| File | Purpose |\n|------|---------|\n| `py-src/data_formulator/errors.py` | `ErrorCode` enum + `AppError` exception |\n| `py-src/data_formulator/error_handler.py` | Global handlers, `classify_and_wrap_llm_error`, `stream_error_event` |\n| `py-src/data_formulator/security/log_sanitizer.py` | `sanitize_url`, `sanitize_params`, `redact_token`, `SensitiveDataFilter` |\n| `py-src/data_formulator/routes/tables.py` | `classify_and_raise_db_error` (database/workspace errors) |\n| `py-src/data_formulator/data_connector.py` | `classify_and_raise_connector_error` (connector errors) |\n| `py-src/data_formulator/security/sanitize.py` | `classify_llm_error` (internal), `sanitize_error_message` |\n| `src/app/apiClient.ts` | `apiRequest`, `streamRequest`, `parseStreamLine`, `ApiRequestError` |\n| `src/app/errorHandler.ts` | `handleApiError` |\n| `src/app/errorCodes.ts` | `ERROR_CODE_I18N_MAP`, `getErrorMessage` |\n| `src/i18n/locales/{en,zh}/errors.json` | Error message translations |\n",".cursor/skills/language-injection/SKILL.md":"---\r\nname: language-injection\r\ndescription: LLM Agent 多语言注入规范。在修改 Agent 提示词、添加新的 Agent 端点、处理用户可见的后端消息（message_code）时使用。\r\n---\r\n\r\n# Language Injection for Agent Prompts\r\n\r\nAuthoritative developer guide: `docs/dev-guides/6-i18n-language-injection.md`.\r\n\r\n> **Prerequisites**: Read `docs/dev-guides/6-i18n-language-injection.md` before changing Agent prompts, Agent routes, backend user-visible messages, or frontend i18n strings.\r\n> If your work introduces new language injection patterns or conventions, update this file and related dev-guides accordingly.\r\n\r\n## Architecture\r\n\r\n```\r\nFrontend i18n.language → Accept-Language header → get_language_instruction()\r\n │\r\n build_language_instruction()\r\n (agents/agent_language.py)\r\n │\r\n ┌────────────┴────────────┐\r\n ▼ ▼\r\n mode=\"full\" mode=\"compact\"\r\n (text-heavy agents) (code-gen agents)\r\n```\r\n\r\n### Core Modules\r\n\r\n| Module | Role |\r\n|--------|------|\r\n| `agents/agent_language.py` | `build_language_instruction(lang, mode)` — generates prompt fragments; `inject_language_instruction()` — injects into system prompts; supports 20 languages; returns `\"\"` for English |\r\n| `routes/agents.py` → `get_language_instruction()` | Reads `Accept-Language` header, delegates to `build_language_instruction` |\r\n| `routes/agents.py` → `_get_ui_lang()` | Extracts primary language code from `Accept-Language` header |\r\n| `src/app/utils.tsx` → `fetchWithIdentity()` | Sets `Accept-Language` header on every API request from `i18n.language` |\r\n| `src/app/utils.tsx` → `translateBackend()` | Translates backend `message_code` / `content_code` using frontend i18n |\r\n\r\n## Code Examples\r\n\r\n### Route handler — inject language\r\n\r\n```python\r\n# In a Flask route handler:\r\nlang_instruction = get_language_instruction(mode=\"compact\")\r\nlang_suffix = f\"\\n\\n{lang_instruction}\" if lang_instruction else \"\"\r\n\r\nmessages = [\r\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\" + lang_suffix},\r\n {\"role\": \"user\", \"content\": user_input},\r\n]\r\n```\r\n\r\n### Agent constructor — use inject_language_instruction()\r\n\r\n```python\r\nfrom data_formulator.agents.agent_language import inject_language_instruction\r\n\r\n# Simple append (most agents)\r\nsystem_prompt = inject_language_instruction(system_prompt, language_instruction)\r\n\r\n# Insert before a marker (complex prompts)\r\nsystem_prompt = inject_language_instruction(\r\n system_prompt, language_instruction,\r\n marker=\"**About the execution environment:**\"\r\n)\r\n```\r\n\r\n### Python-side user-visible messages — message_code pattern\r\n\r\nFor fixed strings in Python that appear in the UI, do NOT translate in Python.\r\nReturn a `message_code` and let the frontend translate:\r\n\r\n```python\r\n# In an Agent or route handler:\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\", # English fallback\r\n \"message_code\": \"agent.emptyDataframe\", # frontend i18n key\r\n}\r\n\r\n# With parameters:\r\nresult = {\r\n \"status\": \"error\",\r\n \"content\": f\"Fields not found: {missing}\",\r\n \"content_code\": \"agent.fieldsNotFound\",\r\n \"content_params\": {\"missing\": missing, \"available\": available},\r\n}\r\n```\r\n\r\nFrontend consumption:\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n```\r\n\r\nTranslation keys go in `src/i18n/locales/{en,zh}/messages.json` under `messages.agent.*`.\r\n\r\n## Anti-Patterns (with explanations)\r\n\r\n| Pattern | Why it's wrong |\r\n|---------|---------------|\r\n| `os.environ.get(\"DF_DEFAULT_LANGUAGE\")` | Process-level — all users get same language; breaks multi-user |\r\n| Global LLM client interceptor | Hidden behavior; can't distinguish full/compact mode; fragile string detection |\r\n| New `MessageBuilder` class | Duplicates `agent_language.py`; creates parallel conflicting abstractions |\r\n| Hardcoded `\"回答请使用中文\"` in prompts | Not configurable; skips the mode system; breaks for other languages |\r\n| Backend-side translation dict (`agent_messages.py`) | Forces adding every new language to Python; translations should all live in `src/i18n/locales/` |\r\n| Hardcoded English UI strings in `.tsx` without `t()` | Not translatable; use `useTranslation` + `t('key')` |\r\n\r\n## Adding a New Language\r\n\r\n1. Add language code + display name to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py`.\r\n2. Optionally add extra rules to `LANGUAGE_EXTRA_RULES` (e.g. simplified vs traditional Chinese).\r\n3. Add frontend translations in `src/i18n/locales//` — copy an existing locale folder as template.\r\n4. No Agent code changes needed — the existing flow picks up new languages automatically.\r\n",".cursor/skills/path-safety/SKILL.md":"---\r\nname: path-safety\r\ndescription: 服务端路径安全与文件访问编码规范。在编写文件下载路由、Agent 工具（文件读取/目录列出）、数据连接器/Loader、Workspace 路径操作、沙箱配置时使用。\r\n---\r\n\r\n# Path Safety — 服务端安全编码规范\r\n\r\n> **来源**：`docs/dev-guides/8-path-safety.md`（正式开发规范）+ `design-docs/issues/002-arbitrary-file-read-audit.md`（安全审计复核）。\r\n> 本文档提炼了 6 条必须遵守的编码规范。违反任一条即可能引入路径穿越（LFI）漏洞。\r\n\r\n---\r\n\r\n## R1. 文件下载：用 `ConfinedDir.resolve()` + `send_file`，禁用 `send_from_directory`\r\n\r\n**原因**：`send_from_directory(dir, user_input)` 内部会对 `user_input` 二次解析路径，与前置安全检查形成 TOCTOU 不一致。\r\n\r\n```python\r\n# ❌ BAD — 安全检查用 resolved target，发送用原始 filename，两次解析不一致\r\ntarget = (scratch_dir / filename).resolve()\r\ntarget.relative_to(scratch_dir.resolve()) # 检查通过\r\nreturn send_from_directory(str(scratch_dir), filename) # 再次解析\r\n\r\n# ✅ GOOD — 检查和发送用同一个 resolved path\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target) # 直接用已验证的路径\r\n```\r\n\r\n`send_file(Path)` 会根据扩展名自动推断 MIME type，无需额外处理。\r\n\r\n---\r\n\r\n## R2. 路径安全检查：用 `ConfinedDir`，禁止 `str.startswith`\r\n\r\n**原因**：`str(path).startswith(str(root))` 存在前缀碰撞缺陷（如 `/workspace` vs `/workspace_evil`）。\r\n\r\n```python\r\n# ❌ BAD\r\nif not str(resolved).startswith(str(root_resolved) + os.sep):\r\n raise ValueError(\"escape\")\r\n\r\n# ✅ GOOD — 统一走 ConfinedDir，内部使用 Path.is_relative_to()\r\njail = ConfinedDir(root_resolved, mkdir=False)\r\ntarget = jail.resolve(user_input)\r\n```\r\n\r\n---\r\n\r\n## R3. Agent 工具复用 `Workspace.confined_*`\r\n\r\n**原因**：Agent 工具参数由 LLM 生成，必须视为间接用户输入。不要在工具内手写 `Path(root) / rel_path` 或 `resolve() + relative_to()`；入口处复用 Workspace 暴露的 `ConfinedDir`。\r\n\r\n```python\r\n# ❌ BAD — 手写路径拼接和校验\r\ndef _tool_read_file(self, args, workspace_path):\r\n target = (workspace_path / rel_path).resolve()\r\n target.relative_to(workspace_path)\r\n\r\n# ✅ GOOD — 入口拿到 ConfinedDir，工具只调用 jail.resolve()\r\ndef _execute_tool(self, name, args):\r\n workspace_jail = self.workspace.confined_root\r\n scratch_jail = self.workspace.confined_scratch\r\n return self._tool_read_file(args, workspace_jail)\r\n\r\ndef _tool_read_file(self, args, workspace_jail):\r\n target = workspace_jail.resolve(args.get(\"path\", \"\"))\r\n```\r\n\r\n---\r\n\r\n## R4. 优先使用 `ConfinedDir`，禁止裸路径拼接\r\n\r\n**原因**：`Path(root) / user_input` 是路径穿越的高频入口。`ConfinedDir` 封装了三层防御（拒绝绝对路径 → 拒绝 `..` 段 → `resolve` + `is_relative_to`）。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\n# ❌ BAD — 手动拼接 + 手动校验，容易遗漏\r\nlocal_file = tmp_path / blob_relative_name\r\nlocal_file.parent.mkdir(parents=True, exist_ok=True)\r\nlocal_file.write_bytes(data)\r\n\r\n# ✅ GOOD — ConfinedDir 自动校验 + 创建父目录\r\njail = ConfinedDir(tmp_path, mkdir=False)\r\njail.write(blob_relative_name, data) # 自动校验 + 写入\r\n```\r\n\r\n### 已有安全 API 的层次关系\r\n\r\n```\r\n用户输入（filename / relative_path / blob key）\r\n │\r\n ▼\r\nsafe_data_filename() / secure_filename() ← 第一层：输入清洗\r\n │\r\n ▼\r\nConfinedDir.resolve() ← 第二层：路径约束\r\n │\r\n ▼\r\n安全的 Path 对象\r\n```\r\n\r\n使用 `Workspace.get_file_path()` 的场景不需要手动调用 `ConfinedDir`，因为它内部已包含等价的校验。\r\n\r\n---\r\n\r\n## R5. 宿主文件系统访问必须设部署模式守卫\r\n\r\n**原因**：桌面单用户模式允许访问本机文件系统（预期行为），但多用户/云部署下等于开放服务器读权限。\r\n\r\n**规范**：任何新的 Loader / Connector 如果涉及直接读取宿主文件系统（不通过 Workspace API），必须：\r\n\r\n1. 在 `data_loader/__init__.py` 的 `_enforce_deployment_restrictions()` 中注册禁用规则\r\n2. 确保 `create_connector()` 会拒绝已禁用的类型\r\n\r\n```python\r\n# data_loader/__init__.py — 参考 local_folder 的处理方式\r\ndef _enforce_deployment_restrictions():\r\n backend = os.environ.get(\"WORKSPACE_BACKEND\", \"local\")\r\n if backend != \"local\":\r\n for key in (\"local_folder\", \"your_new_local_loader\"):\r\n if key in DATA_LOADERS:\r\n del DATA_LOADERS[key]\r\n DISABLED_LOADERS[key] = f\"{key} disabled in multi-user mode\"\r\n```\r\n\r\n**判断标准**：如果 Loader 的构造函数接受一个用户可控的本机路径（如 `root_dir`），它就需要部署守卫。\r\n\r\n---\r\n\r\n## R6. 多用户部署必须启用沙箱\r\n\r\n**原因**：`not_a_sandbox` 模式下 LLM 生成的代码在宿主进程直接执行，可绕过所有路径检查。\r\n\r\n`app.py` 已在启动时检测此配置并输出 `logger.critical` 警告。新增的沙箱模式或部署脚本应确保：\r\n\r\n- `WORKSPACE_BACKEND != \"local\"` 时，`SANDBOX` 必须为 `docker` 或 `local`\r\n- CI/CD 部署模板中默认设置 `SANDBOX=docker`\r\n\r\n---\r\n\r\n## 速查：新增代码时的安全检查清单\r\n\r\n| 场景 | 必须做的事 |\r\n|------|-----------|\r\n| 新增文件下载路由 | 用 `ConfinedDir.resolve()` 得到路径，再 `send_file(resolved_path)`；不用 `send_from_directory` |\r\n| 新增 Agent 工具（读文件/列目录） | 入口复用 `workspace.confined_root` / `workspace.confined_scratch`，工具内只调用 `jail.resolve()` |\r\n| 路径包含判断 | 用 `ConfinedDir.resolve()`，不要手写 `Path.is_relative_to()` 或 `str.startswith()` |\r\n| `Path(root) / variable` 模式 | 改用 `ConfinedDir` 或 `Workspace.get_file_path()` |\r\n| 新增本机文件系统 Loader | 在 `_enforce_deployment_restrictions()` 中注册多用户禁用 |\r\n| 部署配置 | 多用户模式必须 `SANDBOX=docker` 或 `SANDBOX=local` |\r\n\r\n---\r\n\r\n## 参考文档\r\n\r\n- `docs/dev-guides/8-path-safety.md` — 服务端路径安全开发规范\r\n- `design-docs/6-path-safety-confined-dir.md` — 剩余未完成实现项状态页\r\n- `design-docs/issues/002-arbitrary-file-read-audit.md` — 安全审计复核报告\r\n- `py-src/data_formulator/security/path_safety.py` — ConfinedDir 源码\r\n",".github/copilot-instructions.md":"# Project Instructions\n\n## Python Environment\n\n- Always use `uv` instead of `pip` for installing packages (e.g. `uv pip install`, `uv pip install -e .`).\n- Use `uv run` to execute Python scripts and modules (e.g. `uv run python script.py`, `uv run pytest`).\n- The virtual environment is at `.venv/`. Activate it with `source .venv/bin/activate` if needed.\n- The Python source is under `py-src/`. The project is installed in editable mode via `uv pip install -e .`.\n","examples/plugins/README.md":"# Data Formulator — Data Loader Plugins\n\nDrop-in Python files that add new data-source connectors to Data\nFormulator without modifying its source code. If the built-in\nconnectors don't cover your data source (an internal warehouse, a SaaS\nAPI, a niche database), write a small plugin and DF will pick it up on\nthe next restart.\n\nThis folder contains **example plugins**. Treat them as templates: copy\none, rename it, and adapt the body.\n\n---\n\n## Quick start (3 steps)\n\n1. **Find your plugin directory.** It lives under your Data Formulator\n home dir:\n\n ```\n $DATA_FORMULATOR_HOME/plugins/\n ```\n\n If `DATA_FORMULATOR_HOME` is not set, DF defaults to\n `~/.data_formulator/`, so the plugin dir is `~/.data_formulator/plugins/`.\n\n Power users can point somewhere else with `DF_PLUGIN_DIR`\n (highest precedence) — useful for sharing one plugin folder across\n multiple DF installs.\n\n2. **Copy an example into it.** For instance:\n\n ```bash\n mkdir -p \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins\"\n cp examples/plugins/sqlite_data_loader.py \\\n \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins/\"\n ```\n\n3. **Restart Data Formulator.** The new connector appears in the UI\n automatically. No registry edits, no rebuilds.\n\nTo verify it loaded, check the startup log for a line like:\n\n```\nINFO ... Plugin loader 'sqlite' registered from sqlite_data_loader.py\nINFO ... Plugin scan complete: 1 registered, 0 failed (dir=..., reason=WORKSPACE_BACKEND=local)\n```\n\n### Plugin directory resolution order\n\n| Precedence | Source | Default |\n| ---------- | ----------------------------------------- | ----------------------------- |\n| 1 | `DF_PLUGIN_DIR` env var (explicit override) | — |\n| 2 | `$DATA_FORMULATOR_HOME/plugins` | — |\n| 3 | Fallback | `~/.data_formulator/plugins/` |\n\n---\n\n## File-name contract\n\n| Filename | Registry key |\n| --------------------------------- | -------------- |\n| `sqlite_data_loader.py` | `sqlite` |\n| `acme_warehouse_data_loader.py` | `acme_warehouse` |\n| `notion_data_loader.py` | `notion` |\n\nRules:\n\n* The filename **must** end in `_data_loader.py`.\n* The prefix becomes the registry key — keep it lowercase, no spaces.\n* If the key matches a built-in (e.g. `mysql_data_loader.py`), the\n plugin **overrides** the built-in. Useful for hot-patching.\n\n---\n\n## What goes inside the file\n\nEach plugin defines exactly one class that subclasses\n[`ExternalDataLoader`](../../py-src/data_formulator/data_loader/external_data_loader.py).\nThe minimum surface area:\n\n```python\nfrom data_formulator.data_loader.external_data_loader import (\n ExternalDataLoader, MAX_IMPORT_ROWS,\n)\nimport pyarrow as pa\n\nclass MyLoader(ExternalDataLoader):\n\n # Optional: human-friendly UI label. Without this, the registry key\n # is title-cased (``\"my_warehouse\"`` → ``\"My Warehouse\"``). Override\n # to fix awkward casing (``\"SQLite\"``, ``\"BigQuery\"``).\n DISPLAY_NAME = \"My Warehouse\"\n\n @staticmethod\n def list_params() -> list[dict]:\n \"\"\"Declare connection-form fields. The UI auto-renders this.\"\"\"\n return [\n {\"name\": \"endpoint\", \"type\": \"string\", \"required\": True,\n \"tier\": \"connection\", \"description\": \"Server URL\"},\n {\"name\": \"token\", \"type\": \"string\", \"required\": True,\n \"tier\": \"auth\", \"sensitive\": True, \"description\": \"API token\"},\n ]\n\n @staticmethod\n def auth_instructions() -> str:\n \"\"\"Markdown help text shown next to the form.\"\"\"\n return \"Get your API token from https://example.com/settings/tokens\"\n\n def __init__(self, params: dict):\n self.params = params\n # validate + open connection here\n\n def list_tables(self, table_filter: str | None = None) -> list[dict]:\n \"\"\"Return catalog: [{name, metadata: {columns, row_count}}, ...]\"\"\"\n ...\n\n def fetch_data_as_arrow(self, source_table: str,\n import_options: dict | None = None) -> pa.Table:\n \"\"\"Read rows. Honour import_options['size'] up to MAX_IMPORT_ROWS.\"\"\"\n ...\n```\n\nLook at [`sqlite_data_loader.py`](sqlite_data_loader.py) for a runnable\nimplementation (~170 lines, stdlib only).\n\n### `list_params()` field reference\n\n| Key | Meaning |\n| ------------- | ---------------------------------------------------------------- |\n| `name` | Parameter key passed into `__init__(params)` |\n| `type` | `\"string\"`, `\"int\"`, `\"bool\"`, `\"password\"` |\n| `required` | If `True`, DF rejects connections that omit it |\n| `default` | Pre-filled value in the form |\n| `sensitive` | If `True`, value is redacted from logs / stored metadata |\n| `tier` | `\"auth\"`, `\"connection\"`, or `\"filter\"` — groups fields visually |\n| `description` | Help text shown under the field |\n\n---\n\n## Security gate — please read\n\nPlugins execute arbitrary Python in the server process. To prevent\naccidental code execution in shared deployments, the plugin scanner is\n**enabled only in single-user local mode**:\n\n* `WORKSPACE_BACKEND` unset or `local` → scanner runs.\n* `WORKSPACE_BACKEND` is anything else → scanner is skipped silently.\n* To opt in for a hosted deployment, set `DF_ALLOW_PLUGINS=1`\n **and** make sure the plugin directory is writable only by trusted\n administrators.\n\n---\n\n## Debugging a plugin that won't load\n\nThe scanner reports every failure two ways:\n\n1. **In the log**, with a full traceback:\n\n ```\n WARNING Failed to load plugin foo_data_loader.py: ... ModuleNotFoundError ...\n ```\n\n2. **In `DISABLED_LOADERS`**, which the frontend surfaces as a greyed-out\n connector with the reason. Common reasons:\n\n | Message | Fix |\n | ------------------------------------------------------ | --- |\n | `missing dependency 'X' (pip install X)` | `uv pip install X` in DF's venv |\n | `no ExternalDataLoader subclass found` | Add a `class Foo(ExternalDataLoader)` to the file |\n | `could not create import spec` | File isn't valid Python or has wrong extension |\n | `: ` | Anything else raised during `import` — check the traceback |\n\nThe plugin scanner cleans `sys.modules` on failure, so simply restarting\nDF after editing the file picks up the fix — no manual cleanup needed.\n\n---\n\n## Adding third-party dependencies\n\nIf your plugin needs a package that isn't already in DF's environment:\n\n```bash\n# from the data-formulator repo root\nuv pip install \n```\n\nThe plugin file is loaded into the same Python process as DF, so\nanything importable from DF's venv is importable from your plugin.\n\n---\n\n## For AI coding agents\n\nWhen asked to **add a new data source** to a user's Data Formulator\ninstall, prefer writing a plugin file over modifying DF's source tree:\n\n1. Read [`sqlite_data_loader.py`](sqlite_data_loader.py) and the\n `ExternalDataLoader` ABC in\n `py-src/data_formulator/data_loader/external_data_loader.py` to\n understand the required methods.\n2. Create a new file `_data_loader.py` in this folder (or\n directly in `~/.data_formulator/plugins/`).\n3. Subclass `ExternalDataLoader`. Implement at minimum:\n `list_params`, `auth_instructions`, `__init__`, `list_tables`,\n `fetch_data_as_arrow`.\n4. Return data as a `pyarrow.Table` from `fetch_data_as_arrow` — do\n **not** convert to pandas in the hot path.\n5. Quote identifiers when building SQL (see `_quote_ident` in the\n SQLite example) to avoid injection vulnerabilities.\n6. Respect `import_options['size']` and cap at `MAX_IMPORT_ROWS`.\n7. If the source has credentials, mark those params `sensitive: True`\n and `tier: \"auth\"` so DF redacts them from stored metadata.\n8. Do not modify the user's data — open read-only connections where\n the source supports it.\n\nAfter writing the file, verify it loads with:\n\n```bash\nDF_PLUGIN_DIR= uv run python -c \\\n \"from data_formulator import data_loader as dl; \\\n print(dl.PLUGIN_LOADERS, dl.DISABLED_LOADERS)\"\n```\n","py-src/data_formulator/analyst/skills/core/SKILL.md":"---\nname: core\ndescription: >-\n The analyst's built-in capabilities: data-inspection tools and the\n always-available actions (visualize and ask_user).\nwhen_to_use: Always loaded by default — this is the agent's baseline.\nalways_on: true\ntools:\n - execute_python_script\n - inspect_source_data\nactions:\n - visualize\n - ask_user\n---\n\n# Core capabilities\n\nThis describes the built-in **inspection tools** you use to gather data and the\nalways-available **actions** you take on it. The overall loop, your action\nbudget, and the one-action-per-turn rule are covered in your system\ninstructions — this section is about *what* each tool and action does and how\nto use it well.\n\n## Tools (for data gathering)\n\n- **execute_python_script(code)** — run a general-purpose Python script to\n inspect data, compute stats, transform tables, or verify assumptions. Its\n stdout is returned to you (use `print()`); the script is for *your* analysis\n and its output is never shown to the user. pandas, numpy, duckdb, sklearn,\n scipy are available. **Important**: each call runs in a fresh namespace —\n variables do NOT persist between calls, so combine related steps into a\n single script.\n- **inspect_source_data(table_names)** — get schema, stats, and sample rows for\n source tables (cheaper than `execute_python_script` for basic inspection).\n- **load_skill(name)** — load a skill's instructions into context so you can use\n the action it unlocks (see the Skills section of your system instructions).\n\nThese are inspection tools — their results come back to you and are never shown\nto the user; call as many as you need, then take an action or give your final\nanswer.\n\nYou analyse data that is **already in the workspace**. If the user's question\nrequires connected data that isn't present, call `load_skill(\"data-loading\")`\nand follow that skill's discovery and immutable proposal workflow in this same\nconversation. Do not hand off to the standalone Data Loading agent.\n\nThe initial context already includes sample rows and statistics for each table.\nIf the data is straightforward, go straight to the action without calling\ntools. Tool results are returned to you before you act.\n\n## Actions\n\nCall an action as a tool call when you want to act on the data. Actions are\n**sequential**: take **one at a time**, then read the result it returns before\ndeciding the next — each action's outcome shapes the next one (the chart you draw\nnext depends on what this one reveals), so emitting several at once would decide\nthe later ones blind. After each result you choose what to do — take another\naction, or stop. **You end your turn by replying with plain text and no\naction**: that is your closing answer when you expect nothing further. When you\nwant the user to reply — a freeform question, a clarification you need before\nacting, or **clickable choices** — use the `ask_user` action instead. It renders\na question widget and pauses for their reply, keeping the conversation in the\nsame turn (plain text ends the run, so the user's next message would start\nfresh without this context).\n\n**Match the response to what the user asked for.** Two different cases:\n\n- **A direct answer** — they asked a question, so answer it. Length follows the\n question: one line when that settles it, more when it genuinely takes more.\n- **A finding after you acted** — they asked for the work, not a write-up, so\n this is unsolicited. The artifact already shows what it shows; add only what\n they'd miss by looking at it, and default to short.\n\nOpen with the point rather than announcing one is coming, and don't close by\nrestating what you just said.\nNever narrate what you're about to do or recap a chart's axes; let the artifact\nspeak for itself. When an action pauses for the user, give enough context to\nexplain what you found and what their choices mean.\n\n### `visualize` — chart a transform\n\nRun code that produces a DataFrame and render it as a chart. You then observe the\nresult and decide your next move.\n\n- `display_instruction` — ≤12 words; the question/hypothesis the chart\n investigates (don't recap x/y/color — those are visible). Wrap a **column** in\n `**…**` if it anchors the question.\n- `title` — a concise, neutral analytical heading naming the subject, measure,\n and analytical lens, such as “Year-over-year price change peaks.” Prefer a\n stable description of the view over a takeaway claim or narrated trend. Do\n not mention the chart type, imply causality, or editorialize. This field is\n required; put interpretation in the closing response instead.\n- `subtitle` — concise supporting context not already clear from the title or\n axes. Use one phrase of at most 16 words to provide contextual details. Do\n not restate the measure or analytical lens named in the title.\n- `code` — Python producing a DataFrame assigned to `output_variable`.\n- `output_variable` — snake_case name the code assigns.\n- `chart` — `{chart_type, encodings:{x,y,…}, config:{}}` (chart_type from the\n chart type reference).\n- `input_tables` — workspace table names, as listed in the available-tables\n context, that the code reads.\n- `field_metadata` — field → semantic annotation. Include units, index\n baselines, intrinsic domains, and ordinal order when supported by the data;\n never invent a unit. Distinguish percentages from percentage points and\n identifiers from quantities.\n- `field_display_names` — field → concise human-readable label for axes,\n legends, and table headers. Expand technical names, preserve established\n domain abbreviations, include units when useful, and use the user's language.\n\nSilently classify the analytical intent before choosing a chart: comparison,\ntrend, distribution, relationship, composition, deviation, ranking,\nuncertainty, or spatial pattern. Choose encodings and chart type from that\nintent and the data shape. Set ordering deliberately: chronological for time,\nsemantic order for ordinal fields, and measure order for rankings. Avoid line\ncharts or legends with excessive series, labels that collide, and color that\ndoes not encode additional information; aggregate, bin, facet, or limit\ncategories when needed without hiding material data.\n\n### `ask_user` — ask the user and pause for their reply (pauses the run)\n\nAsk the user something and pause for their input. Reach for this on **any** turn\nwhere you want a reply — a choice to make, a clarification you need before\nacting, or a brief statement paired with clickable follow-ups they can react to.\nPrefer it over ending your turn with a plain-text question: plain text ends the\nrun (the user's next message starts a fresh turn without this context), while\n`ask_user` keeps the conversation in the same turn.\n\n- `questions` — 1–3 items, each something the user **acts on**: a choice\n (`single_choice` with `options`) or an open question they type an answer to\n (`free_text`). Put your reasoning, rationale, and context in your reply text —\n **not** here. Never add a `questions` item that only states a rationale or\n explanation with nothing for the user to answer or click.\n- each question: `text` (wrap a **column** in `**…**`), `responseType`\n (`single_choice` when you offer `options`, else `free_text` — the user types\n their own open-ended answer, not a slot for your exposition), `required`\n (`true` when the run depends on the answer, `false` for an optional follow-up),\n and `options` (plain-text choices, **at most 3** — just the most likely\n answers; the user can always type a freeform reply, so don't enumerate every\n case).\n\nThis is **terminal**: the run pauses after it and resumes when the user replies.\n\n## Choosing what to do\n\nMatch the response depth to the user's request. Create charts that materially\ncontribute to the answer, and stop when the answer is sufficient.\n\n- For conceptual or informational questions, answer directly when a chart would\n not improve the answer.\n- For specific analytical questions, create the view or views needed to answer\n them clearly.\n- For diagnostic or exploratory questions, follow relevant findings across\n multiple views when doing so adds meaningful insight.\n- If essential intent is unclear, use `ask_user` rather than guessing.\n- *Missing data* (needs tables not in the workspace):\n `load_skill(\"data-loading\")`, discover the source, and propose immutable\n loading options inline.\n- *Report / write-up request* (e.g. \"write a report on X\", \"summarize the findings\n as a narrative\"): this needs the **report** skill — `load_skill(\"report\")` and\n follow it to commit the `write_report` action. **Do this as your very first\n move when charts already exist** (see `[AVAILABLE CHARTS]` / the thread): don't\n re-create them — load the report skill straight away and embed the existing\n charts by id. Only produce a new chart first if the report genuinely needs one\n that isn't there yet (0–3, judgment-based), then load the skill.\n\nFollow explicit requests about scope, depth, and format. **Never** repeat a\nvisualization already in the trajectory or in another thread.\n\n## Chart Creation Guide\n\nThe following reference material applies when you call the `visualize` tool.\n\n### A. Code Execution Rules\n\n**About the execution environment:**\n- You can use BOTH DuckDB SQL and pandas operations in the same script\n- The script will run in the workspace data directory (all data files are in the current directory)\n- Each table in [CONTEXT] has a **file path** (e.g., `student_exam.parquet`, `sales.csv`). Use EXACTLY that path to load data:\n - `.parquet`: `pd.read_parquet('file.parquet')` or DuckDB `read_parquet('file.parquet')`\n - `.csv`: `pd.read_csv('file.csv')` or DuckDB `read_csv_auto('file.csv')`\n - `.json`: `pd.read_json('file.json')`\n - `.xlsx`/`.xls`: `pd.read_excel('file.xlsx')`\n - `.txt`: `pd.read_csv('file.txt', sep='\\t')`\n- **IMPORTANT:** Use the exact filename from the context — do NOT change the file extension or assume all files are parquet.\n- **Allowed libraries:** pandas, numpy, duckdb, math, datetime, json, statistics, collections, re, sklearn, scipy, random, itertools, functools, operator, time\n- **Not allowed:** matplotlib, plotly, seaborn, requests, subprocess, os, sys, io, or any other library not listed above.\n- File system access (open, write) and network access are also forbidden.\n\n**When to use DuckDB vs pandas:**\n- **Prefer plain pandas** for most tasks — it's simpler and more readable.\n- Only use DuckDB when the dataset is very large and you need efficient SQL aggregations, filtering, joins, or window functions.\n- You can combine both: DuckDB for initial loading/filtering on large files, then pandas for complex operations.\n\n**Code structure:** standalone script (no function wrapper), imports at top. **CRITICAL:** The final result DataFrame MUST be assigned to the exact variable name you specified in `\"output_variable\"` — the system uses this name to extract the result. For example, if your output_variable is `sales_by_region`, the script must contain `sales_by_region = ...`.\n\n**DuckDB notes:**\n- Escape single quotes with '' (not \\')\n- No Unicode escapes (\\u0400); use character ranges directly: [а-яА-Я]\n- Cast date columns explicitly: `CAST(col AS DATE)`, `CAST(col AS TIMESTAMP)`\n- For complex datetime operations, load data first then use pandas datetime functions\n- Critical identifier quoting rule:\n * If a table/column name contains non-ASCII characters (e.g., Chinese, Japanese, Korean, Cyrillic, etc.), spaces, or punctuation,\n you MUST wrap it in double quotes, e.g. SELECT \"金额\" FROM \"客户表\".\n * Never output placeholder identifiers like your_table_name, your_column, your_condition.\n\n**Datetime handling:**\n- `date` columns contain date-only values (YYYY-MM-DD). `datetime` columns contain date+time (ISO 8601).\n- `time` columns contain time-only values (HH:mm:ss). `duration` columns are time intervals.\n- Year → number. Year-month / year-month-day → string (\"2020-01\" / \"2020-01-01\").\n- Hour alone → number. Hour:min or h:m:s → string. Never return raw datetime objects.\n\n### B. Chart Type Reference\n\nThe `chart_type` value in the `visualize` action MUST be one of the names listed\nbelow (exact spelling, including capitalization). When a row lists multiple\nnames, pick whichever fits the \"when to use\" hint best.\n\n**Choosing a chart — prefer simple, escalate when it fits.** Reach for the\n**Everyday** set first: it answers most questions and is the safest, most\nlegible choice. But when the data or question genuinely fits a **Specialized**\ntype (a distribution's shape, a cumulative curve, a rank race, a before→after,\na geographic pattern…), prefer it — a well-matched specialized chart is more\ninsightful than forcing a generic one. Don't pick a specialized type for\nnovelty; use it because its \"when to use\" condition is met.\n\n**Everyday — reach for these first**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Scatter Plot | x, y, color, size, facet | opacity (0.1–1.0) | Relationships between two quantitative fields |\n| Regression | x, y, color, size, facet | regressionMethod (\"linear\",\"log\",\"exp\",\"pow\",\"quad\",\"poly\"), polyOrder (2–10) | Trend line over scatter; one line per color group |\n| Bar Chart / Stacked Bar Chart / Lollipop Chart / Waterfall Chart | x, y, color, facet | — | Bar: categorical comparison (auto-stacks when color is set). Stacked Bar: explicit stacked totals, color = the stack. Lollipop: cleaner for ranked lists / sparse categories. Waterfall: cumulative gain/loss, each bar starts where the previous ended |\n| Grouped Bar Chart | x, y, group, facet | — | Side-by-side bars across a second categorical dimension |\n| Line Chart | x, y, color, strokeDash, facet | interpolate (\"linear\",\"monotone\",\"step\") | Trends over an ordered (usually temporal) x-axis |\n| Area Chart | x, y, color, facet | — | Magnitude over ordered x; auto-stacks when color is set |\n| Histogram / Density Plot | x, color, facet | — | Distribution of one quantitative field. Histogram: discrete bins, auto-binned. Density Plot: smooth KDE curve |\n| Boxplot | x, y, color, facet | — | Distribution summary (median/quartiles/outliers) by category |\n| Pie Chart | size, color, facet | innerRadius (0–100; 0=pie, >0=donut) | Part-of-whole with ≤7 categories. Wedge value goes on **size**, not **theta** |\n| Heatmap | x, y, color, facet | colorScheme — sequential (\"viridis\",\"blues\",\"reds\",\"oranges\",\"greens\") or diverging (\"blueorange\",\"redblue\") | Matrix / 2D density; color encodes the quantitative cell value |\n\n**Specialized — use when the data/question fits the \"when to use\"**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Connected Scatter Plot | x, y, order, color, facet | — | Two quantitative fields traced in sequence — needs an `order` field (e.g. time) so points are joined in order, not by x |\n| Ranged Dot Plot | x, y, color, facet | — | Min–max range or two-point comparison per category |\n| Violin Plot | x, y, color, facet | — | Distribution SHAPE (KDE silhouette) by category; better than a boxplot when data is multimodal. x = category, y = value |\n| Strip Plot | x, y, color, size, facet | — | Every individual point by category (jittered); good for small/medium n where raw values matter, not just a summary |\n| ECDF Plot | x, color, facet | — | Cumulative distribution of one quantitative field. Pass the RAW field on x (do NOT pre-compute the CDF); color for per-group curves |\n| Bump Chart | x, y, color, facet | — | How RANKINGS change over ordered x; y = rank, color = entity (long-form: one row per entity × x) |\n| Slope Chart | x, y, color, facet | — | Change between exactly TWO points (before → after) per entity; x = the two labels, y = value, color = entity |\n| Streamgraph | x, y, color, facet | — | Several series' magnitude over ordered x, stacked around a center baseline (color = series) — theme/volume shifts over time |\n| Range Area Chart | x, y, y2, color, facet | — | A shaded band between a lower (y) and upper (y2) bound over ordered x — e.g. min–max or a confidence interval |\n| Rose Chart | x, y, color, facet | — | Cyclical/categorical magnitude as angular wedges (polar bars); x = category/angle, y = value |\n| Pyramid Chart | x, y, color, facet | — | Back-to-back bars split by a binary group (e.g. population by age × sex); y = category, x = value, color = the two-sided group |\n| Radar Chart | x, y, color, facet | — | Multi-metric profile/comparison; x = metric name, y = value, color = entity (long-form data) |\n| Bar Table | x, y, color, facet | — | Ranked horizontal table with inline bars; one row per category. y = category, x = value |\n| KPI Card | metric, value, goal | — | \"Big number\" dashboard tile(s); one row per tile. `value` must be pre-aggregated; `goal` is optional |\n| Candlestick Chart | x, open, high, low, close, facet | — | OHLC financial data |\n| Map | longitude, latitude, color, size | projection (\"mercator\",\"equalEarth\",\"naturalEarth1\",\"orthographic\",\"albersUsa\"), projectionCenter ([lon,lat]) | Geographic POINTS/bubbles by lon/lat (use projection \"albersUsa\" for a US-only map) |\n| Choropleth | id, color, facet | region (\"world\",\"usa\",…) | Filled REGIONS shaded by value; `id` = the region key (country/state name or code), color = the quantitative value |\n\n**Critical chart rules:**\n- **Scatter Plot**: use config opacity (0.1–1.0) for dense data instead of encoding opacity.\n- **Regression**: trend line is automatic — do NOT compute regression coefficients/predictions in Python. Use `color` to get separate trend lines per group.\n- **Bar Chart**: x=categorical, y=quantitative (vertical bars). Swap x↔y for horizontal bars. Same-x rows are auto-stacked when `color` is set.\n- **Grouped Bar Chart**: use the `group` channel (not `color`) for side-by-side bars.\n- **Histogram**: do NOT pre-bin in Python — pass the raw quantitative field on `x` and the chart bins automatically. Pre-aggregating gives wrong bin widths.\n- **Line Chart**: use `strokeDash` to differentiate line styles (e.g. actual vs forecast).\n- **Pie Chart**: use the `size` channel (not `theta`) for wedge values. Avoid when >7–8 categories.\n- **Radar Chart**: data must be long-form — one row per (entity, metric, value). If your data is wide-form (one column per metric), melt it first in the Python step.\n- **Heatmap**: pick `colorScheme` by the meaning of the values. Use a **sequential** scheme (viridis/blues/reds/oranges/greens) for single-direction magnitudes (counts, rates, prices, scores — higher is simply more). Use a **diverging** scheme (blueorange/redblue) ONLY when the values have a meaningful center to read away from (e.g. profit/loss around 0, change vs. a baseline, temperature around freezing).\n- **Bar Table**: y is the category column to rank; x is the quantitative value driving bar length. Don't sort in Python — the template sorts.\n- **KPI Card**: channels are `metric`, `value`, `goal` (not x/y). One DataFrame row = one tile. The `value` column must already contain the final number to display (aggregate upstream in the Python step).\n- **Candlestick Chart**: requires `open`, `high`, `low`, `close` columns.\n- **Connected Scatter Plot**: provide an `order` field (usually time) so points are joined in sequence, not by x-order.\n- **ECDF Plot**: pass the RAW quantitative field on `x` — the chart computes the cumulative curve; do NOT pre-compute it in Python.\n- **Range Area Chart**: `y` is the lower bound and `y2` the upper bound of the band.\n- **Bump / Slope Chart**: long-form data — one row per (entity, x); `color` is the entity. Slope's `x` has exactly two categories (before/after).\n- **Violin Plot**: like Boxplot but shows the full distribution shape; x = category, y = value.\n- **Map / Choropleth**: `Map` plots points via `longitude` / `latitude` (set projection `\"albersUsa\"` for the US); `Choropleth` fills regions — put the region key on `id` and the value on `color`, not `x` / `y`.\n- **facet**: available for nearly all chart types; use a low-cardinality categorical field.\n- All fields in `encodings` must also appear in `output_fields`. Typically use 2–3 channels (x, y, color/size).\n\n### C. Semantic Type Reference\n\nChoose the most specific type that fits. Only annotate fields used in chart encodings.\n\n| Category | Types |\n|---|---|\n| Temporal | DateTime, Date, Time, Timestamp, Year, Quarter, Month, Week, Day, Hour, YearMonth, YearQuarter, YearWeek, Decade, Duration |\n| Monetary measures | Amount, Price |\n| Physical measures | Quantity, Temperature |\n| Proportion | Percentage |\n| Signed/diverging | Profit, PercentageChange, Sentiment, Correlation |\n| Generic measures | Count, Number |\n| Discrete numeric | Rank, Score |\n| Identifier | ID |\n| Geographic | Latitude, Longitude, Country, State, City, Region, Address, ZipCode |\n| Entity names | Category, Name |\n| Coded categorical | Status, Boolean, Direction |\n| Binned ranges | Range |\n| Fallback | Unknown |\n\nKey guidelines:\n- Use **Amount** for summed monetary totals, **Price** for per-unit prices, **Profit** for values that can be negative.\n- Use **Temperature** (not Quantity) for temperature — it has special diverging behavior.\n- Use **Year** (not Number) for columns like \"year\" with values 2020, 2021.\n\n### D. Statistical Analysis Guide\n\n- **Regression**: use chart_type \"Regression\" — the trend line is automatic, do NOT compute regression values in Python code. Configure method via `{\"regressionMethod\": \"linear\"}` (options: \"linear\", \"log\", \"exp\", \"pow\", \"quad\", \"poly\"; for poly add `{\"polyOrder\": 3}`).\n- **Forecasting**: compute predicted future values in Python. Use Line Chart with strokeDash to distinguish actual vs forecast, and color for series grouping.\n- **Clustering**: compute cluster assignments in Python. Output [x, y, cluster_id]. Use Scatter Plot with color → cluster_id.\n","py-src/data_formulator/analyst/skills/core/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"execute_python_script\",\n \"description\": \"Execute a general-purpose Python script in the sandbox. Here you use it to inspect data, compute statistics, transform tables, or verify assumptions before you act — write results to stdout with print() and that output is returned to you (it is NOT shown to the user). The script is for your own analysis, not for producing the final visualization. pandas, numpy, duckdb, sklearn, scipy are available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"purpose\": {\n \"type\": \"string\",\n \"description\": \"One-sentence description of what this script does and why (shown to user as progress).\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python script to execute. Use print() to surface output.\"\n }\n },\n \"required\": [\"purpose\", \"code\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_source_data\",\n \"description\": \"Get a detailed summary of one or more source tables — schema, field-level statistics, and sample rows. Cheaper than execute_python_script for basic data inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"table_names\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of workspace table names, as listed in the available-tables context, to inspect.\"\n }\n },\n \"required\": [\"table_names\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"visualize\",\n \"description\": \"Commit a data transform + chart: run code producing a DataFrame and render it. The agent observes the result and continues.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\": \"A concise, neutral analytical heading that names the subject, measure, and analytical lens, such as 'Year-over-year price change peaks'. Prefer a stable description of the view over a takeaway claim or narrated trend. Do not mention the chart type, imply causality, or editorialize. Shown as the chart heading.\"\n },\n \"subtitle\": {\n \"type\": \"string\",\n \"description\": \"Concise supporting context not already clear from the title or axes. Use one phrase of at most 16 words to provide contextual details. Do not restate the measure or analytical lens named in the title.\"\n },\n \"display_instruction\": {\n \"type\": \"string\",\n \"description\": \"≤12 words. State the question or hypothesis the chart investigates — don't recap the chart spec (x/y/color/split are already visible). Wrap a **column** in ** ** if it anchors the question.\"\n },\n \"input_tables\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Workspace table names, as listed in the available-tables context, that the code reads.\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python code producing a DataFrame assigned to output_variable.\"\n },\n \"output_variable\": {\n \"type\": \"string\",\n \"description\": \"snake_case name of the DataFrame variable the code assigns.\"\n },\n \"chart\": {\n \"type\": \"object\",\n \"description\": \"Chart spec: {chart_type, encodings:{x,y,...}, config:{}}. chart_type from the chart type reference.\"\n },\n \"field_metadata\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> SemanticType for the output columns.\"\n },\n \"field_display_names\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> human-readable display name for chart axes and table headers.\"\n }\n },\n \"required\": [\"title\", \"code\", \"output_variable\", \"chart\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"ask_user\",\n \"description\": \"Ask the user something and pause for their reply — the run resumes in the same turn with their answer in context. Use this for ANY turn where you want the user to respond: a choice to make, a clarification you need before acting, or a brief statement paired with clickable follow-ups. Put your reasoning, rationale, and context in your normal reply text, not inside this call. Prefer this over ending your turn with a plain-text question: plain text ends the run and the user's next message starts a fresh turn without this context, whereas ask_user keeps the conversation going. Reserve plain text (no action) for your final answer when you expect nothing further.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"thought\": {\n \"type\": \"string\",\n \"description\": \"Brief rationale (not shown to the user).\"\n },\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"1–3 things the user acts on: a choice (single_choice with options) or an open question they type an answer to (free_text). Put rationale, reasoning, and context in your reply text, not here — never add an item that only states an explanation with nothing for the user to answer or click. An explanation is allowed only as a short statement paired with clickable chart-producing follow-ups (required=false with options).\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The question, or (for an optional follow-up) a short statement. Keep a statement to 1–3 grounded sentences and pair it with clickable follow-up options. Wrap a **column** in ** ** to highlight it.\"\n },\n \"responseType\": {\n \"type\": \"string\",\n \"enum\": [\"single_choice\", \"free_text\"],\n \"description\": \"single_choice when you offer options; free_text when the user types their own open-ended answer (not a slot for your own exposition).\"\n },\n \"required\": {\n \"type\": \"boolean\",\n \"description\": \"false for an explanation / optional follow-up; true for a clarification the run depends on.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Plain-text choices, at most 3. Keep them to the few most likely answers — the user can always type a freeform reply, so don't try to enumerate every case. For a clarification these are answers; for an explanation these are short chart-producing follow-up prompts the user might click next (≤8 words each, phrased as the user would say them).\"\n }\n },\n \"required\": [\"text\"]\n }\n }\n },\n \"required\": [\"questions\"]\n }\n }\n }\n]\n","py-src/data_formulator/analyst/skills/data-loading/SKILL.md":"---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.","py-src/data_formulator/analyst/skills/data-loading/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]","py-src/data_formulator/analyst/skills/data_loading/SKILL.md":"---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.","py-src/data_formulator/analyst/skills/data_loading/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]","py-src/data_formulator/analyst/skills/report/SKILL.md":"---\nname: report\ndescription: >-\n Turn an exploration (threads, findings, charts) into a single Markdown\n report — note, blog post, executive summary, KPI dashboard, slide brief, or\n multi-section analytical report, with embedded charts.\nwhen_to_use: >-\n The user asks to write up / summarize / report on what they explored, or\n wants a shareable narrative document built from the charts and findings in\n the data thread. Not for producing a single new chart (use visualize).\nalways_on: false\ntools:\n - inspect_chart\nactions:\n - write_report\n---\n\n# Skill: Report writing\n\nYou are a data journalist / analyst who creates insightful, well-organized\nreports based on data explorations. The output is a single Markdown document\nthat may play many roles — short note, blog post, executive summary, dashboard,\nmulti-section report, FAQ, slide-style brief, etc. Adapt structure and length to\nwhat the user actually asks for; do not force a fixed template.\n\n## Emitting the report (the `write_report` action)\n\nFirst inspect whatever charts and data you need (see below), then write the\nentire report and commit it by **calling the `write_report` tool** — it is the\ncommitting action that ends this turn. Its `report` argument carries the\n**full Markdown** of the finished report:\n\n- `report` — the complete report in Markdown: headings, prose, tables, and\n embedded charts via `![caption](chart://chart_id)`.\n\nProduce any charts the report needs **before** calling `write_report`, and do\nall chart/data inspection first — once you call `write_report`, the report is\ndelivered as-is and the run ends.\n\n## Context available to you\n- **[PRIMARY TABLE(S)]** / **[OTHER AVAILABLE TABLES]**: Lightweight schema of datasets.\n- **[FOCUSED THREAD]** (optional): The exploration thread the user is continuing —\n the ordered steps with the user's questions, the agent's thinking, and the\n findings at each step. This is the spine of the story you are telling.\n- **[OTHER THREADS]** (optional): Brief per-step summaries of other exploration\n threads the user ran. These are additional findings worth weaving in.\n- **[AVAILABLE CHARTS]**: List of charts with their type, encodings, and table references.\n\n## Ground the report in the exploration\nThe thread context is your most important input. The user already did real\nanalysis — your job is to turn that journey into a coherent narrative, not to\nsummarize a single chart. Before writing:\n- Read the FOCUSED THREAD and OTHER THREADS to understand the full set of\n questions asked and findings reached.\n- Plan a report that covers the meaningful findings across the exploration,\n not just the last or most obvious chart.\n\n## Inspecting charts and data\nYou have two inspection tools available the whole time: `inspect_chart` and\n`inspect_source_data`. Use them on your own whenever you need to verify a detail\nbefore writing about it — a chart's exact numbers, its data, or a table's\nschema. `inspect_chart` lets you *read* a chart from its encodings, a data\nsample, and the code that produced it (and points you to the backing table so\nyou can interrogate the full data with `execute_python_script`); a rendered\nimage is included only when one is available. Read the charts behind the key\nfindings you present **before** you compose the report.\n\n## Write the report\nWrite the complete report in Markdown and pass it as the `report` argument of the\n`write_report` tool. Do all your inspecting first, then compose the whole\ndocument and make the one `write_report` call.\n\n### Embedding charts (REQUIRED FORMAT — do not change this)\nTo embed a chart image, use markdown image syntax with a `chart://` URL:\n ![Caption describing the chart](chart://chart_id)\n\nExample: `![Monthly trade balance trend](chart://chart-123)`\n\nThe chart_id must match one from [AVAILABLE CHARTS]. Place each chart embed on\nits own line (it renders as a block). You can embed the same chart at most\nonce. Captions are short — one line describing what the chart shows.\n\n### Tables\nFor data tables, write standard markdown tables directly:\n| date | value |\n| --- | --- |\n| 2020-01 | -43.5 |\n\n### Style & structure — adapt to the user's request\nThe user may ask for any of:\n- a short note or social-style summary (a few sentences, one or two charts),\n- a blog post / narrative report (intro → findings → takeaway),\n- an executive summary (key numbers up top, then context),\n- a KPI dashboard / multi-section overview (headings per topic, multiple charts\n arranged with short commentary between them),\n- a slide-style brief (compact sections with bullet points and embedded charts),\n- a deeper analytical report with sub-sections, methodology notes, and caveats.\n\nPick the structure that fits the request and the available material. Match the\nbreadth of the report to the breadth of the exploration: if the user explored\nseveral questions, the report should reflect that — don't collapse a rich\nexploration into a single-chart blurb unless the user explicitly asked for\nsomething that short. Reasonable defaults if the user is vague:\n- Start with a `# Title` that reflects the topic.\n- Group related findings under `##` (and `###` if useful) headings, typically\n one section per key finding / thread.\n- Around each embedded chart, briefly explain what it shows and the key insight.\n- Use bullets / short paragraphs / tables where they help; don't pad.\n- Close with a brief takeaway or summary section if the report is more than a\n few paragraphs. For very short outputs (notes, single-chart blurbs), a closing\n summary is optional.\n\n### Guardrails\n- Write in Markdown. Keep prose tight; let the data and charts carry the weight.\n- Stay faithful to the data — do not invent numbers, comparisons, or causation\n that the data does not actually support.\n- It is fine to flag uncertainty (\"based on the sample shown…\") when appropriate.\n- Embed every chart you discuss; don't reference a chart in prose without showing it.\n","py-src/data_formulator/analyst/skills/report/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_chart\",\n \"description\": \"Read one or more charts. Returns each chart's encodings, a sample of its data, and the transformation code that created it so you can reason about what it shows (use execute_python_script on the backing table for the full data). A rendered PNG is included only when one is available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"chart_ids\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of chart IDs from [AVAILABLE CHARTS] to inspect.\"\n }\n },\n \"required\": [\"chart_ids\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"write_report\",\n \"description\": \"Deliver a Markdown report and end the run. `report` is the full report text (embed charts with ![caption](chart://chart_id)).\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"report\": {\n \"type\": \"string\",\n \"description\": \"The full Markdown report text. Embed charts with ![caption](chart://chart_id) referencing IDs from [AVAILABLE CHARTS].\"\n }\n },\n \"required\": [\"report\"]\n }\n }\n }\n]\n"},"files":{".cursor/rules/backend-test-conventions.mdc":"---\r\ndescription: Backend Python test conventions (pytest)\r\nglobs: tests/backend/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# Backend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/backend/unit/`, `tests/backend/integration/`, `tests/backend/contract/`, or by domain: `tests/backend/routes/`, `tests/backend/errors/`, `tests/backend/agents/`, `tests/backend/data/`, `tests/backend/security/`, `tests/backend/auth/`, `tests/backend/knowledge/`.\r\n- Name files `test_.py` with descriptive snake_case names.\r\n- Shared test data goes in `tests/backend/fixtures/`.\r\n\r\n## File Structure\r\n\r\n```python\r\n\"\"\"One-paragraph summary of what is being tested and why.\r\n\r\nBackground\r\n----------\r\nBrief context about the feature or bug fix these tests validate.\r\n\"\"\"\r\nfrom __future__ import annotations\r\n\r\nimport pytest\r\n\r\npytestmark = [pytest.mark.backend]\r\n```\r\n\r\n## Conventions\r\n\r\n- Always add `pytestmark = [pytest.mark.backend]` at module level.\r\n- Group related tests in classes prefixed with `Test` (e.g. `TestStripImageBlocks`).\r\n- Use `pytest.mark.parametrize` for data-driven tests instead of writing repetitive cases.\r\n- Use `pytest.fixture()` for shared setup; keep fixtures close to where they are used.\r\n- Unit tests must **not** depend on Flask, network, or external services.\r\n- Mock external calls with `unittest.mock.patch` / `MagicMock`; never make real API calls.\r\n- Each test function should verify **one** behavior and have a clear name: `test__`.\r\n\r\n## Running Tests\r\n\r\n- When running pytest, use `-q` (quiet mode) instead of `-v` (verbose mode) for cleaner output.\r\n- **Preferred:** `python -m pytest tests/backend/ -q`\r\n- **Avoid:** `python -m pytest tests/backend/ -v 2>&1`\r\n\r\n## Error Handling Test Requirements\r\n\r\nApplication errors return **HTTP 200** with `status: \"error\"` in the body.\r\nOnly auth errors (`AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403) and\r\nuncontrolled transport failures (404/413/500) use non-200.\r\n\r\nWhen adding a new route or modifying error handling, include tests for:\r\n\r\n1. **AppError scenarios** — verify HTTP 200 + correct `ErrorCode` in body:\r\n ```python\r\n def test_missing_table_returns_error():\r\n with app.test_client() as client:\r\n resp = client.get('/api/tables/nonexistent')\r\n assert resp.status_code == 200 # application error → always 200\r\n data = resp.get_json()\r\n assert data[\"status\"] == \"error\"\r\n assert data[\"error\"][\"code\"] == \"TABLE_NOT_FOUND\"\r\n ```\r\n\r\n2. **Auth errors** — verify non-200 for authentication/authorization failures:\r\n ```python\r\n def test_access_denied_returns_403():\r\n resp = client.get('/api/protected-resource')\r\n assert resp.status_code == 403\r\n assert resp.get_json()[\"error\"][\"code\"] == \"ACCESS_DENIED\"\r\n ```\r\n\r\n3. **Unexpected exceptions** — verify the global handler catches them (unhandled = 500):\r\n ```python\r\n def test_unexpected_error_returns_500():\r\n with patch('module.function', side_effect=RuntimeError(\"boom\")):\r\n resp = client.get('/api/endpoint')\r\n assert resp.status_code == 500\r\n assert resp.get_json()[\"error\"][\"code\"] == \"INTERNAL_ERROR\"\r\n ```\r\n\r\n4. **Streaming endpoints** — verify error events in the NDJSON stream:\r\n ```python\r\n def test_stream_error_yields_ndjson_event():\r\n resp = client.post('/api/my-stream', ...)\r\n lines = [json.loads(l) for l in resp.data.decode().strip().split('\\n') if l]\r\n error_lines = [l for l in lines if l.get('type') == 'error']\r\n assert len(error_lines) >= 1\r\n assert error_lines[0]['error']['code'] == 'LLM_TIMEOUT'\r\n ```\r\n\r\n5. **Test fixture** — ensure `register_error_handlers(app)` is called in the test app fixture:\r\n ```python\r\n @pytest.fixture\r\n def app():\r\n from data_formulator.error_handler import register_error_handlers\r\n _app = flask.Flask(__name__)\r\n _app.config[\"TESTING\"] = True\r\n _app.register_blueprint(my_bp)\r\n register_error_handlers(_app)\r\n return _app\r\n ```\r\n\r\nSee `tests/backend/errors/test_errors.py` and `test_error_handler.py` for reference implementations.\r\n\r\n## Example\r\n\r\n```python\r\n# ❌ BAD – vague name, no parametrize, no marker\r\ndef test_it_works():\r\n assert sanitize(\"hello world\") == \"hello_world\"\r\n\r\n# ✅ GOOD\r\npytestmark = [pytest.mark.backend]\r\n\r\n@pytest.mark.parametrize(\"raw,expected\", [\r\n (\"hello world\", \"hello_world\"),\r\n (\"订单明细\", \"订单明细\"),\r\n])\r\ndef test_sanitize_preserves_unicode(raw: str, expected: str) -> None:\r\n assert sanitize(raw) == expected\r\n```\r\n",".cursor/rules/dataframe-serialization.mdc":"# DataFrame Serialization\n\nAll DataFrame-to-records conversion for API responses, streaming events, or\nfrontend-visible data MUST use the centralized helpers in\n`data_formulator.datalake.parquet_utils`:\n\n| Source type | Helper |\n|---|---|\n| `pd.DataFrame` | `df_to_safe_records(df)` |\n| `pa.Table` (Arrow) | `get_sample_rows_from_arrow(table)` |\n\n## Why\n\n`pandas.DataFrame.to_json(orient='records')` defaults to `date_format='epoch'`,\nwhich serializes datetime columns as **epoch milliseconds** (e.g. `1773532800000`).\nThe frontend interprets these as plain numbers and renders them with commas\n(`1,773,532,800,000`) instead of formatted dates.\n\n`df_to_safe_records` enforces `date_format='iso'` and `default_handler=str`,\nensuring datetimes become ISO-8601 strings and exotic types degrade gracefully.\n\n## Banned Patterns\n\n```python\n# BAD — missing date_format, datetimes become epoch numbers\njson.loads(df.to_json(orient='records'))\n\n# BAD — to_dict returns Timestamp objects, not JSON-safe values\ndf.to_dict(orient='records')\n\n# ACCEPTABLE but should be unified for consistency\njson.loads(df.to_json(orient='records', date_format='iso'))\n```\n\n## Correct Pattern\n\n```python\nfrom data_formulator.datalake.parquet_utils import df_to_safe_records\n\nrows = df_to_safe_records(df)\npreview = df_to_safe_records(df.head(5))\n```\n\n## Exceptions\n\nInternal data processing that never reaches the frontend or JSON serialization\n(e.g. Kusto SDK metadata parsing, Vega-Lite spec construction) may use\n`to_dict(orient='records')` directly.\n",".cursor/rules/dev-guides-first.mdc":"---\ndescription: Require reading dev-guides before development and keeping docs in sync\nglobs:\nalwaysApply: true\n---\n\n# Dev Guides First\n\n## Before Starting Any Development\n\nBefore implementing or designing a new feature, module, or significant change, **always\nread the relevant documents in `docs/dev-guides/`** to understand existing conventions:\n\n| Guide | When to Read |\n|-------|-------------|\n| `docs/dev-guides/1-streaming-protocol.md` | Any work on streaming endpoints or NDJSON protocol |\n| `docs/dev-guides/2-log-sanitization.md` | Any work involving logging, credentials, external services, or DataLoaders |\n| `docs/dev-guides/3-data-loader-development.md` | Any work on ExternalDataLoader, DataConnector, or connector routes |\n| `docs/dev-guides/4-authentication-oidc-tokenstore.md` | Any work on OIDC, TokenStore, AUTH_MODE, or SSO flows |\n| `docs/dev-guides/6-i18n-language-injection.md` | Any work on Agent prompts, Agent routes, backend user-visible messages, or frontend i18n |\n| `docs/dev-guides/7-unified-error-handling.md` | Any work on API errors, frontend API calls, streaming error events, or error tests |\n| `docs/dev-guides/8-path-safety.md` | Any work on backend file access, downloads, Workspace paths, Agent tools, DataLoaders, or sandbox config |\n| `docs/dev-guides/10-agent-knowledge-reasoning-log.md` | Any work on Agent knowledge injection, KnowledgeStore, reasoning logs, or experience distillation |\n| `docs/dev-guides/11-catalog-metadata-sync.md` | Any work on catalog sync, catalog_cache, catalog_annotations, metadata merge, Agent catalog tools, or frontend catalog browsing |\n| `docs/dev-guides/12-sandbox-session.md` | Any work on sandbox execution, Agent tool-calling loops, explore/execute_python code execution, or namespace management |\n| `docs/dev-guides/13-unified-row-limits.md` | Any work on row limits, data loading size caps, frontendRowLimit, MAX_IMPORT_ROWS, or DataLoader size parameter |\n| `docs/dev-guides/14-model-capability-runtime-degradation.md` | Any work on LLM Client calls, Agent LLM invocations, model capability checks, reasoning_effort, or image/vision degradation |\n| `docs/dev-guides/15-dataframe-serialization.md` | Any work on DataFrame→JSON serialization, Agent result rows, DataLoader sample_rows, or table API responses |\n| `.cursor/rules/i18n-no-hardcoded-strings.mdc` | Any work adding or changing user-visible strings in `src/` |\n\nAlso check `.cursor/rules/` and `.cursor/skills/` for related coding conventions.\n\n## After Introducing New Conventions\n\nWhen your work establishes a new development pattern, convention, or architectural\ndecision, you MUST update documentation before considering the task complete:\n\n1. **Update existing dev-guide** — if your change extends an existing convention\n (e.g. new sensitive key type → update `2-log-sanitization.md`)\n\n2. **Create new dev-guide** — if your change introduces a new cross-cutting convention\n that future developers must follow. Place in `docs/dev-guides/` with the next number prefix.\n\n3. **Update existing SKILL** — if your change affects how an existing skill works\n (e.g. new error handling pattern → update `error-handling/SKILL.md`)\n\n4. **Create new SKILL** — if your change introduces a reusable development workflow\n that AI agents should follow. Place in `.cursor/skills//SKILL.md`.\n\n5. **Update existing Rule** — if your change adds constraints to an existing rule\n (e.g. new banned pattern → update the relevant `.mdc` file)\n\n6. **Create new Rule** — if your change introduces file-scoped coding conventions\n that should be enforced automatically. Place in `.cursor/rules/.mdc`.\n\n## Documentation Sync Checklist\n\nFor every PR that introduces new patterns:\n\n- [ ] Searched `docs/dev-guides/` for related existing docs\n- [ ] Updated or created dev-guide if introducing cross-cutting conventions\n- [ ] Updated or created `.cursor/skills/` if introducing reusable workflows\n- [ ] Updated or created `.cursor/rules/` if introducing file-scoped constraints\n- [ ] New dev-guide includes a \"New Module Checklist\" section (if applicable)\n\n## File Naming Conventions\n\n- **dev-guides**: `-.md` (e.g. `3-data-loader-development.md`)\n- **skills**: `.cursor/skills//SKILL.md`\n- **rules**: `.cursor/rules/.mdc`\n",".cursor/rules/error-response-safety.mdc":"---\ndescription: Prevent information exposure through exception messages in HTTP responses\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Error Response Safety\n\nNever return raw exception text (`str(e)`, `f\"...{e}\"`) directly in HTTP responses.\nPython exceptions may contain stack traces, file paths, database connection strings,\nAPI keys, or internal IP addresses — all of which are security risks (CWE-209).\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full error handling contract.\n\n## Unified Error System\n\nNew error paths should use the `AppError` / `error_handler` infrastructure from\n`errors.py` and `error_handler.py`. Legacy `message` / `error_message` bodies are\nhistorical formats; do not add compatibility branches for them in new code.\nAll application-controlled errors return **HTTP 200** with `status: \"error\"` in the body.\nOnly auth errors (401/403) and uncontrolled transport errors (404/413/500) use non-200.\n\n### Non-streaming routes\n\nRaise `AppError` for new code or when structured error codes are useful. The\nglobal error handler returns HTTP 200 + JSON error envelope:\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\n\n# Business error — raise with appropriate code and safe message\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# LLM / external API error — use classify_and_wrap_llm_error\nfrom data_formulator.error_handler import classify_and_wrap_llm_error\n\nexcept Exception as e:\n raise classify_and_wrap_llm_error(e) from e\n```\n\n### Streaming routes\n\nUse `stream_error_event()` to yield a unified NDJSON error line.\nValidation before the stream starts returns `200 application/json` (not NDJSON):\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\nexcept AppError as e:\n yield stream_error_event(e)\nexcept Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n### Real examples from migrated endpoints\n\n```python\n# /data-agent-streaming — standard NDJSON error event\nexcept Exception as e:\n logger.error(\"Error in data-agent-streaming\", exc_info=e)\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# /generate-report-chat — same pattern\nexcept Exception as e:\n logger.exception(\"generate-report-chat failed\")\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# Validation failure before stream starts — use 200 JSON, not NDJSON\nif not request.is_json:\n return stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\"))\n```\n\n### Database / workspace errors (tables.py)\n\nUse `classify_and_raise_db_error()` instead of the legacy `sanitize_db_error_message()`:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\nexcept Exception as e:\n classify_and_raise_db_error(e) # raises AppError with safe message\n```\n\n### Connector errors (data_connector.py)\n\nUse `classify_and_raise_connector_error()` instead of the legacy `_sanitize_error()`:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e) # raises AppError with safe message\n```\n\n### Legacy functions (deprecated)\n\nThese functions still exist as wrappers but should NOT be used in new code:\n- `sanitize_db_error_message()` → use `classify_and_raise_db_error()`\n- `_sanitize_error()` → use `classify_and_raise_connector_error()`\n- `safe_error_response()` / `sanitize_error_message()` → use `raise AppError(...)`\n- `classify_llm_error()` is used internally by `classify_and_wrap_llm_error()`\n\n## Rules\n\n1. **Application errors** — HTTP 200 + `status: \"error\"`; prefer `error: {...}` for new code and never expose exception details.\n2. **Auth errors** — `AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403 (transport layer interception).\n3. **Transport errors** (uncontrolled) — only `404` (no route), `413` (body limit), `500` (unhandled crash) may be non-200.\n4. **Streaming errors** — always use `stream_error_event()` for NDJSON error lines.\n5. **Logging** — always log the full exception server-side (`logger.warning` / `logger.error`\n with `exc_info=True` when needed). The client never needs the stack trace.\n6. **Debug detail** — `AppError.detail` is only included in responses when `app.debug is True`.\n\n## Bare except Policy\n\n- **Never** use naked `except:` — always specify at least `except Exception:`\n- Use specific exception types when possible (`except (ValueError, TypeError):`)\n- Always log caught exceptions: `logger.warning(...)` for degraded operations, `logger.debug(...)` for expected failures\n- Process cleanup code (`terminate`, `join`, `close`) is exempt from logging requirements\n\n## Common Mistakes\n\n```python\n# ❌ BAD — raw exception leaks internal details\nreturn jsonify({\"message\": str(e)}), 500\nreturn jsonify({\"error\": f\"Failed: {e}\"}), 502\nyield json.dumps({\"type\": \"error\", \"error\": str(e)}) + \"\\n\"\n\n# ❌ BAD — non-200 HTTP status code for application errors\nreturn jsonify({\"status\": \"error\", \"error_message\": \"...\"}), 400\n\n# ❌ BAD — naked except catches SystemExit/KeyboardInterrupt\nexcept:\n pass\n\n# ❌ BAD — silent swallow without logging\nexcept Exception:\n pass\n\n# ✅ GOOD — non-streaming (global handler returns HTTP 200)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# ✅ GOOD — streaming\nyield stream_error_event(AppError(ErrorCode.LLM_TIMEOUT, \"Timeout\", retry=True))\n\n# ✅ GOOD — graceful degradation with logging\nexcept Exception as e:\n logger.warning(\"Could not read schema for %s\", table_name, exc_info=e)\n\n# ✅ GOOD — specific exception types\nexcept (ValueError, TypeError):\n pass # safe fallback\n```\n",".cursor/rules/frontend-test-conventions.mdc":"---\r\ndescription: Frontend TypeScript test conventions (Vitest)\r\nglobs: tests/frontend/**/*.test.{ts,tsx}\r\nalwaysApply: false\r\n---\r\n\r\n# Frontend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/frontend/unit/` mirroring the `src/` structure:\r\n - `tests/frontend/unit/data/` → tests for `src/data/`\r\n - `tests/frontend/unit/app/` → tests for `src/app/`\r\n - `tests/frontend/unit/views/` → tests for `src/views/`\r\n- Name files `.test.ts` (or `.test.tsx` for React rendering tests).\r\n\r\n## File Structure\r\n\r\n```typescript\r\nimport { describe, it, expect } from 'vitest';\r\n// For React rendering tests:\r\n// import { render } from '@testing-library/react';\r\n\r\nimport { myFunction } from '../../../../src/';\r\n\r\ndescribe('myFunction', () => {\r\n it('should handle ', () => {\r\n expect(myFunction(input)).toBe(expected);\r\n });\r\n});\r\n```\r\n\r\n## Conventions\r\n\r\n- Import `describe`, `it`, `expect` explicitly from `vitest` (globals are enabled but explicit imports improve readability).\r\n- Use `@testing-library/react` and `@testing-library/jest-dom` for component rendering tests.\r\n- Prefer testing **exported pure functions** over testing internal component state.\r\n- When component logic is complex, extract it into an exported helper and test that directly.\r\n- Group tests with `describe` blocks; use section comments (`// --- Null cases ---`) for clarity.\r\n- One assertion per `it` block when possible; name tests as `should `.\r\n- Do **not** import from `node_modules` internals; only use public API.\r\n- Keep tests independent — no shared mutable state between `it` blocks.\r\n\r\n## Example\r\n\r\n```typescript\r\n// ❌ BAD – no describe, vague test name\r\nimport { expect, test } from 'vitest';\r\ntest('works', () => { expect(fn(1)).toBe(2); });\r\n\r\n// ✅ GOOD\r\nimport { describe, it, expect } from 'vitest';\r\nimport { checkIsLikelyTextOnlyModel } from '../../../../src/views/DataLoadingThread';\r\n\r\ndescribe('checkIsLikelyTextOnlyModel', () => {\r\n it('returns true for deepseek-chat', () => {\r\n expect(checkIsLikelyTextOnlyModel('deepseek-chat')).toBe(true);\r\n });\r\n\r\n it('returns false for undefined', () => {\r\n expect(checkIsLikelyTextOnlyModel(undefined)).toBe(false);\r\n });\r\n});\r\n```\r\n",".cursor/rules/i18n-no-hardcoded-strings.mdc":"---\r\ndescription: No hardcoded UI strings — use i18n translation keys\r\nglobs: src/**/*.{ts,tsx}\r\nalwaysApply: true\r\n---\r\n\r\n# i18n: No Hardcoded UI Strings\r\n\r\nAll user-visible text in the frontend MUST go through the i18n system. Never hardcode Chinese, English, or any other language string directly in components.\r\n\r\n## How to Use\r\n\r\n```tsx\r\nimport { useTranslation } from 'react-i18next';\r\n\r\nconst { t } = useTranslation();\r\n\r\n// ✅ GOOD\r\n\r\n\r\n\r\n// ❌ BAD\r\n\r\n\r\n\r\n```\r\n\r\n## Non-Component Contexts (thunks, utilities, helpers)\r\n\r\nWhen you cannot use hooks (Redux thunks, plain `.ts` utilities, store helpers), import the i18n instance directly:\r\n\r\n```ts\r\nimport i18n from '../i18n';\r\n\r\n// ✅ GOOD — works outside React components\r\ni18n.t('messages.rowLimitReached', { count: 20000 })\r\n\r\n// ❌ BAD — hooks only work inside React function components\r\nconst { t } = useTranslation(); // will crash in a thunk\r\n```\r\n\r\nThe same rule applies: every user-visible string must use `i18n.t()` with a translation key, never a hardcoded literal.\r\n\r\n## Translation Files\r\n\r\n- English: `src/i18n/locales/en/.json`\r\n- Chinese: `src/i18n/locales/zh/.json`\r\n- Namespaces: `common`, `upload`, `chart`, `model`, `encoding`, `messages`, `navigation`, `dataLoading`, `errors`\r\n\r\nWhen adding a new key, add it to **both** `en` and `zh` locale files. Pick the namespace that fits; create a new namespace only if none applies.\r\n\r\n## What Counts as User-Visible\r\n\r\nMust use `t()`: button labels, tooltips, placeholders, error messages, dialog titles, tab names, toast notifications, table headers, empty-state text.\r\n\r\nMay stay hardcoded: log messages (`console.log`), error messages thrown but never displayed, internal constants, CSS class names, test IDs.\r\n\r\n**Cross-stack sentinel values** — strings shared between frontend and backend as identity markers (e.g. `UNTITLED_SESSION` in `dfSlice.ts` used by both Redux state and backend workspace API) must NOT be translated. They are internal constants, not display text. Translate them only at the UI rendering layer by mapping the sentinel to `t('...')`.\r\n\r\n## Backend Messages with message_code\r\n\r\nWhen the backend returns a user-visible message with a `message_code` (or `content_code`, `error_code`), use `translateBackend()` from `src/app/utils.tsx` to translate it:\r\n\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\n\r\n// Translate a backend error message\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n\r\n// Translate options with parallel code arrays\r\nconst options = translateBackendOptions(rawOptions, event.option_codes);\r\n```\r\n\r\nThe translation keys live in `messages.json` under the `agent` section. Always add **both** `en` and `zh` entries.\r\n",".cursor/rules/implementation-review-checklist.mdc":"---\ndescription: Require structural code review after implementation before final response\nglobs:\nalwaysApply: true\n---\n\n# Implementation Review Checklist\n\nAfter any substantive implementation, before the final response, review the\nuncommitted diff from a maintainer/code-review perspective. Do not only verify\nthat the feature works.\n\n## Required Checks\n\n- Inspect `git diff --stat` and relevant diffs to understand the actual blast radius.\n- Check responsibility boundaries: frontend UI, API route, service/helper, storage/session, and tests should each own the right part of the behavior.\n- Check API semantics: endpoint names, HTTP methods, response shapes, and side effects must match the product action. Similar UI actions should call the same backend API.\n- Check frontend/backend state consistency: after mutations, make sure Redux/local state, backend session/token/vault state, and subsequent list/status APIs agree.\n- Check duplication: if two paths clear the same state or perform the same API flow, extract a small helper or route instead of copying logic.\n- Check regression surfaces: refresh/reload/list/status paths must respect new flags or state, not just the primary click path.\n- Check tests: add or update focused tests for the changed contract; include at least one test for the bug/regression path when practical.\n- Check docs/rules/dev-guides when the change introduces or changes a cross-cutting convention.\n\n## Reporting\n\nIf issues are found during this review, fix clear issues before finalizing. If a\ntrade-off remains, call it out explicitly in the final response with the risk and\nwhy it was left as-is.\n",".cursor/rules/incremental-development-cadence.mdc":"---\ndescription: Enforce step-by-step implementation with design-doc checks, tests, and i18n after each step\nglobs:\nalwaysApply: true\n---\n\n# Incremental Development Cadence\n\nMulti-step tasks MUST be implemented **one step at a time**. After each logical unit (function, endpoint, component), complete all cross-cutting concerns before moving on.\n\n## After Each Step\n\n1. **Verify against design doc** — compare with `design-docs/` or issue spec; raise deviations before continuing\n2. **Add tests** — write/update tests for the just-implemented code, run and confirm green\n3. **Add i18n keys** — if user-visible text was introduced, add `en` + `zh` keys immediately\n4. **Update docs** — if a new cross-cutting convention was introduced, update dev-guide/Rule immediately\n\n## Banned Patterns\n\n- ❌ Implementing 3+ functions/endpoints before writing any tests\n- ❌ Sweeping all i18n keys in a final pass after all code is done\n- ❌ Checking design docs only after the entire implementation is complete\n- ❌ Mentioning \"tests and i18n still needed\" only in the final response\n",".cursor/rules/language-injection-conventions.mdc":"---\r\ndescription: Language injection conventions for LLM Agent prompts\r\nglobs: py-src/data_formulator/agents/**/*.py,py-src/data_formulator/routes/agents.py\r\nalwaysApply: false\r\n---\r\n\r\n# Language Injection Conventions\r\n\r\nLanguage flows per-request: `Frontend i18n → Accept-Language header → get_language_instruction() → system prompt`.\r\n\r\n## LLM Prompt Language Injection\r\n\r\n1. **User-facing LLM output** MUST inject language via `get_language_instruction(mode=...)` in the route handler.\r\n2. **Mode selection:** `\"full\"` for text-heavy agents, `\"compact\"` for code-generation agents and short-text endpoints.\r\n3. **Inject into system prompt only** — use `inject_language_instruction()` from `agent_language.py`. Never inject into user messages.\r\n4. **Do NOT inject** for non-user-facing calls (health checks, internal tool calls).\r\n5. **Do NOT duplicate** — if upstream messages already contain language instruction, skip.\r\n6. **Do NOT** use env vars, global interceptors, or hardcoded language strings (e.g. `\"回答请使用中文\"`) — route handlers should use `get_language_instruction(mode=...)`.\r\n7. **New language?** Add to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py` and add locale files in `src/i18n/locales//`.\r\n\r\n## Python-Side User-Visible Messages (message_code pattern)\r\n\r\nFor **fixed strings in Python** that are shown in the UI (error messages, clarify options, completion summaries), do NOT translate in Python. Instead:\r\n\r\n1. Keep the English string as the default value.\r\n2. Add a `message_code` (or `content_code`, `error_code`, `summary_code`) field with a key like `\"agent.someKey\"`.\r\n3. Optionally add `message_params` for interpolation.\r\n4. The frontend translates using `translateBackend(fallback, code, params)` from `src/app/utils.tsx`.\r\n5. Add the translation key to `src/i18n/locales/{en,zh}/messages.json` under the `agent` section.\r\n\r\n```python\r\n# ✅ GOOD — backend returns code, frontend translates\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\",\r\n \"message_code\": \"agent.emptyDataframe\",\r\n}\r\n\r\n# ❌ BAD — backend-side translation dict\r\nyield {\"message\": translate_in_python(\"empty_df\", lang)}\r\n```\r\n\r\nFor detailed architecture and anti-pattern explanations, see `docs/dev-guides/6-i18n-language-injection.md`.\r\n",".cursor/rules/log-sanitization.mdc":"---\ndescription: Prevent sensitive data leakage in server-side log messages\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Log Sanitization\n\nServer-side logs must never contain passwords, tokens, API keys, or other credentials\nin plain text. This rule complements `error-response-safety.mdc` (which covers client\nresponses) by addressing the **logging** side.\n\n## Architecture: Defense in Depth\n\nTwo layers protect against sensitive data in logs:\n\n1. **Layer 1 — Explicit utilities** (call-site): developers use `sanitize_url()`,\n `sanitize_params()`, `redact_token()` from `security/log_sanitizer.py`.\n2. **Layer 2 — SensitiveDataFilter** (global): a `logging.Filter` registered on all\n handlers in `configure_logging()` auto-redacts patterns that slip through.\n\nLayer 1 is the primary defense; Layer 2 is a safety net. Both are required.\n\n## When to Use Each Utility\n\n| Data Type | Utility | Example |\n|-----------|---------|---------|\n| `dict` with password/secret/token keys | `sanitize_params(params)` | `log.info(\"Init: %s\", sanitize_params(params))` |\n| URL (may have `user:pass@host` or sensitive query params) | `sanitize_url(url)` | `logger.info(\"Connecting to %s\", sanitize_url(url))` |\n| Bearer / access token | `redact_token(token)` | `logger.debug(\"Token: %s\", redact_token(token))` |\n| Normal string (no secrets) | Nothing needed | `logger.info(\"Processing table %s\", name)` |\n\n## Rules\n\n1. **Never log raw `params` / `config` dicts** that may contain `password`, `secret`,\n `api_key`, `token`, or `connection_string` keys. Always use `sanitize_params()`.\n\n2. **Never log full tokens or API keys.** Use `redact_token()` if you need to identify\n a token in logs.\n\n3. **Prefer `type(exc).__name__`** over `str(exc)` in warning/info logs when the\n exception may contain connection strings or upstream response bodies. Use\n `exc_info=True` for full tracebacks at ERROR level (the filter sanitizes `exc_text`).\n\n4. **URL logging** — use `sanitize_url()` for any URL from configuration or\n environment variables (issuer URLs, JWKS URLs, database URLs, API base URLs).\n It masks embedded credentials and sensitive query parameters such as\n `password`, `client_secret`, `token`, and `api_key`.\n\n5. **Use `%s`-style formatting** instead of f-strings for log messages. This allows\n the filter to intercept arguments before formatting and provides better\n compatibility with log aggregation tools.\n\n## Common Mistakes\n\n```python\n# BAD — dict may contain password\nlog.info(f\"Connecting with {params}\")\nlog.info(\"Connecting with %s\", params)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_params\nlog.info(\"Connecting with %s\", sanitize_params(params))\n\n# BAD — URL may embed credentials\nlogger.warning(\"Failed for %s\", discovery_url)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_url\nlogger.warning(\"Failed for %s\", sanitize_url(discovery_url))\n\n# BAD — exception may leak connection string or upstream body\nlogger.warning(\"Error: %s\", exc)\n\n# GOOD — log type name at warning level, full traceback at error level\nlogger.warning(\"Error: %s\", type(exc).__name__)\nlogger.error(\"Unexpected failure\", exc_info=True)\n\n# BAD — full token in logs\nlogger.debug(\"Using token %s\", token)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import redact_token\nlogger.debug(\"Using token %s\", redact_token(token))\n```\n\n## Disabling for Local Debug\n\nSet `LOG_SANITIZE=false` to disable the global filter during local debugging.\nThis should **never** be used in production or CI.\n\n## Adding New Sensitive Keys\n\nIf your module introduces a new credential key name (e.g. `custom_auth_token`),\nadd it to `SENSITIVE_KEYS` in `security/log_sanitizer.py` and add a corresponding\nregex pattern to `_SENSITIVE_KEY_NAMES` if needed.\n\n## New Module Checklist\n\nWhen creating a new module or feature that handles credentials or external services:\n\n- [ ] Audit all `logger.*()` calls — do any log connection params, URLs, or tokens?\n- [ ] Use `sanitize_params()` for any dict that may contain credentials\n- [ ] Use `sanitize_url()` for any URL from config/env\n- [ ] Use `redact_token()` for any token/key value\n- [ ] If introducing new credential key names, update `SENSITIVE_KEYS`\n- [ ] Run `python -m pytest tests/backend/security/test_log_sanitizer.py` to verify\n",".cursor/rules/package-manager-conventions.mdc":"---\r\ndescription: Use Yarn only, never npm/pnpm\r\nglobs: package.json, yarn.lock\r\nalwaysApply: true\r\n---\r\n\r\n# Package Manager Rules\r\n\r\n- Use Yarn v1.22.22 only - never use npm or pnpm\r\n- Never manually edit yarn.lock\r\n- Keep yarn.lock changes minimal when adding deps\r\n- Registry must be https://registry.yarnpkg.com\r\n",".cursor/rules/path-safety.mdc":"---\r\ndescription: 路径安全编码规范 — 编辑路由/Agent/Loader/Workspace/知识库代码时自动提醒\r\nglobs: py-src/data_formulator/{routes,agents,data_loader,datalake,security,knowledge}/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# 路径安全编码规范\r\n\r\n编辑此目录下的文件时，请遵守以下安全规则（详见 `docs/dev-guides/8-path-safety.md` 和 `.cursor/skills/path-safety/SKILL.md`）：\r\n\r\n## ConfinedDir 是唯一的路径约束原语\r\n\r\n所有接受用户可控路径的代码，**必须**通过 `ConfinedDir` 实例访问文件系统。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\njail = ConfinedDir(root_dir, mkdir=False)\r\ntry:\r\n target = jail.resolve(user_input)\r\nexcept ValueError:\r\n return {\"error\": \"Access denied\"}\r\n```\r\n\r\n## 禁止的模式\r\n\r\n```python\r\n# ❌ BAD — 裸路径拼接\r\ntarget = Path(root) / user_input\r\n\r\n# ❌ BAD — 手写 resolve + relative_to（已弃用，必须用 ConfinedDir）\r\ntarget = (root / user_input).resolve()\r\ntry:\r\n target.relative_to(root)\r\nexcept ValueError: ...\r\n\r\n# ❌ BAD — 手写 is_relative_to（已弃用，必须用 ConfinedDir）\r\nresolved = (root / user_input).resolve()\r\nif not resolved.is_relative_to(root.resolve()): ...\r\n\r\n# ❌ BAD — 路径比较用 str.startswith\r\nif str(target).startswith(str(root)): ...\r\n```\r\n\r\n## 正确的模式\r\n\r\n```python\r\n# ✅ GOOD — Agent 工具\r\nworkspace_jail = self.workspace.confined_root\r\nscratch_jail = self.workspace.confined_scratch\r\ntarget = workspace_jail.resolve(rel_path)\r\n\r\n# ✅ GOOD — 文件下载 route\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target)\r\n\r\n# ✅ GOOD — 文件上传（两层防御）\r\nsafe_name = secure_filename(raw_filename)\r\ntarget = scratch_jail.resolve(safe_name)\r\n\r\n# ✅ GOOD — Workspace data 文件\r\npath = workspace.get_file_path(filename) # 内部使用 ConfinedDir\r\n```\r\n\r\n## 部署守卫\r\n\r\n- 新增读取宿主文件系统的 Loader 必须在 `_enforce_deployment_restrictions()` 注册多用户禁用\r\n- 多用户模式（`WORKSPACE_BACKEND != \"local\"`）必须启用沙箱\r\n",".cursor/rules/test-driven-workflow.mdc":"---\r\ndescription: 测试优先开发与测试保护规则。确保新功能/Bug 修复必须有测试覆盖，测试失败时先诊断后修改。\r\nglobs: \r\nalwaysApply: true\r\n---\r\n\r\n# Test-Driven Workflow\r\n\r\n## 1. 测试优先：新功能 / Bug 修复必须有测试\r\n\r\n开发新功能或修复 Bug 时，**必须同步编写测试**，不能只提交业务代码：\r\n\r\n- **新功能** → 先写测试（或至少同步写），描述期望行为，再实现代码使测试通过\r\n- **Bug 修复** → 先写复现测试，确认测试失败，再修复代码使测试通过\r\n- **重构** → 确保重构前已有测试覆盖，重构后测试仍然全部通过\r\n\r\n完成实现后，主动运行相关测试验证：\r\n- 后端：`python -m pytest tests/backend/ -q`\r\n- 前端：`yarn test`\r\n\r\n## 2. 测试保护：不要轻易修改测试代码\r\n\r\n测试失败时，**禁止直接修改测试代码来让测试通过**。必须执行以下流程：\r\n\r\n1. **复现并定位**：运行失败的测试，确认失败现象\r\n2. **诊断分类**：\r\n - A：测试本身有问题（断言错误、setup 不正确）\r\n - B：业务实现有 Bug\r\n - C：需求/规格变更导致测试过时\r\n3. **给出方案**：至少提供两个可选方案（改测试 / 改实现 / 补规格），说明各自影响和风险\r\n4. **等待确认**：由用户决定采用哪个方案，再执行修改\r\n\r\n**绝对禁止**：\r\n- 测试一红就直接改测试代码\r\n- 删除或跳过（`@pytest.mark.skip`）失败的测试来\"解决\"问题\r\n- 修改测试的断言值来匹配错误的实现\r\n\r\n## 3. 精准修改：不动无关测试\r\n\r\n与 Karpathy 编码准则的\"精准修改\"一致：\r\n\r\n- 不\"顺手改进\"与当前任务无关的测试代码\r\n- 不重构正在通过的测试\r\n- 不改变已有测试的断言逻辑，除非被明确要求\r\n",".cursor/rules/unified-error-protocol.mdc":"---\ndescription: Unified error handling protocol for backend and frontend\nglobs: py-src/**/*.py,src/**/*.{ts,tsx}\nalwaysApply: false\n---\n\n# Unified Error Protocol\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full developer guide.\n\n## HTTP Status Code Policy\n\n**All application-controlled errors return HTTP 200** with `status: \"error\"` in the body.\nThis applies to both non-streaming JSON APIs and streaming pre-flight errors.\n\nOnly these scenarios use non-200:\n- `401` — `AUTH_REQUIRED` / `AUTH_EXPIRED` (so transport layer can intercept)\n- `403` — `ACCESS_DENIED` (so transport layer can intercept)\n- `404` — Flask routing: no matching route handler\n- `413` — WSGI: request body exceeds `MAX_CONTENT_LENGTH`\n- `500` — Unhandled exception that escaped all error handling\n\n**Streaming NDJSON APIs:**\n- Pre-stream validation failure: HTTP `200` + `application/json` (use `stream_preflight_error()`)\n- In-stream error: NDJSON `{\"type\": \"error\", \"error\": {...}}`\n- In-stream warning: NDJSON `{\"type\": \"warning\", \"warning\": {...}}`\n\n## Backend: Non-streaming JSON Format\n\n```jsonc\n// Success (HTTP 200)\n{\"status\": \"success\", \"data\": {...}}\n\n// Error (HTTP 200 for business errors, 401/403 for auth errors)\n{\n \"status\": \"error\",\n \"error\": {\n \"code\": \"TABLE_NOT_FOUND\",\n \"message\": \"Table not found\",\n \"retry\": false,\n \"detail\": \"...\" // Only in debug mode\n }\n}\n```\n\n## Backend: Streaming NDJSON Format\n\nAll streaming endpoints use `application/x-ndjson`. Each line is one JSON object:\n\n```jsonc\n{\"type\": \"text_delta\", \"data\": {...}}\n{\"type\": \"error\", \"error\": {\"code\": \"LLM_TIMEOUT\", \"message\": \"...\", \"retry\": true}}\n{\"type\": \"done\", \"data\": {...}}\n```\n\n## Backend: How to Handle Errors\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok, stream_preflight_error\nfrom data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n\n# Non-streaming: success\nreturn json_ok(data)\n\n# Non-streaming: error (global handler returns HTTP 200, auth errors get 401/403)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# Streaming: pre-flight error (always HTTP 200)\nreturn stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Bad input\"))\n\n# Streaming: in-stream error\nyield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n## Frontend: How to Consume APIs\n\n```typescript\nimport { apiRequest, streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\n// Non-streaming — apiRequest checks body.status (main path) and HTTP status (auth/infra)\ntry {\n const { data } = await apiRequest(url, options);\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n\n// Streaming — streamRequest handles pre-flight JSON errors and NDJSON events\ntry {\n for await (const event of streamRequest(url, options, signal)) {\n if (event.type === 'error') { /* handle inline */ }\n // ...process events...\n }\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n```\n\nAPI consumers MUST use `apiRequest()` / `streamRequest()` plus `handleApiError()`.\nDirect `fetchWithIdentity()` is reserved for lower-level clients and explicit protocol\nexceptions such as file downloads, blob/CSV responses, SPA/OIDC redirects, or third-party URLs.\n\n## Prohibited Patterns\n\n### Backend\n- ❌ `return jsonify({\"status\": \"ok\", \"data\": data})` — use `json_ok(data)` instead\n- ❌ `return jsonify({\"error_message\": str(e)})` — use `raise AppError(...)` instead\n- ❌ `yield json.dumps({\"type\": \"error\", \"error\": str(e)})` — use `stream_error_event()`\n- ❌ Bare `except:` or `except Exception: pass` — always log or propagate\n- ❌ `yield 'error: ' + json.dumps(...)` — non-standard prefix\n\n### Frontend\n- ❌ `.catch(() => {})` — either add a comment explaining why (best-effort), or use `handleApiError`\n- ❌ `.catch(console.error)` without user feedback — dispatch `addMessages` or use `handleApiError`\n- ❌ Raw `fetch()` for `/api/` URLs — use `fetchWithIdentity` (identity headers + 401 retry)\n- ❌ RTK thunk without `.rejected` handler — always add `.addCase(thunk.rejected, ...)` with `addMessages`\n- ❌ New code using `fetchWithIdentity().json()` — use `apiRequest()` instead\n\n### RTK Thunk Rejected Handler Pattern\n\n```typescript\n.addCase(myThunk.rejected, (state, action) => {\n if (action.error?.name !== 'AbortError') {\n state.messages.push({\n timestamp: Date.now(), type: 'warning',\n component: 'my-component',\n value: 'Human-readable error description',\n });\n }\n})\n```\n\n## Migrated Streaming Endpoints\n\nAll streaming endpoints now use `application/x-ndjson` and `stream_error_event()`:\n\n| Endpoint | Old format | New format | Status |\n|----|-----|---|-----|\n| `/data-agent-streaming` | `{status:\"error\", error_message}` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/get-recommendation-questions` | `error: {json}` prefix + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/generate-report-chat` | `data: {json}` SSE prefix + `text/event-stream` | Pure NDJSON + `application/x-ndjson` | ✅ Done |\n| `/data-loading-chat` | `str(e)` in error field | `stream_error_event()` safe message | ✅ Done |\n| `/clean-data-stream` | `\\n{json}\\n` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n\n## Complete ErrorCode Reference\n\n```\nAUTH_REQUIRED, AUTH_EXPIRED, ACCESS_DENIED\nINVALID_REQUEST, TABLE_NOT_FOUND, FILE_PARSE_ERROR, FILE_TOO_LARGE, VALIDATION_ERROR\nLLM_AUTH_FAILED, LLM_RATE_LIMIT, LLM_CONTEXT_TOO_LONG, LLM_MODEL_NOT_FOUND\nLLM_TIMEOUT, LLM_SERVICE_ERROR, LLM_CONTENT_FILTERED, LLM_UNKNOWN_ERROR\nDB_CONNECTION_FAILED, DB_QUERY_ERROR, DATA_LOAD_ERROR, CONNECTOR_ERROR\nCODE_EXECUTION_ERROR, AGENT_ERROR\nCATALOG_SYNC_TIMEOUT, CATALOG_NOT_FOUND, ANNOTATION_CONFLICT, ANNOTATION_INVALID_PATCH\nINTERNAL_ERROR, SERVICE_UNAVAILABLE\n```\n\n## Adding a New Error Code\n\n1. Add constant to `py-src/data_formulator/errors.py` `ErrorCode` class (no HTTP mapping needed, defaults to 200)\n2. Add i18n mapping to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`\n3. Add translations to `src/i18n/locales/en/errors.json` AND `zh/errors.json`\n\n## New Endpoint Checklist\n\nWhen adding a new API endpoint, verify:\n- [ ] Success: `json_ok(data)` → `{\"status\": \"success\", \"data\": ...}`, HTTP 200\n- [ ] Error: `raise AppError(...)` → HTTP 200 + error body (auth errors get 401/403)\n- [ ] Streaming pre-flight: `stream_preflight_error(...)` → HTTP 200 + `application/json`\n- [ ] Streaming in-stream: `stream_error_event(...)` via NDJSON\n- [ ] No `str(e)` in any response body\n- [ ] No naked `except:` — use specific exception types or `except Exception as e:` with logging\n- [ ] Frontend uses `apiRequest()` (non-streaming) or `streamRequest()` (streaming)\n- [ ] Error test cases assert HTTP 200 + `body.status == \"error\"` (auth: 401/403)\n",".cursor/skills/error-handling/SKILL.md":"---\nname: error-handling\ndescription: 统一错误处理系统。在添加 API 端点、修改错误处理、添加前端 API 调用、编写错误相关测试时使用。\n---\n\n# Error Handling Skill\n\nUnified error handling system for DF. Use when adding API endpoints, modifying error handling, or adding frontend API calls.\n\n> **Prerequisites**: Read `docs/dev-guides/7-unified-error-handling.md` before changing API error behavior.\n> Read `docs/dev-guides/2-log-sanitization.md` when the work involves logging, credentials, external services, or DataLoaders.\n> If your work introduces new error handling patterns or conventions, update this file and related dev-guides accordingly.\n\n## Architecture Overview\n\n```\nFrontend Backend\n──────── ───────\napiClient.ts errors.py\n├── apiRequest() ←── JSON ──── ├── ErrorCode (enum)\n├── streamRequest() ←── NDJSON ── └── AppError (exception)\n└── parseStreamLine()\n error_handler.py\nerrorCodes.ts ├── register_error_handlers(app)\n└── getErrorMessage() ├── classify_and_wrap_llm_error()\n └── stream_error_event()\nerrorHandler.ts\n└── handleApiError() security/sanitize.py\n └── classify_llm_error() (internal)\nMessageSnackbar ← dfSlice.messages\n```\n\n## Protocol Snapshot\n\nUse this contract for all new or reworked DF APIs:\n\n| Scenario | HTTP | Shape |\n|----------|------|-------|\n| Non-streaming success | `200` | `{\"status\": \"success\", \"data\": ...}` |\n| Non-streaming business/validation error | `200` | `{\"status\": \"error\", \"error\": {\"code\", \"message\", \"retry\", \"request_id\"}}` |\n| Non-streaming auth/authorization error | `401` / `403` | same structured error body |\n| Streaming preflight error | `200` | `application/json` + `{\"status\": \"error\", \"error\": ...}` |\n| Streaming in-flight fatal error | `200` | NDJSON line: `{\"type\": \"error\", \"error\": ...}` |\n| No Flask route / too large / unhandled crash | `404` / `413` / `500` | transport-level error |\n\nDo not use HTTP `400`/`422` for application validation errors in new code.\nDo not convert in-flight NDJSON errors to `status: \"error\"`; once the stream has\nstarted, event `type` is the protocol discriminator.\n\n## Backend: Adding a New API Endpoint\n\n### HTTP Status Code Policy\n\n**Application-controlled business and validation errors return HTTP 200** with\n`status: \"error\"` in the body. Only these use non-200:\n- `401`/`403` — auth errors (`AUTH_REQUIRED`, `AUTH_EXPIRED`, `ACCESS_DENIED`)\n- `404` — no matching Flask route\n- `413` — WSGI body limit exceeded\n- `500` — unhandled exception (program bug)\n\n### Non-streaming endpoint\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok\n\n@bp.route('/my-endpoint', methods=['POST'])\ndef my_endpoint():\n content = request.get_json()\n if not content.get('required_field'):\n raise AppError(ErrorCode.INVALID_REQUEST, \"Missing required_field\")\n\n try:\n result = do_work(content)\n except SomeBusinessError as e:\n raise AppError(ErrorCode.DATA_LOAD_ERROR, \"Failed to load data\") from e\n except Exception as e:\n from data_formulator.error_handler import classify_and_wrap_llm_error\n raise classify_and_wrap_llm_error(e) from e\n\n return json_ok(result)\n# Global handler returns: HTTP 200 + {\"status\": \"error\", \"error\": {code, message, retry}}\n# Auth errors (AUTH_REQUIRED/AUTH_EXPIRED/ACCESS_DENIED) return 401/403\n```\n\nLegacy `{\"status\": \"error\", \"message\": \"...\"}`, `error_message`, bare `{error}`,\nand `status: \"ok\"` responses are historical formats. Do not add new compatibility\nbranches for them; migrate the route to `json_ok()` / `AppError` before using\n`apiRequest()`.\n\n### Streaming endpoint\n\nValidation MUST be outside the generator. Failures return 200 JSON (not NDJSON).\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\n@bp.route('/my-stream', methods=['POST'])\ndef my_stream():\n if not request.is_json:\n return stream_preflight_error(\n AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\")\n )\n\n content = request.get_json()\n client = get_client(content['model'])\n\n def generate():\n try:\n for event in agent.run(...):\n yield json.dumps(event, ensure_ascii=False) + \"\\n\"\n except Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n return Response(stream_with_context(generate()), mimetype='application/x-ndjson')\n```\n\nStreaming runtime errors intentionally use `{\"type\": \"error\", \"error\": ...}`.\nThey cannot use a top-level `status` envelope because the HTTP response and NDJSON\nevent stream have already started.\n\n## Frontend: Consuming an API\n\n### Non-streaming\n\n```typescript\nimport { apiRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n const { data } = await apiRequest(getUrls().MY_ENDPOINT, {\n method: 'POST',\n body: JSON.stringify(payload),\n headers: { 'Content-Type': 'application/json' },\n });\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\nFor UI loading state, model the request lifecycle explicitly with\n`LoadableState` from `src/app/loadableState.ts`. Do not infer loading from\nmissing data (`!data`), because failed requests may legitimately leave data\nempty while loading has ended.\n\n### Streaming\n\n```typescript\nimport { streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n for await (const event of streamRequest(url, options, abortController.signal)) {\n switch (event.type) {\n case 'text_delta':\n break;\n case 'error':\n // Error arrived mid-stream — show inline in component.\n break;\n case 'done':\n break;\n }\n }\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\n### With callbacks\n\n```typescript\nhandleApiError(e, 'MyComponent', {\n onAuth: () => redirectToLogin(), // AUTH_REQUIRED / AUTH_EXPIRED\n onRetryable: () => retryOperation(), // LLM_RATE_LIMIT / LLM_TIMEOUT\n silent: true, // don't show Snackbar (component handles display)\n});\n```\n\n### Migration and special cases\n\nDF API consumers should use `apiRequest()` / `streamRequest()` and\n`handleApiError()`. Direct `fetchWithIdentity()` is for lower-level client helpers\nand explicit protocol exceptions such as file downloads, blob/CSV responses,\nOIDC redirects, SPA fallback, or third-party URLs.\n\nDo not apply the normal JSON API protocol mechanically to file downloads / CSV\nstreaming, SPA fallback, OIDC redirect flows, frontend fetches to third-party\nURLs, or errors after a streaming response has already started. Check the route's\nprotocol first, then preserve safe error bodies and avoid `str(exc)` exposure.\n\n## Adding a New Error Code\n\n1. **Backend** — Add to `py-src/data_formulator/errors.py` `ErrorCode`:\n ```python\n MY_NEW_ERROR = \"MY_NEW_ERROR\"\n ```\n No HTTP mapping needed — defaults to HTTP 200. Only add to `ERROR_CODE_HTTP_STATUS` if it's an auth code.\n\n2. **Frontend mapping** — Add to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`:\n ```typescript\n MY_NEW_ERROR: 'errors.myNewError',\n ```\n\n3. **Translations** — Add to both locale files:\n - `src/i18n/locales/en/errors.json`: `\"myNewError\": \"English message\"`\n - `src/i18n/locales/zh/errors.json`: `\"myNewError\": \"中文消息\"`\n\n## Migrated Endpoints Reference\n\nAll streaming endpoints are now on the unified protocol:\n\n| Endpoint | Format | Notes |\n|----------|--------|-------|\n| `/data-agent-streaming` | NDJSON + `stream_error_event()` | Emits top-level `type` events; errors use `{type:\"error\", error:{...}}` |\n| `/get-recommendation-questions` | NDJSON + `stream_error_event()` | Was `error: {json}` prefix |\n| `/generate-report-chat` | Pure NDJSON + `stream_error_event()` | Was SSE `data: {json}` prefix |\n| `/data-loading-chat` | NDJSON + `stream_error_event()` | `str(e)` removed |\n| `/clean-data-stream` | NDJSON + `stream_error_event()` | Was `\\n{json}\\n` format |\n\nNon-streaming endpoints:\n\n| Endpoint | Error Format | Notes |\n|----------|-------------|-------|\n| `/chart-insight` | `AppError` → HTTP 200 + `{status:\"error\", error:{code,message,retry}}` | Fully migrated. Frontend uses `fetchChartInsight` rejected reducer. |\n| All migrated endpoints | `AppError` → HTTP 200 + unified error body | credentials, knowledge, sessions, tables, agents |\n| `/derive-data`, `/refine-data`, `/sort-data`, `/process-data-on-load`, `/test-model` | `json_ok()` / `AppError` | Migrated to new format |\n\n## Empty Catch Policy\n\nNot all `.catch(() => {})` are bugs. Use this decision tree:\n\n1. **User-initiated action** (delete, refresh, submit) → must notify with `addMessages` or `handleApiError()`\n2. **Background/best-effort fetch** (connector list on mount, session list) → OK to swallow, but add a comment\n3. **RTK thunks** → always add `.rejected` handler with `addMessages`\n4. **AbortError** → filter out with `if (action.error?.name !== 'AbortError')`\n\n## Frontend Stream Parsing Pattern\n\nWhen consuming a migrated streaming endpoint, handle the current NDJSON event\nformat directly:\n\n```typescript\nconst data = JSON.parse(line);\nif (data.type === 'error') {\n const errMsg = data.error?.message || 'Unknown error';\n // show to user...\n}\n```\n\n## Backend: Database/Workspace Errors (tables.py)\n\nFor table CRUD endpoints, use the specialized classifier:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\n@tables_bp.route('/my-table-op', methods=['POST'])\ndef my_table_op():\n try:\n result = workspace.do_something()\n return jsonify({\"status\": \"success\", \"data\": result})\n except Exception as e:\n classify_and_raise_db_error(e)\n```\n\n`classify_and_raise_db_error` maps common DB errors to appropriate `AppError` codes\n(returned as HTTP 200 by the global handler, except ACCESS_DENIED → 403):\n- \"Table does not exist\" → `TABLE_NOT_FOUND` (HTTP 200)\n- \"Table already exists\" → `INVALID_REQUEST` (HTTP 200)\n- \"Permission denied\" → `ACCESS_DENIED` (HTTP 403)\n- Other → `CONNECTOR_ERROR` (HTTP 200)\n\n## Backend: Connector Errors (data_connector.py)\n\nFor connector endpoints, use:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e, operation=\"preview\")\n```\n\nConnector/DataLoader classification is intentionally simple and lives in\n`data_formulator.data_loader.connector_errors`. It maps common failures to a\nsmall stable set: `INVALID_REQUEST`, `CONNECTOR_AUTH_FAILED`, `AUTH_EXPIRED`,\n`ACCESS_DENIED`, `DB_CONNECTION_FAILED`, `DB_QUERY_ERROR`, `DATA_LOAD_ERROR`,\nor `CONNECTOR_ERROR`. Do not add endpoint-local string matching unless the\nclassifier cannot reasonably cover the category.\n\nAll JSON errors include `error.request_id` and an `X-Request-Id` response\nheader. Show/copy this ID for users when reporting backend failures; do not\nshow raw exception text in production. Unhandled 500 responses must never\ninclude raw tracebacks, even in debug mode; return a safe category plus\n`request_id` and keep full stack traces in server logs only.\n\n## Debugging Error Propagation\n\nWhen an error isn't reaching the frontend:\n\n1. **Check backend logs** — is the error logged?\n2. **Check response format**:\n - Non-streaming: `{\"status\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n - Streaming: one line `{\"type\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n3. **Check Content-Type** — streaming must be `application/x-ndjson`, not `application/json` or `text/event-stream`\n4. **Check frontend parser** — is the consumer looking for `data.type === 'error'`?\n5. **Check global handler** — verify `register_error_handlers(app)` is called in `app.py`\n6. **Check blueprint handlers** — blueprint-level `errorhandler(Exception)` takes priority over global handlers\n\nLegacy `message` / `error_message` bodies are protocol violations on migrated API paths.\n\n## Log Sanitization (Sensitive Data in Server Logs)\n\nServer-side logs must never leak passwords, tokens, API keys, or connection strings.\nThe project uses a defense-in-depth approach with two layers.\n\n### Layer 1: Explicit Utilities (call-site)\n\n```python\nfrom data_formulator.security.log_sanitizer import (\n sanitize_url, sanitize_params, redact_token,\n)\n\n# Dict with credentials → sanitize_params()\nlog.info(\"Connecting with: %s\", sanitize_params(params))\n\n# URL that may embed credentials → sanitize_url()\nlogger.info(\"Issuer: %s\", sanitize_url(issuer_url))\n\n# Token/API key → redact_token()\nlogger.debug(\"Token: %s\", redact_token(token))\n```\n\n### Layer 2: SensitiveDataFilter (global safety net)\n\nRegistered in `app.py:configure_logging()`. Automatically redacts:\n- URL credentials (`://user:pass@host`)\n- `Bearer` tokens\n- `password=xxx`, `api_key=xxx`, `secret=xxx` patterns\n- JWT-like base64 strings\n- Python dict repr with sensitive keys\n\nDisable with `LOG_SANITIZE=false` for local debugging only.\n\n### When to Use What\n\n| Data | Utility | Why not just filter? |\n|------|---------|---------------------|\n| `dict` with password keys | `sanitize_params()` | Filter can't identify arbitrary password values in dict repr |\n| URL from config/env | `sanitize_url()` | Explicit is clearer; filter is backup |\n| Token/key value | `redact_token()` | Explicit is clearer; filter is backup |\n| Normal text | Nothing | Filter handles edge cases |\n\n### New Module Checklist\n\nWhen adding a module that handles credentials or external services:\n\n1. Audit all `logger.*()` calls for credential/URL/token logging\n2. Use `sanitize_params()` for dicts, `sanitize_url()` for URLs, `redact_token()` for tokens\n3. Prefer `type(exc).__name__` over `str(exc)` in warning-level logs\n4. If introducing new credential key names, add to `SENSITIVE_KEYS` in `log_sanitizer.py`\n\n## Key Files\n\n| File | Purpose |\n|------|---------|\n| `py-src/data_formulator/errors.py` | `ErrorCode` enum + `AppError` exception |\n| `py-src/data_formulator/error_handler.py` | Global handlers, `classify_and_wrap_llm_error`, `stream_error_event` |\n| `py-src/data_formulator/security/log_sanitizer.py` | `sanitize_url`, `sanitize_params`, `redact_token`, `SensitiveDataFilter` |\n| `py-src/data_formulator/routes/tables.py` | `classify_and_raise_db_error` (database/workspace errors) |\n| `py-src/data_formulator/data_connector.py` | `classify_and_raise_connector_error` (connector errors) |\n| `py-src/data_formulator/security/sanitize.py` | `classify_llm_error` (internal), `sanitize_error_message` |\n| `src/app/apiClient.ts` | `apiRequest`, `streamRequest`, `parseStreamLine`, `ApiRequestError` |\n| `src/app/errorHandler.ts` | `handleApiError` |\n| `src/app/errorCodes.ts` | `ERROR_CODE_I18N_MAP`, `getErrorMessage` |\n| `src/i18n/locales/{en,zh}/errors.json` | Error message translations |\n",".cursor/skills/language-injection/SKILL.md":"---\r\nname: language-injection\r\ndescription: LLM Agent 多语言注入规范。在修改 Agent 提示词、添加新的 Agent 端点、处理用户可见的后端消息（message_code）时使用。\r\n---\r\n\r\n# Language Injection for Agent Prompts\r\n\r\nAuthoritative developer guide: `docs/dev-guides/6-i18n-language-injection.md`.\r\n\r\n> **Prerequisites**: Read `docs/dev-guides/6-i18n-language-injection.md` before changing Agent prompts, Agent routes, backend user-visible messages, or frontend i18n strings.\r\n> If your work introduces new language injection patterns or conventions, update this file and related dev-guides accordingly.\r\n\r\n## Architecture\r\n\r\n```\r\nFrontend i18n.language → Accept-Language header → get_language_instruction()\r\n │\r\n build_language_instruction()\r\n (agents/agent_language.py)\r\n │\r\n ┌────────────┴────────────┐\r\n ▼ ▼\r\n mode=\"full\" mode=\"compact\"\r\n (text-heavy agents) (code-gen agents)\r\n```\r\n\r\n### Core Modules\r\n\r\n| Module | Role |\r\n|--------|------|\r\n| `agents/agent_language.py` | `build_language_instruction(lang, mode)` — generates prompt fragments; `inject_language_instruction()` — injects into system prompts; supports 20 languages; returns `\"\"` for English |\r\n| `routes/agents.py` → `get_language_instruction()` | Reads `Accept-Language` header, delegates to `build_language_instruction` |\r\n| `routes/agents.py` → `_get_ui_lang()` | Extracts primary language code from `Accept-Language` header |\r\n| `src/app/utils.tsx` → `fetchWithIdentity()` | Sets `Accept-Language` header on every API request from `i18n.language` |\r\n| `src/app/utils.tsx` → `translateBackend()` | Translates backend `message_code` / `content_code` using frontend i18n |\r\n\r\n## Code Examples\r\n\r\n### Route handler — inject language\r\n\r\n```python\r\n# In a Flask route handler:\r\nlang_instruction = get_language_instruction(mode=\"compact\")\r\nlang_suffix = f\"\\n\\n{lang_instruction}\" if lang_instruction else \"\"\r\n\r\nmessages = [\r\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\" + lang_suffix},\r\n {\"role\": \"user\", \"content\": user_input},\r\n]\r\n```\r\n\r\n### Agent constructor — use inject_language_instruction()\r\n\r\n```python\r\nfrom data_formulator.agents.agent_language import inject_language_instruction\r\n\r\n# Simple append (most agents)\r\nsystem_prompt = inject_language_instruction(system_prompt, language_instruction)\r\n\r\n# Insert before a marker (complex prompts)\r\nsystem_prompt = inject_language_instruction(\r\n system_prompt, language_instruction,\r\n marker=\"**About the execution environment:**\"\r\n)\r\n```\r\n\r\n### Python-side user-visible messages — message_code pattern\r\n\r\nFor fixed strings in Python that appear in the UI, do NOT translate in Python.\r\nReturn a `message_code` and let the frontend translate:\r\n\r\n```python\r\n# In an Agent or route handler:\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\", # English fallback\r\n \"message_code\": \"agent.emptyDataframe\", # frontend i18n key\r\n}\r\n\r\n# With parameters:\r\nresult = {\r\n \"status\": \"error\",\r\n \"content\": f\"Fields not found: {missing}\",\r\n \"content_code\": \"agent.fieldsNotFound\",\r\n \"content_params\": {\"missing\": missing, \"available\": available},\r\n}\r\n```\r\n\r\nFrontend consumption:\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n```\r\n\r\nTranslation keys go in `src/i18n/locales/{en,zh}/messages.json` under `messages.agent.*`.\r\n\r\n## Anti-Patterns (with explanations)\r\n\r\n| Pattern | Why it's wrong |\r\n|---------|---------------|\r\n| `os.environ.get(\"DF_DEFAULT_LANGUAGE\")` | Process-level — all users get same language; breaks multi-user |\r\n| Global LLM client interceptor | Hidden behavior; can't distinguish full/compact mode; fragile string detection |\r\n| New `MessageBuilder` class | Duplicates `agent_language.py`; creates parallel conflicting abstractions |\r\n| Hardcoded `\"回答请使用中文\"` in prompts | Not configurable; skips the mode system; breaks for other languages |\r\n| Backend-side translation dict (`agent_messages.py`) | Forces adding every new language to Python; translations should all live in `src/i18n/locales/` |\r\n| Hardcoded English UI strings in `.tsx` without `t()` | Not translatable; use `useTranslation` + `t('key')` |\r\n\r\n## Adding a New Language\r\n\r\n1. Add language code + display name to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py`.\r\n2. Optionally add extra rules to `LANGUAGE_EXTRA_RULES` (e.g. simplified vs traditional Chinese).\r\n3. Add frontend translations in `src/i18n/locales//` — copy an existing locale folder as template.\r\n4. No Agent code changes needed — the existing flow picks up new languages automatically.\r\n",".cursor/skills/path-safety/SKILL.md":"---\r\nname: path-safety\r\ndescription: 服务端路径安全与文件访问编码规范。在编写文件下载路由、Agent 工具（文件读取/目录列出）、数据连接器/Loader、Workspace 路径操作、沙箱配置时使用。\r\n---\r\n\r\n# Path Safety — 服务端安全编码规范\r\n\r\n> **来源**：`docs/dev-guides/8-path-safety.md`（正式开发规范）+ `design-docs/issues/002-arbitrary-file-read-audit.md`（安全审计复核）。\r\n> 本文档提炼了 6 条必须遵守的编码规范。违反任一条即可能引入路径穿越（LFI）漏洞。\r\n\r\n---\r\n\r\n## R1. 文件下载：用 `ConfinedDir.resolve()` + `send_file`，禁用 `send_from_directory`\r\n\r\n**原因**：`send_from_directory(dir, user_input)` 内部会对 `user_input` 二次解析路径，与前置安全检查形成 TOCTOU 不一致。\r\n\r\n```python\r\n# ❌ BAD — 安全检查用 resolved target，发送用原始 filename，两次解析不一致\r\ntarget = (scratch_dir / filename).resolve()\r\ntarget.relative_to(scratch_dir.resolve()) # 检查通过\r\nreturn send_from_directory(str(scratch_dir), filename) # 再次解析\r\n\r\n# ✅ GOOD — 检查和发送用同一个 resolved path\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target) # 直接用已验证的路径\r\n```\r\n\r\n`send_file(Path)` 会根据扩展名自动推断 MIME type，无需额外处理。\r\n\r\n---\r\n\r\n## R2. 路径安全检查：用 `ConfinedDir`，禁止 `str.startswith`\r\n\r\n**原因**：`str(path).startswith(str(root))` 存在前缀碰撞缺陷（如 `/workspace` vs `/workspace_evil`）。\r\n\r\n```python\r\n# ❌ BAD\r\nif not str(resolved).startswith(str(root_resolved) + os.sep):\r\n raise ValueError(\"escape\")\r\n\r\n# ✅ GOOD — 统一走 ConfinedDir，内部使用 Path.is_relative_to()\r\njail = ConfinedDir(root_resolved, mkdir=False)\r\ntarget = jail.resolve(user_input)\r\n```\r\n\r\n---\r\n\r\n## R3. Agent 工具复用 `Workspace.confined_*`\r\n\r\n**原因**：Agent 工具参数由 LLM 生成，必须视为间接用户输入。不要在工具内手写 `Path(root) / rel_path` 或 `resolve() + relative_to()`；入口处复用 Workspace 暴露的 `ConfinedDir`。\r\n\r\n```python\r\n# ❌ BAD — 手写路径拼接和校验\r\ndef _tool_read_file(self, args, workspace_path):\r\n target = (workspace_path / rel_path).resolve()\r\n target.relative_to(workspace_path)\r\n\r\n# ✅ GOOD — 入口拿到 ConfinedDir，工具只调用 jail.resolve()\r\ndef _execute_tool(self, name, args):\r\n workspace_jail = self.workspace.confined_root\r\n scratch_jail = self.workspace.confined_scratch\r\n return self._tool_read_file(args, workspace_jail)\r\n\r\ndef _tool_read_file(self, args, workspace_jail):\r\n target = workspace_jail.resolve(args.get(\"path\", \"\"))\r\n```\r\n\r\n---\r\n\r\n## R4. 优先使用 `ConfinedDir`，禁止裸路径拼接\r\n\r\n**原因**：`Path(root) / user_input` 是路径穿越的高频入口。`ConfinedDir` 封装了三层防御（拒绝绝对路径 → 拒绝 `..` 段 → `resolve` + `is_relative_to`）。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\n# ❌ BAD — 手动拼接 + 手动校验，容易遗漏\r\nlocal_file = tmp_path / blob_relative_name\r\nlocal_file.parent.mkdir(parents=True, exist_ok=True)\r\nlocal_file.write_bytes(data)\r\n\r\n# ✅ GOOD — ConfinedDir 自动校验 + 创建父目录\r\njail = ConfinedDir(tmp_path, mkdir=False)\r\njail.write(blob_relative_name, data) # 自动校验 + 写入\r\n```\r\n\r\n### 已有安全 API 的层次关系\r\n\r\n```\r\n用户输入（filename / relative_path / blob key）\r\n │\r\n ▼\r\nsafe_data_filename() / secure_filename() ← 第一层：输入清洗\r\n │\r\n ▼\r\nConfinedDir.resolve() ← 第二层：路径约束\r\n │\r\n ▼\r\n安全的 Path 对象\r\n```\r\n\r\n使用 `Workspace.get_file_path()` 的场景不需要手动调用 `ConfinedDir`，因为它内部已包含等价的校验。\r\n\r\n---\r\n\r\n## R5. 宿主文件系统访问必须设部署模式守卫\r\n\r\n**原因**：桌面单用户模式允许访问本机文件系统（预期行为），但多用户/云部署下等于开放服务器读权限。\r\n\r\n**规范**：任何新的 Loader / Connector 如果涉及直接读取宿主文件系统（不通过 Workspace API），必须：\r\n\r\n1. 在 `data_loader/__init__.py` 的 `_enforce_deployment_restrictions()` 中注册禁用规则\r\n2. 确保 `create_connector()` 会拒绝已禁用的类型\r\n\r\n```python\r\n# data_loader/__init__.py — 参考 local_folder 的处理方式\r\ndef _enforce_deployment_restrictions():\r\n backend = os.environ.get(\"WORKSPACE_BACKEND\", \"local\")\r\n if backend != \"local\":\r\n for key in (\"local_folder\", \"your_new_local_loader\"):\r\n if key in DATA_LOADERS:\r\n del DATA_LOADERS[key]\r\n DISABLED_LOADERS[key] = f\"{key} disabled in multi-user mode\"\r\n```\r\n\r\n**判断标准**：如果 Loader 的构造函数接受一个用户可控的本机路径（如 `root_dir`），它就需要部署守卫。\r\n\r\n---\r\n\r\n## R6. 多用户部署必须启用沙箱\r\n\r\n**原因**：`not_a_sandbox` 模式下 LLM 生成的代码在宿主进程直接执行，可绕过所有路径检查。\r\n\r\n`app.py` 已在启动时检测此配置并输出 `logger.critical` 警告。新增的沙箱模式或部署脚本应确保：\r\n\r\n- `WORKSPACE_BACKEND != \"local\"` 时，`SANDBOX` 必须为 `docker` 或 `local`\r\n- CI/CD 部署模板中默认设置 `SANDBOX=docker`\r\n\r\n---\r\n\r\n## 速查：新增代码时的安全检查清单\r\n\r\n| 场景 | 必须做的事 |\r\n|------|-----------|\r\n| 新增文件下载路由 | 用 `ConfinedDir.resolve()` 得到路径，再 `send_file(resolved_path)`；不用 `send_from_directory` |\r\n| 新增 Agent 工具（读文件/列目录） | 入口复用 `workspace.confined_root` / `workspace.confined_scratch`，工具内只调用 `jail.resolve()` |\r\n| 路径包含判断 | 用 `ConfinedDir.resolve()`，不要手写 `Path.is_relative_to()` 或 `str.startswith()` |\r\n| `Path(root) / variable` 模式 | 改用 `ConfinedDir` 或 `Workspace.get_file_path()` |\r\n| 新增本机文件系统 Loader | 在 `_enforce_deployment_restrictions()` 中注册多用户禁用 |\r\n| 部署配置 | 多用户模式必须 `SANDBOX=docker` 或 `SANDBOX=local` |\r\n\r\n---\r\n\r\n## 参考文档\r\n\r\n- `docs/dev-guides/8-path-safety.md` — 服务端路径安全开发规范\r\n- `design-docs/6-path-safety-confined-dir.md` — 剩余未完成实现项状态页\r\n- `design-docs/issues/002-arbitrary-file-read-audit.md` — 安全审计复核报告\r\n- `py-src/data_formulator/security/path_safety.py` — ConfinedDir 源码\r\n",".github/copilot-instructions.md":"# Project Instructions\n\n## Python Environment\n\n- Always use `uv` instead of `pip` for installing packages (e.g. `uv pip install`, `uv pip install -e .`).\n- Use `uv run` to execute Python scripts and modules (e.g. `uv run python script.py`, `uv run pytest`).\n- The virtual environment is at `.venv/`. Activate it with `source .venv/bin/activate` if needed.\n- The Python source is under `py-src/`. The project is installed in editable mode via `uv pip install -e .`.\n","examples/plugins/README.md":"# Data Formulator — Data Loader Plugins\n\nDrop-in Python files that add new data-source connectors to Data\nFormulator without modifying its source code. If the built-in\nconnectors don't cover your data source (an internal warehouse, a SaaS\nAPI, a niche database), write a small plugin and DF will pick it up on\nthe next restart.\n\nThis folder contains **example plugins**. Treat them as templates: copy\none, rename it, and adapt the body.\n\n---\n\n## Quick start (3 steps)\n\n1. **Find your plugin directory.** It lives under your Data Formulator\n home dir:\n\n ```\n $DATA_FORMULATOR_HOME/plugins/\n ```\n\n If `DATA_FORMULATOR_HOME` is not set, DF defaults to\n `~/.data_formulator/`, so the plugin dir is `~/.data_formulator/plugins/`.\n\n Power users can point somewhere else with `DF_PLUGIN_DIR`\n (highest precedence) — useful for sharing one plugin folder across\n multiple DF installs.\n\n2. **Copy an example into it.** For instance:\n\n ```bash\n mkdir -p \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins\"\n cp examples/plugins/sqlite_data_loader.py \\\n \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins/\"\n ```\n\n3. **Restart Data Formulator.** The new connector appears in the UI\n automatically. No registry edits, no rebuilds.\n\nTo verify it loaded, check the startup log for a line like:\n\n```\nINFO ... Plugin loader 'sqlite' registered from sqlite_data_loader.py\nINFO ... Plugin scan complete: 1 registered, 0 failed (dir=..., reason=WORKSPACE_BACKEND=local)\n```\n\n### Plugin directory resolution order\n\n| Precedence | Source | Default |\n| ---------- | ----------------------------------------- | ----------------------------- |\n| 1 | `DF_PLUGIN_DIR` env var (explicit override) | — |\n| 2 | `$DATA_FORMULATOR_HOME/plugins` | — |\n| 3 | Fallback | `~/.data_formulator/plugins/` |\n\n---\n\n## File-name contract\n\n| Filename | Registry key |\n| --------------------------------- | -------------- |\n| `sqlite_data_loader.py` | `sqlite` |\n| `acme_warehouse_data_loader.py` | `acme_warehouse` |\n| `notion_data_loader.py` | `notion` |\n\nRules:\n\n* The filename **must** end in `_data_loader.py`.\n* The prefix becomes the registry key — keep it lowercase, no spaces.\n* If the key matches a built-in (e.g. `mysql_data_loader.py`), the\n plugin **overrides** the built-in. Useful for hot-patching.\n\n---\n\n## What goes inside the file\n\nEach plugin defines exactly one class that subclasses\n[`ExternalDataLoader`](../../py-src/data_formulator/data_loader/external_data_loader.py).\nThe minimum surface area:\n\n```python\nfrom data_formulator.data_loader.external_data_loader import (\n ExternalDataLoader, MAX_IMPORT_ROWS,\n)\nimport pyarrow as pa\n\nclass MyLoader(ExternalDataLoader):\n\n # Optional: human-friendly UI label. Without this, the registry key\n # is title-cased (``\"my_warehouse\"`` → ``\"My Warehouse\"``). Override\n # to fix awkward casing (``\"SQLite\"``, ``\"BigQuery\"``).\n DISPLAY_NAME = \"My Warehouse\"\n\n @staticmethod\n def list_params() -> list[dict]:\n \"\"\"Declare connection-form fields. The UI auto-renders this.\"\"\"\n return [\n {\"name\": \"endpoint\", \"type\": \"string\", \"required\": True,\n \"tier\": \"connection\", \"description\": \"Server URL\"},\n {\"name\": \"token\", \"type\": \"string\", \"required\": True,\n \"tier\": \"auth\", \"sensitive\": True, \"description\": \"API token\"},\n ]\n\n @staticmethod\n def auth_instructions() -> str:\n \"\"\"Markdown help text shown next to the form.\"\"\"\n return \"Get your API token from https://example.com/settings/tokens\"\n\n def __init__(self, params: dict):\n self.params = params\n # validate + open connection here\n\n def list_tables(self, table_filter: str | None = None) -> list[dict]:\n \"\"\"Return catalog: [{name, metadata: {columns, row_count}}, ...]\"\"\"\n ...\n\n def fetch_data_as_arrow(self, source_table: str,\n import_options: dict | None = None) -> pa.Table:\n \"\"\"Read rows. Honour import_options['size'] up to MAX_IMPORT_ROWS.\"\"\"\n ...\n```\n\nLook at [`sqlite_data_loader.py`](sqlite_data_loader.py) for a runnable\nimplementation (~170 lines, stdlib only).\n\n### `list_params()` field reference\n\n| Key | Meaning |\n| ------------- | ---------------------------------------------------------------- |\n| `name` | Parameter key passed into `__init__(params)` |\n| `type` | `\"string\"`, `\"int\"`, `\"bool\"`, `\"password\"` |\n| `required` | If `True`, DF rejects connections that omit it |\n| `default` | Pre-filled value in the form |\n| `sensitive` | If `True`, value is redacted from logs / stored metadata |\n| `tier` | `\"auth\"`, `\"connection\"`, or `\"filter\"` — groups fields visually |\n| `description` | Help text shown under the field |\n\n---\n\n## Security gate — please read\n\nPlugins execute arbitrary Python in the server process. To prevent\naccidental code execution in shared deployments, the plugin scanner is\n**enabled only in single-user local mode**:\n\n* `WORKSPACE_BACKEND` unset or `local` → scanner runs.\n* `WORKSPACE_BACKEND` is anything else → scanner is skipped silently.\n* To opt in for a hosted deployment, set `DF_ALLOW_PLUGINS=1`\n **and** make sure the plugin directory is writable only by trusted\n administrators.\n\n---\n\n## Debugging a plugin that won't load\n\nThe scanner reports every failure two ways:\n\n1. **In the log**, with a full traceback:\n\n ```\n WARNING Failed to load plugin foo_data_loader.py: ... ModuleNotFoundError ...\n ```\n\n2. **In `DISABLED_LOADERS`**, which the frontend surfaces as a greyed-out\n connector with the reason. Common reasons:\n\n | Message | Fix |\n | ------------------------------------------------------ | --- |\n | `missing dependency 'X' (pip install X)` | `uv pip install X` in DF's venv |\n | `no ExternalDataLoader subclass found` | Add a `class Foo(ExternalDataLoader)` to the file |\n | `could not create import spec` | File isn't valid Python or has wrong extension |\n | `: ` | Anything else raised during `import` — check the traceback |\n\nThe plugin scanner cleans `sys.modules` on failure, so simply restarting\nDF after editing the file picks up the fix — no manual cleanup needed.\n\n---\n\n## Adding third-party dependencies\n\nIf your plugin needs a package that isn't already in DF's environment:\n\n```bash\n# from the data-formulator repo root\nuv pip install \n```\n\nThe plugin file is loaded into the same Python process as DF, so\nanything importable from DF's venv is importable from your plugin.\n\n---\n\n## For AI coding agents\n\nWhen asked to **add a new data source** to a user's Data Formulator\ninstall, prefer writing a plugin file over modifying DF's source tree:\n\n1. Read [`sqlite_data_loader.py`](sqlite_data_loader.py) and the\n `ExternalDataLoader` ABC in\n `py-src/data_formulator/data_loader/external_data_loader.py` to\n understand the required methods.\n2. Create a new file `_data_loader.py` in this folder (or\n directly in `~/.data_formulator/plugins/`).\n3. Subclass `ExternalDataLoader`. Implement at minimum:\n `list_params`, `auth_instructions`, `__init__`, `list_tables`,\n `fetch_data_as_arrow`.\n4. Return data as a `pyarrow.Table` from `fetch_data_as_arrow` — do\n **not** convert to pandas in the hot path.\n5. Quote identifiers when building SQL (see `_quote_ident` in the\n SQLite example) to avoid injection vulnerabilities.\n6. Respect `import_options['size']` and cap at `MAX_IMPORT_ROWS`.\n7. If the source has credentials, mark those params `sensitive: True`\n and `tier: \"auth\"` so DF redacts them from stored metadata.\n8. Do not modify the user's data — open read-only connections where\n the source supports it.\n\nAfter writing the file, verify it loads with:\n\n```bash\nDF_PLUGIN_DIR= uv run python -c \\\n \"from data_formulator import data_loader as dl; \\\n print(dl.PLUGIN_LOADERS, dl.DISABLED_LOADERS)\"\n```\n","py-src/data_formulator/analyst/skills/core/SKILL.md":"---\nname: core\ndescription: >-\n The analyst's built-in capabilities: data-inspection tools and the\n always-available actions (visualize and ask_user).\nwhen_to_use: Always loaded by default — this is the agent's baseline.\nalways_on: true\ntools:\n - execute_python_script\n - inspect_source_data\nactions:\n - visualize\n - ask_user\n---\n\n# Core capabilities\n\nThis describes the built-in **inspection tools** you use to gather data and the\nalways-available **actions** you take on it. The overall loop, your action\nbudget, and the one-action-per-turn rule are covered in your system\ninstructions — this section is about *what* each tool and action does and how\nto use it well.\n\n## Tools (for data gathering)\n\n- **execute_python_script(code)** — run a general-purpose Python script to\n inspect data, compute stats, transform tables, or verify assumptions. Its\n stdout is returned to you (use `print()`); the script is for *your* analysis\n and its output is never shown to the user. pandas, numpy, duckdb, sklearn,\n scipy are available. **Important**: each call runs in a fresh namespace —\n variables do NOT persist between calls, so combine related steps into a\n single script.\n- **inspect_source_data(table_names)** — get schema, stats, and sample rows for\n source tables (cheaper than `execute_python_script` for basic inspection).\n- **load_skill(name)** — load a skill's instructions into context so you can use\n the action it unlocks (see the Skills section of your system instructions).\n\nThese are inspection tools — their results come back to you and are never shown\nto the user; call as many as you need, then take an action or give your final\nanswer.\n\nYou analyse data that is **already in the workspace**. If the user's question\nrequires connected data that isn't present, call `load_skill(\"data-loading\")`\nand follow that skill's discovery and immutable proposal workflow in this same\nconversation. Do not hand off to the standalone Data Loading agent.\n\nThe initial context already includes sample rows and statistics for each table.\nIf the data is straightforward, go straight to the action without calling\ntools. Tool results are returned to you before you act.\n\n## Actions\n\nCall an action as a tool call when you want to act on the data. Actions are\n**sequential**: take **one at a time**, then read the result it returns before\ndeciding the next — each action's outcome shapes the next one (the chart you draw\nnext depends on what this one reveals), so emitting several at once would decide\nthe later ones blind. After each result you choose what to do — take another\naction, or stop. **You end your turn by replying with plain text and no\naction**: that is your closing answer when you expect nothing further. When you\nwant the user to reply — a freeform question, a clarification you need before\nacting, or **clickable choices** — use the `ask_user` action instead. It renders\na question widget and pauses for their reply, keeping the conversation in the\nsame turn (plain text ends the run, so the user's next message would start\nfresh without this context).\n\n**Match the response to what the user asked for.** Two different cases:\n\n- **A direct answer** — they asked a question, so answer it. Length follows the\n question: one line when that settles it, more when it genuinely takes more.\n- **A finding after you acted** — they asked for the work, not a write-up, so\n this is unsolicited. The artifact already shows what it shows; add only what\n they'd miss by looking at it, and default to short.\n\nOpen with the point rather than announcing one is coming, and don't close by\nrestating what you just said.\nNever narrate what you're about to do or recap a chart's axes; let the artifact\nspeak for itself. When an action pauses for the user, give enough context to\nexplain what you found and what their choices mean.\n\n### `visualize` — chart a transform\n\nRun code that produces a DataFrame and render it as a chart. You then observe the\nresult and decide your next move.\n\n- `display_instruction` — ≤12 words; the question/hypothesis the chart\n investigates (don't recap x/y/color — those are visible). Wrap a **column** in\n `**…**` if it anchors the question.\n- `title` — a concise, neutral analytical heading naming the subject, measure,\n and analytical lens, such as “Year-over-year price change peaks.” Prefer a\n stable description of the view over a takeaway claim or narrated trend. Do\n not mention the chart type, imply causality, or editorialize. This field is\n required; put interpretation in the closing response instead.\n- `subtitle` — concise supporting context not already clear from the title or\n axes. Use one phrase of at most 16 words to provide contextual details. Do\n not restate the measure or analytical lens named in the title.\n- `code` — Python producing a DataFrame assigned to `output_variable`.\n- `output_variable` — snake_case name the code assigns.\n- `chart` — `{chart_type, encodings:{x,y,…}, config:{}}` (chart_type from the\n chart type reference).\n- `input_tables` — workspace table names, as listed in the available-tables\n context, that the code reads.\n- `field_metadata` — field → semantic annotation. Include units, index\n baselines, intrinsic domains, and ordinal order when supported by the data;\n never invent a unit. Distinguish percentages from percentage points and\n identifiers from quantities.\n- `field_display_names` — field → concise human-readable label for axes,\n legends, and table headers. Expand technical names, preserve established\n domain abbreviations, include units when useful, and use the user's language.\n\nSilently classify the analytical intent before choosing a chart: comparison,\ntrend, distribution, relationship, composition, deviation, ranking,\nuncertainty, or spatial pattern. Choose encodings and chart type from that\nintent and the data shape. Set ordering deliberately: chronological for time,\nsemantic order for ordinal fields, and measure order for rankings. Avoid line\ncharts or legends with excessive series, labels that collide, and color that\ndoes not encode additional information; aggregate, bin, facet, or limit\ncategories when needed without hiding material data.\n\n### `ask_user` — ask the user and pause for their reply (pauses the run)\n\nAsk the user something and pause for their input. Reach for this on **any** turn\nwhere you want a reply — a choice to make, a clarification you need before\nacting, or a brief statement paired with clickable follow-ups they can react to.\nPrefer it over ending your turn with a plain-text question: plain text ends the\nrun (the user's next message starts a fresh turn without this context), while\n`ask_user` keeps the conversation in the same turn.\n\n- `questions` — 1–3 items, each something the user **acts on**: a choice\n (`single_choice` with `options`) or an open question they type an answer to\n (`free_text`). Put your reasoning, rationale, and context in your reply text —\n **not** here. Never add a `questions` item that only states a rationale or\n explanation with nothing for the user to answer or click.\n- each question: `text` (wrap a **column** in `**…**`), `responseType`\n (`single_choice` when you offer `options`, else `free_text` — the user types\n their own open-ended answer, not a slot for your exposition), `required`\n (`true` when the run depends on the answer, `false` for an optional follow-up),\n and `options` (plain-text choices, **at most 3** — just the most likely\n answers; the user can always type a freeform reply, so don't enumerate every\n case).\n\nThis is **terminal**: the run pauses after it and resumes when the user replies.\n\n## Choosing what to do\n\nMatch the response depth to the user's request. Create charts that materially\ncontribute to the answer, and stop when the answer is sufficient.\n\n- For conceptual or informational questions, answer directly when a chart would\n not improve the answer.\n- For specific analytical questions, create the view or views needed to answer\n them clearly.\n- For diagnostic or exploratory questions, follow relevant findings across\n multiple views when doing so adds meaningful insight.\n- If essential intent is unclear, use `ask_user` rather than guessing.\n- *Missing data* (needs tables not in the workspace):\n `load_skill(\"data-loading\")`, discover the source, and propose immutable\n loading options inline.\n- *Report / write-up request* (e.g. \"write a report on X\", \"summarize the findings\n as a narrative\"): this needs the **report** skill — `load_skill(\"report\")` and\n follow it to commit the `write_report` action. **Do this as your very first\n move when charts already exist** (see `[AVAILABLE CHARTS]` / the thread): don't\n re-create them — load the report skill straight away and embed the existing\n charts by id. Only produce a new chart first if the report genuinely needs one\n that isn't there yet (0–3, judgment-based), then load the skill.\n\nFollow explicit requests about scope, depth, and format. **Never** repeat a\nvisualization already in the trajectory or in another thread.\n\n## Chart Creation Guide\n\nThe following reference material applies when you call the `visualize` tool.\n\n### A. Code Execution Rules\n\n**About the execution environment:**\n- You can use BOTH DuckDB SQL and pandas operations in the same script\n- The script will run in the workspace data directory (all data files are in the current directory)\n- Each table in [CONTEXT] has a **file path** (e.g., `student_exam.parquet`, `sales.csv`). Use EXACTLY that path to load data:\n - `.parquet`: `pd.read_parquet('file.parquet')` or DuckDB `read_parquet('file.parquet')`\n - `.csv`: `pd.read_csv('file.csv')` or DuckDB `read_csv_auto('file.csv')`\n - `.json`: `pd.read_json('file.json')`\n - `.xlsx`/`.xls`: `pd.read_excel('file.xlsx')`\n - `.txt`: `pd.read_csv('file.txt', sep='\\t')`\n- **IMPORTANT:** Use the exact filename from the context — do NOT change the file extension or assume all files are parquet.\n- **Allowed libraries:** pandas, numpy, duckdb, math, datetime, json, statistics, collections, re, sklearn, scipy, random, itertools, functools, operator, time\n- **Not allowed:** matplotlib, plotly, seaborn, requests, subprocess, os, sys, io, or any other library not listed above.\n- File system access (open, write) and network access are also forbidden.\n\n**When to use DuckDB vs pandas:**\n- **Prefer plain pandas** for most tasks — it's simpler and more readable.\n- Only use DuckDB when the dataset is very large and you need efficient SQL aggregations, filtering, joins, or window functions.\n- You can combine both: DuckDB for initial loading/filtering on large files, then pandas for complex operations.\n\n**Code structure:** standalone script (no function wrapper), imports at top. **CRITICAL:** The final result DataFrame MUST be assigned to the exact variable name you specified in `\"output_variable\"` — the system uses this name to extract the result. For example, if your output_variable is `sales_by_region`, the script must contain `sales_by_region = ...`.\n\n**DuckDB notes:**\n- Escape single quotes with '' (not \\')\n- No Unicode escapes (\\u0400); use character ranges directly: [а-яА-Я]\n- Cast date columns explicitly: `CAST(col AS DATE)`, `CAST(col AS TIMESTAMP)`\n- For complex datetime operations, load data first then use pandas datetime functions\n- Critical identifier quoting rule:\n * If a table/column name contains non-ASCII characters (e.g., Chinese, Japanese, Korean, Cyrillic, etc.), spaces, or punctuation,\n you MUST wrap it in double quotes, e.g. SELECT \"金额\" FROM \"客户表\".\n * Never output placeholder identifiers like your_table_name, your_column, your_condition.\n\n**Datetime handling:**\n- `date` columns contain date-only values (YYYY-MM-DD). `datetime` columns contain date+time (ISO 8601).\n- `time` columns contain time-only values (HH:mm:ss). `duration` columns are time intervals.\n- Year → number. Year-month / year-month-day → string (\"2020-01\" / \"2020-01-01\").\n- Hour alone → number. Hour:min or h:m:s → string. Never return raw datetime objects.\n\n### B. Chart Type Reference\n\nThe `chart_type` value in the `visualize` action MUST be one of the names listed\nbelow (exact spelling, including capitalization). When a row lists multiple\nnames, pick whichever fits the \"when to use\" hint best.\n\n**Choosing a chart — prefer simple, escalate when it fits.** Reach for the\n**Everyday** set first: it answers most questions and is the safest, most\nlegible choice. But when the data or question genuinely fits a **Specialized**\ntype (a distribution's shape, a cumulative curve, a rank race, a before→after,\na geographic pattern…), prefer it — a well-matched specialized chart is more\ninsightful than forcing a generic one. Don't pick a specialized type for\nnovelty; use it because its \"when to use\" condition is met.\n\n**Everyday — reach for these first**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Scatter Plot | x, y, color, size, facet | opacity (0.1–1.0) | Relationships between two quantitative fields |\n| Regression | x, y, color, size, facet | regressionMethod (\"linear\",\"log\",\"exp\",\"pow\",\"quad\",\"poly\"), polyOrder (2–10) | Trend line over scatter; one line per color group |\n| Bar Chart / Stacked Bar Chart / Lollipop Chart / Waterfall Chart | x, y, color, facet | — | Bar: categorical comparison (auto-stacks when color is set). Stacked Bar: explicit stacked totals, color = the stack. Lollipop: cleaner for ranked lists / sparse categories. Waterfall: cumulative gain/loss, each bar starts where the previous ended |\n| Grouped Bar Chart | x, y, group, facet | — | Side-by-side bars across a second categorical dimension |\n| Line Chart | x, y, color, strokeDash, facet | interpolate (\"linear\",\"monotone\",\"step\") | Trends over an ordered (usually temporal) x-axis |\n| Area Chart | x, y, color, facet | — | Magnitude over ordered x; auto-stacks when color is set |\n| Histogram / Density Plot | x, color, facet | — | Distribution of one quantitative field. Histogram: discrete bins, auto-binned. Density Plot: smooth KDE curve |\n| Boxplot | x, y, color, facet | — | Distribution summary (median/quartiles/outliers) by category |\n| Pie Chart | size, color, facet | innerRadius (0–100; 0=pie, >0=donut) | Part-of-whole with ≤7 categories. Wedge value goes on **size**, not **theta** |\n| Heatmap | x, y, color, facet | colorScheme — sequential (\"viridis\",\"blues\",\"reds\",\"oranges\",\"greens\") or diverging (\"blueorange\",\"redblue\") | Matrix / 2D density; color encodes the quantitative cell value |\n\n**Specialized — use when the data/question fits the \"when to use\"**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Connected Scatter Plot | x, y, order, color, facet | — | Two quantitative fields traced in sequence — needs an `order` field (e.g. time) so points are joined in order, not by x |\n| Ranged Dot Plot | x, y, color, facet | — | Min–max range or two-point comparison per category |\n| Violin Plot | x, y, color, facet | — | Distribution SHAPE (KDE silhouette) by category; better than a boxplot when data is multimodal. x = category, y = value |\n| Strip Plot | x, y, color, size, facet | — | Every individual point by category (jittered); good for small/medium n where raw values matter, not just a summary |\n| ECDF Plot | x, color, facet | — | Cumulative distribution of one quantitative field. Pass the RAW field on x (do NOT pre-compute the CDF); color for per-group curves |\n| Bump Chart | x, y, color, facet | — | How RANKINGS change over ordered x; y = rank, color = entity (long-form: one row per entity × x) |\n| Slope Chart | x, y, color, facet | — | Change between exactly TWO points (before → after) per entity; x = the two labels, y = value, color = entity |\n| Streamgraph | x, y, color, facet | — | Several series' magnitude over ordered x, stacked around a center baseline (color = series) — theme/volume shifts over time |\n| Range Area Chart | x, y, y2, color, facet | — | A shaded band between a lower (y) and upper (y2) bound over ordered x — e.g. min–max or a confidence interval |\n| Rose Chart | x, y, color, facet | — | Cyclical/categorical magnitude as angular wedges (polar bars); x = category/angle, y = value |\n| Pyramid Chart | x, y, color, facet | — | Back-to-back bars split by a binary group (e.g. population by age × sex); y = category, x = value, color = the two-sided group |\n| Radar Chart | x, y, color, facet | — | Multi-metric profile/comparison; x = metric name, y = value, color = entity (long-form data) |\n| Bar Table | x, y, color, facet | — | Ranked horizontal table with inline bars; one row per category. y = category, x = value |\n| KPI Card | metric, value, goal | — | \"Big number\" dashboard tile(s); one row per tile. `value` must be pre-aggregated; `goal` is optional |\n| Candlestick Chart | x, open, high, low, close, facet | — | OHLC financial data |\n| Map | longitude, latitude, color, size | projection (\"mercator\",\"equalEarth\",\"naturalEarth1\",\"orthographic\",\"albersUsa\"), projectionCenter ([lon,lat]) | Geographic POINTS/bubbles by lon/lat (use projection \"albersUsa\" for a US-only map) |\n| Choropleth | id, color, facet | region (\"world\",\"usa\",…) | Filled REGIONS shaded by value; `id` = the region key (country/state name or code), color = the quantitative value |\n\n**Critical chart rules:**\n- **Scatter Plot**: use config opacity (0.1–1.0) for dense data instead of encoding opacity.\n- **Regression**: trend line is automatic — do NOT compute regression coefficients/predictions in Python. Use `color` to get separate trend lines per group.\n- **Bar Chart**: x=categorical, y=quantitative (vertical bars). Swap x↔y for horizontal bars. Same-x rows are auto-stacked when `color` is set.\n- **Grouped Bar Chart**: use the `group` channel (not `color`) for side-by-side bars.\n- **Histogram**: do NOT pre-bin in Python — pass the raw quantitative field on `x` and the chart bins automatically. Pre-aggregating gives wrong bin widths.\n- **Line Chart**: use `strokeDash` to differentiate line styles (e.g. actual vs forecast).\n- **Pie Chart**: use the `size` channel (not `theta`) for wedge values. Avoid when >7–8 categories.\n- **Radar Chart**: data must be long-form — one row per (entity, metric, value). If your data is wide-form (one column per metric), melt it first in the Python step.\n- **Heatmap**: pick `colorScheme` by the meaning of the values. Use a **sequential** scheme (viridis/blues/reds/oranges/greens) for single-direction magnitudes (counts, rates, prices, scores — higher is simply more). Use a **diverging** scheme (blueorange/redblue) ONLY when the values have a meaningful center to read away from (e.g. profit/loss around 0, change vs. a baseline, temperature around freezing).\n- **Bar Table**: y is the category column to rank; x is the quantitative value driving bar length. Don't sort in Python — the template sorts.\n- **KPI Card**: channels are `metric`, `value`, `goal` (not x/y). One DataFrame row = one tile. The `value` column must already contain the final number to display (aggregate upstream in the Python step).\n- **Candlestick Chart**: requires `open`, `high`, `low`, `close` columns.\n- **Connected Scatter Plot**: provide an `order` field (usually time) so points are joined in sequence, not by x-order.\n- **ECDF Plot**: pass the RAW quantitative field on `x` — the chart computes the cumulative curve; do NOT pre-compute it in Python.\n- **Range Area Chart**: `y` is the lower bound and `y2` the upper bound of the band.\n- **Bump / Slope Chart**: long-form data — one row per (entity, x); `color` is the entity. Slope's `x` has exactly two categories (before/after).\n- **Violin Plot**: like Boxplot but shows the full distribution shape; x = category, y = value.\n- **Map / Choropleth**: `Map` plots points via `longitude` / `latitude` (set projection `\"albersUsa\"` for the US); `Choropleth` fills regions — put the region key on `id` and the value on `color`, not `x` / `y`.\n- **facet**: available for nearly all chart types; use a low-cardinality categorical field.\n- All fields in `encodings` must also appear in `output_fields`. Typically use 2–3 channels (x, y, color/size).\n\n### C. Semantic Type Reference\n\nChoose the most specific type that fits. Only annotate fields used in chart encodings.\n\n| Category | Types |\n|---|---|\n| Temporal | DateTime, Date, Time, Timestamp, Year, Quarter, Month, Week, Day, Hour, YearMonth, YearQuarter, YearWeek, Decade, Duration |\n| Monetary measures | Amount, Price |\n| Physical measures | Quantity, Temperature |\n| Proportion | Percentage |\n| Signed/diverging | Profit, PercentageChange, Sentiment, Correlation |\n| Generic measures | Count, Number |\n| Discrete numeric | Rank, Score |\n| Identifier | ID |\n| Geographic | Latitude, Longitude, Country, State, City, Region, Address, ZipCode |\n| Entity names | Category, Name |\n| Coded categorical | Status, Boolean, Direction |\n| Binned ranges | Range |\n| Fallback | Unknown |\n\nKey guidelines:\n- Use **Amount** for summed monetary totals, **Price** for per-unit prices, **Profit** for values that can be negative.\n- Use **Temperature** (not Quantity) for temperature — it has special diverging behavior.\n- Use **Year** (not Number) for columns like \"year\" with values 2020, 2021.\n\n### D. Statistical Analysis Guide\n\n- **Regression**: use chart_type \"Regression\" — the trend line is automatic, do NOT compute regression values in Python code. Configure method via `{\"regressionMethod\": \"linear\"}` (options: \"linear\", \"log\", \"exp\", \"pow\", \"quad\", \"poly\"; for poly add `{\"polyOrder\": 3}`).\n- **Forecasting**: compute predicted future values in Python. Use Line Chart with strokeDash to distinguish actual vs forecast, and color for series grouping.\n- **Clustering**: compute cluster assignments in Python. Output [x, y, cluster_id]. Use Scatter Plot with color → cluster_id.\n","py-src/data_formulator/analyst/skills/core/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"execute_python_script\",\n \"description\": \"Execute a general-purpose Python script in the sandbox. Here you use it to inspect data, compute statistics, transform tables, or verify assumptions before you act — write results to stdout with print() and that output is returned to you (it is NOT shown to the user). The script is for your own analysis, not for producing the final visualization. pandas, numpy, duckdb, sklearn, scipy are available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"purpose\": {\n \"type\": \"string\",\n \"description\": \"One-sentence description of what this script does and why (shown to user as progress).\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python script to execute. Use print() to surface output.\"\n }\n },\n \"required\": [\"purpose\", \"code\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_source_data\",\n \"description\": \"Get a detailed summary of one or more source tables — schema, field-level statistics, and sample rows. Cheaper than execute_python_script for basic data inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"table_names\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of workspace table names, as listed in the available-tables context, to inspect.\"\n }\n },\n \"required\": [\"table_names\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"visualize\",\n \"description\": \"Commit a data transform + chart: run code producing a DataFrame and render it. The agent observes the result and continues.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\": \"A concise, neutral analytical heading that names the subject, measure, and analytical lens, such as 'Year-over-year price change peaks'. Prefer a stable description of the view over a takeaway claim or narrated trend. Do not mention the chart type, imply causality, or editorialize. Shown as the chart heading.\"\n },\n \"subtitle\": {\n \"type\": \"string\",\n \"description\": \"Concise supporting context not already clear from the title or axes. Use one phrase of at most 16 words to provide contextual details. Do not restate the measure or analytical lens named in the title.\"\n },\n \"display_instruction\": {\n \"type\": \"string\",\n \"description\": \"≤12 words. State the question or hypothesis the chart investigates — don't recap the chart spec (x/y/color/split are already visible). Wrap a **column** in ** ** if it anchors the question.\"\n },\n \"input_tables\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Workspace table names, as listed in the available-tables context, that the code reads.\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python code producing a DataFrame assigned to output_variable.\"\n },\n \"output_variable\": {\n \"type\": \"string\",\n \"description\": \"snake_case name of the DataFrame variable the code assigns.\"\n },\n \"chart\": {\n \"type\": \"object\",\n \"description\": \"Chart spec: {chart_type, encodings:{x,y,...}, config:{}}. chart_type from the chart type reference.\"\n },\n \"field_metadata\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> SemanticType for the output columns.\"\n },\n \"field_display_names\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> human-readable display name for chart axes and table headers.\"\n }\n },\n \"required\": [\"title\", \"code\", \"output_variable\", \"chart\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"ask_user\",\n \"description\": \"Ask the user something and pause for their reply — the run resumes in the same turn with their answer in context. Use this for ANY turn where you want the user to respond: a choice to make, a clarification you need before acting, or a brief statement paired with clickable follow-ups. Put your reasoning, rationale, and context in your normal reply text, not inside this call. Prefer this over ending your turn with a plain-text question: plain text ends the run and the user's next message starts a fresh turn without this context, whereas ask_user keeps the conversation going. Reserve plain text (no action) for your final answer when you expect nothing further.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"thought\": {\n \"type\": \"string\",\n \"description\": \"Brief rationale (not shown to the user).\"\n },\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"1–3 things the user acts on: a choice (single_choice with options) or an open question they type an answer to (free_text). Put rationale, reasoning, and context in your reply text, not here — never add an item that only states an explanation with nothing for the user to answer or click. An explanation is allowed only as a short statement paired with clickable chart-producing follow-ups (required=false with options).\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The question, or (for an optional follow-up) a short statement. Keep a statement to 1–3 grounded sentences and pair it with clickable follow-up options. Wrap a **column** in ** ** to highlight it.\"\n },\n \"responseType\": {\n \"type\": \"string\",\n \"enum\": [\"single_choice\", \"free_text\"],\n \"description\": \"single_choice when you offer options; free_text when the user types their own open-ended answer (not a slot for your own exposition).\"\n },\n \"required\": {\n \"type\": \"boolean\",\n \"description\": \"false for an explanation / optional follow-up; true for a clarification the run depends on.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Plain-text choices, at most 3. Keep them to the few most likely answers — the user can always type a freeform reply, so don't try to enumerate every case. For a clarification these are answers; for an explanation these are short chart-producing follow-up prompts the user might click next (≤8 words each, phrased as the user would say them).\"\n }\n },\n \"required\": [\"text\"]\n }\n }\n },\n \"required\": [\"questions\"]\n }\n }\n }\n]\n","py-src/data_formulator/analyst/skills/data-loading/SKILL.md":"---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.","py-src/data_formulator/analyst/skills/data-loading/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]","py-src/data_formulator/analyst/skills/data_loading/SKILL.md":"---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.","py-src/data_formulator/analyst/skills/data_loading/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]","py-src/data_formulator/analyst/skills/report/SKILL.md":"---\nname: report\ndescription: >-\n Turn an exploration (threads, findings, charts) into a single Markdown\n report — note, blog post, executive summary, KPI dashboard, slide brief, or\n multi-section analytical report, with embedded charts.\nwhen_to_use: >-\n The user asks to write up / summarize / report on what they explored, or\n wants a shareable narrative document built from the charts and findings in\n the data thread. Not for producing a single new chart (use visualize).\nalways_on: false\ntools:\n - inspect_chart\nactions:\n - write_report\n---\n\n# Skill: Report writing\n\nYou are a data journalist / analyst who creates insightful, well-organized\nreports based on data explorations. The output is a single Markdown document\nthat may play many roles — short note, blog post, executive summary, dashboard,\nmulti-section report, FAQ, slide-style brief, etc. Adapt structure and length to\nwhat the user actually asks for; do not force a fixed template.\n\n## Emitting the report (the `write_report` action)\n\nFirst inspect whatever charts and data you need (see below), then write the\nentire report and commit it by **calling the `write_report` tool** — it is the\ncommitting action that ends this turn. Its `report` argument carries the\n**full Markdown** of the finished report:\n\n- `report` — the complete report in Markdown: headings, prose, tables, and\n embedded charts via `![caption](chart://chart_id)`.\n\nProduce any charts the report needs **before** calling `write_report`, and do\nall chart/data inspection first — once you call `write_report`, the report is\ndelivered as-is and the run ends.\n\n## Context available to you\n- **[PRIMARY TABLE(S)]** / **[OTHER AVAILABLE TABLES]**: Lightweight schema of datasets.\n- **[FOCUSED THREAD]** (optional): The exploration thread the user is continuing —\n the ordered steps with the user's questions, the agent's thinking, and the\n findings at each step. This is the spine of the story you are telling.\n- **[OTHER THREADS]** (optional): Brief per-step summaries of other exploration\n threads the user ran. These are additional findings worth weaving in.\n- **[AVAILABLE CHARTS]**: List of charts with their type, encodings, and table references.\n\n## Ground the report in the exploration\nThe thread context is your most important input. The user already did real\nanalysis — your job is to turn that journey into a coherent narrative, not to\nsummarize a single chart. Before writing:\n- Read the FOCUSED THREAD and OTHER THREADS to understand the full set of\n questions asked and findings reached.\n- Plan a report that covers the meaningful findings across the exploration,\n not just the last or most obvious chart.\n\n## Inspecting charts and data\nYou have two inspection tools available the whole time: `inspect_chart` and\n`inspect_source_data`. Use them on your own whenever you need to verify a detail\nbefore writing about it — a chart's exact numbers, its data, or a table's\nschema. `inspect_chart` lets you *read* a chart from its encodings, a data\nsample, and the code that produced it (and points you to the backing table so\nyou can interrogate the full data with `execute_python_script`); a rendered\nimage is included only when one is available. Read the charts behind the key\nfindings you present **before** you compose the report.\n\n## Write the report\nWrite the complete report in Markdown and pass it as the `report` argument of the\n`write_report` tool. Do all your inspecting first, then compose the whole\ndocument and make the one `write_report` call.\n\n### Embedding charts (REQUIRED FORMAT — do not change this)\nTo embed a chart image, use markdown image syntax with a `chart://` URL:\n ![Caption describing the chart](chart://chart_id)\n\nExample: `![Monthly trade balance trend](chart://chart-123)`\n\nThe chart_id must match one from [AVAILABLE CHARTS]. Place each chart embed on\nits own line (it renders as a block). You can embed the same chart at most\nonce. Captions are short — one line describing what the chart shows.\n\n### Tables\nFor data tables, write standard markdown tables directly:\n| date | value |\n| --- | --- |\n| 2020-01 | -43.5 |\n\n### Style & structure — adapt to the user's request\nThe user may ask for any of:\n- a short note or social-style summary (a few sentences, one or two charts),\n- a blog post / narrative report (intro → findings → takeaway),\n- an executive summary (key numbers up top, then context),\n- a KPI dashboard / multi-section overview (headings per topic, multiple charts\n arranged with short commentary between them),\n- a slide-style brief (compact sections with bullet points and embedded charts),\n- a deeper analytical report with sub-sections, methodology notes, and caveats.\n\nPick the structure that fits the request and the available material. Match the\nbreadth of the report to the breadth of the exploration: if the user explored\nseveral questions, the report should reflect that — don't collapse a rich\nexploration into a single-chart blurb unless the user explicitly asked for\nsomething that short. Reasonable defaults if the user is vague:\n- Start with a `# Title` that reflects the topic.\n- Group related findings under `##` (and `###` if useful) headings, typically\n one section per key finding / thread.\n- Around each embedded chart, briefly explain what it shows and the key insight.\n- Use bullets / short paragraphs / tables where they help; don't pad.\n- Close with a brief takeaway or summary section if the report is more than a\n few paragraphs. For very short outputs (notes, single-chart blurbs), a closing\n summary is optional.\n\n### Guardrails\n- Write in Markdown. Keep prose tight; let the data and charts carry the weight.\n- Stay faithful to the data — do not invent numbers, comparisons, or causation\n that the data does not actually support.\n- It is fine to flag uncertainty (\"based on the sample shown…\") when appropriate.\n- Embed every chart you discuss; don't reference a chart in prose without showing it.\n","py-src/data_formulator/analyst/skills/report/tools.json":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_chart\",\n \"description\": \"Read one or more charts. Returns each chart's encodings, a sample of its data, and the transformation code that created it so you can reason about what it shows (use execute_python_script on the backing table for the full data). A rendered PNG is included only when one is available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"chart_ids\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of chart IDs from [AVAILABLE CHARTS] to inspect.\"\n }\n },\n \"required\": [\"chart_ids\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"write_report\",\n \"description\": \"Deliver a Markdown report and end the run. `report` is the full report text (embed charts with ![caption](chart://chart_id)).\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"report\": {\n \"type\": \"string\",\n \"description\": \"The full Markdown report text. Embed charts with ![caption](chart://chart_id) referencing IDs from [AVAILABLE CHARTS].\"\n }\n },\n \"required\": [\"report\"]\n }\n }\n }\n]\n"},"items":[{"name":"copilot-instructions.md","path":".github/copilot-instructions.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.github/copilot-instructions.md","title":"GitHub Copilot Instructions","category":"copilot-instructions","format":"markdown","content":"# Project Instructions\n\n## Python Environment\n\n- Always use `uv` instead of `pip` for installing packages (e.g. `uv pip install`, `uv pip install -e .`).\n- Use `uv run` to execute Python scripts and modules (e.g. `uv run python script.py`, `uv run pytest`).\n- The virtual environment is at `.venv/`. Activate it with `source .venv/bin/activate` if needed.\n- The Python source is under `py-src/`. The project is installed in editable mode via `uv pip install -e .`.\n","isInternal":false,"tokens":122,"sizeBytes":465},{"name":"backend-test-conventions.mdc","path":".cursor/rules/backend-test-conventions.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/backend-test-conventions.mdc","title":"Cursor Rule: backend-test-conventions","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: Backend Python test conventions (pytest)\r\nglobs: tests/backend/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# Backend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/backend/unit/`, `tests/backend/integration/`, `tests/backend/contract/`, or by domain: `tests/backend/routes/`, `tests/backend/errors/`, `tests/backend/agents/`, `tests/backend/data/`, `tests/backend/security/`, `tests/backend/auth/`, `tests/backend/knowledge/`.\r\n- Name files `test_.py` with descriptive snake_case names.\r\n- Shared test data goes in `tests/backend/fixtures/`.\r\n\r\n## File Structure\r\n\r\n```python\r\n\"\"\"One-paragraph summary of what is being tested and why.\r\n\r\nBackground\r\n----------\r\nBrief context about the feature or bug fix these tests validate.\r\n\"\"\"\r\nfrom __future__ import annotations\r\n\r\nimport pytest\r\n\r\npytestmark = [pytest.mark.backend]\r\n```\r\n\r\n## Conventions\r\n\r\n- Always add `pytestmark = [pytest.mark.backend]` at module level.\r\n- Group related tests in classes prefixed with `Test` (e.g. `TestStripImageBlocks`).\r\n- Use `pytest.mark.parametrize` for data-driven tests instead of writing repetitive cases.\r\n- Use `pytest.fixture()` for shared setup; keep fixtures close to where they are used.\r\n- Unit tests must **not** depend on Flask, network, or external services.\r\n- Mock external calls with `unittest.mock.patch` / `MagicMock`; never make real API calls.\r\n- Each test function should verify **one** behavior and have a clear name: `test__`.\r\n\r\n## Running Tests\r\n\r\n- When running pytest, use `-q` (quiet mode) instead of `-v` (verbose mode) for cleaner output.\r\n- **Preferred:** `python -m pytest tests/backend/ -q`\r\n- **Avoid:** `python -m pytest tests/backend/ -v 2>&1`\r\n\r\n## Error Handling Test Requirements\r\n\r\nApplication errors return **HTTP 200** with `status: \"error\"` in the body.\r\nOnly auth errors (`AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403) and\r\nuncontrolled transport failures (404/413/500) use non-200.\r\n\r\nWhen adding a new route or modifying error handling, include tests for:\r\n\r\n1. **AppError scenarios** — verify HTTP 200 + correct `ErrorCode` in body:\r\n ```python\r\n def test_missing_table_returns_error():\r\n with app.test_client() as client:\r\n resp = client.get('/api/tables/nonexistent')\r\n assert resp.status_code == 200 # application error → always 200\r\n data = resp.get_json()\r\n assert data[\"status\"] == \"error\"\r\n assert data[\"error\"][\"code\"] == \"TABLE_NOT_FOUND\"\r\n ```\r\n\r\n2. **Auth errors** — verify non-200 for authentication/authorization failures:\r\n ```python\r\n def test_access_denied_returns_403():\r\n resp = client.get('/api/protected-resource')\r\n assert resp.status_code == 403\r\n assert resp.get_json()[\"error\"][\"code\"] == \"ACCESS_DENIED\"\r\n ```\r\n\r\n3. **Unexpected exceptions** — verify the global handler catches them (unhandled = 500):\r\n ```python\r\n def test_unexpected_error_returns_500():\r\n with patch('module.function', side_effect=RuntimeError(\"boom\")):\r\n resp = client.get('/api/endpoint')\r\n assert resp.status_code == 500\r\n assert resp.get_json()[\"error\"][\"code\"] == \"INTERNAL_ERROR\"\r\n ```\r\n\r\n4. **Streaming endpoints** — verify error events in the NDJSON stream:\r\n ```python\r\n def test_stream_error_yields_ndjson_event():\r\n resp = client.post('/api/my-stream', ...)\r\n lines = [json.loads(l) for l in resp.data.decode().strip().split('\\n') if l]\r\n error_lines = [l for l in lines if l.get('type') == 'error']\r\n assert len(error_lines) >= 1\r\n assert error_lines[0]['error']['code'] == 'LLM_TIMEOUT'\r\n ```\r\n\r\n5. **Test fixture** — ensure `register_error_handlers(app)` is called in the test app fixture:\r\n ```python\r\n @pytest.fixture\r\n def app():\r\n from data_formulator.error_handler import register_error_handlers\r\n _app = flask.Flask(__name__)\r\n _app.config[\"TESTING\"] = True\r\n _app.register_blueprint(my_bp)\r\n register_error_handlers(_app)\r\n return _app\r\n ```\r\n\r\nSee `tests/backend/errors/test_errors.py` and `test_error_handler.py` for reference implementations.\r\n\r\n## Example\r\n\r\n```python\r\n# ❌ BAD – vague name, no parametrize, no marker\r\ndef test_it_works():\r\n assert sanitize(\"hello world\") == \"hello_world\"\r\n\r\n# ✅ GOOD\r\npytestmark = [pytest.mark.backend]\r\n\r\n@pytest.mark.parametrize(\"raw,expected\", [\r\n (\"hello world\", \"hello_world\"),\r\n (\"订单明细\", \"订单明细\"),\r\n])\r\ndef test_sanitize_preserves_unicode(raw: str, expected: str) -> None:\r\n assert sanitize(raw) == expected\r\n```\r\n","frontmatter":{"description":"Backend Python test conventions (pytest)","globs":"tests/backend/**/*.py","alwaysApply":false},"isInternal":false,"tokens":1087,"sizeBytes":4654},{"name":"dataframe-serialization.mdc","path":".cursor/rules/dataframe-serialization.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/dataframe-serialization.mdc","title":"Cursor Rule: dataframe-serialization","category":"cursor-rule","format":"mdc","content":"# DataFrame Serialization\n\nAll DataFrame-to-records conversion for API responses, streaming events, or\nfrontend-visible data MUST use the centralized helpers in\n`data_formulator.datalake.parquet_utils`:\n\n| Source type | Helper |\n|---|---|\n| `pd.DataFrame` | `df_to_safe_records(df)` |\n| `pa.Table` (Arrow) | `get_sample_rows_from_arrow(table)` |\n\n## Why\n\n`pandas.DataFrame.to_json(orient='records')` defaults to `date_format='epoch'`,\nwhich serializes datetime columns as **epoch milliseconds** (e.g. `1773532800000`).\nThe frontend interprets these as plain numbers and renders them with commas\n(`1,773,532,800,000`) instead of formatted dates.\n\n`df_to_safe_records` enforces `date_format='iso'` and `default_handler=str`,\nensuring datetimes become ISO-8601 strings and exotic types degrade gracefully.\n\n## Banned Patterns\n\n```python\n# BAD — missing date_format, datetimes become epoch numbers\njson.loads(df.to_json(orient='records'))\n\n# BAD — to_dict returns Timestamp objects, not JSON-safe values\ndf.to_dict(orient='records')\n\n# ACCEPTABLE but should be unified for consistency\njson.loads(df.to_json(orient='records', date_format='iso'))\n```\n\n## Correct Pattern\n\n```python\nfrom data_formulator.datalake.parquet_utils import df_to_safe_records\n\nrows = df_to_safe_records(df)\npreview = df_to_safe_records(df.head(5))\n```\n\n## Exceptions\n\nInternal data processing that never reaches the frontend or JSON serialization\n(e.g. Kusto SDK metadata parsing, Vega-Lite spec construction) may use\n`to_dict(orient='records')` directly.\n","isInternal":false,"tokens":365,"sizeBytes":1530},{"name":"dev-guides-first.mdc","path":".cursor/rules/dev-guides-first.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/dev-guides-first.mdc","title":"Cursor Rule: dev-guides-first","category":"cursor-rule","format":"mdc","content":"---\ndescription: Require reading dev-guides before development and keeping docs in sync\nglobs:\nalwaysApply: true\n---\n\n# Dev Guides First\n\n## Before Starting Any Development\n\nBefore implementing or designing a new feature, module, or significant change, **always\nread the relevant documents in `docs/dev-guides/`** to understand existing conventions:\n\n| Guide | When to Read |\n|-------|-------------|\n| `docs/dev-guides/1-streaming-protocol.md` | Any work on streaming endpoints or NDJSON protocol |\n| `docs/dev-guides/2-log-sanitization.md` | Any work involving logging, credentials, external services, or DataLoaders |\n| `docs/dev-guides/3-data-loader-development.md` | Any work on ExternalDataLoader, DataConnector, or connector routes |\n| `docs/dev-guides/4-authentication-oidc-tokenstore.md` | Any work on OIDC, TokenStore, AUTH_MODE, or SSO flows |\n| `docs/dev-guides/6-i18n-language-injection.md` | Any work on Agent prompts, Agent routes, backend user-visible messages, or frontend i18n |\n| `docs/dev-guides/7-unified-error-handling.md` | Any work on API errors, frontend API calls, streaming error events, or error tests |\n| `docs/dev-guides/8-path-safety.md` | Any work on backend file access, downloads, Workspace paths, Agent tools, DataLoaders, or sandbox config |\n| `docs/dev-guides/10-agent-knowledge-reasoning-log.md` | Any work on Agent knowledge injection, KnowledgeStore, reasoning logs, or experience distillation |\n| `docs/dev-guides/11-catalog-metadata-sync.md` | Any work on catalog sync, catalog_cache, catalog_annotations, metadata merge, Agent catalog tools, or frontend catalog browsing |\n| `docs/dev-guides/12-sandbox-session.md` | Any work on sandbox execution, Agent tool-calling loops, explore/execute_python code execution, or namespace management |\n| `docs/dev-guides/13-unified-row-limits.md` | Any work on row limits, data loading size caps, frontendRowLimit, MAX_IMPORT_ROWS, or DataLoader size parameter |\n| `docs/dev-guides/14-model-capability-runtime-degradation.md` | Any work on LLM Client calls, Agent LLM invocations, model capability checks, reasoning_effort, or image/vision degradation |\n| `docs/dev-guides/15-dataframe-serialization.md` | Any work on DataFrame→JSON serialization, Agent result rows, DataLoader sample_rows, or table API responses |\n| `.cursor/rules/i18n-no-hardcoded-strings.mdc` | Any work adding or changing user-visible strings in `src/` |\n\nAlso check `.cursor/rules/` and `.cursor/skills/` for related coding conventions.\n\n## After Introducing New Conventions\n\nWhen your work establishes a new development pattern, convention, or architectural\ndecision, you MUST update documentation before considering the task complete:\n\n1. **Update existing dev-guide** — if your change extends an existing convention\n (e.g. new sensitive key type → update `2-log-sanitization.md`)\n\n2. **Create new dev-guide** — if your change introduces a new cross-cutting convention\n that future developers must follow. Place in `docs/dev-guides/` with the next number prefix.\n\n3. **Update existing SKILL** — if your change affects how an existing skill works\n (e.g. new error handling pattern → update `error-handling/SKILL.md`)\n\n4. **Create new SKILL** — if your change introduces a reusable development workflow\n that AI agents should follow. Place in `.cursor/skills//SKILL.md`.\n\n5. **Update existing Rule** — if your change adds constraints to an existing rule\n (e.g. new banned pattern → update the relevant `.mdc` file)\n\n6. **Create new Rule** — if your change introduces file-scoped coding conventions\n that should be enforced automatically. Place in `.cursor/rules/.mdc`.\n\n## Documentation Sync Checklist\n\nFor every PR that introduces new patterns:\n\n- [ ] Searched `docs/dev-guides/` for related existing docs\n- [ ] Updated or created dev-guide if introducing cross-cutting conventions\n- [ ] Updated or created `.cursor/skills/` if introducing reusable workflows\n- [ ] Updated or created `.cursor/rules/` if introducing file-scoped constraints\n- [ ] New dev-guide includes a \"New Module Checklist\" section (if applicable)\n\n## File Naming Conventions\n\n- **dev-guides**: `-.md` (e.g. `3-data-loader-development.md`)\n- **skills**: `.cursor/skills//SKILL.md`\n- **rules**: `.cursor/rules/.mdc`\n","frontmatter":{"description":"Require reading dev-guides before development and keeping docs in sync","globs":null,"alwaysApply":true},"isInternal":false,"tokens":1018,"sizeBytes":4333},{"name":"error-response-safety.mdc","path":".cursor/rules/error-response-safety.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/error-response-safety.mdc","title":"Cursor Rule: error-response-safety","category":"cursor-rule","format":"mdc","content":"---\ndescription: Prevent information exposure through exception messages in HTTP responses\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Error Response Safety\n\nNever return raw exception text (`str(e)`, `f\"...{e}\"`) directly in HTTP responses.\nPython exceptions may contain stack traces, file paths, database connection strings,\nAPI keys, or internal IP addresses — all of which are security risks (CWE-209).\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full error handling contract.\n\n## Unified Error System\n\nNew error paths should use the `AppError` / `error_handler` infrastructure from\n`errors.py` and `error_handler.py`. Legacy `message` / `error_message` bodies are\nhistorical formats; do not add compatibility branches for them in new code.\nAll application-controlled errors return **HTTP 200** with `status: \"error\"` in the body.\nOnly auth errors (401/403) and uncontrolled transport errors (404/413/500) use non-200.\n\n### Non-streaming routes\n\nRaise `AppError` for new code or when structured error codes are useful. The\nglobal error handler returns HTTP 200 + JSON error envelope:\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\n\n# Business error — raise with appropriate code and safe message\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# LLM / external API error — use classify_and_wrap_llm_error\nfrom data_formulator.error_handler import classify_and_wrap_llm_error\n\nexcept Exception as e:\n raise classify_and_wrap_llm_error(e) from e\n```\n\n### Streaming routes\n\nUse `stream_error_event()` to yield a unified NDJSON error line.\nValidation before the stream starts returns `200 application/json` (not NDJSON):\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\nexcept AppError as e:\n yield stream_error_event(e)\nexcept Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n### Real examples from migrated endpoints\n\n```python\n# /data-agent-streaming — standard NDJSON error event\nexcept Exception as e:\n logger.error(\"Error in data-agent-streaming\", exc_info=e)\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# /generate-report-chat — same pattern\nexcept Exception as e:\n logger.exception(\"generate-report-chat failed\")\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# Validation failure before stream starts — use 200 JSON, not NDJSON\nif not request.is_json:\n return stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\"))\n```\n\n### Database / workspace errors (tables.py)\n\nUse `classify_and_raise_db_error()` instead of the legacy `sanitize_db_error_message()`:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\nexcept Exception as e:\n classify_and_raise_db_error(e) # raises AppError with safe message\n```\n\n### Connector errors (data_connector.py)\n\nUse `classify_and_raise_connector_error()` instead of the legacy `_sanitize_error()`:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e) # raises AppError with safe message\n```\n\n### Legacy functions (deprecated)\n\nThese functions still exist as wrappers but should NOT be used in new code:\n- `sanitize_db_error_message()` → use `classify_and_raise_db_error()`\n- `_sanitize_error()` → use `classify_and_raise_connector_error()`\n- `safe_error_response()` / `sanitize_error_message()` → use `raise AppError(...)`\n- `classify_llm_error()` is used internally by `classify_and_wrap_llm_error()`\n\n## Rules\n\n1. **Application errors** — HTTP 200 + `status: \"error\"`; prefer `error: {...}` for new code and never expose exception details.\n2. **Auth errors** — `AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403 (transport layer interception).\n3. **Transport errors** (uncontrolled) — only `404` (no route), `413` (body limit), `500` (unhandled crash) may be non-200.\n4. **Streaming errors** — always use `stream_error_event()` for NDJSON error lines.\n5. **Logging** — always log the full exception server-side (`logger.warning` / `logger.error`\n with `exc_info=True` when needed). The client never needs the stack trace.\n6. **Debug detail** — `AppError.detail` is only included in responses when `app.debug is True`.\n\n## Bare except Policy\n\n- **Never** use naked `except:` — always specify at least `except Exception:`\n- Use specific exception types when possible (`except (ValueError, TypeError):`)\n- Always log caught exceptions: `logger.warning(...)` for degraded operations, `logger.debug(...)` for expected failures\n- Process cleanup code (`terminate`, `join`, `close`) is exempt from logging requirements\n\n## Common Mistakes\n\n```python\n# ❌ BAD — raw exception leaks internal details\nreturn jsonify({\"message\": str(e)}), 500\nreturn jsonify({\"error\": f\"Failed: {e}\"}), 502\nyield json.dumps({\"type\": \"error\", \"error\": str(e)}) + \"\\n\"\n\n# ❌ BAD — non-200 HTTP status code for application errors\nreturn jsonify({\"status\": \"error\", \"error_message\": \"...\"}), 400\n\n# ❌ BAD — naked except catches SystemExit/KeyboardInterrupt\nexcept:\n pass\n\n# ❌ BAD — silent swallow without logging\nexcept Exception:\n pass\n\n# ✅ GOOD — non-streaming (global handler returns HTTP 200)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# ✅ GOOD — streaming\nyield stream_error_event(AppError(ErrorCode.LLM_TIMEOUT, \"Timeout\", retry=True))\n\n# ✅ GOOD — graceful degradation with logging\nexcept Exception as e:\n logger.warning(\"Could not read schema for %s\", table_name, exc_info=e)\n\n# ✅ GOOD — specific exception types\nexcept (ValueError, TypeError):\n pass # safe fallback\n```\n","frontmatter":{"description":"Prevent information exposure through exception messages in HTTP responses","globs":"py-src/**/*.py","alwaysApply":false},"isInternal":false,"tokens":1377,"sizeBytes":6029},{"name":"frontend-test-conventions.mdc","path":".cursor/rules/frontend-test-conventions.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/frontend-test-conventions.mdc","title":"Cursor Rule: frontend-test-conventions","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: Frontend TypeScript test conventions (Vitest)\r\nglobs: tests/frontend/**/*.test.{ts,tsx}\r\nalwaysApply: false\r\n---\r\n\r\n# Frontend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/frontend/unit/` mirroring the `src/` structure:\r\n - `tests/frontend/unit/data/` → tests for `src/data/`\r\n - `tests/frontend/unit/app/` → tests for `src/app/`\r\n - `tests/frontend/unit/views/` → tests for `src/views/`\r\n- Name files `.test.ts` (or `.test.tsx` for React rendering tests).\r\n\r\n## File Structure\r\n\r\n```typescript\r\nimport { describe, it, expect } from 'vitest';\r\n// For React rendering tests:\r\n// import { render } from '@testing-library/react';\r\n\r\nimport { myFunction } from '../../../../src/';\r\n\r\ndescribe('myFunction', () => {\r\n it('should handle ', () => {\r\n expect(myFunction(input)).toBe(expected);\r\n });\r\n});\r\n```\r\n\r\n## Conventions\r\n\r\n- Import `describe`, `it`, `expect` explicitly from `vitest` (globals are enabled but explicit imports improve readability).\r\n- Use `@testing-library/react` and `@testing-library/jest-dom` for component rendering tests.\r\n- Prefer testing **exported pure functions** over testing internal component state.\r\n- When component logic is complex, extract it into an exported helper and test that directly.\r\n- Group tests with `describe` blocks; use section comments (`// --- Null cases ---`) for clarity.\r\n- One assertion per `it` block when possible; name tests as `should `.\r\n- Do **not** import from `node_modules` internals; only use public API.\r\n- Keep tests independent — no shared mutable state between `it` blocks.\r\n\r\n## Example\r\n\r\n```typescript\r\n// ❌ BAD – no describe, vague test name\r\nimport { expect, test } from 'vitest';\r\ntest('works', () => { expect(fn(1)).toBe(2); });\r\n\r\n// ✅ GOOD\r\nimport { describe, it, expect } from 'vitest';\r\nimport { checkIsLikelyTextOnlyModel } from '../../../../src/views/DataLoadingThread';\r\n\r\ndescribe('checkIsLikelyTextOnlyModel', () => {\r\n it('returns true for deepseek-chat', () => {\r\n expect(checkIsLikelyTextOnlyModel('deepseek-chat')).toBe(true);\r\n });\r\n\r\n it('returns false for undefined', () => {\r\n expect(checkIsLikelyTextOnlyModel(undefined)).toBe(false);\r\n });\r\n});\r\n```\r\n","frontmatter":{"description":"Frontend TypeScript test conventions (Vitest)","globs":"tests/frontend/**/*.test.{ts,tsx}","alwaysApply":false},"isInternal":false,"tokens":540,"sizeBytes":2283},{"name":"i18n-no-hardcoded-strings.mdc","path":".cursor/rules/i18n-no-hardcoded-strings.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/i18n-no-hardcoded-strings.mdc","title":"Cursor Rule: i18n-no-hardcoded-strings","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: No hardcoded UI strings — use i18n translation keys\r\nglobs: src/**/*.{ts,tsx}\r\nalwaysApply: true\r\n---\r\n\r\n# i18n: No Hardcoded UI Strings\r\n\r\nAll user-visible text in the frontend MUST go through the i18n system. Never hardcode Chinese, English, or any other language string directly in components.\r\n\r\n## How to Use\r\n\r\n```tsx\r\nimport { useTranslation } from 'react-i18next';\r\n\r\nconst { t } = useTranslation();\r\n\r\n// ✅ GOOD\r\n\r\n\r\n\r\n// ❌ BAD\r\n\r\n\r\n\r\n```\r\n\r\n## Non-Component Contexts (thunks, utilities, helpers)\r\n\r\nWhen you cannot use hooks (Redux thunks, plain `.ts` utilities, store helpers), import the i18n instance directly:\r\n\r\n```ts\r\nimport i18n from '../i18n';\r\n\r\n// ✅ GOOD — works outside React components\r\ni18n.t('messages.rowLimitReached', { count: 20000 })\r\n\r\n// ❌ BAD — hooks only work inside React function components\r\nconst { t } = useTranslation(); // will crash in a thunk\r\n```\r\n\r\nThe same rule applies: every user-visible string must use `i18n.t()` with a translation key, never a hardcoded literal.\r\n\r\n## Translation Files\r\n\r\n- English: `src/i18n/locales/en/.json`\r\n- Chinese: `src/i18n/locales/zh/.json`\r\n- Namespaces: `common`, `upload`, `chart`, `model`, `encoding`, `messages`, `navigation`, `dataLoading`, `errors`\r\n\r\nWhen adding a new key, add it to **both** `en` and `zh` locale files. Pick the namespace that fits; create a new namespace only if none applies.\r\n\r\n## What Counts as User-Visible\r\n\r\nMust use `t()`: button labels, tooltips, placeholders, error messages, dialog titles, tab names, toast notifications, table headers, empty-state text.\r\n\r\nMay stay hardcoded: log messages (`console.log`), error messages thrown but never displayed, internal constants, CSS class names, test IDs.\r\n\r\n**Cross-stack sentinel values** — strings shared between frontend and backend as identity markers (e.g. `UNTITLED_SESSION` in `dfSlice.ts` used by both Redux state and backend workspace API) must NOT be translated. They are internal constants, not display text. Translate them only at the UI rendering layer by mapping the sentinel to `t('...')`.\r\n\r\n## Backend Messages with message_code\r\n\r\nWhen the backend returns a user-visible message with a `message_code` (or `content_code`, `error_code`), use `translateBackend()` from `src/app/utils.tsx` to translate it:\r\n\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\n\r\n// Translate a backend error message\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n\r\n// Translate options with parallel code arrays\r\nconst options = translateBackendOptions(rawOptions, event.option_codes);\r\n```\r\n\r\nThe translation keys live in `messages.json` under the `agent` section. Always add **both** `en` and `zh` entries.\r\n","frontmatter":{"description":"No hardcoded UI strings — use i18n translation keys","globs":"src/**/*.{ts,tsx}","alwaysApply":true},"isInternal":false,"tokens":686,"sizeBytes":2927},{"name":"implementation-review-checklist.mdc","path":".cursor/rules/implementation-review-checklist.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/implementation-review-checklist.mdc","title":"Cursor Rule: implementation-review-checklist","category":"cursor-rule","format":"mdc","content":"---\ndescription: Require structural code review after implementation before final response\nglobs:\nalwaysApply: true\n---\n\n# Implementation Review Checklist\n\nAfter any substantive implementation, before the final response, review the\nuncommitted diff from a maintainer/code-review perspective. Do not only verify\nthat the feature works.\n\n## Required Checks\n\n- Inspect `git diff --stat` and relevant diffs to understand the actual blast radius.\n- Check responsibility boundaries: frontend UI, API route, service/helper, storage/session, and tests should each own the right part of the behavior.\n- Check API semantics: endpoint names, HTTP methods, response shapes, and side effects must match the product action. Similar UI actions should call the same backend API.\n- Check frontend/backend state consistency: after mutations, make sure Redux/local state, backend session/token/vault state, and subsequent list/status APIs agree.\n- Check duplication: if two paths clear the same state or perform the same API flow, extract a small helper or route instead of copying logic.\n- Check regression surfaces: refresh/reload/list/status paths must respect new flags or state, not just the primary click path.\n- Check tests: add or update focused tests for the changed contract; include at least one test for the bug/regression path when practical.\n- Check docs/rules/dev-guides when the change introduces or changes a cross-cutting convention.\n\n## Reporting\n\nIf issues are found during this review, fix clear issues before finalizing. If a\ntrade-off remains, call it out explicitly in the final response with the risk and\nwhy it was left as-is.\n","frontmatter":{"description":"Require structural code review after implementation before final response","globs":null,"alwaysApply":true},"isInternal":false,"tokens":322,"sizeBytes":1634},{"name":"incremental-development-cadence.mdc","path":".cursor/rules/incremental-development-cadence.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/incremental-development-cadence.mdc","title":"Cursor Rule: incremental-development-cadence","category":"cursor-rule","format":"mdc","content":"---\ndescription: Enforce step-by-step implementation with design-doc checks, tests, and i18n after each step\nglobs:\nalwaysApply: true\n---\n\n# Incremental Development Cadence\n\nMulti-step tasks MUST be implemented **one step at a time**. After each logical unit (function, endpoint, component), complete all cross-cutting concerns before moving on.\n\n## After Each Step\n\n1. **Verify against design doc** — compare with `design-docs/` or issue spec; raise deviations before continuing\n2. **Add tests** — write/update tests for the just-implemented code, run and confirm green\n3. **Add i18n keys** — if user-visible text was introduced, add `en` + `zh` keys immediately\n4. **Update docs** — if a new cross-cutting convention was introduced, update dev-guide/Rule immediately\n\n## Banned Patterns\n\n- ❌ Implementing 3+ functions/endpoints before writing any tests\n- ❌ Sweeping all i18n keys in a final pass after all code is done\n- ❌ Checking design docs only after the entire implementation is complete\n- ❌ Mentioning \"tests and i18n still needed\" only in the final response\n","frontmatter":{"description":"Enforce step-by-step implementation with design-doc checks, tests, and i18n after each step","globs":null,"alwaysApply":true},"isInternal":false,"tokens":255,"sizeBytes":1083},{"name":"language-injection-conventions.mdc","path":".cursor/rules/language-injection-conventions.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/language-injection-conventions.mdc","title":"Cursor Rule: language-injection-conventions","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: Language injection conventions for LLM Agent prompts\r\nglobs: py-src/data_formulator/agents/**/*.py,py-src/data_formulator/routes/agents.py\r\nalwaysApply: false\r\n---\r\n\r\n# Language Injection Conventions\r\n\r\nLanguage flows per-request: `Frontend i18n → Accept-Language header → get_language_instruction() → system prompt`.\r\n\r\n## LLM Prompt Language Injection\r\n\r\n1. **User-facing LLM output** MUST inject language via `get_language_instruction(mode=...)` in the route handler.\r\n2. **Mode selection:** `\"full\"` for text-heavy agents, `\"compact\"` for code-generation agents and short-text endpoints.\r\n3. **Inject into system prompt only** — use `inject_language_instruction()` from `agent_language.py`. Never inject into user messages.\r\n4. **Do NOT inject** for non-user-facing calls (health checks, internal tool calls).\r\n5. **Do NOT duplicate** — if upstream messages already contain language instruction, skip.\r\n6. **Do NOT** use env vars, global interceptors, or hardcoded language strings (e.g. `\"回答请使用中文\"`) — route handlers should use `get_language_instruction(mode=...)`.\r\n7. **New language?** Add to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py` and add locale files in `src/i18n/locales//`.\r\n\r\n## Python-Side User-Visible Messages (message_code pattern)\r\n\r\nFor **fixed strings in Python** that are shown in the UI (error messages, clarify options, completion summaries), do NOT translate in Python. Instead:\r\n\r\n1. Keep the English string as the default value.\r\n2. Add a `message_code` (or `content_code`, `error_code`, `summary_code`) field with a key like `\"agent.someKey\"`.\r\n3. Optionally add `message_params` for interpolation.\r\n4. The frontend translates using `translateBackend(fallback, code, params)` from `src/app/utils.tsx`.\r\n5. Add the translation key to `src/i18n/locales/{en,zh}/messages.json` under the `agent` section.\r\n\r\n```python\r\n# ✅ GOOD — backend returns code, frontend translates\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\",\r\n \"message_code\": \"agent.emptyDataframe\",\r\n}\r\n\r\n# ❌ BAD — backend-side translation dict\r\nyield {\"message\": translate_in_python(\"empty_df\", lang)}\r\n```\r\n\r\nFor detailed architecture and anti-pattern explanations, see `docs/dev-guides/6-i18n-language-injection.md`.\r\n","frontmatter":{"description":"Language injection conventions for LLM Agent prompts","globs":"py-src/data_formulator/agents/**/*.py,py-src/data_formulator/routes/agents.py","alwaysApply":false},"isInternal":false,"tokens":536,"sizeBytes":2321},{"name":"log-sanitization.mdc","path":".cursor/rules/log-sanitization.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/log-sanitization.mdc","title":"Cursor Rule: log-sanitization","category":"cursor-rule","format":"mdc","content":"---\ndescription: Prevent sensitive data leakage in server-side log messages\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Log Sanitization\n\nServer-side logs must never contain passwords, tokens, API keys, or other credentials\nin plain text. This rule complements `error-response-safety.mdc` (which covers client\nresponses) by addressing the **logging** side.\n\n## Architecture: Defense in Depth\n\nTwo layers protect against sensitive data in logs:\n\n1. **Layer 1 — Explicit utilities** (call-site): developers use `sanitize_url()`,\n `sanitize_params()`, `redact_token()` from `security/log_sanitizer.py`.\n2. **Layer 2 — SensitiveDataFilter** (global): a `logging.Filter` registered on all\n handlers in `configure_logging()` auto-redacts patterns that slip through.\n\nLayer 1 is the primary defense; Layer 2 is a safety net. Both are required.\n\n## When to Use Each Utility\n\n| Data Type | Utility | Example |\n|-----------|---------|---------|\n| `dict` with password/secret/token keys | `sanitize_params(params)` | `log.info(\"Init: %s\", sanitize_params(params))` |\n| URL (may have `user:pass@host` or sensitive query params) | `sanitize_url(url)` | `logger.info(\"Connecting to %s\", sanitize_url(url))` |\n| Bearer / access token | `redact_token(token)` | `logger.debug(\"Token: %s\", redact_token(token))` |\n| Normal string (no secrets) | Nothing needed | `logger.info(\"Processing table %s\", name)` |\n\n## Rules\n\n1. **Never log raw `params` / `config` dicts** that may contain `password`, `secret`,\n `api_key`, `token`, or `connection_string` keys. Always use `sanitize_params()`.\n\n2. **Never log full tokens or API keys.** Use `redact_token()` if you need to identify\n a token in logs.\n\n3. **Prefer `type(exc).__name__`** over `str(exc)` in warning/info logs when the\n exception may contain connection strings or upstream response bodies. Use\n `exc_info=True` for full tracebacks at ERROR level (the filter sanitizes `exc_text`).\n\n4. **URL logging** — use `sanitize_url()` for any URL from configuration or\n environment variables (issuer URLs, JWKS URLs, database URLs, API base URLs).\n It masks embedded credentials and sensitive query parameters such as\n `password`, `client_secret`, `token`, and `api_key`.\n\n5. **Use `%s`-style formatting** instead of f-strings for log messages. This allows\n the filter to intercept arguments before formatting and provides better\n compatibility with log aggregation tools.\n\n## Common Mistakes\n\n```python\n# BAD — dict may contain password\nlog.info(f\"Connecting with {params}\")\nlog.info(\"Connecting with %s\", params)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_params\nlog.info(\"Connecting with %s\", sanitize_params(params))\n\n# BAD — URL may embed credentials\nlogger.warning(\"Failed for %s\", discovery_url)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_url\nlogger.warning(\"Failed for %s\", sanitize_url(discovery_url))\n\n# BAD — exception may leak connection string or upstream body\nlogger.warning(\"Error: %s\", exc)\n\n# GOOD — log type name at warning level, full traceback at error level\nlogger.warning(\"Error: %s\", type(exc).__name__)\nlogger.error(\"Unexpected failure\", exc_info=True)\n\n# BAD — full token in logs\nlogger.debug(\"Using token %s\", token)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import redact_token\nlogger.debug(\"Using token %s\", redact_token(token))\n```\n\n## Disabling for Local Debug\n\nSet `LOG_SANITIZE=false` to disable the global filter during local debugging.\nThis should **never** be used in production or CI.\n\n## Adding New Sensitive Keys\n\nIf your module introduces a new credential key name (e.g. `custom_auth_token`),\nadd it to `SENSITIVE_KEYS` in `security/log_sanitizer.py` and add a corresponding\nregex pattern to `_SENSITIVE_KEY_NAMES` if needed.\n\n## New Module Checklist\n\nWhen creating a new module or feature that handles credentials or external services:\n\n- [ ] Audit all `logger.*()` calls — do any log connection params, URLs, or tokens?\n- [ ] Use `sanitize_params()` for any dict that may contain credentials\n- [ ] Use `sanitize_url()` for any URL from config/env\n- [ ] Use `redact_token()` for any token/key value\n- [ ] If introducing new credential key names, update `SENSITIVE_KEYS`\n- [ ] Run `python -m pytest tests/backend/security/test_log_sanitizer.py` to verify\n","frontmatter":{"description":"Prevent sensitive data leakage in server-side log messages","globs":"py-src/**/*.py","alwaysApply":false},"isInternal":false,"tokens":1025,"sizeBytes":4324},{"name":"package-manager-conventions.mdc","path":".cursor/rules/package-manager-conventions.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/package-manager-conventions.mdc","title":"Cursor Rule: package-manager-conventions","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: Use Yarn only, never npm/pnpm\r\nglobs: package.json, yarn.lock\r\nalwaysApply: true\r\n---\r\n\r\n# Package Manager Rules\r\n\r\n- Use Yarn v1.22.22 only - never use npm or pnpm\r\n- Never manually edit yarn.lock\r\n- Keep yarn.lock changes minimal when adding deps\r\n- Registry must be https://registry.yarnpkg.com\r\n","frontmatter":{"description":"Use Yarn only, never npm/pnpm","globs":"package.json, yarn.lock","alwaysApply":true},"isInternal":false,"tokens":82,"sizeBytes":317},{"name":"path-safety.mdc","path":".cursor/rules/path-safety.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/path-safety.mdc","title":"Cursor Rule: path-safety","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: 路径安全编码规范 — 编辑路由/Agent/Loader/Workspace/知识库代码时自动提醒\r\nglobs: py-src/data_formulator/{routes,agents,data_loader,datalake,security,knowledge}/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# 路径安全编码规范\r\n\r\n编辑此目录下的文件时，请遵守以下安全规则（详见 `docs/dev-guides/8-path-safety.md` 和 `.cursor/skills/path-safety/SKILL.md`）：\r\n\r\n## ConfinedDir 是唯一的路径约束原语\r\n\r\n所有接受用户可控路径的代码，**必须**通过 `ConfinedDir` 实例访问文件系统。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\njail = ConfinedDir(root_dir, mkdir=False)\r\ntry:\r\n target = jail.resolve(user_input)\r\nexcept ValueError:\r\n return {\"error\": \"Access denied\"}\r\n```\r\n\r\n## 禁止的模式\r\n\r\n```python\r\n# ❌ BAD — 裸路径拼接\r\ntarget = Path(root) / user_input\r\n\r\n# ❌ BAD — 手写 resolve + relative_to（已弃用，必须用 ConfinedDir）\r\ntarget = (root / user_input).resolve()\r\ntry:\r\n target.relative_to(root)\r\nexcept ValueError: ...\r\n\r\n# ❌ BAD — 手写 is_relative_to（已弃用，必须用 ConfinedDir）\r\nresolved = (root / user_input).resolve()\r\nif not resolved.is_relative_to(root.resolve()): ...\r\n\r\n# ❌ BAD — 路径比较用 str.startswith\r\nif str(target).startswith(str(root)): ...\r\n```\r\n\r\n## 正确的模式\r\n\r\n```python\r\n# ✅ GOOD — Agent 工具\r\nworkspace_jail = self.workspace.confined_root\r\nscratch_jail = self.workspace.confined_scratch\r\ntarget = workspace_jail.resolve(rel_path)\r\n\r\n# ✅ GOOD — 文件下载 route\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target)\r\n\r\n# ✅ GOOD — 文件上传（两层防御）\r\nsafe_name = secure_filename(raw_filename)\r\ntarget = scratch_jail.resolve(safe_name)\r\n\r\n# ✅ GOOD — Workspace data 文件\r\npath = workspace.get_file_path(filename) # 内部使用 ConfinedDir\r\n```\r\n\r\n## 部署守卫\r\n\r\n- 新增读取宿主文件系统的 Loader 必须在 `_enforce_deployment_restrictions()` 注册多用户禁用\r\n- 多用户模式（`WORKSPACE_BACKEND != \"local\"`）必须启用沙箱\r\n","frontmatter":{"description":"路径安全编码规范 — 编辑路由/Agent/Loader/Workspace/知识库代码时自动提醒","globs":"py-src/data_formulator/{routes,agents,data_loader,datalake,security,knowledge}/**/*.py","alwaysApply":false},"isInternal":false,"tokens":605,"sizeBytes":2162},{"name":"test-driven-workflow.mdc","path":".cursor/rules/test-driven-workflow.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/test-driven-workflow.mdc","title":"Cursor Rule: test-driven-workflow","category":"cursor-rule","format":"mdc","content":"---\r\ndescription: 测试优先开发与测试保护规则。确保新功能/Bug 修复必须有测试覆盖，测试失败时先诊断后修改。\r\nglobs: \r\nalwaysApply: true\r\n---\r\n\r\n# Test-Driven Workflow\r\n\r\n## 1. 测试优先：新功能 / Bug 修复必须有测试\r\n\r\n开发新功能或修复 Bug 时，**必须同步编写测试**，不能只提交业务代码：\r\n\r\n- **新功能** → 先写测试（或至少同步写），描述期望行为，再实现代码使测试通过\r\n- **Bug 修复** → 先写复现测试，确认测试失败，再修复代码使测试通过\r\n- **重构** → 确保重构前已有测试覆盖，重构后测试仍然全部通过\r\n\r\n完成实现后，主动运行相关测试验证：\r\n- 后端：`python -m pytest tests/backend/ -q`\r\n- 前端：`yarn test`\r\n\r\n## 2. 测试保护：不要轻易修改测试代码\r\n\r\n测试失败时，**禁止直接修改测试代码来让测试通过**。必须执行以下流程：\r\n\r\n1. **复现并定位**：运行失败的测试，确认失败现象\r\n2. **诊断分类**：\r\n - A：测试本身有问题（断言错误、setup 不正确）\r\n - B：业务实现有 Bug\r\n - C：需求/规格变更导致测试过时\r\n3. **给出方案**：至少提供两个可选方案（改测试 / 改实现 / 补规格），说明各自影响和风险\r\n4. **等待确认**：由用户决定采用哪个方案，再执行修改\r\n\r\n**绝对禁止**：\r\n- 测试一红就直接改测试代码\r\n- 删除或跳过（`@pytest.mark.skip`）失败的测试来\"解决\"问题\r\n- 修改测试的断言值来匹配错误的实现\r\n\r\n## 3. 精准修改：不动无关测试\r\n\r\n与 Karpathy 编码准则的\"精准修改\"一致：\r\n\r\n- 不\"顺手改进\"与当前任务无关的测试代码\r\n- 不重构正在通过的测试\r\n- 不改变已有测试的断言逻辑，除非被明确要求\r\n","frontmatter":{"description":"测试优先开发与测试保护规则。确保新功能/Bug 修复必须有测试覆盖，测试失败时先诊断后修改。","globs":null,"alwaysApply":true},"isInternal":false,"tokens":600,"sizeBytes":1838},{"name":"unified-error-protocol.mdc","path":".cursor/rules/unified-error-protocol.mdc","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/rules/unified-error-protocol.mdc","title":"Cursor Rule: unified-error-protocol","category":"cursor-rule","format":"mdc","content":"---\ndescription: Unified error handling protocol for backend and frontend\nglobs: py-src/**/*.py,src/**/*.{ts,tsx}\nalwaysApply: false\n---\n\n# Unified Error Protocol\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full developer guide.\n\n## HTTP Status Code Policy\n\n**All application-controlled errors return HTTP 200** with `status: \"error\"` in the body.\nThis applies to both non-streaming JSON APIs and streaming pre-flight errors.\n\nOnly these scenarios use non-200:\n- `401` — `AUTH_REQUIRED` / `AUTH_EXPIRED` (so transport layer can intercept)\n- `403` — `ACCESS_DENIED` (so transport layer can intercept)\n- `404` — Flask routing: no matching route handler\n- `413` — WSGI: request body exceeds `MAX_CONTENT_LENGTH`\n- `500` — Unhandled exception that escaped all error handling\n\n**Streaming NDJSON APIs:**\n- Pre-stream validation failure: HTTP `200` + `application/json` (use `stream_preflight_error()`)\n- In-stream error: NDJSON `{\"type\": \"error\", \"error\": {...}}`\n- In-stream warning: NDJSON `{\"type\": \"warning\", \"warning\": {...}}`\n\n## Backend: Non-streaming JSON Format\n\n```jsonc\n// Success (HTTP 200)\n{\"status\": \"success\", \"data\": {...}}\n\n// Error (HTTP 200 for business errors, 401/403 for auth errors)\n{\n \"status\": \"error\",\n \"error\": {\n \"code\": \"TABLE_NOT_FOUND\",\n \"message\": \"Table not found\",\n \"retry\": false,\n \"detail\": \"...\" // Only in debug mode\n }\n}\n```\n\n## Backend: Streaming NDJSON Format\n\nAll streaming endpoints use `application/x-ndjson`. Each line is one JSON object:\n\n```jsonc\n{\"type\": \"text_delta\", \"data\": {...}}\n{\"type\": \"error\", \"error\": {\"code\": \"LLM_TIMEOUT\", \"message\": \"...\", \"retry\": true}}\n{\"type\": \"done\", \"data\": {...}}\n```\n\n## Backend: How to Handle Errors\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok, stream_preflight_error\nfrom data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n\n# Non-streaming: success\nreturn json_ok(data)\n\n# Non-streaming: error (global handler returns HTTP 200, auth errors get 401/403)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# Streaming: pre-flight error (always HTTP 200)\nreturn stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Bad input\"))\n\n# Streaming: in-stream error\nyield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n## Frontend: How to Consume APIs\n\n```typescript\nimport { apiRequest, streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\n// Non-streaming — apiRequest checks body.status (main path) and HTTP status (auth/infra)\ntry {\n const { data } = await apiRequest(url, options);\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n\n// Streaming — streamRequest handles pre-flight JSON errors and NDJSON events\ntry {\n for await (const event of streamRequest(url, options, signal)) {\n if (event.type === 'error') { /* handle inline */ }\n // ...process events...\n }\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n```\n\nAPI consumers MUST use `apiRequest()` / `streamRequest()` plus `handleApiError()`.\nDirect `fetchWithIdentity()` is reserved for lower-level clients and explicit protocol\nexceptions such as file downloads, blob/CSV responses, SPA/OIDC redirects, or third-party URLs.\n\n## Prohibited Patterns\n\n### Backend\n- ❌ `return jsonify({\"status\": \"ok\", \"data\": data})` — use `json_ok(data)` instead\n- ❌ `return jsonify({\"error_message\": str(e)})` — use `raise AppError(...)` instead\n- ❌ `yield json.dumps({\"type\": \"error\", \"error\": str(e)})` — use `stream_error_event()`\n- ❌ Bare `except:` or `except Exception: pass` — always log or propagate\n- ❌ `yield 'error: ' + json.dumps(...)` — non-standard prefix\n\n### Frontend\n- ❌ `.catch(() => {})` — either add a comment explaining why (best-effort), or use `handleApiError`\n- ❌ `.catch(console.error)` without user feedback — dispatch `addMessages` or use `handleApiError`\n- ❌ Raw `fetch()` for `/api/` URLs — use `fetchWithIdentity` (identity headers + 401 retry)\n- ❌ RTK thunk without `.rejected` handler — always add `.addCase(thunk.rejected, ...)` with `addMessages`\n- ❌ New code using `fetchWithIdentity().json()` — use `apiRequest()` instead\n\n### RTK Thunk Rejected Handler Pattern\n\n```typescript\n.addCase(myThunk.rejected, (state, action) => {\n if (action.error?.name !== 'AbortError') {\n state.messages.push({\n timestamp: Date.now(), type: 'warning',\n component: 'my-component',\n value: 'Human-readable error description',\n });\n }\n})\n```\n\n## Migrated Streaming Endpoints\n\nAll streaming endpoints now use `application/x-ndjson` and `stream_error_event()`:\n\n| Endpoint | Old format | New format | Status |\n|----|-----|---|-----|\n| `/data-agent-streaming` | `{status:\"error\", error_message}` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/get-recommendation-questions` | `error: {json}` prefix + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/generate-report-chat` | `data: {json}` SSE prefix + `text/event-stream` | Pure NDJSON + `application/x-ndjson` | ✅ Done |\n| `/data-loading-chat` | `str(e)` in error field | `stream_error_event()` safe message | ✅ Done |\n| `/clean-data-stream` | `\\n{json}\\n` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n\n## Complete ErrorCode Reference\n\n```\nAUTH_REQUIRED, AUTH_EXPIRED, ACCESS_DENIED\nINVALID_REQUEST, TABLE_NOT_FOUND, FILE_PARSE_ERROR, FILE_TOO_LARGE, VALIDATION_ERROR\nLLM_AUTH_FAILED, LLM_RATE_LIMIT, LLM_CONTEXT_TOO_LONG, LLM_MODEL_NOT_FOUND\nLLM_TIMEOUT, LLM_SERVICE_ERROR, LLM_CONTENT_FILTERED, LLM_UNKNOWN_ERROR\nDB_CONNECTION_FAILED, DB_QUERY_ERROR, DATA_LOAD_ERROR, CONNECTOR_ERROR\nCODE_EXECUTION_ERROR, AGENT_ERROR\nCATALOG_SYNC_TIMEOUT, CATALOG_NOT_FOUND, ANNOTATION_CONFLICT, ANNOTATION_INVALID_PATCH\nINTERNAL_ERROR, SERVICE_UNAVAILABLE\n```\n\n## Adding a New Error Code\n\n1. Add constant to `py-src/data_formulator/errors.py` `ErrorCode` class (no HTTP mapping needed, defaults to 200)\n2. Add i18n mapping to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`\n3. Add translations to `src/i18n/locales/en/errors.json` AND `zh/errors.json`\n\n## New Endpoint Checklist\n\nWhen adding a new API endpoint, verify:\n- [ ] Success: `json_ok(data)` → `{\"status\": \"success\", \"data\": ...}`, HTTP 200\n- [ ] Error: `raise AppError(...)` → HTTP 200 + error body (auth errors get 401/403)\n- [ ] Streaming pre-flight: `stream_preflight_error(...)` → HTTP 200 + `application/json`\n- [ ] Streaming in-stream: `stream_error_event(...)` via NDJSON\n- [ ] No `str(e)` in any response body\n- [ ] No naked `except:` — use specific exception types or `except Exception as e:` with logging\n- [ ] Frontend uses `apiRequest()` (non-streaming) or `streamRequest()` (streaming)\n- [ ] Error test cases assert HTTP 200 + `body.status == \"error\"` (auth: 401/403)\n","frontmatter":{"description":"Unified error handling protocol for backend and frontend","globs":"py-src/**/*.py,src/**/*.{ts,tsx}","alwaysApply":false},"isInternal":false,"tokens":1810,"sizeBytes":7012},{"name":"SKILL.md","path":".cursor/skills/error-handling/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/skills/error-handling/SKILL.md","title":"error-handling","category":"anthropic-skill","format":"markdown","content":"---\nname: error-handling\ndescription: 统一错误处理系统。在添加 API 端点、修改错误处理、添加前端 API 调用、编写错误相关测试时使用。\n---\n\n# Error Handling Skill\n\nUnified error handling system for DF. Use when adding API endpoints, modifying error handling, or adding frontend API calls.\n\n> **Prerequisites**: Read `docs/dev-guides/7-unified-error-handling.md` before changing API error behavior.\n> Read `docs/dev-guides/2-log-sanitization.md` when the work involves logging, credentials, external services, or DataLoaders.\n> If your work introduces new error handling patterns or conventions, update this file and related dev-guides accordingly.\n\n## Architecture Overview\n\n```\nFrontend Backend\n──────── ───────\napiClient.ts errors.py\n├── apiRequest() ←── JSON ──── ├── ErrorCode (enum)\n├── streamRequest() ←── NDJSON ── └── AppError (exception)\n└── parseStreamLine()\n error_handler.py\nerrorCodes.ts ├── register_error_handlers(app)\n└── getErrorMessage() ├── classify_and_wrap_llm_error()\n └── stream_error_event()\nerrorHandler.ts\n└── handleApiError() security/sanitize.py\n └── classify_llm_error() (internal)\nMessageSnackbar ← dfSlice.messages\n```\n\n## Protocol Snapshot\n\nUse this contract for all new or reworked DF APIs:\n\n| Scenario | HTTP | Shape |\n|----------|------|-------|\n| Non-streaming success | `200` | `{\"status\": \"success\", \"data\": ...}` |\n| Non-streaming business/validation error | `200` | `{\"status\": \"error\", \"error\": {\"code\", \"message\", \"retry\", \"request_id\"}}` |\n| Non-streaming auth/authorization error | `401` / `403` | same structured error body |\n| Streaming preflight error | `200` | `application/json` + `{\"status\": \"error\", \"error\": ...}` |\n| Streaming in-flight fatal error | `200` | NDJSON line: `{\"type\": \"error\", \"error\": ...}` |\n| No Flask route / too large / unhandled crash | `404` / `413` / `500` | transport-level error |\n\nDo not use HTTP `400`/`422` for application validation errors in new code.\nDo not convert in-flight NDJSON errors to `status: \"error\"`; once the stream has\nstarted, event `type` is the protocol discriminator.\n\n## Backend: Adding a New API Endpoint\n\n### HTTP Status Code Policy\n\n**Application-controlled business and validation errors return HTTP 200** with\n`status: \"error\"` in the body. Only these use non-200:\n- `401`/`403` — auth errors (`AUTH_REQUIRED`, `AUTH_EXPIRED`, `ACCESS_DENIED`)\n- `404` — no matching Flask route\n- `413` — WSGI body limit exceeded\n- `500` — unhandled exception (program bug)\n\n### Non-streaming endpoint\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok\n\n@bp.route('/my-endpoint', methods=['POST'])\ndef my_endpoint():\n content = request.get_json()\n if not content.get('required_field'):\n raise AppError(ErrorCode.INVALID_REQUEST, \"Missing required_field\")\n\n try:\n result = do_work(content)\n except SomeBusinessError as e:\n raise AppError(ErrorCode.DATA_LOAD_ERROR, \"Failed to load data\") from e\n except Exception as e:\n from data_formulator.error_handler import classify_and_wrap_llm_error\n raise classify_and_wrap_llm_error(e) from e\n\n return json_ok(result)\n# Global handler returns: HTTP 200 + {\"status\": \"error\", \"error\": {code, message, retry}}\n# Auth errors (AUTH_REQUIRED/AUTH_EXPIRED/ACCESS_DENIED) return 401/403\n```\n\nLegacy `{\"status\": \"error\", \"message\": \"...\"}`, `error_message`, bare `{error}`,\nand `status: \"ok\"` responses are historical formats. Do not add new compatibility\nbranches for them; migrate the route to `json_ok()` / `AppError` before using\n`apiRequest()`.\n\n### Streaming endpoint\n\nValidation MUST be outside the generator. Failures return 200 JSON (not NDJSON).\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\n@bp.route('/my-stream', methods=['POST'])\ndef my_stream():\n if not request.is_json:\n return stream_preflight_error(\n AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\")\n )\n\n content = request.get_json()\n client = get_client(content['model'])\n\n def generate():\n try:\n for event in agent.run(...):\n yield json.dumps(event, ensure_ascii=False) + \"\\n\"\n except Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n return Response(stream_with_context(generate()), mimetype='application/x-ndjson')\n```\n\nStreaming runtime errors intentionally use `{\"type\": \"error\", \"error\": ...}`.\nThey cannot use a top-level `status` envelope because the HTTP response and NDJSON\nevent stream have already started.\n\n## Frontend: Consuming an API\n\n### Non-streaming\n\n```typescript\nimport { apiRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n const { data } = await apiRequest(getUrls().MY_ENDPOINT, {\n method: 'POST',\n body: JSON.stringify(payload),\n headers: { 'Content-Type': 'application/json' },\n });\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\nFor UI loading state, model the request lifecycle explicitly with\n`LoadableState` from `src/app/loadableState.ts`. Do not infer loading from\nmissing data (`!data`), because failed requests may legitimately leave data\nempty while loading has ended.\n\n### Streaming\n\n```typescript\nimport { streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n for await (const event of streamRequest(url, options, abortController.signal)) {\n switch (event.type) {\n case 'text_delta':\n break;\n case 'error':\n // Error arrived mid-stream — show inline in component.\n break;\n case 'done':\n break;\n }\n }\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\n### With callbacks\n\n```typescript\nhandleApiError(e, 'MyComponent', {\n onAuth: () => redirectToLogin(), // AUTH_REQUIRED / AUTH_EXPIRED\n onRetryable: () => retryOperation(), // LLM_RATE_LIMIT / LLM_TIMEOUT\n silent: true, // don't show Snackbar (component handles display)\n});\n```\n\n### Migration and special cases\n\nDF API consumers should use `apiRequest()` / `streamRequest()` and\n`handleApiError()`. Direct `fetchWithIdentity()` is for lower-level client helpers\nand explicit protocol exceptions such as file downloads, blob/CSV responses,\nOIDC redirects, SPA fallback, or third-party URLs.\n\nDo not apply the normal JSON API protocol mechanically to file downloads / CSV\nstreaming, SPA fallback, OIDC redirect flows, frontend fetches to third-party\nURLs, or errors after a streaming response has already started. Check the route's\nprotocol first, then preserve safe error bodies and avoid `str(exc)` exposure.\n\n## Adding a New Error Code\n\n1. **Backend** — Add to `py-src/data_formulator/errors.py` `ErrorCode`:\n ```python\n MY_NEW_ERROR = \"MY_NEW_ERROR\"\n ```\n No HTTP mapping needed — defaults to HTTP 200. Only add to `ERROR_CODE_HTTP_STATUS` if it's an auth code.\n\n2. **Frontend mapping** — Add to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`:\n ```typescript\n MY_NEW_ERROR: 'errors.myNewError',\n ```\n\n3. **Translations** — Add to both locale files:\n - `src/i18n/locales/en/errors.json`: `\"myNewError\": \"English message\"`\n - `src/i18n/locales/zh/errors.json`: `\"myNewError\": \"中文消息\"`\n\n## Migrated Endpoints Reference\n\nAll streaming endpoints are now on the unified protocol:\n\n| Endpoint | Format | Notes |\n|----------|--------|-------|\n| `/data-agent-streaming` | NDJSON + `stream_error_event()` | Emits top-level `type` events; errors use `{type:\"error\", error:{...}}` |\n| `/get-recommendation-questions` | NDJSON + `stream_error_event()` | Was `error: {json}` prefix |\n| `/generate-report-chat` | Pure NDJSON + `stream_error_event()` | Was SSE `data: {json}` prefix |\n| `/data-loading-chat` | NDJSON + `stream_error_event()` | `str(e)` removed |\n| `/clean-data-stream` | NDJSON + `stream_error_event()` | Was `\\n{json}\\n` format |\n\nNon-streaming endpoints:\n\n| Endpoint | Error Format | Notes |\n|----------|-------------|-------|\n| `/chart-insight` | `AppError` → HTTP 200 + `{status:\"error\", error:{code,message,retry}}` | Fully migrated. Frontend uses `fetchChartInsight` rejected reducer. |\n| All migrated endpoints | `AppError` → HTTP 200 + unified error body | credentials, knowledge, sessions, tables, agents |\n| `/derive-data`, `/refine-data`, `/sort-data`, `/process-data-on-load`, `/test-model` | `json_ok()` / `AppError` | Migrated to new format |\n\n## Empty Catch Policy\n\nNot all `.catch(() => {})` are bugs. Use this decision tree:\n\n1. **User-initiated action** (delete, refresh, submit) → must notify with `addMessages` or `handleApiError()`\n2. **Background/best-effort fetch** (connector list on mount, session list) → OK to swallow, but add a comment\n3. **RTK thunks** → always add `.rejected` handler with `addMessages`\n4. **AbortError** → filter out with `if (action.error?.name !== 'AbortError')`\n\n## Frontend Stream Parsing Pattern\n\nWhen consuming a migrated streaming endpoint, handle the current NDJSON event\nformat directly:\n\n```typescript\nconst data = JSON.parse(line);\nif (data.type === 'error') {\n const errMsg = data.error?.message || 'Unknown error';\n // show to user...\n}\n```\n\n## Backend: Database/Workspace Errors (tables.py)\n\nFor table CRUD endpoints, use the specialized classifier:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\n@tables_bp.route('/my-table-op', methods=['POST'])\ndef my_table_op():\n try:\n result = workspace.do_something()\n return jsonify({\"status\": \"success\", \"data\": result})\n except Exception as e:\n classify_and_raise_db_error(e)\n```\n\n`classify_and_raise_db_error` maps common DB errors to appropriate `AppError` codes\n(returned as HTTP 200 by the global handler, except ACCESS_DENIED → 403):\n- \"Table does not exist\" → `TABLE_NOT_FOUND` (HTTP 200)\n- \"Table already exists\" → `INVALID_REQUEST` (HTTP 200)\n- \"Permission denied\" → `ACCESS_DENIED` (HTTP 403)\n- Other → `CONNECTOR_ERROR` (HTTP 200)\n\n## Backend: Connector Errors (data_connector.py)\n\nFor connector endpoints, use:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e, operation=\"preview\")\n```\n\nConnector/DataLoader classification is intentionally simple and lives in\n`data_formulator.data_loader.connector_errors`. It maps common failures to a\nsmall stable set: `INVALID_REQUEST`, `CONNECTOR_AUTH_FAILED`, `AUTH_EXPIRED`,\n`ACCESS_DENIED`, `DB_CONNECTION_FAILED`, `DB_QUERY_ERROR`, `DATA_LOAD_ERROR`,\nor `CONNECTOR_ERROR`. Do not add endpoint-local string matching unless the\nclassifier cannot reasonably cover the category.\n\nAll JSON errors include `error.request_id` and an `X-Request-Id` response\nheader. Show/copy this ID for users when reporting backend failures; do not\nshow raw exception text in production. Unhandled 500 responses must never\ninclude raw tracebacks, even in debug mode; return a safe category plus\n`request_id` and keep full stack traces in server logs only.\n\n## Debugging Error Propagation\n\nWhen an error isn't reaching the frontend:\n\n1. **Check backend logs** — is the error logged?\n2. **Check response format**:\n - Non-streaming: `{\"status\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n - Streaming: one line `{\"type\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n3. **Check Content-Type** — streaming must be `application/x-ndjson`, not `application/json` or `text/event-stream`\n4. **Check frontend parser** — is the consumer looking for `data.type === 'error'`?\n5. **Check global handler** — verify `register_error_handlers(app)` is called in `app.py`\n6. **Check blueprint handlers** — blueprint-level `errorhandler(Exception)` takes priority over global handlers\n\nLegacy `message` / `error_message` bodies are protocol violations on migrated API paths.\n\n## Log Sanitization (Sensitive Data in Server Logs)\n\nServer-side logs must never leak passwords, tokens, API keys, or connection strings.\nThe project uses a defense-in-depth approach with two layers.\n\n### Layer 1: Explicit Utilities (call-site)\n\n```python\nfrom data_formulator.security.log_sanitizer import (\n sanitize_url, sanitize_params, redact_token,\n)\n\n# Dict with credentials → sanitize_params()\nlog.info(\"Connecting with: %s\", sanitize_params(params))\n\n# URL that may embed credentials → sanitize_url()\nlogger.info(\"Issuer: %s\", sanitize_url(issuer_url))\n\n# Token/API key → redact_token()\nlogger.debug(\"Token: %s\", redact_token(token))\n```\n\n### Layer 2: SensitiveDataFilter (global safety net)\n\nRegistered in `app.py:configure_logging()`. Automatically redacts:\n- URL credentials (`://user:pass@host`)\n- `Bearer` tokens\n- `password=xxx`, `api_key=xxx`, `secret=xxx` patterns\n- JWT-like base64 strings\n- Python dict repr with sensitive keys\n\nDisable with `LOG_SANITIZE=false` for local debugging only.\n\n### When to Use What\n\n| Data | Utility | Why not just filter? |\n|------|---------|---------------------|\n| `dict` with password keys | `sanitize_params()` | Filter can't identify arbitrary password values in dict repr |\n| URL from config/env | `sanitize_url()` | Explicit is clearer; filter is backup |\n| Token/key value | `redact_token()` | Explicit is clearer; filter is backup |\n| Normal text | Nothing | Filter handles edge cases |\n\n### New Module Checklist\n\nWhen adding a module that handles credentials or external services:\n\n1. Audit all `logger.*()` calls for credential/URL/token logging\n2. Use `sanitize_params()` for dicts, `sanitize_url()` for URLs, `redact_token()` for tokens\n3. Prefer `type(exc).__name__` over `str(exc)` in warning-level logs\n4. If introducing new credential key names, add to `SENSITIVE_KEYS` in `log_sanitizer.py`\n\n## Key Files\n\n| File | Purpose |\n|------|---------|\n| `py-src/data_formulator/errors.py` | `ErrorCode` enum + `AppError` exception |\n| `py-src/data_formulator/error_handler.py` | Global handlers, `classify_and_wrap_llm_error`, `stream_error_event` |\n| `py-src/data_formulator/security/log_sanitizer.py` | `sanitize_url`, `sanitize_params`, `redact_token`, `SensitiveDataFilter` |\n| `py-src/data_formulator/routes/tables.py` | `classify_and_raise_db_error` (database/workspace errors) |\n| `py-src/data_formulator/data_connector.py` | `classify_and_raise_connector_error` (connector errors) |\n| `py-src/data_formulator/security/sanitize.py` | `classify_llm_error` (internal), `sanitize_error_message` |\n| `src/app/apiClient.ts` | `apiRequest`, `streamRequest`, `parseStreamLine`, `ApiRequestError` |\n| `src/app/errorHandler.ts` | `handleApiError` |\n| `src/app/errorCodes.ts` | `ERROR_CODE_I18N_MAP`, `getErrorMessage` |\n| `src/i18n/locales/{en,zh}/errors.json` | Error message translations |\n","frontmatter":{"name":"error-handling","description":"统一错误处理系统。在添加 API 端点、修改错误处理、添加前端 API 调用、编写错误相关测试时使用。"},"isInternal":false,"tokens":3622,"sizeBytes":15424},{"name":"SKILL.md","path":".cursor/skills/language-injection/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/skills/language-injection/SKILL.md","title":"language-injection","category":"anthropic-skill","format":"markdown","content":"---\r\nname: language-injection\r\ndescription: LLM Agent 多语言注入规范。在修改 Agent 提示词、添加新的 Agent 端点、处理用户可见的后端消息（message_code）时使用。\r\n---\r\n\r\n# Language Injection for Agent Prompts\r\n\r\nAuthoritative developer guide: `docs/dev-guides/6-i18n-language-injection.md`.\r\n\r\n> **Prerequisites**: Read `docs/dev-guides/6-i18n-language-injection.md` before changing Agent prompts, Agent routes, backend user-visible messages, or frontend i18n strings.\r\n> If your work introduces new language injection patterns or conventions, update this file and related dev-guides accordingly.\r\n\r\n## Architecture\r\n\r\n```\r\nFrontend i18n.language → Accept-Language header → get_language_instruction()\r\n │\r\n build_language_instruction()\r\n (agents/agent_language.py)\r\n │\r\n ┌────────────┴────────────┐\r\n ▼ ▼\r\n mode=\"full\" mode=\"compact\"\r\n (text-heavy agents) (code-gen agents)\r\n```\r\n\r\n### Core Modules\r\n\r\n| Module | Role |\r\n|--------|------|\r\n| `agents/agent_language.py` | `build_language_instruction(lang, mode)` — generates prompt fragments; `inject_language_instruction()` — injects into system prompts; supports 20 languages; returns `\"\"` for English |\r\n| `routes/agents.py` → `get_language_instruction()` | Reads `Accept-Language` header, delegates to `build_language_instruction` |\r\n| `routes/agents.py` → `_get_ui_lang()` | Extracts primary language code from `Accept-Language` header |\r\n| `src/app/utils.tsx` → `fetchWithIdentity()` | Sets `Accept-Language` header on every API request from `i18n.language` |\r\n| `src/app/utils.tsx` → `translateBackend()` | Translates backend `message_code` / `content_code` using frontend i18n |\r\n\r\n## Code Examples\r\n\r\n### Route handler — inject language\r\n\r\n```python\r\n# In a Flask route handler:\r\nlang_instruction = get_language_instruction(mode=\"compact\")\r\nlang_suffix = f\"\\n\\n{lang_instruction}\" if lang_instruction else \"\"\r\n\r\nmessages = [\r\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\" + lang_suffix},\r\n {\"role\": \"user\", \"content\": user_input},\r\n]\r\n```\r\n\r\n### Agent constructor — use inject_language_instruction()\r\n\r\n```python\r\nfrom data_formulator.agents.agent_language import inject_language_instruction\r\n\r\n# Simple append (most agents)\r\nsystem_prompt = inject_language_instruction(system_prompt, language_instruction)\r\n\r\n# Insert before a marker (complex prompts)\r\nsystem_prompt = inject_language_instruction(\r\n system_prompt, language_instruction,\r\n marker=\"**About the execution environment:**\"\r\n)\r\n```\r\n\r\n### Python-side user-visible messages — message_code pattern\r\n\r\nFor fixed strings in Python that appear in the UI, do NOT translate in Python.\r\nReturn a `message_code` and let the frontend translate:\r\n\r\n```python\r\n# In an Agent or route handler:\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\", # English fallback\r\n \"message_code\": \"agent.emptyDataframe\", # frontend i18n key\r\n}\r\n\r\n# With parameters:\r\nresult = {\r\n \"status\": \"error\",\r\n \"content\": f\"Fields not found: {missing}\",\r\n \"content_code\": \"agent.fieldsNotFound\",\r\n \"content_params\": {\"missing\": missing, \"available\": available},\r\n}\r\n```\r\n\r\nFrontend consumption:\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n```\r\n\r\nTranslation keys go in `src/i18n/locales/{en,zh}/messages.json` under `messages.agent.*`.\r\n\r\n## Anti-Patterns (with explanations)\r\n\r\n| Pattern | Why it's wrong |\r\n|---------|---------------|\r\n| `os.environ.get(\"DF_DEFAULT_LANGUAGE\")` | Process-level — all users get same language; breaks multi-user |\r\n| Global LLM client interceptor | Hidden behavior; can't distinguish full/compact mode; fragile string detection |\r\n| New `MessageBuilder` class | Duplicates `agent_language.py`; creates parallel conflicting abstractions |\r\n| Hardcoded `\"回答请使用中文\"` in prompts | Not configurable; skips the mode system; breaks for other languages |\r\n| Backend-side translation dict (`agent_messages.py`) | Forces adding every new language to Python; translations should all live in `src/i18n/locales/` |\r\n| Hardcoded English UI strings in `.tsx` without `t()` | Not translatable; use `useTranslation` + `t('key')` |\r\n\r\n## Adding a New Language\r\n\r\n1. Add language code + display name to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py`.\r\n2. Optionally add extra rules to `LANGUAGE_EXTRA_RULES` (e.g. simplified vs traditional Chinese).\r\n3. Add frontend translations in `src/i18n/locales//` — copy an existing locale folder as template.\r\n4. No Agent code changes needed — the existing flow picks up new languages automatically.\r\n","frontmatter":{"name":"language-injection","description":"LLM Agent 多语言注入规范。在修改 Agent 提示词、添加新的 Agent 端点、处理用户可见的后端消息（message_code）时使用。"},"isInternal":false,"tokens":1078,"sizeBytes":5214},{"name":"SKILL.md","path":".cursor/skills/path-safety/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/.cursor/skills/path-safety/SKILL.md","title":"path-safety","category":"anthropic-skill","format":"markdown","content":"---\r\nname: path-safety\r\ndescription: 服务端路径安全与文件访问编码规范。在编写文件下载路由、Agent 工具（文件读取/目录列出）、数据连接器/Loader、Workspace 路径操作、沙箱配置时使用。\r\n---\r\n\r\n# Path Safety — 服务端安全编码规范\r\n\r\n> **来源**：`docs/dev-guides/8-path-safety.md`（正式开发规范）+ `design-docs/issues/002-arbitrary-file-read-audit.md`（安全审计复核）。\r\n> 本文档提炼了 6 条必须遵守的编码规范。违反任一条即可能引入路径穿越（LFI）漏洞。\r\n\r\n---\r\n\r\n## R1. 文件下载：用 `ConfinedDir.resolve()` + `send_file`，禁用 `send_from_directory`\r\n\r\n**原因**：`send_from_directory(dir, user_input)` 内部会对 `user_input` 二次解析路径，与前置安全检查形成 TOCTOU 不一致。\r\n\r\n```python\r\n# ❌ BAD — 安全检查用 resolved target，发送用原始 filename，两次解析不一致\r\ntarget = (scratch_dir / filename).resolve()\r\ntarget.relative_to(scratch_dir.resolve()) # 检查通过\r\nreturn send_from_directory(str(scratch_dir), filename) # 再次解析\r\n\r\n# ✅ GOOD — 检查和发送用同一个 resolved path\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target) # 直接用已验证的路径\r\n```\r\n\r\n`send_file(Path)` 会根据扩展名自动推断 MIME type，无需额外处理。\r\n\r\n---\r\n\r\n## R2. 路径安全检查：用 `ConfinedDir`，禁止 `str.startswith`\r\n\r\n**原因**：`str(path).startswith(str(root))` 存在前缀碰撞缺陷（如 `/workspace` vs `/workspace_evil`）。\r\n\r\n```python\r\n# ❌ BAD\r\nif not str(resolved).startswith(str(root_resolved) + os.sep):\r\n raise ValueError(\"escape\")\r\n\r\n# ✅ GOOD — 统一走 ConfinedDir，内部使用 Path.is_relative_to()\r\njail = ConfinedDir(root_resolved, mkdir=False)\r\ntarget = jail.resolve(user_input)\r\n```\r\n\r\n---\r\n\r\n## R3. Agent 工具复用 `Workspace.confined_*`\r\n\r\n**原因**：Agent 工具参数由 LLM 生成，必须视为间接用户输入。不要在工具内手写 `Path(root) / rel_path` 或 `resolve() + relative_to()`；入口处复用 Workspace 暴露的 `ConfinedDir`。\r\n\r\n```python\r\n# ❌ BAD — 手写路径拼接和校验\r\ndef _tool_read_file(self, args, workspace_path):\r\n target = (workspace_path / rel_path).resolve()\r\n target.relative_to(workspace_path)\r\n\r\n# ✅ GOOD — 入口拿到 ConfinedDir，工具只调用 jail.resolve()\r\ndef _execute_tool(self, name, args):\r\n workspace_jail = self.workspace.confined_root\r\n scratch_jail = self.workspace.confined_scratch\r\n return self._tool_read_file(args, workspace_jail)\r\n\r\ndef _tool_read_file(self, args, workspace_jail):\r\n target = workspace_jail.resolve(args.get(\"path\", \"\"))\r\n```\r\n\r\n---\r\n\r\n## R4. 优先使用 `ConfinedDir`，禁止裸路径拼接\r\n\r\n**原因**：`Path(root) / user_input` 是路径穿越的高频入口。`ConfinedDir` 封装了三层防御（拒绝绝对路径 → 拒绝 `..` 段 → `resolve` + `is_relative_to`）。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\n# ❌ BAD — 手动拼接 + 手动校验，容易遗漏\r\nlocal_file = tmp_path / blob_relative_name\r\nlocal_file.parent.mkdir(parents=True, exist_ok=True)\r\nlocal_file.write_bytes(data)\r\n\r\n# ✅ GOOD — ConfinedDir 自动校验 + 创建父目录\r\njail = ConfinedDir(tmp_path, mkdir=False)\r\njail.write(blob_relative_name, data) # 自动校验 + 写入\r\n```\r\n\r\n### 已有安全 API 的层次关系\r\n\r\n```\r\n用户输入（filename / relative_path / blob key）\r\n │\r\n ▼\r\nsafe_data_filename() / secure_filename() ← 第一层：输入清洗\r\n │\r\n ▼\r\nConfinedDir.resolve() ← 第二层：路径约束\r\n │\r\n ▼\r\n安全的 Path 对象\r\n```\r\n\r\n使用 `Workspace.get_file_path()` 的场景不需要手动调用 `ConfinedDir`，因为它内部已包含等价的校验。\r\n\r\n---\r\n\r\n## R5. 宿主文件系统访问必须设部署模式守卫\r\n\r\n**原因**：桌面单用户模式允许访问本机文件系统（预期行为），但多用户/云部署下等于开放服务器读权限。\r\n\r\n**规范**：任何新的 Loader / Connector 如果涉及直接读取宿主文件系统（不通过 Workspace API），必须：\r\n\r\n1. 在 `data_loader/__init__.py` 的 `_enforce_deployment_restrictions()` 中注册禁用规则\r\n2. 确保 `create_connector()` 会拒绝已禁用的类型\r\n\r\n```python\r\n# data_loader/__init__.py — 参考 local_folder 的处理方式\r\ndef _enforce_deployment_restrictions():\r\n backend = os.environ.get(\"WORKSPACE_BACKEND\", \"local\")\r\n if backend != \"local\":\r\n for key in (\"local_folder\", \"your_new_local_loader\"):\r\n if key in DATA_LOADERS:\r\n del DATA_LOADERS[key]\r\n DISABLED_LOADERS[key] = f\"{key} disabled in multi-user mode\"\r\n```\r\n\r\n**判断标准**：如果 Loader 的构造函数接受一个用户可控的本机路径（如 `root_dir`），它就需要部署守卫。\r\n\r\n---\r\n\r\n## R6. 多用户部署必须启用沙箱\r\n\r\n**原因**：`not_a_sandbox` 模式下 LLM 生成的代码在宿主进程直接执行，可绕过所有路径检查。\r\n\r\n`app.py` 已在启动时检测此配置并输出 `logger.critical` 警告。新增的沙箱模式或部署脚本应确保：\r\n\r\n- `WORKSPACE_BACKEND != \"local\"` 时，`SANDBOX` 必须为 `docker` 或 `local`\r\n- CI/CD 部署模板中默认设置 `SANDBOX=docker`\r\n\r\n---\r\n\r\n## 速查：新增代码时的安全检查清单\r\n\r\n| 场景 | 必须做的事 |\r\n|------|-----------|\r\n| 新增文件下载路由 | 用 `ConfinedDir.resolve()` 得到路径，再 `send_file(resolved_path)`；不用 `send_from_directory` |\r\n| 新增 Agent 工具（读文件/列目录） | 入口复用 `workspace.confined_root` / `workspace.confined_scratch`，工具内只调用 `jail.resolve()` |\r\n| 路径包含判断 | 用 `ConfinedDir.resolve()`，不要手写 `Path.is_relative_to()` 或 `str.startswith()` |\r\n| `Path(root) / variable` 模式 | 改用 `ConfinedDir` 或 `Workspace.get_file_path()` |\r\n| 新增本机文件系统 Loader | 在 `_enforce_deployment_restrictions()` 中注册多用户禁用 |\r\n| 部署配置 | 多用户模式必须 `SANDBOX=docker` 或 `SANDBOX=local` |\r\n\r\n---\r\n\r\n## 参考文档\r\n\r\n- `docs/dev-guides/8-path-safety.md` — 服务端路径安全开发规范\r\n- `design-docs/6-path-safety-confined-dir.md` — 剩余未完成实现项状态页\r\n- `design-docs/issues/002-arbitrary-file-read-audit.md` — 安全审计复核报告\r\n- `py-src/data_formulator/security/path_safety.py` — ConfinedDir 源码\r\n","frontmatter":{"name":"path-safety","description":"服务端路径安全与文件访问编码规范。在编写文件下载路由、Agent 工具（文件读取/目录列出）、数据连接器/Loader、Workspace 路径操作、沙箱配置时使用。"},"isInternal":false,"tokens":1994,"sizeBytes":6581},{"name":"SKILL.md","path":"py-src/data_formulator/analyst/skills/core/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/core/SKILL.md","title":"core","category":"anthropic-skill","format":"markdown","content":"---\nname: core\ndescription: >-\n The analyst's built-in capabilities: data-inspection tools and the\n always-available actions (visualize and ask_user).\nwhen_to_use: Always loaded by default — this is the agent's baseline.\nalways_on: true\ntools:\n - execute_python_script\n - inspect_source_data\nactions:\n - visualize\n - ask_user\n---\n\n# Core capabilities\n\nThis describes the built-in **inspection tools** you use to gather data and the\nalways-available **actions** you take on it. The overall loop, your action\nbudget, and the one-action-per-turn rule are covered in your system\ninstructions — this section is about *what* each tool and action does and how\nto use it well.\n\n## Tools (for data gathering)\n\n- **execute_python_script(code)** — run a general-purpose Python script to\n inspect data, compute stats, transform tables, or verify assumptions. Its\n stdout is returned to you (use `print()`); the script is for *your* analysis\n and its output is never shown to the user. pandas, numpy, duckdb, sklearn,\n scipy are available. **Important**: each call runs in a fresh namespace —\n variables do NOT persist between calls, so combine related steps into a\n single script.\n- **inspect_source_data(table_names)** — get schema, stats, and sample rows for\n source tables (cheaper than `execute_python_script` for basic inspection).\n- **load_skill(name)** — load a skill's instructions into context so you can use\n the action it unlocks (see the Skills section of your system instructions).\n\nThese are inspection tools — their results come back to you and are never shown\nto the user; call as many as you need, then take an action or give your final\nanswer.\n\nYou analyse data that is **already in the workspace**. If the user's question\nrequires connected data that isn't present, call `load_skill(\"data-loading\")`\nand follow that skill's discovery and immutable proposal workflow in this same\nconversation. Do not hand off to the standalone Data Loading agent.\n\nThe initial context already includes sample rows and statistics for each table.\nIf the data is straightforward, go straight to the action without calling\ntools. Tool results are returned to you before you act.\n\n## Actions\n\nCall an action as a tool call when you want to act on the data. Actions are\n**sequential**: take **one at a time**, then read the result it returns before\ndeciding the next — each action's outcome shapes the next one (the chart you draw\nnext depends on what this one reveals), so emitting several at once would decide\nthe later ones blind. After each result you choose what to do — take another\naction, or stop. **You end your turn by replying with plain text and no\naction**: that is your closing answer when you expect nothing further. When you\nwant the user to reply — a freeform question, a clarification you need before\nacting, or **clickable choices** — use the `ask_user` action instead. It renders\na question widget and pauses for their reply, keeping the conversation in the\nsame turn (plain text ends the run, so the user's next message would start\nfresh without this context).\n\n**Match the response to what the user asked for.** Two different cases:\n\n- **A direct answer** — they asked a question, so answer it. Length follows the\n question: one line when that settles it, more when it genuinely takes more.\n- **A finding after you acted** — they asked for the work, not a write-up, so\n this is unsolicited. The artifact already shows what it shows; add only what\n they'd miss by looking at it, and default to short.\n\nOpen with the point rather than announcing one is coming, and don't close by\nrestating what you just said.\nNever narrate what you're about to do or recap a chart's axes; let the artifact\nspeak for itself. When an action pauses for the user, give enough context to\nexplain what you found and what their choices mean.\n\n### `visualize` — chart a transform\n\nRun code that produces a DataFrame and render it as a chart. You then observe the\nresult and decide your next move.\n\n- `display_instruction` — ≤12 words; the question/hypothesis the chart\n investigates (don't recap x/y/color — those are visible). Wrap a **column** in\n `**…**` if it anchors the question.\n- `title` — a concise, neutral analytical heading naming the subject, measure,\n and analytical lens, such as “Year-over-year price change peaks.” Prefer a\n stable description of the view over a takeaway claim or narrated trend. Do\n not mention the chart type, imply causality, or editorialize. This field is\n required; put interpretation in the closing response instead.\n- `subtitle` — concise supporting context not already clear from the title or\n axes. Use one phrase of at most 16 words to provide contextual details. Do\n not restate the measure or analytical lens named in the title.\n- `code` — Python producing a DataFrame assigned to `output_variable`.\n- `output_variable` — snake_case name the code assigns.\n- `chart` — `{chart_type, encodings:{x,y,…}, config:{}}` (chart_type from the\n chart type reference).\n- `input_tables` — workspace table names, as listed in the available-tables\n context, that the code reads.\n- `field_metadata` — field → semantic annotation. Include units, index\n baselines, intrinsic domains, and ordinal order when supported by the data;\n never invent a unit. Distinguish percentages from percentage points and\n identifiers from quantities.\n- `field_display_names` — field → concise human-readable label for axes,\n legends, and table headers. Expand technical names, preserve established\n domain abbreviations, include units when useful, and use the user's language.\n\nSilently classify the analytical intent before choosing a chart: comparison,\ntrend, distribution, relationship, composition, deviation, ranking,\nuncertainty, or spatial pattern. Choose encodings and chart type from that\nintent and the data shape. Set ordering deliberately: chronological for time,\nsemantic order for ordinal fields, and measure order for rankings. Avoid line\ncharts or legends with excessive series, labels that collide, and color that\ndoes not encode additional information; aggregate, bin, facet, or limit\ncategories when needed without hiding material data.\n\n### `ask_user` — ask the user and pause for their reply (pauses the run)\n\nAsk the user something and pause for their input. Reach for this on **any** turn\nwhere you want a reply — a choice to make, a clarification you need before\nacting, or a brief statement paired with clickable follow-ups they can react to.\nPrefer it over ending your turn with a plain-text question: plain text ends the\nrun (the user's next message starts a fresh turn without this context), while\n`ask_user` keeps the conversation in the same turn.\n\n- `questions` — 1–3 items, each something the user **acts on**: a choice\n (`single_choice` with `options`) or an open question they type an answer to\n (`free_text`). Put your reasoning, rationale, and context in your reply text —\n **not** here. Never add a `questions` item that only states a rationale or\n explanation with nothing for the user to answer or click.\n- each question: `text` (wrap a **column** in `**…**`), `responseType`\n (`single_choice` when you offer `options`, else `free_text` — the user types\n their own open-ended answer, not a slot for your exposition), `required`\n (`true` when the run depends on the answer, `false` for an optional follow-up),\n and `options` (plain-text choices, **at most 3** — just the most likely\n answers; the user can always type a freeform reply, so don't enumerate every\n case).\n\nThis is **terminal**: the run pauses after it and resumes when the user replies.\n\n## Choosing what to do\n\nMatch the response depth to the user's request. Create charts that materially\ncontribute to the answer, and stop when the answer is sufficient.\n\n- For conceptual or informational questions, answer directly when a chart would\n not improve the answer.\n- For specific analytical questions, create the view or views needed to answer\n them clearly.\n- For diagnostic or exploratory questions, follow relevant findings across\n multiple views when doing so adds meaningful insight.\n- If essential intent is unclear, use `ask_user` rather than guessing.\n- *Missing data* (needs tables not in the workspace):\n `load_skill(\"data-loading\")`, discover the source, and propose immutable\n loading options inline.\n- *Report / write-up request* (e.g. \"write a report on X\", \"summarize the findings\n as a narrative\"): this needs the **report** skill — `load_skill(\"report\")` and\n follow it to commit the `write_report` action. **Do this as your very first\n move when charts already exist** (see `[AVAILABLE CHARTS]` / the thread): don't\n re-create them — load the report skill straight away and embed the existing\n charts by id. Only produce a new chart first if the report genuinely needs one\n that isn't there yet (0–3, judgment-based), then load the skill.\n\nFollow explicit requests about scope, depth, and format. **Never** repeat a\nvisualization already in the trajectory or in another thread.\n\n## Chart Creation Guide\n\nThe following reference material applies when you call the `visualize` tool.\n\n### A. Code Execution Rules\n\n**About the execution environment:**\n- You can use BOTH DuckDB SQL and pandas operations in the same script\n- The script will run in the workspace data directory (all data files are in the current directory)\n- Each table in [CONTEXT] has a **file path** (e.g., `student_exam.parquet`, `sales.csv`). Use EXACTLY that path to load data:\n - `.parquet`: `pd.read_parquet('file.parquet')` or DuckDB `read_parquet('file.parquet')`\n - `.csv`: `pd.read_csv('file.csv')` or DuckDB `read_csv_auto('file.csv')`\n - `.json`: `pd.read_json('file.json')`\n - `.xlsx`/`.xls`: `pd.read_excel('file.xlsx')`\n - `.txt`: `pd.read_csv('file.txt', sep='\\t')`\n- **IMPORTANT:** Use the exact filename from the context — do NOT change the file extension or assume all files are parquet.\n- **Allowed libraries:** pandas, numpy, duckdb, math, datetime, json, statistics, collections, re, sklearn, scipy, random, itertools, functools, operator, time\n- **Not allowed:** matplotlib, plotly, seaborn, requests, subprocess, os, sys, io, or any other library not listed above.\n- File system access (open, write) and network access are also forbidden.\n\n**When to use DuckDB vs pandas:**\n- **Prefer plain pandas** for most tasks — it's simpler and more readable.\n- Only use DuckDB when the dataset is very large and you need efficient SQL aggregations, filtering, joins, or window functions.\n- You can combine both: DuckDB for initial loading/filtering on large files, then pandas for complex operations.\n\n**Code structure:** standalone script (no function wrapper), imports at top. **CRITICAL:** The final result DataFrame MUST be assigned to the exact variable name you specified in `\"output_variable\"` — the system uses this name to extract the result. For example, if your output_variable is `sales_by_region`, the script must contain `sales_by_region = ...`.\n\n**DuckDB notes:**\n- Escape single quotes with '' (not \\')\n- No Unicode escapes (\\u0400); use character ranges directly: [а-яА-Я]\n- Cast date columns explicitly: `CAST(col AS DATE)`, `CAST(col AS TIMESTAMP)`\n- For complex datetime operations, load data first then use pandas datetime functions\n- Critical identifier quoting rule:\n * If a table/column name contains non-ASCII characters (e.g., Chinese, Japanese, Korean, Cyrillic, etc.), spaces, or punctuation,\n you MUST wrap it in double quotes, e.g. SELECT \"金额\" FROM \"客户表\".\n * Never output placeholder identifiers like your_table_name, your_column, your_condition.\n\n**Datetime handling:**\n- `date` columns contain date-only values (YYYY-MM-DD). `datetime` columns contain date+time (ISO 8601).\n- `time` columns contain time-only values (HH:mm:ss). `duration` columns are time intervals.\n- Year → number. Year-month / year-month-day → string (\"2020-01\" / \"2020-01-01\").\n- Hour alone → number. Hour:min or h:m:s → string. Never return raw datetime objects.\n\n### B. Chart Type Reference\n\nThe `chart_type` value in the `visualize` action MUST be one of the names listed\nbelow (exact spelling, including capitalization). When a row lists multiple\nnames, pick whichever fits the \"when to use\" hint best.\n\n**Choosing a chart — prefer simple, escalate when it fits.** Reach for the\n**Everyday** set first: it answers most questions and is the safest, most\nlegible choice. But when the data or question genuinely fits a **Specialized**\ntype (a distribution's shape, a cumulative curve, a rank race, a before→after,\na geographic pattern…), prefer it — a well-matched specialized chart is more\ninsightful than forcing a generic one. Don't pick a specialized type for\nnovelty; use it because its \"when to use\" condition is met.\n\n**Everyday — reach for these first**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Scatter Plot | x, y, color, size, facet | opacity (0.1–1.0) | Relationships between two quantitative fields |\n| Regression | x, y, color, size, facet | regressionMethod (\"linear\",\"log\",\"exp\",\"pow\",\"quad\",\"poly\"), polyOrder (2–10) | Trend line over scatter; one line per color group |\n| Bar Chart / Stacked Bar Chart / Lollipop Chart / Waterfall Chart | x, y, color, facet | — | Bar: categorical comparison (auto-stacks when color is set). Stacked Bar: explicit stacked totals, color = the stack. Lollipop: cleaner for ranked lists / sparse categories. Waterfall: cumulative gain/loss, each bar starts where the previous ended |\n| Grouped Bar Chart | x, y, group, facet | — | Side-by-side bars across a second categorical dimension |\n| Line Chart | x, y, color, strokeDash, facet | interpolate (\"linear\",\"monotone\",\"step\") | Trends over an ordered (usually temporal) x-axis |\n| Area Chart | x, y, color, facet | — | Magnitude over ordered x; auto-stacks when color is set |\n| Histogram / Density Plot | x, color, facet | — | Distribution of one quantitative field. Histogram: discrete bins, auto-binned. Density Plot: smooth KDE curve |\n| Boxplot | x, y, color, facet | — | Distribution summary (median/quartiles/outliers) by category |\n| Pie Chart | size, color, facet | innerRadius (0–100; 0=pie, >0=donut) | Part-of-whole with ≤7 categories. Wedge value goes on **size**, not **theta** |\n| Heatmap | x, y, color, facet | colorScheme — sequential (\"viridis\",\"blues\",\"reds\",\"oranges\",\"greens\") or diverging (\"blueorange\",\"redblue\") | Matrix / 2D density; color encodes the quantitative cell value |\n\n**Specialized — use when the data/question fits the \"when to use\"**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Connected Scatter Plot | x, y, order, color, facet | — | Two quantitative fields traced in sequence — needs an `order` field (e.g. time) so points are joined in order, not by x |\n| Ranged Dot Plot | x, y, color, facet | — | Min–max range or two-point comparison per category |\n| Violin Plot | x, y, color, facet | — | Distribution SHAPE (KDE silhouette) by category; better than a boxplot when data is multimodal. x = category, y = value |\n| Strip Plot | x, y, color, size, facet | — | Every individual point by category (jittered); good for small/medium n where raw values matter, not just a summary |\n| ECDF Plot | x, color, facet | — | Cumulative distribution of one quantitative field. Pass the RAW field on x (do NOT pre-compute the CDF); color for per-group curves |\n| Bump Chart | x, y, color, facet | — | How RANKINGS change over ordered x; y = rank, color = entity (long-form: one row per entity × x) |\n| Slope Chart | x, y, color, facet | — | Change between exactly TWO points (before → after) per entity; x = the two labels, y = value, color = entity |\n| Streamgraph | x, y, color, facet | — | Several series' magnitude over ordered x, stacked around a center baseline (color = series) — theme/volume shifts over time |\n| Range Area Chart | x, y, y2, color, facet | — | A shaded band between a lower (y) and upper (y2) bound over ordered x — e.g. min–max or a confidence interval |\n| Rose Chart | x, y, color, facet | — | Cyclical/categorical magnitude as angular wedges (polar bars); x = category/angle, y = value |\n| Pyramid Chart | x, y, color, facet | — | Back-to-back bars split by a binary group (e.g. population by age × sex); y = category, x = value, color = the two-sided group |\n| Radar Chart | x, y, color, facet | — | Multi-metric profile/comparison; x = metric name, y = value, color = entity (long-form data) |\n| Bar Table | x, y, color, facet | — | Ranked horizontal table with inline bars; one row per category. y = category, x = value |\n| KPI Card | metric, value, goal | — | \"Big number\" dashboard tile(s); one row per tile. `value` must be pre-aggregated; `goal` is optional |\n| Candlestick Chart | x, open, high, low, close, facet | — | OHLC financial data |\n| Map | longitude, latitude, color, size | projection (\"mercator\",\"equalEarth\",\"naturalEarth1\",\"orthographic\",\"albersUsa\"), projectionCenter ([lon,lat]) | Geographic POINTS/bubbles by lon/lat (use projection \"albersUsa\" for a US-only map) |\n| Choropleth | id, color, facet | region (\"world\",\"usa\",…) | Filled REGIONS shaded by value; `id` = the region key (country/state name or code), color = the quantitative value |\n\n**Critical chart rules:**\n- **Scatter Plot**: use config opacity (0.1–1.0) for dense data instead of encoding opacity.\n- **Regression**: trend line is automatic — do NOT compute regression coefficients/predictions in Python. Use `color` to get separate trend lines per group.\n- **Bar Chart**: x=categorical, y=quantitative (vertical bars). Swap x↔y for horizontal bars. Same-x rows are auto-stacked when `color` is set.\n- **Grouped Bar Chart**: use the `group` channel (not `color`) for side-by-side bars.\n- **Histogram**: do NOT pre-bin in Python — pass the raw quantitative field on `x` and the chart bins automatically. Pre-aggregating gives wrong bin widths.\n- **Line Chart**: use `strokeDash` to differentiate line styles (e.g. actual vs forecast).\n- **Pie Chart**: use the `size` channel (not `theta`) for wedge values. Avoid when >7–8 categories.\n- **Radar Chart**: data must be long-form — one row per (entity, metric, value). If your data is wide-form (one column per metric), melt it first in the Python step.\n- **Heatmap**: pick `colorScheme` by the meaning of the values. Use a **sequential** scheme (viridis/blues/reds/oranges/greens) for single-direction magnitudes (counts, rates, prices, scores — higher is simply more). Use a **diverging** scheme (blueorange/redblue) ONLY when the values have a meaningful center to read away from (e.g. profit/loss around 0, change vs. a baseline, temperature around freezing).\n- **Bar Table**: y is the category column to rank; x is the quantitative value driving bar length. Don't sort in Python — the template sorts.\n- **KPI Card**: channels are `metric`, `value`, `goal` (not x/y). One DataFrame row = one tile. The `value` column must already contain the final number to display (aggregate upstream in the Python step).\n- **Candlestick Chart**: requires `open`, `high`, `low`, `close` columns.\n- **Connected Scatter Plot**: provide an `order` field (usually time) so points are joined in sequence, not by x-order.\n- **ECDF Plot**: pass the RAW quantitative field on `x` — the chart computes the cumulative curve; do NOT pre-compute it in Python.\n- **Range Area Chart**: `y` is the lower bound and `y2` the upper bound of the band.\n- **Bump / Slope Chart**: long-form data — one row per (entity, x); `color` is the entity. Slope's `x` has exactly two categories (before/after).\n- **Violin Plot**: like Boxplot but shows the full distribution shape; x = category, y = value.\n- **Map / Choropleth**: `Map` plots points via `longitude` / `latitude` (set projection `\"albersUsa\"` for the US); `Choropleth` fills regions — put the region key on `id` and the value on `color`, not `x` / `y`.\n- **facet**: available for nearly all chart types; use a low-cardinality categorical field.\n- All fields in `encodings` must also appear in `output_fields`. Typically use 2–3 channels (x, y, color/size).\n\n### C. Semantic Type Reference\n\nChoose the most specific type that fits. Only annotate fields used in chart encodings.\n\n| Category | Types |\n|---|---|\n| Temporal | DateTime, Date, Time, Timestamp, Year, Quarter, Month, Week, Day, Hour, YearMonth, YearQuarter, YearWeek, Decade, Duration |\n| Monetary measures | Amount, Price |\n| Physical measures | Quantity, Temperature |\n| Proportion | Percentage |\n| Signed/diverging | Profit, PercentageChange, Sentiment, Correlation |\n| Generic measures | Count, Number |\n| Discrete numeric | Rank, Score |\n| Identifier | ID |\n| Geographic | Latitude, Longitude, Country, State, City, Region, Address, ZipCode |\n| Entity names | Category, Name |\n| Coded categorical | Status, Boolean, Direction |\n| Binned ranges | Range |\n| Fallback | Unknown |\n\nKey guidelines:\n- Use **Amount** for summed monetary totals, **Price** for per-unit prices, **Profit** for values that can be negative.\n- Use **Temperature** (not Quantity) for temperature — it has special diverging behavior.\n- Use **Year** (not Number) for columns like \"year\" with values 2020, 2021.\n\n### D. Statistical Analysis Guide\n\n- **Regression**: use chart_type \"Regression\" — the trend line is automatic, do NOT compute regression values in Python code. Configure method via `{\"regressionMethod\": \"linear\"}` (options: \"linear\", \"log\", \"exp\", \"pow\", \"quad\", \"poly\"; for poly add `{\"polyOrder\": 3}`).\n- **Forecasting**: compute predicted future values in Python. Use Line Chart with strokeDash to distinguish actual vs forecast, and color for series grouping.\n- **Clustering**: compute cluster assignments in Python. Output [x, y, cluster_id]. Use Scatter Plot with color → cluster_id.\n","frontmatter":{"name":"core","description":"The analyst's built-in capabilities: data-inspection tools and the always-available actions (visualize and ask_user).","when_to_use":"Always loaded by default — this is the agent's baseline.","always_on":true,"tools":["execute_python_script","inspect_source_data"],"actions":["visualize","ask_user"]},"isInternal":false,"tokens":5284,"sizeBytes":21991},{"name":"tools.json","path":"py-src/data_formulator/analyst/skills/core/tools.json","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/core/tools.json","title":"Core Skill","category":"anthropic-skill","format":"json","content":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"execute_python_script\",\n \"description\": \"Execute a general-purpose Python script in the sandbox. Here you use it to inspect data, compute statistics, transform tables, or verify assumptions before you act — write results to stdout with print() and that output is returned to you (it is NOT shown to the user). The script is for your own analysis, not for producing the final visualization. pandas, numpy, duckdb, sklearn, scipy are available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"purpose\": {\n \"type\": \"string\",\n \"description\": \"One-sentence description of what this script does and why (shown to user as progress).\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python script to execute. Use print() to surface output.\"\n }\n },\n \"required\": [\"purpose\", \"code\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_source_data\",\n \"description\": \"Get a detailed summary of one or more source tables — schema, field-level statistics, and sample rows. Cheaper than execute_python_script for basic data inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"table_names\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of workspace table names, as listed in the available-tables context, to inspect.\"\n }\n },\n \"required\": [\"table_names\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"visualize\",\n \"description\": \"Commit a data transform + chart: run code producing a DataFrame and render it. The agent observes the result and continues.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\": \"A concise, neutral analytical heading that names the subject, measure, and analytical lens, such as 'Year-over-year price change peaks'. Prefer a stable description of the view over a takeaway claim or narrated trend. Do not mention the chart type, imply causality, or editorialize. Shown as the chart heading.\"\n },\n \"subtitle\": {\n \"type\": \"string\",\n \"description\": \"Concise supporting context not already clear from the title or axes. Use one phrase of at most 16 words to provide contextual details. Do not restate the measure or analytical lens named in the title.\"\n },\n \"display_instruction\": {\n \"type\": \"string\",\n \"description\": \"≤12 words. State the question or hypothesis the chart investigates — don't recap the chart spec (x/y/color/split are already visible). Wrap a **column** in ** ** if it anchors the question.\"\n },\n \"input_tables\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Workspace table names, as listed in the available-tables context, that the code reads.\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python code producing a DataFrame assigned to output_variable.\"\n },\n \"output_variable\": {\n \"type\": \"string\",\n \"description\": \"snake_case name of the DataFrame variable the code assigns.\"\n },\n \"chart\": {\n \"type\": \"object\",\n \"description\": \"Chart spec: {chart_type, encodings:{x,y,...}, config:{}}. chart_type from the chart type reference.\"\n },\n \"field_metadata\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> SemanticType for the output columns.\"\n },\n \"field_display_names\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> human-readable display name for chart axes and table headers.\"\n }\n },\n \"required\": [\"title\", \"code\", \"output_variable\", \"chart\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"ask_user\",\n \"description\": \"Ask the user something and pause for their reply — the run resumes in the same turn with their answer in context. Use this for ANY turn where you want the user to respond: a choice to make, a clarification you need before acting, or a brief statement paired with clickable follow-ups. Put your reasoning, rationale, and context in your normal reply text, not inside this call. Prefer this over ending your turn with a plain-text question: plain text ends the run and the user's next message starts a fresh turn without this context, whereas ask_user keeps the conversation going. Reserve plain text (no action) for your final answer when you expect nothing further.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"thought\": {\n \"type\": \"string\",\n \"description\": \"Brief rationale (not shown to the user).\"\n },\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"1–3 things the user acts on: a choice (single_choice with options) or an open question they type an answer to (free_text). Put rationale, reasoning, and context in your reply text, not here — never add an item that only states an explanation with nothing for the user to answer or click. An explanation is allowed only as a short statement paired with clickable chart-producing follow-ups (required=false with options).\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The question, or (for an optional follow-up) a short statement. Keep a statement to 1–3 grounded sentences and pair it with clickable follow-up options. Wrap a **column** in ** ** to highlight it.\"\n },\n \"responseType\": {\n \"type\": \"string\",\n \"enum\": [\"single_choice\", \"free_text\"],\n \"description\": \"single_choice when you offer options; free_text when the user types their own open-ended answer (not a slot for your own exposition).\"\n },\n \"required\": {\n \"type\": \"boolean\",\n \"description\": \"false for an explanation / optional follow-up; true for a clarification the run depends on.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Plain-text choices, at most 3. Keep them to the few most likely answers — the user can always type a freeform reply, so don't try to enumerate every case. For a clarification these are answers; for an explanation these are short chart-producing follow-up prompts the user might click next (≤8 words each, phrased as the user would say them).\"\n }\n },\n \"required\": [\"text\"]\n }\n }\n },\n \"required\": [\"questions\"]\n }\n }\n }\n]\n","isInternal":false,"tokens":1512,"sizeBytes":7170},{"name":"SKILL.md","path":"py-src/data_formulator/analyst/skills/data_loading/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/data_loading/SKILL.md","title":"data-loading","category":"anthropic-skill","format":"markdown","content":"---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.","frontmatter":{"name":"data-loading","description":"Discover connected data sources, add new data connectors through a user-confirmed form, inspect table metadata, and run bounded read-only probes when the current workspace data is insufficient.","when_to_use":"The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.","always_on":false,"tools":["list_data","find_data","describe_data","probe_data","list_connectors","describe_connector"],"actions":["propose_data_operation","propose_connection"]},"isInternal":false,"tokens":1240,"sizeBytes":5656},{"name":"tools.json","path":"py-src/data_formulator/analyst/skills/data_loading/tools.json","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/data_loading/tools.json","title":"Data_loading Skill","category":"anthropic-skill","format":"json","content":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]","isInternal":false,"tokens":2024,"sizeBytes":9415},{"name":"SKILL.md","path":"py-src/data_formulator/analyst/skills/data-loading/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/data-loading/SKILL.md","title":"data-loading","category":"anthropic-skill","format":"markdown","content":"---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.","frontmatter":{"name":"data-loading","description":"Discover connected data sources, add new data connectors through a user-confirmed form, inspect table metadata, and run bounded read-only probes when the current workspace data is insufficient.","when_to_use":"The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.","always_on":false,"tools":["list_data","find_data","describe_data","probe_data","list_connectors","describe_connector"],"actions":["propose_data_operation","propose_connection"]},"isInternal":false,"tokens":1240,"sizeBytes":5656},{"name":"tools.json","path":"py-src/data_formulator/analyst/skills/data-loading/tools.json","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/data-loading/tools.json","title":"Data-loading Skill","category":"anthropic-skill","format":"json","content":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]","isInternal":false,"tokens":2024,"sizeBytes":9415},{"name":"SKILL.md","path":"py-src/data_formulator/analyst/skills/report/SKILL.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/report/SKILL.md","title":"report","category":"anthropic-skill","format":"markdown","content":"---\nname: report\ndescription: >-\n Turn an exploration (threads, findings, charts) into a single Markdown\n report — note, blog post, executive summary, KPI dashboard, slide brief, or\n multi-section analytical report, with embedded charts.\nwhen_to_use: >-\n The user asks to write up / summarize / report on what they explored, or\n wants a shareable narrative document built from the charts and findings in\n the data thread. Not for producing a single new chart (use visualize).\nalways_on: false\ntools:\n - inspect_chart\nactions:\n - write_report\n---\n\n# Skill: Report writing\n\nYou are a data journalist / analyst who creates insightful, well-organized\nreports based on data explorations. The output is a single Markdown document\nthat may play many roles — short note, blog post, executive summary, dashboard,\nmulti-section report, FAQ, slide-style brief, etc. Adapt structure and length to\nwhat the user actually asks for; do not force a fixed template.\n\n## Emitting the report (the `write_report` action)\n\nFirst inspect whatever charts and data you need (see below), then write the\nentire report and commit it by **calling the `write_report` tool** — it is the\ncommitting action that ends this turn. Its `report` argument carries the\n**full Markdown** of the finished report:\n\n- `report` — the complete report in Markdown: headings, prose, tables, and\n embedded charts via `![caption](chart://chart_id)`.\n\nProduce any charts the report needs **before** calling `write_report`, and do\nall chart/data inspection first — once you call `write_report`, the report is\ndelivered as-is and the run ends.\n\n## Context available to you\n- **[PRIMARY TABLE(S)]** / **[OTHER AVAILABLE TABLES]**: Lightweight schema of datasets.\n- **[FOCUSED THREAD]** (optional): The exploration thread the user is continuing —\n the ordered steps with the user's questions, the agent's thinking, and the\n findings at each step. This is the spine of the story you are telling.\n- **[OTHER THREADS]** (optional): Brief per-step summaries of other exploration\n threads the user ran. These are additional findings worth weaving in.\n- **[AVAILABLE CHARTS]**: List of charts with their type, encodings, and table references.\n\n## Ground the report in the exploration\nThe thread context is your most important input. The user already did real\nanalysis — your job is to turn that journey into a coherent narrative, not to\nsummarize a single chart. Before writing:\n- Read the FOCUSED THREAD and OTHER THREADS to understand the full set of\n questions asked and findings reached.\n- Plan a report that covers the meaningful findings across the exploration,\n not just the last or most obvious chart.\n\n## Inspecting charts and data\nYou have two inspection tools available the whole time: `inspect_chart` and\n`inspect_source_data`. Use them on your own whenever you need to verify a detail\nbefore writing about it — a chart's exact numbers, its data, or a table's\nschema. `inspect_chart` lets you *read* a chart from its encodings, a data\nsample, and the code that produced it (and points you to the backing table so\nyou can interrogate the full data with `execute_python_script`); a rendered\nimage is included only when one is available. Read the charts behind the key\nfindings you present **before** you compose the report.\n\n## Write the report\nWrite the complete report in Markdown and pass it as the `report` argument of the\n`write_report` tool. Do all your inspecting first, then compose the whole\ndocument and make the one `write_report` call.\n\n### Embedding charts (REQUIRED FORMAT — do not change this)\nTo embed a chart image, use markdown image syntax with a `chart://` URL:\n ![Caption describing the chart](chart://chart_id)\n\nExample: `![Monthly trade balance trend](chart://chart-123)`\n\nThe chart_id must match one from [AVAILABLE CHARTS]. Place each chart embed on\nits own line (it renders as a block). You can embed the same chart at most\nonce. Captions are short — one line describing what the chart shows.\n\n### Tables\nFor data tables, write standard markdown tables directly:\n| date | value |\n| --- | --- |\n| 2020-01 | -43.5 |\n\n### Style & structure — adapt to the user's request\nThe user may ask for any of:\n- a short note or social-style summary (a few sentences, one or two charts),\n- a blog post / narrative report (intro → findings → takeaway),\n- an executive summary (key numbers up top, then context),\n- a KPI dashboard / multi-section overview (headings per topic, multiple charts\n arranged with short commentary between them),\n- a slide-style brief (compact sections with bullet points and embedded charts),\n- a deeper analytical report with sub-sections, methodology notes, and caveats.\n\nPick the structure that fits the request and the available material. Match the\nbreadth of the report to the breadth of the exploration: if the user explored\nseveral questions, the report should reflect that — don't collapse a rich\nexploration into a single-chart blurb unless the user explicitly asked for\nsomething that short. Reasonable defaults if the user is vague:\n- Start with a `# Title` that reflects the topic.\n- Group related findings under `##` (and `###` if useful) headings, typically\n one section per key finding / thread.\n- Around each embedded chart, briefly explain what it shows and the key insight.\n- Use bullets / short paragraphs / tables where they help; don't pad.\n- Close with a brief takeaway or summary section if the report is more than a\n few paragraphs. For very short outputs (notes, single-chart blurbs), a closing\n summary is optional.\n\n### Guardrails\n- Write in Markdown. Keep prose tight; let the data and charts carry the weight.\n- Stay faithful to the data — do not invent numbers, comparisons, or causation\n that the data does not actually support.\n- It is fine to flag uncertainty (\"based on the sample shown…\") when appropriate.\n- Embed every chart you discuss; don't reference a chart in prose without showing it.\n","frontmatter":{"name":"report","description":"Turn an exploration (threads, findings, charts) into a single Markdown report — note, blog post, executive summary, KPI dashboard, slide brief, or multi-section analytical report, with embedded charts.","when_to_use":"The user asks to write up / summarize / report on what they explored, or wants a shareable narrative document built from the charts and findings in the data thread. Not for producing a single new chart (use visualize).","always_on":false,"tools":["inspect_chart"],"actions":["write_report"]},"isInternal":false,"tokens":1345,"sizeBytes":5967},{"name":"tools.json","path":"py-src/data_formulator/analyst/skills/report/tools.json","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/py-src/data_formulator/analyst/skills/report/tools.json","title":"Report Skill","category":"anthropic-skill","format":"json","content":"[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_chart\",\n \"description\": \"Read one or more charts. Returns each chart's encodings, a sample of its data, and the transformation code that created it so you can reason about what it shows (use execute_python_script on the backing table for the full data). A rendered PNG is included only when one is available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"chart_ids\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of chart IDs from [AVAILABLE CHARTS] to inspect.\"\n }\n },\n \"required\": [\"chart_ids\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"write_report\",\n \"description\": \"Deliver a Markdown report and end the run. `report` is the full report text (embed charts with ![caption](chart://chart_id)).\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"report\": {\n \"type\": \"string\",\n \"description\": \"The full Markdown report text. Embed charts with ![caption](chart://chart_id) referencing IDs from [AVAILABLE CHARTS].\"\n }\n },\n \"required\": [\"report\"]\n }\n }\n }\n]\n","isInternal":false,"tokens":303,"sizeBytes":1287},{"name":"README.md","path":"examples/plugins/README.md","rawUrl":"https://raw.githubusercontent.com/microsoft/data-formulator/HEAD/examples/plugins/README.md","title":"plugins Documentation","category":"plugin-manifest","format":"markdown","content":"# Data Formulator — Data Loader Plugins\n\nDrop-in Python files that add new data-source connectors to Data\nFormulator without modifying its source code. If the built-in\nconnectors don't cover your data source (an internal warehouse, a SaaS\nAPI, a niche database), write a small plugin and DF will pick it up on\nthe next restart.\n\nThis folder contains **example plugins**. Treat them as templates: copy\none, rename it, and adapt the body.\n\n---\n\n## Quick start (3 steps)\n\n1. **Find your plugin directory.** It lives under your Data Formulator\n home dir:\n\n ```\n $DATA_FORMULATOR_HOME/plugins/\n ```\n\n If `DATA_FORMULATOR_HOME` is not set, DF defaults to\n `~/.data_formulator/`, so the plugin dir is `~/.data_formulator/plugins/`.\n\n Power users can point somewhere else with `DF_PLUGIN_DIR`\n (highest precedence) — useful for sharing one plugin folder across\n multiple DF installs.\n\n2. **Copy an example into it.** For instance:\n\n ```bash\n mkdir -p \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins\"\n cp examples/plugins/sqlite_data_loader.py \\\n \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins/\"\n ```\n\n3. **Restart Data Formulator.** The new connector appears in the UI\n automatically. No registry edits, no rebuilds.\n\nTo verify it loaded, check the startup log for a line like:\n\n```\nINFO ... Plugin loader 'sqlite' registered from sqlite_data_loader.py\nINFO ... Plugin scan complete: 1 registered, 0 failed (dir=..., reason=WORKSPACE_BACKEND=local)\n```\n\n### Plugin directory resolution order\n\n| Precedence | Source | Default |\n| ---------- | ----------------------------------------- | ----------------------------- |\n| 1 | `DF_PLUGIN_DIR` env var (explicit override) | — |\n| 2 | `$DATA_FORMULATOR_HOME/plugins` | — |\n| 3 | Fallback | `~/.data_formulator/plugins/` |\n\n---\n\n## File-name contract\n\n| Filename | Registry key |\n| --------------------------------- | -------------- |\n| `sqlite_data_loader.py` | `sqlite` |\n| `acme_warehouse_data_loader.py` | `acme_warehouse` |\n| `notion_data_loader.py` | `notion` |\n\nRules:\n\n* The filename **must** end in `_data_loader.py`.\n* The prefix becomes the registry key — keep it lowercase, no spaces.\n* If the key matches a built-in (e.g. `mysql_data_loader.py`), the\n plugin **overrides** the built-in. Useful for hot-patching.\n\n---\n\n## What goes inside the file\n\nEach plugin defines exactly one class that subclasses\n[`ExternalDataLoader`](../../py-src/data_formulator/data_loader/external_data_loader.py).\nThe minimum surface area:\n\n```python\nfrom data_formulator.data_loader.external_data_loader import (\n ExternalDataLoader, MAX_IMPORT_ROWS,\n)\nimport pyarrow as pa\n\nclass MyLoader(ExternalDataLoader):\n\n # Optional: human-friendly UI label. Without this, the registry key\n # is title-cased (``\"my_warehouse\"`` → ``\"My Warehouse\"``). Override\n # to fix awkward casing (``\"SQLite\"``, ``\"BigQuery\"``).\n DISPLAY_NAME = \"My Warehouse\"\n\n @staticmethod\n def list_params() -> list[dict]:\n \"\"\"Declare connection-form fields. The UI auto-renders this.\"\"\"\n return [\n {\"name\": \"endpoint\", \"type\": \"string\", \"required\": True,\n \"tier\": \"connection\", \"description\": \"Server URL\"},\n {\"name\": \"token\", \"type\": \"string\", \"required\": True,\n \"tier\": \"auth\", \"sensitive\": True, \"description\": \"API token\"},\n ]\n\n @staticmethod\n def auth_instructions() -> str:\n \"\"\"Markdown help text shown next to the form.\"\"\"\n return \"Get your API token from https://example.com/settings/tokens\"\n\n def __init__(self, params: dict):\n self.params = params\n # validate + open connection here\n\n def list_tables(self, table_filter: str | None = None) -> list[dict]:\n \"\"\"Return catalog: [{name, metadata: {columns, row_count}}, ...]\"\"\"\n ...\n\n def fetch_data_as_arrow(self, source_table: str,\n import_options: dict | None = None) -> pa.Table:\n \"\"\"Read rows. Honour import_options['size'] up to MAX_IMPORT_ROWS.\"\"\"\n ...\n```\n\nLook at [`sqlite_data_loader.py`](sqlite_data_loader.py) for a runnable\nimplementation (~170 lines, stdlib only).\n\n### `list_params()` field reference\n\n| Key | Meaning |\n| ------------- | ---------------------------------------------------------------- |\n| `name` | Parameter key passed into `__init__(params)` |\n| `type` | `\"string\"`, `\"int\"`, `\"bool\"`, `\"password\"` |\n| `required` | If `True`, DF rejects connections that omit it |\n| `default` | Pre-filled value in the form |\n| `sensitive` | If `True`, value is redacted from logs / stored metadata |\n| `tier` | `\"auth\"`, `\"connection\"`, or `\"filter\"` — groups fields visually |\n| `description` | Help text shown under the field |\n\n---\n\n## Security gate — please read\n\nPlugins execute arbitrary Python in the server process. To prevent\naccidental code execution in shared deployments, the plugin scanner is\n**enabled only in single-user local mode**:\n\n* `WORKSPACE_BACKEND` unset or `local` → scanner runs.\n* `WORKSPACE_BACKEND` is anything else → scanner is skipped silently.\n* To opt in for a hosted deployment, set `DF_ALLOW_PLUGINS=1`\n **and** make sure the plugin directory is writable only by trusted\n administrators.\n\n---\n\n## Debugging a plugin that won't load\n\nThe scanner reports every failure two ways:\n\n1. **In the log**, with a full traceback:\n\n ```\n WARNING Failed to load plugin foo_data_loader.py: ... ModuleNotFoundError ...\n ```\n\n2. **In `DISABLED_LOADERS`**, which the frontend surfaces as a greyed-out\n connector with the reason. Common reasons:\n\n | Message | Fix |\n | ------------------------------------------------------ | --- |\n | `missing dependency 'X' (pip install X)` | `uv pip install X` in DF's venv |\n | `no ExternalDataLoader subclass found` | Add a `class Foo(ExternalDataLoader)` to the file |\n | `could not create import spec` | File isn't valid Python or has wrong extension |\n | `: ` | Anything else raised during `import` — check the traceback |\n\nThe plugin scanner cleans `sys.modules` on failure, so simply restarting\nDF after editing the file picks up the fix — no manual cleanup needed.\n\n---\n\n## Adding third-party dependencies\n\nIf your plugin needs a package that isn't already in DF's environment:\n\n```bash\n# from the data-formulator repo root\nuv pip install \n```\n\nThe plugin file is loaded into the same Python process as DF, so\nanything importable from DF's venv is importable from your plugin.\n\n---\n\n## For AI coding agents\n\nWhen asked to **add a new data source** to a user's Data Formulator\ninstall, prefer writing a plugin file over modifying DF's source tree:\n\n1. Read [`sqlite_data_loader.py`](sqlite_data_loader.py) and the\n `ExternalDataLoader` ABC in\n `py-src/data_formulator/data_loader/external_data_loader.py` to\n understand the required methods.\n2. Create a new file `_data_loader.py` in this folder (or\n directly in `~/.data_formulator/plugins/`).\n3. Subclass `ExternalDataLoader`. Implement at minimum:\n `list_params`, `auth_instructions`, `__init__`, `list_tables`,\n `fetch_data_as_arrow`.\n4. Return data as a `pyarrow.Table` from `fetch_data_as_arrow` — do\n **not** convert to pandas in the hot path.\n5. Quote identifiers when building SQL (see `_quote_ident` in the\n SQLite example) to avoid injection vulnerabilities.\n6. Respect `import_options['size']` and cap at `MAX_IMPORT_ROWS`.\n7. If the source has credentials, mark those params `sensitive: True`\n and `tier: \"auth\"` so DF redacts them from stored metadata.\n8. Do not modify the user's data — open read-only connections where\n the source supports it.\n\nAfter writing the file, verify it loads with:\n\n```bash\nDF_PLUGIN_DIR= uv run python -c \\\n \"from data_formulator import data_loader as dl; \\\n print(dl.PLUGIN_LOADERS, dl.DISABLED_LOADERS)\"\n```\n","isInternal":false,"tokens":1910,"sizeBytes":8496}],"systemPromptSnippet":"\n\n# Project Instructions\n\n## Python Environment\n\n- Always use `uv` instead of `pip` for installing packages (e.g. `uv pip install`, `uv pip install -e .`).\n- Use `uv run` to execute Python scripts and modules (e.g. `uv run python script.py`, `uv run pytest`).\n- The virtual environment is at `.venv/`. Activate it with `source .venv/bin/activate` if needed.\n- The Python source is under `py-src/`. The project is installed in editable mode via `uv pip install -e .`.\n\n\n---\r\ndescription: Backend Python test conventions (pytest)\r\nglobs: tests/backend/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# Backend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/backend/unit/`, `tests/backend/integration/`, `tests/backend/contract/`, or by domain: `tests/backend/routes/`, `tests/backend/errors/`, `tests/backend/agents/`, `tests/backend/data/`, `tests/backend/security/`, `tests/backend/auth/`, `tests/backend/knowledge/`.\r\n- Name files `test_.py` with descriptive snake_case names.\r\n- Shared test data goes in `tests/backend/fixtures/`.\r\n\r\n## File Structure\r\n\r\n```python\r\n\"\"\"One-paragraph summary of what is being tested and why.\r\n\r\nBackground\r\n----------\r\nBrief context about the feature or bug fix these tests validate.\r\n\"\"\"\r\nfrom __future__ import annotations\r\n\r\nimport pytest\r\n\r\npytestmark = [pytest.mark.backend]\r\n```\r\n\r\n## Conventions\r\n\r\n- Always add `pytestmark = [pytest.mark.backend]` at module level.\r\n- Group related tests in classes prefixed with `Test` (e.g. `TestStripImageBlocks`).\r\n- Use `pytest.mark.parametrize` for data-driven tests instead of writing repetitive cases.\r\n- Use `pytest.fixture()` for shared setup; keep fixtures close to where they are used.\r\n- Unit tests must **not** depend on Flask, network, or external services.\r\n- Mock external calls with `unittest.mock.patch` / `MagicMock`; never make real API calls.\r\n- Each test function should verify **one** behavior and have a clear name: `test__`.\r\n\r\n## Running Tests\r\n\r\n- When running pytest, use `-q` (quiet mode) instead of `-v` (verbose mode) for cleaner output.\r\n- **Preferred:** `python -m pytest tests/backend/ -q`\r\n- **Avoid:** `python -m pytest tests/backend/ -v 2>&1`\r\n\r\n## Error Handling Test Requirements\r\n\r\nApplication errors return **HTTP 200** with `status: \"error\"` in the body.\r\nOnly auth errors (`AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403) and\r\nuncontrolled transport failures (404/413/500) use non-200.\r\n\r\nWhen adding a new route or modifying error handling, include tests for:\r\n\r\n1. **AppError scenarios** — verify HTTP 200 + correct `ErrorCode` in body:\r\n ```python\r\n def test_missing_table_returns_error():\r\n with app.test_client() as client:\r\n resp = client.get('/api/tables/nonexistent')\r\n assert resp.status_code == 200 # application error → always 200\r\n data = resp.get_json()\r\n assert data[\"status\"] == \"error\"\r\n assert data[\"error\"][\"code\"] == \"TABLE_NOT_FOUND\"\r\n ```\r\n\r\n2. **Auth errors** — verify non-200 for authentication/authorization failures:\r\n ```python\r\n def test_access_denied_returns_403():\r\n resp = client.get('/api/protected-resource')\r\n assert resp.status_code == 403\r\n assert resp.get_json()[\"error\"][\"code\"] == \"ACCESS_DENIED\"\r\n ```\r\n\r\n3. **Unexpected exceptions** — verify the global handler catches them (unhandled = 500):\r\n ```python\r\n def test_unexpected_error_returns_500():\r\n with patch('module.function', side_effect=RuntimeError(\"boom\")):\r\n resp = client.get('/api/endpoint')\r\n assert resp.status_code == 500\r\n assert resp.get_json()[\"error\"][\"code\"] == \"INTERNAL_ERROR\"\r\n ```\r\n\r\n4. **Streaming endpoints** — verify error events in the NDJSON stream:\r\n ```python\r\n def test_stream_error_yields_ndjson_event():\r\n resp = client.post('/api/my-stream', ...)\r\n lines = [json.loads(l) for l in resp.data.decode().strip().split('\\n') if l]\r\n error_lines = [l for l in lines if l.get('type') == 'error']\r\n assert len(error_lines) >= 1\r\n assert error_lines[0]['error']['code'] == 'LLM_TIMEOUT'\r\n ```\r\n\r\n5. **Test fixture** — ensure `register_error_handlers(app)` is called in the test app fixture:\r\n ```python\r\n @pytest.fixture\r\n def app():\r\n from data_formulator.error_handler import register_error_handlers\r\n _app = flask.Flask(__name__)\r\n _app.config[\"TESTING\"] = True\r\n _app.register_blueprint(my_bp)\r\n register_error_handlers(_app)\r\n return _app\r\n ```\r\n\r\nSee `tests/backend/errors/test_errors.py` and `test_error_handler.py` for reference implementations.\r\n\r\n## Example\r\n\r\n```python\r\n# ❌ BAD – vague name, no parametrize, no marker\r\ndef test_it_works():\r\n assert sanitize(\"hello world\") == \"hello_world\"\r\n\r\n# ✅ GOOD\r\npytestmark = [pytest.mark.backend]\r\n\r\n@pytest.mark.parametrize(\"raw,expected\", [\r\n (\"hello world\", \"hello_world\"),\r\n (\"订单明细\", \"订单明细\"),\r\n])\r\ndef test_sanitize_preserves_unicode(raw: str, expected: str) -> None:\r\n assert sanitize(raw) == expected\r\n```\n\n\n# DataFrame Serialization\n\nAll DataFrame-to-records conversion for API responses, streaming events, or\nfrontend-visible data MUST use the centralized helpers in\n`data_formulator.datalake.parquet_utils`:\n\n| Source type | Helper |\n|---|---|\n| `pd.DataFrame` | `df_to_safe_records(df)` |\n| `pa.Table` (Arrow) | `get_sample_rows_from_arrow(table)` |\n\n## Why\n\n`pandas.DataFrame.to_json(orient='records')` defaults to `date_format='epoch'`,\nwhich serializes datetime columns as **epoch milliseconds** (e.g. `1773532800000`).\nThe frontend interprets these as plain numbers and renders them with commas\n(`1,773,532,800,000`) instead of formatted dates.\n\n`df_to_safe_records` enforces `date_format='iso'` and `default_handler=str`,\nensuring datetimes become ISO-8601 strings and exotic types degrade gracefully.\n\n## Banned Patterns\n\n```python\n# BAD — missing date_format, datetimes become epoch numbers\njson.loads(df.to_json(orient='records'))\n\n# BAD — to_dict returns Timestamp objects, not JSON-safe values\ndf.to_dict(orient='records')\n\n# ACCEPTABLE but should be unified for consistency\njson.loads(df.to_json(orient='records', date_format='iso'))\n```\n\n## Correct Pattern\n\n```python\nfrom data_formulator.datalake.parquet_utils import df_to_safe_records\n\nrows = df_to_safe_records(df)\npreview = df_to_safe_records(df.head(5))\n```\n\n## Exceptions\n\nInternal data processing that never reaches the frontend or JSON serialization\n(e.g. Kusto SDK metadata parsing, Vega-Lite spec construction) may use\n`to_dict(orient='records')` directly.\n\n\n---\ndescription: Require reading dev-guides before development and keeping docs in sync\nglobs:\nalwaysApply: true\n---\n\n# Dev Guides First\n\n## Before Starting Any Development\n\nBefore implementing or designing a new feature, module, or significant change, **always\nread the relevant documents in `docs/dev-guides/`** to understand existing conventions:\n\n| Guide | When to Read |\n|-------|-------------|\n| `docs/dev-guides/1-streaming-protocol.md` | Any work on streaming endpoints or NDJSON protocol |\n| `docs/dev-guides/2-log-sanitization.md` | Any work involving logging, credentials, external services, or DataLoaders |\n| `docs/dev-guides/3-data-loader-development.md` | Any work on ExternalDataLoader, DataConnector, or connector routes |\n| `docs/dev-guides/4-authentication-oidc-tokenstore.md` | Any work on OIDC, TokenStore, AUTH_MODE, or SSO flows |\n| `docs/dev-guides/6-i18n-language-injection.md` | Any work on Agent prompts, Agent routes, backend user-visible messages, or frontend i18n |\n| `docs/dev-guides/7-unified-error-handling.md` | Any work on API errors, frontend API calls, streaming error events, or error tests |\n| `docs/dev-guides/8-path-safety.md` | Any work on backend file access, downloads, Workspace paths, Agent tools, DataLoaders, or sandbox config |\n| `docs/dev-guides/10-agent-knowledge-reasoning-log.md` | Any work on Agent knowledge injection, KnowledgeStore, reasoning logs, or experience distillation |\n| `docs/dev-guides/11-catalog-metadata-sync.md` | Any work on catalog sync, catalog_cache, catalog_annotations, metadata merge, Agent catalog tools, or frontend catalog browsing |\n| `docs/dev-guides/12-sandbox-session.md` | Any work on sandbox execution, Agent tool-calling loops, explore/execute_python code execution, or namespace management |\n| `docs/dev-guides/13-unified-row-limits.md` | Any work on row limits, data loading size caps, frontendRowLimit, MAX_IMPORT_ROWS, or DataLoader size parameter |\n| `docs/dev-guides/14-model-capability-runtime-degradation.md` | Any work on LLM Client calls, Agent LLM invocations, model capability checks, reasoning_effort, or image/vision degradation |\n| `docs/dev-guides/15-dataframe-serialization.md` | Any work on DataFrame→JSON serialization, Agent result rows, DataLoader sample_rows, or table API responses |\n| `.cursor/rules/i18n-no-hardcoded-strings.mdc` | Any work adding or changing user-visible strings in `src/` |\n\nAlso check `.cursor/rules/` and `.cursor/skills/` for related coding conventions.\n\n## After Introducing New Conventions\n\nWhen your work establishes a new development pattern, convention, or architectural\ndecision, you MUST update documentation before considering the task complete:\n\n1. **Update existing dev-guide** — if your change extends an existing convention\n (e.g. new sensitive key type → update `2-log-sanitization.md`)\n\n2. **Create new dev-guide** — if your change introduces a new cross-cutting convention\n that future developers must follow. Place in `docs/dev-guides/` with the next number prefix.\n\n3. **Update existing SKILL** — if your change affects how an existing skill works\n (e.g. new error handling pattern → update `error-handling/SKILL.md`)\n\n4. **Create new SKILL** — if your change introduces a reusable development workflow\n that AI agents should follow. Place in `.cursor/skills//SKILL.md`.\n\n5. **Update existing Rule** — if your change adds constraints to an existing rule\n (e.g. new banned pattern → update the relevant `.mdc` file)\n\n6. **Create new Rule** — if your change introduces file-scoped coding conventions\n that should be enforced automatically. Place in `.cursor/rules/.mdc`.\n\n## Documentation Sync Checklist\n\nFor every PR that introduces new patterns:\n\n- [ ] Searched `docs/dev-guides/` for related existing docs\n- [ ] Updated or created dev-guide if introducing cross-cutting conventions\n- [ ] Updated or created `.cursor/skills/` if introducing reusable workflows\n- [ ] Updated or created `.cursor/rules/` if introducing file-scoped constraints\n- [ ] New dev-guide includes a \"New Module Checklist\" section (if applicable)\n\n## File Naming Conventions\n\n- **dev-guides**: `-.md` (e.g. `3-data-loader-development.md`)\n- **skills**: `.cursor/skills//SKILL.md`\n- **rules**: `.cursor/rules/.mdc`\n\n\n---\ndescription: Prevent information exposure through exception messages in HTTP responses\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Error Response Safety\n\nNever return raw exception text (`str(e)`, `f\"...{e}\"`) directly in HTTP responses.\nPython exceptions may contain stack traces, file paths, database connection strings,\nAPI keys, or internal IP addresses — all of which are security risks (CWE-209).\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full error handling contract.\n\n## Unified Error System\n\nNew error paths should use the `AppError` / `error_handler` infrastructure from\n`errors.py` and `error_handler.py`. Legacy `message` / `error_message` bodies are\nhistorical formats; do not add compatibility branches for them in new code.\nAll application-controlled errors return **HTTP 200** with `status: \"error\"` in the body.\nOnly auth errors (401/403) and uncontrolled transport errors (404/413/500) use non-200.\n\n### Non-streaming routes\n\nRaise `AppError` for new code or when structured error codes are useful. The\nglobal error handler returns HTTP 200 + JSON error envelope:\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\n\n# Business error — raise with appropriate code and safe message\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# LLM / external API error — use classify_and_wrap_llm_error\nfrom data_formulator.error_handler import classify_and_wrap_llm_error\n\nexcept Exception as e:\n raise classify_and_wrap_llm_error(e) from e\n```\n\n### Streaming routes\n\nUse `stream_error_event()` to yield a unified NDJSON error line.\nValidation before the stream starts returns `200 application/json` (not NDJSON):\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\nexcept AppError as e:\n yield stream_error_event(e)\nexcept Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n### Real examples from migrated endpoints\n\n```python\n# /data-agent-streaming — standard NDJSON error event\nexcept Exception as e:\n logger.error(\"Error in data-agent-streaming\", exc_info=e)\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# /generate-report-chat — same pattern\nexcept Exception as e:\n logger.exception(\"generate-report-chat failed\")\n from data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n# Validation failure before stream starts — use 200 JSON, not NDJSON\nif not request.is_json:\n return stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\"))\n```\n\n### Database / workspace errors (tables.py)\n\nUse `classify_and_raise_db_error()` instead of the legacy `sanitize_db_error_message()`:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\nexcept Exception as e:\n classify_and_raise_db_error(e) # raises AppError with safe message\n```\n\n### Connector errors (data_connector.py)\n\nUse `classify_and_raise_connector_error()` instead of the legacy `_sanitize_error()`:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e) # raises AppError with safe message\n```\n\n### Legacy functions (deprecated)\n\nThese functions still exist as wrappers but should NOT be used in new code:\n- `sanitize_db_error_message()` → use `classify_and_raise_db_error()`\n- `_sanitize_error()` → use `classify_and_raise_connector_error()`\n- `safe_error_response()` / `sanitize_error_message()` → use `raise AppError(...)`\n- `classify_llm_error()` is used internally by `classify_and_wrap_llm_error()`\n\n## Rules\n\n1. **Application errors** — HTTP 200 + `status: \"error\"`; prefer `error: {...}` for new code and never expose exception details.\n2. **Auth errors** — `AUTH_REQUIRED`/`AUTH_EXPIRED` → 401, `ACCESS_DENIED` → 403 (transport layer interception).\n3. **Transport errors** (uncontrolled) — only `404` (no route), `413` (body limit), `500` (unhandled crash) may be non-200.\n4. **Streaming errors** — always use `stream_error_event()` for NDJSON error lines.\n5. **Logging** — always log the full exception server-side (`logger.warning` / `logger.error`\n with `exc_info=True` when needed). The client never needs the stack trace.\n6. **Debug detail** — `AppError.detail` is only included in responses when `app.debug is True`.\n\n## Bare except Policy\n\n- **Never** use naked `except:` — always specify at least `except Exception:`\n- Use specific exception types when possible (`except (ValueError, TypeError):`)\n- Always log caught exceptions: `logger.warning(...)` for degraded operations, `logger.debug(...)` for expected failures\n- Process cleanup code (`terminate`, `join`, `close`) is exempt from logging requirements\n\n## Common Mistakes\n\n```python\n# ❌ BAD — raw exception leaks internal details\nreturn jsonify({\"message\": str(e)}), 500\nreturn jsonify({\"error\": f\"Failed: {e}\"}), 502\nyield json.dumps({\"type\": \"error\", \"error\": str(e)}) + \"\\n\"\n\n# ❌ BAD — non-200 HTTP status code for application errors\nreturn jsonify({\"status\": \"error\", \"error_message\": \"...\"}), 400\n\n# ❌ BAD — naked except catches SystemExit/KeyboardInterrupt\nexcept:\n pass\n\n# ❌ BAD — silent swallow without logging\nexcept Exception:\n pass\n\n# ✅ GOOD — non-streaming (global handler returns HTTP 200)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# ✅ GOOD — streaming\nyield stream_error_event(AppError(ErrorCode.LLM_TIMEOUT, \"Timeout\", retry=True))\n\n# ✅ GOOD — graceful degradation with logging\nexcept Exception as e:\n logger.warning(\"Could not read schema for %s\", table_name, exc_info=e)\n\n# ✅ GOOD — specific exception types\nexcept (ValueError, TypeError):\n pass # safe fallback\n```\n\n\n---\r\ndescription: Frontend TypeScript test conventions (Vitest)\r\nglobs: tests/frontend/**/*.test.{ts,tsx}\r\nalwaysApply: false\r\n---\r\n\r\n# Frontend Test Conventions\r\n\r\n## File Location & Naming\r\n\r\n- Place tests under `tests/frontend/unit/` mirroring the `src/` structure:\r\n - `tests/frontend/unit/data/` → tests for `src/data/`\r\n - `tests/frontend/unit/app/` → tests for `src/app/`\r\n - `tests/frontend/unit/views/` → tests for `src/views/`\r\n- Name files `.test.ts` (or `.test.tsx` for React rendering tests).\r\n\r\n## File Structure\r\n\r\n```typescript\r\nimport { describe, it, expect } from 'vitest';\r\n// For React rendering tests:\r\n// import { render } from '@testing-library/react';\r\n\r\nimport { myFunction } from '../../../../src/';\r\n\r\ndescribe('myFunction', () => {\r\n it('should handle ', () => {\r\n expect(myFunction(input)).toBe(expected);\r\n });\r\n});\r\n```\r\n\r\n## Conventions\r\n\r\n- Import `describe`, `it`, `expect` explicitly from `vitest` (globals are enabled but explicit imports improve readability).\r\n- Use `@testing-library/react` and `@testing-library/jest-dom` for component rendering tests.\r\n- Prefer testing **exported pure functions** over testing internal component state.\r\n- When component logic is complex, extract it into an exported helper and test that directly.\r\n- Group tests with `describe` blocks; use section comments (`// --- Null cases ---`) for clarity.\r\n- One assertion per `it` block when possible; name tests as `should `.\r\n- Do **not** import from `node_modules` internals; only use public API.\r\n- Keep tests independent — no shared mutable state between `it` blocks.\r\n\r\n## Example\r\n\r\n```typescript\r\n// ❌ BAD – no describe, vague test name\r\nimport { expect, test } from 'vitest';\r\ntest('works', () => { expect(fn(1)).toBe(2); });\r\n\r\n// ✅ GOOD\r\nimport { describe, it, expect } from 'vitest';\r\nimport { checkIsLikelyTextOnlyModel } from '../../../../src/views/DataLoadingThread';\r\n\r\ndescribe('checkIsLikelyTextOnlyModel', () => {\r\n it('returns true for deepseek-chat', () => {\r\n expect(checkIsLikelyTextOnlyModel('deepseek-chat')).toBe(true);\r\n });\r\n\r\n it('returns false for undefined', () => {\r\n expect(checkIsLikelyTextOnlyModel(undefined)).toBe(false);\r\n });\r\n});\r\n```\n\n\n---\r\ndescription: No hardcoded UI strings — use i18n translation keys\r\nglobs: src/**/*.{ts,tsx}\r\nalwaysApply: true\r\n---\r\n\r\n# i18n: No Hardcoded UI Strings\r\n\r\nAll user-visible text in the frontend MUST go through the i18n system. Never hardcode Chinese, English, or any other language string directly in components.\r\n\r\n## How to Use\r\n\r\n```tsx\r\nimport { useTranslation } from 'react-i18next';\r\n\r\nconst { t } = useTranslation();\r\n\r\n// ✅ GOOD\r\n\r\n\r\n\r\n// ❌ BAD\r\n\r\n\r\n\r\n```\r\n\r\n## Non-Component Contexts (thunks, utilities, helpers)\r\n\r\nWhen you cannot use hooks (Redux thunks, plain `.ts` utilities, store helpers), import the i18n instance directly:\r\n\r\n```ts\r\nimport i18n from '../i18n';\r\n\r\n// ✅ GOOD — works outside React components\r\ni18n.t('messages.rowLimitReached', { count: 20000 })\r\n\r\n// ❌ BAD — hooks only work inside React function components\r\nconst { t } = useTranslation(); // will crash in a thunk\r\n```\r\n\r\nThe same rule applies: every user-visible string must use `i18n.t()` with a translation key, never a hardcoded literal.\r\n\r\n## Translation Files\r\n\r\n- English: `src/i18n/locales/en/.json`\r\n- Chinese: `src/i18n/locales/zh/.json`\r\n- Namespaces: `common`, `upload`, `chart`, `model`, `encoding`, `messages`, `navigation`, `dataLoading`, `errors`\r\n\r\nWhen adding a new key, add it to **both** `en` and `zh` locale files. Pick the namespace that fits; create a new namespace only if none applies.\r\n\r\n## What Counts as User-Visible\r\n\r\nMust use `t()`: button labels, tooltips, placeholders, error messages, dialog titles, tab names, toast notifications, table headers, empty-state text.\r\n\r\nMay stay hardcoded: log messages (`console.log`), error messages thrown but never displayed, internal constants, CSS class names, test IDs.\r\n\r\n**Cross-stack sentinel values** — strings shared between frontend and backend as identity markers (e.g. `UNTITLED_SESSION` in `dfSlice.ts` used by both Redux state and backend workspace API) must NOT be translated. They are internal constants, not display text. Translate them only at the UI rendering layer by mapping the sentinel to `t('...')`.\r\n\r\n## Backend Messages with message_code\r\n\r\nWhen the backend returns a user-visible message with a `message_code` (or `content_code`, `error_code`), use `translateBackend()` from `src/app/utils.tsx` to translate it:\r\n\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\n\r\n// Translate a backend error message\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n\r\n// Translate options with parallel code arrays\r\nconst options = translateBackendOptions(rawOptions, event.option_codes);\r\n```\r\n\r\nThe translation keys live in `messages.json` under the `agent` section. Always add **both** `en` and `zh` entries.\n\n\n---\ndescription: Require structural code review after implementation before final response\nglobs:\nalwaysApply: true\n---\n\n# Implementation Review Checklist\n\nAfter any substantive implementation, before the final response, review the\nuncommitted diff from a maintainer/code-review perspective. Do not only verify\nthat the feature works.\n\n## Required Checks\n\n- Inspect `git diff --stat` and relevant diffs to understand the actual blast radius.\n- Check responsibility boundaries: frontend UI, API route, service/helper, storage/session, and tests should each own the right part of the behavior.\n- Check API semantics: endpoint names, HTTP methods, response shapes, and side effects must match the product action. Similar UI actions should call the same backend API.\n- Check frontend/backend state consistency: after mutations, make sure Redux/local state, backend session/token/vault state, and subsequent list/status APIs agree.\n- Check duplication: if two paths clear the same state or perform the same API flow, extract a small helper or route instead of copying logic.\n- Check regression surfaces: refresh/reload/list/status paths must respect new flags or state, not just the primary click path.\n- Check tests: add or update focused tests for the changed contract; include at least one test for the bug/regression path when practical.\n- Check docs/rules/dev-guides when the change introduces or changes a cross-cutting convention.\n\n## Reporting\n\nIf issues are found during this review, fix clear issues before finalizing. If a\ntrade-off remains, call it out explicitly in the final response with the risk and\nwhy it was left as-is.\n\n\n---\ndescription: Enforce step-by-step implementation with design-doc checks, tests, and i18n after each step\nglobs:\nalwaysApply: true\n---\n\n# Incremental Development Cadence\n\nMulti-step tasks MUST be implemented **one step at a time**. After each logical unit (function, endpoint, component), complete all cross-cutting concerns before moving on.\n\n## After Each Step\n\n1. **Verify against design doc** — compare with `design-docs/` or issue spec; raise deviations before continuing\n2. **Add tests** — write/update tests for the just-implemented code, run and confirm green\n3. **Add i18n keys** — if user-visible text was introduced, add `en` + `zh` keys immediately\n4. **Update docs** — if a new cross-cutting convention was introduced, update dev-guide/Rule immediately\n\n## Banned Patterns\n\n- ❌ Implementing 3+ functions/endpoints before writing any tests\n- ❌ Sweeping all i18n keys in a final pass after all code is done\n- ❌ Checking design docs only after the entire implementation is complete\n- ❌ Mentioning \"tests and i18n still needed\" only in the final response\n\n\n---\r\ndescription: Language injection conventions for LLM Agent prompts\r\nglobs: py-src/data_formulator/agents/**/*.py,py-src/data_formulator/routes/agents.py\r\nalwaysApply: false\r\n---\r\n\r\n# Language Injection Conventions\r\n\r\nLanguage flows per-request: `Frontend i18n → Accept-Language header → get_language_instruction() → system prompt`.\r\n\r\n## LLM Prompt Language Injection\r\n\r\n1. **User-facing LLM output** MUST inject language via `get_language_instruction(mode=...)` in the route handler.\r\n2. **Mode selection:** `\"full\"` for text-heavy agents, `\"compact\"` for code-generation agents and short-text endpoints.\r\n3. **Inject into system prompt only** — use `inject_language_instruction()` from `agent_language.py`. Never inject into user messages.\r\n4. **Do NOT inject** for non-user-facing calls (health checks, internal tool calls).\r\n5. **Do NOT duplicate** — if upstream messages already contain language instruction, skip.\r\n6. **Do NOT** use env vars, global interceptors, or hardcoded language strings (e.g. `\"回答请使用中文\"`) — route handlers should use `get_language_instruction(mode=...)`.\r\n7. **New language?** Add to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py` and add locale files in `src/i18n/locales//`.\r\n\r\n## Python-Side User-Visible Messages (message_code pattern)\r\n\r\nFor **fixed strings in Python** that are shown in the UI (error messages, clarify options, completion summaries), do NOT translate in Python. Instead:\r\n\r\n1. Keep the English string as the default value.\r\n2. Add a `message_code` (or `content_code`, `error_code`, `summary_code`) field with a key like `\"agent.someKey\"`.\r\n3. Optionally add `message_params` for interpolation.\r\n4. The frontend translates using `translateBackend(fallback, code, params)` from `src/app/utils.tsx`.\r\n5. Add the translation key to `src/i18n/locales/{en,zh}/messages.json` under the `agent` section.\r\n\r\n```python\r\n# ✅ GOOD — backend returns code, frontend translates\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\",\r\n \"message_code\": \"agent.emptyDataframe\",\r\n}\r\n\r\n# ❌ BAD — backend-side translation dict\r\nyield {\"message\": translate_in_python(\"empty_df\", lang)}\r\n```\r\n\r\nFor detailed architecture and anti-pattern explanations, see `docs/dev-guides/6-i18n-language-injection.md`.\n\n\n---\ndescription: Prevent sensitive data leakage in server-side log messages\nglobs: py-src/**/*.py\nalwaysApply: false\n---\n\n# Log Sanitization\n\nServer-side logs must never contain passwords, tokens, API keys, or other credentials\nin plain text. This rule complements `error-response-safety.mdc` (which covers client\nresponses) by addressing the **logging** side.\n\n## Architecture: Defense in Depth\n\nTwo layers protect against sensitive data in logs:\n\n1. **Layer 1 — Explicit utilities** (call-site): developers use `sanitize_url()`,\n `sanitize_params()`, `redact_token()` from `security/log_sanitizer.py`.\n2. **Layer 2 — SensitiveDataFilter** (global): a `logging.Filter` registered on all\n handlers in `configure_logging()` auto-redacts patterns that slip through.\n\nLayer 1 is the primary defense; Layer 2 is a safety net. Both are required.\n\n## When to Use Each Utility\n\n| Data Type | Utility | Example |\n|-----------|---------|---------|\n| `dict` with password/secret/token keys | `sanitize_params(params)` | `log.info(\"Init: %s\", sanitize_params(params))` |\n| URL (may have `user:pass@host` or sensitive query params) | `sanitize_url(url)` | `logger.info(\"Connecting to %s\", sanitize_url(url))` |\n| Bearer / access token | `redact_token(token)` | `logger.debug(\"Token: %s\", redact_token(token))` |\n| Normal string (no secrets) | Nothing needed | `logger.info(\"Processing table %s\", name)` |\n\n## Rules\n\n1. **Never log raw `params` / `config` dicts** that may contain `password`, `secret`,\n `api_key`, `token`, or `connection_string` keys. Always use `sanitize_params()`.\n\n2. **Never log full tokens or API keys.** Use `redact_token()` if you need to identify\n a token in logs.\n\n3. **Prefer `type(exc).__name__`** over `str(exc)` in warning/info logs when the\n exception may contain connection strings or upstream response bodies. Use\n `exc_info=True` for full tracebacks at ERROR level (the filter sanitizes `exc_text`).\n\n4. **URL logging** — use `sanitize_url()` for any URL from configuration or\n environment variables (issuer URLs, JWKS URLs, database URLs, API base URLs).\n It masks embedded credentials and sensitive query parameters such as\n `password`, `client_secret`, `token`, and `api_key`.\n\n5. **Use `%s`-style formatting** instead of f-strings for log messages. This allows\n the filter to intercept arguments before formatting and provides better\n compatibility with log aggregation tools.\n\n## Common Mistakes\n\n```python\n# BAD — dict may contain password\nlog.info(f\"Connecting with {params}\")\nlog.info(\"Connecting with %s\", params)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_params\nlog.info(\"Connecting with %s\", sanitize_params(params))\n\n# BAD — URL may embed credentials\nlogger.warning(\"Failed for %s\", discovery_url)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import sanitize_url\nlogger.warning(\"Failed for %s\", sanitize_url(discovery_url))\n\n# BAD — exception may leak connection string or upstream body\nlogger.warning(\"Error: %s\", exc)\n\n# GOOD — log type name at warning level, full traceback at error level\nlogger.warning(\"Error: %s\", type(exc).__name__)\nlogger.error(\"Unexpected failure\", exc_info=True)\n\n# BAD — full token in logs\nlogger.debug(\"Using token %s\", token)\n\n# GOOD\nfrom data_formulator.security.log_sanitizer import redact_token\nlogger.debug(\"Using token %s\", redact_token(token))\n```\n\n## Disabling for Local Debug\n\nSet `LOG_SANITIZE=false` to disable the global filter during local debugging.\nThis should **never** be used in production or CI.\n\n## Adding New Sensitive Keys\n\nIf your module introduces a new credential key name (e.g. `custom_auth_token`),\nadd it to `SENSITIVE_KEYS` in `security/log_sanitizer.py` and add a corresponding\nregex pattern to `_SENSITIVE_KEY_NAMES` if needed.\n\n## New Module Checklist\n\nWhen creating a new module or feature that handles credentials or external services:\n\n- [ ] Audit all `logger.*()` calls — do any log connection params, URLs, or tokens?\n- [ ] Use `sanitize_params()` for any dict that may contain credentials\n- [ ] Use `sanitize_url()` for any URL from config/env\n- [ ] Use `redact_token()` for any token/key value\n- [ ] If introducing new credential key names, update `SENSITIVE_KEYS`\n- [ ] Run `python -m pytest tests/backend/security/test_log_sanitizer.py` to verify\n\n\n---\r\ndescription: Use Yarn only, never npm/pnpm\r\nglobs: package.json, yarn.lock\r\nalwaysApply: true\r\n---\r\n\r\n# Package Manager Rules\r\n\r\n- Use Yarn v1.22.22 only - never use npm or pnpm\r\n- Never manually edit yarn.lock\r\n- Keep yarn.lock changes minimal when adding deps\r\n- Registry must be https://registry.yarnpkg.com\n\n\n---\r\ndescription: 路径安全编码规范 — 编辑路由/Agent/Loader/Workspace/知识库代码时自动提醒\r\nglobs: py-src/data_formulator/{routes,agents,data_loader,datalake,security,knowledge}/**/*.py\r\nalwaysApply: false\r\n---\r\n\r\n# 路径安全编码规范\r\n\r\n编辑此目录下的文件时，请遵守以下安全规则（详见 `docs/dev-guides/8-path-safety.md` 和 `.cursor/skills/path-safety/SKILL.md`）：\r\n\r\n## ConfinedDir 是唯一的路径约束原语\r\n\r\n所有接受用户可控路径的代码，**必须**通过 `ConfinedDir` 实例访问文件系统。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\njail = ConfinedDir(root_dir, mkdir=False)\r\ntry:\r\n target = jail.resolve(user_input)\r\nexcept ValueError:\r\n return {\"error\": \"Access denied\"}\r\n```\r\n\r\n## 禁止的模式\r\n\r\n```python\r\n# ❌ BAD — 裸路径拼接\r\ntarget = Path(root) / user_input\r\n\r\n# ❌ BAD — 手写 resolve + relative_to（已弃用，必须用 ConfinedDir）\r\ntarget = (root / user_input).resolve()\r\ntry:\r\n target.relative_to(root)\r\nexcept ValueError: ...\r\n\r\n# ❌ BAD — 手写 is_relative_to（已弃用，必须用 ConfinedDir）\r\nresolved = (root / user_input).resolve()\r\nif not resolved.is_relative_to(root.resolve()): ...\r\n\r\n# ❌ BAD — 路径比较用 str.startswith\r\nif str(target).startswith(str(root)): ...\r\n```\r\n\r\n## 正确的模式\r\n\r\n```python\r\n# ✅ GOOD — Agent 工具\r\nworkspace_jail = self.workspace.confined_root\r\nscratch_jail = self.workspace.confined_scratch\r\ntarget = workspace_jail.resolve(rel_path)\r\n\r\n# ✅ GOOD — 文件下载 route\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target)\r\n\r\n# ✅ GOOD — 文件上传（两层防御）\r\nsafe_name = secure_filename(raw_filename)\r\ntarget = scratch_jail.resolve(safe_name)\r\n\r\n# ✅ GOOD — Workspace data 文件\r\npath = workspace.get_file_path(filename) # 内部使用 ConfinedDir\r\n```\r\n\r\n## 部署守卫\r\n\r\n- 新增读取宿主文件系统的 Loader 必须在 `_enforce_deployment_restrictions()` 注册多用户禁用\r\n- 多用户模式（`WORKSPACE_BACKEND != \"local\"`）必须启用沙箱\n\n\n---\r\ndescription: 测试优先开发与测试保护规则。确保新功能/Bug 修复必须有测试覆盖，测试失败时先诊断后修改。\r\nglobs: \r\nalwaysApply: true\r\n---\r\n\r\n# Test-Driven Workflow\r\n\r\n## 1. 测试优先：新功能 / Bug 修复必须有测试\r\n\r\n开发新功能或修复 Bug 时，**必须同步编写测试**，不能只提交业务代码：\r\n\r\n- **新功能** → 先写测试（或至少同步写），描述期望行为，再实现代码使测试通过\r\n- **Bug 修复** → 先写复现测试，确认测试失败，再修复代码使测试通过\r\n- **重构** → 确保重构前已有测试覆盖，重构后测试仍然全部通过\r\n\r\n完成实现后，主动运行相关测试验证：\r\n- 后端：`python -m pytest tests/backend/ -q`\r\n- 前端：`yarn test`\r\n\r\n## 2. 测试保护：不要轻易修改测试代码\r\n\r\n测试失败时，**禁止直接修改测试代码来让测试通过**。必须执行以下流程：\r\n\r\n1. **复现并定位**：运行失败的测试，确认失败现象\r\n2. **诊断分类**：\r\n - A：测试本身有问题（断言错误、setup 不正确）\r\n - B：业务实现有 Bug\r\n - C：需求/规格变更导致测试过时\r\n3. **给出方案**：至少提供两个可选方案（改测试 / 改实现 / 补规格），说明各自影响和风险\r\n4. **等待确认**：由用户决定采用哪个方案，再执行修改\r\n\r\n**绝对禁止**：\r\n- 测试一红就直接改测试代码\r\n- 删除或跳过（`@pytest.mark.skip`）失败的测试来\"解决\"问题\r\n- 修改测试的断言值来匹配错误的实现\r\n\r\n## 3. 精准修改：不动无关测试\r\n\r\n与 Karpathy 编码准则的\"精准修改\"一致：\r\n\r\n- 不\"顺手改进\"与当前任务无关的测试代码\r\n- 不重构正在通过的测试\r\n- 不改变已有测试的断言逻辑，除非被明确要求\n\n\n---\ndescription: Unified error handling protocol for backend and frontend\nglobs: py-src/**/*.py,src/**/*.{ts,tsx}\nalwaysApply: false\n---\n\n# Unified Error Protocol\n\nSee `docs/dev-guides/7-unified-error-handling.md` for the full developer guide.\n\n## HTTP Status Code Policy\n\n**All application-controlled errors return HTTP 200** with `status: \"error\"` in the body.\nThis applies to both non-streaming JSON APIs and streaming pre-flight errors.\n\nOnly these scenarios use non-200:\n- `401` — `AUTH_REQUIRED` / `AUTH_EXPIRED` (so transport layer can intercept)\n- `403` — `ACCESS_DENIED` (so transport layer can intercept)\n- `404` — Flask routing: no matching route handler\n- `413` — WSGI: request body exceeds `MAX_CONTENT_LENGTH`\n- `500` — Unhandled exception that escaped all error handling\n\n**Streaming NDJSON APIs:**\n- Pre-stream validation failure: HTTP `200` + `application/json` (use `stream_preflight_error()`)\n- In-stream error: NDJSON `{\"type\": \"error\", \"error\": {...}}`\n- In-stream warning: NDJSON `{\"type\": \"warning\", \"warning\": {...}}`\n\n## Backend: Non-streaming JSON Format\n\n```jsonc\n// Success (HTTP 200)\n{\"status\": \"success\", \"data\": {...}}\n\n// Error (HTTP 200 for business errors, 401/403 for auth errors)\n{\n \"status\": \"error\",\n \"error\": {\n \"code\": \"TABLE_NOT_FOUND\",\n \"message\": \"Table not found\",\n \"retry\": false,\n \"detail\": \"...\" // Only in debug mode\n }\n}\n```\n\n## Backend: Streaming NDJSON Format\n\nAll streaming endpoints use `application/x-ndjson`. Each line is one JSON object:\n\n```jsonc\n{\"type\": \"text_delta\", \"data\": {...}}\n{\"type\": \"error\", \"error\": {\"code\": \"LLM_TIMEOUT\", \"message\": \"...\", \"retry\": true}}\n{\"type\": \"done\", \"data\": {...}}\n```\n\n## Backend: How to Handle Errors\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok, stream_preflight_error\nfrom data_formulator.error_handler import stream_error_event, classify_and_wrap_llm_error\n\n# Non-streaming: success\nreturn json_ok(data)\n\n# Non-streaming: error (global handler returns HTTP 200, auth errors get 401/403)\nraise AppError(ErrorCode.TABLE_NOT_FOUND, \"Table not found\")\n\n# Streaming: pre-flight error (always HTTP 200)\nreturn stream_preflight_error(AppError(ErrorCode.INVALID_REQUEST, \"Bad input\"))\n\n# Streaming: in-stream error\nyield stream_error_event(classify_and_wrap_llm_error(e))\n```\n\n## Frontend: How to Consume APIs\n\n```typescript\nimport { apiRequest, streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\n// Non-streaming — apiRequest checks body.status (main path) and HTTP status (auth/infra)\ntry {\n const { data } = await apiRequest(url, options);\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n\n// Streaming — streamRequest handles pre-flight JSON errors and NDJSON events\ntry {\n for await (const event of streamRequest(url, options, signal)) {\n if (event.type === 'error') { /* handle inline */ }\n // ...process events...\n }\n} catch (e) {\n handleApiError(e, 'my-component');\n}\n```\n\nAPI consumers MUST use `apiRequest()` / `streamRequest()` plus `handleApiError()`.\nDirect `fetchWithIdentity()` is reserved for lower-level clients and explicit protocol\nexceptions such as file downloads, blob/CSV responses, SPA/OIDC redirects, or third-party URLs.\n\n## Prohibited Patterns\n\n### Backend\n- ❌ `return jsonify({\"status\": \"ok\", \"data\": data})` — use `json_ok(data)` instead\n- ❌ `return jsonify({\"error_message\": str(e)})` — use `raise AppError(...)` instead\n- ❌ `yield json.dumps({\"type\": \"error\", \"error\": str(e)})` — use `stream_error_event()`\n- ❌ Bare `except:` or `except Exception: pass` — always log or propagate\n- ❌ `yield 'error: ' + json.dumps(...)` — non-standard prefix\n\n### Frontend\n- ❌ `.catch(() => {})` — either add a comment explaining why (best-effort), or use `handleApiError`\n- ❌ `.catch(console.error)` without user feedback — dispatch `addMessages` or use `handleApiError`\n- ❌ Raw `fetch()` for `/api/` URLs — use `fetchWithIdentity` (identity headers + 401 retry)\n- ❌ RTK thunk without `.rejected` handler — always add `.addCase(thunk.rejected, ...)` with `addMessages`\n- ❌ New code using `fetchWithIdentity().json()` — use `apiRequest()` instead\n\n### RTK Thunk Rejected Handler Pattern\n\n```typescript\n.addCase(myThunk.rejected, (state, action) => {\n if (action.error?.name !== 'AbortError') {\n state.messages.push({\n timestamp: Date.now(), type: 'warning',\n component: 'my-component',\n value: 'Human-readable error description',\n });\n }\n})\n```\n\n## Migrated Streaming Endpoints\n\nAll streaming endpoints now use `application/x-ndjson` and `stream_error_event()`:\n\n| Endpoint | Old format | New format | Status |\n|----|-----|---|-----|\n| `/data-agent-streaming` | `{status:\"error\", error_message}` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/get-recommendation-questions` | `error: {json}` prefix + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n| `/generate-report-chat` | `data: {json}` SSE prefix + `text/event-stream` | Pure NDJSON + `application/x-ndjson` | ✅ Done |\n| `/data-loading-chat` | `str(e)` in error field | `stream_error_event()` safe message | ✅ Done |\n| `/clean-data-stream` | `\\n{json}\\n` + `application/json` | `stream_error_event()` + `application/x-ndjson` | ✅ Done |\n\n## Complete ErrorCode Reference\n\n```\nAUTH_REQUIRED, AUTH_EXPIRED, ACCESS_DENIED\nINVALID_REQUEST, TABLE_NOT_FOUND, FILE_PARSE_ERROR, FILE_TOO_LARGE, VALIDATION_ERROR\nLLM_AUTH_FAILED, LLM_RATE_LIMIT, LLM_CONTEXT_TOO_LONG, LLM_MODEL_NOT_FOUND\nLLM_TIMEOUT, LLM_SERVICE_ERROR, LLM_CONTENT_FILTERED, LLM_UNKNOWN_ERROR\nDB_CONNECTION_FAILED, DB_QUERY_ERROR, DATA_LOAD_ERROR, CONNECTOR_ERROR\nCODE_EXECUTION_ERROR, AGENT_ERROR\nCATALOG_SYNC_TIMEOUT, CATALOG_NOT_FOUND, ANNOTATION_CONFLICT, ANNOTATION_INVALID_PATCH\nINTERNAL_ERROR, SERVICE_UNAVAILABLE\n```\n\n## Adding a New Error Code\n\n1. Add constant to `py-src/data_formulator/errors.py` `ErrorCode` class (no HTTP mapping needed, defaults to 200)\n2. Add i18n mapping to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`\n3. Add translations to `src/i18n/locales/en/errors.json` AND `zh/errors.json`\n\n## New Endpoint Checklist\n\nWhen adding a new API endpoint, verify:\n- [ ] Success: `json_ok(data)` → `{\"status\": \"success\", \"data\": ...}`, HTTP 200\n- [ ] Error: `raise AppError(...)` → HTTP 200 + error body (auth errors get 401/403)\n- [ ] Streaming pre-flight: `stream_preflight_error(...)` → HTTP 200 + `application/json`\n- [ ] Streaming in-stream: `stream_error_event(...)` via NDJSON\n- [ ] No `str(e)` in any response body\n- [ ] No naked `except:` — use specific exception types or `except Exception as e:` with logging\n- [ ] Frontend uses `apiRequest()` (non-streaming) or `streamRequest()` (streaming)\n- [ ] Error test cases assert HTTP 200 + `body.status == \"error\"` (auth: 401/403)\n\n\n---\nname: error-handling\ndescription: 统一错误处理系统。在添加 API 端点、修改错误处理、添加前端 API 调用、编写错误相关测试时使用。\n---\n\n# Error Handling Skill\n\nUnified error handling system for DF. Use when adding API endpoints, modifying error handling, or adding frontend API calls.\n\n> **Prerequisites**: Read `docs/dev-guides/7-unified-error-handling.md` before changing API error behavior.\n> Read `docs/dev-guides/2-log-sanitization.md` when the work involves logging, credentials, external services, or DataLoaders.\n> If your work introduces new error handling patterns or conventions, update this file and related dev-guides accordingly.\n\n## Architecture Overview\n\n```\nFrontend Backend\n──────── ───────\napiClient.ts errors.py\n├── apiRequest() ←── JSON ──── ├── ErrorCode (enum)\n├── streamRequest() ←── NDJSON ── └── AppError (exception)\n└── parseStreamLine()\n error_handler.py\nerrorCodes.ts ├── register_error_handlers(app)\n└── getErrorMessage() ├── classify_and_wrap_llm_error()\n └── stream_error_event()\nerrorHandler.ts\n└── handleApiError() security/sanitize.py\n └── classify_llm_error() (internal)\nMessageSnackbar ← dfSlice.messages\n```\n\n## Protocol Snapshot\n\nUse this contract for all new or reworked DF APIs:\n\n| Scenario | HTTP | Shape |\n|----------|------|-------|\n| Non-streaming success | `200` | `{\"status\": \"success\", \"data\": ...}` |\n| Non-streaming business/validation error | `200` | `{\"status\": \"error\", \"error\": {\"code\", \"message\", \"retry\", \"request_id\"}}` |\n| Non-streaming auth/authorization error | `401` / `403` | same structured error body |\n| Streaming preflight error | `200` | `application/json` + `{\"status\": \"error\", \"error\": ...}` |\n| Streaming in-flight fatal error | `200` | NDJSON line: `{\"type\": \"error\", \"error\": ...}` |\n| No Flask route / too large / unhandled crash | `404` / `413` / `500` | transport-level error |\n\nDo not use HTTP `400`/`422` for application validation errors in new code.\nDo not convert in-flight NDJSON errors to `status: \"error\"`; once the stream has\nstarted, event `type` is the protocol discriminator.\n\n## Backend: Adding a New API Endpoint\n\n### HTTP Status Code Policy\n\n**Application-controlled business and validation errors return HTTP 200** with\n`status: \"error\"` in the body. Only these use non-200:\n- `401`/`403` — auth errors (`AUTH_REQUIRED`, `AUTH_EXPIRED`, `ACCESS_DENIED`)\n- `404` — no matching Flask route\n- `413` — WSGI body limit exceeded\n- `500` — unhandled exception (program bug)\n\n### Non-streaming endpoint\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import json_ok\n\n@bp.route('/my-endpoint', methods=['POST'])\ndef my_endpoint():\n content = request.get_json()\n if not content.get('required_field'):\n raise AppError(ErrorCode.INVALID_REQUEST, \"Missing required_field\")\n\n try:\n result = do_work(content)\n except SomeBusinessError as e:\n raise AppError(ErrorCode.DATA_LOAD_ERROR, \"Failed to load data\") from e\n except Exception as e:\n from data_formulator.error_handler import classify_and_wrap_llm_error\n raise classify_and_wrap_llm_error(e) from e\n\n return json_ok(result)\n# Global handler returns: HTTP 200 + {\"status\": \"error\", \"error\": {code, message, retry}}\n# Auth errors (AUTH_REQUIRED/AUTH_EXPIRED/ACCESS_DENIED) return 401/403\n```\n\nLegacy `{\"status\": \"error\", \"message\": \"...\"}`, `error_message`, bare `{error}`,\nand `status: \"ok\"` responses are historical formats. Do not add new compatibility\nbranches for them; migrate the route to `json_ok()` / `AppError` before using\n`apiRequest()`.\n\n### Streaming endpoint\n\nValidation MUST be outside the generator. Failures return 200 JSON (not NDJSON).\n\n```python\nfrom data_formulator.errors import AppError, ErrorCode\nfrom data_formulator.error_handler import (\n classify_and_wrap_llm_error,\n stream_error_event,\n stream_preflight_error,\n)\n\n@bp.route('/my-stream', methods=['POST'])\ndef my_stream():\n if not request.is_json:\n return stream_preflight_error(\n AppError(ErrorCode.INVALID_REQUEST, \"Invalid request\")\n )\n\n content = request.get_json()\n client = get_client(content['model'])\n\n def generate():\n try:\n for event in agent.run(...):\n yield json.dumps(event, ensure_ascii=False) + \"\\n\"\n except Exception as e:\n yield stream_error_event(classify_and_wrap_llm_error(e))\n\n return Response(stream_with_context(generate()), mimetype='application/x-ndjson')\n```\n\nStreaming runtime errors intentionally use `{\"type\": \"error\", \"error\": ...}`.\nThey cannot use a top-level `status` envelope because the HTTP response and NDJSON\nevent stream have already started.\n\n## Frontend: Consuming an API\n\n### Non-streaming\n\n```typescript\nimport { apiRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n const { data } = await apiRequest(getUrls().MY_ENDPOINT, {\n method: 'POST',\n body: JSON.stringify(payload),\n headers: { 'Content-Type': 'application/json' },\n });\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\nFor UI loading state, model the request lifecycle explicitly with\n`LoadableState` from `src/app/loadableState.ts`. Do not infer loading from\nmissing data (`!data`), because failed requests may legitimately leave data\nempty while loading has ended.\n\n### Streaming\n\n```typescript\nimport { streamRequest } from '../app/apiClient';\nimport { handleApiError } from '../app/errorHandler';\n\ntry {\n for await (const event of streamRequest(url, options, abortController.signal)) {\n switch (event.type) {\n case 'text_delta':\n break;\n case 'error':\n // Error arrived mid-stream — show inline in component.\n break;\n case 'done':\n break;\n }\n }\n} catch (e) {\n handleApiError(e, 'MyComponent');\n}\n```\n\n### With callbacks\n\n```typescript\nhandleApiError(e, 'MyComponent', {\n onAuth: () => redirectToLogin(), // AUTH_REQUIRED / AUTH_EXPIRED\n onRetryable: () => retryOperation(), // LLM_RATE_LIMIT / LLM_TIMEOUT\n silent: true, // don't show Snackbar (component handles display)\n});\n```\n\n### Migration and special cases\n\nDF API consumers should use `apiRequest()` / `streamRequest()` and\n`handleApiError()`. Direct `fetchWithIdentity()` is for lower-level client helpers\nand explicit protocol exceptions such as file downloads, blob/CSV responses,\nOIDC redirects, SPA fallback, or third-party URLs.\n\nDo not apply the normal JSON API protocol mechanically to file downloads / CSV\nstreaming, SPA fallback, OIDC redirect flows, frontend fetches to third-party\nURLs, or errors after a streaming response has already started. Check the route's\nprotocol first, then preserve safe error bodies and avoid `str(exc)` exposure.\n\n## Adding a New Error Code\n\n1. **Backend** — Add to `py-src/data_formulator/errors.py` `ErrorCode`:\n ```python\n MY_NEW_ERROR = \"MY_NEW_ERROR\"\n ```\n No HTTP mapping needed — defaults to HTTP 200. Only add to `ERROR_CODE_HTTP_STATUS` if it's an auth code.\n\n2. **Frontend mapping** — Add to `src/app/errorCodes.ts` `ERROR_CODE_I18N_MAP`:\n ```typescript\n MY_NEW_ERROR: 'errors.myNewError',\n ```\n\n3. **Translations** — Add to both locale files:\n - `src/i18n/locales/en/errors.json`: `\"myNewError\": \"English message\"`\n - `src/i18n/locales/zh/errors.json`: `\"myNewError\": \"中文消息\"`\n\n## Migrated Endpoints Reference\n\nAll streaming endpoints are now on the unified protocol:\n\n| Endpoint | Format | Notes |\n|----------|--------|-------|\n| `/data-agent-streaming` | NDJSON + `stream_error_event()` | Emits top-level `type` events; errors use `{type:\"error\", error:{...}}` |\n| `/get-recommendation-questions` | NDJSON + `stream_error_event()` | Was `error: {json}` prefix |\n| `/generate-report-chat` | Pure NDJSON + `stream_error_event()` | Was SSE `data: {json}` prefix |\n| `/data-loading-chat` | NDJSON + `stream_error_event()` | `str(e)` removed |\n| `/clean-data-stream` | NDJSON + `stream_error_event()` | Was `\\n{json}\\n` format |\n\nNon-streaming endpoints:\n\n| Endpoint | Error Format | Notes |\n|----------|-------------|-------|\n| `/chart-insight` | `AppError` → HTTP 200 + `{status:\"error\", error:{code,message,retry}}` | Fully migrated. Frontend uses `fetchChartInsight` rejected reducer. |\n| All migrated endpoints | `AppError` → HTTP 200 + unified error body | credentials, knowledge, sessions, tables, agents |\n| `/derive-data`, `/refine-data`, `/sort-data`, `/process-data-on-load`, `/test-model` | `json_ok()` / `AppError` | Migrated to new format |\n\n## Empty Catch Policy\n\nNot all `.catch(() => {})` are bugs. Use this decision tree:\n\n1. **User-initiated action** (delete, refresh, submit) → must notify with `addMessages` or `handleApiError()`\n2. **Background/best-effort fetch** (connector list on mount, session list) → OK to swallow, but add a comment\n3. **RTK thunks** → always add `.rejected` handler with `addMessages`\n4. **AbortError** → filter out with `if (action.error?.name !== 'AbortError')`\n\n## Frontend Stream Parsing Pattern\n\nWhen consuming a migrated streaming endpoint, handle the current NDJSON event\nformat directly:\n\n```typescript\nconst data = JSON.parse(line);\nif (data.type === 'error') {\n const errMsg = data.error?.message || 'Unknown error';\n // show to user...\n}\n```\n\n## Backend: Database/Workspace Errors (tables.py)\n\nFor table CRUD endpoints, use the specialized classifier:\n\n```python\nfrom data_formulator.routes.tables import classify_and_raise_db_error\n\n@tables_bp.route('/my-table-op', methods=['POST'])\ndef my_table_op():\n try:\n result = workspace.do_something()\n return jsonify({\"status\": \"success\", \"data\": result})\n except Exception as e:\n classify_and_raise_db_error(e)\n```\n\n`classify_and_raise_db_error` maps common DB errors to appropriate `AppError` codes\n(returned as HTTP 200 by the global handler, except ACCESS_DENIED → 403):\n- \"Table does not exist\" → `TABLE_NOT_FOUND` (HTTP 200)\n- \"Table already exists\" → `INVALID_REQUEST` (HTTP 200)\n- \"Permission denied\" → `ACCESS_DENIED` (HTTP 403)\n- Other → `CONNECTOR_ERROR` (HTTP 200)\n\n## Backend: Connector Errors (data_connector.py)\n\nFor connector endpoints, use:\n\n```python\nfrom data_formulator.data_connector import classify_and_raise_connector_error\n\nexcept Exception as e:\n classify_and_raise_connector_error(e, operation=\"preview\")\n```\n\nConnector/DataLoader classification is intentionally simple and lives in\n`data_formulator.data_loader.connector_errors`. It maps common failures to a\nsmall stable set: `INVALID_REQUEST`, `CONNECTOR_AUTH_FAILED`, `AUTH_EXPIRED`,\n`ACCESS_DENIED`, `DB_CONNECTION_FAILED`, `DB_QUERY_ERROR`, `DATA_LOAD_ERROR`,\nor `CONNECTOR_ERROR`. Do not add endpoint-local string matching unless the\nclassifier cannot reasonably cover the category.\n\nAll JSON errors include `error.request_id` and an `X-Request-Id` response\nheader. Show/copy this ID for users when reporting backend failures; do not\nshow raw exception text in production. Unhandled 500 responses must never\ninclude raw tracebacks, even in debug mode; return a safe category plus\n`request_id` and keep full stack traces in server logs only.\n\n## Debugging Error Propagation\n\nWhen an error isn't reaching the frontend:\n\n1. **Check backend logs** — is the error logged?\n2. **Check response format**:\n - Non-streaming: `{\"status\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n - Streaming: one line `{\"type\": \"error\", \"error\": {\"code\": ..., \"message\": ...}}`\n3. **Check Content-Type** — streaming must be `application/x-ndjson`, not `application/json` or `text/event-stream`\n4. **Check frontend parser** — is the consumer looking for `data.type === 'error'`?\n5. **Check global handler** — verify `register_error_handlers(app)` is called in `app.py`\n6. **Check blueprint handlers** — blueprint-level `errorhandler(Exception)` takes priority over global handlers\n\nLegacy `message` / `error_message` bodies are protocol violations on migrated API paths.\n\n## Log Sanitization (Sensitive Data in Server Logs)\n\nServer-side logs must never leak passwords, tokens, API keys, or connection strings.\nThe project uses a defense-in-depth approach with two layers.\n\n### Layer 1: Explicit Utilities (call-site)\n\n```python\nfrom data_formulator.security.log_sanitizer import (\n sanitize_url, sanitize_params, redact_token,\n)\n\n# Dict with credentials → sanitize_params()\nlog.info(\"Connecting with: %s\", sanitize_params(params))\n\n# URL that may embed credentials → sanitize_url()\nlogger.info(\"Issuer: %s\", sanitize_url(issuer_url))\n\n# Token/API key → redact_token()\nlogger.debug(\"Token: %s\", redact_token(token))\n```\n\n### Layer 2: SensitiveDataFilter (global safety net)\n\nRegistered in `app.py:configure_logging()`. Automatically redacts:\n- URL credentials (`://user:pass@host`)\n- `Bearer` tokens\n- `password=xxx`, `api_key=xxx`, `secret=xxx` patterns\n- JWT-like base64 strings\n- Python dict repr with sensitive keys\n\nDisable with `LOG_SANITIZE=false` for local debugging only.\n\n### When to Use What\n\n| Data | Utility | Why not just filter? |\n|------|---------|---------------------|\n| `dict` with password keys | `sanitize_params()` | Filter can't identify arbitrary password values in dict repr |\n| URL from config/env | `sanitize_url()` | Explicit is clearer; filter is backup |\n| Token/key value | `redact_token()` | Explicit is clearer; filter is backup |\n| Normal text | Nothing | Filter handles edge cases |\n\n### New Module Checklist\n\nWhen adding a module that handles credentials or external services:\n\n1. Audit all `logger.*()` calls for credential/URL/token logging\n2. Use `sanitize_params()` for dicts, `sanitize_url()` for URLs, `redact_token()` for tokens\n3. Prefer `type(exc).__name__` over `str(exc)` in warning-level logs\n4. If introducing new credential key names, add to `SENSITIVE_KEYS` in `log_sanitizer.py`\n\n## Key Files\n\n| File | Purpose |\n|------|---------|\n| `py-src/data_formulator/errors.py` | `ErrorCode` enum + `AppError` exception |\n| `py-src/data_formulator/error_handler.py` | Global handlers, `classify_and_wrap_llm_error`, `stream_error_event` |\n| `py-src/data_formulator/security/log_sanitizer.py` | `sanitize_url`, `sanitize_params`, `redact_token`, `SensitiveDataFilter` |\n| `py-src/data_formulator/routes/tables.py` | `classify_and_raise_db_error` (database/workspace errors) |\n| `py-src/data_formulator/data_connector.py` | `classify_and_raise_connector_error` (connector errors) |\n| `py-src/data_formulator/security/sanitize.py` | `classify_llm_error` (internal), `sanitize_error_message` |\n| `src/app/apiClient.ts` | `apiRequest`, `streamRequest`, `parseStreamLine`, `ApiRequestError` |\n| `src/app/errorHandler.ts` | `handleApiError` |\n| `src/app/errorCodes.ts` | `ERROR_CODE_I18N_MAP`, `getErrorMessage` |\n| `src/i18n/locales/{en,zh}/errors.json` | Error message translations |\n\n\n---\r\nname: language-injection\r\ndescription: LLM Agent 多语言注入规范。在修改 Agent 提示词、添加新的 Agent 端点、处理用户可见的后端消息（message_code）时使用。\r\n---\r\n\r\n# Language Injection for Agent Prompts\r\n\r\nAuthoritative developer guide: `docs/dev-guides/6-i18n-language-injection.md`.\r\n\r\n> **Prerequisites**: Read `docs/dev-guides/6-i18n-language-injection.md` before changing Agent prompts, Agent routes, backend user-visible messages, or frontend i18n strings.\r\n> If your work introduces new language injection patterns or conventions, update this file and related dev-guides accordingly.\r\n\r\n## Architecture\r\n\r\n```\r\nFrontend i18n.language → Accept-Language header → get_language_instruction()\r\n │\r\n build_language_instruction()\r\n (agents/agent_language.py)\r\n │\r\n ┌────────────┴────────────┐\r\n ▼ ▼\r\n mode=\"full\" mode=\"compact\"\r\n (text-heavy agents) (code-gen agents)\r\n```\r\n\r\n### Core Modules\r\n\r\n| Module | Role |\r\n|--------|------|\r\n| `agents/agent_language.py` | `build_language_instruction(lang, mode)` — generates prompt fragments; `inject_language_instruction()` — injects into system prompts; supports 20 languages; returns `\"\"` for English |\r\n| `routes/agents.py` → `get_language_instruction()` | Reads `Accept-Language` header, delegates to `build_language_instruction` |\r\n| `routes/agents.py` → `_get_ui_lang()` | Extracts primary language code from `Accept-Language` header |\r\n| `src/app/utils.tsx` → `fetchWithIdentity()` | Sets `Accept-Language` header on every API request from `i18n.language` |\r\n| `src/app/utils.tsx` → `translateBackend()` | Translates backend `message_code` / `content_code` using frontend i18n |\r\n\r\n## Code Examples\r\n\r\n### Route handler — inject language\r\n\r\n```python\r\n# In a Flask route handler:\r\nlang_instruction = get_language_instruction(mode=\"compact\")\r\nlang_suffix = f\"\\n\\n{lang_instruction}\" if lang_instruction else \"\"\r\n\r\nmessages = [\r\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\" + lang_suffix},\r\n {\"role\": \"user\", \"content\": user_input},\r\n]\r\n```\r\n\r\n### Agent constructor — use inject_language_instruction()\r\n\r\n```python\r\nfrom data_formulator.agents.agent_language import inject_language_instruction\r\n\r\n# Simple append (most agents)\r\nsystem_prompt = inject_language_instruction(system_prompt, language_instruction)\r\n\r\n# Insert before a marker (complex prompts)\r\nsystem_prompt = inject_language_instruction(\r\n system_prompt, language_instruction,\r\n marker=\"**About the execution environment:**\"\r\n)\r\n```\r\n\r\n### Python-side user-visible messages — message_code pattern\r\n\r\nFor fixed strings in Python that appear in the UI, do NOT translate in Python.\r\nReturn a `message_code` and let the frontend translate:\r\n\r\n```python\r\n# In an Agent or route handler:\r\nyield {\r\n \"type\": \"error\",\r\n \"message\": \"Output DataFrame is empty (0 rows).\", # English fallback\r\n \"message_code\": \"agent.emptyDataframe\", # frontend i18n key\r\n}\r\n\r\n# With parameters:\r\nresult = {\r\n \"status\": \"error\",\r\n \"content\": f\"Fields not found: {missing}\",\r\n \"content_code\": \"agent.fieldsNotFound\",\r\n \"content_params\": {\"missing\": missing, \"available\": available},\r\n}\r\n```\r\n\r\nFrontend consumption:\r\n```tsx\r\nimport { translateBackend } from '../app/utils';\r\nconst msg = translateBackend(event.message, event.message_code, event.message_params);\r\n```\r\n\r\nTranslation keys go in `src/i18n/locales/{en,zh}/messages.json` under `messages.agent.*`.\r\n\r\n## Anti-Patterns (with explanations)\r\n\r\n| Pattern | Why it's wrong |\r\n|---------|---------------|\r\n| `os.environ.get(\"DF_DEFAULT_LANGUAGE\")` | Process-level — all users get same language; breaks multi-user |\r\n| Global LLM client interceptor | Hidden behavior; can't distinguish full/compact mode; fragile string detection |\r\n| New `MessageBuilder` class | Duplicates `agent_language.py`; creates parallel conflicting abstractions |\r\n| Hardcoded `\"回答请使用中文\"` in prompts | Not configurable; skips the mode system; breaks for other languages |\r\n| Backend-side translation dict (`agent_messages.py`) | Forces adding every new language to Python; translations should all live in `src/i18n/locales/` |\r\n| Hardcoded English UI strings in `.tsx` without `t()` | Not translatable; use `useTranslation` + `t('key')` |\r\n\r\n## Adding a New Language\r\n\r\n1. Add language code + display name to `LANGUAGE_DISPLAY_NAMES` in `agents/agent_language.py`.\r\n2. Optionally add extra rules to `LANGUAGE_EXTRA_RULES` (e.g. simplified vs traditional Chinese).\r\n3. Add frontend translations in `src/i18n/locales//` — copy an existing locale folder as template.\r\n4. No Agent code changes needed — the existing flow picks up new languages automatically.\n\n\n---\r\nname: path-safety\r\ndescription: 服务端路径安全与文件访问编码规范。在编写文件下载路由、Agent 工具（文件读取/目录列出）、数据连接器/Loader、Workspace 路径操作、沙箱配置时使用。\r\n---\r\n\r\n# Path Safety — 服务端安全编码规范\r\n\r\n> **来源**：`docs/dev-guides/8-path-safety.md`（正式开发规范）+ `design-docs/issues/002-arbitrary-file-read-audit.md`（安全审计复核）。\r\n> 本文档提炼了 6 条必须遵守的编码规范。违反任一条即可能引入路径穿越（LFI）漏洞。\r\n\r\n---\r\n\r\n## R1. 文件下载：用 `ConfinedDir.resolve()` + `send_file`，禁用 `send_from_directory`\r\n\r\n**原因**：`send_from_directory(dir, user_input)` 内部会对 `user_input` 二次解析路径，与前置安全检查形成 TOCTOU 不一致。\r\n\r\n```python\r\n# ❌ BAD — 安全检查用 resolved target，发送用原始 filename，两次解析不一致\r\ntarget = (scratch_dir / filename).resolve()\r\ntarget.relative_to(scratch_dir.resolve()) # 检查通过\r\nreturn send_from_directory(str(scratch_dir), filename) # 再次解析\r\n\r\n# ✅ GOOD — 检查和发送用同一个 resolved path\r\nscratch_jail = workspace.confined_scratch\r\ntarget = scratch_jail.resolve(filename)\r\nreturn send_file(target) # 直接用已验证的路径\r\n```\r\n\r\n`send_file(Path)` 会根据扩展名自动推断 MIME type，无需额外处理。\r\n\r\n---\r\n\r\n## R2. 路径安全检查：用 `ConfinedDir`，禁止 `str.startswith`\r\n\r\n**原因**：`str(path).startswith(str(root))` 存在前缀碰撞缺陷（如 `/workspace` vs `/workspace_evil`）。\r\n\r\n```python\r\n# ❌ BAD\r\nif not str(resolved).startswith(str(root_resolved) + os.sep):\r\n raise ValueError(\"escape\")\r\n\r\n# ✅ GOOD — 统一走 ConfinedDir，内部使用 Path.is_relative_to()\r\njail = ConfinedDir(root_resolved, mkdir=False)\r\ntarget = jail.resolve(user_input)\r\n```\r\n\r\n---\r\n\r\n## R3. Agent 工具复用 `Workspace.confined_*`\r\n\r\n**原因**：Agent 工具参数由 LLM 生成，必须视为间接用户输入。不要在工具内手写 `Path(root) / rel_path` 或 `resolve() + relative_to()`；入口处复用 Workspace 暴露的 `ConfinedDir`。\r\n\r\n```python\r\n# ❌ BAD — 手写路径拼接和校验\r\ndef _tool_read_file(self, args, workspace_path):\r\n target = (workspace_path / rel_path).resolve()\r\n target.relative_to(workspace_path)\r\n\r\n# ✅ GOOD — 入口拿到 ConfinedDir，工具只调用 jail.resolve()\r\ndef _execute_tool(self, name, args):\r\n workspace_jail = self.workspace.confined_root\r\n scratch_jail = self.workspace.confined_scratch\r\n return self._tool_read_file(args, workspace_jail)\r\n\r\ndef _tool_read_file(self, args, workspace_jail):\r\n target = workspace_jail.resolve(args.get(\"path\", \"\"))\r\n```\r\n\r\n---\r\n\r\n## R4. 优先使用 `ConfinedDir`，禁止裸路径拼接\r\n\r\n**原因**：`Path(root) / user_input` 是路径穿越的高频入口。`ConfinedDir` 封装了三层防御（拒绝绝对路径 → 拒绝 `..` 段 → `resolve` + `is_relative_to`）。\r\n\r\n```python\r\nfrom data_formulator.security.path_safety import ConfinedDir\r\n\r\n# ❌ BAD — 手动拼接 + 手动校验，容易遗漏\r\nlocal_file = tmp_path / blob_relative_name\r\nlocal_file.parent.mkdir(parents=True, exist_ok=True)\r\nlocal_file.write_bytes(data)\r\n\r\n# ✅ GOOD — ConfinedDir 自动校验 + 创建父目录\r\njail = ConfinedDir(tmp_path, mkdir=False)\r\njail.write(blob_relative_name, data) # 自动校验 + 写入\r\n```\r\n\r\n### 已有安全 API 的层次关系\r\n\r\n```\r\n用户输入（filename / relative_path / blob key）\r\n │\r\n ▼\r\nsafe_data_filename() / secure_filename() ← 第一层：输入清洗\r\n │\r\n ▼\r\nConfinedDir.resolve() ← 第二层：路径约束\r\n │\r\n ▼\r\n安全的 Path 对象\r\n```\r\n\r\n使用 `Workspace.get_file_path()` 的场景不需要手动调用 `ConfinedDir`，因为它内部已包含等价的校验。\r\n\r\n---\r\n\r\n## R5. 宿主文件系统访问必须设部署模式守卫\r\n\r\n**原因**：桌面单用户模式允许访问本机文件系统（预期行为），但多用户/云部署下等于开放服务器读权限。\r\n\r\n**规范**：任何新的 Loader / Connector 如果涉及直接读取宿主文件系统（不通过 Workspace API），必须：\r\n\r\n1. 在 `data_loader/__init__.py` 的 `_enforce_deployment_restrictions()` 中注册禁用规则\r\n2. 确保 `create_connector()` 会拒绝已禁用的类型\r\n\r\n```python\r\n# data_loader/__init__.py — 参考 local_folder 的处理方式\r\ndef _enforce_deployment_restrictions():\r\n backend = os.environ.get(\"WORKSPACE_BACKEND\", \"local\")\r\n if backend != \"local\":\r\n for key in (\"local_folder\", \"your_new_local_loader\"):\r\n if key in DATA_LOADERS:\r\n del DATA_LOADERS[key]\r\n DISABLED_LOADERS[key] = f\"{key} disabled in multi-user mode\"\r\n```\r\n\r\n**判断标准**：如果 Loader 的构造函数接受一个用户可控的本机路径（如 `root_dir`），它就需要部署守卫。\r\n\r\n---\r\n\r\n## R6. 多用户部署必须启用沙箱\r\n\r\n**原因**：`not_a_sandbox` 模式下 LLM 生成的代码在宿主进程直接执行，可绕过所有路径检查。\r\n\r\n`app.py` 已在启动时检测此配置并输出 `logger.critical` 警告。新增的沙箱模式或部署脚本应确保：\r\n\r\n- `WORKSPACE_BACKEND != \"local\"` 时，`SANDBOX` 必须为 `docker` 或 `local`\r\n- CI/CD 部署模板中默认设置 `SANDBOX=docker`\r\n\r\n---\r\n\r\n## 速查：新增代码时的安全检查清单\r\n\r\n| 场景 | 必须做的事 |\r\n|------|-----------|\r\n| 新增文件下载路由 | 用 `ConfinedDir.resolve()` 得到路径，再 `send_file(resolved_path)`；不用 `send_from_directory` |\r\n| 新增 Agent 工具（读文件/列目录） | 入口复用 `workspace.confined_root` / `workspace.confined_scratch`，工具内只调用 `jail.resolve()` |\r\n| 路径包含判断 | 用 `ConfinedDir.resolve()`，不要手写 `Path.is_relative_to()` 或 `str.startswith()` |\r\n| `Path(root) / variable` 模式 | 改用 `ConfinedDir` 或 `Workspace.get_file_path()` |\r\n| 新增本机文件系统 Loader | 在 `_enforce_deployment_restrictions()` 中注册多用户禁用 |\r\n| 部署配置 | 多用户模式必须 `SANDBOX=docker` 或 `SANDBOX=local` |\r\n\r\n---\r\n\r\n## 参考文档\r\n\r\n- `docs/dev-guides/8-path-safety.md` — 服务端路径安全开发规范\r\n- `design-docs/6-path-safety-confined-dir.md` — 剩余未完成实现项状态页\r\n- `design-docs/issues/002-arbitrary-file-read-audit.md` — 安全审计复核报告\r\n- `py-src/data_formulator/security/path_safety.py` — ConfinedDir 源码\n\n\n---\nname: core\ndescription: >-\n The analyst's built-in capabilities: data-inspection tools and the\n always-available actions (visualize and ask_user).\nwhen_to_use: Always loaded by default — this is the agent's baseline.\nalways_on: true\ntools:\n - execute_python_script\n - inspect_source_data\nactions:\n - visualize\n - ask_user\n---\n\n# Core capabilities\n\nThis describes the built-in **inspection tools** you use to gather data and the\nalways-available **actions** you take on it. The overall loop, your action\nbudget, and the one-action-per-turn rule are covered in your system\ninstructions — this section is about *what* each tool and action does and how\nto use it well.\n\n## Tools (for data gathering)\n\n- **execute_python_script(code)** — run a general-purpose Python script to\n inspect data, compute stats, transform tables, or verify assumptions. Its\n stdout is returned to you (use `print()`); the script is for *your* analysis\n and its output is never shown to the user. pandas, numpy, duckdb, sklearn,\n scipy are available. **Important**: each call runs in a fresh namespace —\n variables do NOT persist between calls, so combine related steps into a\n single script.\n- **inspect_source_data(table_names)** — get schema, stats, and sample rows for\n source tables (cheaper than `execute_python_script` for basic inspection).\n- **load_skill(name)** — load a skill's instructions into context so you can use\n the action it unlocks (see the Skills section of your system instructions).\n\nThese are inspection tools — their results come back to you and are never shown\nto the user; call as many as you need, then take an action or give your final\nanswer.\n\nYou analyse data that is **already in the workspace**. If the user's question\nrequires connected data that isn't present, call `load_skill(\"data-loading\")`\nand follow that skill's discovery and immutable proposal workflow in this same\nconversation. Do not hand off to the standalone Data Loading agent.\n\nThe initial context already includes sample rows and statistics for each table.\nIf the data is straightforward, go straight to the action without calling\ntools. Tool results are returned to you before you act.\n\n## Actions\n\nCall an action as a tool call when you want to act on the data. Actions are\n**sequential**: take **one at a time**, then read the result it returns before\ndeciding the next — each action's outcome shapes the next one (the chart you draw\nnext depends on what this one reveals), so emitting several at once would decide\nthe later ones blind. After each result you choose what to do — take another\naction, or stop. **You end your turn by replying with plain text and no\naction**: that is your closing answer when you expect nothing further. When you\nwant the user to reply — a freeform question, a clarification you need before\nacting, or **clickable choices** — use the `ask_user` action instead. It renders\na question widget and pauses for their reply, keeping the conversation in the\nsame turn (plain text ends the run, so the user's next message would start\nfresh without this context).\n\n**Match the response to what the user asked for.** Two different cases:\n\n- **A direct answer** — they asked a question, so answer it. Length follows the\n question: one line when that settles it, more when it genuinely takes more.\n- **A finding after you acted** — they asked for the work, not a write-up, so\n this is unsolicited. The artifact already shows what it shows; add only what\n they'd miss by looking at it, and default to short.\n\nOpen with the point rather than announcing one is coming, and don't close by\nrestating what you just said.\nNever narrate what you're about to do or recap a chart's axes; let the artifact\nspeak for itself. When an action pauses for the user, give enough context to\nexplain what you found and what their choices mean.\n\n### `visualize` — chart a transform\n\nRun code that produces a DataFrame and render it as a chart. You then observe the\nresult and decide your next move.\n\n- `display_instruction` — ≤12 words; the question/hypothesis the chart\n investigates (don't recap x/y/color — those are visible). Wrap a **column** in\n `**…**` if it anchors the question.\n- `title` — a concise, neutral analytical heading naming the subject, measure,\n and analytical lens, such as “Year-over-year price change peaks.” Prefer a\n stable description of the view over a takeaway claim or narrated trend. Do\n not mention the chart type, imply causality, or editorialize. This field is\n required; put interpretation in the closing response instead.\n- `subtitle` — concise supporting context not already clear from the title or\n axes. Use one phrase of at most 16 words to provide contextual details. Do\n not restate the measure or analytical lens named in the title.\n- `code` — Python producing a DataFrame assigned to `output_variable`.\n- `output_variable` — snake_case name the code assigns.\n- `chart` — `{chart_type, encodings:{x,y,…}, config:{}}` (chart_type from the\n chart type reference).\n- `input_tables` — workspace table names, as listed in the available-tables\n context, that the code reads.\n- `field_metadata` — field → semantic annotation. Include units, index\n baselines, intrinsic domains, and ordinal order when supported by the data;\n never invent a unit. Distinguish percentages from percentage points and\n identifiers from quantities.\n- `field_display_names` — field → concise human-readable label for axes,\n legends, and table headers. Expand technical names, preserve established\n domain abbreviations, include units when useful, and use the user's language.\n\nSilently classify the analytical intent before choosing a chart: comparison,\ntrend, distribution, relationship, composition, deviation, ranking,\nuncertainty, or spatial pattern. Choose encodings and chart type from that\nintent and the data shape. Set ordering deliberately: chronological for time,\nsemantic order for ordinal fields, and measure order for rankings. Avoid line\ncharts or legends with excessive series, labels that collide, and color that\ndoes not encode additional information; aggregate, bin, facet, or limit\ncategories when needed without hiding material data.\n\n### `ask_user` — ask the user and pause for their reply (pauses the run)\n\nAsk the user something and pause for their input. Reach for this on **any** turn\nwhere you want a reply — a choice to make, a clarification you need before\nacting, or a brief statement paired with clickable follow-ups they can react to.\nPrefer it over ending your turn with a plain-text question: plain text ends the\nrun (the user's next message starts a fresh turn without this context), while\n`ask_user` keeps the conversation in the same turn.\n\n- `questions` — 1–3 items, each something the user **acts on**: a choice\n (`single_choice` with `options`) or an open question they type an answer to\n (`free_text`). Put your reasoning, rationale, and context in your reply text —\n **not** here. Never add a `questions` item that only states a rationale or\n explanation with nothing for the user to answer or click.\n- each question: `text` (wrap a **column** in `**…**`), `responseType`\n (`single_choice` when you offer `options`, else `free_text` — the user types\n their own open-ended answer, not a slot for your exposition), `required`\n (`true` when the run depends on the answer, `false` for an optional follow-up),\n and `options` (plain-text choices, **at most 3** — just the most likely\n answers; the user can always type a freeform reply, so don't enumerate every\n case).\n\nThis is **terminal**: the run pauses after it and resumes when the user replies.\n\n## Choosing what to do\n\nMatch the response depth to the user's request. Create charts that materially\ncontribute to the answer, and stop when the answer is sufficient.\n\n- For conceptual or informational questions, answer directly when a chart would\n not improve the answer.\n- For specific analytical questions, create the view or views needed to answer\n them clearly.\n- For diagnostic or exploratory questions, follow relevant findings across\n multiple views when doing so adds meaningful insight.\n- If essential intent is unclear, use `ask_user` rather than guessing.\n- *Missing data* (needs tables not in the workspace):\n `load_skill(\"data-loading\")`, discover the source, and propose immutable\n loading options inline.\n- *Report / write-up request* (e.g. \"write a report on X\", \"summarize the findings\n as a narrative\"): this needs the **report** skill — `load_skill(\"report\")` and\n follow it to commit the `write_report` action. **Do this as your very first\n move when charts already exist** (see `[AVAILABLE CHARTS]` / the thread): don't\n re-create them — load the report skill straight away and embed the existing\n charts by id. Only produce a new chart first if the report genuinely needs one\n that isn't there yet (0–3, judgment-based), then load the skill.\n\nFollow explicit requests about scope, depth, and format. **Never** repeat a\nvisualization already in the trajectory or in another thread.\n\n## Chart Creation Guide\n\nThe following reference material applies when you call the `visualize` tool.\n\n### A. Code Execution Rules\n\n**About the execution environment:**\n- You can use BOTH DuckDB SQL and pandas operations in the same script\n- The script will run in the workspace data directory (all data files are in the current directory)\n- Each table in [CONTEXT] has a **file path** (e.g., `student_exam.parquet`, `sales.csv`). Use EXACTLY that path to load data:\n - `.parquet`: `pd.read_parquet('file.parquet')` or DuckDB `read_parquet('file.parquet')`\n - `.csv`: `pd.read_csv('file.csv')` or DuckDB `read_csv_auto('file.csv')`\n - `.json`: `pd.read_json('file.json')`\n - `.xlsx`/`.xls`: `pd.read_excel('file.xlsx')`\n - `.txt`: `pd.read_csv('file.txt', sep='\\t')`\n- **IMPORTANT:** Use the exact filename from the context — do NOT change the file extension or assume all files are parquet.\n- **Allowed libraries:** pandas, numpy, duckdb, math, datetime, json, statistics, collections, re, sklearn, scipy, random, itertools, functools, operator, time\n- **Not allowed:** matplotlib, plotly, seaborn, requests, subprocess, os, sys, io, or any other library not listed above.\n- File system access (open, write) and network access are also forbidden.\n\n**When to use DuckDB vs pandas:**\n- **Prefer plain pandas** for most tasks — it's simpler and more readable.\n- Only use DuckDB when the dataset is very large and you need efficient SQL aggregations, filtering, joins, or window functions.\n- You can combine both: DuckDB for initial loading/filtering on large files, then pandas for complex operations.\n\n**Code structure:** standalone script (no function wrapper), imports at top. **CRITICAL:** The final result DataFrame MUST be assigned to the exact variable name you specified in `\"output_variable\"` — the system uses this name to extract the result. For example, if your output_variable is `sales_by_region`, the script must contain `sales_by_region = ...`.\n\n**DuckDB notes:**\n- Escape single quotes with '' (not \\')\n- No Unicode escapes (\\u0400); use character ranges directly: [а-яА-Я]\n- Cast date columns explicitly: `CAST(col AS DATE)`, `CAST(col AS TIMESTAMP)`\n- For complex datetime operations, load data first then use pandas datetime functions\n- Critical identifier quoting rule:\n * If a table/column name contains non-ASCII characters (e.g., Chinese, Japanese, Korean, Cyrillic, etc.), spaces, or punctuation,\n you MUST wrap it in double quotes, e.g. SELECT \"金额\" FROM \"客户表\".\n * Never output placeholder identifiers like your_table_name, your_column, your_condition.\n\n**Datetime handling:**\n- `date` columns contain date-only values (YYYY-MM-DD). `datetime` columns contain date+time (ISO 8601).\n- `time` columns contain time-only values (HH:mm:ss). `duration` columns are time intervals.\n- Year → number. Year-month / year-month-day → string (\"2020-01\" / \"2020-01-01\").\n- Hour alone → number. Hour:min or h:m:s → string. Never return raw datetime objects.\n\n### B. Chart Type Reference\n\nThe `chart_type` value in the `visualize` action MUST be one of the names listed\nbelow (exact spelling, including capitalization). When a row lists multiple\nnames, pick whichever fits the \"when to use\" hint best.\n\n**Choosing a chart — prefer simple, escalate when it fits.** Reach for the\n**Everyday** set first: it answers most questions and is the safest, most\nlegible choice. But when the data or question genuinely fits a **Specialized**\ntype (a distribution's shape, a cumulative curve, a rank race, a before→after,\na geographic pattern…), prefer it — a well-matched specialized chart is more\ninsightful than forcing a generic one. Don't pick a specialized type for\nnovelty; use it because its \"when to use\" condition is met.\n\n**Everyday — reach for these first**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Scatter Plot | x, y, color, size, facet | opacity (0.1–1.0) | Relationships between two quantitative fields |\n| Regression | x, y, color, size, facet | regressionMethod (\"linear\",\"log\",\"exp\",\"pow\",\"quad\",\"poly\"), polyOrder (2–10) | Trend line over scatter; one line per color group |\n| Bar Chart / Stacked Bar Chart / Lollipop Chart / Waterfall Chart | x, y, color, facet | — | Bar: categorical comparison (auto-stacks when color is set). Stacked Bar: explicit stacked totals, color = the stack. Lollipop: cleaner for ranked lists / sparse categories. Waterfall: cumulative gain/loss, each bar starts where the previous ended |\n| Grouped Bar Chart | x, y, group, facet | — | Side-by-side bars across a second categorical dimension |\n| Line Chart | x, y, color, strokeDash, facet | interpolate (\"linear\",\"monotone\",\"step\") | Trends over an ordered (usually temporal) x-axis |\n| Area Chart | x, y, color, facet | — | Magnitude over ordered x; auto-stacks when color is set |\n| Histogram / Density Plot | x, color, facet | — | Distribution of one quantitative field. Histogram: discrete bins, auto-binned. Density Plot: smooth KDE curve |\n| Boxplot | x, y, color, facet | — | Distribution summary (median/quartiles/outliers) by category |\n| Pie Chart | size, color, facet | innerRadius (0–100; 0=pie, >0=donut) | Part-of-whole with ≤7 categories. Wedge value goes on **size**, not **theta** |\n| Heatmap | x, y, color, facet | colorScheme — sequential (\"viridis\",\"blues\",\"reds\",\"oranges\",\"greens\") or diverging (\"blueorange\",\"redblue\") | Matrix / 2D density; color encodes the quantitative cell value |\n\n**Specialized — use when the data/question fits the \"when to use\"**\n\n| chart_type | encodings | config | when to use |\n|---|---|---|---|\n| Connected Scatter Plot | x, y, order, color, facet | — | Two quantitative fields traced in sequence — needs an `order` field (e.g. time) so points are joined in order, not by x |\n| Ranged Dot Plot | x, y, color, facet | — | Min–max range or two-point comparison per category |\n| Violin Plot | x, y, color, facet | — | Distribution SHAPE (KDE silhouette) by category; better than a boxplot when data is multimodal. x = category, y = value |\n| Strip Plot | x, y, color, size, facet | — | Every individual point by category (jittered); good for small/medium n where raw values matter, not just a summary |\n| ECDF Plot | x, color, facet | — | Cumulative distribution of one quantitative field. Pass the RAW field on x (do NOT pre-compute the CDF); color for per-group curves |\n| Bump Chart | x, y, color, facet | — | How RANKINGS change over ordered x; y = rank, color = entity (long-form: one row per entity × x) |\n| Slope Chart | x, y, color, facet | — | Change between exactly TWO points (before → after) per entity; x = the two labels, y = value, color = entity |\n| Streamgraph | x, y, color, facet | — | Several series' magnitude over ordered x, stacked around a center baseline (color = series) — theme/volume shifts over time |\n| Range Area Chart | x, y, y2, color, facet | — | A shaded band between a lower (y) and upper (y2) bound over ordered x — e.g. min–max or a confidence interval |\n| Rose Chart | x, y, color, facet | — | Cyclical/categorical magnitude as angular wedges (polar bars); x = category/angle, y = value |\n| Pyramid Chart | x, y, color, facet | — | Back-to-back bars split by a binary group (e.g. population by age × sex); y = category, x = value, color = the two-sided group |\n| Radar Chart | x, y, color, facet | — | Multi-metric profile/comparison; x = metric name, y = value, color = entity (long-form data) |\n| Bar Table | x, y, color, facet | — | Ranked horizontal table with inline bars; one row per category. y = category, x = value |\n| KPI Card | metric, value, goal | — | \"Big number\" dashboard tile(s); one row per tile. `value` must be pre-aggregated; `goal` is optional |\n| Candlestick Chart | x, open, high, low, close, facet | — | OHLC financial data |\n| Map | longitude, latitude, color, size | projection (\"mercator\",\"equalEarth\",\"naturalEarth1\",\"orthographic\",\"albersUsa\"), projectionCenter ([lon,lat]) | Geographic POINTS/bubbles by lon/lat (use projection \"albersUsa\" for a US-only map) |\n| Choropleth | id, color, facet | region (\"world\",\"usa\",…) | Filled REGIONS shaded by value; `id` = the region key (country/state name or code), color = the quantitative value |\n\n**Critical chart rules:**\n- **Scatter Plot**: use config opacity (0.1–1.0) for dense data instead of encoding opacity.\n- **Regression**: trend line is automatic — do NOT compute regression coefficients/predictions in Python. Use `color` to get separate trend lines per group.\n- **Bar Chart**: x=categorical, y=quantitative (vertical bars). Swap x↔y for horizontal bars. Same-x rows are auto-stacked when `color` is set.\n- **Grouped Bar Chart**: use the `group` channel (not `color`) for side-by-side bars.\n- **Histogram**: do NOT pre-bin in Python — pass the raw quantitative field on `x` and the chart bins automatically. Pre-aggregating gives wrong bin widths.\n- **Line Chart**: use `strokeDash` to differentiate line styles (e.g. actual vs forecast).\n- **Pie Chart**: use the `size` channel (not `theta`) for wedge values. Avoid when >7–8 categories.\n- **Radar Chart**: data must be long-form — one row per (entity, metric, value). If your data is wide-form (one column per metric), melt it first in the Python step.\n- **Heatmap**: pick `colorScheme` by the meaning of the values. Use a **sequential** scheme (viridis/blues/reds/oranges/greens) for single-direction magnitudes (counts, rates, prices, scores — higher is simply more). Use a **diverging** scheme (blueorange/redblue) ONLY when the values have a meaningful center to read away from (e.g. profit/loss around 0, change vs. a baseline, temperature around freezing).\n- **Bar Table**: y is the category column to rank; x is the quantitative value driving bar length. Don't sort in Python — the template sorts.\n- **KPI Card**: channels are `metric`, `value`, `goal` (not x/y). One DataFrame row = one tile. The `value` column must already contain the final number to display (aggregate upstream in the Python step).\n- **Candlestick Chart**: requires `open`, `high`, `low`, `close` columns.\n- **Connected Scatter Plot**: provide an `order` field (usually time) so points are joined in sequence, not by x-order.\n- **ECDF Plot**: pass the RAW quantitative field on `x` — the chart computes the cumulative curve; do NOT pre-compute it in Python.\n- **Range Area Chart**: `y` is the lower bound and `y2` the upper bound of the band.\n- **Bump / Slope Chart**: long-form data — one row per (entity, x); `color` is the entity. Slope's `x` has exactly two categories (before/after).\n- **Violin Plot**: like Boxplot but shows the full distribution shape; x = category, y = value.\n- **Map / Choropleth**: `Map` plots points via `longitude` / `latitude` (set projection `\"albersUsa\"` for the US); `Choropleth` fills regions — put the region key on `id` and the value on `color`, not `x` / `y`.\n- **facet**: available for nearly all chart types; use a low-cardinality categorical field.\n- All fields in `encodings` must also appear in `output_fields`. Typically use 2–3 channels (x, y, color/size).\n\n### C. Semantic Type Reference\n\nChoose the most specific type that fits. Only annotate fields used in chart encodings.\n\n| Category | Types |\n|---|---|\n| Temporal | DateTime, Date, Time, Timestamp, Year, Quarter, Month, Week, Day, Hour, YearMonth, YearQuarter, YearWeek, Decade, Duration |\n| Monetary measures | Amount, Price |\n| Physical measures | Quantity, Temperature |\n| Proportion | Percentage |\n| Signed/diverging | Profit, PercentageChange, Sentiment, Correlation |\n| Generic measures | Count, Number |\n| Discrete numeric | Rank, Score |\n| Identifier | ID |\n| Geographic | Latitude, Longitude, Country, State, City, Region, Address, ZipCode |\n| Entity names | Category, Name |\n| Coded categorical | Status, Boolean, Direction |\n| Binned ranges | Range |\n| Fallback | Unknown |\n\nKey guidelines:\n- Use **Amount** for summed monetary totals, **Price** for per-unit prices, **Profit** for values that can be negative.\n- Use **Temperature** (not Quantity) for temperature — it has special diverging behavior.\n- Use **Year** (not Number) for columns like \"year\" with values 2020, 2021.\n\n### D. Statistical Analysis Guide\n\n- **Regression**: use chart_type \"Regression\" — the trend line is automatic, do NOT compute regression values in Python code. Configure method via `{\"regressionMethod\": \"linear\"}` (options: \"linear\", \"log\", \"exp\", \"pow\", \"quad\", \"poly\"; for poly add `{\"polyOrder\": 3}`).\n- **Forecasting**: compute predicted future values in Python. Use Line Chart with strokeDash to distinguish actual vs forecast, and color for series grouping.\n- **Clustering**: compute cluster assignments in Python. Output [x, y, cluster_id]. Use Scatter Plot with color → cluster_id.\n\n\n[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"execute_python_script\",\n \"description\": \"Execute a general-purpose Python script in the sandbox. Here you use it to inspect data, compute statistics, transform tables, or verify assumptions before you act — write results to stdout with print() and that output is returned to you (it is NOT shown to the user). The script is for your own analysis, not for producing the final visualization. pandas, numpy, duckdb, sklearn, scipy are available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"purpose\": {\n \"type\": \"string\",\n \"description\": \"One-sentence description of what this script does and why (shown to user as progress).\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python script to execute. Use print() to surface output.\"\n }\n },\n \"required\": [\"purpose\", \"code\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_source_data\",\n \"description\": \"Get a detailed summary of one or more source tables — schema, field-level statistics, and sample rows. Cheaper than execute_python_script for basic data inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"table_names\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of workspace table names, as listed in the available-tables context, to inspect.\"\n }\n },\n \"required\": [\"table_names\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"visualize\",\n \"description\": \"Commit a data transform + chart: run code producing a DataFrame and render it. The agent observes the result and continues.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\": \"A concise, neutral analytical heading that names the subject, measure, and analytical lens, such as 'Year-over-year price change peaks'. Prefer a stable description of the view over a takeaway claim or narrated trend. Do not mention the chart type, imply causality, or editorialize. Shown as the chart heading.\"\n },\n \"subtitle\": {\n \"type\": \"string\",\n \"description\": \"Concise supporting context not already clear from the title or axes. Use one phrase of at most 16 words to provide contextual details. Do not restate the measure or analytical lens named in the title.\"\n },\n \"display_instruction\": {\n \"type\": \"string\",\n \"description\": \"≤12 words. State the question or hypothesis the chart investigates — don't recap the chart spec (x/y/color/split are already visible). Wrap a **column** in ** ** if it anchors the question.\"\n },\n \"input_tables\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Workspace table names, as listed in the available-tables context, that the code reads.\"\n },\n \"code\": {\n \"type\": \"string\",\n \"description\": \"Python code producing a DataFrame assigned to output_variable.\"\n },\n \"output_variable\": {\n \"type\": \"string\",\n \"description\": \"snake_case name of the DataFrame variable the code assigns.\"\n },\n \"chart\": {\n \"type\": \"object\",\n \"description\": \"Chart spec: {chart_type, encodings:{x,y,...}, config:{}}. chart_type from the chart type reference.\"\n },\n \"field_metadata\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> SemanticType for the output columns.\"\n },\n \"field_display_names\": {\n \"type\": \"object\",\n \"description\": \"Map of field name -> human-readable display name for chart axes and table headers.\"\n }\n },\n \"required\": [\"title\", \"code\", \"output_variable\", \"chart\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"ask_user\",\n \"description\": \"Ask the user something and pause for their reply — the run resumes in the same turn with their answer in context. Use this for ANY turn where you want the user to respond: a choice to make, a clarification you need before acting, or a brief statement paired with clickable follow-ups. Put your reasoning, rationale, and context in your normal reply text, not inside this call. Prefer this over ending your turn with a plain-text question: plain text ends the run and the user's next message starts a fresh turn without this context, whereas ask_user keeps the conversation going. Reserve plain text (no action) for your final answer when you expect nothing further.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"thought\": {\n \"type\": \"string\",\n \"description\": \"Brief rationale (not shown to the user).\"\n },\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"1–3 things the user acts on: a choice (single_choice with options) or an open question they type an answer to (free_text). Put rationale, reasoning, and context in your reply text, not here — never add an item that only states an explanation with nothing for the user to answer or click. An explanation is allowed only as a short statement paired with clickable chart-producing follow-ups (required=false with options).\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The question, or (for an optional follow-up) a short statement. Keep a statement to 1–3 grounded sentences and pair it with clickable follow-up options. Wrap a **column** in ** ** to highlight it.\"\n },\n \"responseType\": {\n \"type\": \"string\",\n \"enum\": [\"single_choice\", \"free_text\"],\n \"description\": \"single_choice when you offer options; free_text when the user types their own open-ended answer (not a slot for your own exposition).\"\n },\n \"required\": {\n \"type\": \"boolean\",\n \"description\": \"false for an explanation / optional follow-up; true for a clarification the run depends on.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"Plain-text choices, at most 3. Keep them to the few most likely answers — the user can always type a freeform reply, so don't try to enumerate every case. For a clarification these are answers; for an explanation these are short chart-producing follow-up prompts the user might click next (≤8 words each, phrased as the user would say them).\"\n }\n },\n \"required\": [\"text\"]\n }\n }\n },\n \"required\": [\"questions\"]\n }\n }\n }\n]\n\n\n---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.\n\n\n[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]\n\n\n---\nname: data-loading\ndescription: >-\n Discover connected data sources, add new data connectors through a\n user-confirmed form, inspect table metadata, and run bounded read-only probes\n when the current workspace data is insufficient.\nwhen_to_use: >-\n The user's question needs data that is not already available as a workspace\n input, the user asks what connected data is available, or the user wants to\n connect a database, warehouse, or cloud source. Not for analyzing tables\n already listed in the workspace context.\nalways_on: false\ntools:\n - list_data\n - find_data\n - describe_data\n - probe_data\n - list_connectors\n - describe_connector\nactions:\n - propose_data_operation\n - propose_connection\n---\n\n# Skill: Data discovery\n\nThe workspace tables listed in your context are the data already loaded into the\nsystem, and the only data that can be read directly. Everything these tools\nreturn is *not* loaded yet — it lives in a connected source and only becomes\nusable after the user selects a loading option and the server materializes it.\n\nUse these tools to determine whether connected sources contain data needed for\nthe user's goal. They are read-only: discovering, describing, or probing a\nsource does not add anything to the workspace analysis inputs.\n\n## Adding a connector\n\nWhen the user wants to connect a new source, do not merely ask them to navigate\nto settings and do not attempt to connect on their behalf.\n\n1. Call `list_connectors` first because available built-ins and plugins vary by\n deployment. For a broad request such as \"help me connect\", summarize the\n concrete available types and ask which one they use.\n2. Once the source type is known, call `describe_connector` when field or auth\n details are useful.\n3. **When the requested source type is known and available, you MUST call\n `propose_connection` in this same turn.** Do not stop with text such as\n \"I'll open the form\", \"you'll need to provide\", or a list of required\n fields. Only the action opens the form. Include one or two helpful sentences\n alongside the action call explaining what the user should review or supply;\n this text appears above the chat while the form opens on the canvas. Pass\n `prefilled` values the user already supplied, including values parsed from a\n connection string or config snippet. Never invent missing values.\n4. The form is only a proposal. The user reviews it and clicks Connect; the\n action must never connect automatically.\n\nPrefilled values may include credentials the user deliberately supplied. Do not\nrepeat those values in prose or subsequent tool output. They are transient form\nseeds and are removed from persisted UI state.\n\n## Discovery sequence\n\n1. Use `find_data` when the user names a business concept or table. Use\n `list_data` when you need to browse available sources or hierarchy.\n2. Use `describe_data` before relying on columns, types, row counts, or filter\n values. Pass the exact `source_id` and `table_key` returned by discovery.\n3. Use `probe_data` only when metadata is insufficient to choose a useful\n bounded result. Probes are limited, read-only, and may be approximate.\n4. First reconcile discoveries with every table in `[PRIMARY TABLE(S)]`,\n `[OTHER AVAILABLE TABLES]`, or `[AVAILABLE TABLES]`. If the needed data is\n already loaded, use or explain that workspace table instead of proposing it.\n5. When there are genuinely missing useful alternatives, call\n `propose_data_operation` with one\n to three complete immutable plans. This pauses for the user's choice; it\n does not load data yet.\n\n## Proposing loading options\n\nWrite your answer as **message text alongside the call** — that prose is what\nthe user reads, so it carries the whole answer. Do not put it in an action\nfield, and do not leave the call bare. Say what you went looking for, what you\nactually found, and what each option would give them — enough that they can\nchoose without opening a single preview. Two to four sentences; more when the\noptions differ in ways that matter (grain, coverage, freshness, joins needed),\nfewer when the choice is obvious. Name real tables and columns you saw during\ndiscovery, and say plainly when an option is a compromise or when you'd pick one\nyourself. Write it as you'd say it to a colleague, not as a schema summary.\n\n- Each `option` is a complete alternative: a concise action label (2–6 words)\n and one or more tables. The labels are buttons, not sentences — the\n reasoning belongs in your message text. The application displays table\n previews separately, so don't list columns as a substitute for explaining.\n- Use only source IDs, table keys, columns, and values grounded by discovery.\n- For a whole table, omit `query`. Use the optional raw-row query only when the\n request needs filters, projection, ordering, or an intentional limit. It uses\n the same `filters` / `columns` / `order_by` / `limit` vocabulary as\n `probe_data`, without aggregation.\n- Do not invent operation IDs, plan IDs, or hashes. The server creates them.\n- Never propose an exact connector query already represented by a workspace\n table. The server also enforces this using persisted load provenance.\n\n## Grounding rules\n\n- Never invent source IDs, table keys, columns, or category values.\n- Prefer cached catalog discovery before a live probe.\n- Treat probe rows as evidence for planning, not as analysis input data.\n- Keep queries structured and bounded. Do not generate source-specific SQL.\n- If a source is unavailable or permissions changed, report the tool result and\n ask the user for the needed connection or choose another source.\n\n\n[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_data\",\n \"description\": \"Browse cached connected-source catalogs. With no arguments, list source summaries. With source_id, list its top-level entries. Add path to browse direct children and filter for a case-insensitive substring match.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\", \"description\": \"Connected source identifier. Omit for source summaries.\" },\n \"path\": { \"type\": \"array\", \"items\": { \"type\": \"string\" }, \"description\": \"Hierarchy path segments.\" },\n \"filter\": { \"type\": \"string\", \"description\": \"Substring filter on direct children.\" }\n },\n \"required\": []\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"find_data\",\n \"description\": \"Regex search across cached connected-source catalogs and optionally existing workspace tables. Returns exact source_id and table_key values for follow-up inspection.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": { \"type\": \"string\", \"description\": \"Case-insensitive regex. Plain keywords work as literals.\" },\n \"scope\": { \"type\": \"string\", \"description\": \"all, workspace, connected, a source_id, or source_id:path/segments.\" },\n \"exclude\": { \"type\": \"string\", \"description\": \"Optional table-name exclusion regex.\" },\n \"fields\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\", \"enum\": [\"name\", \"description\", \"columns\"] },\n \"description\": \"Fields to search. Omit for all.\"\n },\n \"limit\": { \"type\": \"integer\" }\n },\n \"required\": [\"query\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_data\",\n \"description\": \"Read cached metadata, columns, types, description, and row count for one discovered table.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"probe_data\",\n \"description\": \"Run a bounded read-only structured query against one connected table. Use only after describe_data. Results are evidence for planning and do not become workspace inputs.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"group_by\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"aggregates\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"op\": { \"type\": \"string\", \"enum\": [\"count\", \"count_distinct\", \"sum\", \"avg\", \"min\", \"max\"] },\n \"column\": { \"type\": \"string\" },\n \"as\": { \"type\": \"string\" }\n },\n \"required\": [\"op\"]\n }\n },\n \"order_by\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\" }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"list_connectors\",\n \"description\": \"List connector types available in this deployment. Call this before propose_connection because built-ins, plugins, and missing dependencies vary by deployment. If the user's requested type is present, you MUST call propose_connection in the same turn; do not merely say you will open a form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {}\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"describe_connector\",\n \"description\": \"Return setup fields and authentication choices for one source_type returned by list_connectors. After this, call propose_connection in the same turn; describing fields does not open the form.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_connection\",\n \"description\": \"REQUIRED terminal action when the user wants an available connector and its source_type is known. This is the only operation that opens the user-confirmed add-connector form on the canvas. Call list_connectors first. Prefill only values the user supplied; never invent credentials or connect automatically.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_type\": { \"type\": \"string\", \"description\": \"Connector type key returned by list_connectors.\" },\n \"prefilled\": {\n \"type\": \"object\",\n \"description\": \"Optional connector field values already supplied by the user. Values seed the live form and must not be repeated in prose.\",\n \"additionalProperties\": {}\n }\n },\n \"required\": [\"source_type\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"propose_data_operation\",\n \"description\": \"Offer one to three complete immutable connected-data loading alternatives and pause for the user's selection. Discovery must ground every source, table, filter, and sort field. This does not execute a load.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"response\": {\n \"type\": \"string\",\n \"description\": \"Fallback only. Leave empty when you narrate in your message text, which is what the user reads.\"\n },\n \"options\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 3,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"label\": {\n \"type\": \"string\",\n \"description\": \"Concise action label, ideally 2-6 words.\"\n },\n \"tables\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"source_id\": { \"type\": \"string\" },\n \"table_key\": { \"type\": \"string\" },\n \"query\": {\n \"type\": \"object\",\n \"description\": \"Optional raw-row subset. Omit to load the whole table subject to server limits.\",\n \"properties\": {\n \"filters\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"op\": { \"type\": \"string\", \"enum\": [\"EQ\", \"NEQ\", \"GT\", \"GTE\", \"LT\", \"LTE\", \"IN\", \"ILIKE\", \"BETWEEN\", \"IS_NULL\"] },\n \"value\": {}\n },\n \"required\": [\"column\", \"op\"]\n }\n },\n \"columns\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } },\n \"order_by\": {\n \"type\": \"array\",\n \"maxItems\": 1,\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"column\": { \"type\": \"string\" },\n \"dir\": { \"type\": \"string\", \"enum\": [\"asc\", \"desc\"] }\n },\n \"required\": [\"column\"]\n }\n },\n \"limit\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n },\n \"required\": [\"source_id\", \"table_key\"]\n }\n }\n },\n \"required\": [\"label\", \"tables\"]\n }\n }\n },\n \"required\": [\"options\"]\n }\n }\n }\n]\n\n\n---\nname: report\ndescription: >-\n Turn an exploration (threads, findings, charts) into a single Markdown\n report — note, blog post, executive summary, KPI dashboard, slide brief, or\n multi-section analytical report, with embedded charts.\nwhen_to_use: >-\n The user asks to write up / summarize / report on what they explored, or\n wants a shareable narrative document built from the charts and findings in\n the data thread. Not for producing a single new chart (use visualize).\nalways_on: false\ntools:\n - inspect_chart\nactions:\n - write_report\n---\n\n# Skill: Report writing\n\nYou are a data journalist / analyst who creates insightful, well-organized\nreports based on data explorations. The output is a single Markdown document\nthat may play many roles — short note, blog post, executive summary, dashboard,\nmulti-section report, FAQ, slide-style brief, etc. Adapt structure and length to\nwhat the user actually asks for; do not force a fixed template.\n\n## Emitting the report (the `write_report` action)\n\nFirst inspect whatever charts and data you need (see below), then write the\nentire report and commit it by **calling the `write_report` tool** — it is the\ncommitting action that ends this turn. Its `report` argument carries the\n**full Markdown** of the finished report:\n\n- `report` — the complete report in Markdown: headings, prose, tables, and\n embedded charts via `![caption](chart://chart_id)`.\n\nProduce any charts the report needs **before** calling `write_report`, and do\nall chart/data inspection first — once you call `write_report`, the report is\ndelivered as-is and the run ends.\n\n## Context available to you\n- **[PRIMARY TABLE(S)]** / **[OTHER AVAILABLE TABLES]**: Lightweight schema of datasets.\n- **[FOCUSED THREAD]** (optional): The exploration thread the user is continuing —\n the ordered steps with the user's questions, the agent's thinking, and the\n findings at each step. This is the spine of the story you are telling.\n- **[OTHER THREADS]** (optional): Brief per-step summaries of other exploration\n threads the user ran. These are additional findings worth weaving in.\n- **[AVAILABLE CHARTS]**: List of charts with their type, encodings, and table references.\n\n## Ground the report in the exploration\nThe thread context is your most important input. The user already did real\nanalysis — your job is to turn that journey into a coherent narrative, not to\nsummarize a single chart. Before writing:\n- Read the FOCUSED THREAD and OTHER THREADS to understand the full set of\n questions asked and findings reached.\n- Plan a report that covers the meaningful findings across the exploration,\n not just the last or most obvious chart.\n\n## Inspecting charts and data\nYou have two inspection tools available the whole time: `inspect_chart` and\n`inspect_source_data`. Use them on your own whenever you need to verify a detail\nbefore writing about it — a chart's exact numbers, its data, or a table's\nschema. `inspect_chart` lets you *read* a chart from its encodings, a data\nsample, and the code that produced it (and points you to the backing table so\nyou can interrogate the full data with `execute_python_script`); a rendered\nimage is included only when one is available. Read the charts behind the key\nfindings you present **before** you compose the report.\n\n## Write the report\nWrite the complete report in Markdown and pass it as the `report` argument of the\n`write_report` tool. Do all your inspecting first, then compose the whole\ndocument and make the one `write_report` call.\n\n### Embedding charts (REQUIRED FORMAT — do not change this)\nTo embed a chart image, use markdown image syntax with a `chart://` URL:\n ![Caption describing the chart](chart://chart_id)\n\nExample: `![Monthly trade balance trend](chart://chart-123)`\n\nThe chart_id must match one from [AVAILABLE CHARTS]. Place each chart embed on\nits own line (it renders as a block). You can embed the same chart at most\nonce. Captions are short — one line describing what the chart shows.\n\n### Tables\nFor data tables, write standard markdown tables directly:\n| date | value |\n| --- | --- |\n| 2020-01 | -43.5 |\n\n### Style & structure — adapt to the user's request\nThe user may ask for any of:\n- a short note or social-style summary (a few sentences, one or two charts),\n- a blog post / narrative report (intro → findings → takeaway),\n- an executive summary (key numbers up top, then context),\n- a KPI dashboard / multi-section overview (headings per topic, multiple charts\n arranged with short commentary between them),\n- a slide-style brief (compact sections with bullet points and embedded charts),\n- a deeper analytical report with sub-sections, methodology notes, and caveats.\n\nPick the structure that fits the request and the available material. Match the\nbreadth of the report to the breadth of the exploration: if the user explored\nseveral questions, the report should reflect that — don't collapse a rich\nexploration into a single-chart blurb unless the user explicitly asked for\nsomething that short. Reasonable defaults if the user is vague:\n- Start with a `# Title` that reflects the topic.\n- Group related findings under `##` (and `###` if useful) headings, typically\n one section per key finding / thread.\n- Around each embedded chart, briefly explain what it shows and the key insight.\n- Use bullets / short paragraphs / tables where they help; don't pad.\n- Close with a brief takeaway or summary section if the report is more than a\n few paragraphs. For very short outputs (notes, single-chart blurbs), a closing\n summary is optional.\n\n### Guardrails\n- Write in Markdown. Keep prose tight; let the data and charts carry the weight.\n- Stay faithful to the data — do not invent numbers, comparisons, or causation\n that the data does not actually support.\n- It is fine to flag uncertainty (\"based on the sample shown…\") when appropriate.\n- Embed every chart you discuss; don't reference a chart in prose without showing it.\n\n\n[\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"inspect_chart\",\n \"description\": \"Read one or more charts. Returns each chart's encodings, a sample of its data, and the transformation code that created it so you can reason about what it shows (use execute_python_script on the backing table for the full data). A rendered PNG is included only when one is available.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"chart_ids\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" },\n \"description\": \"List of chart IDs from [AVAILABLE CHARTS] to inspect.\"\n }\n },\n \"required\": [\"chart_ids\"]\n }\n }\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"write_report\",\n \"description\": \"Deliver a Markdown report and end the run. `report` is the full report text (embed charts with ![caption](chart://chart_id)).\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"report\": {\n \"type\": \"string\",\n \"description\": \"The full Markdown report text. Embed charts with ![caption](chart://chart_id) referencing IDs from [AVAILABLE CHARTS].\"\n }\n },\n \"required\": [\"report\"]\n }\n }\n }\n]\n\n\n# Data Formulator — Data Loader Plugins\n\nDrop-in Python files that add new data-source connectors to Data\nFormulator without modifying its source code. If the built-in\nconnectors don't cover your data source (an internal warehouse, a SaaS\nAPI, a niche database), write a small plugin and DF will pick it up on\nthe next restart.\n\nThis folder contains **example plugins**. Treat them as templates: copy\none, rename it, and adapt the body.\n\n---\n\n## Quick start (3 steps)\n\n1. **Find your plugin directory.** It lives under your Data Formulator\n home dir:\n\n ```\n $DATA_FORMULATOR_HOME/plugins/\n ```\n\n If `DATA_FORMULATOR_HOME` is not set, DF defaults to\n `~/.data_formulator/`, so the plugin dir is `~/.data_formulator/plugins/`.\n\n Power users can point somewhere else with `DF_PLUGIN_DIR`\n (highest precedence) — useful for sharing one plugin folder across\n multiple DF installs.\n\n2. **Copy an example into it.** For instance:\n\n ```bash\n mkdir -p \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins\"\n cp examples/plugins/sqlite_data_loader.py \\\n \"${DATA_FORMULATOR_HOME:-$HOME/.data_formulator}/plugins/\"\n ```\n\n3. **Restart Data Formulator.** The new connector appears in the UI\n automatically. No registry edits, no rebuilds.\n\nTo verify it loaded, check the startup log for a line like:\n\n```\nINFO ... Plugin loader 'sqlite' registered from sqlite_data_loader.py\nINFO ... Plugin scan complete: 1 registered, 0 failed (dir=..., reason=WORKSPACE_BACKEND=local)\n```\n\n### Plugin directory resolution order\n\n| Precedence | Source | Default |\n| ---------- | ----------------------------------------- | ----------------------------- |\n| 1 | `DF_PLUGIN_DIR` env var (explicit override) | — |\n| 2 | `$DATA_FORMULATOR_HOME/plugins` | — |\n| 3 | Fallback | `~/.data_formulator/plugins/` |\n\n---\n\n## File-name contract\n\n| Filename | Registry key |\n| --------------------------------- | -------------- |\n| `sqlite_data_loader.py` | `sqlite` |\n| `acme_warehouse_data_loader.py` | `acme_warehouse` |\n| `notion_data_loader.py` | `notion` |\n\nRules:\n\n* The filename **must** end in `_data_loader.py`.\n* The prefix becomes the registry key — keep it lowercase, no spaces.\n* If the key matches a built-in (e.g. `mysql_data_loader.py`), the\n plugin **overrides** the built-in. Useful for hot-patching.\n\n---\n\n## What goes inside the file\n\nEach plugin defines exactly one class that subclasses\n[`ExternalDataLoader`](../../py-src/data_formulator/data_loader/external_data_loader.py).\nThe minimum surface area:\n\n```python\nfrom data_formulator.data_loader.external_data_loader import (\n ExternalDataLoader, MAX_IMPORT_ROWS,\n)\nimport pyarrow as pa\n\nclass MyLoader(ExternalDataLoader):\n\n # Optional: human-friendly UI label. Without this, the registry key\n # is title-cased (``\"my_warehouse\"`` → ``\"My Warehouse\"``). Override\n # to fix awkward casing (``\"SQLite\"``, ``\"BigQuery\"``).\n DISPLAY_NAME = \"My Warehouse\"\n\n @staticmethod\n def list_params() -> list[dict]:\n \"\"\"Declare connection-form fields. The UI auto-renders this.\"\"\"\n return [\n {\"name\": \"endpoint\", \"type\": \"string\", \"required\": True,\n \"tier\": \"connection\", \"description\": \"Server URL\"},\n {\"name\": \"token\", \"type\": \"string\", \"required\": True,\n \"tier\": \"auth\", \"sensitive\": True, \"description\": \"API token\"},\n ]\n\n @staticmethod\n def auth_instructions() -> str:\n \"\"\"Markdown help text shown next to the form.\"\"\"\n return \"Get your API token from https://example.com/settings/tokens\"\n\n def __init__(self, params: dict):\n self.params = params\n # validate + open connection here\n\n def list_tables(self, table_filter: str | None = None) -> list[dict]:\n \"\"\"Return catalog: [{name, metadata: {columns, row_count}}, ...]\"\"\"\n ...\n\n def fetch_data_as_arrow(self, source_table: str,\n import_options: dict | None = None) -> pa.Table:\n \"\"\"Read rows. Honour import_options['size'] up to MAX_IMPORT_ROWS.\"\"\"\n ...\n```\n\nLook at [`sqlite_data_loader.py`](sqlite_data_loader.py) for a runnable\nimplementation (~170 lines, stdlib only).\n\n### `list_params()` field reference\n\n| Key | Meaning |\n| ------------- | ---------------------------------------------------------------- |\n| `name` | Parameter key passed into `__init__(params)` |\n| `type` | `\"string\"`, `\"int\"`, `\"bool\"`, `\"password\"` |\n| `required` | If `True`, DF rejects connections that omit it |\n| `default` | Pre-filled value in the form |\n| `sensitive` | If `True`, value is redacted from logs / stored metadata |\n| `tier` | `\"auth\"`, `\"connection\"`, or `\"filter\"` — groups fields visually |\n| `description` | Help text shown under the field |\n\n---\n\n## Security gate — please read\n\nPlugins execute arbitrary Python in the server process. To prevent\naccidental code execution in shared deployments, the plugin scanner is\n**enabled only in single-user local mode**:\n\n* `WORKSPACE_BACKEND` unset or `local` → scanner runs.\n* `WORKSPACE_BACKEND` is anything else → scanner is skipped silently.\n* To opt in for a hosted deployment, set `DF_ALLOW_PLUGINS=1`\n **and** make sure the plugin directory is writable only by trusted\n administrators.\n\n---\n\n## Debugging a plugin that won't load\n\nThe scanner reports every failure two ways:\n\n1. **In the log**, with a full traceback:\n\n ```\n WARNING Failed to load plugin foo_data_loader.py: ... ModuleNotFoundError ...\n ```\n\n2. **In `DISABLED_LOADERS`**, which the frontend surfaces as a greyed-out\n connector with the reason. Common reasons:\n\n | Message | Fix |\n | ------------------------------------------------------ | --- |\n | `missing dependency 'X' (pip install X)` | `uv pip install X` in DF's venv |\n | `no ExternalDataLoader subclass found` | Add a `class Foo(ExternalDataLoader)` to the file |\n | `could not create import spec` | File isn't valid Python or has wrong extension |\n | `: ` | Anything else raised during `import` — check the traceback |\n\nThe plugin scanner cleans `sys.modules` on failure, so simply restarting\nDF after editing the file picks up the fix — no manual cleanup needed.\n\n---\n\n## Adding third-party dependencies\n\nIf your plugin needs a package that isn't already in DF's environment:\n\n```bash\n# from the data-formulator repo root\nuv pip install \n```\n\nThe plugin file is loaded into the same Python process as DF, so\nanything importable from DF's venv is importable from your plugin.\n\n---\n\n## For AI coding agents\n\nWhen asked to **add a new data source** to a user's Data Formulator\ninstall, prefer writing a plugin file over modifying DF's source tree:\n\n1. Read [`sqlite_data_loader.py`](sqlite_data_loader.py) and the\n `ExternalDataLoader` ABC in\n `py-src/data_formulator/data_loader/external_data_loader.py` to\n understand the required methods.\n2. Create a new file `_data_loader.py` in this folder (or\n directly in `~/.data_formulator/plugins/`).\n3. Subclass `ExternalDataLoader`. Implement at minimum:\n `list_params`, `auth_instructions`, `__init__`, `list_tables`,\n `fetch_data_as_arrow`.\n4. Return data as a `pyarrow.Table` from `fetch_data_as_arrow` — do\n **not** convert to pandas in the hot path.\n5. Quote identifiers when building SQL (see `_quote_ident` in the\n SQLite example) to avoid injection vulnerabilities.\n6. Respect `import_options['size']` and cap at `MAX_IMPORT_ROWS`.\n7. If the source has credentials, mark those params `sensitive: True`\n and `tier: \"auth\"` so DF redacts them from stored metadata.\n8. Do not modify the user's data — open read-only connections where\n the source supports it.\n\nAfter writing the file, verify it loads with:\n\n```bash\nDF_PLUGIN_DIR= uv run python -c \\\n \"from data_formulator import data_loader as dl; \\\n print(dl.PLUGIN_LOADERS, dl.DISABLED_LOADERS)\"\n```\n"}