{"owner":"andrewyng","repo":"context-hub","hasSkills":true,"totalSkillsCount":62,"totalTokensCount":147169,"categories":["anthropic-skill","subagent-persona","project-spec","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["cli/skills/get-api-docs/SKILL.md","content/autogen/docs/package/python/references/agents.md","content/autogen/docs/package/python/references/group-chat.md","content/datahub/docs/sdk/python/references/assertions.md","content/datahub/docs/sdk/python/references/entities.md","content/datahub/docs/sdk/python/references/lineage.md","content/datahub/docs/sdk/python/references/search.md","content/django/docs/extensions/python/DOC.md","content/edubase/docs/api/references/permissions.md","content/edubase/docs/api/references/question-types.md","content/edubase/docs/api/references/quiz-sets.md","content/edubase/docs/api/references/scoring.md","content/edubase/docs/api/references/tagging.md","content/edubase/docs/api/references/users.md","content/edubase/docs/api/references/webhooks.md","content/flutter/skills/state-management/bloc-cubit/SKILL.md","content/flutter/skills/state-management/riverpod/SKILL.md","content/gemini/docs/live/javascript/references/advanced.md","content/gemini/docs/live/python/references/advanced.md","content/grpcio/docs/tools/python/DOC.md","content/landingai/skills/ade/document-extraction/SKILL.md","content/landingai/skills/ade/document-extraction/references/chunk-types.md","content/landingai/skills/ade/document-extraction/references/extraction-schemas.md","content/landingai/skills/ade/document-extraction/references/file-formats.md","content/landingai/skills/ade/document-extraction/references/troubleshooting.md","content/landingai/skills/ade/document-extraction/references/use-cases.md","content/landingai/skills/ade/document-workflows/SKILL.md","content/landingai/skills/ade/document-workflows/references/batch-processing.md","content/landingai/skills/ade/document-workflows/references/database-integration.md","content/landingai/skills/ade/document-workflows/references/rag-pipelines.md","content/landingai/skills/ade/document-workflows/references/schema-catalog.md","content/landingai/skills/ade/document-workflows/references/table-stitching.md","content/landingai/skills/ade/document-workflows/references/visualization.md","content/langchain/docs/agents/python/DOC.md","content/mypy/docs/extensions/python/DOC.md","content/olakai/skills/integrate/SKILL.md","content/olakai/skills/new-project/SKILL.md","content/playwright-community/skills/login-flows/SKILL.md","content/rails/docs/activerecord/references/associations.md","content/rails/docs/activerecord/references/migrations.md","content/rails/docs/activerecord/references/queries.md","content/rspec/docs/package/references/matchers.md","content/rspec/docs/package/references/mocks.md","content/snowflake/docs/cortex-code/references/commands.md","content/snowflake/docs/cortex-code/references/configuration.md","content/snowflake/docs/cortex-code/references/mcp-and-agents.md","content/snowflake/docs/cortex-code/references/snowflake-tools.md","content/sourceparts/docs/parts-mcp/python/references/tools.md","content/sourceparts/docs/parts-mcp/python/references/workflows.md","content/sourceparts/skills/electronics-sourcing/SKILL.md","content/tavily/skills/tavily-best-practices/SKILL.md","content/tavily/skills/tavily-best-practices/references/crawl.md","content/tavily/skills/tavily-best-practices/references/extract.md","content/tavily/skills/tavily-best-practices/references/integrations.md","content/tavily/skills/tavily-best-practices/references/research.md","content/tavily/skills/tavily-best-practices/references/sdk.md","content/tavily/skills/tavily-best-practices/references/search.md","content/vectara/docs/api/python/references/advanced-query.md","content/vectara/docs/api/python/references/agents.md","content/vectara/docs/api/python/references/data-management.md","docs/design.md","llms.txt"],"skills":{"cli/skills/get-api-docs/SKILL.md":"---\nname: get-api-docs\ndescription: >\n Use this skill to get documentation for third-party APIs, SDKs or libraries before writing\n code that uses them to ensure you have the latest, most accurate documentation. This is a\n better way to find documentation than doing web search. This includes when a user asks for\n tasks like \"use the OpenAI API\", \"call the Stripe API\", \"use the Anthropic SDK\", \"query Pinecone\",\n or any other time the user asks you to write code against an external service and you need\n current API reference. Fetch the docs with chub before answering, rather than relying on\n your pre-trained knowledge, which may be outdated because of recent changes to these APIs.\n Be sure to use this skill when the user asks for the latest docs, latest API behavior, or\n explicitly mentions chub or Context Hub. Ensure `chub` is available, run `chub --help`, then\n follow the instructions there.\n\n---\n\n# Get API Docs via chub\n\nWhen you need documentation for a library or API, fetch it with the `chub` CLI\nrather than guessing from training data. This gives you the current, correct API.\n\n## Step 1 - Ensure chub exists and get the latest instructions for using it\n\nRun\n\n```bash\nchub --help\n```\n\nIf this results in an error showing that chub does not exist, then install it first, and run it\nagain. If the environment has node, npm, network access, and package-manager permission,\ninstall it as follows:\n\n```bash\nnpm install -g @aisuite/chub\n```\n\n## Step 2 — Find the right documents\n\nIf there is a conflict, follow the instructions from the output of chub --help since that will\nbe the latest guidance. However, a key command for searching for the right documentation is chub search:\n\n```bash\nchub search \"\" --json\n```\n\nPick the best-matching `id` from the results (e.g. `openai/chat`, `anthropic/sdk`,\n`stripe/api`). If nothing matches, try some broader terms.\n\n## Step 3 — Fetch the docs\n\nAgain, follow the latest instructions from chub --help instead of this if there is a conflict. But a\nkey command for fetching the relevant docs is:\n\n```bash\nchub get --lang py # or --lang js, --lang ts\n```\n\nRemember to include the --lang parameter.\n\n## Step 4 — Use the docs and leave feedback\n\nRead the fetched content and use it to write accurate code or answer the question.\nDo not rely on memorized API shapes — use what the docs say.\n\nFinally, after completing the task, if you discovered something not in the doc — a gotcha,\nworkaround, version quirk, or project-specific detail — save it so future sessions\nstart smarter:\n\n```bash\nchub annotate \"Webhook verification requires raw body — do not parse before verifying\"\n```\n\nAnnotations are local, persist across sessions, and appear automatically on future\n`chub get` calls. Keep notes concise and actionable. Don't repeat what's already in\nthe doc.\n\nAdditionally, to help all developers, give feedback on the documentation using chub feedback, following\nthe instructions in chub --help . Do not include secrets, source code, private architecture details,\nor other sensitive information in feedback comments.\n","content/autogen/docs/package/python/references/agents.md":"# AG2 Agent API Reference\n\n## LLMConfig\n\nWraps model configuration for any supported provider. Always pass a dict:\n\n```python\nfrom autogen.llm_config import LLMConfig\n\n# OpenAI\nllm_config = LLMConfig({\"api_type\": \"openai\", \"model\": \"gpt-4.1-mini\", \"api_key\": \"your-api-key\"})\n\n# Google\nllm_config = LLMConfig({\"api_type\": \"google\", \"model\": \"gemini-3.1-flash-lite-preview\"})\n\n# Anthropic\nllm_config = LLMConfig({\"api_type\": \"anthropic\", \"model\": \"claude-sonnet-4-20250514\"})\n```\n\n| Key | Required | Description |\n|---|---|---|\n| `api_type` | Yes | Provider identifier: `\"openai\"`, `\"google\"`, `\"anthropic\"`, etc. |\n| `model` | Yes | Model name as the provider expects it |\n| `api_key` | No | API key; omit if set in environment variables |\n| `base_url` | No | Custom endpoint URL (for proxies or self-hosted models) |\n| `temperature` | No | Sampling temperature |\n| `max_tokens` | No | Maximum tokens in response |\n\n### Loading from JSON file\n\n```python\nllm_config = LLMConfig.from_json(\n path=\"OAI_CONFIG_LIST\",\n file_location=\".\",\n filter_dict={\"model\": [\"gpt-4.1-mini\"]},\n)\n```\n\nUse when a project stores multi-model credentials in an `OAI_CONFIG_LIST` file.\n\n### Using as context manager\n\n```python\nwith LLMConfig({\"api_type\": \"openai\", \"model\": \"gpt-4.1-mini\"}) as config:\n agent = ConversableAgent(name=\"assistant\", llm_config=config)\n```\n\nNote: Each provider requires its corresponding extra to be installed (e.g., `autogen[openai]`, `autogen[google]`, `autogen[anthropic]`). Without the extra, model calls will fail at runtime.\n\n## ConversableAgent\n\nThe base class for all AG2 agents. Every agent is a `ConversableAgent` or extends one.\n\n### Constructor\n\n```python\nfrom autogen import ConversableAgent\n\nagent = ConversableAgent(\n name=\"my_agent\",\n system_message=\"You are a helpful assistant.\",\n llm_config=llm_config,\n human_input_mode=\"NEVER\",\n code_execution_config=False,\n functions=[my_tool_fn],\n)\n```\n\n| Param | Type | Default | Description |\n|---|---|---|---|\n| `name` | `str` | required | Unique agent identifier |\n| `system_message` | `str \\| None` | `\"\"` | Role/personality prompt sent to the LLM |\n| `llm_config` | `LLMConfig \\| dict \\| False \\| None` | `None` | LLM configuration; `False` disables LLM |\n| `human_input_mode` | `\"ALWAYS\" \\| \"NEVER\" \\| \"TERMINATE\"` | `\"TERMINATE\"` | When to solicit human input |\n| `code_execution_config` | `dict \\| False` | `False` | Code executor settings; `False` disables |\n| `functions` | `list[Callable]` | `None` | Tool functions the agent can call |\n| `is_termination_msg` | `Callable[[dict], bool] \\| None` | `None` | Predicate to end the conversation |\n| `max_consecutive_auto_reply` | `int \\| None` | `None` | Auto-reply limit before requesting human input |\n| `description` | `str \\| None` | `None` | Description used by group chat manager for speaker selection |\n\n## UserProxyAgent\n\nPre-configured to represent a human or execute code on their behalf. Extends `ConversableAgent`.\n\n```python\nfrom autogen import UserProxyAgent\n\nuser_proxy = UserProxyAgent(\n name=\"user_proxy\",\n code_execution_config={\"executor\": executor},\n)\n```\n\nKey defaults that differ from `ConversableAgent`:\n- `human_input_mode=\"ALWAYS\"` — prompts for human input by default\n- `llm_config=False` — no LLM backing\n- `code_execution_config={}` — code execution enabled by default\n\nUse when: you need a human-in-the-loop checkpoint, tool execution relay, or code execution sandbox.\n\n## run()\n\nThe primary method for single-agent and two-agent conversations. For 3+ agents, use `run_group_chat()` (see `references/group-chat.md`). Non-blocking — runs the chat in a background thread and returns a `RunResponse` immediately.\n\n```python\nresult = assistant.run(\n message=\"Write a function that groups strings by first letter.\",\n max_turns=3,\n)\nresult.process() # execute and stream output to console\n\nprint(result.summary)\n```\n\n| Param | Type | Default | Description |\n|---|---|---|---|\n| `message` | `str \\| dict \\| Callable \\| None` | `None` | Initial message to start the chat |\n| `recipient` | `ConversableAgent \\| None` | `None` | Target agent; `None` enables single-agent mode |\n| `max_turns` | `int \\| None` | `None` | Maximum conversation turns |\n| `user_input` | `bool \\| None` | `False` | Whether to allow user input during execution |\n| `tools` | `Tool \\| Iterable[Tool] \\| None` | `None` | Tools available (single-agent mode only) |\n| `clear_history` | `bool` | `True` | Clear chat history before starting |\n| `silent` | `bool \\| None` | `False` | Suppress console output |\n| `summary_method` | `str \\| Callable \\| None` | `\"last_msg\"` | How to generate the summary (`\"last_msg\"`, `\"reflection_with_llm\"`, or callable) |\n| `msg_to` | `str \\| None` | `\"agent\"` | Direction of initial message: `\"agent\"` or `\"user\"` |\n\n### Single-agent mode\n\nWhen `recipient=None`, `run()` creates a temporary executor agent automatically:\n\n```python\nresult = assistant.run(\n message=\"Explain Python decorators.\",\n max_turns=1,\n)\nresult.process()\n```\n\n### Two-agent mode\n\nPass a `recipient` to create a direct conversation between two agents:\n\n```python\nresult = user_proxy.run(\n recipient=assistant,\n message=\"Summarize the advantages of asyncio.\",\n max_turns=4,\n)\nresult.process()\n```\n\n### RunResponse\n\n`run()` returns a `RunResponse`. Call `.process()` to execute, then access results:\n\n```python\nresult = assistant.run(message=\"Explain decorators.\", max_turns=2)\nresult.process()\n\nprint(result.summary) # conversation summary\nprint(result.cost) # token usage and cost breakdown\nprint(result.last_speaker) # name of the last agent that spoke\n```\n\n| Property | Type | Description |\n|---|---|---|\n| `.summary` | `str \\| None` | Conversation summary (controlled by `summary_method`) |\n| `.cost` | `Cost \\| None` | Token usage and cost; `.cost.usage_including_cached_inference` and `.cost.usage_excluding_cached_inference` each have `.total_cost` |\n| `.last_speaker` | `str \\| None` | Name of the last agent that spoke |\n| `.context_variables` | `ContextVariables \\| None` | Shared context state |\n| `.events` | `Iterable[BaseEvent]` | Stream of conversation events |\n| `.messages` | `Iterable[Message]` | Chat messages exchanged |\n\nPass a custom `EventProcessorProtocol` to `.process(processor)` for custom event handling.\n\n## run_iter()\n\nStepped execution — the background thread pauses after each event until you advance. Use for event-driven UIs, custom logging, or streaming integrations.\n\n```python\nfrom autogen.events import TextEvent, TerminationEvent\n\nfor event in assistant.run_iter(\n message=\"Write a haiku about Python.\",\n yield_on=[TextEvent, TerminationEvent],\n):\n if isinstance(event, TextEvent):\n print(f\"[{event.source}]: {event.content}\")\n elif isinstance(event, TerminationEvent):\n print(\"Done.\")\n```\n\n| Param | Type | Default | Description |\n|---|---|---|---|\n| `yield_on` | `Sequence[type[BaseEvent]] \\| None` | `None` | Event types to yield; `None` yields all |\n\nAll other params match `run()`. Common event types:\n\n| Event | Description |\n|---|---|\n| `TextEvent` | Agent produced text output |\n| `ToolCallEvent` | Agent invoked a tool |\n| `TerminationEvent` | Conversation ended |\n| `InputRequestEvent` | Agent is requesting user input |\n\n## Tool Registration\n\nTool execution works differently depending on whether you use group chat or two-agent chat.\n\n### Group chat: automatic tool execution\n\nIn `run_group_chat()`, tool execution is automatic. Register tools on the agent via `functions` and AG2 handles execution internally through a built-in `_Group_Tool_Executor` agent:\n\n```python\nfrom typing import Annotated\n\ndef search_web(query: Annotated[str, \"The search query\"]) -> str:\n \"\"\"Search the web and return results.\"\"\"\n return do_search(query)\n\nresearcher = ConversableAgent(\n name=\"researcher\",\n llm_config=llm_config,\n functions=[search_web],\n description=\"Researches topics using web search.\",\n)\nwriter = ConversableAgent(name=\"writer\", llm_config=llm_config,\n description=\"Writes content based on research.\")\neditor = ConversableAgent(name=\"editor\", llm_config=llm_config,\n description=\"Edits and polishes written content.\")\n\n# In group chat, tool calls are executed automatically\nresult = run_group_chat(\n pattern=AutoPattern(\n initial_agent=researcher,\n agents=[researcher, writer, editor],\n group_manager_args={\"llm_config\": llm_config},\n ),\n messages=\"Research and write about AG2 documentation.\",\n)\nresult.process()\n```\n\n### Two-agent chat: split registration\n\nWith `run()` and two agents, you must explicitly split tool registration — one agent proposes the tool call (LLM), and the other executes it:\n\n```python\nfrom typing import Annotated\nfrom autogen import ConversableAgent, UserProxyAgent\n\nassistant = ConversableAgent(name=\"assistant\", llm_config=llm_config)\nuser_proxy = UserProxyAgent(name=\"user_proxy\", code_execution_config=False)\n\n@user_proxy.register_for_execution()\n@assistant.register_for_llm(description=\"Search the web\")\ndef search_web(query: Annotated[str, \"The search query\"]) -> str:\n return do_search(query)\n\nresult = user_proxy.run(\n recipient=assistant,\n message=\"Search for AG2 documentation.\",\n)\nresult.process()\n```\n\n| Decorator | Purpose |\n|---|---|\n| `@agent.register_for_llm(description=...)` | Makes the tool available in the agent's LLM tool list (agent proposes the call) |\n| `@agent.register_for_execution(name=...)` | Registers the function for actual execution by that agent (agent runs it) |\n\nThe LLM agent decides when to call the tool; the execution agent runs it and returns the result.\n\n### Single-agent mode: inline tools\n\nWhen using `run()` without a recipient, pass tools directly:\n\n```python\nfrom typing import Annotated\n\ndef search_web(query: Annotated[str, \"The search query\"]) -> str:\n \"\"\"Search the web and return results.\"\"\"\n return do_search(query)\n\nassistant = ConversableAgent(name=\"assistant\", llm_config=llm_config)\n\nresult = assistant.run(\n message=\"Search for AG2 documentation.\",\n tools=[search_web],\n)\nresult.process()\n```\n\n### Defining tool functions\n\nUse `Annotated` types with descriptions on every parameter. The function docstring becomes the tool description the LLM sees:\n\n```python\nfrom typing import Annotated, Any\n\ndef check_payment(\n vendor: Annotated[str, \"The vendor name\"],\n amount: Annotated[float, \"The transaction amount\"],\n) -> dict[str, Any]:\n \"\"\"Check if a payment has been processed.\"\"\"\n return {\"status\": \"processed\", \"vendor\": vendor}\n```\n\n### Tools with context variables\n\nName a parameter `context_variables: ContextVariables` and AG2 injects the shared context automatically:\n\n```python\nfrom autogen.agentchat.group.context_variables import ContextVariables\n\ndef get_user_name(context_variables: ContextVariables) -> str:\n \"\"\"Get the current user's name from context.\"\"\"\n return context_variables.get(\"user_name\", \"Unknown\")\n```\n\n## initiate_chat() (legacy)\n\nBlocking two-agent chat. Suitable for simple synchronous scripts:\n\n```python\nresult = user_proxy.initiate_chat(\n assistant,\n message=\"Explain list comprehensions.\",\n max_turns=3,\n)\nprint(result.summary)\n```\n\nReturns a `ChatResult` with `.chat_history`, `.summary`, `.cost`, `.human_input`.\n\nNote: Prefer `run()` + `.process()` for new code. `run()` supports single-agent mode, event streaming, and the `RunResponse` interface. `initiate_chat()` is retained for backward compatibility.\n","content/autogen/docs/package/python/references/group-chat.md":"# AG2 Group Chat API Reference\n\n## run_group_chat()\n\nThe primary function for orchestrating 3+ agents. For single-agent or two-agent conversations, use `run()` (see `references/agents.md`). Non-blocking — runs in a background thread and returns a `RunResponse` immediately.\n\n```python\nfrom autogen import ConversableAgent\nfrom autogen.agentchat import run_group_chat\nfrom autogen.agentchat.group.patterns import AutoPattern\nfrom autogen.llm_config import LLMConfig\n\nllm_config = LLMConfig({\"api_type\": \"google\", \"model\": \"gemini-3.1-flash-lite-preview\"})\n\nplanner = ConversableAgent(name=\"planner\", llm_config=llm_config,\n description=\"Plans implementation steps and breaks down tasks.\")\nreviewer = ConversableAgent(name=\"reviewer\", llm_config=llm_config,\n description=\"Reviews work for correctness and quality.\")\ncoder = ConversableAgent(name=\"coder\", llm_config=llm_config,\n description=\"Writes Python code based on the plan.\")\n\npattern = AutoPattern(\n initial_agent=planner,\n agents=[planner, reviewer, coder],\n group_manager_args={\"llm_config\": llm_config},\n)\n\nresult = run_group_chat(\n pattern=pattern,\n messages=\"Design a REST API for a todo app.\",\n max_rounds=10,\n)\nresult.process()\n\nprint(result.summary)\n```\n\n| Param | Type | Default | Description |\n|---|---|---|---|\n| `pattern` | `Pattern` | required | Orchestration pattern controlling agent selection |\n| `messages` | `str \\| list[dict]` | required | Initial message(s) to start the group chat |\n| `max_rounds` | `int` | `20` | Maximum conversation rounds before termination |\n\n### RunResponse\n\n`run_group_chat()` returns a `RunResponse`. Call `.process()` to execute, then access results:\n\n```python\nresult = run_group_chat(pattern=pattern, messages=\"Design an API.\")\nresult.process()\n\nprint(result.summary) # conversation summary\nprint(result.cost) # token usage and cost breakdown\nprint(result.last_speaker) # name of the last agent that spoke\nprint(result.context_variables.to_dict()) # final shared state\n```\n\n| Property | Type | Description |\n|---|---|---|\n| `.summary` | `str \\| None` | Conversation summary (controlled by `summary_method`) |\n| `.cost` | `Cost \\| None` | Token usage and cost; `.cost.usage_including_cached_inference` and `.cost.usage_excluding_cached_inference` each have `.total_cost` |\n| `.last_speaker` | `str \\| None` | Name of the last agent that spoke |\n| `.context_variables` | `ContextVariables \\| None` | Final shared state after the conversation |\n| `.events` | `Iterable[BaseEvent]` | Stream of all conversation events |\n| `.messages` | `Iterable[Message]` | Chat messages exchanged |\n\n`run()` returns the same `RunResponse` — see `references/agents.md` for details.\n\n## run_group_chat_iter()\n\nStepped iteration over group chat events. Use for streaming UIs or custom event processing.\n\n```python\nfrom autogen.agentchat import run_group_chat_iter\nfrom autogen.events import TextEvent, TerminationEvent\n\nfor event in run_group_chat_iter(\n pattern=pattern,\n messages=\"Plan a deployment strategy.\",\n max_rounds=15,\n yield_on=[TextEvent, TerminationEvent],\n):\n if isinstance(event, TextEvent):\n print(f\"[{event.source}]: {event.content}\")\n```\n\n| Param | Type | Default | Description |\n|---|---|---|---|\n| `yield_on` | `Sequence[type[BaseEvent]] \\| None` | `None` | Event types to yield; `None` yields all |\n\nAll other params match `run_group_chat()`. Common event types:\n\n| Event | Description |\n|---|---|\n| `TextEvent` | Agent produced text output |\n| `ToolCallEvent` | Agent invoked a tool |\n| `GroupChatRunChatEvent` | Group chat round completed |\n| `TerminationEvent` | Conversation ended |\n\n## Tool Execution in Group Chat\n\nIn group chat, tool execution is automatic. Register tools on an agent via `functions` and AG2 handles execution internally — no need for a separate executor agent or split registration:\n\n```python\nfrom typing import Annotated\n\ndef lookup_order(order_id: Annotated[str, \"The order ID\"]) -> str:\n \"\"\"Look up an order by ID.\"\"\"\n return f\"Order {order_id}: shipped\"\n\nagent = ConversableAgent(\n name=\"support\",\n llm_config=llm_config,\n functions=[lookup_order],\n)\n```\n\nThis differs from two-agent `run()` where you must split registration between an LLM agent and an execution agent. See `references/agents.md` for the two-agent pattern.\n\n## Patterns\n\nPatterns control how agents take turns in a group chat. All patterns share a common base:\n\n| Param | Type | Default | Description |\n|---|---|---|---|\n| `initial_agent` | `ConversableAgent` | required | Agent that speaks first |\n| `agents` | `list[ConversableAgent]` | required | All participating agents |\n| `user_agent` | `ConversableAgent \\| None` | `None` | Human-in-the-loop agent |\n| `group_manager_args` | `dict \\| None` | `None` | Config for the internal group manager (e.g., `llm_config`, `is_termination_msg`) |\n| `context_variables` | `ContextVariables \\| None` | `None` | Shared state across agents |\n| `group_after_work` | `TransitionTarget \\| None` | `TerminateTarget()` | Default action when no handoff triggers |\n| `summary_method` | `str \\| Callable \\| None` | `\"last_msg\"` | How to generate the conversation summary |\n\n### AutoPattern\n\nLLM-based agent selection. The group manager uses its LLM to choose the next speaker based on conversation context. Easiest to start with.\n\nImportant: Set `description` on each agent — the group manager uses agent descriptions to decide who speaks next. Without descriptions, it uses the system_message which can be too verbose.\n\n```python\nfrom autogen.agentchat.group.patterns import AutoPattern\n\nplanner = ConversableAgent(name=\"planner\", llm_config=llm_config,\n description=\"Plans implementation steps and breaks down tasks.\")\ncoder = ConversableAgent(name=\"coder\", llm_config=llm_config,\n description=\"Writes Python code based on the plan.\")\nreviewer = ConversableAgent(name=\"reviewer\", llm_config=llm_config,\n description=\"Reviews code and plans for correctness and quality.\")\n\npattern = AutoPattern(\n initial_agent=planner,\n agents=[planner, coder, reviewer],\n group_manager_args={\"llm_config\": llm_config},\n)\n```\n\n`group_after_work` is always `GroupManagerTarget` (LLM picks the next agent). Requires an LLM config on the group manager.\n\nUse when: you want the LLM to dynamically route between agents without explicit handoff rules.\n\n### DefaultPattern\n\nExplicit handoff-driven routing. Agents must configure their own transitions via `agent.handoffs`. No automatic next-speaker selection.\n\n```python\nfrom autogen.agentchat.group.patterns import DefaultPattern\nfrom autogen.agentchat.group import OnCondition, AfterWork\n\nplanner.handoffs.add_llm_condition(\n OnCondition(target=coder, condition=\"The plan is ready to implement.\")\n)\ncoder.handoffs.add_llm_condition(\n OnCondition(target=reviewer, condition=\"Code is ready for review.\")\n)\nreviewer.handoffs.set_after_work(AfterWork(target=planner))\n\npattern = DefaultPattern(\n initial_agent=planner,\n agents=[planner, coder, reviewer],\n)\n```\n\nUse when: you need deterministic control over agent transitions.\n\n### RoundRobinPattern\n\nAgents speak in list order, cycling back to the first. No LLM-based selection.\n\n```python\nfrom autogen.agentchat.group.patterns import RoundRobinPattern\n\npattern = RoundRobinPattern(\n initial_agent=agent_a,\n agents=[agent_a, agent_b, agent_c],\n)\n```\n\nUse when: every agent must contribute each round in a fixed order.\n\n### RandomPattern\n\nEach turn, a random agent is selected. No LLM-based selection.\n\n```python\nfrom autogen.agentchat.group.patterns import RandomPattern\n\npattern = RandomPattern(\n initial_agent=agent_a,\n agents=[agent_a, agent_b, agent_c],\n)\n```\n\nUse when: you want diverse perspectives without ordering bias.\n\n### ManualPattern\n\nPrompts the user to select the next agent each turn.\n\n```python\nfrom autogen.agentchat.group.patterns import ManualPattern\n\npattern = ManualPattern(\n initial_agent=agent_a,\n agents=[agent_a, agent_b, agent_c],\n user_agent=human,\n)\n```\n\nUse when: a human operator needs full control over the conversation flow.\n\n## Handoffs\n\nHandoffs control how agents transition to each other in `DefaultPattern` (or when adding custom transitions in any pattern).\n\n### OnCondition\n\nLLM-evaluated transition. The condition is translated into a tool the LLM can call to trigger the handoff.\n\n```python\nfrom autogen.agentchat.group import OnCondition\n\nplanner.handoffs.add_llm_condition(\n OnCondition(\n target=reviewer,\n condition=\"The plan is complete and ready for review.\",\n )\n)\n```\n\n| Param | Type | Description |\n|---|---|---|\n| `target` | `TransitionTarget` | Agent or target to hand off to |\n| `condition` | `str` | Natural language condition evaluated by the LLM |\n\n### OnContextCondition\n\nContext variable-based transition. Evaluated without an LLM call — checked before `OnCondition`.\n\n```python\nfrom autogen.agentchat.group import OnContextCondition\n\nplanner.handoffs.add_context_condition(\n OnContextCondition(\n target=reviewer,\n condition=lambda ctx: ctx.get(\"plan_ready\", False),\n )\n)\n```\n\n### AfterWork\n\nFallback target when no `OnCondition` or `OnContextCondition` triggers.\n\n```python\nfrom autogen.agentchat.group import AfterWork\n\nreviewer.handoffs.set_after_work(AfterWork(target=planner))\n```\n\n### Transition Targets\n\nTargets control where the conversation goes next. Used in `OnCondition`, `AfterWork`, and `ReplyResult`.\n\n| Target | Description |\n|---|---|\n| `AgentTarget(agent)` | Hand off to a specific agent instance |\n| `AgentNameTarget(\"name\")` | Hand off to an agent by name (string) |\n| `RevertToUserTarget()` | Return control to the user agent |\n| `StayTarget()` | Current agent continues speaking |\n| `TerminateTarget()` | End the group chat |\n| `GroupManagerTarget()` | Let the group manager LLM pick the next agent |\n| `RandomAgentTarget(agents)` | Randomly select from a list of agents |\n| `FunctionTarget(fn)` | Call a function to determine the next target dynamically |\n\nAll targets import from `autogen.agentchat.group`.\n\n#### AgentTarget\n\nThe most common target — routes to a specific agent:\n\n```python\nfrom autogen.agentchat.group import AgentTarget, OnCondition\n\nplanner.handoffs.add_llm_condition(\n OnCondition(target=AgentTarget(coder), condition=\"Plan is ready to implement.\")\n)\n```\n\nYou can also pass an agent instance directly to `target=` in `OnCondition` or `AfterWork` — AG2 wraps it in `AgentTarget` automatically.\n\n#### FunctionTarget\n\nCalls a function at transition time to dynamically decide where to go. The function receives the last message and context variables, and must return a `FunctionTargetResult`:\n\n```python\nfrom autogen.agentchat.group import FunctionTarget, OnCondition\nfrom autogen.agentchat.group.targets.function_target import FunctionTargetResult\n\ndef route_next(output: str, context_variables: ContextVariables) -> FunctionTargetResult:\n if context_variables.get(\"needs_review\", False):\n return FunctionTargetResult(target=AgentTarget(reviewer))\n return FunctionTargetResult(target=AgentTarget(coder))\n\nplanner.handoffs.set_after_work(\n AfterWork(target=FunctionTarget(route_next))\n)\n```\n\n| `FunctionTargetResult` field | Type | Description |\n|---|---|---|\n| `target` | `TransitionTarget` | Required — the next target to transition to |\n| `messages` | `list \\| str \\| None` | Optional messages to broadcast to specific agents |\n| `context_variables` | `ContextVariables \\| None` | Optional context updates to merge |\n\nUse when: the next agent depends on runtime state that can't be expressed as a static condition.\n\n### Handoffs API\n\nAll methods return `self` for chaining:\n\n```python\nagent.handoffs \\\n .add_llm_condition(OnCondition(target=other, condition=\"Ready.\")) \\\n .set_after_work(AfterWork(target=TerminateTarget()))\n```\n\n| Method | Description |\n|---|---|\n| `.add_llm_condition(condition)` | Add an LLM-evaluated transition |\n| `.add_context_condition(condition)` | Add a context variable-based transition |\n| `.set_after_work(target)` | Set unconditional fallback (replaces existing) |\n| `.add_after_work(condition)` | Add conditional fallback |\n| `.add(condition)` | Type-dispatched — adds any condition type |\n| `.add_many(conditions)` | Add multiple conditions at once |\n| `.clear()` | Remove all handoffs |\n\n## ReplyResult\n\nTool functions can return a `ReplyResult` to control both the response message and which agent speaks next. This is the primary way tools influence routing in group chats.\n\n```python\nfrom autogen.agentchat.group.reply_result import ReplyResult\nfrom autogen.agentchat.group import AgentTarget\n\ndef process_order(order_id: Annotated[str, \"The order ID\"],\n context_variables: ContextVariables) -> ReplyResult:\n \"\"\"Process an order and route to the appropriate agent.\"\"\"\n result = do_processing(order_id)\n context_variables.set(\"order_status\", result[\"status\"])\n\n if result[\"status\"] == \"needs_review\":\n return ReplyResult(\n message=f\"Order {order_id} needs manual review.\",\n target=AgentTarget(reviewer),\n context_variables=context_variables,\n )\n return ReplyResult(\n message=f\"Order {order_id} processed successfully.\",\n target=AgentTarget(shipping_agent),\n )\n```\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `message` | `str` | required | The tool's response text (what the LLM sees) |\n| `target` | `TransitionTarget \\| None` | `None` | Which agent to hand off to next |\n| `context_variables` | `ContextVariables \\| None` | `None` | Context updates to merge into the group's shared state |\n\nA `ReplyResult` without a `target` lets the normal handoff/pattern logic decide the next speaker while still updating context variables:\n\n```python\ndef fetch_data(query: Annotated[str, \"The query\"]) -> ReplyResult:\n \"\"\"Fetch data without influencing routing.\"\"\"\n result = do_fetch(query)\n return ReplyResult(message=result)\n```\n\nWhen `target` is set, `ReplyResult` overrides the normal routing. Without `ReplyResult` at all, a tool returns a plain string and the pattern decides the next speaker.\n\nNote: Only the `message` field is visible to the LLM in conversation history. The `target` and `context_variables` are consumed by the orchestration layer silently.\n\n## ContextVariables\n\nDict-like shared state accessible by all agents and tool functions in a group chat.\n\n```python\nfrom autogen.agentchat.group.context_variables import ContextVariables\n\nctx = ContextVariables({\"user_name\": \"Alice\", \"step\": 1})\n\n# Dict-like access\nctx[\"step\"] = 2\nname = ctx.get(\"user_name\", \"Unknown\")\ndel ctx[\"step\"]\n```\n\n| Method | Description |\n|---|---|\n| `.get(key, default)` | Get value with fallback |\n| `.set(key, value)` | Set a value |\n| `.remove(key)` | Remove a key, returns `bool` |\n| `.update(dict)` | Merge another dict |\n| `.contains(key)` | Check existence |\n| `.to_dict()` | Export as plain dict |\n\n### Injecting into tool functions\n\nName a parameter `context_variables: ContextVariables` and AG2 provides it automatically:\n\n```python\ndef update_step(context_variables: ContextVariables) -> str:\n \"\"\"Advance to the next workflow step.\"\"\"\n step = context_variables.get(\"step\", 0) + 1\n context_variables.set(\"step\", step)\n return f\"Advanced to step {step}\"\n```\n\n### Passing to a pattern\n\n```python\nctx = ContextVariables({\"project\": \"my-app\"})\n\npattern = DefaultPattern(\n initial_agent=planner,\n agents=[planner, coder, reviewer],\n context_variables=ctx,\n)\n\nresult = run_group_chat(pattern=pattern, messages=\"Start the project.\")\nresult.process()\n\n# Access final context state\nprint(result.context_variables.to_dict())\n```\n\n## Termination\n\n### is_termination_msg\n\nPass a predicate to `group_manager_args` to stop the chat when a condition is met:\n\n```python\ndef should_terminate(msg: dict) -> bool:\n content = msg.get(\"content\", \"\") or \"\"\n return \"DONE\" in content\n\npattern = AutoPattern(\n initial_agent=planner,\n agents=[planner, reviewer, coder], # ensure each agent has a description set\n group_manager_args={\n \"llm_config\": llm_config,\n \"is_termination_msg\": should_terminate,\n },\n)\n```\n\n### max_rounds\n\nSet `max_rounds` in `run_group_chat()` to cap the number of turns:\n\n```python\nresult = run_group_chat(pattern=pattern, messages=\"Go.\", max_rounds=5)\n```\n\n### Agent-level termination\n\nUse `TerminateTarget()` in handoffs to end the group chat from a specific agent:\n\n```python\nfrom autogen.agentchat.group import AfterWork, TerminateTarget\n\nreviewer.handoffs.set_after_work(AfterWork(target=TerminateTarget()))\n```\n\n## initiate_group_chat() (legacy)\n\nBlocking group chat. Returns a tuple instead of `RunResponse`:\n\n```python\nfrom autogen.agentchat import initiate_group_chat\n\nresult, context_variables, last_agent = initiate_group_chat(\n pattern=pattern,\n messages=\"Start planning.\",\n max_rounds=10,\n)\nprint(result.summary)\n```\n\nReturns `(ChatResult, ContextVariables, Agent)`.\n\nNote: Prefer `run_group_chat()` + `.process()` for new code. It returns a unified `RunResponse` with event streaming support. `initiate_group_chat()` is retained for backward compatibility.\n\n## Common Patterns\n\n### Planning and review loop\n\n```python\nplanner = ConversableAgent(name=\"planner\", llm_config=llm_config,\n system_message=\"Create detailed implementation plans.\",\n description=\"Plans implementation steps and breaks down tasks.\")\ncoder = ConversableAgent(name=\"coder\", llm_config=llm_config,\n system_message=\"Implement the plan in Python.\",\n description=\"Writes Python code based on the plan.\")\nreviewer = ConversableAgent(name=\"reviewer\", llm_config=llm_config,\n system_message=\"Review plans and code. Say APPROVED when satisfied.\",\n description=\"Reviews code and plans for correctness.\")\n\ndef is_approved(msg):\n return \"APPROVED\" in (msg.get(\"content\", \"\") or \"\")\n\npattern = AutoPattern(\n initial_agent=planner,\n agents=[planner, coder, reviewer],\n group_manager_args={\"llm_config\": llm_config, \"is_termination_msg\": is_approved},\n)\n\nresult = run_group_chat(pattern=pattern, messages=\"Plan a user auth system.\")\nresult.process()\n```\n\n### Explicit handoff chain\n\n```python\nfrom autogen.agentchat.group import OnCondition, AfterWork, TerminateTarget\nfrom autogen.agentchat.group.patterns import DefaultPattern\n\nresearcher = ConversableAgent(name=\"researcher\", llm_config=llm_config)\nwriter = ConversableAgent(name=\"writer\", llm_config=llm_config)\neditor = ConversableAgent(name=\"editor\", llm_config=llm_config)\n\nresearcher.handoffs.add_llm_condition(\n OnCondition(target=writer, condition=\"Research is complete.\")\n)\nwriter.handoffs.add_llm_condition(\n OnCondition(target=editor, condition=\"Draft is written.\")\n)\neditor.handoffs.set_after_work(AfterWork(target=TerminateTarget()))\n\npattern = DefaultPattern(\n initial_agent=researcher,\n agents=[researcher, writer, editor],\n)\n\nresult = run_group_chat(pattern=pattern, messages=\"Write about async Python.\")\nresult.process()\n```\n\n### Human-in-the-loop\n\n```python\nfrom autogen import UserProxyAgent\n\nhuman = UserProxyAgent(name=\"human\")\n\npattern = AutoPattern(\n initial_agent=planner,\n agents=[planner, coder, reviewer], # ensure each agent has a description set\n user_agent=human,\n group_manager_args={\"llm_config\": llm_config},\n)\n\nresult = run_group_chat(pattern=pattern, messages=\"Build a CLI tool.\")\nresult.process()\n```\n","content/datahub/docs/sdk/python/references/assertions.md":"# Bulk Data Quality Assertions\n\nCreate and manage smart assertions at scale using the DataHub Cloud Python SDK (`acryl-datahub-cloud`).\n\nThe actor making API calls needs `Edit Assertions` and `Edit Monitors` privileges.\n\n## Setup\n\n```python\nfrom datahub.sdk import DataHubClient, FilterDsl as F\nfrom datahub.metadata.urns import DatasetUrn\n\nclient = DataHubClient(server=\"\", token=\"\")\n```\n\n## Discover Target Tables\n\n### By specific URNs\n\n```python\ntable_urns = [\n \"urn:li:dataset:(urn:li:dataPlatform:snowflake,database.schema.users,PROD)\",\n \"urn:li:dataset:(urn:li:dataPlatform:snowflake,database.schema.orders,PROD)\",\n]\ndatasets = [DatasetUrn.from_string(urn) for urn in table_urns]\n```\n\n### By search pattern\n\n```python\nresults = client.search.get_urns(\n filter=F.and_(\n F.entity_type(\"dataset\"),\n F.platform(\"snowflake\"),\n F.env(\"PROD\"),\n )\n)\ndatasets = [DatasetUrn.from_string(str(urn)) for urn in results]\n```\n\n### By tag or domain\n\n```python\ncritical_datasets = client.search.get_urns(\n filter=F.and_(\n F.entity_type(\"dataset\"),\n F.tag(\"urn:li:tag:critical\"),\n )\n)\n```\n\n## Create Table-Level Assertions\n\n### Freshness assertions\n\n```python\ndef create_freshness_assertions(datasets, client):\n for dataset_urn in datasets:\n try:\n assertion = client.assertions.sync_smart_freshness_assertion(\n dataset_urn=dataset_urn,\n display_name=\"Freshness Anomaly Monitor\",\n detection_mechanism=\"information_schema\",\n sensitivity=\"medium\",\n tags=[\"automated\", \"freshness\", \"data_quality\"],\n enabled=True,\n )\n print(f\"Created freshness assertion for {dataset_urn.name}: {assertion.urn}\")\n except Exception as e:\n print(f\"Failed for {dataset_urn.name}: {e}\")\n```\n\n### Volume assertions\n\n```python\ndef create_volume_assertions(datasets, client):\n for dataset_urn in datasets:\n try:\n assertion = client.assertions.sync_smart_volume_assertion(\n dataset_urn=dataset_urn,\n display_name=\"Smart Volume Check\",\n detection_mechanism=\"information_schema\",\n sensitivity=\"medium\",\n tags=[\"automated\", \"volume\", \"data_quality\"],\n schedule=\"0 */6 * * *\",\n enabled=True,\n )\n print(f\"Created volume assertion for {dataset_urn.name}: {assertion.urn}\")\n except Exception as e:\n print(f\"Failed for {dataset_urn.name}: {e}\")\n```\n\n### SQL assertions\n\n```python\ndef create_sql_assertions(datasets, client):\n sql_queries = {\n \"row_count\": \"SELECT COUNT(*) FROM {table_name}\",\n \"null_check\": \"SELECT COUNT(*) FROM {table_name} WHERE id IS NULL\",\n }\n\n for dataset_urn in datasets:\n for query_name, query_template in sql_queries.items():\n try:\n statement = query_template.format(table_name=dataset_urn.name)\n assertion = client.assertions.sync_smart_sql_assertion(\n dataset_urn=dataset_urn,\n display_name=f\"Smart SQL - {query_name}\",\n statement=statement,\n sensitivity=\"medium\",\n tags=[\"automated\", \"smart_sql\", query_name],\n schedule=\"0 */6 * * *\",\n enabled=True,\n )\n print(f\"Created SQL assertion '{query_name}' for {dataset_urn.name}: {assertion.urn}\")\n except Exception as e:\n print(f\"Failed SQL assertion '{query_name}' for {dataset_urn.name}: {e}\")\n```\n\n## Get Column Information\n\n```python\ndef get_dataset_columns(client, dataset_urn):\n try:\n dataset = client.entities.get(dataset_urn)\n if dataset and dataset.schema:\n return [\n {\n \"name\": field.field_path,\n \"type\": field.native_type,\n }\n for field in dataset.schema\n ]\n return []\n except Exception as e:\n print(f\"Failed to get columns for {dataset_urn}: {e}\")\n return []\n\ndataset_columns = {}\nfor dataset_urn in datasets:\n columns = get_dataset_columns(client, dataset_urn)\n dataset_columns[str(dataset_urn)] = columns\n```\n\n## Create Column-Level Assertions\n\n```python\nimport fnmatch\n\nASSERTION_RULES = {\n \"null_checks\": {\n \"column_patterns\": [\"id\", \"*_id\", \"user_id\", \"email\"],\n \"metric_type\": \"null_count\",\n },\n \"unique_checks\": {\n \"column_patterns\": [\"*_id\", \"email\", \"username\"],\n \"metric_type\": \"unique_count\",\n },\n \"empty_checks\": {\n \"column_patterns\": [\"name\", \"description\", \"title\"],\n \"metric_type\": \"empty_count\",\n },\n}\n\ndef should_apply_rule(column_name, rule_config):\n return any(\n fnmatch.fnmatch(column_name.lower(), pattern.lower())\n for pattern in rule_config[\"column_patterns\"]\n )\n\ndef create_column_assertions(datasets, columns_dict, client):\n for dataset_urn in datasets:\n columns = columns_dict.get(str(dataset_urn), [])\n for column in columns:\n for rule_name, rule_config in ASSERTION_RULES.items():\n if should_apply_rule(column[\"name\"], rule_config):\n try:\n client.assertions.sync_smart_column_metric_assertion(\n dataset_urn=dataset_urn,\n column_name=column[\"name\"],\n metric_type=rule_config[\"metric_type\"],\n display_name=f\"{rule_name} - {column['name']}\",\n detection_mechanism=\"all_rows_query_datahub_dataset_profile\",\n tags=[\"automated\", \"column_quality\", rule_name],\n enabled=True,\n )\n print(f\"Created {rule_name} for {dataset_urn.name}.{column['name']}\")\n except Exception as e:\n print(f\"Failed {rule_name} for {dataset_urn.name}.{column['name']}: {e}\")\n```\n\n## Update Existing Assertions\n\nPass the existing assertion URN to update rather than create:\n\n```python\nupdated = client.assertions.sync_smart_freshness_assertion(\n dataset_urn=dataset_urn,\n urn=existing_assertion_urn,\n sensitivity=\"high\",\n tags=[\"automated\", \"freshness\", \"updated\"],\n enabled=True,\n)\n```\n\n## Batch Processing\n\n```python\nimport time\n\ndef batch_create_assertions(datasets, client, batch_size=10, delay_seconds=1.0):\n results = {\"successful\": [], \"failed\": []}\n\n for i in range(0, len(datasets), batch_size):\n batch = datasets[i : i + batch_size]\n for dataset_urn in batch:\n try:\n assertion = client.assertions.sync_smart_freshness_assertion(\n dataset_urn=dataset_urn,\n tags=[\"batch_created\", \"automated\"],\n enabled=True,\n )\n results[\"successful\"].append(\n {\"dataset\": str(dataset_urn), \"assertion\": str(assertion.urn)}\n )\n except Exception as e:\n results[\"failed\"].append({\"dataset\": str(dataset_urn), \"error\": str(e)})\n\n if i + batch_size < len(datasets):\n time.sleep(delay_seconds)\n\n return results\n```\n\n## Important Considerations\n\n- **Single-threaded per dataset:** Always process assertions for a given dataset in a single thread to avoid race conditions.\n- **Entities must exist:** Target datasets must already be present in DataHub before creating assertions.\n- **Sensitivity levels:** `\"low\"`, `\"medium\"`, `\"high\"` control anomaly detection thresholds.\n- **Detection mechanisms:** Use `\"information_schema\"` for table-level, `\"all_rows_query_datahub_dataset_profile\"` for column-level.\n- **Tags:** Plain tag names are auto-converted to URNs (e.g., `\"critical\"` becomes `\"urn:li:tag:critical\"`).\n- **Wait for processing:** Writes are async via Kafka. Wait for the previous batch to be reflected in the UI before re-running sync operations.\n","content/datahub/docs/sdk/python/references/entities.md":"# Entity Reference\n\nAdvanced information & guidelines for specific entity types.\n\n## Containers\n\nDatasets are organized physically using the container entity (e.g. databases, schemas, folders).\n\n```python\nfrom datahub.sdk import Container\nfrom datahub.emitter.mcp_builder import DatabaseKey, SchemaKey\n\ndatabase_key = DatabaseKey(\n platform=\"snowflake\",\n instance=\"production\",\n database=\"analytics_db\",\n)\n\ndatabase_container = Container(\n database_key,\n display_name=\"Analytics Database\",\n description=\"Main analytics database\",\n subtype=\"Database\",\n)\nclient.entities.upsert(database_container)\n\nschema_key = SchemaKey(\n platform=\"snowflake\",\n instance=\"production\",\n database=\"analytics_db\",\n schema=\"reporting\",\n)\n\nschema_container = Container(\n schema_key,\n display_name=\"Reporting Schema\",\n description=\"Schema for reporting tables and views\",\n subtype=\"Schema\",\n parent_container=database_key,\n domain=\"urn:li:domain:analytics\",\n)\nclient.entities.upsert(schema_container)\n```\n\n## Documents\n\nDocuments store knowledge content — tutorials, runbooks, FAQs, or references to external docs in systems like Notion or Confluence.\n\n### Native document (stored in DataHub)\n\n```python\nfrom datahub.sdk import Document\n\ndoc = Document.create_document(\n id=\"getting-started-guide\",\n title=\"Getting Started with DataHub\",\n text=\"# Welcome\\n\\nThis guide will help you get started...\",\n tags=[\"onboarding\"],\n domain=\"urn:li:domain:data-governance\",\n)\nclient.entities.upsert(doc)\n```\n\n### External document (Notion, Confluence, etc.)\n\n```python\ndoc = Document.create_external_document(\n id=\"notion-team-handbook\",\n title=\"Engineering Handbook\",\n platform=\"notion\",\n external_url=\"https://notion.so/team/engineering-handbook\",\n external_id=\"notion-page-abc123\",\n text=\"Summary of the handbook for search indexing...\",\n)\nclient.entities.upsert(doc)\n```\n\n### AI-only context document\n\nDocuments hidden from global search/sidebar, accessible only through related assets:\n\n```python\ndoc = Document.create_document(\n id=\"orders-dataset-context\",\n title=\"Orders Dataset Context\",\n text=\"This dataset contains daily order summaries...\",\n show_in_global_context=False,\n related_assets=[\"urn:li:dataset:(urn:li:dataPlatform:snowflake,orders,PROD)\"],\n)\nclient.entities.upsert(doc)\n```\n\n### Document properties\n\n- `doc.set_title(\"New Title\")` / `doc.title`\n- `doc.set_text(\"New content\")` / `doc.text`\n- `doc.publish()` / `doc.unpublish()` / `doc.status`\n- `doc.set_parent_document(\"urn:li:document:parent-doc\")` — hierarchical organization\n- `doc.add_related_asset(\"urn:li:dataset:...\")` / `doc.set_related_assets([...])`\n- `doc.add_related_document(\"urn:li:document:...\")` / `doc.set_related_documents([...])`\n- `doc.hide_from_global_context()` / `doc.show_in_global_search()`\n","content/datahub/docs/sdk/python/references/lineage.md":"# Lineage Operations\n\nAdvanced lineage patterns using the DataHub Python SDK `LineageClient`.\n\n## Cross-Entity Lineage\n\nThe `add_lineage()` method supports lineage between any combination of:\n\n- Dataset to Dataset\n- Dataset to Chart\n- Dataset to DataJob\n- DataJob to DataJob\n- DataJob to Dataset\n- Dashboard to Chart\n- Dashboard to Dataset\n- Dashboard to Dashboard\n\n```python\nclient.lineage.add_lineage(\n upstream=\"urn:li:dataset:(urn:li:dataPlatform:snowflake,raw.events,PROD)\",\n downstream=\"urn:li:chart:(looker,chart_events_overview)\",\n)\n```\n\n## Column Lineage Options\n\nThe `column_lineage` parameter on `add_lineage()` accepts several forms:\n\n```python\n# Auto-match columns by name (fuzzy matching)\nclient.lineage.add_lineage(\n upstream=\"urn:li:dataset:(urn:li:dataPlatform:postgres,users,PROD)\",\n downstream=\"urn:li:dataset:(urn:li:dataPlatform:snowflake,users_replica,PROD)\",\n column_lineage=\"auto_fuzzy\",\n)\n\n# Strict auto-match (exact column name match)\nclient.lineage.add_lineage(\n upstream=upstream_urn,\n downstream=downstream_urn,\n column_lineage=\"auto_strict\",\n)\n\n# Explicit column mapping (downstream_col -> [upstream_cols])\nclient.lineage.add_lineage(\n upstream=upstream_urn,\n downstream=downstream_urn,\n column_lineage={\n \"user_id\": [\"id\"],\n \"full_name\": [\"first_name\", \"last_name\"],\n },\n)\n\n# No column lineage (default)\nclient.lineage.add_lineage(\n upstream=upstream_urn,\n downstream=downstream_urn,\n column_lineage=False,\n)\n```\n\n## DataJob Lineage\n\nConnect pipelines (Airflow tasks, Spark jobs) to their inputs and outputs:\n\n```python\n# Datajob to downstream dataset\nclient.lineage.add_lineage(\n upstream=\"urn:li:dataJob:(urn:li:dataFlow:(airflow,etl_pipeline,PROD),load_users)\",\n downstream=\"urn:li:dataset:(urn:li:dataPlatform:snowflake,analytics.users,PROD)\",\n)\n\n# Dataset to datajob (upstream input)\nclient.lineage.add_lineage(\n upstream=\"urn:li:dataset:(urn:li:dataPlatform:postgres,users,PROD)\",\n downstream=\"urn:li:dataJob:(urn:li:dataFlow:(airflow,etl_pipeline,PROD),load_users)\",\n)\n```\n\n## SQL-Based Lineage Inference\n\nLet the SDK parse SQL to automatically extract upstream/downstream relationships:\n\n```python\nclient.lineage.infer_lineage_from_sql(\n query_text=\"\"\"\n INSERT INTO analytics.daily_revenue\n SELECT date, SUM(amount) AS revenue\n FROM raw.orders\n JOIN raw.products ON orders.product_id = products.id\n GROUP BY date\n \"\"\",\n platform=\"snowflake\",\n env=\"PROD\",\n default_db=\"my_database\",\n default_schema=\"public\",\n)\n```\n\nParameters:\n\n| Parameter | Description |\n|-----------|-------------|\n| `query_text` | The SQL query to parse |\n| `platform` | Data platform identifier (e.g., `\"snowflake\"`, `\"bigquery\"`) |\n| `platform_instance` | Optional platform instance |\n| `env` | Environment, defaults to `\"PROD\"` |\n| `default_db` | Default database for unqualified table names |\n| `default_schema` | Default schema for unqualified table names |\n| `override_dialect` | Override SQL dialect for parsing |\n\n## Filtered Lineage\n\nCombine `get_lineage()` with search filters to narrow results:\n\n```python\nfrom datahub.sdk import FilterDsl as F\n\nresults = client.lineage.get_lineage(\n source_urn=dataset_urn,\n direction=\"downstream\",\n max_hops=3,\n filter=F.and_(\n F.entity_type(\"dataset\"),\n F.platform(\"snowflake\"),\n ),\n)\n```\n\n## LineageResult Object\n\nEach result from `get_lineage()` is a `LineageResult`:\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `urn` | `str` | URN of the related entity |\n| `type` | `str` | Entity type (dataset, chart, etc.) |\n| `hops` | `int` | Number of hops from the source |\n| `direction` | `\"upstream\" \\| \"downstream\"` | Traversal direction |\n| `platform` | `str \\| None` | Platform name |\n| `name` | `str \\| None` | Entity display name |\n| `description` | `str \\| None` | Entity description |\n| `paths` | `List[LineagePath] \\| None` | Lineage paths with intermediate entities |\n\n## View Lineage (Auto-Parsed)\n\nDatasets with `view_definition` set automatically extract upstream lineage from the SQL:\n\n```python\nfrom datahub.sdk import Dataset\n\nview = Dataset(\n platform=\"snowflake\",\n name=\"analytics.reporting.sales_summary\",\n view_definition=\"SELECT * FROM analytics.raw.sales WHERE year = 2024\",\n subtype=\"view\",\n)\n# view.upstreams is auto-populated with analytics.raw.sales\n\nclient.entities.upsert(view)\n```\n\nSet `parse_view_lineage=False` to disable auto-parsing.\n","content/datahub/docs/sdk/python/references/search.md":"# Search API\n\nDetailed reference for searching and discovering metadata with the DataHub Python SDK.\n\n## Overview\n\nAll search goes through `client.search.get_urns()`, which returns an iterable of URNs. You can search by text query, structured filters, or both.\n\n```python\nfrom datahub.sdk import DataHubClient, FilterDsl as F\n\nclient = DataHubClient.from_env()\nresults = client.search.get_urns(query=\"...\", filter=F.platform(\"...\"))\n```\n\n## Filter-Based Search\n\nStructured filters scope results by platform, entity type, domain, and other metadata fields.\n\n### Platform\n\nFind all assets on a specific data platform.\n\n```python\nresults = client.search.get_urns(filter=F.platform(\"snowflake\"))\n\n# Multiple platforms\nresults = client.search.get_urns(filter=F.platform([\"snowflake\", \"bigquery\"]))\n```\n\nUse when: you know which warehouse or system the asset lives in.\n\n### Entity Type\n\nNarrow to a specific entity type.\n\n```python\nresults = client.search.get_urns(filter=F.entity_type(\"dataset\"))\nresults = client.search.get_urns(filter=F.entity_type(\"dashboard\"))\nresults = client.search.get_urns(filter=F.entity_type(\"chart\"))\nresults = client.search.get_urns(filter=F.entity_type(\"document\"))\nresults = client.search.get_urns(filter=F.entity_type(\"container\"))\nresults = client.search.get_urns(filter=F.entity_type(\"dataFlow\"))\nresults = client.search.get_urns(filter=F.entity_type(\"dataJob\"))\nresults = client.search.get_urns(filter=F.entity_type(\"glossaryTerm\"))\n```\n\nUse when: you need a specific kind of asset, not everything matching a keyword.\n\n### Entity Subtype\n\nFilter by subtype within an entity type (e.g., ML Experiment, View, Topic).\n\n```python\nresults = client.search.get_urns(\n filter=F.and_(F.platform(\"mlflow\"), F.entity_subtype(\"ML Experiment\"))\n)\n```\n\nUse when: a platform has multiple subtypes and you need a specific one.\n\n### Domain\n\nFind assets within a business domain.\n\n```python\nresults = client.search.get_urns(filter=F.domain(\"urn:li:domain:marketing\"))\n\n# Multiple domains\nresults = client.search.get_urns(filter=F.domain([\"urn:li:domain:marketing\", \"urn:li:domain:sales\"]))\n```\n\nUse when: organizing or auditing assets by business function. Preferred over env-based filtering for logical organization.\n\n### Container\n\nFind assets within a specific container (database, schema, folder).\n\n```python\nresults = client.search.get_urns(filter=F.container(\"urn:li:container:prod_analytics\"))\n\n# Direct descendants only (not nested containers)\nresults = client.search.get_urns(\n filter=F.container(\"urn:li:container:prod_analytics\", direct_descendants_only=True)\n)\n```\n\nUse when: browsing assets within a specific database or schema hierarchy.\n\n### Owner\n\nFind assets owned by a specific user or group.\n\n```python\nresults = client.search.get_urns(filter=F.owner(\"urn:li:corpuser:jane\"))\nresults = client.search.get_urns(filter=F.owner(\"urn:li:corpGroup:data-eng\"))\n```\n\nUse when: auditing ownership or finding a team's assets.\n\n### Tag\n\nFind assets with a specific tag.\n\n```python\nresults = client.search.get_urns(filter=F.tag(\"urn:li:tag:critical\"))\nresults = client.search.get_urns(filter=F.tag([\"urn:li:tag:pii\", \"urn:li:tag:sensitive\"]))\n```\n\nUse when: finding assets by classification label.\n\n### Glossary Term\n\nFind assets annotated with a business glossary term.\n\n```python\nresults = client.search.get_urns(filter=F.glossary_term(\"urn:li:glossaryTerm:PII\"))\n```\n\nUse when: finding all assets governed by a specific business concept.\n\n### Environment\n\nFind assets in a specific environment.\n\n```python\nresults = client.search.get_urns(filter=F.env(\"PROD\"))\n```\n\nNote: prefer using domains or container hierarchies for logical organization over env-based filtering.\n\n### Custom Properties\n\nFind assets with a specific custom property value.\n\n```python\nresults = client.search.get_urns(\n filter=F.has_custom_property(\"department\", \"sales\")\n)\n```\n\nUse when: searching by user-defined metadata that doesn't fit into tags or glossary terms.\n\n### Deletion Status\n\nInclude or exclude soft-deleted assets.\n\n```python\nresults = client.search.get_urns(filter=F.soft_deleted(\"NOT_SOFT_DELETED\"))\n```\n\n### Custom Field Conditions\n\nFor advanced filtering on any `@Searchable`-annotated field in DataHub's metadata models.\n\n```python\nresults = client.search.get_urns(\n filter=F.custom_filter(field=\"urn\", condition=\"CONTAIN\", values=[\"example_dataset\"])\n)\n\nresults = client.search.get_urns(\n filter=F.custom_filter(field=\"name\", condition=\"START_WITH\", values=[\"prod_\"])\n)\n```\n\n#### Supported conditions\n\n| Condition | Description |\n|-----------|-------------|\n| `EQUAL` | Exact match |\n| `CONTAIN` | Substring match |\n| `START_WITH` | Prefix match |\n| `END_WITH` | Suffix match |\n| `GREATER_THAN` | Numeric/timestamp comparison |\n| `GREATER_THAN_OR_EQUAL_TO` | Numeric/timestamp comparison |\n| `LESS_THAN` | Numeric/timestamp comparison |\n| `LESS_THAN_OR_EQUAL_TO` | Numeric/timestamp comparison |\n| `IN` | Match any value in a list |\n| `EXISTS` | Field is present |\n\n#### Common searchable fields\n\nFields annotated with `@Searchable` in DataHub's PDL models can be used with `custom_filter`. Common ones include: `urn`, `name`, `description`, `env`, `platform`.\n\nThe exact searchable fields vary by entity type. Refer to the PDL model files in the [DataHub repo](https://github.com/datahub-project/datahub/tree/master/metadata-models/src/main/pegasus) for the full list.\n\n## Common Patterns\n\n### Find datasets tagged as critical but missing an owner\n\n```python\ncritical = client.search.get_urns(filter=F.tag(\"urn:li:tag:critical\"))\nfor urn in critical:\n entity = client.entities.get(urn)\n if not entity.owners:\n print(f\"Missing owner: {urn}\")\n```\n\n## Caching\n\n`get_urns()` accepts a `skip_cache=True` parameter to bypass the search cache when you need fresh results immediately after a write:\n\n```python\nresults = client.search.get_urns(query=\"new_table\", skip_cache=True)\n```\n","content/django/docs/extensions/python/DOC.md":"---\nname: extensions\ndescription: \"django-extensions package guide for Python projects using Django management commands, model mixins, and developer tooling\"\nmetadata:\n languages: \"python\"\n versions: \"4.1\"\n revision: 2\n updated-on: \"2026-03-12\"\n source: maintainer\n tags: \"django,django-extensions,management-commands,shell,debugging,orm\"\n---\n\n# django-extensions Python Package Guide\n\n## Golden Rule\n\nUse `django-extensions` as Django project tooling, not as a standalone runtime API.\n\n- Install the package version that matches your Django support window.\n- Add `\"django_extensions\"` to `INSTALLED_APPS`.\n- Reach for its `manage.py` commands first: `shell_plus`, `runserver_plus`, `graph_models`, `runscript`, `show_urls`, and `show_permissions`.\n- Treat command-specific dependencies as optional add-ons, not as globally required.\n\nThis guide covers `django-extensions==4.1`.\n\n## What django-extensions Adds\n\n`django-extensions` is a grab bag of developer-facing Django helpers:\n\n- management commands for local debugging, schema inspection, data export, and repetitive project tasks\n- model and field mixins such as `TimeStampedModel` and `TitleSlugDescriptionModel`\n- class-based-view permission mixins\n- template debugging helpers and other project utilities\n\nFor most teams, the package is mainly about faster local development and maintenance workflows.\n\n## Version-Sensitive Notes\n\n- The package version covered here and the current latest PyPI release are both `4.1`.\n- The upstream docs URL you were given, `https://django-extensions.readthedocs.io/en/latest/`, is already rendering `4.2` docs. For `4.1`, prefer `https://django-extensions.readthedocs.io/en/stable/`.\n- The upstream README and PyPI metadata for the `4.1` line require `Django 4.2 or later`.\n- The `4.1` release added the `show_permissions` command, added per-app styling support for `graph_models`, and included a fix around `JSONField` and `bulk_update()`.\n- If you copy examples from `latest/`, verify they still exist in `stable/` before assuming they are valid for `4.1`.\n\n## Installation\n\n### pip\n\n```bash\npip install django-extensions==4.1\n```\n\n### uv\n\n```bash\nuv add django-extensions==4.1\n```\n\n### Poetry\n\n```bash\npoetry add django-extensions==4.1\n```\n\n### Base Requirements\n\nFor the `4.1` line:\n\n- Django: `4.2+`\n- Python/Django support policy: upstream says it follows Django-supported Python and Django versions\n\nCommon optional dependencies:\n\n- `werkzeug` for `runserver_plus`\n- `pyOpenSSL` if you want HTTPS cert generation with `runserver_plus --cert-file`\n- Graphviz plus either `pygraphviz` or `pydot` for rendered `graph_models` output\n- `ipython`, `bpython`, or `ptpython` for richer `shell_plus` sessions\n\n## Setup\n\nAdd the app to your Django settings:\n\n```python\nINSTALLED_APPS = [\n # ...\n \"django_extensions\",\n]\n```\n\nThen confirm the extra commands are available:\n\n```bash\npython manage.py help\n```\n\nIf a command needs an optional dependency and it is missing, the command raises an error when you invoke it.\n\n## Core Usage\n\n### `shell_plus`\n\n`shell_plus` starts a Django-aware shell and auto-imports models. It is usually the fastest way to inspect data or prototype ORM queries.\n\n```bash\npython manage.py shell_plus\npython manage.py shell_plus --ipython\npython manage.py shell_plus --plain\npython manage.py shell_plus --print-sql\npython manage.py shell_plus --lab\n```\n\nUseful settings:\n\n```python\nSHELL_PLUS = \"ipython\"\nSHELL_PLUS_PRINT_SQL = True\nSHELL_PLUS_PRINT_SQL_TRUNCATE = 1000\nSHELL_PLUS_DONT_LOAD = [\"sites\", \"blog.pictures\"]\nSHELL_PLUS_MODEL_ALIASES = {\n \"blog\": {\"Messages\": \"blog_messages\"},\n}\nSHELL_PLUS_IMPORTS = [\n \"from myapp.services import build_report\",\n]\n```\n\nUse `SHELL_PLUS_DONT_LOAD`, aliases, or a collision resolver when models from different apps share names.\n\n### `runserver_plus`\n\n`runserver_plus` wraps Django's development server with the Werkzeug debugger.\n\n```bash\npip install werkzeug\npython manage.py runserver_plus\npython manage.py runserver_plus 0.0.0.0:8000\npython manage.py runserver_plus --cert-file /tmp/dev-cert.crt\n```\n\nUseful settings:\n\n```python\nRUNSERVER_PLUS_SERVER_ADDRESS_PORT = \"0.0.0.0:8000\"\nRUNSERVER_PLUS_PRINT_SQL = True\nRUNSERVER_PLUS_PRINT_SQL_TRUNCATE = 1000\nRUNSERVER_PLUS_EXTRA_FILES = []\nRUNSERVER_PLUS_TRUSTED_HOSTS = [\".localhost\", \"127.0.0.1\"]\n```\n\nImportant behavior:\n\n- this debugger can execute code inside traceback frames\n- Werkzeug 3.0.3+ restricts debugger hosts unless `RUNSERVER_PLUS_TRUSTED_HOSTS` allows them\n- this command is for local development only\n\n### `graph_models`\n\n`graph_models` is the quickest way to visualize model relationships or export a diagram during a refactor.\n\n```bash\npython manage.py graph_models -a > my_project.dot\npython manage.py graph_models -a -g -o my_project.png\npython manage.py graph_models users billing auth > domain.dot\npython manage.py graph_models -a --app-style path/to/style.json -o styled.png\n```\n\nBackends:\n\n```bash\npip install pygraphviz\n# or\npip install pyparsing pydot\n```\n\nUseful defaults:\n\n```python\nGRAPH_MODELS = {\n \"all_applications\": True,\n \"group_models\": True,\n \"app_labels\": [\"users\", \"billing\", \"auth\"],\n}\n```\n\nIf image rendering fails, generate plain DOT output first. Most failures come from missing Graphviz system libraries rather than Django itself.\n\n### `runscript`\n\nUse `runscript` for small, repeatable Django-context automation jobs that are too specific for a management command.\n\nProject layout:\n\n```text\nscripts/\n __init__.py\n delete_stale_sessions.py\n```\n\nExample script:\n\n```python\nfrom django.utils import timezone\n\nfrom sessions.models import Session\n\ndef run(*args):\n cutoff = timezone.now()\n Session.objects.filter(expires_at__lt=cutoff).delete()\n```\n\nRun it with:\n\n```bash\npython manage.py runscript delete_stale_sessions\npython manage.py runscript delete_stale_sessions --script-args dry-run\n```\n\n`scripts/__init__.py` is required. Without it, Django will not discover the scripts package.\n\n### Jobs scheduling helpers\n\n`django-extensions` can scaffold and run jobs, but it does not schedule them by itself.\n\nCreate the job layout:\n\n```bash\npython manage.py create_jobs myapp\n```\n\nThat creates a `jobs/` package with directories such as `hourly/`, `daily/`, `weekly/`, `monthly/`, and `yearly/`.\n\nEach job lives in its own module and exposes a `Job` class derived from one of the base job classes.\n\nRun jobs:\n\n```bash\npython manage.py runjob myjob\npython manage.py runjobs hourly\npython manage.py runjobs daily\npython manage.py runjobs -l\n```\n\nTypical cron wiring:\n\n```cron\n@hourly /path/to/project/manage.py runjobs hourly\n@daily /path/to/project/manage.py runjobs daily\n```\n\n### Model and field extensions\n\nIf you want convenience base classes instead of more management commands, start with the model extensions.\n\nExample:\n\n```python\nfrom django.db import models\nfrom django_extensions.db.models import TimeStampedModel, TitleSlugDescriptionModel\n\nclass Article(TimeStampedModel, TitleSlugDescriptionModel):\n body = models.TextField()\n```\n\nCommonly useful pieces:\n\n- `TimeStampedModel` for `created` and `modified`\n- `ActivatorModel` for active/inactive scheduling fields and helpers\n- `TitleDescriptionModel` and `TitleSlugDescriptionModel` for common content patterns\n\nFor new Django `4.2+` projects, prefer Django's built-in `models.JSONField` unless you are maintaining older code that already depends on the extension field.\n\n### `show_permissions`\n\n`show_permissions` is a good `4.1`-specific command to inspect generated model permissions.\n\n```bash\npython manage.py show_permissions\npython manage.py show_permissions blog\npython manage.py show_permissions blog.Post\npython manage.py show_permissions --all\npython manage.py show_permissions --app-label blog\n```\n\nUse it when debugging custom auth flows, role-based access, or unexpected permission rows after migrations.\n\n## Configuration And Auth\n\n- There is no external authentication or API client setup. Configuration is standard Django settings plus optional Python packages for specific commands.\n- Keep `django-extensions` in a development dependency group unless you intentionally use its commands in deployed admin or maintenance environments.\n- Command-specific settings only affect those commands. For example, `SHELL_PLUS_*` settings do not change normal app imports.\n- If you expose anything over the network for local debugging, treat `runserver_plus` as unsafe outside a trusted machine.\n\n## Common Pitfalls\n\n- Package name and Django app name differ: install `django-extensions`, configure `django_extensions`.\n- `latest/` docs are version-drifting. For `4.1`, prefer the `stable/` docs root and the `4.1` release page.\n- `runserver_plus` is not safe for production or shared environments.\n- `graph_models` often fails because Graphviz or the rendering backend is not installed correctly.\n- `runscript` and jobs discovery require real Python packages with `__init__.py` files.\n- Jobs do not run automatically. `create_jobs` only scaffolds files.\n- `shell_plus` can import colliding model names; set aliases or skip modules instead of relying on import order.\n- Some commands are destructive or invasive, such as `reset_db`. Do not use them casually on a non-local database.\n\n## Recommended Agent Workflow\n\n1. Confirm the project's Django version before pinning `django-extensions`.\n2. Install `django-extensions==4.1` and add `\"django_extensions\"` to `INSTALLED_APPS`.\n3. Use `shell_plus` for ORM exploration, `runserver_plus` for local traceback debugging, and `graph_models` for schema inspection.\n4. If a command errors immediately, check for optional dependencies before changing app code.\n5. If docs disagree, prefer the `4.1` PyPI/release metadata and `stable/` docs over `latest/`.\n\n## Official Sources\n\n- Stable docs root: https://django-extensions.readthedocs.io/en/stable/\n- Installation instructions: https://django-extensions.readthedocs.io/en/stable/installation_instructions.html\n- `shell_plus`: https://django-extensions.readthedocs.io/en/latest/shell_plus.html\n- `runserver_plus`: https://django-extensions.readthedocs.io/en/latest/runserver_plus.html\n- `graph_models`: https://django-extensions.readthedocs.io/en/latest/graph_models.html\n- `runscript`: https://django-extensions.readthedocs.io/en/latest/runscript.html\n- Jobs scheduling: https://django-extensions.readthedocs.io/en/latest/jobs_scheduling.html\n- Model extensions: https://django-extensions.readthedocs.io/en/stable/model_extensions.html\n- `show_permissions`: https://django-extensions.readthedocs.io/en/latest/show_permissions.html\n- PyPI release page: https://pypi.org/project/django-extensions/4.1/\n- Upstream repository: https://github.com/django-extensions/django-extensions\n- Upstream releases: https://github.com/django-extensions/django-extensions/releases/tag/4.1\n","content/edubase/docs/api/references/permissions.md":"# Content Permissions\n\nCheck, grant, or revoke permissions on any content type. Permissions control who can view, modify, or manage content in EduBase.\n\n## Supported Content Types\n\nPermissions can be managed for: `class`, `course`, `event`, `exam`, `integration`, `organization`, `quiz`, `scorm`, `tag`, `video`.\n\n## Permission Levels\n\n| Level | Description |\n|-------|-------------|\n| `view` | View the content |\n| `report` | View reports and analytics |\n| `control` | Control settings (e.g. start/pause exams) |\n| `modify` | Edit and modify content |\n| `grant` | Grant permissions to others |\n| `admin` | Full administrative access |\n| `finances` | Financial access (events only) |\n\n## Check Permission\n\n```bash\ncurl -d \"app={app}&secret={secret}&exam={exam_id}&user={user_id}&permission=modify\" \\\n https://www.edubase.net/api/v1/exam:permission\n```\n\nReturns:\n```json\n{\n \"user\": \"...\",\n \"content\": {\n \"type\": \"exam\",\n \"code\": \"...\",\n \"id\": null\n },\n \"status\": {\n \"permission\": true,\n \"rule\": true\n }\n}\n```\n\n- `permission`: User has this permission (directly or inherited)\n- `rule`: A specific permission rule exists with these exact parameters\n\n## Grant Permission\n\n```bash\ncurl -X POST -d \"app={app}&secret={secret}&exam={exam_id}&user={user_id}&permission=modify\" \\\n https://www.edubase.net/api/v1/exam:permission\n```\n\nReturns: `{\"user\":\"...\",\"content\":{...},\"success\":true}`\n\n## Revoke Permission\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&exam={exam_id}&user={user_id}&permission=modify\" \\\n https://www.edubase.net/api/v1/exam:permission\n```\n\nReturns: `{\"user\":\"...\",\"content\":{...},\"success\":true}`\n\n## Transfer Ownership\n\nTransfer full ownership of content to another user:\n\n```bash\ncurl -X POST -d \"app={app}&secret={secret}&exam={exam_id}&user={user_id}\" \\\n https://www.edubase.net/api/v1/exam:transfer\n```\n\nReturns: `{\"user\":\"...\",\"content\":{...},\"success\":true}`\n\n## Endpoint Patterns\n\nAll content types follow the same pattern. Replace `{type}` with the content type:\n\n| Operation | Method | Endpoint |\n|-----------|--------|----------|\n| Check permission | GET | `/{type}:permission` |\n| Grant permission | POST | `/{type}:permission` |\n| Revoke permission | DELETE | `/{type}:permission` |\n| Transfer ownership | POST | `/{type}:transfer` |\n\n### Examples for Different Content Types\n\n```bash\n# Quiz permissions\ncurl -d \"app={app}&secret={secret}&quiz={quiz_id}&user={user_id}&permission=view\" \\\n https://www.edubase.net/api/v1/quiz:permission\n\n# Class permissions\ncurl -X POST -d \"app={app}&secret={secret}&class={class_id}&user={user_id}&permission=report\" \\\n https://www.edubase.net/api/v1/class:permission\n\n# Organization transfer\ncurl -X POST -d \"app={app}&secret={secret}&organization={org_id}&user={user_id}\" \\\n https://www.edubase.net/api/v1/organization:transfer\n\n# SCORM permissions\ncurl -X DELETE -d \"app={app}&secret={secret}&scorm={scorm_id}&user={user_id}&permission=modify\" \\\n https://www.edubase.net/api/v1/scorm:permission\n\n# Video permissions\ncurl -d \"app={app}&secret={secret}&video={video_id}&user={user_id}&permission=admin\" \\\n https://www.edubase.net/api/v1/video:permission\n```\n\n## Required Parameters\n\n| Parameter | Description |\n|-----------|-------------|\n| `{type}` | Content identification string (e.g. `exam`, `quiz`, `class`) |\n| `user` | User identification string |\n| `permission` | Permission level (not required for transfer) |\n","content/edubase/docs/api/references/question-types.md":"# Question Types & Parametric Generation\n\n## Question Types Reference\n\n### Basic Types\n\n**GENERIC** — Strict exact matching including spaces and punctuation. Use for technical answers where precision matters (code keywords, chemical formulas).\n\n**TEXT** — Text input with flexible matching. Ignores spaces and punctuation (` apple.` matches `apple`). Good for fill-in-the-blank, vocabulary.\n\n**NUMERICAL** — Numeric value validation. Handles integers, decimals, fractions (a/b), constants (pi, e). Supports interval responses `{from}-{to}`. Configure precision with `decimals` field (default: 2). Use `tolerance` for acceptable error ranges.\n\n**DATE/TIME** — Calendar date validation. Adjustable precision: year (`YYYY`), month (`MM/YYYY`), day (`MM/DD/YYYY`). Configure with `datetime_precision`. Flexible input parsing — spaces, punctuation, slashes don't matter. Supports BC/AD dates and range responses.\n\n**EXPRESSION** — Mathematical expression evaluation. Compares formulas symbolically/numerically. Supports parametrization. Configure with `expression_check` (`RANDOM`, `EXPLICIT`, `COMPARE`), `expression_variable` (default: `x`), `expression_decimals`, `expression_functions` (`+`/`-` to enable/disable function input).\n\n**FREE-TEXT** — Extended text with semi-automatic grading. Supports keyword-based auto-rules. Falls back to manual grading for complex answers. Use `answer_format=code:python` (or other language) for syntax-highlighted display.\n\n**READING** — Non-assessed text display. No scoring, no answer required. Used as the first question in a group to provide context for subsequent questions.\n\n### Choice-Based Types\n\n**CHOICE** — Single correct answer. Options randomized by default.\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/question\" \\\n --data \"app={app}&secret={secret}\" \\\n --data \"id=CHOICE_EXAMPLE\" \\\n --data \"type=choice\" \\\n --data \"question=What is the capital of France?\" \\\n --data \"answer=Paris\" \\\n --data \"options=London &&& Berlin &&& Madrid\"\n```\n\n**MULTIPLE-CHOICE** — Multiple correct answers. Can limit max selections.\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/question\" \\\n --data \"app={app}&secret={secret}\" \\\n --data \"id=MULTI_CHOICE\" \\\n --data \"type=multiple-choice\" \\\n --data \"question=Which are citrus fruits?\" \\\n --data \"answer=Lemon &&& Orange\" \\\n --data \"options=Apple &&& Banana &&& Grape\"\n```\n\n**TRUE/FALSE** — Statement evaluation. True statements in `answer`, false in `options`. Displayed in random order. Optionally add a third option.\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/question\" \\\n --data \"app={app}&secret={secret}\" \\\n --data \"id=TRUE_FALSE\" \\\n --data \"type=true/false\" \\\n --data \"answer=The Earth orbits the Sun &&& Water boils at 100°C at sea level\" \\\n --data \"options=The Sun orbits the Earth &&& Sound travels faster than light\"\n```\n\n**ORDER** — Sequence arrangement. Items from `answer` displayed randomly; user must arrange in correct order.\n\n### Grouping Types\n\n**GROUPING** — Assign elements to predefined groups. Groups in `answer`, elements in `answer_label` or via triple-arrow (`>>>`) in `answer`.\n\n**PAIRING** — Match elements to pairs. Same syntax as grouping.\n\n### Matrix & Set Types\n\n**MATRIX** — Matrix/vector evaluation with numerical validation. Format: `[a11; a12 | a21; a22]` (semicolons separate columns, pipes separate rows).\n\n**MATRIX:GENERIC** — Matrix with strict exact matching (like GENERIC type). Each element requires exact match including spaces and punctuation.\n\n**MATRIX:EXPRESSION** — Each matrix element evaluated as an expression. For rotation matrices, Jacobians, etc.\n\n**SET** — Unordered collection of numbers. Order and repetition don't matter.\n\n**SET:TEXT** — Unordered collection of text elements.\n\n### Special Types\n\n**HOTSPOT** — Mark areas on an image. Not available to all users.\n\n**FILE** — File upload with semi-automatic grading. Not available to all users.\n\n## Parametric Question Generation\n\nParameters create unique question variants per student. Define in the `parameters` field, separated by `&&&`. Reference in question text and answers with `{name}`.\n\n### Parameter Types\n\n**FIX** — Constant value:\n```\n{pi; FIX; 3.14159}\n```\n\n**INTEGER** — Random whole number:\n```\n{a; INTEGER} # any integer\n{a; INTEGER; 1; 100} # between 1 and 100\n{a; INTEGER; -; -; [10-20]; [14-16]} # 10-20, excluding 14-16\n```\n\n**FLOAT** — Random decimal:\n```\n{x; FLOAT; 2} # 2 decimal places\n{x; FLOAT; 3; 0; 1} # between 0 and 1, 3 decimals\n```\n\n**FORMULA** — Computed from other parameters:\n```\n{d; FORMULA; {b}^2-4*{a}*{c}} # discriminant\n{result; FORMULA; {a}+{b}; 2} # with precision\n```\n\n**LIST** — Random selection from values:\n```\n{animal; LIST; dog; cat; snake; camel}\n```\n\n**PERMUTATION** — Creates indexed sub-parameters `{name_1}`, `{name_2}`, etc., all guaranteed different:\n```\n{primes; PERMUTATION; 2; 3; 5; 7}\n# Use as {primes_1}, {primes_2} — guaranteed to be different values\n```\n\n**FORMAT** — Format another parameter for display:\n```\n{pp; FORMAT; p; NUMBER; 1} # round to 1 decimal\n{pp; FORMAT; p; NUMBERTEXT} # number as text\n{pp; FORMAT; p; ROMAN} # as Roman numeral\n```\n\n### Constraints\n\nEnsure valid parameter combinations:\n\n```bash\n--data \"constraints={b}^2-4*{a}*{c}>0\"\n```\n\nMultiple constraints separated by `&&&`. Allowed relations: `<`, `<=`, `=`, `>=`, `>`, `<>`.\n\nIf constraints are too restrictive and generation frequently fails, EduBase may auto-deactivate the question.\n\n### Syncing LIST Parameters\n\nWhen using multiple LIST parameters that should stay in sync (e.g. country-capital pairs):\n\n```bash\n--data \"parameters={country; LIST; France; Germany; Italy} &&& {capital; LIST; Paris; Berlin; Rome}\" \\\n--data \"parameters_sync=+\"\n```\n\nWith `parameters_sync=+`, if `{country}` picks index 2 (Italy), `{capital}` also picks index 2 (Rome).\n\n### Full Parametric Example\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/question\" \\\n --data \"app={app}&secret={secret}\" \\\n --data \"id=QUADRATIC_DISCRIMINANT\" \\\n --data \"type=numerical\" \\\n --data \"question=For the equation {a}x² + {b}x + {c} = 0, calculate the discriminant.\" \\\n --data \"question_format=LATEX\" \\\n --data \"answer={d}\" \\\n --data \"parameters={a; INTEGER; 1; 5} &&& {b; INTEGER; -10; 10} &&& {c; INTEGER; -10; 10} &&& {d; FORMULA; {b}^2-4*{a}*{c}}\" \\\n --data \"constraints={d}>0\" \\\n --data \"subject=Mathematics\" \\\n --data \"category=Algebra\" \\\n --data \"difficulty=3\" \\\n --data \"hint=The discriminant formula is b²-4ac\" \\\n --data \"solution=D = {b}² - 4·{a}·{c} = {d}\"\n```\n\n## Question Text Formatting\n\n### LaTeX\n\nEnable with `question_format=LATEX`. Inline: `$$...$$`. Block: `$$$$...$$$$`.\n\n```bash\n--data \"question=Calculate $$\\sqrt{x^2 + y^2}$$ for x={a}, y={b}.\" \\\n--data \"question_format=LATEX\"\n```\n\n### EduTags\n\nBold: `[[B]]...[[/B]]`, Italic: `[[I]]...[[/I]]`, Underline: `[[U]]...[[/U]]`, Subscript: `[[SUB]]...[[/SUB]]`, Superscript: `[[SUP]]...[[/SUP]]`.\n\nCode: `[[CODE]]...[[/CODE]]` (inline), `[[CODEBLOCK]]...[[/CODEBLOCK]]` (block), `[[LINES]]...[[/LINES]]` (with line numbers).\n\nColor: `[[COLOR:red]]...[[/COLOR]]`, Background: `[[BACKGROUND:yellow]]...[[/BACKGROUND]]`.\n\n### Tables\n\nUse `[[..]]` format: `[[Header 1; Header 2 | Data 1; Data 2]]` (semicolons = columns, pipes = rows).\n\n### Images\n\n`[[IMAGE:filename.png]]` — file must be provided in the `images` field with that exact name.\n\n### Answer Placeholders\n\n`[[___]]` (three underscores) — visual placeholder for answer fields within question text (fill-in-the-gaps style).\n\n### Quick Expressions\n\nUse triple-wave `~~~...~~~` for inline calculations: `The area is ~~~{r}*{r}*pi~~~`.\n\n## Options Ordering\n\n### Fixed Ordering\n\n`options_fix` field values:\n- `all`: Answers first, then options\n- `abc`: Alphabetical sort of all items\n- `first:N`: Place first N options at the end\n- `last:N`: Place last N options at the end\n- `answers`: Place all answers at the end\n\n### Custom Ordering\n\n`options_order` field — reference items by position:\n\n```bash\n--data \"options_order=OPTION:0 &&& ANSWER:0 &&& OPTION:1 &&& ANSWER:1\"\n```\n\nAll answers and options must be referenced exactly once.\n\n## Expression Evaluation Functions\n\nAvailable in `expression` and `matrix:expression` types, in `FORMULA` parameters, and in `constraints`:\n\n**Basic**: `sqrt(x)`, `abs(x)`, `round(x)`, `floor(x)`, `ceil(x)`\n\n**Logarithmic**: `ln(x)`, `log(x)` (base 10), `log10(x)`\n\n**Trigonometric**: `sin(x)`, `cos(x)`, `tan(x)`, `csc(x)`, `sec(x)`, `arcsin(x)`/`asin(x)`, `arccos(x)`/`acos(x)`, `arctan(x)`/`atan(x)`\n\n**Hyperbolic**: `sinh(x)`, `cosh(x)`, `tanh(x)`, `arcsinh(x)`/`asinh(x)`, `arccosh(x)`/`acosh(x)`, `arctanh(x)`/`atanh(x)`\n\n**Conversions**: `degree2radian(x)`, `radian2degree(x)`, `number2binary(x)`, `binary2number(x)`, `number2roman(x)`, `roman2number(x)`, and octal/hexadecimal variants.\n\n**Two-parameter** (use semicolon separator): `min(a;b)`, `max(a;b)`, `mod(n;i)`, `fmod(n;i)`, `div(a;b)`, `intdiv(a;b)`\n\n## Additional Question Fields\n\n### Numerical & Date Validation\n\n| Field | Applicable Types | Description |\n|-------|------------------|-------------|\n| `decimals` | NUMERICAL, MATRIX, SET | Number of decimal places (default: 2) |\n| `tolerance` | NUMERICAL, MATRIX, SET | Validation tolerance. Values: `ABSOLUTE:N` (e.g. ±0.1), `RELATIVE:N%` (e.g. ±5%), `QUOTIENT` (integer multiple), `QUOTIENT2` (scalar multiple) |\n| `numerical_range` | NUMERICAL | Enable interval answers `{from}-{to}` with `+` |\n| `datetime_precision` | DATE/TIME | Granularity: `YEAR`, `MONTH`, `DAY` (default) |\n| `datetime_range` | DATE/TIME | Enable date range answers with `+` |\n\n### Choice & True/False Options\n\n| Field | Applicable Types | Description |\n|-------|------------------|-------------|\n| `maximum_choices` | MULTIPLE-CHOICE | Limit maximum number of selections |\n| `truefalse_third_options` | TRUE/FALSE | Add a third option (e.g. \"Cannot determine\") |\n| `truefalse_third_options_label` | TRUE/FALSE | Label for the third option |\n\n### Free-Text & File Options\n\n| Field | Applicable Types | Description |\n|-------|------------------|-------------|\n| `freetext_characters` | FREE-TEXT | Character limit for response |\n| `freetext_words` | FREE-TEXT | Word limit for response |\n| `freetext_rules` | FREE-TEXT | Auto-grading keyword rules |\n| `file_count` | FILE | Number of files allowed |\n| `file_types` | FILE | Allowed file extensions |\n\n### Hotspot Options\n\n| Field | Applicable Types | Description |\n|-------|------------------|-------------|\n| `hotspot_image` | HOTSPOT | Image for marking areas |\n| `hotspot_zones` | HOTSPOT | Zone definitions for valid areas |\n\n### Organization & Metadata\n\n| Field | Description |\n|-------|-------------|\n| `label` | Instance-level categorization (predefined by administrators) |\n| `tags` | User-defined tags, separated by `&&&` |\n| `ai` | Set to any value to mark question as AI-generated |\n| `group` | Question group name (when uploading to Quiz set) |","content/edubase/docs/api/references/quiz-sets.md":"# Quiz Sets\n\nCreate and manage Quiz sets (question collections) and assign questions to them.\n\n## List Quiz Sets\n\n```bash\ncurl -d \"app={app}&secret={secret}\" \\\n https://www.edubase.net/api/v1/quizes\n```\n\nReturns owned and managed Quiz sets. Supports `search`, `limit` (default: 16), and `page` parameters.\n\nOutput per quiz: `quiz` (ID), `id` (external ID if set), `name`.\n\n## Get Quiz Set\n\n```bash\ncurl -d \"app={app}&secret={secret}&quiz={quiz_id}\" \\\n https://www.edubase.net/api/v1/quiz\n```\n\nReturns: `quiz`, `id` (external), `name`.\n\n## Create Quiz Set\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/quiz\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"title=Introduction to Physics\" \\\n --data \"description=Basic physics concepts quiz\" \\\n --data \"mode=TEST\" \\\n --data \"type=set\"\n# Returns: {\"quiz\":\"...\"}\n```\n\n| Field | Required | Description |\n|-------|----------|-------------|\n| `title` | Yes | Quiz set title |\n| `language` | No | Quiz language |\n| `id` | No | External unique identifier (max 64 chars) |\n| `description` | No | Short description |\n| `copy_settings` | No | Quiz ID to copy settings from |\n| `copy_questions` | No | Quiz ID to copy questions from |\n| `mode` | No | `TEST` (all questions at once, default) or `TURNS` (one at a time) |\n| `type` | No | `set` (practice), `exam` (examination), `private` (testing) |\n\n## Delete Quiz Set\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&quiz={quiz_id}\" \\\n https://www.edubase.net/api/v1/quiz\n```\n\n## List Questions in Quiz\n\n```bash\ncurl -d \"app={app}&secret={secret}&quiz={quiz_id}\" \\\n https://www.edubase.net/api/v1/quiz:questions\n```\n\nReturns list of questions and question groups:\n\n```json\n[\n {\"question\": \"...\", \"id\": \"MATH_001\", \"active\": true},\n {\"group\": \"Advanced Problems\", \"active\": true},\n {\"question\": \"...\", \"id\": null, \"active\": false}\n]\n```\n\n## Assign Questions to Quiz\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/quiz:questions\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"quiz={quiz_id}\" \\\n --data \"questions=q1,q2,q3\"\n```\n\nTo assign to a specific question group within the quiz:\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/quiz:questions\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"quiz={quiz_id}\" \\\n --data \"group=Advanced Problems\" \\\n --data \"questions=q4,q5\"\n```\n\n## Remove Questions from Quiz\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&quiz={quiz_id}&questions=q1,q2\" \\\n https://www.edubase.net/api/v1/quiz:questions\n```\n\nTo remove from a specific group:\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&quiz={quiz_id}&group=Advanced Problems&questions=q4\" \\\n https://www.edubase.net/api/v1/quiz:questions\n```\n\n## Question Groups\n\nQuestion groups organize questions within a Quiz set. They support three modes:\n\n- **Random selection**: Randomly pick N questions from the group\n- **Sequential**: Questions appear in order\n- **Complex (mixed)**: All questions appear as one \"big\" question with sub-parts\n\nWhen uploading questions via the API, use the `group` field to assign them to a group:\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/question\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"id=PHYSICS_MOTION_Q1\" \\\n --data \"type=numerical\" \\\n --data \"question=Calculate velocity given distance {d}m and time {t}s.\" \\\n --data \"answer={d}/{t}\" \\\n --data \"parameters={d; INTEGER; 10; 100} &&& {t; INTEGER; 1; 10}\" \\\n --data \"group=Kinematics Questions\"\n```\n\nIf the group doesn't exist, it's created automatically as a complex task. Configure group settings (type, question count, scoring) through the EduBase UI.\n\n### Use Cases\n\n**Balanced difficulty distribution:**\nCreate groups by difficulty, randomly select from each to ensure consistent exam difficulty.\n\n**Reading comprehension:**\nGroup a passage (READING type) with its comprehension questions (CHOICE/TEXT types) to keep them together.\n\n**Multi-part problems:**\nGroup related sub-questions that share parameters or context.\n","content/edubase/docs/api/references/scoring.md":"# Scoring & Grading\n\n## Points\n\nSet maximum points with `points` field (default: 1). For questions with multiple answers, points are distributed according to the subscoring method.\n\n## Partial Credit (Subscoring)\n\nThe `subscoring` field controls how partial credit is calculated. Not applicable for `choice`, `reading`, and `free-text` types.\n\n**PROPORTIONAL** (default) — Points awarded proportionally to correct answers.\n\n**LINEAR_SUBSTRACTED:N** — Linear scoring with N points subtracted per error.\n\n```bash\n--data \"subscoring=LINEAR_SUBSTRACTED:2\"\n```\n\n**CUSTOM** — Custom point distribution. Requires `subpoints` field with percentages separated by `&&&`.\n\n```bash\n# First answer worth 50%, second 25%, third 25%\n--data \"points=4\" \\\n--data \"subscoring=CUSTOM\" \\\n--data \"subpoints=50 &&& 25 &&& 25\"\n# Yields: 2 points for first, 1 for second, 1 for third\n```\n\n**NONE** — All-or-nothing. No partial credit.\n\n## Penalty Scoring\n\n### Penalty Methods\n\n`penalty_scoring` field:\n- `DEFAULT`: Standard penalty behavior (varies by type)\n- `PER_ANSWER`: Penalty applied for each incorrect answer\n- `PER_QUESTION`: Penalty applied once per question\n\n### Penalty Points\n\n`penalty_points` — Points deducted for completely incorrect answers. No penalty for partially correct or unanswered questions. Use positive values.\n\n```bash\n--data \"penalty_scoring=PER_ANSWER\" \\\n--data \"penalty_points=2\"\n```\n\n## Assistance Penalties\n\n### Hint Penalties\n\n`hint_penalty` field:\n- `NONE`: No penalty (default)\n- `ONCE:N`: Single deduction regardless of hints used (N as percentage)\n- `PER-HELP:N`: Deduction per hint used\n\n```bash\n# 10% penalty per hint used\n--data \"hint_penalty=PER-HELP:10%\"\n# or equivalently\n--data \"hint_penalty=PER-HELP:0.1\"\n```\n\n### Solution Penalties\n\n`solution_penalty` — Same format as hint_penalty. Applied when student views solution steps.\n\n```bash\n# 50% penalty for viewing solution (any number of steps)\n--data \"solution_penalty=ONCE:50%\"\n```\n\n### Video Penalties\n\n`video_penalty` — Same format as hint_penalty, except `PER-HELP` not available.\n\n```bash\n--data \"video_penalty=ONCE:15%\"\n```\n\n## Manual Scoring\n\n`manual_scoring` field (not applicable for `reading` and `free-text`):\n- `NO`: Never (default)\n- `NOT_CORRECT`: Only manually score incorrect answers\n- `ALWAYS`: Always require manual scoring\n\n## Full Scoring Example\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/question\" \\\n --data \"app={app}&secret={secret}\" \\\n --data \"id=SCORED_PROBLEM\" \\\n --data \"type=expression\" \\\n --data \"question=Find the area of a circle with radius {r}.\" \\\n --data \"answer=pi*{r}^2\" \\\n --data \"parameters={r; INTEGER; 2; 10}\" \\\n --data \"points=10\" \\\n --data \"hint=Think about the formula for circle area &&& Remember that area involves squaring the radius\" \\\n --data \"solution=The formula for circle area is pi*r^2 = pi*{r}^2\" \\\n --data \"penalty_scoring=PER_ANSWER\" \\\n --data \"penalty_points=3\" \\\n --data \"hint_penalty=PER-HELP:10%\" \\\n --data \"solution_penalty=ONCE:50%\"\n```\n\nThis configuration:\n- Awards up to 10 points for a correct expression\n- Deducts 3 points for a completely wrong answer\n- Each hint used costs 10% of 10 = 1 point\n- Viewing solution costs 50% of 10 = 5 points (once, regardless of steps viewed)","content/edubase/docs/api/references/tagging.md":"# Content Tagging\n\nTags organize and categorize content in EduBase. Create tags via the EduBase UI, then attach them to content via API.\n\n## Supported Content Types\n\nTags can be attached to: `class`, `course`, `event`, `exam`, `integration`, `organization`, `quiz`, `scorm`, `video`.\n\n## List Tags\n\n```bash\ncurl -d \"app={app}&secret={secret}\" https://www.edubase.net/api/v1/tags\n```\n\nReturns: `[{\"tag\":\"...\",\"id\":null,\"name\":\"...\"}, ...]`\n\nSupports `search`, `limit` (default: 16), and `page` parameters.\n\n## Get Tag Details\n\n```bash\ncurl -d \"app={app}&secret={secret}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/tag\n```\n\nReturns:\n```json\n{\n \"tag\": \"...\",\n \"id\": null,\n \"name\": \"Mathematics\",\n \"color\": \"#FF5733\",\n \"icon\": \"fa-book\"\n}\n```\n\n## List Tags on Content\n\n```bash\ncurl -d \"app={app}&secret={secret}&exam={exam_id}\" \\\n https://www.edubase.net/api/v1/exam:tags\n```\n\nReturns: `[{\"tag\":\"...\",\"name\":\"...\"}, ...]`\n\n## Check Tag Attachment\n\n```bash\ncurl -d \"app={app}&secret={secret}&exam={exam_id}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/exam:tag\n```\n\nReturns:\n```json\n{\n \"tag\": \"...\",\n \"content\": {\n \"type\": \"exam\",\n \"code\": \"...\",\n \"id\": null\n },\n \"status\": true\n}\n```\n\n## Attach Tag to Content\n\n```bash\ncurl -X POST -d \"app={app}&secret={secret}&exam={exam_id}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/exam:tag\n```\n\nReturns: `{\"tag\":\"...\",\"content\":{...},\"success\":true}`\n\n## Detach Tag from Content\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&exam={exam_id}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/exam:tag\n```\n\nReturns: `{\"tag\":\"...\",\"content\":{...},\"success\":true}`\n\n## Endpoint Patterns\n\nAll content types follow the same pattern. Replace `{type}` with the content type:\n\n| Operation | Method | Endpoint |\n|-----------|--------|----------|\n| List tags on content | GET | `/{type}:tags` |\n| Check tag attachment | GET | `/{type}:tag` |\n| Attach tag | POST | `/{type}:tag` |\n| Detach tag | DELETE | `/{type}:tag` |\n\n### Examples for Different Content Types\n\n```bash\n# List tags on a quiz\ncurl -d \"app={app}&secret={secret}&quiz={quiz_id}\" \\\n https://www.edubase.net/api/v1/quiz:tags\n\n# Attach tag to class\ncurl -X POST -d \"app={app}&secret={secret}&class={class_id}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/class:tag\n\n# Check tag on organization\ncurl -d \"app={app}&secret={secret}&organization={org_id}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/organization:tag\n\n# Detach tag from video\ncurl -X DELETE -d \"app={app}&secret={secret}&video={video_id}&tag={tag_id}\" \\\n https://www.edubase.net/api/v1/video:tag\n\n# List tags on SCORM\ncurl -d \"app={app}&secret={secret}&scorm={scorm_id}\" \\\n https://www.edubase.net/api/v1/scorm:tags\n```\n\n## Required Parameters\n\n| Parameter | Description |\n|-----------|-------------|\n| `{type}` | Content identification string (e.g. `exam`, `quiz`, `class`) |\n| `tag` | Tag identification string |\n","content/edubase/docs/api/references/users.md":"# Users\n\nCreate and manage user accounts, generate login links, and assume user identity for API operations.\n\n## List Users\n\n```bash\ncurl -d \"app={app}&secret={secret}\" \\\n https://www.edubase.net/api/v1/users\n```\n\nReturns managed, non-generated users. Supports `search`, `limit` (default: 16), and `page` parameters.\n\n## Get User\n\n```bash\n# Get current user (API owner)\ncurl -d \"app={app}&secret={secret}\" \\\n https://www.edubase.net/api/v1/user:me\n\n# Get specific user\ncurl -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user\n```\n\nReturns: `user` (ID), `name`, `status` (enabled/disabled), `exam` (whether it's an exam-only account).\n\n## Create User\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/user\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"username=jsmith\" \\\n --data \"first_name=John\" \\\n --data \"last_name=Smith\" \\\n --data \"email=john.smith@example.com\"\n# Returns: {\"user\":\"...\",\"username\":\"jsmith\",\"password\":\"...\"}\n```\n\n### Required Fields\n\n| Field | Description |\n|-------|-------------|\n| `username` | 4-64 characters |\n| `first_name` | 1-64 characters |\n| `last_name` | 1-64 characters |\n| `email` | Valid email address |\n\n### Optional Fields\n\n| Field | Description |\n|-------|-------------|\n| `password` | 4-64 chars (auto-generated if omitted) |\n| `full_name` | Override auto-generated full name |\n| `display_name` | Override display name |\n| `phone` | Format: `+prefix number` (e.g. `+1 1234567890`) |\n| `gender` | `male`, `female`, `other` |\n| `birthdate` | `YYYY-MM-DD` |\n| `exam` | `true` for exam-only accounts (default: `false`) |\n| `group` | User group name |\n| `template` | Account template: `corporate_test`, `corporate_user`, `corporate_reporter`, `corporate_supervisor`, `corporate_teacher`, `corporate_admin`, `streaming` |\n| `language` | ISO 639-1 code (default: API owner's language) |\n| `timezone` | Timezone (default: API owner's timezone) |\n| `color` | Favorite color: `default`, `branding`, `red`, `blue`, `yellow`, `green`, `purple`, `gray` |\n| `must_change_password` | `true` to force password change on first login |\n| `notify` | `true` to send welcome email/SMS |\n| `custom_{field}` | Custom field data (if configured for instance) |\n\n## Update User\n\n```bash\n# Enable/disable user\ncurl -X PATCH -d \"app={app}&secret={secret}&user={user_id}&active=false\" \\\n https://www.edubase.net/api/v1/user\n```\n\n## Delete User\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user\n```\n\n## User Name\n\n```bash\n# Get name\ncurl -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user:name\n\n# Update name\ncurl -X POST \"https://www.edubase.net/api/v1/user:name\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"user={user_id}\" \\\n --data \"first_name=Jane\" \\\n --data \"last_name=Doe\"\n```\n\n## User Group\n\n```bash\n# Get group\ncurl -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user:group\n\n# Update group\ncurl -X POST -d \"app={app}&secret={secret}&user={user_id}&group=teachers\" \\\n https://www.edubase.net/api/v1/user:group\n```\n\n## Search User\n\nLook up users by email, username, or code:\n\n```bash\ncurl -d \"app={app}&secret={secret}&query=john@example.com\" \\\n https://www.edubase.net/api/v1/user:search\n# Returns: {\"user\":\"...\",\"exam\":false}\n```\n\n## Login Links\n\nGenerate magic login links for users. Links can be single-use or reusable.\n\n### Get Existing Link\n\n```bash\ncurl -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user:login\n```\n\n### Generate New Link\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/user:login\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"user={user_id}\" \\\n --data \"expires=7\" \\\n --data \"logins=1\"\n# Returns: {\"user\":\"...\",\"url\":\"...\",\"valid\":\"...\",\"count\":1}\n```\n\n| Field | Description |\n|-------|-------------|\n| `expires` | Days (1-30) or date `YYYY-MM-DD` (default: 1 day) |\n| `logins` | Max uses, up to 255 (default: unlimited) |\n| `redirect` | URI path or `[{content_type}:{tag}]` to redirect after login |\n| `exam` | Exam ID to redirect user to (user must be on exam) |\n| `template` | Login link template identifier |\n| `short` | `true` for shortened eduba.se link (if enabled) |\n\n### Invalidate Link\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&user={user_id}&url={login_url}\" \\\n https://www.edubase.net/api/v1/user:login\n```\n\n## Assume User\n\nPerform API operations as a different user. Useful for administrative tasks.\n\n### Request Assume Token\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/user:assume\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"user={user_id}\"\n# Returns: {\"user\":\"...\",\"token\":\"...\",\"valid\":\"...\"}\n```\n\nThe `user` field accepts: user identification string, username, or email address. Optionally include `password` or user secret for additional verification.\n\n### Use Assume Token\n\nInclude the token in subsequent requests:\n\n```bash\ncurl -d \"app={app}&secret={secret}&assume={token}\" \\\n https://www.edubase.net/api/v1/exams\n```\n\n### Revoke Assume Token\n\nAlways revoke tokens when done:\n\n```bash\ncurl -X DELETE -d \"app={app}&secret={secret}&token={assume_token}\" \\\n https://www.edubase.net/api/v1/user:assume\n```\n\n## User Organizations\n\n```bash\n# List user's organizations\ncurl -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user:organizations\n\n# Add user to organizations\ncurl -X POST \"https://www.edubase.net/api/v1/user:organizations\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"user={user_id}\" \\\n --data \"organizations=org1,org2\" \\\n --data \"permission_organization=teacher\" \\\n --data \"permission_content=modify\"\n\n# Remove user from organizations\ncurl -X DELETE -d \"app={app}&secret={secret}&user={user_id}&organizations=org1,org2\" \\\n https://www.edubase.net/api/v1/user:organizations\n```\n\n## User Classes\n\n```bash\n# List user's classes\ncurl -d \"app={app}&secret={secret}&user={user_id}\" \\\n https://www.edubase.net/api/v1/user:classes\n\n# Add user to classes\ncurl -X POST -d \"app={app}&secret={secret}&user={user_id}&classes=cls1,cls2&expires=30\" \\\n https://www.edubase.net/api/v1/user:classes\n\n# Remove user from classes\ncurl -X DELETE -d \"app={app}&secret={secret}&user={user_id}&classes=cls1,cls2\" \\\n https://www.edubase.net/api/v1/user:classes\n```\n","content/edubase/docs/api/references/webhooks.md":"# Webhooks\n\nWebhooks notify external systems about events in EduBase. Configured per organization.\n\n## Create a Webhook\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/organization:webhook\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"organization={org_id}\" \\\n --data \"name=Exam Results Webhook\" \\\n --data \"trigger_event=exam-play-result\" \\\n --data \"endpoint=https://example.com/webhook/edubase\" \\\n --data \"method=POST\" \\\n --data \"authentication=key\" \\\n --data \"authentication_send=bearer\" \\\n --data \"authentication_key=my_secret_token\" \\\n --data \"retry=error\"\n```\n\nReturns: `{\"organization\":\"...\",\"webhook\":\"...\"}`\n\n## Trigger Events\n\n| Event | Description |\n|-------|-------------|\n| `exam-play-result` | User (org member) completes an exam |\n| `quiz-play-result` | User (org member) completes a quiz in practice mode |\n| `api` | Manual trigger via API — useful for testing |\n\n## Authentication Options\n\n`authentication` field:\n- `none`: No authentication (default)\n- `key`: Use a secret key/password\n\nWhen `authentication=key`, configure how to send it:\n\n`authentication_send` field:\n- `header`: As a custom header (`authentication_send_header` specifies header name)\n- `bearer`: As Bearer token in Authorization header\n- `data`: As a data field (`authentication_send_data` specifies field name)\n\n```bash\n# Bearer token\n--data \"authentication=key\" \\\n--data \"authentication_send=bearer\" \\\n--data \"authentication_key=mysecret\"\n\n# Custom header\n--data \"authentication=key\" \\\n--data \"authentication_send=header\" \\\n--data \"authentication_send_header=X-Webhook-Secret\" \\\n--data \"authentication_key=mysecret\"\n\n# Data field\n--data \"authentication=key\" \\\n--data \"authentication_send=data\" \\\n--data \"authentication_send_data=webhook_token\" \\\n--data \"authentication_key=mysecret\"\n```\n\n## Additional Options\n\n- `extra_data`: JSON string sent with every notification\n- `retry`: `none` (no retry) or `error` (delayed retry on failure, default)\n\n## Manage Webhooks\n\n```bash\n# Get webhook details\ncurl -d \"app={app}&secret={secret}&organization={org_id}&webhook={webhook_id}\" \\\n https://www.edubase.net/api/v1/organization:webhook\n\n# Enable/disable\ncurl -X PATCH -d \"app={app}&secret={secret}&organization={org_id}&webhook={webhook_id}&active=false\" \\\n https://www.edubase.net/api/v1/organization:webhook\n\n# Delete\ncurl -X DELETE -d \"app={app}&secret={secret}&organization={org_id}&webhook={webhook_id}\" \\\n https://www.edubase.net/api/v1/organization:webhook\n```\n\n## Test a Webhook\n\nTrigger an `api`-type webhook manually with optional custom payload:\n\n```bash\ncurl -X POST \"https://www.edubase.net/api/v1/organization:webhook:trigger\" \\\n --data \"app={app}\" \\\n --data \"secret={secret}\" \\\n --data \"organization={org_id}\" \\\n --data \"webhook={webhook_id}\" \\\n --data 'data={\"test\": true, \"message\": \"Hello from EduBase\"}'\n```\n\nOnly triggers webhooks where `trigger_event` is set to `api`.","content/flutter/skills/state-management/bloc-cubit/SKILL.md":"---\nname: bloc-cubit\ndescription: \"Use when working with Flutter Bloc/Cubit state management. Covers when to choose Bloc vs Cubit, how to use bloc and flutter_bloc together, lifecycle, testing, and safe defaults.\"\nmetadata:\n revision: 1\n updated-on: \"2026-04-04\"\n source: official\n tags: \"flutter,dart,bloc,cubit,flutter_bloc,state-management,testing,architecture\"\n---\n\n# Flutter Bloc/Cubit State Management\n\nUse this skill when building Flutter state management with `bloc` and `flutter_bloc`.\n\n## Core Rule\n\n- Use `Cubit` for simple, direct state updates.\n- Use `Bloc` for event-driven flows, transitions, and replayable business logic.\n- Use `bloc` in Dart-only projects.\n- Use `flutter_bloc` in Flutter apps when you need widgets like `BlocProvider`, `BlocBuilder`, or `BlocListener`.\n- If you install `flutter_bloc`, you do not need to add `bloc` separately in a Flutter app because `flutter_bloc` depends on it.\n\n## Decision Guide\n\nChoose `Cubit` when:\n- state changes are simple method calls\n- you do not need events\n- the feature is local and low complexity\n- examples include counters, toggles, filters, form flags, and theme mode\n\nChoose `Bloc` when:\n- user actions should be modeled as explicit events\n- the flow has loading, success, and failure transitions\n- the logic benefits from clear state machines\n- examples include auth, pagination, checkout, sync, and multi-step workflows\n\n## Required Project Setup\n\nFor Dart-only code:\n- add `bloc`\n- do not add `flutter_bloc` unless Flutter widgets are needed\n\nFor Flutter UI code:\n- add `flutter_bloc`\n- let it bring `bloc` transitively\n- use `BlocProvider` at the feature boundary\n- use `BlocBuilder` for rebuilds and `BlocListener` for side effects\n\n## Implementation Pattern\n\nPrefer this structure:\n- repository or service owns I/O\n- bloc/cubit owns state and orchestration\n- UI only dispatches actions and renders state\n\nKeep state immutable.\nKeep events explicit when using `Bloc`.\nKeep one bloc or cubit per feature responsibility.\n\n## Lifecycle Rules\n\n- Close manually created blocs and cubits with `close()`.\n- Do not manually close instances owned by `BlocProvider`.\n- Use `BlocProvider` or `MultiBlocProvider` to let Flutter manage disposal.\n- If you create a bloc/cubit with `new` or a constructor outside the widget tree, you own its lifecycle.\n\n## UI Binding Rules\n\nUse `BlocBuilder` when the widget should rebuild from state.\nUse `BlocListener` when the widget should react without rebuilding.\nUse `BlocConsumer` only when both are needed in one place.\nUse `buildWhen` and `listenWhen` when rebuilds or listeners need narrowing.\nUse `BlocSelector` when only one field should drive rebuilds.\n\n## Testing Rules\n\n- Test `Cubit` by calling methods and asserting emitted states.\n- Test `Bloc` by adding events and asserting the transition sequence.\n- Mock repositories at the boundary, not inside the bloc logic.\n- Test side effects separately from rendering logic.\n\n## Common Pitfalls\n\n- Do not put network calls directly in widgets.\n- Do not use `Bloc` for trivial local state.\n- Do not add both `bloc` and `flutter_bloc` in a Flutter app when only `flutter_bloc` is needed.\n- Do not forget `close()` for manually managed instances.\n- Do not emit duplicate states unless the transition is meaningful.\n- Do not let a single bloc grow into an app-wide dumping ground.\n\n## What To Prefer In Answers\n\nWhen writing code or advising on design:\n- show the smallest working Bloc or Cubit first\n- mention why Bloc or Cubit was chosen\n- mention whether the dependency should be `bloc` or `flutter_bloc`\n- include cleanup and testing notes if lifecycle is manual\n- keep examples aligned with the current Flutter state management docs and the bloc package docs\n\n## Minimal Reference Checklist\n\n- `bloc` = core logic package\n- `flutter_bloc` = Flutter UI integration package\n- `Cubit` = method-based updates\n- `Bloc` = event-based transitions\n- manual creation = manual `close()`\n- provider-owned instance = no manual `close()`\n","content/flutter/skills/state-management/riverpod/SKILL.md":"---\nname: riverpod\ndescription: \"Use when working with Flutter Riverpod state management. Covers providers, consumers, refs, containers, overrides, async state, code generation, testing, and safe defaults.\"\nmetadata:\n revision: 1\n updated-on: \"2026-04-04\"\n source: official\n tags: \"flutter,dart,riverpod,flutter_riverpod,providers,async,testing,codegen\"\n---\n\n# Flutter Riverpod State Management\n\nUse this skill when building Flutter state management with `riverpod` and `flutter_riverpod`.\n\n## Core Rule\n\n- Use `Provider` for read-only values and dependency wiring.\n- Use `StateProvider` for small mutable state.\n- Use `FutureProvider` for one-shot async loading.\n- Use `StreamProvider` for reactive streams.\n- Use `Notifier` or `AsyncNotifier` for feature state that needs mutation methods.\n- Use `autoDispose` for short-lived state.\n- Use `family` when provider output depends on an argument.\n- Use `flutter_riverpod` in Flutter apps.\n- If you use `flutter_riverpod`, you usually do not add `riverpod` separately because the Flutter package brings it in transitively.\n- Current stable line is Riverpod 3.x (`riverpod` 3.2.1, `flutter_riverpod` 3.3.1).\n\n## Decision Guide\n\nChoose Riverpod when:\n- you want provider-based dependency injection and reactive state in one model\n- state should be derived from other providers\n- async data is a major part of the feature\n- test overrides and scoped state matter\n- you want fewer classes than a Bloc-heavy approach\n\nChoose a different pattern when:\n- the feature is a tiny local state toggle and a simple widget would be enough\n- the logic is better expressed as an explicit event machine\n\n## Required Project Setup\n\nFor Dart-only code:\n- add `riverpod`\n\nFor Flutter UI code:\n- add `flutter_riverpod`\n- wrap the app in `ProviderScope`\n- use `ConsumerWidget`, `Consumer`, or `ConsumerStatefulWidget` where `ref` is needed\n\nIf code generation is used:\n- add `riverpod_annotation`\n- add `riverpod_generator`\n- add `build_runner`\n\n## Implementation Pattern\n\nPrefer this structure:\n- providers own state, dependencies, and derived values\n- repositories and APIs stay outside widgets\n- UI reads state with `ref.watch`\n- UI performs imperative reads with `ref.read`\n- container-level overrides are used for tests, demos, and environment-specific wiring\n\nKeep providers small and composable.\nKeep state immutable unless the provider type is specifically meant for mutation.\nKeep feature logic in providers or notifiers, not in widgets.\n\n## Lifecycle Rules\n\n- Use `autoDispose` for short-lived screens and transient queries.\n- Use `ref.keepAlive()` only when you want cached state to survive temporarily.\n- Prefer provider disposal over manual cleanup.\n- In tests and pure Dart code, dispose `ProviderContainer` when finished.\n\n## UI Binding Rules\n\nUse `ConsumerWidget` when the whole widget depends on providers.\nUse `Consumer` when only a small subtree needs access to `ref`.\nUse `ConsumerStatefulWidget` when you need widget lifecycle plus `ref`.\nUse `select` to reduce rebuilds.\nUse `ProviderListener` or `ref.listen` for side effects.\n\n## Testing Rules\n\n- Test providers through `ProviderContainer`.\n- Override dependencies instead of mocking provider internals.\n- Assert `AsyncValue` states directly for async providers.\n- Dispose containers in tests.\n\n## Common Pitfalls\n\n- Do not call `ref.watch` inside callbacks; use `ref.read` there.\n- Do not skip `ProviderScope` at the app root.\n- Do not use `StateProvider` for large feature state.\n- Do not put business logic in widgets.\n- Do not rebuild the whole tree when a single field changes; use `select`.\n- Do not model async loading manually when `FutureProvider` or `AsyncNotifier` fits.\n- Do not forget provider overrides for tests and previews.\n\n## What To Prefer In Answers\n\nWhen writing code or advising on design:\n- show the smallest working provider first\n- mention whether `ProviderScope` is required\n- mention whether the provider should be `autoDispose` or cached\n- mention whether a `family` is needed for arguments\n- include test override notes when dependencies are external\n- keep examples aligned with the current Riverpod docs and Flutter integration patterns\n- if code generation is involved, prefer matching `riverpod_annotation` / `riverpod_generator` versions from the same 3.x release line\n\n## Minimal Reference Checklist\n\n- `ProviderScope` = app root setup\n- `ref.watch` = rebuild on change\n- `ref.read` = imperative access\n- `select` = narrower rebuilds\n- `autoDispose` = short-lived state\n- `family` = parameterized provider\n- `ProviderContainer` = test/headless container\n","content/gemini/docs/live/javascript/references/advanced.md":"# Gemini Live API — Advanced Topics (JavaScript/TypeScript)\n\n## Raw WebSocket API (No SDK)\n\nFor environments where the SDK is unavailable, connect directly via WebSocket.\n\n### Endpoint\n\n```\nwss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY\n```\n\n### Setup Message (first message after connect)\n\n```json\n{\n \"config\": {\n \"model\": \"models/gemini-3.1-flash-live-preview\",\n \"responseModalities\": [\"AUDIO\"],\n \"systemInstruction\": {\n \"parts\": [{\"text\": \"You are a helpful assistant.\"}]\n }\n }\n}\n```\n\n### Sending Audio\n\n```json\n{\n \"realtimeInput\": {\n \"audio\": {\n \"data\": \"\",\n \"mimeType\": \"audio/pcm;rate=16000\"\n }\n }\n}\n```\n\n### Sending Text\n\n```json\n{ \"realtimeInput\": { \"text\": \"Hello, how are you?\" } }\n```\n\n### Tool Response\n\n```json\n{\n \"toolResponse\": {\n \"functionResponses\": [\n { \"name\": \"func_name\", \"id\": \"call_id\", \"response\": { \"result\": \"ok\" } }\n ]\n }\n}\n```\n\n### Server Message Fields\n\n| Field | Description |\n|---|---|\n| `serverContent.modelTurn.parts[].inlineData.data` | Base64 audio |\n| `serverContent.inputTranscription.text` | User speech transcription |\n| `serverContent.outputTranscription.text` | Model speech transcription |\n| `serverContent.interrupted` | User interrupted the model |\n| `serverContent.turnComplete` | Model finished its turn |\n| `serverContent.generationComplete` | Generation fully complete |\n| `toolCall.functionCalls[]` | Function calls (name, id, args) |\n| `goAway.timeLeft` | Time before server disconnects |\n| `sessionResumptionUpdate.newHandle` | Handle for session resumption |\n\n## Ephemeral Tokens\n\nShort-lived tokens for direct browser-to-API connections. Created server-side\n(typically in Python or Node.js backend), passed to the browser client.\n\n### Browser WebSocket Endpoint (with ephemeral token)\n\n```\nwss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token={token}\n```\n\n### Browser Usage Example\n\n```typescript\n// Token received from your backend (endpoint must be authenticated + rate-limited)\nconst ephemeralToken = await fetch('/api/gemini-token').then(r => r.json());\n\nconst ws = new WebSocket(\n `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token=${ephemeralToken.token}`\n);\n\nws.onopen = () => {\n ws.send(JSON.stringify({\n config: {\n model: 'models/gemini-3.1-flash-live-preview',\n responseModalities: ['AUDIO'],\n },\n }));\n};\n```\n\nTokens can be locked to specific configurations with `liveConnectConstraints`\nfor additional security.\n\n## Async Function Calling (Gemini 2.5 Flash Live Only)\n\n**Not supported on Gemini 3.1 Flash Live.** Allows the model to continue\ninteracting while a function runs in the background.\n\nSet `behavior: 'NON_BLOCKING'` on the function declaration:\n\n```typescript\ntools: [{\n functionDeclarations: [{\n name: 'slowLookup',\n description: 'A slow database lookup',\n parameters: { type: 'object', properties: { query: { type: 'string' } } },\n behavior: 'NON_BLOCKING',\n }],\n}]\n```\n\nWhen sending the function response, set `scheduling`:\n\n| Scheduling | Behavior |\n|---|---|\n| `INTERRUPT` | Model interrupts current output to address result |\n| `WHEN_IDLE` | Model addresses result after finishing current output |\n| `SILENT` | Model absorbs result silently for later use |\n\n## Model Feature Comparison\n\n| Feature | 3.1 Flash Live | 2.5 Flash Live |\n|---|---|---|\n| Thinking config | `thinkingLevel` (minimal/low/medium/high) | `thinkingBudget` (token count) |\n| Response events | Single event, multiple parts | Each event, one part |\n| `sendClientContent` | Initial history only | Throughout conversation |\n| Turn coverage default | `TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO` | `TURN_INCLUDES_ONLY_ACTIVITY` |\n| Async function calling | No | Yes (`NON_BLOCKING`) |\n| Proactive audio (v1alpha) | No | Yes |\n| Affective dialog (v1alpha) | No | Yes |\n| Context window | 128k tokens | 32k tokens |\n","content/gemini/docs/live/python/references/advanced.md":"# Gemini Live API — Advanced Topics (Python)\n\n## Raw WebSocket API (No SDK)\n\nFor environments where the SDK is unavailable, connect directly via WebSocket.\n\n### Endpoint\n\n```\nwss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY\n```\n\n### Setup Message (first message after connect)\n\n```json\n{\n \"config\": {\n \"model\": \"models/gemini-3.1-flash-live-preview\",\n \"responseModalities\": [\"AUDIO\"],\n \"systemInstruction\": {\n \"parts\": [{\"text\": \"You are a helpful assistant.\"}]\n }\n }\n}\n```\n\n### Sending Audio\n\n```json\n{\n \"realtimeInput\": {\n \"audio\": {\n \"data\": \"\",\n \"mimeType\": \"audio/pcm;rate=16000\"\n }\n }\n}\n```\n\n### Sending Text\n\n```json\n{ \"realtimeInput\": { \"text\": \"Hello, how are you?\" } }\n```\n\n### Tool Response\n\n```json\n{\n \"toolResponse\": {\n \"functionResponses\": [\n { \"name\": \"func_name\", \"id\": \"call_id\", \"response\": { \"result\": \"ok\" } }\n ]\n }\n}\n```\n\n### Server Message Fields\n\nResponses are `BidiGenerateContentServerMessage` JSON:\n\n| Field | Description |\n|---|---|\n| `serverContent.modelTurn.parts[].inlineData.data` | Base64 audio |\n| `serverContent.inputTranscription.text` | User speech transcription |\n| `serverContent.outputTranscription.text` | Model speech transcription |\n| `serverContent.interrupted` | User interrupted the model |\n| `serverContent.turnComplete` | Model finished its turn |\n| `serverContent.generationComplete` | Generation fully complete |\n| `toolCall.functionCalls[]` | Function calls (name, id, args) |\n| `goAway.timeLeft` | Time before server disconnects |\n| `sessionResumptionUpdate.newHandle` | Handle for session resumption |\n| `usageMetadata.totalTokenCount` | Token usage |\n\n## Ephemeral Tokens\n\nShort-lived tokens for direct browser-to-API connections without exposing the\nAPI key. Created server-side, passed to the client.\n\n```python\nimport datetime\nfrom google import genai\n\n# Must use v1alpha for ephemeral tokens\nclient = genai.Client(http_options={\"api_version\": \"v1alpha\"})\n\nnow = datetime.datetime.now(tz=datetime.timezone.utc)\ntoken = client.auth_tokens.create(config={\n \"uses\": 1,\n \"expire_time\": now + datetime.timedelta(minutes=30),\n \"new_session_expire_time\": now + datetime.timedelta(minutes=1),\n \"http_options\": {\"api_version\": \"v1alpha\"},\n})\n\n# token.name is the ephemeral token string\n# Pass to client for WebSocket connection:\n# wss://...BidiGenerateContentConstrained?access_token={token.name}\n```\n\nTokens can be locked to specific configurations with `live_connect_constraints`\nfor additional security.\n\n### Ephemeral Token WebSocket Endpoint\n\n```\nwss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token={token}\n```\n\n## Async Function Calling (Gemini 2.5 Flash Live Only)\n\n**Not supported on Gemini 3.1 Flash Live.** Allows the model to continue\ninteracting while a function executes in the background.\n\nSet `behavior: \"NON_BLOCKING\"` on the function declaration:\n\n```python\ntools = [{\n \"function_declarations\": [{\n \"name\": \"slow_lookup\",\n \"description\": \"A slow database lookup\",\n \"parameters\": {\"type\": \"object\", \"properties\": {\"query\": {\"type\": \"string\"}}},\n \"behavior\": \"NON_BLOCKING\",\n }]\n}]\n```\n\nWhen sending the `FunctionResponse`, set a `scheduling` field:\n\n| Scheduling | Behavior |\n|---|---|\n| `INTERRUPT` | Model interrupts current output to address result |\n| `WHEN_IDLE` | Model addresses result after finishing current output |\n| `SILENT` | Model absorbs result silently for later use |\n\n## Model Feature Comparison\n\n| Feature | 3.1 Flash Live | 2.5 Flash Live |\n|---|---|---|\n| Thinking config | `thinkingLevel` (minimal/low/medium/high) | `thinkingBudget` (token count) |\n| Response events | Single event, multiple parts | Each event, one part |\n| `send_client_content` | Initial history only | Throughout conversation |\n| Turn coverage default | `TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO` | `TURN_INCLUDES_ONLY_ACTIVITY` |\n| Async function calling | No | Yes (`NON_BLOCKING`) |\n| Proactive audio (v1alpha) | No | Yes (`proactive_audio: true`) |\n| Affective dialog (v1alpha) | No | Yes (`enable_affective_dialog: true`) |\n| Context window | 128k tokens | 32k tokens |\n\n## Session Limits\n\n| Constraint | Value |\n|---|---|\n| Audio-only session | 15 min |\n| Audio + video session | 2 min |\n| WebSocket connection | ~10 min (use session resumption) |\n| Context window (native audio) | 128k tokens |\n| Audio token accumulation | ~25 tokens/second |\n| Supported languages | 97 |\n","content/grpcio/docs/tools/python/DOC.md":"---\nname: tools\ndescription: \"grpcio-tools package guide for generating Python protobuf modules and gRPC stubs from .proto files\"\nmetadata:\n languages: \"python\"\n versions: \"1.78.0\"\n revision: 1\n updated-on: \"2026-03-12\"\n source: maintainer\n tags: \"grpcio-tools,grpc,protobuf,protoc,python,codegen\"\n---\n\n# grpcio-tools Python Package Guide\n\nUse `grpcio-tools` to compile `.proto` files into Python protobuf message modules and gRPC service stubs. The PyPI package name is `grpcio-tools`, but the command module you run is `grpc_tools`.\n\n## Golden Rule\n\n- Use `grpcio-tools` for code generation, not as the gRPC runtime.\n- Generate all Python outputs from the same command: `--python_out`, `--pyi_out`, and `--grpc_python_out`.\n- Install the runtime packages separately in any environment that imports the generated files: `protobuf` for `*_pb2.py` and `grpcio` for `*_pb2_grpc.py`.\n\n## Install\n\nCode generation only:\n\n```bash\npython -m pip install \"grpcio-tools==1.78.0\"\n```\n\nRuntime dependencies for applications that import generated code:\n\n```bash\npython -m pip install \"protobuf\" \"grpcio==1.78.0\"\n```\n\nIf one environment both generates and runs the code, install all three:\n\n```bash\npython -m pip install \"protobuf\" \"grpcio==1.78.0\" \"grpcio-tools==1.78.0\"\n```\n\n## Initialization And Auth\n\n`grpcio-tools` is a local compiler wrapper around `protoc` and the Python gRPC plugin. It does not create a client, does not open network connections, and has no package-specific authentication or environment-variable setup.\n\nAuthentication and channel setup belong to the generated gRPC client or server you run with `grpcio`.\n\n## Minimal `.proto` File\n\n```proto\nsyntax = \"proto3\";\n\npackage helloworld;\n\nservice Greeter {\n rpc SayHello(HelloRequest) returns (HelloReply) {}\n}\n\nmessage HelloRequest {\n string name = 1;\n}\n\nmessage HelloReply {\n string message = 1;\n}\n```\n\nSave this as `protos/helloworld.proto`.\n\n## Generate Python Code\n\nSet an explicit proto root and output directory so the generated imports stay predictable:\n\n```bash\nexport PROTO_ROOT=./protos\nexport PY_OUT=./src\n\npython -m grpc_tools.protoc \\\n -I\"$PROTO_ROOT\" \\\n --python_out=\"$PY_OUT\" \\\n --pyi_out=\"$PY_OUT\" \\\n --grpc_python_out=\"$PY_OUT\" \\\n \"$PROTO_ROOT/helloworld.proto\"\n```\n\nThis writes:\n\n- `src/helloworld_pb2.py` for protobuf messages and descriptors\n- `src/helloworld_pb2.pyi` for typing information\n- `src/helloworld_pb2_grpc.py` for the gRPC stub, servicer base class, and server registration helper\n\n## Use The Generated Files\n\nAfter generation, import the message module and the gRPC helper module together:\n\n```python\nimport helloworld_pb2\nimport helloworld_pb2_grpc\n```\n\nThe gRPC helper module imports the protobuf module, so both files need to stay importable on the same Python module path.\n\n## Minimal Server\n\nThis requires `grpcio` in the runtime environment.\n\n```python\nfrom concurrent import futures\n\nimport grpc\n\nimport helloworld_pb2\nimport helloworld_pb2_grpc\n\n\nclass Greeter(helloworld_pb2_grpc.GreeterServicer):\n def SayHello(self, request, context):\n return helloworld_pb2.HelloReply(message=f\"Hello, {request.name}\")\n\n\ndef serve() -> None:\n server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))\n helloworld_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)\n server.add_insecure_port(\"[::]:50051\")\n server.start()\n server.wait_for_termination()\n\n\nif __name__ == \"__main__\":\n serve()\n```\n\n## Minimal Client\n\nUse an environment variable for the target address so the client can move between local development and deployed environments without code changes:\n\n```bash\nexport GREETER_ADDR=localhost:50051\n```\n\n```python\nimport os\n\nimport grpc\n\nimport helloworld_pb2\nimport helloworld_pb2_grpc\n\n\ntarget = os.environ.get(\"GREETER_ADDR\", \"localhost:50051\")\n\nwith grpc.insecure_channel(target) as channel:\n stub = helloworld_pb2_grpc.GreeterStub(channel)\n reply = stub.SayHello(helloworld_pb2.HelloRequest(name=\"Ada\"))\n print(reply.message)\n```\n\nFor production traffic, switch to `grpc.secure_channel(...)` and the credential helpers documented in the gRPC auth guide.\n\n## Common Pitfalls\n\n- `pip install grpcio-tools`, but run `python -m grpc_tools.protoc`.\n- `grpcio-tools` does not run RPC clients or servers by itself. Generated `*_pb2_grpc.py` files expect the `grpcio` runtime.\n- `*_pb2.py` and `*_pb2_grpc.py` are a pair. If one moves or lands in a different import root, the generated imports break.\n- Output paths matter. Run generation from a stable working directory and keep `-I` and the output directories aligned with your Python package layout.\n- If your project needs package-prefixed generated imports, use the custom `-Ipackage=...` mapping documented in the Python basics tutorial instead of patching the generated files manually.\n\n## Version Notes\n\n- This entry covers PyPI package version `1.78.0`.\n- The gRPC documentation site and the PyPI package page can move on different patch cadences. If you are debugging generated-code differences, compare the installed package version with the version shown in the hosted Python docs.\n\n## Official Sources\n\n- gRPC Python landing page: https://grpc.io/docs/languages/python/\n- gRPC Python basics tutorial: https://grpc.io/docs/languages/python/basics/\n- gRPC Python generated-code reference: https://grpc.io/docs/languages/python/generated-code/\n- gRPC Python API landing page: https://grpc.io/docs/languages/python/api/\n- Canonical hosted Python API docs: https://grpc.github.io/grpc/python/\n- PyPI package page: https://pypi.org/project/grpcio-tools/\n","content/landingai/skills/ade/document-extraction/SKILL.md":"---\nname: document-extraction\ndescription: Use this skill for intelligent document processing and content extraction using LandingAI's Agentic Document Extraction (ADE). Trigger when users need to (1) Parse documents (PDFs, images, spreadsheets, presentations) into structured Markdown with layout understanding, (2) Extract specific structured data from documents using schemas (invoice fields, form data, table data, etc.), (3) Classify and separate multi-document batches by type (invoices vs receipts, statements vs forms, etc.), (4) Process large documents asynchronously (up to 1GB/1000 pages), (5) Get visual grounding (bounding boxes, page numbers) for extracted content — use when users mention bounding boxes, word locations, grounding, highlighting extracted content, or showing where data appears in a document. Use this skill when the task involves understanding document content for a set of documents. In particular this skill can help you write code that run on sets of documents. This will increase speed, and reduce the cost of loading the documents on the Agent context window because you can use a single script to extract the information needed.\n---\n\n# Document Extraction (ADE)\n\n## Overview\n\nLandingAI's Agentic Document Extraction (ADE) is a document processing SaaS that parses, extracts, and classifies documents without requiring templates or training. It provides three main capabilities:\n\n1. **Parse**: Convert documents into structured Markdown with hierarchical JSON representation\n2. **Extract**: Pull specific structured data using JSON schemas or Pydantic models\n3. **Split**: Classify and separate multi-document batches by type\n\n**Key Benefits:**\n- No ML training or templates required\n- Layout-agnostic parsing (works with any document structure)\n- Supports 20+ file formats (PDF, images, spreadsheets, presentations)\n- Precise visual grounding (bounding boxes, page numbers)\n- Multiple models optimized for different document types\n\n## Quick Start\n\n### 1. Installation\n\nNever install packages globally without user approval. Always check for a local Python environment first.\n\n```\n1. .venv/bin/python — uv-managed (this project)\n2. venv/bin/python — standard Python venv\n3. uv run python — if pyproject.toml exists\n4. poetry run python — if poetry.lock exists\n5. python3 — system fallback; warn the user\n```\nUse the local environment to install: `landingai-ade`, `python-dotenv`\n\n### 2. API Key Setup\n\nThe user may have already setup a `.env` file in the same directory as the `document-extraction` skill with the API key. You MUST check this path first (ls -la .*/skills/document-extraction/.env). Also try checking on the same directory as this SKILL.md file.\n\nIf not, provide instructions to create one. The script below will search for `.env` in common locations and load it.\n\n```bash\n.venv/bin/python - << 'EOF'\nimport os\nfrom pathlib import Path\nfrom dotenv import load_dotenv\n\n# Load API key: prefer existing env var, then .env file lookup\nif os.environ.get(\"VISION_AGENT_API_KEY\"):\n print(\"API key found in existing environment variable\")\nelse:\n def _find_env():\n for d in [Path.cwd().resolve(), *Path.cwd().resolve().parents]:\n for candidate in [\n # ADD the directory where the document-extraction skill is located\n d / '.env',\n d / 'document-extraction/.env',\n d / 'skills/document-extraction/.env',\n ]:\n if candidate.is_file():\n return candidate\n return None\n env = _find_env()\n if env:\n load_dotenv(env)\n print(f\"API key loaded from: {env}\")\n else:\n print(\"Warning: VISION_AGENT_API_KEY not set and no .env found\")\nEOF\n```\n\nIf not key is found instruct the user to get an API key from [https://va.landing.ai/settings/api-key](https://va.landing.ai/settings/api-key)\n\nCopy `.env-sample` to `.env` and add your API key:\n\n```bash\ncp .env-sample .env\n```\n\nEdit `.env` and add your key:\n```\nVISION_AGENT_API_KEY=your_actual_api_key_here\n```\n\n**Note:** The `.env` file is gitignored for security. Advanced users can also set the environment variable directly: `export VISION_AGENT_API_KEY=`\n\n**EU Endpoint:** If using the EU endpoint, set `environment=\"eu\"` when initializing the client.\n\n### 3. Basic Parse Example\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv() # Load API key from .env\n\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\n# Parse a document\nresponse = client.parse(\n document=Path(\"document.pdf\"),\n model=\"dpt-2-latest\"\n)\n\n# Access results\nprint(f\"Pages: {response.metadata.page_count}\")\nprint(f\"Chunks: {len(response.chunks)}\")\nprint(\"\\nMarkdown output:\")\nprint(response.markdown[:500]) # First 500 chars\n\n# Save Markdown for extraction\nwith open(\"output.md\", \"w\", encoding=\"utf-8\") as f:\n f.write(response.markdown)\n```\n\n### 4. Basic Extract Example\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\nfrom pydantic import BaseModel, Field\nfrom pathlib import Path\n\n# Define extraction schema using Pydantic\nclass Invoice(BaseModel):\n invoice_number: str = Field(description=\"Invoice number\")\n invoice_date: str = Field(description=\"Invoice date\")\n total_amount: float = Field(description=\"Total amount in USD\")\n vendor_name: str = Field(description=\"Vendor name\")\n\n# Convert to JSON schema\nschema = pydantic_to_json_schema(Invoice)\n\nclient = LandingAIADE()\n\n# Extract from parsed markdown\nresponse = client.extract(\n schema=schema,\n markdown=Path(\"output.md\"), # From parse step\n model=\"extract-latest\"\n)\n\n# Access extracted data\nprint(response.extraction)\n# Output: {'invoice_number': 'INV-12345', 'invoice_date': '2024-01-15', ...}\n\n# Check extraction metadata (traceability)\nprint(response.extraction_metadata)\n```\n\n## Document Parsing\n\n### Parse Local Files\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\nresponse = client.parse(\n document=Path(\"/path/to/document.pdf\"),\n model=\"dpt-2-latest\"\n)\n\n# Work with chunks\nfor chunk in response.chunks:\n print(f\"Type: {chunk.type}, Page: {chunk.grounding.page}\")\n print(f\"Content: {chunk.markdown[:100]}...\")\n```\n\n### Parse Remote URLs\n\n```python\nresponse = client.parse(\n document_url=\"https://example.com/document.pdf\",\n model=\"dpt-2-latest\"\n)\n```\n\n### Parse Spreadsheets\n\nSpreadsheets (CSV, XLSX) return a **different response type** than documents. Key differences:\n\n| Field | Documents (`ParseResponse`) | Spreadsheets (`SpreadsheetParseResponse`) |\n|---|---|---|\n| `metadata.page_count` | ✓ | ✗ (uses `sheet_count`, `total_rows`, `total_cells`, `total_chunks`, `total_images`) |\n| `splits[].pages` | ✓ | ✗ (uses `sheets` — array of sheet indices) |\n| `grounding` (top-level) | ✓ | ✗ (not present for spreadsheets) |\n| Chunk grounding | Always present | Optional (null for table chunks, present for embedded image chunks) |\n\n```python\nresponse = client.parse(\n document=Path(\"data.xlsx\"),\n model=\"dpt-2-latest\"\n)\n\n# Spreadsheet metadata\nprint(f\"Sheets: {response.metadata.sheet_count}\")\nprint(f\"Total rows: {response.metadata.total_rows}\")\nprint(f\"Total cells: {response.metadata.total_cells}\")\n\n# Splits use 'sheets' instead of 'pages'\nfor split in response.splits:\n print(f\"Sheet indices: {split.sheets}\")\n print(f\"Markdown: {split.markdown[:200]}...\")\n```\n\n### Model Selection\n\nChoose the right model for your documents:\n\n| Model | Best For | Chunk Types |\n|-------|----------|-------------|\n| **dpt-2-latest** | Complex documents with logos, signatures, ID cards | text, table, figure, logo, card, attestation, scan_code, marginalia |\n| **dpt-2-mini** | Simple, digitally-native documents (faster, cheaper) | text, table, figure, marginalia |\n| **dpt-1** | ⚠️ **Deprecated March 31, 2026** — migrate to dpt-2 | text, table, figure, marginalia |\n\n**Recommendation:** Use `dpt-2-latest` unless you have simple documents where cost/speed is critical.\n\n**Version Pinning:** For production, use dated versions (e.g., `dpt-2-20260302`) for reproducibility.\n\n### Parse Large Files (Async)\n\nFor files up to 1 GB or 6,000 pages, use Parse Jobs:\n\n```python\nimport time\nfrom dotenv import load_dotenv\nload_dotenv()\n\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\n# Step 1: Create parse job\njob = client.parse_jobs.create(\n document=Path(\"large_document.pdf\"),\n model=\"dpt-2-latest\"\n)\n\njob_id = job.job_id\nprint(f\"Job {job_id} created\")\n\n# Step 2: Poll for completion\nwhile True:\n response = client.parse_jobs.get(job_id)\n if response.status == \"completed\":\n print(f\"Job {job_id} completed\")\n break\n print(f\"Progress: {response.progress * 100:.0f}%\")\n time.sleep(5)\n\n# Step 3: Access results\n# Results are in response.data (or response.output_url for large results)\nif response.data:\n print(f\"Chunks: {len(response.data.chunks)}\")\n with open(\"output.md\", \"w\", encoding=\"utf-8\") as f:\n f.write(response.data.markdown)\nelif response.output_url:\n # Results > 1MB are returned as a presigned URL\n print(f\"Download results from: {response.output_url}\")\n```\n\n**Job Status Response Fields:**\n- `job_id`, `status` (pending, processing, completed, failed, cancelled), `progress` (0-1)\n- `data`: The `ParseResponse` (or `SpreadsheetParseResponse`) when complete and result < 1MB\n- `output_url`: Presigned S3 URL when result > 1MB or when `output_save_url` was used. Expires after 1 hour; a new URL is generated on each GET.\n- `metadata`: Same as sync parse (`filename`, `page_count`, `duration_ms`, etc.)\n- `failure_reason`: Error message if job failed\n\n### Zero Data Retention (ZDR)\n\nIf ZDR is enabled for your organization, you must provide an `output_save_url` where parsed results will be saved. The results will not be returned in the API response. ZDR is not enabled by default. Typically `output_save_url` is a presigned url with write permissions to your S3 bucket, but you can also use other storage solutions that support file uploads via HTTP PUT requests.\n\n```python\njob = client.parse_jobs.create(\n document=Path(\"sensitive_document.pdf\"),\n model=\"dpt-2-latest\",\n output_save_url=\"https://your-bucket.s3.amazonaws.com/output.json\"\n)\n```\n\n### List Parse Jobs\n\nList all async parse jobs with optional pagination and status filtering:\n\n```python\n# List recent jobs\njobs_response = client.parse_jobs.list(page=0, page_size=10)\nfor job in jobs_response.jobs:\n print(f\"{job.job_id}: {job.status} ({job.progress:.0%})\")\n\n# Filter by status\ncompleted = client.parse_jobs.list(status=\"completed\", page_size=5)\nprint(f\"Completed jobs: {len(completed.jobs)}, more: {completed.has_more}\")\n```\n\n**Available status filters:** `pending`, `processing`, `completed`, `failed`, `cancelled`\n\n### Understanding Parse Outputs\n\nParse returns a `ParseResponse` with:\n\n- **`markdown`**: Complete document in Markdown with HTML anchor tags\n- **`chunks`**: Array of extracted elements (each with unique ID, type, content, and per-chunk grounding)\n- **`grounding`**: Dictionary mapping element IDs to detailed location data (page, bounding box, grounding type, and table cell position). See [JSON Response](#json-response) for structure.\n- **`metadata`**: Processing info — `filename`, `org_id`, `page_count`, `duration_ms`, `credit_usage` (float), `job_id`, `version`, `failed_pages`\n- **`splits`**: Array of split objects grouping chunks. Always present — contains a single `\"full\"` split by default, or per-page splits if `split=\"page\"` was used. **Note:** Parse splits use a `class` field (values: `\"full\"` or `\"page\"`), which is different from the Split API's `classification` field.\n\n**Common chunk types**: `text`, `table`, `figure`, `logo`, `card`, `attestation`, `scan_code`, `marginalia`\n\nFor detailed chunk type reference, see [references/chunk-types.md](references/chunk-types.md)\n\n> **Anchor tag prefix in `chunk.markdown`:** Every chunk's `markdown` field\n> is prefixed with an HTML anchor tag embedding the chunk UUID:\n> `\\n\\nActual content…`. This is how the full document\n> markdown links back to individual chunks. Strip it before string matching,\n> display, or RAG indexing:\n>\n> ```python\n> import re\n> _ANCHOR_RE = re.compile(r\"]*>\\s*\", re.IGNORECASE)\n>\n> def chunk_text(ch) -> str:\n> \"\"\"Return clean chunk markdown without the anchor prefix.\"\"\"\n> return _ANCHOR_RE.sub(\"\", ch.markdown or \"\").strip()\n>\n> # Example: fingerprint match against a section of the full markdown\n> intro_chunks = [ch for ch in response.chunks\n> if chunk_text(ch)[:80] in intro_markdown]\n> ```\n\n### Saving Parse Responses\n\nThe SDK provides a built-in `save_to` parameter on `parse()`, `extract()`, and `split()` that automatically saves the JSON response to a folder:\n\n```python\nfrom pathlib import Path\n\n# Parse and auto-save response JSON to output/ folder\nresponse = client.parse(\n document=Path(\"document.pdf\"),\n model=\"dpt-2-latest\",\n save_to=\"output/\" # Creates output/document_parse_output.json\n)\n\n# Response is still returned normally for immediate use\nprint(response.markdown[:200])\n```\n\nThe `save_to` parameter:\n- Creates the folder if it doesn't exist\n- Names the file `{input_filename}_{method}_output.json` (e.g., `document_parse_output.json`)\n- Works on `client.parse()`, `client.extract()`, and `client.split()`\n- Is a **client-side convenience** — it saves the full response locally after the API call\n\nFor manual serialization (e.g., custom filenames or selective saving), use `model_dump()`:\n\n```python\nimport json\n\nresponse_dict = response.model_dump()\nwith open(\"parse_response.json\", \"w\", encoding=\"utf-8\") as f:\n json.dump(response_dict, f, indent=2, ensure_ascii=False)\n\n# Save markdown separately for extraction\nwith open(\"document_parsed.md\", \"w\", encoding=\"utf-8\") as f:\n f.write(response.markdown)\n```\n\n**Important:** Always use `model_dump()` to serialize the complete response. Do not manually construct dictionaries with selected fields, as you may miss important data like the `splits` array or complete grounding information.\n\n### Parse Parameters\n\n```python\nresponse = client.parse(\n document=Path(\"document.pdf\"),\n model=\"dpt-2-latest\",\n split=\"page\", # Optional: organize chunks by page\n password=\"secret\", # Optional: decrypt protected files (ZDR only)\n save_to=\"output/\", # Optional: auto-save response JSON\n)\n```\n\n### Parse Password-Protected Files\n\nOrganizations with [Zero Data Retention (ZDR)](https://docs.landing.ai/ade/zdr) enabled can parse password-protected files by passing the `password` parameter. Supported formats: PDF, DOC, DOCX, ODT, PPT, PPTX, XLSX.\n\n```python\n# Sync parse\nresponse = client.parse(\n document=Path(\"encrypted.pdf\"),\n password=\"document_password\",\n model=\"dpt-2-latest\"\n)\n\n# Async parse jobs\njob = client.parse_jobs.create(\n document=Path(\"encrypted.pdf\"),\n password=\"document_password\",\n model=\"dpt-2-latest\"\n)\n```\n\n> **Note:** Without ZDR the API returns HTTP 422. If the password is wrong the API\n> returns HTTP 422 with a decryption error. The parameter is ignored for unencrypted documents.\n\n## Structured Data Extraction\n\n### Schema Definition\n\nDefine what to extract using JSON Schema or Pydantic models.\n\n**Pydantic approach (recommended for Python):**\n\n```python\nfrom pydantic import BaseModel, Field\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nclass BankStatement(BaseModel):\n account_holder: str = Field(description=\"Account holder name\")\n account_number: str = Field(description=\"Account number\")\n beginning_balance: float = Field(description=\"Beginning balance in USD\")\n ending_balance: float = Field(description=\"Ending balance in USD\")\n\nschema = pydantic_to_json_schema(BankStatement)\n```\n\n**JSON Schema approach:**\n\n```python\nschema = {\n \"type\": \"object\",\n \"properties\": {\n \"account_holder\": {\n \"type\": \"string\",\n \"description\": \"Account holder name\"\n },\n \"account_number\": {\n \"type\": \"string\",\n \"description\": \"Account number\"\n },\n \"beginning_balance\": {\n \"type\": \"number\",\n \"description\": \"Beginning balance in USD\"\n },\n \"ending_balance\": {\n \"type\": \"number\",\n \"description\": \"Ending balance in USD\"\n }\n },\n \"required\": [\"account_holder\", \"account_number\"]\n}\n```\n\n### Extraction Workflow\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\n# Step 1: Parse document\nparse_response = client.parse(\n document=Path(\"bank_statement.pdf\"),\n model=\"dpt-2-latest\"\n)\n\n# Save markdown\nwith open(\"parsed.md\", \"w\", encoding=\"utf-8\") as f:\n f.write(parse_response.markdown)\n\n# Step 2: Extract structured data\nextract_response = client.extract(\n schema=schema, # Your JSON schema\n markdown=Path(\"parsed.md\"),\n model=\"extract-latest\"\n)\n\n# Access extracted data\nprint(extract_response.extraction)\n\n# Check traceability (which chunks provided each field)\nfor field, metadata in extract_response.extraction_metadata.items():\n print(f\"{field}: from chunks {metadata.chunk_ids}\")\n```\n\n### Extract from URL\n\nYou can extract from a remotely-hosted Markdown file using `markdown_url`:\n\n```python\nextract_response = client.extract(\n schema=schema,\n markdown_url=\"https://example.com/parsed_document.md\",\n model=\"extract-latest\"\n)\n```\n\n### Common Schema Patterns\n\nFor detailed schema patterns, see [references/extraction-schemas.md](references/extraction-schemas.md)\n\n**Nested objects:**\n```python\nclass Address(BaseModel):\n street: str\n city: str\n zip_code: str\n\nclass Invoice(BaseModel):\n invoice_number: str\n billing_address: Address # Nested object\n```\n\n**Arrays (lists):**\n```python\nclass LineItem(BaseModel):\n description: str\n quantity: int\n amount: float\n\nclass Invoice(BaseModel):\n invoice_number: str\n line_items: list[LineItem] # Array of objects\n```\n\n**Enums (restricted values):**\n```python\nclass BankStatement(BaseModel):\n account_type: str = Field(\n description=\"Account type\",\n enum=[\"Checking\", \"Savings\"] # Only these values allowed\n )\n```\n\n**Nullable fields:**\n```python\nclass Patient(BaseModel):\n first_name: str\n middle_name: str | None = Field(default=None) # Optional field\n last_name: str\n```\n\n### Document Classification\n\nClassify documents before extracting type-specific fields:\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nfrom landingai_ade import LandingAIADE\nfrom pydantic import BaseModel, Field\nfrom landingai_ade.lib import pydantic_to_json_schema\nfrom pathlib import Path\n\n# Step 1: Define classification schema\nclass DocumentType(BaseModel):\n document_type: str = Field(\n description=\"Document classification\",\n enum=[\"Invoice\", \"Receipt\", \"Bank Statement\", \"Other\"]\n )\n\nclient = LandingAIADE()\n\n# Step 2: Parse document\nparse_response = client.parse(\n document=Path(\"document.pdf\"),\n model=\"dpt-2-latest\"\n)\n\n# Step 3: Classify document\nclassification_schema = pydantic_to_json_schema(DocumentType)\nclassification_response = client.extract(\n schema=classification_schema,\n markdown=parse_response.markdown,\n model=\"extract-latest\"\n)\n\ndoc_type = classification_response.extraction[\"document_type\"]\nprint(f\"Classified as: {doc_type}\")\n\n# Step 4: Extract based on type\nif doc_type == \"Invoice\":\n schema = pydantic_to_json_schema(InvoiceSchema)\nelif doc_type == \"Receipt\":\n schema = pydantic_to_json_schema(ReceiptSchema)\nelse:\n print(\"Unsupported document type\")\n exit()\n\n# Extract type-specific fields\nextract_response = client.extract(\n schema=schema,\n markdown=parse_response.markdown,\n model=\"extract-latest\"\n)\n```\n\n## Document Classification & Splitting\n\n### When to Use Split API\n\nUse the Split API when you have multi-document batches on single file that need to be separated:\n\n- Financial services: Separate bank statements, utility bills, ID documents\n- Healthcare: Split intake forms, medical reports, medication lists\n- Accounting: Separate multiple invoices and receipts\n- Academic: Separate article bodies from references\n\n### Split Classification\n\nDefine how to classify and separate documents using `split_class`:\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\n# Step 1: Parse multi-document PDF\nparse_response = client.parse(\n document=Path(\"batch.pdf\"),\n model=\"dpt-2-latest\"\n)\n\n# Step 2: Define split classes\nsplit_classes = [\n {\n \"name\": \"Invoice\",\n \"description\": \"Commercial invoices with itemized charges\",\n \"identifier\": \"Invoice Number\" # Separate by invoice number\n },\n {\n \"name\": \"Receipt\",\n \"description\": \"Payment receipts showing transaction details\",\n \"identifier\": \"Receipt Date\"\n },\n {\n \"name\": \"Bank Statement\",\n \"description\": \"Monthly bank account statements\"\n }\n]\n\n# Step 3: Split document\nsplit_response = client.split(\n markdown=parse_response.markdown,\n split_class=split_classes\n)\n\n# Step 4: Process each split\nfor split in split_response.splits:\n print(f\"Type: {split.classification}\")\n print(f\"Identifier: {split.identifier}\")\n print(f\"Pages: {split.pages}\")\n print(f\"Content: {split.markdowns[0][:200]}...\")\n```\n\n**Split Class Components:**\n- **name** (required): Document classification label (e.g., \"Invoice\")\n- **description** (optional): Context for classification (more detail = better accuracy)\n- **identifier** (optional): Field that makes each instance unique (creates separate split per unique value)\n- **Limit:** Maximum 19 split classes per request\n\n**Split from URL:** You can also split from a remotely-hosted Markdown file:\n\n```python\nsplit_response = client.split(\n markdown_url=\"https://example.com/parsed_document.md\",\n split_class=split_classes\n)\n```\n\n## Output Formats\n\n### Markdown\n\nADE converts documents to structured Markdown:\n\n```markdown\n# Document Title\n\n## Section 1\n\nParagraph text...\n\n| Column 1 | Column 2 |\n|----------|----------|\n| Data 1 | Data 2 |\n\n<::Caption: Bar chart showing quarterly revenue::>\n```\n\n**Features:**\n- HTML anchor tags for traceability (link to chunk IDs)\n- Special delimiters for visual elements: `<::Caption: description::>`\n- HTML tables for spreadsheet data\n- Preserved structure and hierarchy\n\n### JSON Response\n\nParse returns structured JSON with five top-level fields:\n\n```json\n{\n \"markdown\": \"# Document...\",\n \"chunks\": [\n {\n \"id\": \"7d58c5cf-e4f5-4a7e-ba34-0cd7bc6a6506\",\n \"type\": \"text\",\n \"markdown\": \"Content...\",\n \"grounding\": {\n \"page\": 0,\n \"box\": { \"left\": 0.1, \"top\": 0.2, \"right\": 0.9, \"bottom\": 0.3 }\n }\n }\n ],\n \"splits\": [\n {\n \"class\": \"full\",\n \"identifier\": \"full\",\n \"pages\": [0],\n \"markdown\": \"# Document...\",\n \"chunks\": [\"7d58c5cf-e4f5-4a7e-ba34-0cd7bc6a6506\"]\n }\n ],\n \"grounding\": {\n \"7d58c5cf-e4f5-4a7e-ba34-0cd7bc6a6506\": {\n \"box\": { \"left\": 0.1, \"top\": 0.2, \"right\": 0.9, \"bottom\": 0.3 },\n \"page\": 0,\n \"type\": \"chunkText\",\n \"confidence\": 0.95,\n \"low_confidence_spans\": []\n },\n \"0-1\": {\n \"box\": { \"left\": 0.15, \"top\": 0.4, \"right\": 0.85, \"bottom\": 0.7 },\n \"page\": 0,\n \"type\": \"table\"\n },\n \"0-2\": {\n \"box\": { \"left\": 0.15, \"top\": 0.4, \"right\": 0.5, \"bottom\": 0.55 },\n \"page\": 0,\n \"type\": \"tableCell\",\n \"position\": { \"row\": 0, \"col\": 0, \"rowspan\": 1, \"colspan\": 1,\n \"chunk_id\": \"ef24b1ea-...\" }\n }\n },\n \"metadata\": {\n \"filename\": \"document.pdf\",\n \"org_id\": \"org-123\",\n \"page_count\": 5,\n \"duration_ms\": 1500,\n \"credit_usage\": 2.0,\n \"job_id\": \"abc-123\",\n \"version\": \"dpt-2-20260302\",\n \"failed_pages\": []\n }\n}\n```\n\n**Top-level `grounding`** is a dictionary keyed by element ID (UUID for chunks, `{page}-{base62}` for tables/cells). Each value contains `box`, `page`, `type`, and optionally `confidence` and `low_confidence_spans` (see [Confidence Scores](#confidence-scores)). Table cell entries also include a `position` field (see [Grounding and Traceability](#grounding-and-traceability)).\n\n#### Grounding Type Mapping\n\nGrounding types use a `chunk` prefix to distinguish them from chunk types. The `table` and `tableCell` types are grounding-only (no corresponding chunk type):\n\n| Grounding Type | Chunk Type | Description |\n|---|---|---|\n| `chunkText` | `text` | Text content |\n| `chunkTable` | `table` | Table chunk (overall location) |\n| `chunkFigure` | `figure` | Figures and images |\n| `chunkMarginalia` | `marginalia` | Headers, footers, page numbers |\n| `chunkLogo` | `logo` | Company logos (DPT-2) |\n| `chunkCard` | `card` | ID cards, licenses (DPT-2) |\n| `chunkAttestation` | `attestation` | Signatures, stamps (DPT-2) |\n| `chunkScanCode` | `scan_code` | QR codes, barcodes (DPT-2) |\n| `table` | _(grounding only)_ | HTML `` element within a table chunk |\n| `tableCell` | _(grounding only)_ | Individual cell within a table |\n\nExtract returns:\n\n```json\n{\n \"extraction\": {\n \"invoice_number\": \"INV-12345\",\n \"total\": 1250.00\n },\n \"extraction_metadata\": {\n \"invoice_number\": {\n \"chunk_ids\": [\"chunk-uuid-1\"]\n },\n \"total\": {\n \"chunk_ids\": [\"chunk-uuid-2\"],\n \"cell_ids\": [\"2-a5\"]\n }\n },\n \"metadata\": {\n \"filename\": \"markdown.md\",\n \"org_id\": \"org-123\",\n \"duration_ms\": 850,\n \"credit_usage\": 1.0,\n \"job_id\": \"abc-456\",\n \"version\": \"extract-20251024\",\n \"schema_violation_error\": null,\n \"fallback_model_version\": null\n }\n}\n```\n\n**Extract Metadata Fields:**\n- **`schema_violation_error`**: `null` when extraction matches schema. Contains a detailed error message when the extracted data doesn't fully conform (HTTP 206 response). Extraction still returns partial data and consumes credits.\n- **`fallback_model_version`**: `null` normally. Contains the model version actually used when the initial extraction attempt failed with the requested version and a fallback was used.\n\n### Grounding and Traceability\n\nEvery parsed element includes precise location information in the top-level `grounding` dictionary:\n\n- **Page references**: Zero-indexed page numbers\n- **Bounding boxes**: Normalized coordinates (0-1) for position\n - `left`, `top`, `right`, `bottom`\n - Convert to pixels: multiply by image dimensions\n- **Element IDs**: UUID for chunks, `{page}-{base62}` for tables and table cells\n - Table/cell IDs use sequential base62 numbering per page: `0-1`, `0-2`, ..., `0-9`, `0-a`, ..., `0-z`, `0-A`, ..., `0-Z`, `0-10`, etc.\n - Numbering restarts on each page (e.g., first table on page 1 → `1-1`)\n- **Grounding types**: Each entry has a `type` field using prefixed names (e.g., `chunkText`, `chunkTable`). See [Grounding Type Mapping](#grounding-type-mapping).\n- **Table cell position**: `tableCell` entries include a `position` object with `row`, `col` (zero-indexed), `rowspan`, `colspan`, and `chunk_id` (the parent table chunk UUID)\n- **Extraction metadata**: Shows which chunks/cells provided each field\n\n**Per-chunk grounding** (on each chunk object) contains only `box` and `page`. The **top-level grounding dictionary** adds `type` and, for table cells, `position`.\n\n**Example:**\n```python\n# Per-chunk grounding (basic location)\nfor chunk in response.chunks:\n print(f\"Chunk {chunk.id} on page {chunk.grounding.page}\")\n bbox = chunk.grounding.box\n print(f\"Location: ({bbox.left}, {bbox.top}) to ({bbox.right}, {bbox.bottom})\")\n\n# Top-level grounding (detailed, with type and position)\n# NOTE: grounding values are Pydantic models — use attribute access, not dict access\nfor elem_id, info in response.grounding.items():\n print(f\"{elem_id}: type={info.type}, page={info.page}\")\n if info.type == \"tableCell\" and info.position:\n print(f\" Cell at row={info.position.row}, col={info.position.col}\")\n```\n\n> **Important:** `response.grounding` is a `Dict[str, Grounding]` — the outer container is a dict (so `.items()`, `.get()` work), but each **value** is a Pydantic model. Use **attribute access** (`info.type`, `info.box.left`) not dict access (`info[\"type\"]`).\n\n### Confidence Scores {#confidence-scores}\n\nTop-level grounding entries may include confidence information:\n\n- **`confidence`** (`float | None`): Overall confidence score (0.0–1.0) for the chunk's transcription\n- **`low_confidence_spans`** (`list | None`): Specific text spans with low confidence, each containing:\n - `confidence` (`float`): Span-level confidence score\n - `text` (`str`): The low-confidence text\n - `span` (`list`): Position markers within the chunk\n\n```python\n# Access confidence scores from top-level grounding\nfor elem_id, info in response.grounding.items():\n if info.confidence is not None:\n print(f\"{elem_id}: confidence={info.confidence:.2f}\")\n for span in info.low_confidence_spans or []:\n print(f\" Low confidence ({span.confidence:.2f}): \"\n f\"'{span.text}'\")\n```\n\n**Notes:**\n- Confidence is only present in **top-level grounding** (not per-chunk grounding)\n- Not all grounding entries will have confidence (e.g., `table`/`tableCell` types may not)\n- Use confidence scores to flag chunks that may need human review\n\n## Best Practices\n\n### Model Selection\n\n- **Use dpt-2-latest** for most documents (complex layouts, logos, signatures)\n- **Use dpt-2-mini** for simple, digitally-native documents (faster, cheaper)\n- **Pin versions in production** for reproducibility (e.g., `dpt-2-20260302`)\n- **Use extract-latest** for extraction (automatically uses newest model)\n- **Do NOT use dpt-1** — deprecated March 31, 2026; migrate to dpt-2\n\n### Schema Design\n\n- **Be specific**: Use descriptive field names (`invoice_number` not `number`)\n- **Add descriptions**: Include format requirements (\"in USD\", \"as YYYY-MM-DD\")\n- **Keep it simple**: Start with few fields, add more as needed\n- **Limit complexity**: Under 30 properties for optimal performance\n- **Match document structure**: Order fields as they appear in document\n\nFor detailed schema patterns, see [references/extraction-schemas.md](references/extraction-schemas.md)\n\n### Error Handling\n\n```python\ntry:\n response = client.parse(document=Path(\"doc.pdf\"), model=\"dpt-2-latest\")\nexcept Exception as e:\n print(f\"Parse error: {e}\")\n # Handle error (check file format, file size, API key, etc.)\n\ntry:\n extract_response = client.extract(schema=schema, markdown=response.markdown)\nexcept Exception as e:\n print(f\"Extract error: {e}\")\n # Handle error (check schema validity, markdown format, etc.)\n```\n\n### Handling Partial Results (HTTP 206)\n\nBoth Parse and Extract APIs can return HTTP 206 (Partial Content) when processing partially succeeds:\n\n**Parse 206**: Some pages failed to parse. Check `metadata.failed_pages`:\n```python\nresponse = client.parse(document=Path(\"doc.pdf\"), model=\"dpt-2-latest\")\nif response.metadata.failed_pages:\n print(f\"Failed pages: {response.metadata.failed_pages}\")\n # Remaining pages were parsed successfully\n```\n\n**Extract 206**: Extraction completed but data doesn't fully match schema. Check `metadata.schema_violation_error`:\n```python\nresponse = client.extract(schema=schema, markdown=markdown)\nerr = response.metadata.schema_violation_error\nif err:\n print(f\"Schema violation: {err}\")\n # Extraction still returns partial data; credits are consumed\n```\n\n**Note:** 206 responses still consume credits. The API returns the best results it could produce.\n\n### Performance\n\n- **Large files**: Use Parse Jobs API (async) for files > 50 pages or > 10 MB\n- **Batch processing**: Process documents in parallel when possible\n- **Cache parse results**: Save markdown to avoid re-parsing for multiple extractions\n- **Optimize parsing**: Use the `split=\"page\"` parameter when you need page-level organization\n\n### File Formats\n\n- **Prefer PDF** for native documents (no conversion needed)\n- **Use high-resolution images** (300+ DPI) for better OCR\n- **Password-protected files**: Use the `password` parameter (requires ZDR). Without ZDR, remove password protection before parsing\n- **Test conversion** for DOCX/PPTX files (layout may change)\n\nFor complete file format reference, see [references/file-formats.md](references/file-formats.md)\n\n## Use Cases\n\nSee [references/use-cases.md](references/use-cases.md) for complete worked examples: invoice processing, form data extraction, multi-document classification, table extraction, and figure cropping with PyMuPDF.\n\n## Troubleshooting\n\nSee [references/troubleshooting.md](references/troubleshooting.md) for HTTP error codes, parse failures, extraction accuracy issues, schema validation errors, and performance guidance.\n\n## Links\n\n### Official Documentation\n- [LandingAI ADE Documentation](https://docs.landing.ai/ade/)\n- [Parse API Reference](https://docs.landing.ai/api-reference/tools/ade-parse)\n- [Extract API Reference](https://docs.landing.ai/api-reference/tools/ade-extract)\n- [Split API Reference](https://docs.landing.ai/api-reference/tools/ade-split)\n- [Python Library (GitHub)](https://github.com/landing-ai/ade-python)\n\n### API Key\n- [Get API Key](https://va.landing.ai/settings/api-key)\n\n### Reference Files\n- [Extraction Schema Patterns](references/extraction-schemas.md) - Detailed schema examples\n- [Chunk Types Reference](references/chunk-types.md) - Complete chunk type guide\n- [File Formats](references/file-formats.md) - Supported formats and considerations\n- [Use Cases](references/use-cases.md) - Worked examples for invoices, forms, tables, and figure extraction\n- [Troubleshooting](references/troubleshooting.md) - HTTP error codes and common issues\n","content/landingai/skills/ade/document-extraction/references/chunk-types.md":"# Chunk Types Reference\n\n## Overview\n\nA **chunk** is a discrete element extracted from a document during parsing. When you send a document to ADE, it analyzes the content, breaks it down into meaningful elements, and returns each as a chunk with structured metadata describing its content and location.\n\n## What is Semantic Chunking?\n\nADE uses **semantic chunking**, which intelligently groups content based on meaning rather than just layout or formatting. Instead of splitting at arbitrary points (like fixed lengths or paragraph breaks), ADE identifies coherent units of information and extracts them as individual chunks.\n\n**Benefits:**\n- Enables efficient processing of large documents (avoids token limits)\n- Improves retrieval granularity for downstream tasks\n- Supports semantic search and embeddings\n- Maintains human readability and logical relationships\n\n## Chunk Type Comparison by Model\n\n| Chunk Type | DPT-1 ⚠️ | DPT-2 | Confidence | Description |\n|------------|-----------|-------|------------|-------------|\n| text | ✓ | ✓ | ✓ | Paragraphs, headings, lists, forms, code |\n| table | ✓ | ✓ | ✗ | Grids of data, receipts, spreadsheets |\n| figure | ✓ | ✓ | ✓ | Images, graphs, flowcharts, diagrams |\n| marginalia | ✓ | ✓ | ✓ | Headers, footers, page numbers, margin notes |\n| logo | ✗ | ✓ | ✓ | Company logos |\n| card | ✗ | ✓ | ✓ | ID cards, driver licenses |\n| attestation | ✗ | ✓ | ✓ | Signatures, stamps, seals |\n| scan_code | ✗ | ✓ | ✓ | QR codes, barcodes |\n\n> ⚠️ **DPT-1 Deprecation:** DPT-1 will be removed on **March 31, 2026**. Migrate to DPT-2 now.\n\n**Note:** DPT-2 provides more granular chunk types. In DPT-1, logos, QR codes, barcodes, stamps, signatures, and ID cards are all classified as `figure`. The **Confidence** column indicates which chunk types support confidence scores in top-level grounding (Preview feature).\n\n## Chunk Type Details\n\n### text\n\n**Description:** Content consisting entirely of characters (letters and numbers).\n\n**Includes:**\n- Paragraphs\n- Titles and headings\n- Lists (bulleted, numbered)\n- Form fields\n- Checkboxes and radio buttons\n- Equations\n- Code blocks\n- Handwritten text\n\n**Key-Value Pairs:** If text contains form fields with key-value pairs, the extracted data is returned as key-value pairs separated by line breaks (`\\n`).\n\n**Example Output:**\n```markdown\n## Solar Energy Benefits\n\nSolar energy is a renewable and clean source of power that has numerous advantages:\n- Reduces electricity bills\n- Environmentally friendly\n- Low maintenance costs\n- Energy independence\n```\n\n### table\n\n**Description:** Grids of rows and columns containing data.\n\n**Includes:**\n- Traditional tables with gridlines\n- Well-aligned data without gridlines (e.g., receipt line items)\n- Spreadsheet data\n\n**Note:** ADE doesn't require gridlines to be present. It interprets well-aligned sets of data as tables.\n\n**Example Output:**\n```markdown\n| Item | Quantity | Price |\n|------|----------|-------|\n| Coffee | 2 | $5.00 |\n| Sandwich | 1 | $8.50 |\n| **Total** | | **$13.50** |\n```\n\n### marginalia\n\n**Description:** Text in the top, bottom, or side margins of a document.\n\n**Includes:**\n- Page headers\n- Page footers\n- Page numbers\n- Handwritten notes in margins\n- Line numbers\n\n**Example Output:**\n```markdown\n_Confidential Report - Page 3_\n```\n\n### figure\n\n**Description:** Visual or graphical non-text content.\n\n**Includes:**\n- Pictures and photographs\n- Graphs (bar, line, pie charts)\n- Flowcharts\n- Diagrams\n\n**DPT-1 also includes:** logos, QR codes, barcodes, stamps, signatures, ID cards (these have dedicated types in DPT-2)\n\n**Example Output:**\n```markdown\n<::Caption: Bar chart showing quarterly revenue growth from Q1 to Q4::>\n```\n\n### logo\n\n**Description:** Company logos and branding elements.\n\n**Availability:** DPT-2 only\n\n**Example Output:**\n```markdown\n<::Caption: Landing AI logo::>\n```\n\n### card\n\n**Description:** Identification cards and licenses.\n\n**Includes:**\n- ID cards\n- Driver licenses\n\n**Availability:** DPT-2 only\n\n**Example Output:**\n```markdown\n<::Caption: Driver's license with photo and personal information::>\n```\n\n### attestation\n\n**Description:** Elements that serve as authentication or approval.\n\n**Includes:**\n- Signatures (handwritten or digital)\n- Stamps\n- Seals\n\n**Availability:** DPT-2 only\n\n**Example Output:**\n```markdown\n<::Caption: Handwritten signature::>\n```\n\n### scan_code\n\n**Description:** Machine-readable codes.\n\n**Includes:**\n- QR codes\n- Barcodes (UPC, Code 39, Code 128, etc.)\n\n**Availability:** DPT-2 only\n\n**Example Output:**\n```markdown\n<::Caption: Barcode::>\n```\n\n## Working with Chunks\n\n### Accessing Chunks in Python\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\nresponse = client.parse(\n document=Path(\"document.pdf\"),\n model=\"dpt-2-latest\"\n)\n\n# Access all chunks\nfor chunk in response.chunks:\n print(f\"Type: {chunk.type}\")\n print(f\"ID: {chunk.id}\")\n print(f\"Content: {chunk.markdown}\")\n print(f\"Page: {chunk.grounding.page}\")\n print(\"---\")\n```\n\n### Filtering by Chunk Type\n\n```python\n# Get all text chunks\ntext_chunks = [chunk for chunk in response.chunks if chunk.type == 'text']\n\n# Get all tables\ntables = [chunk for chunk in response.chunks if chunk.type == 'table']\n\n# Get all figures\nfigures = [chunk for chunk in response.chunks if chunk.type == 'figure']\n```\n\n### Filtering by Page\n\n```python\n# Get all chunks from page 0 (first page)\npage_0_chunks = [chunk for chunk in response.chunks\n if chunk.grounding.page == 0]\n```\n\n### Accessing Chunk Location (Grounding)\n\n```python\nfor chunk in response.chunks:\n print(f\"Chunk ID: {chunk.id}\")\n print(f\"Page: {chunk.grounding.page}\")\n print(f\"Bounding box: {chunk.grounding.box}\")\n # Bounding box format: {left, top, right, bottom}\n # Values are normalized 0-1\n```\n\n### Grounding Type Mapping\n\nThe top-level `grounding` dictionary in the Parse response uses **grounding types** (prefixed with `chunk`) rather than chunk types. This allows the grounding dictionary to also include table-specific entries (`table`, `tableCell`) that don't correspond to chunk types.\n\n| Grounding Type | Chunk Type | Notes |\n|---|---|---|\n| `chunkText` | `text` | |\n| `chunkTable` | `table` | Overall table location |\n| `chunkFigure` | `figure` | |\n| `chunkMarginalia` | `marginalia` | |\n| `chunkLogo` | `logo` | DPT-2 only |\n| `chunkCard` | `card` | DPT-2 only |\n| `chunkAttestation` | `attestation` | DPT-2 only |\n| `chunkScanCode` | `scan_code` | DPT-2 only |\n| `table` | _(grounding only)_ | HTML `
` element within a table chunk |\n| `tableCell` | _(grounding only)_ | Individual cell; includes `position` (row, col, rowspan, colspan, chunk_id) |\n\n**Example — looking up grounding type for a chunk:**\n```python\nfor chunk in response.chunks:\n grounding_entry = response.grounding.get(chunk.id)\n if grounding_entry:\n print(f\"Chunk type: {chunk.type}\")\n print(f\"Grounding type: {grounding_entry.type}\")\n # e.g. chunk type \"text\" → grounding type \"chunkText\"\n```\n\n## Common Use Cases by Chunk Type\n\n### text\n- Extract document content for RAG systems\n- Build searchable document indices\n- Extract form field values\n- Process questionnaires and surveys\n\n### table\n- Extract financial data from statements\n- Process invoices and receipts\n- Parse spreadsheet data\n- Extract structured data from reports\n\n### figure\n- Index images for visual search\n- Generate image captions for accessibility\n- Extract diagrams for documentation\n- Archive visual content\n\n### marginalia\n- Track document metadata (page numbers, headers)\n- Extract document versioning information\n- Identify document sections from headers\n\n### logo (DPT-2)\n- Brand recognition and classification\n- Document authenticity verification\n- Company identification from documents\n\n### card (DPT-2)\n- KYC (Know Your Customer) processing\n- Identity verification workflows\n- License verification\n\n### attestation (DPT-2)\n- Contract validation\n- Approval workflow tracking\n- Signature verification\n- Document authenticity checks\n\n### scan_code (DPT-2)\n- Inventory tracking (barcodes)\n- Payment processing (QR codes)\n- Product identification\n- Link extraction from QR codes\n\n## Deprecated Chunk Types\n\n**Note:** The following chunk types were deprecated and consolidated (as of May 22, 2025):\n\n**Consolidated into `marginalia`:**\n- `page_header`\n- `page_footer`\n- `page_number`\n\n**Consolidated into `text`:**\n- `title`\n- `form`\n- `key_value`\n\n**Action Required:** If your code references these deprecated types, update to use the new consolidated types (`marginalia` or `text`).\n\n## Best Practices\n\n### 1. Choose the Right Model\n- Use **DPT-2** for documents with logos, signatures, ID cards, or barcodes\n- Use **DPT-2 mini** for simple, digitally-native documents\n\n### 2. Filter Chunks by Type\nFilter chunks to focus on relevant content for your use case:\n```python\n# Extract only text and tables for data extraction\ndata_chunks = [c for c in response.chunks\n if c.type in ['text', 'table']]\n```\n\n### 3. Use Chunk IDs for Traceability\nEach chunk has a unique ID that can be referenced in extraction metadata:\n```python\n# Extract data and trace back to source chunks\nextract_response = client.extract(schema=schema, markdown=response.markdown)\nfor field, metadata in extract_response.extraction_metadata.items():\n print(f\"{field} extracted from chunks: {metadata.chunk_ids}\")\n```\n\n### 4. Handle Visual Elements\nFor figures, logos, attestations, and scan_codes, the markdown includes a caption:\n```python\n# Check if chunk is visual\nvisual_types = ['figure', 'logo', 'card', 'attestation', 'scan_code']\nif chunk.type in visual_types:\n print(f\"Visual element: {chunk.markdown}\")\n # Caption format: <::Caption: description::>\n```\n\n## References\n\n- [Official Chunk Types Documentation](https://docs.landing.ai/ade/ade-chunk-types)\n- [Parse Models (DPT-1 vs DPT-2)](https://docs.landing.ai/ade/ade-parse-models)\n- [JSON Response Structure](https://docs.landing.ai/ade/ade-json-response)\n","content/landingai/skills/ade/document-extraction/references/extraction-schemas.md":"# Extraction Schema Patterns\n\nThis reference provides patterns and examples for creating extraction schemas for the LandingAI ADE Extract API.\n\n## Overview\n\nExtraction schemas are JSON Schema objects that define what structured data should be extracted from parsed documents. You can use either JSON Schema format (for API calls) or Pydantic models (when using the Python library).\n\n## Basic Structure\n\nEvery extraction schema must follow this structure:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"field_name\": {\n \"type\": \"string\",\n \"description\": \"Description of what to extract\"\n }\n },\n \"required\": [\"field_name\"]\n}\n```\n\n**Key points:**\n- Top-level `type` must be `\"object\"`\n- Define fields in the `properties` object\n- Use `required` array for mandatory fields\n- Add `description` for better accuracy\n\n## Supported Field Types\n\n| Type | Description | Example Use Case |\n|------|-------------|------------------|\n| `string` | Text values | Names, addresses, IDs |\n| `number` | Numeric values with decimals | Prices, amounts, percentages |\n| `integer` | Whole numbers | Counts, quantities |\n| `boolean` | True/false values | Checkboxes, yes/no questions |\n| `array` | Lists of items | Line items, charges, addresses |\n| `object` | Nested structures | Address with street/city/zip |\n\n## Common Patterns\n\n### 1. Basic Field Extraction\n\nExtract simple fields from a document:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"patient_name\": {\n \"type\": \"string\",\n \"description\": \"The name of the patient\"\n },\n \"doctor\": {\n \"type\": \"string\",\n \"description\": \"Primary care physician of the patient\"\n },\n \"copay\": {\n \"type\": \"number\",\n \"description\": \"Copay that the patient is required to pay before services are rendered\"\n }\n },\n \"required\": [\"patient_name\"]\n}\n```\n\n### 2. Nested Objects\n\nExtract hierarchical data with up to 5 levels of nesting:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"invoice\": {\n \"type\": \"object\",\n \"properties\": {\n \"number\": {\n \"type\": \"string\",\n \"description\": \"Invoice number\"\n },\n \"date\": {\n \"type\": \"string\",\n \"description\": \"Invoice date in YYYY-MM-DD format\"\n },\n \"total\": {\n \"type\": \"number\",\n \"description\": \"Total amount in USD\"\n }\n }\n },\n \"vendor\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"address\": {\"type\": \"string\"},\n \"phone\": {\"type\": \"string\"}\n }\n }\n }\n}\n```\n\n### 3. Arrays (Lists)\n\nExtract repeating items from tables or lists:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"charges\": {\n \"type\": \"array\",\n \"description\": \"List of charges on the utility bill\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"charge_type\": {\n \"type\": \"string\",\n \"description\": \"Type of charge (e.g., electricity, gas, water)\"\n },\n \"amount\": {\n \"type\": \"number\",\n \"description\": \"Charge amount in USD\"\n },\n \"usage\": {\n \"type\": \"string\",\n \"description\": \"Usage amount with unit (e.g., '450 kWh', '25 CCF')\"\n }\n }\n }\n }\n }\n}\n```\n\n### 4. Enum (Restricted Values)\n\nLimit extracted values to a specific set (string enums only):\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"account_type\": {\n \"type\": \"string\",\n \"enum\": [\"Premium Checking\", \"Standard Checking\"],\n \"description\": \"Bank account type\"\n }\n }\n}\n```\n\n### 5. Document Classification\n\nUse enum to classify documents and extract different fields per type:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"document_type\": {\n \"type\": \"string\",\n \"enum\": [\"Passport\", \"Invoice\", \"Receipt\", \"Other\"]\n }\n },\n \"required\": [\"document_type\"]\n}\n```\n\nAfter classification, make a second extraction call with type-specific schema.\n\n### 6. Nullable Fields\n\n**For extract-20251024 (recommended):**\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"middle_name\": {\n \"type\": \"string\",\n \"nullable\": true,\n \"description\": \"Patient's middle name, if provided\"\n }\n }\n}\n```\n\n**For extract-20250930:**\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"middle_name\": {\n \"type\": [\"string\", \"null\"],\n \"description\": \"Patient's middle name, if provided\"\n }\n }\n}\n```\n\n### 7. Union Types\n\nWhen a field can accept multiple types, use `anyOf` (especially with objects or arrays):\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"field1\": {\"type\": \"string\"},\n \"field2\": {\n \"anyOf\": [\n {\"type\": \"number\"},\n {\"type\": \"object\"}\n ]\n }\n }\n}\n```\n\n> **Validation rule:** Every sub-schema within `anyOf` must include either a `type` or `anyOf` keyword. If a sub-schema is missing both, the API returns a **400 error** identifying the invalid path. For example, `\"anyOf\": [{\"description\": \"a number\"}]` will fail because the sub-schema has no `type`.\n\n## Pydantic Example (Python Library)\n\nWhen using the landingai-ade Python library, use Pydantic models:\n\n```python\nfrom pydantic import BaseModel, Field\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nclass Invoice(BaseModel):\n invoice_number: str = Field(description=\"Invoice number\")\n invoice_date: str = Field(description=\"Invoice date\")\n total_amount: float = Field(description=\"Total amount in USD\")\n vendor_name: str = Field(description=\"Vendor name\")\n\n# Convert to JSON schema\nschema = pydantic_to_json_schema(Invoice)\n```\n\n## Best Practices\n\n### 1. Use Descriptive Field Names\n- **Good**: `invoice_number`, `patient_name`, `total_amount`\n- **Bad**: `number`, `name`, `amount`\n\n### 2. Add Detailed Descriptions\nInclude in descriptions:\n- Exactly what to extract\n- Format requirements (\"in USD\", \"as YYYY-MM-DD\")\n- What to include/exclude (\"excluding tax\", \"including area code\")\n\nExample:\n```json\n{\n \"total_amount\": {\n \"type\": \"number\",\n \"description\": \"Total amount in USD, excluding tax\"\n }\n}\n```\n\n### 3. Match Document Structure\nOrder fields in your schema to match their order in the document.\n\n### 4. Limit Complexity\n- Keep schemas under 30 properties for optimal performance\n- Start with a few fields, add more as needed\n- Keep names short but descriptive\n- Flatten nested arrays when possible\n- Reduce optional properties\n\n### 5. Use Appropriate Types\n- Use `number` for monetary values or calculations\n- Use `integer` for counts\n- Use `array` for repeating items (tables, lists)\n- Use `object` for hierarchical data\n\n## Model-Specific Considerations\n\n### extract-20251024 (Latest, Recommended)\n\n**Supported Keywords:**\n- `type`, `properties`, `required`, `description`, `title`\n- `enum` (string only), `nullable`\n- `array`, `items`, `maxItems`, `minItems`\n- `number`, `maximum`, `minimum`\n- `anyOf`, `$ref`, `$defs`\n- `format`, `propertyOrdering`\n\n**Behavior:**\n- Missing fields return `null` (even if required)\n- Falls back to extract-20250930 if schema is too complex\n\n### extract-20250930 (Previous Version)\n\n**Unsupported Keywords:**\n- `allOf`, `not`, `dependentRequired`, `dependentSchemas`, `if`, `then`, `else`\n\n**Behavior:**\n- Inconsistent handling of missing fields (may return `null`, `0`, empty string, etc.)\n- Use type array for nullable: `\"type\": [\"string\", \"null\"]`\n\n## Troubleshooting\n\n### Schema Validation Errors (422)\n- Ensure top-level type is \"object\"\n- Check JSON syntax\n- Verify required structure\n\n### Partial Extraction (206)\n- Extracted data doesn't match schema\n- API returns partial results and consumes credits\n- Review field types and descriptions\n\n### Low Accuracy\n- Add more detailed descriptions\n- Use more specific field names\n- Match schema to document structure\n- Reduce schema complexity\n\n## Examples by Use Case\n\n### Invoice Processing\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"invoice_number\": {\"type\": \"string\"},\n \"invoice_date\": {\"type\": \"string\", \"description\": \"Date in YYYY-MM-DD format\"},\n \"due_date\": {\"type\": \"string\", \"description\": \"Due date in YYYY-MM-DD format\"},\n \"vendor_name\": {\"type\": \"string\"},\n \"total_amount\": {\"type\": \"number\", \"description\": \"Total in USD\"},\n \"line_items\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"description\": {\"type\": \"string\"},\n \"quantity\": {\"type\": \"integer\"},\n \"unit_price\": {\"type\": \"number\"},\n \"amount\": {\"type\": \"number\"}\n }\n }\n }\n },\n \"required\": [\"invoice_number\", \"total_amount\"]\n}\n```\n\n### Bank Statement\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"account_holder\": {\"type\": \"string\"},\n \"account_number\": {\"type\": \"string\"},\n \"statement_period\": {\"type\": \"string\"},\n \"beginning_balance\": {\"type\": \"number\"},\n \"ending_balance\": {\"type\": \"number\"},\n \"transactions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"date\": {\"type\": \"string\"},\n \"description\": {\"type\": \"string\"},\n \"amount\": {\"type\": \"number\"},\n \"type\": {\"type\": \"string\", \"enum\": [\"Debit\", \"Credit\"]}\n }\n }\n }\n }\n}\n```\n\n### Medical Form\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"patient\": {\n \"type\": \"object\",\n \"properties\": {\n \"first_name\": {\"type\": \"string\"},\n \"middle_name\": {\"type\": \"string\", \"nullable\": true},\n \"last_name\": {\"type\": \"string\"},\n \"date_of_birth\": {\"type\": \"string\"},\n \"insurance_id\": {\"type\": \"string\"}\n }\n },\n \"provider\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"specialty\": {\"type\": \"string\"}\n }\n },\n \"has_allergies\": {\"type\": \"boolean\"},\n \"allergies\": {\n \"type\": \"array\",\n \"items\": {\"type\": \"string\"}\n }\n }\n}\n```\n\n## References\n\n- [Official Extraction Schema Documentation](https://docs.landing.ai/ade/ade-extract-schema-json)\n- [Python Library Documentation](https://docs.landing.ai/ade/ade-python)\n- [Extraction Models](https://docs.landing.ai/ade/ade-extract-models)\n","content/landingai/skills/ade/document-extraction/references/file-formats.md":"# Supported File Formats\n\n## Overview\n\nLandingAI ADE supports 20+ file formats across PDFs, images, documents, presentations, and spreadsheets. This reference details supported formats, limitations, and considerations for each category.\n\n## Quick Reference\n\n| Category | Formats | Notes |\n|----------|---------|-------|\n| **PDF** | PDF | Up to 100 pages in Playground; see rate limits for API |\n| **Images** | JPEG, JPG, PNG, + 15 more | Common formats fully supported |\n| **Documents** | DOC, DOCX, ODT | Converted to PDF before parsing |\n| **Presentations** | PPT, PPTX, ODP | Converted to PDF before parsing |\n| **Spreadsheets** | CSV, XLSX | Up to 10 MB in Playground; no limit in API |\n\n## PDFs\n\n### Supported\n- Standard PDF files (.pdf)\n- Multi-page PDFs (up to 100 pages in Playground)\n- Scanned PDFs (OCR applied automatically)\n\n### API Limits\n- Playground: Up to 100 pages\n- API: See [Rate Limits](https://docs.landing.ai/ade/ade-rate-limits)\n- Parse Jobs (async): Up to 1 GB or 6,000 pages\n\n\n\n### Usage Example\n```python\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\nresponse = client.parse(\n document=Path(\"document.pdf\"),\n model=\"dpt-2-latest\"\n)\n```\n\n## Images\n\n### Fully Supported (Playground + API)\n- **JPEG** - Joint Photographic Experts Group\n- **JPG** - JPEG variant\n- **PNG** - Portable Network Graphics\n\n### API-Only Supported\n- **APNG** - Animated PNG\n- **BMP** - Bitmap\n- **DCX** - Multi-page PCX\n- **DDS** - DirectDraw Surface\n- **DIB** - Device Independent Bitmap\n- **GD** - GD Graphics\n- **GIF** - Graphics Interchange Format\n- **ICNS** - Apple Icon\n- **JP2** - JPEG 2000\n- **PCX** - PC Paintbrush\n- **PPM** - Portable Pixmap\n- **PSD** - Photoshop Document\n- **TGA** - Truevision Graphics Adapter\n- **TIFF** - Tagged Image File Format\n- **WEBP** - Web Picture format\n\n### Considerations\n- All images are processed with OCR if they contain text\n- Images with complex layouts benefit from DPT-2\n- Scanned documents work best as images or PDFs\n\n### Usage Example\n```python\n# Parse an image file\nresponse = client.parse(\n document=Path(\"receipt.jpg\"),\n model=\"dpt-2-latest\"\n)\n\n# Parse from URL\nresponse = client.parse(\n document_url=\"https://example.com/invoice.png\",\n model=\"dpt-2-latest\"\n)\n```\n\n## Text Documents\n\n### Supported Formats\n- **DOC** - Microsoft Word (legacy)\n- **DOCX** - Microsoft Word\n- **ODT** - OpenDocument Text (LibreOffice)\n\n### Important: File Conversion\n\n**All text documents are converted to PDF before parsing.**\n\n**Impact:**\n- Layout may change during conversion\n- Unsupported fonts are replaced (may cause text wrapping changes)\n- Page count may increase or decrease\n- Text may overflow onto additional pages\n\n**Parsing Quality:**\nDespite layout changes, ADE still parses content correctly. The semantic chunking and content extraction work as expected.\n\n### Best Practices\n- Test with sample documents to understand conversion impact\n- For critical layout preservation, convert to PDF manually first\n- Use DPT-2 for documents with complex formatting\n\n### Usage Example\n```python\n# Parse Word document\nresponse = client.parse(\n document=Path(\"contract.docx\"),\n model=\"dpt-2-latest\"\n)\n```\n\n## Presentations\n\n### Supported Formats\n- **PPT** - Microsoft PowerPoint (legacy)\n- **PPTX** - Microsoft PowerPoint\n- **ODP** - OpenDocument Presentation (LibreOffice)\n\n### Important: File Conversion\n\n**All presentations are converted to PDF before parsing.**\n\n**Impact:**\n- Each slide becomes a page in the PDF\n- Animations and transitions are lost\n- Layout may change (same considerations as text documents)\n- Speaker notes may not be preserved\n\n**Parsing Quality:**\nSlide content, text, images, and tables are extracted correctly.\n\n### Best Practices\n- Test presentation conversion with sample files\n- For critical slides, export to PDF manually first\n- Use DPT-2 for slides with complex graphics or logos\n\n### Usage Example\n```python\n# Parse PowerPoint\nresponse = client.parse(\n document=Path(\"presentation.pptx\"),\n model=\"dpt-2-latest\"\n)\n```\n\n## Spreadsheets\n\n### Supported Formats\n- **CSV** - Comma-Separated Values\n- **XLSX** - Microsoft Excel\n\n### Limits\n\n| Environment | CSV Limit | XLSX Limit | Sheets/Rows/Columns |\n|-------------|-----------|------------|---------------------|\n| Playground | 10 MB | 10 MB | Unlimited |\n| API/Library | Unlimited | Unlimited | Unlimited |\n\n**Note:** In Playground, a render limit applies and only a truncated version is displayed. This does **not** affect parsing results.\n\n### How Spreadsheets are Parsed\n\n**Each table/sheet:**\n- Extracted as `table` chunk type\n- Cell-level IDs generated for traceability\n- Converted to HTML tables in Markdown\n- Grounding includes cell positions\n\n### Best Practices\n- Use CSV for simple data (faster processing)\n- Use XLSX for multi-sheet workbooks\n- For large spreadsheets, use Parse Jobs API (async)\n- Filter specific sheets if only subset needed\n\n### Usage Example\n```python\n# Parse Excel file\nresponse = client.parse(\n document=Path(\"sales_data.xlsx\"),\n model=\"dpt-2-latest\"\n)\n\n# Access table chunks\ntables = [chunk for chunk in response.chunks if chunk.type == 'table']\nfor table in tables:\n print(table.markdown) # HTML table\n```\n\n## Loading from Bytes\n\n### Python Library Support\n\nYou can load documents from bytes (useful for API responses, web uploads, or in-memory processing):\n\n```python\nfrom pathlib import Path\n\n# Load PDF from bytes\nwith open(\"document.pdf\", \"rb\") as f:\n pdf_bytes = f.read()\n\nresponse = client.parse(\n document=pdf_bytes, # Pass bytes directly\n model=\"dpt-2-latest\"\n)\n\n# Load image from bytes\nwith open(\"image.jpg\", \"rb\") as f:\n image_bytes = f.read()\n\nresponse = client.parse(\n document=image_bytes,\n model=\"dpt-2-latest\"\n)\n```\n\n**Use Cases:**\n- Processing files from web uploads without saving to disk\n- Working with files from cloud storage APIs\n- Processing encrypted files after decryption in memory\n\n## Format Selection Guide\n\n### When to Use Each Format\n\n| Format | Best For | Considerations |\n|--------|----------|----------------|\n| **PDF** | Any document type, forms, reports | Native format - no conversion |\n| **Images** | Receipts, photos, scanned docs | Use high resolution for better OCR |\n| **DOCX** | Contracts, reports, letters | Layout may change during conversion |\n| **PPTX** | Slide content extraction | Animations lost |\n| **XLSX** | Financial data, tables, lists | Best for structured data |\n| **CSV** | Simple tabular data | Fast processing |\n\n## Troubleshooting\n\n### Error: \"document closed or encrypted\"\n**Cause:** Password-protected document\n**Solution:** Pass `password=\"...\"` parameter (requires ZDR enabled). Without ZDR, remove password protection before parsing.\n\n### Poor OCR Results from Images\n**Possible Causes:**\n- Low resolution (< 300 DPI)\n- Poor image quality\n- Blurry or skewed text\n\n**Solutions:**\n- Increase image resolution\n- Ensure good lighting for scanned documents\n- Use deskewing tools if needed\n\n### Unexpected Layout Changes (DOCX/PPTX)\n**Cause:** File conversion to PDF\n**Solutions:**\n- Convert to PDF manually with preferred tool\n- Test conversion with sample files\n- Accept layout changes if content extraction is primary goal\n\n### Large File Processing Slow\n**Solutions:**\n- Use Parse Jobs API for files > 50 pages\n- Compress images before processing\n- Split multi-document PDFs if possible\n\n### Spreadsheet Too Large\n**Solutions:**\n- Filter to specific sheets before parsing\n- Split large XLSX into smaller files\n- Use CSV format if possible (faster)\n- Use Parse Jobs API (async processing)\n\n## API vs Playground Support\n\n| Feature | Playground | API/Library |\n|---------|------------|-------------|\n| PDF Pages | Up to 100 | See rate limits |\n| Common Images | ✓ | ✓ |\n| Extended Images | ✗ | ✓ |\n| Documents | ✓ | ✓ |\n| Presentations | ✓ | ✓ |\n| Spreadsheets | Up to 10 MB | Unlimited |\n| Parse Jobs | ✗ | ✓ |\n| Bytes Loading | ✗ | ✓ |\n\n## References\n\n- [Official File Types Documentation](https://docs.landing.ai/ade/ade-file-types)\n- [Rate Limits](https://docs.landing.ai/ade/ade-rate-limits)\n- [Parse Jobs for Large Files](https://docs.landing.ai/ade/ade-parse-async)\n- [Python Library](https://docs.landing.ai/ade/ade-python)\n","content/landingai/skills/ade/document-extraction/references/troubleshooting.md":"# Troubleshooting\n\n## HTTP Error Codes\n\n| Code | Meaning | Common Causes | Action |\n|------|---------|---------------|--------|\n| **400** | Bad Request | `anyOf` sub-schema missing `type`/`anyOf` keyword; invalid parameter | Fix schema per error message |\n| **401** | Unauthorized | Missing or invalid `VISION_AGENT_API_KEY` | Check `.env` file and key validity |\n| **413** | Payload Too Large | File exceeds sync parse limit | Use Parse Jobs API for large files |\n| **422** | Unprocessable Entity | Invalid JSON schema; unsupported keywords; top-level type not `\"object\"`; password-protected file without ZDR or with wrong password | Validate schema structure; check password; enable ZDR |\n| **429** | Rate Limited | Too many concurrent requests | Add retry with exponential backoff |\n| **206** | Partial Content | Some pages failed (parse) or schema violation (extract) | Check `metadata.failed_pages` or `metadata.schema_violation_error` |\n\n## Parse Failures\n\n- **Password-protected file**: Pass `password=\"...\"` parameter (requires ZDR). Without ZDR, remove password protection before parsing\n- **Unsupported format**: Check [file formats reference](file-formats.md)\n- **File too large**: Use Parse Jobs API (`client.parse_jobs.create()`) for files > 50 pages or > 10 MB\n- **Poor OCR quality**: Use high-resolution scans (300+ DPI); consider `dpt-2-latest` over `dpt-2-mini` for scanned docs\n\n## Low Extraction Accuracy\n\n- Add more detailed field descriptions (include format hints: \"as YYYY-MM-DD\", \"in USD\")\n- Use more specific field names (`invoice_total_usd` rather than `total`)\n- Match schema field order to how data appears in the document\n- Reduce schema complexity — stay under 30 properties for best results\n- Try `model=\"extract-20251024\"` if the latest model misses fields it should return as `null`\n\n## Missing Fields\n\n- Verify the field actually exists in the document\n- Check that the field description clearly identifies the data\n- `extract-20251024` returns `null` for absent fields; `extract-latest` may omit them entirely\n- Check `extraction_metadata` — if the field has `chunk_ids`, the model found it but may have returned an unexpected value\n\n## Schema Validation Errors (HTTP 422)\n\n- Top-level schema must have `\"type\": \"object\"`\n- `anyOf` / `oneOf` sub-schemas each need their own `type` or `anyOf` keyword\n- Avoid unsupported JSON Schema keywords (e.g., `if`/`then`, `$ref`)\n- Use `pydantic_to_json_schema()` from `landingai_ade.lib` for reliable schema generation\n\n## Performance Issues\n\n- Use `dpt-2-mini` for simple, digitally-native documents (faster and cheaper)\n- Enable Parse Jobs (`client.parse_jobs.create()`) for large files to avoid timeouts\n- Process documents in parallel with `ThreadPoolExecutor` — see [document-workflows batch-processing.md](../../document-workflows/references/batch-processing.md)\n- Cache parse results (save `response.markdown` to disk) when running multiple extractions on the same document\n\n## Partial Results (HTTP 206)\n\n**Parse 206** — Some pages failed:\n```python\nresponse = client.parse(document=Path(\"doc.pdf\"), model=\"dpt-2-latest\")\nif response.metadata.failed_pages:\n print(f\"Failed pages: {response.metadata.failed_pages}\")\n # Remaining pages were parsed successfully; credits are consumed\n```\n\n**Extract 206** — Schema violation:\n```python\nresponse = client.extract(schema=schema, markdown=markdown)\nerr = response.metadata.schema_violation_error\nif err:\n print(f\"Schema violation: {err}\")\n # Partial data is still returned; credits are consumed\n```\n","content/landingai/skills/ade/document-extraction/references/use-cases.md":"# Use Cases\n\nCommon document processing patterns using LandingAI ADE. All examples assume the client and dependencies are set up per `SKILL.md`.\n\n## Invoice Processing\n\n```python\nfrom pydantic import BaseModel, Field\nfrom landingai_ade.lib import pydantic_to_json_schema\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclass LineItem(BaseModel):\n description: str = Field(description=\"Item description\")\n quantity: int = Field(description=\"Quantity\")\n unit_price: float = Field(description=\"Unit price in USD\")\n amount: float = Field(description=\"Line total in USD\")\n\nclass Invoice(BaseModel):\n invoice_number: str = Field(description=\"Invoice number\")\n invoice_date: str = Field(description=\"Invoice date as YYYY-MM-DD\")\n vendor_name: str = Field(description=\"Vendor company name\")\n vendor_address: str = Field(description=\"Vendor address\")\n total_amount: float = Field(description=\"Total amount due in USD\")\n line_items: list[LineItem] = Field(description=\"List of line items\")\n\nclient = LandingAIADE()\n\nparse_response = client.parse(document=Path(\"invoice.pdf\"), model=\"dpt-2-latest\")\nextract_response = client.extract(\n schema=pydantic_to_json_schema(Invoice),\n markdown=parse_response.markdown,\n model=\"extract-latest\"\n)\n\ninvoice = extract_response.extraction\nprint(f\"Invoice #{invoice['invoice_number']} — ${invoice['total_amount']}\")\nfor item in invoice['line_items']:\n print(f\" {item['description']}: {item['quantity']} x ${item['unit_price']}\")\n```\n\n## Form Data Extraction\n\n```python\nfrom pydantic import BaseModel, Field\nfrom typing import Optional\n\nclass PatientIntake(BaseModel):\n patient_name: str = Field(description=\"Full patient name\")\n date_of_birth: str = Field(description=\"Date of birth as YYYY-MM-DD\")\n insurance_id: str = Field(description=\"Insurance ID number\")\n emergency_contact: str = Field(description=\"Emergency contact name and phone\")\n allergies: list[str] = Field(description=\"List of known allergies\")\n has_existing_conditions: bool = Field(description=\"Whether patient has existing conditions\")\n primary_complaint: Optional[str] = Field(default=None, description=\"Primary complaint or reason for visit\")\n\n# Parse and extract\nparse_response = client.parse(document=Path(\"intake_form.pdf\"), model=\"dpt-2-latest\")\nextract_response = client.extract(\n schema=pydantic_to_json_schema(PatientIntake),\n markdown=parse_response.markdown,\n model=\"extract-latest\"\n)\nprint(extract_response.extraction)\n```\n\n## Multi-Document Classification and Extraction\n\nSplit a batch PDF into document types, then extract type-specific fields from each:\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\n# Step 1: Parse the batch\nparse_response = client.parse(document=Path(\"batch.pdf\"), model=\"dpt-2-latest\")\n\n# Step 2: Split by document type\nsplit_response = client.split(\n markdown=parse_response.markdown,\n split_class=[\n {\"name\": \"Invoice\", \"description\": \"Commercial invoice with line items\", \"identifier\": \"Invoice Number\"},\n {\"name\": \"Receipt\", \"description\": \"Payment receipt\", \"identifier\": \"Receipt Date\"},\n {\"name\": \"Bank Statement\", \"description\": \"Monthly bank account statement\"}\n ]\n)\n\n# Step 3: Extract from each split based on its classification\nfor split in split_response.splits:\n print(f\"Type: {split.classification}, Pages: {split.pages}\")\n if split.classification == \"Invoice\":\n extract_response = client.extract(\n schema=pydantic_to_json_schema(Invoice),\n markdown=split.markdowns[0],\n model=\"extract-latest\"\n )\n print(f\" Invoice #{extract_response.extraction['invoice_number']}\")\n elif split.classification == \"Bank Statement\":\n # Use bank statement schema\n pass\n```\n\n## Table Extraction\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\nimport json\n\nclient = LandingAIADE()\n\n# Parse document or spreadsheet\nresponse = client.parse(document=Path(\"data.xlsx\"), model=\"dpt-2-latest\")\n\n# Filter table chunks\ntables = [chunk for chunk in response.chunks if chunk.type == \"table\"]\nprint(f\"Found {len(tables)} tables\")\n\nfor i, table in enumerate(tables, start=1):\n print(f\"\\nTable {i} on page {table.grounding.page}:\")\n print(table.markdown) # HTML table — parse with pandas or BeautifulSoup\n\n# Save as CSV using pandas\nimport pandas as pd\nfrom io import StringIO\n\nfor i, table in enumerate(tables, start=1):\n try:\n dfs = pd.read_html(StringIO(table.markdown))\n if dfs:\n dfs[0].to_csv(f\"table_{i:02d}.csv\", index=False)\n print(f\"Saved table_{i:02d}.csv\")\n except Exception as e:\n print(f\"Table {i}: could not parse as HTML ({e})\")\n```\n\n> **Multi-page tables:** When a table spans multiple pages, ADE emits separate chunks per page\n> and may represent some pages as plain text instead of table chunks. See\n> [Table Stitching](../../document-workflows/references/table-stitching.md) in the\n> `document-workflows` skill for three approaches to merge them into a single output.\n\n## Figure Extraction with Cropping\n\nExtract figures from PDFs as individual PNG files using bounding boxes:\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nimport fitz # PyMuPDF — install with: pip install pymupdf\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\n\n# Step 1: Parse the PDF\npdf_path = Path(\"document.pdf\")\nresponse = client.parse(document=pdf_path, model=\"dpt-2-latest\")\n\n# Step 2: Filter figure chunks\nfigure_chunks = [chunk for chunk in response.chunks if chunk.type == \"figure\"]\nprint(f\"Found {len(figure_chunks)} figures\")\n\n# Step 3: Open PDF with PyMuPDF and crop each figure\npdf_doc = fitz.open(pdf_path)\n\nfor idx, chunk in enumerate(figure_chunks, start=1):\n page_num = chunk.grounding.page\n bbox = chunk.grounding.box # Always present — API guarantees grounding on returned chunks\n\n page = pdf_doc[page_num]\n\n # Convert normalized coordinates (0-1) to absolute pixel coordinates\n x0 = bbox.left * page.rect.width\n y0 = bbox.top * page.rect.height\n x1 = bbox.right * page.rect.width\n y1 = bbox.bottom * page.rect.height\n\n # Render at 2x zoom for quality\n zoom = 2.0\n mat = fitz.Matrix(zoom, zoom)\n pix = page.get_pixmap(matrix=mat, clip=fitz.Rect(x0, y0, x1, y1))\n\n output_path = f\"figure_{idx:02d}_page{page_num + 1}.png\"\n pix.save(output_path)\n print(f\"Figure {idx}: saved as {output_path}\")\n\n # IMPORTANT: Read back the first output PNG and visually verify it shows the right content\n # before continuing. Page indexing bugs are easy to miss without a visual check.\n\npdf_doc.close()\n```\n\n**Key Points:**\n- Bounding boxes use normalized coordinates (0-1); multiply by page dimensions to get pixels\n- Every chunk returned by the API is guaranteed to have `grounding.box`\n- Use `zoom=2.0` or higher for crisp output\n- Page numbers are zero-indexed in ADE\n- After saving the first PNG, read it back and confirm it shows the expected figure\n","content/landingai/skills/ade/document-workflows/SKILL.md":"---\nname: document-workflows\ndescription: >\n Use this skill for building end-to-end document processing workflows and\n pipelines using LandingAI ADE. Trigger when users need to:\n (1) Process batches of documents in parallel or async,\n (2) Build classify-then-extract pipelines for mixed document types,\n (3) Prepare parsed documents for RAG systems with chunking and vector DB ingestion,\n (4) Load extraction results into databases like Snowflake or export to CSV/DataFrames,\n (5) Visualize extraction results: draw bounding box overlays on pages, crop\n chunk images, or highlight/annotate specific words or phrases found in documents,\n (6) Build Streamlit or web UIs for document processing,\n (7) Find and highlight specific terms within document sections using word-level\n grounding (e.g. highlight \"L2S\" in the Introduction, redact PII, annotate\n extracted values on the original page).\n This skill complements the document-extraction skill which covers ADE SDK basics.\n Use document-extraction to write code that executes parse/extract/split operations with more precision and less cost than adding the document image to the prompt and asking the LLM to find the relevant info.\n Use document-workflows when composing those operations into pipelines,\n or when you need visualization, annotation, or word-level grounding on\n parsed documents.\n---\n\n# Document Workflows — ADE Pipeline Patterns\n\n## Overview\n\nThis skill provides **reusable building blocks** for composing LandingAI ADE\nprimitives (parse, extract, split) into production-ready document processing\npipelines. It complements the `document-extraction` skill:\n\n| Concern | `document-extraction` | `document-workflows` |\n|---------|----------------------|---------------------|\n| Scope | ADE SDK API: parse, extract, split, grounding | End-to-end pipelines: batch, RAG, DB, classify-route |\n| When | Need to call a single ADE operation | Need to compose operations into a workflow |\n| Code | SDK method calls with parameters | Complete functions with error handling, parallelism |\n| Deps | `landingai-ade` only | + workflow-specific libs (pandas, chromadb, etc.) |\n\n**Philosophy:** Organize by *workflow pattern* (batch, RAG, DB insertion),\nnot by document type. The same pattern applies whether documents are invoices,\nutility bills, or medical forms.\n\n---\n\n## Step 0 (mandatory) — Pre-Flight Document Exploration {#pre-flight}\n\n**Run this before writing any pipeline code** whenever working with documents\nwhose internal structure has not already been inspected in this session.\n\n> **Rule: never write section-detection, heading-matching, or text-search code\n> without first running Tool 2 (diagnostic parse) on the sample document.\n> Heading format is document-specific and cannot be inferred from the task\n> description or document type alone — the only reliable way to know it is to\n> look at the actual ADE output.**\n>\n> Common surprises: a paper's \"Introduction\" heading may appear as\n> `1. Introduction` (plain text, no `#`), `## Introduction`, `INTRODUCTION`\n> (all-caps), or embedded inside a text chunk with body copy. Getting this\n> wrong means a silent failure (zero chunks matched) that requires a full\n> re-parse to debug.\n\nRun Tool 1 (visual render) and Tool 2 (diagnostic parse) on 1–3 representative\nsample documents before writing any code. This takes under a minute and\nprevents debugging iterations that a pre-flight would have avoided.\n\n### Tool 1 — Visual page render\n\nRender 1–2 pages as PNG and read them as visual context. No ADE credits used,\nbut each PNG consumes context tokens. Use when layout is ambiguous or document\norigin is unknown (handwriting? scan? form?).\n\n```bash\n.venv/bin/python - << 'EOF'\nimport pymupdf\nfrom pathlib import Path\nfrom PIL import Image\n\npdf = Path('path/to/sample.pdf')\nout_dir = Path('/tmp/ade_preflight'); out_dir.mkdir(exist_ok=True)\ndoc = pymupdf.open(pdf)\nfor pg in range(min(2, len(doc))): # first 2 pages only\n pix = doc[pg].get_pixmap(matrix=pymupdf.Matrix(1.5, 1.5)) # 108 DPI\n img = Image.frombytes(\"RGB\", [pix.width, pix.height], pix.samples)\n out = out_dir / f\"{pdf.stem}_page{pg + 1}.png\"\n img.save(out)\n print(out)\ndoc.close()\nEOF\n```\n\nThen read the saved PNGs. Immediately answers:\n- Are headings **bold text** (→ ADE may output plain-text heading, not `# Heading`)\n- Is the document handwritten or scanned? → Tesseract OCR needed, not PyMuPDF\n- Single-column or two-column layout?\n- Any noise: running headers, page numbers, watermarks, stamps?\n\n### Tool 2 — ADE diagnostic parse\n\nParses 1 sample and prints markdown structure + chunk inventory. Uses ADE\ncredits — keep to **1–3 samples only**, never the full corpus.\n\n```bash\n.venv/bin/python - << 'EOF'\nimport os\nfrom pathlib import Path\nfrom collections import Counter\nfrom dotenv import load_dotenv\n\n# Load API key: prefer existing env var, then .env file lookup\nload_dotenv() # Load API key from .env. Add a path to the .env if needed.\n\nfrom landingai_ade import LandingAIADE\nclient = LandingAIADE()\npr = client.parse(document=Path('path/to/sample.pdf'))\n\nprint(\"=== MARKDOWN (first 80 lines) ===\")\nfor i, ln in enumerate(pr.markdown.splitlines()[:80], 1):\n print(f\"{i:3}: {ln}\")\n\nprint(\"\\n=== CHUNKS ===\")\nfor ch in pr.chunks:\n txt = (ch.markdown or '').replace('\\n', ' ')[:70]\n b = ch.grounding.box\n print(f\"p{ch.grounding.page} {ch.type:12} \"\n f\"l={b.left:.2f} t={b.top:.2f} r={b.right:.2f} b={b.bottom:.2f} | {txt}\")\n\nprint(f\"\\nPages: {pr.metadata.page_count} \"\n f\"Chunks: {len(pr.chunks)} \"\n f\"Types: {dict(Counter(ch.type for ch in pr.chunks))}\")\nEOF\n```\n\n> **Cost note:** Save the parse result with `pr.model_dump()` to a JSON file\n> after the first run. Load it for later development instead of calling\n> `client.parse()` again. Only re-parse when the document set changes.\n\n### What to look for\n\n| Observation | Implication |\n|-------------|-------------|\n| Heading is `1. Introduction` (plain text, no `#`) | ADE markdown won't use ATX header → use ADE extract, not regex |\n| Heading format varies across docs (`# INTRO` in one, `1. Intro` in another) | Regex will break on some docs → use ADE extract for robustness |\n| Every `ch.markdown` starts with `` | Strip anchor before string matching or display |\n| Two-column: chunks on same page with `l=0.07` vs `l=0.50` | Text order is left column then right; sections may span both |\n| Chunk text cut mid-word at page break | Section spans pages; collect chunks from multiple pages |\n| `marginalia` chunks at `t<0.08` or `t>0.90` | Running headers / page numbers → exclude from content extraction |\n| Scanned / handwritten content visible in page image | PyMuPDF text extraction won't work → use Tesseract OCR |\n\n### Tool 3 — Post-Crop Visual Verification (mandatory for bounding-box workflows) {#post-crop-verification}\n\nAfter producing any bounding-box crop or overlay (figure extraction, chunk\ncropping, table cell extraction, word-level grounding), **read back at least\none output PNG as an image** and describe what you see. Compare your\ndescription against the user's request. This catches:\n\n- **Wrong-page bugs** — ADE page numbers are 0-indexed; an off-by-one error\n lands the crop on an adjacent page with completely different content\n- **Wrong-region bugs** — coordinate system mismatches that crop blank space\n or an unrelated section\n\n> **Rule: never declare a crop workflow complete without visually reading at\n> least one output PNG and confirming its content matches the user's request.**\n\n#### Verification steps\n\n1. Save the first crop as PNG (the workflow already does this)\n2. Read the PNG file as an image (use the `read_file` tool on the PNG path)\n3. Describe what you see: what content, table, figure, or text appears?\n4. Compare against the user's request:\n - User asked for \"the Events table\" → does the crop show an Events table?\n - User asked for \"Figure 3\" → does the crop show a chart/diagram?\n - User asked for \"Introduction section\" → does the crop show intro text?\n5. If the description doesn't match → investigate page indexing and\n bounding-box coordinates before continuing\n6. Only proceed with remaining crops after the first one is verified\n\n#### Why LLM vision, not heuristics\n\nA blank-check heuristic (e.g. \"mean brightness > 250 → blank\") catches only\nthe most obvious failures. The agent's own vision capability can semantically\nverify: \"this crop shows a bar chart\" vs \"the user asked for a data table.\"\nThis catches wrong-page errors even when the crop contains valid content from\nthe wrong section.\n\n---\n\n## Quick Reference — Building Blocks\n\n| # | Block | Pattern | Reference |\n|---|-------|---------|-----------|\n| 0 | Pre-flight (mandatory) | Render pages + diagnostic parse before building | [Above](#pre-flight) |\n| 1 | Parse + Save | Single doc → JSON + markdown | [Below](#core-workflow) |\n| 2 | Parse + Extract + Save | Single doc → structured data | [Below](#core-workflow) |\n| 3 | Batch (sync) | ThreadPoolExecutor + tqdm | [batch-processing.md](references/batch-processing.md) |\n| 4 | Batch (async) | AsyncLandingAIADE + aiolimiter | [batch-processing.md](references/batch-processing.md) |\n| 5 | Large files | Parse Jobs API (async polling) | [batch-processing.md](references/batch-processing.md) |\n| 6 | Classify → Extract | Enum classification + schema routing | [Below](#classify-then-extract) |\n| 7 | Results → DataFrame | Flatten nested extraction to tables | [database-integration.md](references/database-integration.md) |\n| 8 | Results → CSV | Summary + per-document export | [database-integration.md](references/database-integration.md) |\n| 9 | Results → Snowflake | 4 normalized tables + COPY upload | [database-integration.md](references/database-integration.md) |\n| 10 | Chunks → RAG CSV | 19-column chunk dataset | [rag-pipelines.md](references/rag-pipelines.md) |\n| 11 | Chunks → ChromaDB | OpenAI embeddings + persistent store | [rag-pipelines.md](references/rag-pipelines.md) |\n| 12 | Chunks → FAISS | LangChain Documents + FAISS index | [rag-pipelines.md](references/rag-pipelines.md) |\n| 13 | RAG query | RetrievalQA chain with sources | [rag-pipelines.md](references/rag-pipelines.md) |\n| 14 | Chunk images | Crop chunks from pages as PNGs | [visualization.md](references/visualization.md) |\n| 15 | Grounding overlay | Color-coded bounding boxes on pages | [visualization.md](references/visualization.md) |\n| 16 | Word-level grounding | OCR + fuzzy match highlighting | [visualization.md](references/visualization.md) |\n| 17 | Section extraction | Named section from markdown (regex or ADE extract) | [Below](#section-extraction) |\n| 18 | Embedding computation | Local (FastEmbed) or API (OpenAI) with best practices | [rag-pipelines.md](references/rag-pipelines.md) |\n| 19 | Hierarchical chunking | Group ADE chunks into semantic units for embedding | [rag-pipelines.md](references/rag-pipelines.md) |\n| 20 | Multi-granularity RAG | Chunk vs hierarchical vs document-level strategy | [rag-pipelines.md](references/rag-pipelines.md) |\n| 21 | Table stitching | Parse-only or parse+extract merge of multi-page tables | [table-stitching.md](references/table-stitching.md) |\n| — | Schema catalog | Ready-to-use Pydantic models | [schema-catalog.md](references/schema-catalog.md) |\n\n---\n\n## Core Workflow: Parse + Extract + Save\n\nThe fundamental two-step ADE pattern. Every other workflow builds on this.\n\n```python\nimport io\nfrom pathlib import Path\nfrom typing import Any, Tuple, Type\n\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\n\ndef parse_extract_save(\n doc_path: Path,\n client: LandingAIADE,\n schema_cls: Type[Any],\n output_dir: str = \"./ade_results\",\n) -> Tuple[Any, Any]:\n \"\"\"Parse a document, extract structured data, save both\n as JSON via save_to. Returns (parse_result, extract_result).\"\"\"\n # Step 1 — Parse (auto-saves {stem}_parse_output.json)\n parse_result = client.parse(\n document=doc_path, save_to=output_dir,\n )\n\n # Step 2 — Extract (auto-saves {stem}_extract_output.json)\n extract_result = client.extract(\n schema=pydantic_to_json_schema(schema_cls),\n markdown=io.BytesIO(\n parse_result.markdown.encode(\"utf-8\")\n ),\n save_to=output_dir,\n )\n return parse_result, extract_result\n```\n\n> **`save_to` parameter:** Available on `parse()`, `extract()`, and `split()`.\n> Creates the folder if needed and writes `{input_filename}_{method}_output.json`.\n> This is a client-side convenience — the full response is saved locally after the API call.\n\n### Parse-Only (no extraction)\n\n```python\ndef parse_and_save(\n doc_path: Path,\n client: LandingAIADE,\n output_dir: str = \"./ade_results\",\n) -> Any:\n return client.parse(\n document=doc_path, save_to=output_dir,\n )\n```\n\n> **Schemas:** See [schema-catalog.md](references/schema-catalog.md) for\n> ready-to-use Pydantic models (invoice, utility bill, bank statement,\n> pay stub, food label, CME certificate, document classifier).\n> See the `document-extraction` skill for schema design rules.\n\n---\n\n## Classify-then-Extract\n\nProcess mixed document types by first classifying, then applying the\nappropriate schema. Two approaches:\n\n### Approach 1: Classification Extraction (any document mix)\n\n```python\nfrom typing import Literal\nfrom pydantic import BaseModel, Field\n\n\nclass DocType(BaseModel):\n type: Literal[\n \"invoice\", \"bank_statement\", \"pay_stub\",\n \"utility_bill\",\n ] = Field(description=\"The type of the document.\")\n\n\n# Map types to schemas (from schema-catalog.md)\nSCHEMA_MAP: dict[str, type] = {\n \"invoice\": InvoiceSchema,\n \"bank_statement\": BankStatementSchema,\n \"pay_stub\": PayStubSchema,\n \"utility_bill\": UtilityBillSchema,\n}\n\n\ndef classify_and_extract(\n doc_path: Path,\n client: LandingAIADE,\n) -> dict:\n \"\"\"Classify a document then extract with the matching\n schema.\"\"\"\n pr = client.parse(document=doc_path)\n\n # Classify using first page\n cls = client.extract(\n schema=pydantic_to_json_schema(DocType),\n markdown=pr.markdown,\n )\n doc_type: str = cls.extraction[\"type\"]\n\n # Extract with type-specific schema\n schema_cls = SCHEMA_MAP[doc_type]\n er = client.extract(\n schema=pydantic_to_json_schema(schema_cls),\n markdown=pr.markdown,\n )\n return {\n \"type\": doc_type,\n \"extraction\": er.extraction,\n \"parse_result\": pr,\n \"extract_result\": er,\n }\n```\n\n### Approach 2: Split API (multi-document PDFs)\n\nWhen a single PDF contains multiple document types (e.g., a packet with\ninvoices + receipts), use the Split API first:\n\n```python\ndef split_classify_extract(\n pdf_path: Path,\n client: LandingAIADE,\n split_classes: list[dict],\n) -> list[dict]:\n \"\"\"Split a multi-doc PDF, classify each split, extract.\"\"\"\n pr = client.parse(document=pdf_path, split=\"page\")\n\n # Split into sub-documents\n split_result = client.split(\n markdown=pr.markdown,\n split_class=split_classes,\n )\n\n results = []\n for split_doc in split_result.splits:\n # Classify\n cls = client.extract(\n schema=pydantic_to_json_schema(DocType),\n markdown=split_doc.markdowns[0],\n )\n doc_type = cls.extraction[\"type\"]\n\n # Extract\n schema_cls = SCHEMA_MAP[doc_type]\n er = client.extract(\n schema=pydantic_to_json_schema(schema_cls),\n markdown=split_doc.markdowns[0],\n )\n results.append({\n \"type\": doc_type,\n \"extraction\": er.extraction,\n \"pages\": split_doc.pages,\n })\n return results\n```\n\n> **Split API parameters:** Use `split_class` (list of dicts with `name`, `description`, `identifier` keys).\n> See the `document-extraction` skill for full Split API reference.\n\n> **When to use Split vs Classification:**\n> - **Split API**: One PDF contains multiple separate documents\n> - **Classification extraction**: Each file is one document, but types vary\n\n---\n\n## Section Extraction\n\nExtract a named section (e.g. \"Introduction\", \"Abstract\") from a parsed\ndocument's markdown. Two approaches — choose based on document diversity\nand whether the extra API cost is justified.\n\n**Decision:** If the diagnostic parse (Tool 2) shows consistent ATX headers (`## Introduction`, `## 2. Methods`) across all your documents, use Approach A. If you see any plain-text numbered headings (`1. Introduction`) or formatting variation across documents, skip Approach A entirely and go straight to Approach B.\n\n| Approach | When to use |\n|----------|-------------|\n| **A — regex** | Uniform, well-structured docs (academic papers, reports). Free, fast. |\n| **B — ADE extract** | Mixed or unpredictable formatting (slides, scanned papers, varied templates). Costs an extra extract credit per document. |\n\n### Approach A — Rule-based regex (free, fast, brittle)\n\nADE may emit headings as ATX markdown (`## 2. Related Work`) or plain-text\n(`1. Introduction`) even within the same document. Handle both patterns:\n\n```python\nimport re\n\ndef find_section(markdown: str, name: str) -> str | None:\n \"\"\"Extract a named section from ADE markdown, handling both ATX\n headers (## Introduction) and plain-text numbered headings\n (1. Introduction) which ADE may emit inconsistently.\"\"\"\n\n # Pattern 1: ATX header (# Introduction, ## 1. Introduction …)\n m = re.search(\n r\"^(#{1,6})\\s+(?:\\d+\\.?\\s+)?\" + re.escape(name) + r\"\\b.*$\",\n markdown, re.IGNORECASE | re.MULTILINE,\n )\n if m:\n level = len(m.group(1))\n end = re.search(r\"^#{1,\" + str(level) + r\"}\\s\",\n markdown[m.end():], re.MULTILINE)\n end_pos = m.end() + (end.start() if end else len(markdown[m.end():]))\n return markdown[m.start():m.end() + end_pos].strip()\n\n # Pattern 2: plain-text numbered heading (1. Introduction)\n m2 = re.search(r\"^(?:\\d+\\.?\\s+)?\" + re.escape(name) + r\"\\s*$\",\n markdown, re.IGNORECASE | re.MULTILINE)\n if m2:\n end2 = re.search(\n r\"^#{1,6}\\s|^(?:\\d+\\.?\\s+)[A-Z][a-zA-Z ]{3,}\\s*$\",\n markdown[m2.end():], re.MULTILINE,\n )\n end_pos = m2.end() + (end2.start() if end2 else len(markdown[m2.end():]))\n return markdown[m2.start():end_pos].strip()\n return None\n```\n\n### Approach B — ADE extract (robust, handles document diversity)\n\nUse ADE's own extraction to semantically locate sections — no regex needed.\nThe LLM understands section meaning even when formatting is inconsistent:\n\n```python\nfrom pydantic import BaseModel, Field\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\nfrom pathlib import Path\n\n\nclass PaperSections(BaseModel):\n abstract: str = Field(\n description=\"The abstract section, plain text only, \"\n \"no markdown formatting or anchor tags.\"\n )\n introduction: str = Field(\n description=\"The introduction section, plain text only, \"\n \"no markdown formatting or anchor tags.\"\n )\n\n\nclient = LandingAIADE()\npr = client.parse(document=Path(\"paper.pdf\"))\ner = client.extract(\n schema=pydantic_to_json_schema(PaperSections),\n markdown=pr.markdown,\n)\nintro_text = er.extraction[\"introduction\"]\n```\n\n> **Cost note:** Each `extract()` call consumes additional credits on top of\n> `parse()`. For high-volume pipelines with uniform document types, Approach A\n> avoids this cost. For diverse or unpredictable documents the accuracy\n> improvement justifies the extra credit.\n\n---\n\n## Multi-Page Table Stitching {#table-stitching}\n\nWhen a table spans multiple pages, ADE may emit it as separate table chunks\nper page — and may emit some pages as plain text instead of table chunks.\nThis inconsistency can occur on **any** page, not just the last one.\n\nThree approaches handle this, with different cost/accuracy/fragility\ntrade-offs:\n\n| Approach | ADE Calls | Handles non-table chunks | Fragility |\n|----------|-----------|--------------------------|-----------|\n| **A — Parse + Extract** | 2 | ✓ LLM reads full markdown | Low — no custom parsing |\n| **B — HTML table parsing** | 1 | ✓ with fallback regex | **High** — requires uniform row structure |\n| **C — pandas read_html** | 1 | ✗ misses non-table chunks | Medium |\n\n**Decision guide:**\n- Use **Approach A** when accuracy is paramount and cost is secondary\n- Use **Approach B** when rows are highly uniform, document structure is\n predictable, and cost savings justify the fragility of regex-based parsing\n- Use **Approach C** for quick prototyping or when missing some rows is\n acceptable\n\n### Pre-flight additions for table stitching\n\nBefore choosing an approach, run the diagnostic parse (Tool 2) and check:\n\n| What to check | How | Why |\n|---------------|-----|-----|\n| Chunk types per page | Count `type == \"table\"` vs `\"text\"` per page | Any page may have inconsistent types |\n| Column count consistency | Compare column counts across table chunks | Inconsistent counts may indicate different tables |\n| Header row presence | Check first row of each table chunk | Needed for detection and row filtering |\n| Non-target tables | Look for summary/metadata tables with same column count | Must distinguish target from others |\n| Row uniformity | Compare row structure across pages | Low uniformity makes Approach B fragile |\n\n### Domain-specific semantic checks\n\nAfter stitching, add validation checks that leverage domain knowledge:\n- **Financial:** running balances, column totals = sum of rows\n- **Inventory:** quantity conservation across rows\n- **Time-series:** chronological ordering, no sequence gaps\n- **Scientific:** consistent units, monotonic IDs\n\nThese checks serve as both **validation** (confirming correctness) and\n**disambiguation** (resolving structural ambiguity in parsed output).\n\n> **Full code** for all three approaches with reusable patterns:\n> see [table-stitching.md](references/table-stitching.md).\n\n---\n\n## Batch Processing\n\nTwo patterns depending on scale. Both include per-document error handling.\n\n### Quick: ThreadPoolExecutor (sync)\n\n```python\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\nfrom tqdm import tqdm\n\n\ndef batch_process(\n files: list[Path],\n schema_cls: type,\n max_workers: int = 4,\n) -> list[tuple[Path, Any, Any]]:\n client = LandingAIADE()\n results: list[tuple[Path, Any, Any]] = []\n with ThreadPoolExecutor(max_workers=max_workers) as pool:\n futures = {\n pool.submit(\n parse_extract_save, fp, client, schema_cls\n ): fp\n for fp in files\n }\n for fut in tqdm(\n as_completed(futures), total=len(futures)\n ):\n fp = futures[fut]\n try:\n results.append((fp, *fut.result()))\n except Exception as e:\n print(f\"FAILED {fp.name}: {e}\")\n return results\n```\n\n### Scalable: AsyncLandingAIADE (async)\n\n```python\nimport asyncio\nfrom aiolimiter import AsyncLimiter\nfrom landingai_ade import AsyncLandingAIADE\n\n\nasync def batch_parse_async(\n files: list[Path],\n rate_limit: int = 30,\n) -> list[dict]:\n client = AsyncLandingAIADE()\n limiter = AsyncLimiter(rate_limit, 60)\n\n async def _process(fp: Path) -> dict | None:\n try:\n async with limiter:\n return {\n \"path\": fp,\n \"result\": await client.parse(document=fp),\n }\n except Exception as e:\n print(f\"FAILED {fp.name}: {e}\")\n return None\n\n raw = await asyncio.gather(*[_process(fp) for fp in files])\n return [r for r in raw if r]\n```\n\n> **Full code** with output directory organization, CSV export, and chunk\n> image saving: see [batch-processing.md](references/batch-processing.md).\n\n---\n\n## Results to DataFrames and CSV\n\nFlatten nested ADE extraction results into 4 normalized tables:\n\n```python\nimport uuid\nfrom datetime import datetime, timezone\n\n\ndef rows_from_doc(\n file_path: str,\n parse_result: Any,\n extract_result: Any,\n run_id: str = \"\",\n) -> tuple[dict, list[dict], list[dict], dict]:\n \"\"\"Returns (main_row, line_rows, chunk_rows, md_record).\n\n - main_row: flattened top-level fields (nested__field)\n - line_rows: one per list item (line items, transactions)\n - chunk_rows: one per parsed chunk with bounding boxes\n - md_record: full markdown for traceability\n \"\"\"\n doc_uuid = str(uuid.uuid4())\n f = extract_result.extraction\n\n # Flatten top-level fields\n main_row = {\"doc_uuid\": doc_uuid, \"document_name\": Path(file_path).name}\n for k, v in f.items():\n if isinstance(v, dict):\n for sk, sv in v.items():\n main_row[f\"{k}__{sk}\"] = sv\n elif not isinstance(v, list):\n main_row[k] = v\n\n # Extract list fields as line rows\n line_rows = [\n {\"doc_uuid\": doc_uuid, \"list_field\": k, \"line_index\": i, **item}\n for k, v in f.items() if isinstance(v, list)\n for i, item in enumerate(v) if isinstance(item, dict)\n ]\n\n # Chunk rows from parse result\n chunk_rows = [\n {\n \"doc_uuid\": doc_uuid,\n \"chunk_id\": getattr(ch, \"id\", None),\n \"chunk_type\": getattr(ch, \"type\", None),\n \"page\": ch.grounding.page if hasattr(ch, \"grounding\") else None,\n }\n for ch in (parse_result.chunks or [])\n ]\n\n md_record = {\n \"doc_uuid\": doc_uuid,\n \"markdown\": parse_result.markdown,\n }\n return main_row, line_rows, chunk_rows, md_record\n```\n\n> **Full code** with Snowflake upload, UUID traceability, and bounding box\n> columns: see [database-integration.md](references/database-integration.md).\n\n---\n\n## RAG Preparation\n\nQuick path from parsed documents to a queryable RAG system. Two\nembedding options: **local** (free, offline) or **API** (higher quality).\n\n### Option A — Local embeddings with FastEmbed (free)\n\n```python\nimport re\nfrom fastembed import TextEmbedding\n\n\ndef ade_to_embeddings_local(\n parse_results: list[dict],\n model: str = \"BAAI/bge-small-en-v1.5\",\n) -> list[dict]:\n \"\"\"Embed ADE chunks locally. Returns list of dicts with\n text, vector, and grounding metadata.\"\"\"\n embedder = TextEmbedding(model_name=model)\n items: list[dict] = []\n for pr in parse_results:\n for ch in (pr[\"parse_result\"].chunks or []):\n text = re.sub(\n r\"\\s*\", \"\", ch.markdown,\n ).strip()\n if not text:\n continue\n items.append({\n \"text\": text,\n \"source\": pr[\"name\"],\n \"page\": ch.grounding.page,\n \"box\": {\n \"l\": ch.grounding.box.left,\n \"t\": ch.grounding.box.top,\n \"r\": ch.grounding.box.right,\n \"b\": ch.grounding.box.bottom,\n },\n })\n vecs = list(embedder.embed([i[\"text\"] for i in items]))\n for item, vec in zip(items, vecs):\n item[\"vector\"] = vec.tolist()\n return items\n```\n\n### Option B — API embeddings with OpenAI\n\n```python\nfrom langchain.docstore.document import Document\nfrom langchain_community.vectorstores import FAISS\nfrom langchain_openai import OpenAIEmbeddings\n\n\ndef ade_to_rag(\n parse_results: list[dict],\n embedding_model: str = \"text-embedding-3-small\",\n) -> FAISS:\n \"\"\"Convert ADE parse results to a FAISS vector store.\n\n Args:\n parse_results: list of {\"name\": str, \"parse_result\": ParseResponse}\n \"\"\"\n docs = [\n Document(\n page_content=ch.markdown,\n metadata={\n \"source\": item[\"name\"],\n \"chunk_type\": getattr(ch, \"type\", \"\"),\n \"page\": ch.grounding.page if hasattr(ch, \"grounding\") else -1,\n },\n )\n for item in parse_results\n for ch in (item[\"parse_result\"].chunks or [])\n if ch.markdown.strip()\n ]\n return FAISS.from_documents(\n docs, OpenAIEmbeddings(model=embedding_model)\n )\n```\n\n> **Full code** with embedding best practices, hierarchical chunking,\n> multi-granularity strategies, ChromaDB, LangChain RetrievalQA, and\n> CSV export: see [rag-pipelines.md](references/rag-pipelines.md).\n\n**Advanced RAG patterns in [rag-pipelines.md](references/rag-pipelines.md):**\n\n- **Embedding computation** (blocks 18–19) — choosing between local (FastEmbed, free) and API (OpenAI, higher quality) embeddings, including batch sizing and rate limiting\n- **Hierarchical chunking** (block 20) — embed at multiple granularities (chunk, section, document) for hybrid retrieval\n- **Multi-granularity RAG** (block 21) — combine chunk-level precision with document-level context, routing queries to the right embedding level based on scope\n\n---\n\n## Visualization\n\nQuick snippet for bounding box overlays on parsed pages:\n\n```python\nfrom PIL import Image, ImageDraw\nimport pymupdf\n\nCHUNK_COLORS = {\n \"text\": (40, 167, 69),\n \"table\": (0, 123, 255),\n \"figure\": (255, 0, 255),\n \"marginalia\": (111, 66, 193),\n}\n\ndef annotate_page(\n img: Image.Image, chunks: list, page: int,\n) -> Image.Image:\n annotated = img.copy()\n draw = ImageDraw.Draw(annotated)\n w, h = img.size\n for ch in chunks:\n if not hasattr(ch, \"grounding\") or ch.grounding.page != page:\n continue\n box = ch.grounding.box\n color = CHUNK_COLORS.get(getattr(ch, \"type\", \"\"), (200, 200, 200))\n draw.rectangle(\n [int(box.left * w), int(box.top * h),\n int(box.right * w), int(box.bottom * h)],\n outline=color, width=3,\n )\n return annotated\n```\n\n> **Full code** with chunk image cropping, extraction-only overlays, and\n> word-level OCR grounding: see [visualization.md](references/visualization.md).\n\n---\n\n## Streamlit UI Pattern\n\nQuick Streamlit app for interactive document processing:\n\n```python\nimport streamlit as st\nfrom pathlib import Path\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nst.title(\"Document Processor\")\n\nuploaded = st.file_uploader(\n \"Upload document\", type=[\"pdf\", \"png\", \"jpg\"]\n)\nif uploaded:\n # Save temp file\n tmp = Path(f\"/tmp/{uploaded.name}\")\n tmp.write_bytes(uploaded.read())\n\n client = LandingAIADE()\n\n with st.spinner(\"Parsing...\"):\n pr = client.parse(document=tmp)\n\n st.subheader(\"Markdown Preview\")\n st.markdown(pr.markdown[:2000])\n\n st.subheader(\"Chunks\")\n for ch in pr.chunks:\n with st.expander(\n f\"{ch.type} (page {ch.grounding.page})\"\n ):\n st.text(ch.markdown[:500])\n```\n\n\n> **Full Streamlit app** with batch upload, extraction display, and\n> visualization tabs: adapt from the patterns in\n> [batch-processing.md](references/batch-processing.md) and\n> [visualization.md](references/visualization.md).\n\n---\n\n## Dependency Guide\n\n| Workflow | Install |\n|----------|---------|\n| Core (parse + extract) | `pip install landingai-ade` |\n| Batch sync | `pip install landingai-ade tqdm` |\n| Batch async | `pip install landingai-ade aiolimiter` |\n| DataFrames / CSV | `pip install landingai-ade pandas` |\n| Snowflake | `pip install landingai-ade pandas snowflake-connector-python[pandas]` |\n| RAG (local embeddings) | `pip install landingai-ade fastembed` |\n| RAG (ChromaDB) | `pip install landingai-ade chromadb openai` |\n| RAG (FAISS + LangChain) | `pip install landingai-ade langchain langchain-openai langchain-community faiss-cpu` |\n| Visualization | `pip install landingai-ade Pillow pymupdf` |\n| Word-level grounding | `pip install landingai-ade Pillow pymupdf pytesseract fuzzywuzzy` + `tesseract` binary |\n| Streamlit UI | `pip install landingai-ade streamlit` |\n| Schema conversion | `from landingai_ade.lib import pydantic_to_json_schema` (included in landingai-ade) |\n\n---\n\n## Reference Files\n\nRead these for full implementations when building a specific workflow:\n\n- **[schema-catalog.md](references/schema-catalog.md)** — Ready-to-use Pydantic schemas for invoice, utility bill, bank statement, pay stub, food label, CME certificate, and document classification\n- **[batch-processing.md](references/batch-processing.md)** — ThreadPoolExecutor, AsyncLandingAIADE, and Parse Jobs API patterns with full error handling\n- **[rag-pipelines.md](references/rag-pipelines.md)** — Chunks to CSV, ChromaDB ingestion, FAISS + LangChain, and RAG query chains\n- **[database-integration.md](references/database-integration.md)** — DataFrame normalization, Snowflake upload, and CSV export patterns\n- **[visualization.md](references/visualization.md)** — Chunk image cropping, bounding box overlays, and word-level OCR grounding\n- **[table-stitching.md](references/table-stitching.md)** — Parse+Extract (robust), HTML parsing (fragile), and pandas approaches for merging multi-page tables into a single output\n","content/landingai/skills/ade/document-workflows/references/batch-processing.md":"# Batch Processing Patterns\n\nThree approaches for processing multiple documents, from simplest to most\nscalable. All patterns include per-document error handling so one failure\ndoesn't stop the batch.\n\n---\n\n## 1. Sync Parallel — ThreadPoolExecutor\n\nBest for: moderate batches (10–200 docs), simple scripts, notebooks.\n\n```python\nimport io\nimport json\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\nfrom pathlib import Path\nfrom typing import Any, List, Tuple, Type\n\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\nfrom tqdm import tqdm\n\n\ndef parse_extract_save(\n doc_path: Path,\n client: LandingAIADE,\n schema_cls: Type[Any],\n output_dir: Path,\n) -> Tuple[Any, Any]:\n \"\"\"Parse one document, extract with schema, save both\n results as JSON. Returns (parse_result, extract_result).\"\"\"\n output_dir.mkdir(parents=True, exist_ok=True)\n stem = doc_path.stem\n\n # Step 1 — Parse\n parse_result = client.parse(document=doc_path)\n _save_json(\n parse_result, output_dir / f\"parse_{stem}.json\"\n )\n\n # Step 2 — Extract\n json_schema = pydantic_to_json_schema(schema_cls)\n extract_result = client.extract(\n schema=json_schema,\n markdown=io.BytesIO(\n parse_result.markdown.encode(\"utf-8\")\n ),\n )\n _save_json(\n extract_result, output_dir / f\"extract_{stem}.json\"\n )\n return parse_result, extract_result\n\n\ndef _save_json(obj: Any, path: Path) -> None:\n data = (\n obj.model_dump()\n if hasattr(obj, \"model_dump\")\n else obj\n )\n path.write_text(\n json.dumps(data, indent=2, default=str),\n encoding=\"utf-8\",\n )\n\n\ndef batch_parse_extract(\n file_paths: List[Path],\n schema_cls: Type[Any],\n output_dir: Path = Path(\"./ade_results\"),\n max_workers: int = 4,\n api_key: str | None = None,\n) -> List[Tuple[Path, Any, Any]]:\n \"\"\"Process a list of documents in parallel.\n\n Returns list of (path, parse_result, extract_result)\n for successful documents. Failures are printed but\n do not stop the batch.\n \"\"\"\n client = LandingAIADE(\n **({\"apikey\": api_key} if api_key else {})\n )\n results: List[Tuple[Path, Any, Any]] = []\n\n with ThreadPoolExecutor(max_workers=max_workers) as pool:\n futures = {\n pool.submit(\n parse_extract_save,\n fp, client, schema_cls, output_dir,\n ): fp\n for fp in file_paths\n }\n for future in tqdm(\n as_completed(futures),\n total=len(futures),\n desc=\"Processing\",\n ):\n fp = futures[future]\n try:\n pr, er = future.result()\n results.append((fp, pr, er))\n except Exception as exc:\n print(f\"FAILED {fp.name}: {exc}\")\n\n return results\n```\n\n### Usage\n\n```python\nfrom pathlib import Path\nfrom my_schema import InvoiceSchema # or any Pydantic model\n\nfiles = sorted(Path(\"invoices/\").glob(\"*.pdf\"))\nresults = batch_parse_extract(\n files,\n schema_cls=InvoiceSchema,\n output_dir=Path(\"./results\"),\n max_workers=6,\n)\nprint(f\"Processed {len(results)}/{len(files)} documents\")\n```\n\n---\n\n## 2. Async Parallel — AsyncLandingAIADE\n\nBest for: large batches (100+ docs), CLI tools, production pipelines.\nUses `asyncio` + `aiolimiter` for rate-limited concurrency.\n\n```python\nimport asyncio\nimport json\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional\n\nimport pandas as pd\nfrom aiolimiter import AsyncLimiter\nfrom landingai_ade import AsyncLandingAIADE\n\n\nSUPPORTED_EXTS = {\".pdf\", \".png\", \".jpg\", \".jpeg\"}\n\n\ndef collect_files(input_dir: Path) -> List[Path]:\n return sorted(\n p for p in input_dir.glob(\"*\")\n if p.is_file() and p.suffix.lower() in SUPPORTED_EXTS\n )\n\n\nasync def process_document(\n file_path: Path,\n client: AsyncLandingAIADE,\n output_dirs: Dict[str, Path],\n rate_limiter: AsyncLimiter,\n) -> Optional[Dict[str, Any]]:\n \"\"\"Parse one document async, save JSON + markdown.\"\"\"\n try:\n async with rate_limiter:\n result = await client.parse(document=file_path)\n\n stem = file_path.stem\n # Save JSON\n (output_dirs[\"json\"] / f\"{stem}.json\").write_text(\n json.dumps(\n result.model_dump(), indent=2, default=str\n ),\n encoding=\"utf-8\",\n )\n # Save markdown\n (output_dirs[\"markdown\"] / f\"{stem}.md\").write_text(\n result.markdown, encoding=\"utf-8\"\n )\n return {\"path\": file_path, \"result\": result}\n except Exception as exc:\n print(f\"FAILED {file_path.name}: {exc}\")\n return None\n\n\nasync def batch_parse_async(\n input_dir: Path,\n output_dir: Path,\n max_concurrent: int = 10,\n rate_limit: int = 30,\n api_key: str | None = None,\n) -> List[Dict[str, Any]]:\n \"\"\"Parse all documents in input_dir concurrently.\n\n Args:\n input_dir: folder with documents\n output_dir: base output folder (json/, markdown/\n subdirs created automatically)\n max_concurrent: max parallel requests\n rate_limit: max requests per minute\n \"\"\"\n files = collect_files(input_dir)\n if not files:\n print(f\"No documents found in {input_dir}\")\n return []\n\n # Create output subdirectories\n dirs: Dict[str, Path] = {}\n for sub in (\"json\", \"markdown\"):\n d = output_dir / sub\n d.mkdir(parents=True, exist_ok=True)\n dirs[sub] = d\n\n client = AsyncLandingAIADE(\n **({\"apikey\": api_key} if api_key else {})\n )\n limiter = AsyncLimiter(rate_limit, 60)\n\n tasks = [\n process_document(fp, client, dirs, limiter)\n for fp in files\n ]\n raw = await asyncio.gather(*tasks)\n return [r for r in raw if r is not None]\n```\n\n### Usage\n\n```python\nimport asyncio\nfrom pathlib import Path\n\nresults = asyncio.run(\n batch_parse_async(\n input_dir=Path(\"documents/\"),\n output_dir=Path(\"results/\"),\n max_concurrent=10,\n rate_limit=30,\n )\n)\nprint(f\"Parsed {len(results)} documents\")\n```\n\n### Adding Extraction to Async Pipeline\n\n```python\nimport io\nfrom landingai_ade.lib import pydantic_to_json_schema\n\n\nasync def process_with_extraction(\n file_path: Path,\n client: AsyncLandingAIADE,\n schema_cls: type,\n output_dirs: Dict[str, Path],\n rate_limiter: AsyncLimiter,\n) -> Optional[Dict[str, Any]]:\n try:\n async with rate_limiter:\n parse_result = await client.parse(\n document=file_path\n )\n async with rate_limiter:\n extract_result = await client.extract(\n schema=pydantic_to_json_schema(schema_cls),\n markdown=io.BytesIO(\n parse_result.markdown.encode(\"utf-8\")\n ),\n )\n return {\n \"path\": file_path,\n \"parse\": parse_result,\n \"extract\": extract_result,\n }\n except Exception as exc:\n print(f\"FAILED {file_path.name}: {exc}\")\n return None\n```\n\n---\n\n## 3. Large File Processing — Parse Jobs API\n\nBest for: files > 50 MB (up to ~1 GB). Uses async job submission +\npolling instead of synchronous upload.\n\n```python\nimport time\nfrom pathlib import Path\nfrom typing import Any\n\nfrom landingai_ade import LandingAIADE\n\n\ndef parse_large_file(\n file_path: Path,\n client: LandingAIADE,\n poll_interval: int = 10,\n max_wait: int = 600,\n) -> Any:\n \"\"\"Submit a large file as a parse job and poll until\n complete.\n\n Returns the parse result (same shape as client.parse()).\n \"\"\"\n # Step 1 — Submit job\n job = client.parse(\n document=file_path, is_async=True\n )\n job_id = job.request_id\n print(f\"Job submitted: {job_id}\")\n\n # Step 2 — Poll for completion\n elapsed = 0\n while elapsed < max_wait:\n status = client.get_parse_job(job_id)\n if status.status == \"complete\":\n print(f\"Job complete after {elapsed}s\")\n return status.data\n if status.status == \"failed\":\n raise RuntimeError(\n f\"Parse job {job_id} failed: {status}\"\n )\n time.sleep(poll_interval)\n elapsed += poll_interval\n\n raise TimeoutError(\n f\"Job {job_id} not complete after {max_wait}s\"\n )\n```\n\n### Batch Large Files\n\n```python\nfrom concurrent.futures import ThreadPoolExecutor\n\ndef batch_parse_large(\n file_paths: list[Path],\n max_workers: int = 3,\n) -> list[Any]:\n client = LandingAIADE()\n results = []\n with ThreadPoolExecutor(max_workers=max_workers) as pool:\n futures = {\n pool.submit(\n parse_large_file, fp, client\n ): fp\n for fp in file_paths\n }\n for fut in futures:\n try:\n results.append(fut.result())\n except Exception as exc:\n fp = futures[fut]\n print(f\"FAILED {fp.name}: {exc}\")\n return results\n```\n\n---\n\n## Rate Limiting & Error Handling Tips\n\n| Concern | Recommendation |\n|---------|---------------|\n| API rate limits | Use `aiolimiter.AsyncLimiter(30, 60)` for async; limit `max_workers` for sync |\n| Transient failures | Wrap individual doc processing in try/except; log and continue |\n| Large batches (1000+) | Use async pattern with `rate_limit=20`; monitor API response times |\n| Memory | Process results incrementally (save to disk per doc) rather than accumulating in memory |\n| Retries | Add exponential backoff for 429/5xx errors: `tenacity.retry(wait=wait_exponential())` |\n\n### Dependencies\n\n```\n# Sync parallel (ThreadPoolExecutor)\npip install landingai-ade tqdm\n\n# Async parallel\npip install landingai-ade aiolimiter pandas\n\n# Large files — no extra deps beyond landingai-ade\n```\n","content/landingai/skills/ade/document-workflows/references/database-integration.md":"# Database Integration Patterns\n\nPatterns for normalizing ADE extraction results into relational tables\nand loading them into databases. Covers DataFrame normalization, CSV\nexport, and Snowflake insertion.\n\n---\n\n## 1. DataFrame Normalization\n\nADE extraction results are nested dicts. This pattern flattens them into\n4 normalized tables suitable for any relational DB:\n\n| Table | Contents |\n|-------|----------|\n| `main` | One row per document — top-level extracted fields |\n| `line_items` | One row per line item / repeating element |\n| `chunks` | One row per parsed chunk with bounding boxes |\n| `markdown` | One row per document — full markdown for traceability |\n\n```python\nimport uuid\nfrom datetime import datetime, timezone\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional, Tuple\n\n\ndef _dig(obj: Any, *keys: str, default: Any = None) -> Any:\n \"\"\"Safely traverse nested dicts/objects by key path.\"\"\"\n for k in keys:\n if obj is None:\n return default\n if isinstance(obj, dict):\n obj = obj.get(k, default)\n else:\n obj = getattr(obj, k, default)\n return obj\n\n\ndef _to_float(v: Any) -> Optional[float]:\n if v is None:\n return None\n try:\n return float(v)\n except (ValueError, TypeError):\n return None\n\n\ndef rows_from_doc(\n file_path: str,\n parse_result: Any,\n extract_result: Any,\n run_id: str | None = None,\n) -> Tuple[\n Dict[str, Any],\n List[Dict[str, Any]],\n List[Dict[str, Any]],\n Dict[str, Any],\n]:\n \"\"\"Transform ADE parse + extract results into 4 row types.\n\n Returns: (main_row, line_rows, chunk_rows, markdown_record)\n\n Args:\n file_path: original document path\n parse_result: from client.parse()\n extract_result: from client.extract()\n run_id: optional batch run identifier\n \"\"\"\n doc_name = Path(file_path).name\n doc_uuid = str(uuid.uuid4())\n now = datetime.now(timezone.utc).isoformat()\n rid = run_id or doc_uuid\n\n f = extract_result.extraction # dict\n m = getattr(extract_result, \"extraction_metadata\", {})\n\n # --- markdown record ---\n markdown_record = {\n \"run_id\": rid,\n \"doc_uuid\": doc_uuid,\n \"document_name\": doc_name,\n \"processed_at\": now,\n \"markdown\": parse_result.markdown,\n }\n\n # --- chunk rows ---\n chunk_rows: List[Dict[str, Any]] = []\n for ch in (parse_result.chunks or []):\n box = (\n ch.grounding.box\n if hasattr(ch, \"grounding\")\n and hasattr(ch.grounding, \"box\")\n else None\n )\n chunk_rows.append({\n \"run_id\": rid,\n \"doc_uuid\": doc_uuid,\n \"document_name\": doc_name,\n \"chunk_id\": getattr(ch, \"id\", None),\n \"chunk_type\": getattr(ch, \"type\", None),\n \"text\": getattr(ch, \"markdown\", None),\n \"page\": (\n ch.grounding.page\n if hasattr(ch, \"grounding\")\n else None\n ),\n \"box_l\": _to_float(box.left if box else None),\n \"box_t\": _to_float(box.top if box else None),\n \"box_r\": _to_float(box.right if box else None),\n \"box_b\": _to_float(box.bottom if box else None),\n })\n\n # --- main row (flatten top-level fields) ---\n main_row: Dict[str, Any] = {\n \"run_id\": rid,\n \"doc_uuid\": doc_uuid,\n \"document_name\": doc_name,\n \"processed_at\": now,\n }\n # Flatten one level of nesting\n for key, val in f.items():\n if isinstance(val, dict):\n for sub_key, sub_val in val.items():\n main_row[f\"{key}__{sub_key}\"] = sub_val\n elif isinstance(val, list):\n pass # lists go to line_items\n else:\n main_row[key] = val\n\n # --- line item rows ---\n line_rows: List[Dict[str, Any]] = []\n for key, val in f.items():\n if not isinstance(val, list):\n continue\n for idx, item in enumerate(val):\n row: Dict[str, Any] = {\n \"run_id\": rid,\n \"doc_uuid\": doc_uuid,\n \"document_name\": doc_name,\n \"list_field\": key,\n \"line_index\": idx,\n }\n if isinstance(item, dict):\n row.update(item)\n else:\n row[\"value\"] = item\n line_rows.append(row)\n\n return main_row, line_rows, chunk_rows, markdown_record\n```\n\n### Usage — Build DataFrames from a Batch\n\n```python\nimport pandas as pd\nfrom pathlib import Path\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nclient = LandingAIADE()\nrun_id = \"batch_2025_01\"\n\nall_main, all_lines, all_chunks, all_md = [], [], [], []\n\nfor fp in Path(\"invoices/\").glob(\"*.pdf\"):\n pr = client.parse(document=fp)\n er = client.extract(\n schema=pydantic_to_json_schema(InvoiceSchema),\n markdown=pr.markdown,\n )\n main, lines, chunks, md = rows_from_doc(\n str(fp), pr, er, run_id=run_id\n )\n all_main.append(main)\n all_lines.extend(lines)\n all_chunks.extend(chunks)\n all_md.append(md)\n\ndf_main = pd.DataFrame(all_main)\ndf_lines = pd.DataFrame(all_lines)\ndf_chunks = pd.DataFrame(all_chunks)\ndf_md = pd.DataFrame(all_md)\n\n# Save to CSV\nfor name, df in [\n (\"main\", df_main),\n (\"line_items\", df_lines),\n (\"chunks\", df_chunks),\n (\"markdown\", df_md),\n]:\n df.to_csv(f\"{run_id}_{name}.csv\", index=False)\n```\n\n---\n\n## 2. Snowflake Integration\n\nUpload normalized tables to Snowflake using the connector's\n`write_pandas` or staged COPY pattern.\n\n> **Note:** ADE is also available as a **Snowflake Native App** (GA since Nov 2025), which runs ADE directly inside your Snowflake account without data leaving Snowflake. The patterns below use the standard Python SDK connector approach. For the Native App, see [Snowflake Native App docs](https://docs.landing.ai/ade/ade-snowflake).\n\n### Connection Setup\n\n```python\nimport snowflake.connector\nfrom snowflake.connector.pandas_tools import write_pandas\n\n\ndef get_snowflake_conn(\n account: str,\n user: str,\n password: str,\n database: str,\n schema: str,\n warehouse: str,\n role: str = \"SYSADMIN\",\n) -> snowflake.connector.SnowflakeConnection:\n return snowflake.connector.connect(\n account=account,\n user=user,\n password=password,\n database=database,\n schema=schema,\n warehouse=warehouse,\n role=role,\n )\n```\n\n### Table Creation\n\n```sql\n-- Main extraction results (one row per document)\nCREATE TABLE IF NOT EXISTS ade_extractions (\n run_id VARCHAR,\n doc_uuid VARCHAR PRIMARY KEY,\n document_name VARCHAR,\n processed_at TIMESTAMP_TZ,\n -- Add flattened extraction columns here\n -- e.g., invoice_info__invoice_number VARCHAR\n);\n\n-- Line items (one row per repeating element)\nCREATE TABLE IF NOT EXISTS ade_line_items (\n run_id VARCHAR,\n doc_uuid VARCHAR REFERENCES ade_extractions(doc_uuid),\n document_name VARCHAR,\n list_field VARCHAR,\n line_index INTEGER,\n -- Add line item columns here\n);\n\n-- Parsed chunks with bounding boxes\nCREATE TABLE IF NOT EXISTS ade_chunks (\n run_id VARCHAR,\n doc_uuid VARCHAR REFERENCES ade_extractions(doc_uuid),\n document_name VARCHAR,\n chunk_id VARCHAR,\n chunk_type VARCHAR,\n text VARCHAR,\n page INTEGER,\n box_l FLOAT,\n box_t FLOAT,\n box_r FLOAT,\n box_b FLOAT\n);\n\n-- Full markdown for traceability\nCREATE TABLE IF NOT EXISTS ade_markdown (\n run_id VARCHAR,\n doc_uuid VARCHAR REFERENCES ade_extractions(doc_uuid),\n document_name VARCHAR,\n processed_at TIMESTAMP_TZ,\n markdown VARCHAR(16777216)\n);\n```\n\n### Upload DataFrames\n\n```python\ndef upload_to_snowflake(\n conn: snowflake.connector.SnowflakeConnection,\n df_main: \"pd.DataFrame\",\n df_lines: \"pd.DataFrame\",\n df_chunks: \"pd.DataFrame\",\n df_md: \"pd.DataFrame\",\n) -> None:\n \"\"\"Upload all 4 normalized tables to Snowflake.\"\"\"\n # Column names must be UPPER CASE for Snowflake\n for table, df in [\n (\"ADE_EXTRACTIONS\", df_main),\n (\"ADE_LINE_ITEMS\", df_lines),\n (\"ADE_CHUNKS\", df_chunks),\n (\"ADE_MARKDOWN\", df_md),\n ]:\n if df.empty:\n continue\n df.columns = [c.upper() for c in df.columns]\n write_pandas(\n conn, df, table,\n auto_create_table=True,\n overwrite=False,\n )\n print(f\"Uploaded {len(df)} rows to {table}\")\n```\n\n### Full Pipeline: Parse → Extract → Snowflake\n\n```python\nimport io\nimport pandas as pd\nfrom pathlib import Path\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\n\ndef ade_to_snowflake(\n input_dir: Path,\n schema_cls: type,\n sf_conn: \"snowflake.connector.SnowflakeConnection\",\n run_id: str = \"default\",\n) -> int:\n \"\"\"Parse, extract, normalize, and upload to Snowflake.\n\n Returns number of documents processed.\n \"\"\"\n client = LandingAIADE()\n exts = {\".pdf\", \".png\", \".jpg\", \".jpeg\"}\n files = [\n p for p in input_dir.glob(\"*\")\n if p.suffix.lower() in exts\n ]\n\n all_main, all_lines, all_chunks, all_md = (\n [], [], [], []\n )\n for fp in files:\n try:\n pr = client.parse(document=fp)\n er = client.extract(\n schema=pydantic_to_json_schema(schema_cls),\n markdown=io.BytesIO(\n pr.markdown.encode(\"utf-8\")\n ),\n )\n main, lines, chunks, md = rows_from_doc(\n str(fp), pr, er, run_id=run_id\n )\n all_main.append(main)\n all_lines.extend(lines)\n all_chunks.extend(chunks)\n all_md.append(md)\n except Exception as exc:\n print(f\"FAILED {fp.name}: {exc}\")\n\n upload_to_snowflake(\n sf_conn,\n pd.DataFrame(all_main),\n pd.DataFrame(all_lines),\n pd.DataFrame(all_chunks),\n pd.DataFrame(all_md),\n )\n return len(all_main)\n```\n\n---\n\n## 3. CSV Export Patterns\n\n### Summary CSV — One Row per Document\n\n```python\ndef extractions_to_summary_csv(\n results: list[tuple[str, dict]],\n output_path: Path,\n) -> \"pd.DataFrame\":\n \"\"\"Create a summary CSV with one row per document.\n\n Args:\n results: list of (filename, extraction_dict) tuples\n output_path: CSV file path\n \"\"\"\n rows = []\n for name, extraction in results:\n row = {\"document_name\": name}\n for k, v in extraction.items():\n if isinstance(v, dict):\n for sk, sv in v.items():\n row[f\"{k}__{sk}\"] = sv\n elif isinstance(v, list):\n row[f\"{k}__count\"] = len(v)\n else:\n row[k] = v\n rows.append(row)\n df = pd.DataFrame(rows)\n df.to_csv(output_path, index=False)\n return df\n```\n\n### Per-Document JSON + Combined CSV\n\n```python\nimport json\n\ndef save_results(\n file_path: Path,\n parse_result: Any,\n extract_result: Any,\n output_dir: Path,\n) -> None:\n \"\"\"Save individual JSON files + append to combined CSV.\"\"\"\n stem = file_path.stem\n output_dir.mkdir(parents=True, exist_ok=True)\n\n # Individual JSON\n for prefix, obj in [\n (\"parse\", parse_result),\n (\"extract\", extract_result),\n ]:\n data = (\n obj.model_dump()\n if hasattr(obj, \"model_dump\")\n else obj\n )\n (output_dir / f\"{prefix}_{stem}.json\").write_text(\n json.dumps(data, indent=2, default=str),\n encoding=\"utf-8\",\n )\n```\n\n---\n\n## Dependencies\n\n```\n# DataFrame + CSV only\npip install landingai-ade pandas\n\n# Snowflake integration\npip install landingai-ade pandas snowflake-connector-python[pandas]\n\n# Environment variable management\npip install python-dotenv pydantic-settings\n```\n","content/landingai/skills/ade/document-workflows/references/rag-pipelines.md":"# RAG Pipeline Patterns\n\nEnd-to-end patterns for preparing ADE-parsed documents for Retrieval\nAugmented Generation (RAG) systems. Covers embedding computation,\nchunking strategies, vector DB ingestion, and query pipelines.\n\n---\n\n## 1. Chunks to CSV — RAG-Ready Dataset\n\nExtract all chunks from parsed documents into a structured CSV with 19\ncolumns including bounding boxes, sequence info, and metadata. This CSV\ncan feed any vector DB or search index.\n\n> **Grounding-aware records:** Every record includes `page`, `box_l`,\n> `box_t`, `box_r`, `box_b` from ADE's grounding data. Preserve these\n> columns when ingesting into a vector DB — they let you trace retrieval\n> results back to exact document locations for highlighting or citation.\n\n```python\nimport re\nfrom pathlib import Path\nfrom typing import Any, Dict, List\n\nimport pandas as pd\n\n\ndef clean_chunk_text(text: str) -> str:\n \"\"\"Remove anchor tags and strip whitespace.\"\"\"\n cleaned = re.sub(r\"\\s*\", \"\", text)\n return cleaned.strip()\n\n\ndef chunks_to_records(\n parse_result: Any,\n document_name: str,\n model_version: str = \"unknown\",\n) -> List[Dict[str, Any]]:\n \"\"\"Convert parse result chunks to flat dicts.\n\n Each dict has 19 columns suitable for CSV / DataFrame:\n DOCUMENT_NAME, chunk_id, chunk_sequence_number,\n chunk_type, chunk_content_raw, chunk_content,\n chunk_text_length, chunk_word_count, page,\n box_l, box_t, box_r, box_b,\n prev_chunk_id, next_chunk_id, chunk_image_path,\n processed_at, ade_version, model_version\n \"\"\"\n from datetime import datetime, timezone\n\n import landingai_ade\n\n chunks = parse_result.chunks or []\n now = datetime.now(timezone.utc).isoformat()\n records: List[Dict[str, Any]] = []\n\n for idx, ch in enumerate(chunks):\n raw = ch.markdown if hasattr(ch, \"markdown\") else \"\"\n clean = clean_chunk_text(raw)\n box = (\n ch.grounding.box\n if hasattr(ch, \"grounding\")\n and hasattr(ch.grounding, \"box\")\n else None\n )\n page = (\n ch.grounding.page\n if hasattr(ch, \"grounding\")\n else None\n )\n records.append({\n \"DOCUMENT_NAME\": document_name,\n \"chunk_id\": getattr(ch, \"id\", None),\n \"chunk_sequence_number\": idx,\n \"chunk_type\": getattr(ch, \"type\", None),\n \"chunk_content_raw\": raw,\n \"chunk_content\": clean,\n \"chunk_text_length\": len(clean),\n \"chunk_word_count\": len(clean.split()) if clean else 0,\n \"page\": page,\n \"box_l\": box.left if box else None,\n \"box_t\": box.top if box else None,\n \"box_r\": box.right if box else None,\n \"box_b\": box.bottom if box else None,\n \"prev_chunk_id\": (\n chunks[idx - 1].id if idx > 0 else None\n ),\n \"next_chunk_id\": (\n chunks[idx + 1].id\n if idx < len(chunks) - 1\n else None\n ),\n \"chunk_image_path\": None,\n \"processed_at\": now,\n \"ade_version\": landingai_ade.__version__,\n \"model_version\": model_version,\n })\n return records\n\n\ndef batch_chunks_to_csv(\n results: List[Dict[str, Any]],\n output_path: Path,\n) -> pd.DataFrame:\n \"\"\"Combine chunk records from multiple documents into\n one CSV.\n\n Args:\n results: list of dicts with keys 'name' and\n 'parse_result'\n output_path: CSV file path\n \"\"\"\n all_records: List[Dict[str, Any]] = []\n for r in results:\n all_records.extend(\n chunks_to_records(r[\"parse_result\"], r[\"name\"])\n )\n df = pd.DataFrame(all_records)\n df.to_csv(output_path, index=False)\n return df\n```\n\n### Usage\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\nresults = []\nfor fp in Path(\"docs/\").glob(\"*.pdf\"):\n pr = client.parse(document=fp)\n results.append({\"name\": fp.name, \"parse_result\": pr})\n\ndf = batch_chunks_to_csv(results, Path(\"all_chunks.csv\"))\nprint(f\"{len(df)} chunks from {df['DOCUMENT_NAME'].nunique()} docs\")\n```\n\n---\n\n## 2. Vector DB Ingestion — ChromaDB\n\nLocal persistent vector store using OpenAI embeddings. Good for\nprototyping and small-to-medium corpora.\n\n```python\nfrom pathlib import Path\nfrom typing import Any, List\n\nimport chromadb\nfrom chromadb.config import Settings\nfrom openai import OpenAI\n\n\ndef ade_chunks_to_chromadb(\n parse_results: List[dict],\n collection_name: str = \"ade_documents\",\n persist_dir: str = \"./chroma_db\",\n embedding_model: str = \"text-embedding-3-small\",\n) -> chromadb.Collection:\n \"\"\"Ingest ADE chunks into a persistent ChromaDB collection.\n\n Args:\n parse_results: list of dicts with 'name' (str) and\n 'parse_result' (ParseResponse)\n collection_name: ChromaDB collection name\n persist_dir: directory for persistent storage\n embedding_model: OpenAI embedding model name\n\n Returns:\n The ChromaDB collection with all chunks ingested.\n \"\"\"\n openai_client = OpenAI()\n chroma_client = chromadb.PersistentClient(\n path=persist_dir\n )\n collection = chroma_client.get_or_create_collection(\n name=collection_name,\n metadata={\"hnsw:space\": \"cosine\"},\n )\n\n for doc in parse_results:\n name = doc[\"name\"]\n chunks = doc[\"parse_result\"].chunks or []\n\n texts, ids, metadatas = [], [], []\n for ch in chunks:\n text = ch.markdown if hasattr(ch, \"markdown\") else \"\"\n if not text.strip():\n continue\n chunk_id = f\"{name}:{ch.id}\"\n texts.append(text)\n ids.append(chunk_id)\n metadatas.append({\n \"document\": name,\n \"chunk_type\": getattr(ch, \"type\", \"unknown\"),\n \"page\": (\n ch.grounding.page\n if hasattr(ch, \"grounding\")\n else -1\n ),\n })\n\n if not texts:\n continue\n\n # Generate embeddings in batches of 100\n all_embeddings: List[List[float]] = []\n for i in range(0, len(texts), 100):\n batch = texts[i : i + 100]\n resp = openai_client.embeddings.create(\n input=batch, model=embedding_model\n )\n all_embeddings.extend(\n [e.embedding for e in resp.data]\n )\n\n collection.add(\n ids=ids,\n documents=texts,\n embeddings=all_embeddings,\n metadatas=metadatas,\n )\n\n return collection\n```\n\n### Query ChromaDB\n\n```python\ndef query_chromadb(\n collection: chromadb.Collection,\n question: str,\n n_results: int = 5,\n embedding_model: str = \"text-embedding-3-small\",\n) -> dict:\n \"\"\"Query the collection and return matching chunks.\"\"\"\n openai_client = OpenAI()\n resp = openai_client.embeddings.create(\n input=[question], model=embedding_model\n )\n query_embedding = resp.data[0].embedding\n return collection.query(\n query_embeddings=[query_embedding],\n n_results=n_results,\n )\n```\n\n---\n\n## 3. Vector DB Ingestion — FAISS + LangChain\n\nFor LangChain-based RAG pipelines. Uses FAISS for in-memory vector\nsearch.\n\n```python\nfrom typing import Any, List\n\nfrom langchain.docstore.document import Document\nfrom langchain_community.vectorstores import FAISS\nfrom langchain_openai import OpenAIEmbeddings\n\n\ndef ade_to_langchain_docs(\n parse_results: List[dict],\n) -> List[Document]:\n \"\"\"Convert ADE parse results to LangChain Documents.\n\n Each chunk becomes one Document with metadata including\n source document name, chunk type, and page number.\n \"\"\"\n docs: List[Document] = []\n for item in parse_results:\n name = item[\"name\"]\n chunks = item[\"parse_result\"].chunks or []\n for ch in chunks:\n text = ch.markdown if hasattr(ch, \"markdown\") else \"\"\n if not text.strip():\n continue\n docs.append(Document(\n page_content=text,\n metadata={\n \"source\": name,\n \"chunk_type\": getattr(ch, \"type\", \"unknown\"),\n \"chunk_id\": getattr(ch, \"id\", \"\"),\n \"page\": (\n ch.grounding.page\n if hasattr(ch, \"grounding\")\n else -1\n ),\n },\n ))\n return docs\n\n\ndef build_faiss_index(\n documents: List[Document],\n embedding_model: str = \"text-embedding-3-small\",\n) -> FAISS:\n \"\"\"Build a FAISS vector store from LangChain Documents.\"\"\"\n embeddings = OpenAIEmbeddings(model=embedding_model)\n return FAISS.from_documents(documents, embeddings)\n```\n\n### RAG Query with LangChain\n\n```python\nfrom langchain.chains import RetrievalQA\nfrom langchain_openai import ChatOpenAI\n\n\ndef build_rag_chain(\n vectorstore: FAISS,\n model: str = \"gpt-4o-mini\",\n k: int = 5,\n) -> RetrievalQA:\n \"\"\"Build a RetrievalQA chain from a FAISS index.\"\"\"\n retriever = vectorstore.as_retriever(\n search_kwargs={\"k\": k}\n )\n return RetrievalQA.from_chain_type(\n llm=ChatOpenAI(model=model, temperature=0),\n chain_type=\"stuff\",\n retriever=retriever,\n return_source_documents=True,\n )\n\n\n# Usage\nchain = build_rag_chain(vectorstore)\nanswer = chain.invoke({\"query\": \"What is the total revenue?\"})\nprint(answer[\"result\"])\nfor doc in answer[\"source_documents\"]:\n print(f\" - {doc.metadata['source']} p{doc.metadata['page']}\")\n```\n\n---\n\n## 4. Full RAG Pipeline — End to End\n\nCombines parsing, chunking, vector DB, and querying into one flow.\n\n```python\nimport asyncio\nfrom pathlib import Path\n\nfrom langchain.chains import RetrievalQA\nfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings\nfrom langchain_community.vectorstores import FAISS\nfrom landingai_ade import LandingAIADE\n\n\ndef build_rag_from_folder(\n input_dir: Path,\n embedding_model: str = \"text-embedding-3-small\",\n llm_model: str = \"gpt-4o-mini\",\n) -> RetrievalQA:\n \"\"\"One-shot: parse all docs in a folder and build a\n RAG chain ready for querying.\n\n Returns a LangChain RetrievalQA chain.\n \"\"\"\n client = LandingAIADE()\n exts = {\".pdf\", \".png\", \".jpg\", \".jpeg\"}\n files = [\n p for p in input_dir.glob(\"*\")\n if p.suffix.lower() in exts\n ]\n\n # Parse all documents\n parse_results = []\n for fp in files:\n pr = client.parse(document=fp)\n parse_results.append({\"name\": fp.name, \"parse_result\": pr})\n\n # Convert to LangChain docs\n docs = ade_to_langchain_docs(parse_results)\n\n # Build vector store\n embeddings = OpenAIEmbeddings(model=embedding_model)\n vectorstore = FAISS.from_documents(docs, embeddings)\n\n # Build chain\n return RetrievalQA.from_chain_type(\n llm=ChatOpenAI(model=llm_model, temperature=0),\n chain_type=\"stuff\",\n retriever=vectorstore.as_retriever(\n search_kwargs={\"k\": 5}\n ),\n return_source_documents=True,\n )\n```\n\n### Usage\n\n```python\nchain = build_rag_from_folder(Path(\"10k_filings/\"))\nresult = chain.invoke(\n {\"query\": \"What were the main risk factors?\"}\n)\nprint(result[\"result\"])\n```\n\n---\n\n## 5. Embedding Computation\n\nTwo approaches for computing embeddings from ADE chunks: **local**\n(free, offline, fast) and **API-based** (higher quality, paid). Choose\nbased on your cost/quality tradeoff.\n\n### Local Embeddings with FastEmbed\n\nUses [FastEmbed](https://github.com/qdrant/fastembed) to run embedding\nmodels locally. No API key needed, no per-token cost, works offline.\n\n```python\nfrom typing import Any\n\nfrom fastembed import TextEmbedding\n\n\ndef compute_embeddings_local(\n texts: list[str],\n model: str = \"BAAI/bge-small-en-v1.5\",\n) -> list[list[float]]:\n \"\"\"Embed texts locally with FastEmbed (batched).\n\n Default model: BAAI/bge-small-en-v1.5 (384 dims, ~33M params).\n Other options:\n - BAAI/bge-base-en-v1.5 (768 dims, ~110M params)\n - sentence-transformers/all-MiniLM-L6-v2 (384 dims)\n \"\"\"\n embedder = TextEmbedding(model_name=model)\n return [v.tolist() for v in embedder.embed(texts)]\n```\n\n### API Embeddings with OpenAI\n\nHigher quality, especially for domain-specific content. Requires\n`OPENAI_API_KEY` and incurs per-token cost.\n\n```python\nfrom openai import OpenAI\n\n\ndef compute_embeddings_openai(\n texts: list[str],\n model: str = \"text-embedding-3-small\",\n batch_size: int = 100,\n) -> list[list[float]]:\n \"\"\"Embed texts via OpenAI API in batches.\"\"\"\n client = OpenAI()\n all_vecs: list[list[float]] = []\n for i in range(0, len(texts), batch_size):\n resp = client.embeddings.create(\n input=texts[i : i + batch_size], model=model,\n )\n all_vecs.extend(e.embedding for e in resp.data)\n return all_vecs\n```\n\n### Embedding Best Practices\n\n| Practice | Why | Example |\n|----------|-----|---------|\n| **Prepend title/heading** | Gives the embedding semantic context about what the chunk is about | `f\"{title}\\n\\n{body}\"` |\n| **Batch all texts in one call** | Faster than embedding one-by-one; both FastEmbed and OpenAI support batching | `embedder.embed(all_texts)` |\n| **Store model metadata** | Consumers need to know which model produced the vectors to query correctly | `{\"model\": \"bge-small-en-v1.5\", \"dims\": 384}` |\n| **Carry grounding refs** | Enables source attribution — trace retrieval hits back to page + bounding box | `{\"page\": 2, \"box\": {...}}` |\n| **Clean anchor tags first** | ADE chunks contain `` tags that add noise to embeddings | `re.sub(r\"\\s*\", \"\", text)` |\n\n### Self-Describing Embedding Output\n\nAlways store the embedding model name and dimensions alongside the\nvector so downstream consumers can interpret it correctly:\n\n```python\ndef make_embedding_record(\n text: str,\n vector: list[float],\n model: str,\n metadata: dict | None = None,\n) -> dict:\n \"\"\"Wrap a vector with its model info and metadata.\"\"\"\n return {\n \"text\": text,\n \"embedding\": {\n \"model\": model,\n \"dimensions\": len(vector),\n \"vector\": vector,\n },\n \"metadata\": metadata or {},\n }\n```\n\n### Model Selection Guide\n\n| Model | Dims | Cost | Quality | Best for |\n|-------|------|------|---------|----------|\n| `BAAI/bge-small-en-v1.5` | 384 | Free (local) | Good | Prototyping, cost-sensitive, offline |\n| `BAAI/bge-base-en-v1.5` | 768 | Free (local) | Better | Local with higher quality needs |\n| `text-embedding-3-small` | 1536 | ~$0.02/1M tokens | High | Production, mixed-domain content |\n| `text-embedding-3-large` | 3072 | ~$0.13/1M tokens | Highest | Maximum retrieval accuracy |\n\n---\n\n## 6. Multi-Granularity Embedding Strategy\n\nADE chunks are the finest-grained unit, but they're not always the\nright unit for embedding. Choose the granularity that matches your\nretrieval needs.\n\n### Granularity Levels\n\n| Level | Unit | How to build | Best for |\n|-------|------|-------------|----------|\n| **Chunk** | Raw ADE chunk | Direct from `parse_result.chunks` | Tables, figures, forms with independent fields |\n| **Hierarchical** | Group of consecutive chunks | Group by boundary detection (see below) | Narrative docs where answers span paragraphs |\n| **Document** | Full markdown or summary | `parse_result.markdown` or ADE extract summary | Classification, routing, coarse-grained search |\n\n### Chunk-Level (default)\n\nEach ADE chunk gets its own embedding. This is what Sections 2–4 above\nuse. Fine-grained but may split semantic units across multiple vectors.\n\n```python\n# Already shown in Sections 2-4 — each chunk → one embedding\ntexts = [\n clean_chunk_text(ch.markdown)\n for ch in parse_result.chunks\n if ch.markdown.strip()\n and getattr(ch, \"type\", \"\") in {\"text\", \"table\", \"card\"}\n]\n```\n\n### Hierarchical Chunking\n\nGroup consecutive ADE chunks into higher-level semantic units before\nembedding. The grouping boundary is **document-specific** — the pattern\nis always the same but the boundary detection varies:\n\n- **Heading detection** (regex or ADE extract) — for papers, reports\n- **Clause boundaries** (ADE split API or extract) — for contracts\n- **Page boundaries** — simple, works for any document\n- **Fixed-size sliding windows** — N consecutive chunks with overlap\n\nThe abstract pattern:\n\n```python\nfrom dataclasses import dataclass, field\nfrom typing import Any, Callable\n\n\n@dataclass\nclass ChunkGroup:\n \"\"\"A group of consecutive ADE chunks forming a semantic unit.\"\"\"\n label: str\n chunks: list[Any] = field(default_factory=list)\n grounding_refs: list[dict] = field(default_factory=list)\n\n @property\n def text(self) -> str:\n return \"\\n\".join(\n clean_chunk_text(ch.markdown)\n for ch in self.chunks if ch.markdown.strip()\n )\n\n @property\n def embedding_input(self) -> str:\n \"\"\"Prepend label for better embedding quality.\"\"\"\n return f\"{self.label}\\n\\n{self.text}\"\n\n\ndef group_chunks(\n chunks: list[Any],\n is_boundary: Callable[[Any], str | None],\n) -> list[ChunkGroup]:\n \"\"\"Group ADE chunks by a boundary detection function.\n\n Args:\n chunks: ADE parse_result.chunks\n is_boundary: function that returns a group label\n (str) if the chunk starts a new group, or\n None if it continues the current group.\n\n Returns:\n List of ChunkGroup with grounding refs preserved.\n \"\"\"\n groups: list[ChunkGroup] = []\n current: ChunkGroup | None = None\n for ch in chunks:\n label = is_boundary(ch)\n if label is not None:\n current = ChunkGroup(label=label)\n groups.append(current)\n if current is None:\n current = ChunkGroup(label=\"(preamble)\")\n groups.append(current)\n current.chunks.append(ch)\n if hasattr(ch, \"grounding\"):\n b = ch.grounding.box\n current.grounding_refs.append({\n \"page\": ch.grounding.page,\n \"box\": {\n \"left\": b.left, \"top\": b.top,\n \"right\": b.right, \"bottom\": b.bottom,\n },\n })\n return groups\n```\n\n**Example boundary detectors:**\n\n```python\nimport re\n\n# Page-based: new group every page\ndef by_page(ch: Any) -> str | None:\n page = ch.grounding.page if hasattr(ch, \"grounding\") else -1\n return f\"Page {page + 1}\" if not hasattr(by_page, \"_last\") or by_page._last != page else None\n # (simplified — use a closure or class for production)\n\n# Heading-based: new group on ATX or numbered headings\ndef by_heading(ch: Any) -> str | None:\n text = re.sub(r\"\\s*\", \"\", ch.markdown or \"\").strip()\n first_line = text.split(\"\\n\")[0].strip()\n if re.match(r\"^#{1,6}\\s+\", first_line):\n return re.sub(r\"^#{1,6}\\s+\", \"\", first_line)\n if re.match(r\"^\\d+(?:\\.\\d+)*\\.?\\s+[A-Z]\", first_line):\n return first_line\n return None\n```\n\n**Using groups for embedding:**\n\n```python\ngroups = group_chunks(parse_result.chunks, by_heading)\ntexts = [g.embedding_input for g in groups if g.text.strip()]\nvectors = compute_embeddings_local(texts)\n\n# Each group carries grounding_refs for source attribution\nfor g, vec in zip(groups, vectors):\n print(f\"{g.label}: {len(g.grounding_refs)} chunk refs, \"\n f\"{len(vec)} dims\")\n```\n\n### Document-Level\n\nEmbed the full document markdown or a summary. Useful for routing\nqueries to the right document before doing fine-grained search.\n\n```python\n# Full markdown (may be long — consider truncation)\ndoc_text = parse_result.markdown[:8000]\ndoc_vec = compute_embeddings_local([doc_text])[0]\n\n# Or use ADE extract to get a summary first\nfrom landingai_ade.lib import pydantic_to_json_schema\nfrom pydantic import BaseModel, Field\n\nclass DocSummary(BaseModel):\n summary: str = Field(\n description=\"A 2-3 sentence summary of the document.\"\n )\n\ner = client.extract(\n schema=pydantic_to_json_schema(DocSummary),\n markdown=parse_result.markdown,\n)\nsummary_vec = compute_embeddings_local(\n [er.extraction[\"summary\"]]\n)[0]\n```\n\n### Decision Matrix\n\n| Document Type | Recommended | Rationale |\n|--------------|-------------|-----------|\n| Academic papers, reports | Hierarchical (by heading) | Answers span paragraphs within sections |\n| Invoices, forms | Chunk-level | Each field is independent |\n| Mixed document batches | Document-level + chunk-level | Route first, then search within |\n| Contracts, legal docs | Hierarchical (by clause) | Clauses are the natural retrieval unit |\n| Slide decks | Chunk-level (by page) | Each slide is self-contained |\n| Long narratives (books) | Hierarchical (sliding window) | Fixed-size windows with overlap |\n\n---\n\n## Chunk Filtering Tips\n\nNot all chunks are useful for RAG. Filter by type to improve relevance:\n\n```python\n# Keep only text and table chunks (skip logos, scan codes)\nRAG_CHUNK_TYPES = {\"text\", \"table\", \"card\"}\n\ndocs = [\n Document(page_content=ch.markdown, metadata={...})\n for ch in parse_result.chunks\n if getattr(ch, \"type\", \"\") in RAG_CHUNK_TYPES\n and ch.markdown.strip()\n]\n```\n\n---\n\n## Dependencies\n\n```\n# Chunks to CSV only\npip install landingai-ade pandas\n\n# Local embeddings (free, offline)\npip install landingai-ade fastembed\n\n# ChromaDB pipeline (API embeddings)\npip install landingai-ade chromadb openai\n\n# FAISS + LangChain pipeline (API embeddings)\npip install landingai-ade langchain langchain-openai langchain-community faiss-cpu\n\n# Local embeddings + ChromaDB\npip install landingai-ade fastembed chromadb\n\n# Full pipeline (all options)\npip install landingai-ade pandas fastembed chromadb openai langchain langchain-openai langchain-community faiss-cpu\n```\n","content/landingai/skills/ade/document-workflows/references/schema-catalog.md":"# Schema Catalog — Ready-to-Use Pydantic Models\n\nReady-to-use extraction schemas for common document types.\nEach schema is a Pydantic `BaseModel` that can be converted to JSON Schema\nvia `pydantic_to_json_schema` and passed to `client.extract()`.\n\n> **Tip:** ADE supports **one level of nesting**. Use nested `BaseModel`\n> sub-classes for logical grouping, and `List[SubModel]` for repeating items\n> like line items or transactions.\n\n---\n\n## Usage Pattern (all schemas)\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nclient = LandingAIADE()\nparse_result = client.parse(document=path)\nextract_result = client.extract(\n schema=pydantic_to_json_schema(MySchema),\n markdown=parse_result.markdown,\n)\ndata: dict = extract_result.extraction\n```\n\n---\n\n## 1. Invoice Schema\n\n6 nested groups, 30+ fields. Covers invoices from any vendor/country.\n\n```python\nfrom typing import Optional, List\nfrom datetime import date\nfrom pydantic import BaseModel, Field\n\n\nclass DocumentInfo(BaseModel):\n invoice_date_raw: str = Field(\n ...,\n description=(\n \"Invoice date as found in the document.\"\n \" Do not reformat.\"\n ),\n )\n invoice_date: Optional[date] = Field(\n ..., description=\"Invoice date in YYYY-MM-DD.\"\n )\n invoice_number: str = Field(\n ..., description=\"Invoice number.\"\n )\n order_date: Optional[str] = Field(\n None, description=\"Order or purchase date.\"\n )\n po_number: Optional[str] = Field(\n None, description=\"Customer purchase order (PO) number.\"\n )\n status: Optional[str] = Field(\n None,\n description=\"Payment status (e.g., PAID, UNPAID).\",\n )\n\n\nclass CustomerInfo(BaseModel):\n sold_to_name: str = Field(\n ...,\n description=(\n \"Name of the customer billed.\"\n \" Can be a person or an organization.\"\n ),\n )\n sold_to_address: Optional[str] = Field(\n None, description=\"Address of the customer billed.\"\n )\n customer_email: Optional[str] = Field(\n None, description=\"Email address for the customer.\"\n )\n\n\nclass SupplierInfo(BaseModel):\n supplier_name: str = Field(\n ..., description=\"Name of the supplier company.\"\n )\n supplier_address: Optional[str] = Field(\n None, description=\"Address of the supplier.\"\n )\n representative: Optional[str] = Field(\n None, description=\"Sales representative(s).\"\n )\n email: Optional[str] = Field(\n None, description=\"Email address of the supplier.\"\n )\n phone: Optional[str] = Field(\n None, description=\"Phone number of the supplier.\"\n )\n gstin: Optional[str] = Field(\n None, description=\"GSTIN of the supplier (India).\"\n )\n pan: Optional[str] = Field(\n None, description=\"Permanent Account Number (India).\"\n )\n\n\nclass TermsAndShipping(BaseModel):\n payment_terms: Optional[str] = Field(\n None, description=\"Payment terms (e.g., Net 30).\"\n )\n ship_via: Optional[str] = Field(\n None, description=\"Carrier/service (e.g., UPS Ground).\"\n )\n ship_date: Optional[str] = Field(\n None, description=\"Date shipped.\"\n )\n tracking_number: Optional[str] = Field(\n None, description=\"Tracking number.\"\n )\n\n\nclass TotalsSummary(BaseModel):\n currency: Optional[str] = Field(\n None, description=\"ISO currency code.\"\n )\n total_due_raw: Optional[str] = Field(\n None, description=\"Total due as shown in the doc.\"\n )\n total_due: float = Field(\n ..., description=\"Total amount due (numeric, no symbols).\"\n )\n subtotal: Optional[float] = Field(\n None, description=\"Subtotal (numeric).\"\n )\n tax: Optional[float] = Field(\n None, description=\"Tax (numeric).\"\n )\n shipping: Optional[float] = Field(\n None, description=\"Shipping (numeric).\"\n )\n handling_fee: Optional[float] = Field(\n None, description=\"Handling fee (numeric).\"\n )\n\n\nclass LineItem(BaseModel):\n line_number: Optional[str] = Field(\n None, description=\"Printed line number.\"\n )\n sku: Optional[str] = Field(\n None, description=\"SKU / Item code / Part number.\"\n )\n description: str = Field(\n ..., description=\"Item or service description.\"\n )\n quantity: Optional[float] = Field(\n None, description=\"Quantity purchased.\"\n )\n unit_price: Optional[float] = Field(\n None, description=\"Unit price (numeric).\"\n )\n amount: Optional[float] = Field(\n None, description=\"Extended line amount (numeric).\"\n )\n\n\nclass InvoiceSchema(BaseModel):\n invoice_info: DocumentInfo = Field(\n description=\"Key identifiers and dates.\"\n )\n customer_info: CustomerInfo = Field(\n description=\"Details about the customer billed.\"\n )\n company_info: SupplierInfo = Field(\n description=\"Details about the issuing company.\"\n )\n order_details: TermsAndShipping = Field(\n description=\"Payment and shipping information.\"\n )\n totals_summary: TotalsSummary = Field(\n description=\"Financial totals by category.\"\n )\n line_items: List[LineItem] = Field(\n default_factory=list,\n description=\"List of items included in the invoice.\",\n )\n```\n\n---\n\n## 2. Utility Bill Schema\n\nProvider, account, billing summary, electric and gas charges.\n\n```python\nfrom typing import Optional\nfrom pydantic import BaseModel, Field\n\n\nclass ProviderInfo(BaseModel):\n provider: str = Field(\n ..., description=\"Name of the utility provider.\"\n )\n phone_number: Optional[str] = Field(\n None,\n description=\"Customer service phone (XXX-XXX-XXXX).\",\n )\n website: Optional[str] = Field(\n None, description=\"Official website URL.\"\n )\n usage_bar_chart: bool = Field(\n False,\n description=\"Does the bill include a usage trend chart?\",\n )\n\n\nclass AccountInfo(BaseModel):\n account_holder: str = Field(\n ...,\n description=(\n \"Full name of the account holder.\"\n \" May be a person or organization.\"\n ),\n )\n account_number: str = Field(\n ..., description=\"Unique customer account identifier.\"\n )\n service_address: str = Field(\n ...,\n description=(\n \"Full service address on one line\"\n \" (remove newlines, replace with space).\"\n ),\n )\n service_address_city: Optional[str] = Field(\n None, description=\"City of service address.\"\n )\n service_address_state: Optional[str] = Field(\n None, description=\"2-letter state abbreviation.\"\n )\n service_address_zip: Optional[str] = Field(\n None, description=\"5-digit ZIP code.\"\n )\n\n\nclass BillingSummary(BaseModel):\n due_date: str = Field(\n ..., description=\"Payment due date (YYYY-MM-DD).\"\n )\n bill_date: str = Field(\n ..., description=\"Bill issue date (YYYY-MM-DD).\"\n )\n service_start_date: Optional[str] = Field(\n None, description=\"Service period start (MM-DD-YYYY).\"\n )\n service_end_date: Optional[str] = Field(\n None, description=\"Service period end (MM-DD-YYYY).\"\n )\n total_amount_due: str = Field(\n ...,\n description=\"Total amount due including currency symbol.\",\n )\n\n\nclass ElectricCharges(BaseModel):\n meter_number: Optional[str] = Field(\n None,\n description=(\n \"Electric meter identifier.\"\n \" Blank if no electric service.\"\n ),\n )\n usage_kwh: Optional[str] = Field(\n None, description=\"Total kWh for billing period.\"\n )\n total_electric_charges: Optional[str] = Field(\n None,\n description=\"Total electric charges with currency symbol.\",\n )\n\n\nclass GasCharges(BaseModel):\n meter_number: Optional[str] = Field(\n None,\n description=(\n \"Gas meter identifier.\"\n \" Blank if no gas service.\"\n ),\n )\n usage_therms: Optional[str] = Field(\n None, description=\"Total therms for billing period.\"\n )\n total_gas_charges: Optional[str] = Field(\n None,\n description=\"Total gas charges with currency symbol.\",\n )\n\n\nclass UtilityBillSchema(BaseModel):\n provider_info: ProviderInfo = Field(\n description=\"Energy provider details.\"\n )\n account_info: AccountInfo = Field(\n description=\"Account and customer identifiers.\"\n )\n billing_summary: BillingSummary = Field(\n description=\"Charges and due dates.\"\n )\n electric_charges: ElectricCharges = Field(\n description=\"Electric usage and charges.\"\n )\n gas_charges: GasCharges = Field(\n description=\"Gas usage and charges.\"\n )\n```\n\n---\n\n## 3. Bank Statement Schema\n\n```python\nfrom typing import Optional, List\nfrom pydantic import BaseModel, Field\n\n\nclass BankTransaction(BaseModel):\n date: str = Field(\n ..., description=\"Transaction date (YYYY-MM-DD).\"\n )\n description: str = Field(\n ..., description=\"Transaction description.\"\n )\n amount: float = Field(\n ..., description=\"Transaction amount (numeric).\"\n )\n type: Optional[str] = Field(\n None,\n description=\"Transaction type: debit or credit.\",\n )\n\n\nclass BankStatementSchema(BaseModel):\n bank_name: str = Field(\n ..., description=\"Name of the bank.\"\n )\n account_number: str = Field(\n ..., description=\"Bank account number.\"\n )\n statement_period: Optional[str] = Field(\n None,\n description=\"Statement period (e.g., Jan 1 - Jan 31, 2025).\",\n )\n opening_balance: Optional[float] = Field(\n None, description=\"Opening balance (numeric).\"\n )\n closing_balance: float = Field(\n ..., description=\"Closing / current balance (numeric).\"\n )\n total_deposits: Optional[float] = Field(\n None, description=\"Total deposits (numeric).\"\n )\n total_withdrawals: Optional[float] = Field(\n None, description=\"Total withdrawals (numeric).\"\n )\n transactions: List[BankTransaction] = Field(\n default_factory=list,\n description=\"List of transactions in the statement.\",\n )\n```\n\n---\n\n## 3b. Multi-Page Table Schema (for Table Stitching)\n\nUse this pattern when a table spans multiple pages and you want the LLM\nto stitch all rows into a single list via `client.extract()`.\n\n```python\nfrom typing import List, Optional\nfrom pydantic import BaseModel, Field\n\n\nclass TableRow(BaseModel):\n \"\"\"Generic row — customize fields for your table.\"\"\"\n key_column: str = Field(\n description=(\n \"Primary identifier (e.g., date, ID). \"\n \"Use empty string for continuation rows.\"\n )\n )\n description: str = Field(\n description=\"Description or label column.\"\n )\n amount_a: Optional[str] = Field(\n default=None,\n description=(\n \"First amount column (digits and commas only, \"\n \"no currency symbol). \"\n \"Null if not applicable for this row.\"\n ),\n )\n amount_b: Optional[str] = Field(\n default=None,\n description=(\n \"Second amount column. \"\n \"Null if not applicable.\"\n ),\n )\n running_total: str = Field(\n description=(\n \"Running total or balance after this row.\"\n )\n )\n\n\nclass MultiPageTable(BaseModel):\n rows: List[TableRow] = Field(\n description=(\n \"All data rows in order across ALL pages of \"\n \"the document. Include rows from every page, \"\n \"even if some pages render as plain text \"\n \"rather than tables. \"\n \"Skip column-header rows and section-header \"\n \"rows.\"\n )\n )\n```\n\n**Schema design tips for multi-page tables:**\n- Say **\"across ALL pages\"** in the `List` field description\n- Mention **\"even if some pages render as plain text\"**\n- Say **\"Skip column-header rows\"** to avoid duplicated headers\n- Use `Optional[str]` for mutually exclusive amount columns\n- Use `str` (not `float`) for amounts to preserve original formatting\n- Add domain-specific hints in descriptions (e.g., \"running balance\n after this transaction\") to help the LLM resolve ambiguities\n\n> **Full patterns** for three table stitching approaches (parse+extract,\n> HTML parsing, pandas): see\n> [table-stitching.md](table-stitching.md).\n\n---\n\n## 4. Pay Stub Schema\n\n```python\nfrom typing import Optional, List\nfrom pydantic import BaseModel, Field\n\n\nclass Deduction(BaseModel):\n name: str = Field(\n ..., description=\"Deduction name (e.g., Federal Tax).\"\n )\n amount: float = Field(\n ..., description=\"Deduction amount (numeric).\"\n )\n\n\nclass PayStubSchema(BaseModel):\n employee_name: str = Field(\n ..., description=\"Full name of the employee.\"\n )\n employer_name: Optional[str] = Field(\n None, description=\"Name of the employer.\"\n )\n pay_period: str = Field(\n ..., description=\"Pay period covered by this stub.\"\n )\n pay_date: Optional[str] = Field(\n None, description=\"Payment date (YYYY-MM-DD).\"\n )\n gross_pay: float = Field(\n ..., description=\"Gross pay amount (numeric).\"\n )\n net_pay: float = Field(\n ..., description=\"Net pay after deductions (numeric).\"\n )\n total_deductions: Optional[float] = Field(\n None, description=\"Total deductions (numeric).\"\n )\n deductions: List[Deduction] = Field(\n default_factory=list,\n description=\"Itemized deductions.\",\n )\n```\n\n---\n\n## 5. Food / Product Label Schema\n\n27 fields covering identification, weight, certifications, and dietary claims.\n\n```python\nfrom pydantic import BaseModel, Field\n\n\nclass ProductLabelSchema(BaseModel):\n # Identification\n product_name: str = Field(\n ...,\n description=(\n \"Full product name excluding brand\"\n \" as it appears on packaging.\"\n ),\n )\n brand: str = Field(\n ..., description=\"Brand or company name.\"\n )\n product_type: str = Field(\n ...,\n description=(\n \"General category (e.g., yogurt, hot dogs,\"\n \" supplement, beef sticks).\"\n ),\n )\n flavor: str = Field(\n ...,\n description=(\n \"Flavor if applicable. Empty if not found.\"\n ),\n )\n # Weight / Serving\n net_weight_oz: float = Field(\n ..., description=\"Net weight in ounces.\"\n )\n net_weight_g: float = Field(\n ..., description=\"Net weight in grams.\"\n )\n servings_per_container: int = Field(\n ..., description=\"Total servings per container.\"\n )\n serving_size: str = Field(\n ...,\n description=\"Serving size as printed (e.g., '1 stick (45g)').\",\n )\n # Certifications (bool flags)\n is_organic: bool = Field(\n ..., description=\"True if labeled Organic / USDA Organic.\"\n )\n is_non_gmo: bool = Field(\n ..., description=\"True if Non-GMO Project Verified.\"\n )\n is_grass_fed: bool = Field(\n ..., description=\"True if labeled Grass-Fed.\"\n )\n is_kosher: bool = Field(\n ..., description=\"True if Kosher certified.\"\n )\n is_gluten_free: bool = Field(\n ..., description=\"True if labeled Gluten-Free.\"\n )\n # Dietary claims\n is_keto_friendly: bool = Field(\n ..., description=\"True if labeled Keto.\"\n )\n is_dairy_free: bool = Field(\n ..., description=\"True if labeled Dairy-Free.\"\n )\n has_no_added_sugar: bool = Field(\n ...,\n description=\"True if No Added Sugar / Zero Sugar.\",\n )\n```\n\n> **Note:** The full food label schema has 27 fields including\n> `is_pasture_raised`, `is_certified_humane`, `no_antibiotics`,\n> `no_hormones`, `is_regenerative`, `is_paleo_friendly`,\n> `is_whole30_approved`, `is_lactose_free`, `usda_inspected`, etc.\n> Add fields as needed for your use case.\n\n---\n\n## 6. CME / Continuing Education Certificate\n\n```python\nfrom typing import Optional\nfrom pydantic import BaseModel, Field\n\n\nclass CMECertificateSchema(BaseModel):\n recipient_name: str = Field(\n ..., description=\"Full name of the certificate recipient.\"\n )\n issuing_organization: str = Field(\n ...,\n description=\"Organization that issued the certificate.\",\n )\n activity_title: str = Field(\n ...,\n description=\"Title of the educational activity or course.\",\n )\n completion_date: Optional[str] = Field(\n None, description=\"Completion date (YYYY-MM-DD).\"\n )\n credits_earned: float = Field(\n ..., description=\"Number of credits earned (numeric).\"\n )\n credit_type: Optional[str] = Field(\n None,\n description=(\n \"Type of credit (e.g., CME, CE, CPE, CLE).\"\n ),\n )\n certificate_id: Optional[str] = Field(\n None, description=\"Certificate or confirmation number.\"\n )\n accreditation_statement: Optional[str] = Field(\n None,\n description=\"Accreditation or approval statement text.\",\n )\n```\n\n---\n\n## 7. Document Classification Schema\n\nGeneric classifier using `Literal` enum. Adapt the type list to your\ndocument mix.\n\n```python\nfrom typing import Literal\nfrom pydantic import BaseModel, Field\n\n\nclass DocTypeClassification(BaseModel):\n type: Literal[\n \"invoice\",\n \"bank_statement\",\n \"pay_stub\",\n \"utility_bill\",\n \"receipt\",\n \"contract\",\n ] = Field(\n description=\"The type of the document.\",\n title=\"Document Type\",\n )\n```\n\n### Classify-then-Extract Pattern\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nschema_map = {\n \"invoice\": InvoiceSchema,\n \"bank_statement\": BankStatementSchema,\n \"pay_stub\": PayStubSchema,\n \"utility_bill\": UtilityBillSchema,\n}\n\nclient = LandingAIADE()\nparse_result = client.parse(document=path)\n\n# Step 1: Classify\ncls_result = client.extract(\n schema=pydantic_to_json_schema(DocTypeClassification),\n markdown=parse_result.markdown,\n)\ndoc_type: str = cls_result.extraction[\"type\"]\n\n# Step 2: Extract with type-specific schema\nschema_cls = schema_map[doc_type]\nextract_result = client.extract(\n schema=pydantic_to_json_schema(schema_cls),\n markdown=parse_result.markdown,\n)\n```\n\n---\n\n## Tips for Custom Schemas\n\n1. **Use descriptive `description` fields** — ADE uses them as extraction\n instructions. Be specific about format expectations.\n2. **Mark optional fields** with `Optional[T]` and `default=None` to avoid\n extraction failures when a field is absent.\n3. **Use `default_factory=list`** for array fields (line items,\n transactions) to avoid null results.\n4. **Keep nesting to one level** — ADE supports `TopLevel.nested_object`\n but not `TopLevel.nested.deeper`.\n5. **Use `Literal` for enums** — constrains extraction to known values.\n6. **Prefer `float` over `str` for monetary amounts** — easier downstream\n processing. Keep a `_raw` string variant if you need the original format.\n","content/landingai/skills/ade/document-workflows/references/table-stitching.md":"# Table Stitching — Multi-Page Table Extraction\n\nWhen a table spans multiple pages, ADE emits separate chunks per page and\nmay represent some pages as plain text instead of `
` HTML. This\ninconsistency can occur on **any** page — not just the last one. This\nreference covers three approaches to stitch those chunks into a single\noutput, generalized for any document type.\n\n---\n\n## Decision Guide\n\n| Approach | ADE Calls | Handles non-table chunks | Fragility | Best when |\n|----------|-----------|--------------------------|-----------|-----------|\n| **A — Parse + Extract** | 2 | ✓ LLM reads full markdown | Low | Accuracy is paramount; cost is secondary |\n| **B — HTML table parsing** | 1 | ✓ with fallback regex | **High** — requires uniform row structure | Rows are highly uniform; cost savings justify fragility |\n| **C — pandas read_html** | 1 | ✗ misses non-table chunks | Medium | Quick prototyping; missing some rows is acceptable |\n\n---\n\n## Approach A — Parse + Extract (LLM-based)\n\nThe simplest and most robust approach. Parse the document, then call\n`client.extract()` with a schema that describes the full table as a\n`List[RowModel]`. The LLM reads the entire markdown — including any\npages where the table was emitted as plain text — and returns structured\nJSON.\n\n### Schema design for multi-page tables\n\n```python\nfrom typing import List, Optional\nfrom pydantic import BaseModel, Field\n\n\nclass TableRow(BaseModel):\n \"\"\"Customize fields for your specific table.\"\"\"\n key_column: str = Field(\n description=(\n \"Primary identifier (e.g., date, ID, row number). \"\n \"Use empty string for continuation rows.\"\n )\n )\n description: str = Field(\n description=\"Description or label column.\"\n )\n amount_a: Optional[str] = Field(\n default=None,\n description=(\n \"First amount column (digits and commas only, \"\n \"no currency symbol). \"\n \"Null if not applicable for this row.\"\n ),\n )\n amount_b: Optional[str] = Field(\n default=None,\n description=(\n \"Second amount column. \"\n \"Null if not applicable.\"\n ),\n )\n running_total: str = Field(\n description=\"Running total or balance after this row.\"\n )\n\n\nclass DocumentWithTable(BaseModel):\n rows: List[TableRow] = Field(\n description=(\n \"All data rows in order across ALL pages of the \"\n \"document. Include rows from every page of the \"\n \"table, even if some pages render as plain text \"\n \"rather than tables. \"\n \"Skip column-header rows and section-header rows.\"\n )\n )\n```\n\n### Key schema tips\n\n- **Say \"across ALL pages\"** in the `List` field description — this\n tells the LLM to look beyond the first table chunk.\n- **Mention \"even if some pages render as plain text\"** — the LLM\n will then scan text chunks for table-like content.\n- **Say \"Skip column-header rows\"** — continued tables often repeat\n headers on each page.\n- **Use `Optional[str]` for mutually exclusive columns** — e.g., when\n only one of \"debit\" or \"credit\" applies per row.\n- **Use `str` (not `float`) for amounts** to preserve original\n formatting (commas, decimals). Convert downstream if needed.\n\n### Implementation pattern\n\n```python\nimport json\nfrom pathlib import Path\n\nfrom landingai_ade import LandingAIADE\nfrom landingai_ade.lib import pydantic_to_json_schema\n\nclient = LandingAIADE()\n\n# Step 1: Parse (cache the result to avoid re-parsing)\nparse_json = Path(\"output/parsed.json\")\nif parse_json.exists():\n data = json.loads(parse_json.read_text())\n markdown = data[\"markdown\"]\nelse:\n pr = client.parse(document=Path(\"document.pdf\"))\n parse_json.parent.mkdir(parents=True, exist_ok=True)\n parse_json.write_text(\n json.dumps(pr.model_dump(), indent=2, default=str)\n )\n markdown = pr.markdown\n\n# Step 2: Extract with the multi-page table schema\ner = client.extract(\n schema=pydantic_to_json_schema(DocumentWithTable),\n markdown=markdown,\n)\nrows = er.extraction[\"rows\"]\n```\n\n### Pros / Cons\n\n- ✅ Handles all pages uniformly, including plain-text edge cases\n- ✅ No custom parsing code\n- ❌ Two ADE API calls → ~2× credit cost\n- ❌ Slower: two network round-trips\n\n---\n\n## Approach B — HTML Table Parsing (parse-only, fragile)\n\nReuse the cached parse result (zero extra credits). Parse `
`\nelements from the ADE markdown, detect which tables belong to the\ntarget table, merge their rows, then fall back to regex for any\nplain-text rows.\n\n> **Warning:** This approach requires **strong similarity between rows**\n> to write reliable detection and extraction regex. It is brittle when\n> table format varies across documents or even across pages of the same\n> document. Always validate against Approach A on a sample before\n> relying on this in production.\n\n### Generic HTML table parser\n\n```python\nfrom html.parser import HTMLParser\n\n\nclass TableParser(HTMLParser):\n \"\"\"Extract all
elements as list-of-rows-of-cells.\"\"\"\n\n def __init__(self) -> None:\n super().__init__()\n self._tables: list[list[list[str]]] = []\n self._tbl: list[list[str]] | None = None\n self._row: list[str] | None = None\n self._cell: list[str] | None = None\n\n def handle_starttag(self, tag: str, attrs: list) -> None:\n if tag == \"table\":\n self._tbl = []\n elif tag == \"tr\" and self._tbl is not None:\n self._row = []\n elif tag == \"td\" and self._row is not None:\n self._cell = []\n\n def handle_endtag(self, tag: str) -> None:\n if tag == \"table\" and self._tbl is not None:\n self._tables.append(self._tbl)\n self._tbl = None\n elif tag == \"tr\" and self._tbl is not None:\n if self._row is not None:\n self._tbl.append(self._row)\n self._row = None\n elif tag == \"td\" and self._row is not None:\n if self._cell is not None:\n self._row.append(\n \"\".join(self._cell).strip()\n )\n self._cell = None\n\n def handle_data(self, data: str) -> None:\n if self._cell is not None:\n self._cell.append(data)\n\n @property\n def tables(self) -> list[list[list[str]]]:\n return self._tables\n\n\ndef extract_html_tables(\n markdown: str,\n) -> list[list[list[str]]]:\n p = TableParser()\n p.feed(markdown)\n return p.tables\n```\n\n### Table detection strategies\n\nColumn count alone is often insufficient — multiple table types may\nshare the same number of columns. Use content-based signals:\n\n```\nIs there a header row with known column names?\n ├─ YES → Match on header content (most reliable)\n │ e.g., cells[0]==\"Date\" and \"Description\" in cells[1]\n └─ NO → Does the first data cell match a known pattern?\n ├─ YES → Match on first-column pattern\n │ e.g., regex for \"Mon DD\" date format\n └─ NO → Use column count + content heuristics\n (least reliable — last resort)\n```\n\n### Row filtering\n\nAfter detecting the target table, filter out non-data rows:\n\n- **Column-header rows** — repeated on each page (e.g., `cells[0] == \"Date\"`)\n- **Section sub-headers** — account names, category labels\n- **Summary/totals rows** — may need special handling\n\n### Plain-text fallback\n\nWhen ADE emits table rows as a text chunk, use regex to extract them.\nThis is the most fragile part — it requires the text to have a\npredictable structure:\n\n```python\nimport re\n\n# Example: lines starting with a date pattern\nDATE_RE = re.compile(\n r\"^((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\"\n r\"\\s+\\d{1,2})\\s+(.*)\"\n)\nAMOUNT_RE = re.compile(r\"[\\d,]+\\.\\d{2}\")\n\n\ndef parse_text_rows(\n text: str,\n) -> list[dict[str, str]]:\n \"\"\"Extract rows from plain-text table content.\n\n Customize the date/amount patterns for your document type.\n \"\"\"\n rows: list[dict[str, str]] = []\n for line in text.splitlines():\n m = DATE_RE.match(line.strip())\n if not m:\n continue\n date_str, rest = m.group(1), m.group(2)\n amounts = AMOUNT_RE.findall(rest)\n desc = AMOUNT_RE.sub(\"\", rest).strip()\n rows.append({\n \"date\": date_str,\n \"description\": desc,\n \"amounts\": amounts,\n })\n return rows\n```\n\n### Domain-specific semantic checks\n\nAfter stitching, add validation that leverages domain knowledge to\nboth **confirm correctness** and **resolve ambiguity**:\n\n| Domain | Check | Example |\n|--------|-------|---------|\n| Financial | Running balance | `prev ± amount ≈ new_balance` |\n| Financial | Column totals | Sum of rows = reported total |\n| Inventory | Quantity conservation | `in - out = remaining` |\n| Time-series | Chronological order | Dates are monotonically increasing |\n| Scientific | Consistent units | All values in same column share units |\n\nThese checks are especially valuable when plain-text fallback\nproduces amounts that could belong to multiple columns.\n\n### Pros / Cons\n\n- ✅ Single ADE API call → half the credit cost\n- ✅ Fast (parse result is cached; parsing runs locally)\n- ❌ Requires strong row similarity for reliable regex\n- ❌ Brittle — format changes across documents break detection\n- ❌ Plain-text fallback needs domain-specific validation\n\n---\n\n## Approach C — pandas `read_html` (parse-only, quick)\n\nLet pandas do the heavy lifting. Extract `
` HTML strings with\na regex, feed each to `pd.read_html()`, use pandas signals to detect\ntarget tables, then `pd.concat` to merge.\n\n> **Limitation:** This approach cannot recover rows from non-table\n> chunks. If ADE emits some pages as plain text, those rows are lost.\n\n### Implementation pattern\n\n```python\nimport re\nfrom io import StringIO\n\nimport pandas as pd\n\nTABLE_RE = re.compile(r\"]*>.*?
\", re.DOTALL)\n\n\ndef stitch_tables_pandas(\n markdown: str,\n expected_cols: int,\n date_pattern: str = (\n r\"^(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\"\n r\"\\s+\\d+\"\n ),\n) -> pd.DataFrame:\n \"\"\"Extract and merge multi-page tables from ADE markdown.\n\n Args:\n markdown: Full ADE markdown output.\n expected_cols: Expected number of columns in the target table.\n date_pattern: Regex to match date-like values in column 0.\n \"\"\"\n table_htmls = TABLE_RE.findall(markdown)\n target_dfs: list[pd.DataFrame] = []\n\n for html in table_htmls:\n df = pd.read_html(StringIO(html), thousands=\",\")[0]\n if df.shape[1] != expected_cols:\n continue\n col0 = df.iloc[:, 0]\n has_date = bool(\n col0.astype(str).str.match(date_pattern).any()\n )\n has_numeric = (\n df.select_dtypes(include=\"number\").shape[1] >= 1\n )\n if has_date or has_numeric:\n target_dfs.append(df)\n\n if not target_dfs:\n raise ValueError(\"No target tables found\")\n return pd.concat(target_dfs, ignore_index=True)\n```\n\n### Detection signals from pandas\n\n| Signal | How | Use |\n|--------|-----|-----|\n| `df.shape` | Column count | Filter by expected column count |\n| `select_dtypes(\"number\")` | Numeric column count | Tables without headers have auto-inferred numeric cols |\n| `col0.str.match(pattern)` | First-column pattern | Date, ID, or other key column pattern |\n| `col0.str.strip() == \"Header\"` | Header presence | Detect repeated column headers to drop |\n\n### Pitfall: pandas 3.x NaN trap\n\nAfter `pd.read_html`, empty cells are `float('nan')`. In pandas 3.x,\nafter `.astype(str)`, StringDtype keeps them as the `float('nan')`\n**object** rather than the string `\"nan\"`. This means:\n\n```python\n# BROKEN in pandas 3.x:\ncol_str = df.iloc[:, 0].astype(str)\nmask = col_str == \"nan\" # Always False!\n\n# CORRECT — check on the raw column before string conversion:\nmask = pd.isna(df.iloc[:, 0])\n```\n\n### Amount formatting\n\n`pd.read_html(thousands=\",\")` strips commas from numbers during dtype\ninference. If you need comma-formatted output in CSV:\n\n```python\ndef fmt_amount(val: object) -> str:\n if pd.isna(val):\n return \"\"\n if isinstance(val, float):\n return f\"{val:,.2f}\"\n s = str(val).strip()\n try:\n return f\"{float(s):,.2f}\"\n except ValueError:\n return s\n```\n\n### Pros / Cons\n\n- ✅ Least code — `pd.read_html` + `pd.concat` do most of the work\n- ✅ No custom HTML parser or regex\n- ✅ pandas dtype signals give a clear detection path\n- ❌ Cannot recover rows from non-table chunks\n- ❌ `pd.NA` / `float('nan')` trap in pandas 3.x\n- ❌ `thousands=\",\"` strips commas — must re-format for output\n\n---\n\n## Common Pitfalls\n\n1. **ADE may emit any page as plain text** — not just the last page.\n Always check chunk types across all pages during pre-flight.\n\n2. **Column count alone is insufficient** — when multiple table types\n share the same column count, use content-based detection (header\n matching, first-column patterns, dtype signals).\n\n3. **Approach B regex is brittle** — test on multiple documents before\n relying on it. Format variations across documents (or even across\n pages of the same document) will break detection.\n\n4. **Always validate with domain-specific semantic checks** — these\n catch errors that structural parsing misses and resolve ambiguities\n in column assignment.\n\n5. **Cache parse results** — save `pr.model_dump()` to JSON after the\n first parse. Load it for development instead of calling\n `client.parse()` again. Only re-parse when the document changes.\n\n---\n\n## Pre-Flight Checklist for Table Stitching\n\nBefore choosing an approach, run the diagnostic parse and check:\n\n| What to check | How | Why |\n|---------------|-----|-----|\n| Chunk types per page | Count `type == \"table\"` vs `\"text\"` per page | Any page may have inconsistent types |\n| Column count consistency | Compare column counts across table chunks | Inconsistent counts may indicate different tables |\n| Header row presence | Check first row of each table chunk | Needed for detection and row filtering |\n| Non-target tables | Look for summary/metadata tables with same column count | Must distinguish target from others |\n| Row uniformity | Compare row structure across pages | Low uniformity makes Approach B fragile |\n| Plain-text table content | Inspect text chunks for table-like patterns | Determines if fallback is needed |\n","content/landingai/skills/ade/document-workflows/references/visualization.md":"# Visualization Patterns\n\nPatterns for visualizing ADE parse and extraction results: chunk image\ncropping, bounding box overlays, and word-level grounding highlights.\n\n---\n\n## 1. Chunk Image Extraction\n\nCrop individual chunks from document pages using bounding box coordinates.\nUseful for QA, debugging, and building visual search indexes.\n\n```python\nfrom pathlib import Path\nfrom typing import Any, List, Optional\n\nfrom PIL import Image\n\ntry:\n import pymupdf\nexcept ImportError:\n pymupdf = None # type: ignore[assignment]\n\n\ndef save_chunk_images(\n parse_result: Any,\n document_path: Path,\n output_dir: Path,\n zoom: float = 2.0,\n) -> Optional[Path]:\n \"\"\"Crop and save each chunk as a PNG image.\n\n Creates: output_dir//page_/..png\n\n Args:\n parse_result: from client.parse()\n document_path: original document file\n output_dir: base directory for chunk images\n zoom: render scale factor (2.0 = 144 DPI)\n\n Returns:\n Path to created document directory, or None on error.\n \"\"\"\n if pymupdf is None:\n print(\"Install pymupdf: pip install pymupdf\")\n return None\n\n doc_dir = output_dir / document_path.stem\n chunks = parse_result.chunks or []\n\n def _save_page_chunks(\n img: Image.Image,\n page_chunks: List[Any],\n page_num: int,\n ) -> None:\n w, h = img.size\n page_dir = doc_dir / f\"page_{page_num}\"\n page_dir.mkdir(parents=True, exist_ok=True)\n for ch in page_chunks:\n if (\n not hasattr(ch, \"grounding\")\n or ch.grounding.page != page_num\n ):\n continue\n box = ch.grounding.box\n crop = img.crop((\n int(box.left * w),\n int(box.top * h),\n int(box.right * w),\n int(box.bottom * h),\n ))\n fname = f\"{ch.type}.{ch.id}.png\"\n crop.save(page_dir / fname)\n\n try:\n if document_path.suffix.lower() == \".pdf\":\n pdf = pymupdf.open(document_path)\n mat = pymupdf.Matrix(zoom, zoom)\n for page_num in range(len(pdf)):\n pix = pdf[page_num].get_pixmap(matrix=mat)\n img = Image.frombytes(\n \"RGB\", [pix.width, pix.height], pix.samples\n )\n _save_page_chunks(img, chunks, page_num)\n pdf.close()\n else:\n img = Image.open(document_path).convert(\"RGB\")\n _save_page_chunks(img, chunks, 0)\n return doc_dir\n except Exception as exc:\n print(f\"Failed to save chunk images: {exc}\")\n return None\n```\n\n### Usage\n\n```python\nfrom landingai_ade import LandingAIADE\nfrom pathlib import Path\n\nclient = LandingAIADE()\npr = client.parse(document=Path(\"report.pdf\"))\nsave_chunk_images(pr, Path(\"report.pdf\"), Path(\"chunk_images/\"))\n# Creates: chunk_images/report/page_0/text.abc123.png, etc.\n```\n\n---\n\n## 2. Grounding Overlay — Bounding Boxes on Pages\n\nDraw color-coded bounding boxes on rendered page images to show where\neach chunk was detected.\n\n```python\nfrom pathlib import Path\nfrom typing import Any, Dict, Tuple\n\nfrom PIL import Image, ImageDraw\n\ntry:\n import pymupdf\nexcept ImportError:\n pymupdf = None # type: ignore[assignment]\n\n# Color map for chunk types (RGB tuples)\nCHUNK_COLORS: Dict[str, Tuple[int, int, int]] = {\n \"text\": (40, 167, 69), # green\n \"table\": (0, 123, 255), # blue\n \"marginalia\": (111, 66, 193), # purple\n \"figure\": (255, 0, 255), # magenta\n \"logo\": (144, 238, 144), # light green\n \"card\": (255, 165, 0), # orange\n \"attestation\": (0, 255, 255), # cyan\n \"scan_code\": (255, 193, 7), # yellow\n}\nDEFAULT_COLOR = (200, 200, 200)\n\n\ndef render_page_image(\n document_path: Path,\n page_num: int,\n zoom: float = 2.0,\n) -> Image.Image:\n \"\"\"Render a single page as a PIL Image.\"\"\"\n if document_path.suffix.lower() == \".pdf\":\n if pymupdf is None:\n raise ImportError(\"pip install pymupdf\")\n pdf = pymupdf.open(document_path)\n pix = pdf[page_num].get_pixmap(\n matrix=pymupdf.Matrix(zoom, zoom)\n )\n img = Image.frombytes(\n \"RGB\", [pix.width, pix.height], pix.samples\n )\n pdf.close()\n return img\n return Image.open(document_path).convert(\"RGB\")\n\n\ndef annotate_page(\n img: Image.Image,\n chunks: list,\n page_num: int,\n line_width: int = 3,\n) -> Image.Image:\n \"\"\"Draw bounding boxes for all chunks on a page image.\"\"\"\n annotated = img.copy()\n draw = ImageDraw.Draw(annotated)\n w, h = img.size\n\n for ch in chunks:\n if (\n not hasattr(ch, \"grounding\")\n or ch.grounding.page != page_num\n ):\n continue\n box = ch.grounding.box\n color = CHUNK_COLORS.get(\n getattr(ch, \"type\", \"\"), DEFAULT_COLOR\n )\n coords = [\n int(box.left * w),\n int(box.top * h),\n int(box.right * w),\n int(box.bottom * h),\n ]\n draw.rectangle(coords, outline=color, width=line_width)\n return annotated\n\n\ndef visualize_parse(\n parse_result: Any,\n document_path: Path,\n output_dir: Path,\n zoom: float = 2.0,\n) -> None:\n \"\"\"Render all pages with chunk bounding box overlays.\n\n Saves: output_dir//page__annotated.png\n \"\"\"\n doc_dir = output_dir / document_path.stem\n doc_dir.mkdir(parents=True, exist_ok=True)\n chunks = parse_result.chunks or []\n\n # Determine page count\n if document_path.suffix.lower() == \".pdf\":\n pdf = pymupdf.open(document_path)\n n_pages = len(pdf)\n pdf.close()\n else:\n n_pages = 1\n\n for page_num in range(n_pages):\n img = render_page_image(document_path, page_num, zoom)\n annotated = annotate_page(img, chunks, page_num)\n out_path = doc_dir / f\"page_{page_num + 1}_annotated.png\"\n annotated.save(out_path)\n```\n\n### Visualize Extracted Fields Only\n\nShow only the chunks that contributed to extracted fields (using\nextraction metadata references):\n\n```python\ndef visualize_extraction(\n parse_result: Any,\n extract_result: Any,\n document_path: Path,\n output_dir: Path,\n zoom: float = 2.0,\n) -> None:\n \"\"\"Draw boxes only for chunks referenced by extracted\n fields.\"\"\"\n doc_dir = output_dir / document_path.stem\n doc_dir.mkdir(parents=True, exist_ok=True)\n\n # Collect referenced chunk IDs\n meta = getattr(extract_result, \"extraction_metadata\", {})\n if isinstance(meta, dict):\n ref_ids = set()\n for field_meta in meta.values():\n refs = (\n field_meta.get(\"references\", [])\n if isinstance(field_meta, dict)\n else getattr(field_meta, \"references\", [])\n )\n ref_ids.update(refs)\n else:\n ref_ids = set()\n\n # Filter chunks to only referenced ones\n ref_chunks = [\n ch for ch in (parse_result.chunks or [])\n if getattr(ch, \"id\", None) in ref_ids\n ]\n\n if document_path.suffix.lower() == \".pdf\":\n pdf = pymupdf.open(document_path)\n n_pages = len(pdf)\n pdf.close()\n else:\n n_pages = 1\n\n for page_num in range(n_pages):\n img = render_page_image(document_path, page_num, zoom)\n annotated = annotate_page(img, ref_chunks, page_num)\n out = doc_dir / f\"page_{page_num + 1}_annotated.png\"\n annotated.save(out)\n```\n\n---\n\n## 3. Word-Level Grounding\n\nTwo approaches depending on whether the PDF contains native text or is scanned.\n\n| Scenario | Approach |\n|----------|----------|\n| Native text PDF (most PDFs) | **3a** — PyMuPDF native extraction (exact, fast, no extra deps) |\n| Scanned / image-only PDF | **3b** — Tesseract OCR + fuzzy match |\n\n---\n\n## 3a. Native PDF Word Search (preferred)\n\nFor text-based PDFs, PyMuPDF's `get_text(\"words\", clip=rect)` finds words\nexactly with no OCR required. The key pattern is **spatially restricting the\nsearch to specific ADE chunk bounding boxes**, so occurrences in adjacent\nsections on the same page (e.g. an abstract above the introduction) are\nautomatically excluded.\n\n```python\nfrom pathlib import Path\nfrom typing import Any, List, Tuple\n\nfrom PIL import Image, ImageDraw\n\ntry:\n import pymupdf\nexcept ImportError:\n pymupdf = None # type: ignore[assignment]\n\n\ndef find_term_in_chunks(\n pdf_path: Path,\n page_num: int,\n chunks: list, # ADE chunk objects with grounding\n term: str,\n zoom: float = 2.0,\n) -> List[dict]:\n \"\"\"Find *term* only within the ADE chunk bounding boxes on *page_num*.\n\n Uses PyMuPDF native text extraction clipped to each chunk rect so that\n occurrences outside the supplied chunks are ignored.\n\n Returns list of dicts: text, left, top, width, height (pixel coords\n at *zoom* scale, matching the image from render_page_image()).\n \"\"\"\n if pymupdf is None:\n raise ImportError(\"pip install pymupdf\")\n pdf = pymupdf.open(pdf_path)\n page = pdf[page_num]\n pw, ph = page.rect.width, page.rect.height\n\n boxes = []\n for ch in chunks:\n b = ch.grounding.box\n clip = pymupdf.Rect(b.left * pw, b.top * ph, b.right * pw, b.bottom * ph)\n # Each word entry: (x0, y0, x1, y1, \"word\", block_no, line_no, word_no)\n for x0, y0, x1, y1, text, *_ in page.get_text(\"words\", clip=clip):\n if text.strip(\".,;:!?()[]{}\\\"'–—\") == term:\n boxes.append({\n \"text\": text,\n \"left\": int(x0 * zoom),\n \"top\": int(y0 * zoom),\n \"width\": int((x1 - x0) * zoom),\n \"height\": int((y1 - y0) * zoom),\n })\n pdf.close()\n return boxes\n```\n\n### Annotation and Redaction\n\nThe same `annotate_page` function handles both use cases — the only\ndifference is the alpha value of the fill colour:\n\n```python\n# Highlight: semi-transparent colour (text remains readable)\nHIGHLIGHT = (255, 255, 0, 120) # yellow\n\n# Redact: opaque box (text is visually hidden)\nREDACT = (0, 0, 0, 255) # black, fully opaque\n\n\ndef annotate_page(\n page_img: Image.Image,\n boxes: List[dict],\n fill: Tuple[int, int, int, int] = HIGHLIGHT,\n) -> Image.Image:\n \"\"\"Overlay filled rectangles on a page image.\n\n Pass HIGHLIGHT for annotation or REDACT to cover sensitive content.\n Output is a PNG image — not a PDF-native redaction.\n \"\"\"\n rgba = page_img.convert(\"RGBA\")\n overlay = Image.new(\"RGBA\", rgba.size, (0, 0, 0, 0))\n draw = ImageDraw.Draw(overlay)\n for b in boxes:\n draw.rectangle(\n [b[\"left\"], b[\"top\"],\n b[\"left\"] + b[\"width\"], b[\"top\"] + b[\"height\"]],\n fill=fill,\n )\n return Image.alpha_composite(rgba, overlay)\n```\n\n> **PDF-native redaction** (permanently removes underlying text, not just\n> visually covers it) uses `page.add_redact_annot()` /\n> `page.apply_redactions()` from PyMuPDF. That does not depend on ADE and\n> belongs in the `pdf` skill.\n\n### Usage\n\n```python\nfrom landingai_ade import LandingAIADE\n\nclient = LandingAIADE()\npr = client.parse(document=Path(\"paper.pdf\"))\n\n# Select only the chunks you want to search within\ntarget_chunks = [ch for ch in pr.chunks if ch.grounding.page == 1\n and \"introduction\" in (ch.markdown or \"\").lower()]\n\nboxes = find_term_in_chunks(Path(\"paper.pdf\"), page_num=1,\n chunks=target_chunks, term=\"L2S\")\nimg = render_page_image(Path(\"paper.pdf\"), page_num=1)\nhighlighted = annotate_page(img, boxes, fill=HIGHLIGHT)\nhighlighted.convert(\"RGB\").save(\"page_2_highlighted.png\")\n```\n\n---\n\n## 3b. OCR Word-Level Grounding (scanned PDFs)\n\nFor precise highlighting of extracted values within chunks. Uses\nTesseract OCR on chunk crops + fuzzy matching to locate exact words.\n\n> **Requires:** `pytesseract`, `tesseract` system binary, `fuzzywuzzy`\n\n```python\nfrom typing import List, Tuple\n\nfrom PIL import Image, ImageDraw\n\ntry:\n import pytesseract\n from fuzzywuzzy import fuzz\n WORD_GROUNDING_AVAILABLE = True\nexcept ImportError:\n WORD_GROUNDING_AVAILABLE = False\n\n\ndef find_words_in_chunk(\n chunk_image: Image.Image,\n search_text: str,\n confidence_threshold: int = 60,\n fuzzy_threshold: int = 80,\n) -> List[dict]:\n \"\"\"Find word-level bounding boxes matching search_text.\n\n Returns list of dicts with keys: text, left, top, width,\n height, conf, match_score.\n \"\"\"\n if not WORD_GROUNDING_AVAILABLE:\n raise ImportError(\n \"pip install pytesseract fuzzywuzzy python-Levenshtein\"\n )\n\n ocr_data = pytesseract.image_to_data(\n chunk_image, output_type=pytesseract.Output.DICT\n )\n search_words = search_text.lower().split()\n matches: List[dict] = []\n\n for i, word in enumerate(ocr_data[\"text\"]):\n conf = int(ocr_data[\"conf\"][i])\n if conf < confidence_threshold or not word.strip():\n continue\n for sw in search_words:\n score = fuzz.ratio(word.lower(), sw)\n if score >= fuzzy_threshold:\n matches.append({\n \"text\": word,\n \"left\": ocr_data[\"left\"][i],\n \"top\": ocr_data[\"top\"][i],\n \"width\": ocr_data[\"width\"][i],\n \"height\": ocr_data[\"height\"][i],\n \"conf\": conf,\n \"match_score\": score,\n })\n return matches\n\n\ndef highlight_words(\n chunk_image: Image.Image,\n matches: List[dict],\n color: Tuple[int, int, int, int] = (255, 255, 0, 100),\n) -> Image.Image:\n \"\"\"Draw semi-transparent highlights over matched words.\"\"\"\n highlighted = chunk_image.convert(\"RGBA\")\n overlay = Image.new(\"RGBA\", highlighted.size, (0, 0, 0, 0))\n draw = ImageDraw.Draw(overlay)\n\n for m in matches:\n draw.rectangle(\n [\n m[\"left\"],\n m[\"top\"],\n m[\"left\"] + m[\"width\"],\n m[\"top\"] + m[\"height\"],\n ],\n fill=color,\n )\n return Image.alpha_composite(highlighted, overlay)\n```\n\n### Full Word-Level Grounding Pipeline\n\n```python\ndef word_level_grounding(\n parse_result: Any,\n extract_result: Any,\n document_path: Path,\n output_dir: Path,\n zoom: float = 2.0,\n) -> None:\n \"\"\"For each extracted field, find and highlight the exact\n words in the source chunk.\n\n Saves highlighted chunk crops to output_dir.\n \"\"\"\n output_dir.mkdir(parents=True, exist_ok=True)\n meta = getattr(extract_result, \"extraction_metadata\", {})\n extraction = extract_result.extraction\n\n # Build chunk lookup\n chunk_map = {\n ch.id: ch for ch in (parse_result.chunks or [])\n if hasattr(ch, \"id\")\n }\n\n for field_name, field_meta in (\n meta.items() if isinstance(meta, dict) else []\n ):\n refs = (\n field_meta.get(\"references\", [])\n if isinstance(field_meta, dict)\n else getattr(field_meta, \"references\", [])\n )\n if not refs:\n continue\n\n # Get the extracted value\n value = _dig_value(extraction, field_name)\n if not value or not isinstance(value, str):\n continue\n\n chunk_id = refs[0]\n chunk = chunk_map.get(chunk_id)\n if not chunk or not hasattr(chunk, \"grounding\"):\n continue\n\n # Crop the chunk from the page\n page_num = chunk.grounding.page\n page_img = render_page_image(\n document_path, page_num, zoom\n )\n box = chunk.grounding.box\n w, h = page_img.size\n crop = page_img.crop((\n int(box.left * w),\n int(box.top * h),\n int(box.right * w),\n int(box.bottom * h),\n ))\n\n # Find and highlight words\n matches = find_words_in_chunk(crop, str(value))\n if matches:\n highlighted = highlight_words(crop, matches)\n out = output_dir / f\"{field_name}_{chunk_id}.png\"\n highlighted.save(out)\n\n\ndef _dig_value(d: dict, dotted_key: str) -> Any:\n \"\"\"Get value from nested dict using __ separator.\"\"\"\n parts = dotted_key.split(\"__\")\n obj: Any = d\n for p in parts:\n if isinstance(obj, dict):\n obj = obj.get(p)\n else:\n return None\n return obj\n```\n\n---\n\n## Dependencies\n\n```\n# Chunk images + bounding box overlays + native word search (Sections 1–3a)\npip install landingai-ade Pillow pymupdf\n\n# OCR word-level grounding / scanned PDFs (Section 3b)\npip install landingai-ade Pillow pymupdf pytesseract fuzzywuzzy python-Levenshtein\n# Also requires: brew install tesseract (macOS) or apt install tesseract-ocr (Linux)\n```\n","content/langchain/docs/agents/python/DOC.md":"","content/mypy/docs/extensions/python/DOC.md":"---\nname: extensions\ndescription: \"mypy-extensions package guide for legacy mypy and mypyc typing helpers in Python\"\nmetadata:\n languages: \"python\"\n versions: \"1.1.0\"\n revision: 1\n updated-on: \"2026-03-12\"\n source: maintainer\n tags: \"python,typing,mypy,mypyc,types\"\n---\n\n# mypy-extensions Python Package Guide\n\n## Golden Rule\n\nUse `mypy-extensions` only for legacy code or for helpers that are specific to `mypy` and `mypyc`. For new application code, prefer the standard `typing` module or `typing_extensions` when the package itself marks a symbol as deprecated.\n\n## Install\n\n`mypy-extensions` 1.1.0 requires Python 3.8 or later.\n\n```bash\npython -m pip install mypy-extensions==1.1.0\n```\n\nIf you manage dependencies in `pyproject.toml`:\n\n```toml\n[project]\ndependencies = [\n \"mypy-extensions==1.1.0\",\n]\n```\n\nThe distribution name on PyPI is `mypy-extensions`, but the import name in code is `mypy_extensions`.\n\n## Initialization And Environment\n\nThere are no environment variables, API keys, or client objects to configure. Import the helpers you need directly from `mypy_extensions`.\n\n```python\nfrom mypy_extensions import (\n Arg,\n DefaultArg,\n DefaultNamedArg,\n KwArg,\n NamedArg,\n VarArg,\n i64,\n mypyc_attr,\n trait,\n)\n```\n\n## Recommended Modern Replacements\n\nTwo legacy names exposed by `mypy_extensions` are explicitly deprecated in 1.1.0:\n\n- `mypy_extensions.TypedDict` → use `typing.TypedDict` or `typing_extensions.TypedDict`\n- `mypy_extensions.NoReturn` → use `typing.NoReturn` or `typing_extensions.NoReturn`\n\nUse the modern forms in new code:\n\n```python\nfrom typing import NoReturn\nfrom typing_extensions import TypedDict\n\n\nclass UserPayload(TypedDict):\n id: int\n email: str\n\n\ndef fail(message: str) -> NoReturn:\n raise RuntimeError(message)\n```\n\n## Common Workflows\n\n### Describe callable argument kinds for `mypy`\n\n`Arg`, `DefaultArg`, `NamedArg`, `DefaultNamedArg`, `VarArg`, and `KwArg` let `mypy` describe argument kinds inside a `Callable[...]` type. At runtime, these helpers simply return the wrapped type.\n\n```python\nfrom collections.abc import Callable\n\nfrom mypy_extensions import Arg, DefaultNamedArg, KwArg, VarArg\n\nPluginHook = Callable[[\n Arg(str, \"event\"),\n DefaultNamedArg(bool, \"dry_run\"),\n VarArg(bytes),\n KwArg(str),\n], None]\n\n\ndef emit(\n event: str,\n *payload: bytes,\n dry_run: bool = False,\n **labels: str,\n) -> None:\n print(event, payload, dry_run, labels)\n\n\nhook: PluginHook = emit\n```\n\nUse these helpers only in type annotations. They do not enforce argument behavior at runtime.\n\n### Keep a legacy `TypedDict` import working\n\nIf you are maintaining older code that already imports `TypedDict` from `mypy_extensions`, the symbol still exists in 1.1.0, but it is deprecated and emits a `DeprecationWarning` when the typed dict class is created.\n\n```python\nfrom mypy_extensions import TypedDict\n\n\nclass LegacyMovie(TypedDict):\n title: str\n year: int\n\n\nmovie: LegacyMovie = {\"title\": \"Alien\", \"year\": 1979}\n```\n\nPrefer migrating that import to `typing.TypedDict` or `typing_extensions.TypedDict` instead of adding new uses of the legacy form.\n\n### Mark classes for `mypyc`\n\n`trait` and `mypyc_attr(...)` are decorators used by tooling in the `mypy` and `mypyc` ecosystem. At runtime they behave like identity decorators.\n\n```python\nfrom mypy_extensions import mypyc_attr, trait\n\n\n@trait\nclass Visitor:\n def visit_int(self, value: int) -> str:\n raise NotImplementedError\n\n\n@mypyc_attr(allow_interpreted_subclasses=True)\nclass Node:\n def __init__(self, value: int) -> None:\n self.value = value\n```\n\nUse these only if your project already relies on `mypyc` behavior. In ordinary Python execution they do not add runtime validation or behavior.\n\n### Use mypyc fixed-width integer marker types\n\n`i64`, `i32`, `i16`, and `u8` represent native fixed-width integer types for `mypyc`. In normal Python code, they construct plain `int` values and `isinstance(x, i64)` behaves like `isinstance(x, int)`.\n\n```python\nfrom mypy_extensions import i16, i64, u8\n\ncount: i64 = i64(10)\nsmall: i16 = i16(\"12\")\nbyte: u8 = u8(255)\n\nassert isinstance(count, i64)\nassert isinstance(count, int)\nassert isinstance(byte, u8)\n```\n\nThese types are useful only when your code is interpreted by tooling that understands the mypyc-specific integer semantics.\n\n## Common Pitfalls\n\n- Do not install `mypy-extensions` and then import `mypy-extensions`; the Python module name is `mypy_extensions`.\n- Do not use `mypy_extensions.TypedDict` or `mypy_extensions.NoReturn` in new code; both are deprecated in 1.1.0.\n- Do not expect `Arg(...)`, `NamedArg(...)`, `VarArg(...)`, or `KwArg(...)` to change runtime behavior; they are typing helpers, not runtime wrappers.\n- Do not use `isinstance(value, TypedDictSubclass)` or `issubclass(...)` checks with legacy typed dicts; `mypy_extensions.TypedDict` explicitly rejects instance and class checks.\n- Do not expect `i64`, `i32`, `i16`, or `u8` to behave like fixed-width integers in normal Python execution; outside mypyc-specific contexts they behave like `int`.\n- Do not assume legacy typed dict introspection matches modern `typing.TypedDict`; `mypy_extensions.TypedDict` exposes `__annotations__` and `__total__`, but not the newer required/optional key metadata used by modern typed dict implementations.\n\n## Version-Sensitive Notes For 1.1.0\n\n- `mypy-extensions` 1.1.0 declares `Requires-Python >=3.8`.\n- The 1.1.0 wheel advertises Python 3.8 through 3.13 support.\n- `TypedDict` emits a deprecation warning telling you to use `typing.TypedDict` or `typing_extensions.TypedDict`.\n- `NoReturn` is provided through module attribute fallback and emits a deprecation warning telling you to use `typing.NoReturn` or `typing_extensions.NoReturn`.\n- The package surface is intentionally small; if you need broader modern typing features, reach for `typing` or `typing_extensions` rather than expecting this package to mirror them.\n\n## Official Sources\n\n- Maintainer repository: https://github.com/python/mypy_extensions\n- PyPI project: https://pypi.org/project/mypy-extensions/\n- 1.1.0 wheel published on PyPI: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl\n","content/olakai/skills/integrate/SKILL.md":"---\nname: integrate\ndescription: \"Add Olakai monitoring to existing AI code — wrap your LLM client, configure custom KPIs, and validate the integration end-to-end\"\nmetadata:\n revision: 1\n updated-on: \"2026-03-10\"\n source: maintainer\n tags: \"olakai,integration,monitoring,sdk,kpi,governance\"\n---\n\n# Integrate Olakai into Existing AI Code\n\nThis skill guides you through adding Olakai monitoring to an existing AI agent or LLM-powered application with minimal code changes.\n\nFor full SDK documentation, see: https://app.olakai.ai/llms.txt\n\n## Prerequisites\n\n- Existing working AI agent/application using OpenAI, Anthropic, or other LLM\n- Olakai CLI installed and authenticated (`npm install -g olakai-cli && olakai login`)\n- Olakai API key for your agent (get via CLI: `olakai agents get AGENT_ID --json | jq '.apiKey'`)\n- Node.js 18+ (for TypeScript) or Python 3.7+ (for Python)\n\n> **Note:** Each agent can have its own API key. Create one with `olakai agents create --name \"Name\" --with-api-key`\n\n## Why Custom KPIs Are Essential\n\nAdding monitoring is only the first step. **The real value of Olakai comes from tracking custom KPIs specific to your agent's business purpose.**\n\n**Without KPIs configured:**\n- Only basic token counts and request data\n- No aggregated business KPIs on dashboard\n- No alerting capabilities\n- No ROI tracking\n\n**With KPIs configured:**\n- Custom KPIs (items processed, success rates, quality scores)\n- Trend analysis and performance dashboards\n- Threshold-based alerting\n- Business value calculations\n\n> **Plan to configure at least 2-4 KPIs** that answer: \"How do I know this agent is performing well?\"\n\n> **KPIs are unique per agent.** If adding monitoring to an agent that needs the same KPIs as another already-configured agent, you must still create new KPI definitions for this agent. KPIs cannot be shared or reused across agents.\n\n## Understanding the customData to KPI Pipeline\n\nBefore adding monitoring, understand how custom data flows through Olakai:\n\n```\nSDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData\n```\n\n### Critical Rules\n\n| Rule | Consequence |\n|------|-------------|\n| Only CustomDataConfig fields become variables | Unregistered customData fields are NOT usable in KPIs |\n| Formula evaluation is case-insensitive | `stepCount`, `STEPCOUNT`, `StepCount` all work in formulas |\n| NUMBER configs need numeric values | Don't send `\"5\"` (string), send `5` (number) |\n\n> **IMPORTANT**: The SDK accepts any JSON in `customData`, but **only fields registered as CustomDataConfigs are processed**. Unregistered fields are stored but cannot be used in KPIs.\n\n## Quick Start (5-Minute Integration)\n\n### For TypeScript/JavaScript\n\n**1. Install the SDK:**\n```bash\nnpm install @olakai/sdk\n```\n\n**2. Add tracking after your LLM call:**\n\nBefore:\n```typescript\nimport OpenAI from \"openai\";\nconst openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });\n\nconst response = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: userMessage }],\n});\n```\n\nAfter:\n```typescript\nimport OpenAI from \"openai\";\nimport { olakaiConfig, olakai } from \"@olakai/sdk\";\n\nolakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });\n\nconst openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });\n\nconst response = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: userMessage }],\n});\n\n// Track the interaction (fire-and-forget)\nolakai(\"event\", \"ai_activity\", {\n prompt: userMessage,\n response: response.choices[0].message.content,\n tokens: response.usage?.total_tokens,\n userEmail: user.email,\n task: \"Customer Experience\",\n});\n```\n\n### For Python\n\n**1. Install the SDK:**\n```bash\npip install olakai-sdk\n```\n\n**2. Add tracking after your LLM call:**\n\nBefore:\n```python\nfrom openai import OpenAI\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\nresponse = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\"role\": \"user\", \"content\": user_message}],\n)\n```\n\nAfter:\n```python\nfrom openai import OpenAI\nfrom olakaisdk import olakai_config, olakai, OlakaiEventParams\n\nolakai_config(os.getenv(\"OLAKAI_API_KEY\"))\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\nresponse = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\"role\": \"user\", \"content\": user_message}],\n)\n\n# Track the interaction\nolakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=user_message,\n response=response.choices[0].message.content,\n tokens=response.usage.total_tokens,\n userEmail=user.email,\n task=\"Customer Experience\",\n))\n```\n\n---\n\n## Detailed Integration Guide\n\n### Step 1: Identify Your Integration Pattern\n\n**Pattern A: Single LLM Client**\nYou have one OpenAI/Anthropic client used throughout your app.\nUse the fire-and-forget `olakai()` call after each completion.\n\n**Pattern B: Multiple LLM Calls per Request**\nYour agent makes several LLM calls to complete one task.\nUse manual event tracking to aggregate calls into a single event.\n\n**Pattern C: Streaming Responses**\nYou stream LLM responses to users.\nTrack after the stream completes with the full accumulated response.\n\n**Pattern D: Third-Party LLM (not OpenAI/Anthropic)**\nYou use Perplexity, Groq, local models, etc.\nUse manual event tracking via `olakai()` or `olakai_event()`.\n\n### Step 2: Install and Configure\n\n#### TypeScript Setup\n\n```typescript\n// lib/olakai.ts - Initialize once at app startup\nimport { olakaiConfig } from \"@olakai/sdk\";\n\nolakaiConfig({\n apiKey: process.env.OLAKAI_API_KEY!,\n debug: process.env.NODE_ENV === \"development\",\n});\n```\n\n#### Python Setup\n\n```python\n# lib/olakai.py - Initialize once at app startup\nimport os\nfrom olakaisdk import olakai_config\n\nolakai_config(\n api_key=os.getenv(\"OLAKAI_API_KEY\"),\n debug=os.getenv(\"DEBUG\") == \"true\"\n)\n```\n\n### Step 3: Add Context to Calls\n\n#### Adding User Information\n\nTypeScript:\n```typescript\nolakai(\"event\", \"ai_activity\", {\n prompt: userMessage,\n response: aiResponse,\n userEmail: user.email,\n task: \"Customer Experience\",\n});\n```\n\nPython:\n```python\nolakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=user_message,\n response=ai_response,\n userEmail=user.email,\n task=\"Customer Experience\",\n))\n```\n\n#### Grouping Events by Conversation (chatId)\n\nFor assistive AI (chatbots/copilots), use `chatId` to group multiple turns of a conversation together. This is required for CHAT-scoped KPIs that analyze the full conversation.\n\n```typescript\nolakai(\"event\", \"ai_activity\", {\n prompt: userMessage,\n response: aiResponse,\n chatId: conversationId, // groups turns in the same conversation\n userEmail: user.email,\n});\n```\n\n> **When to use `chatId`:** If your agent handles multi-turn conversations and you want KPIs that evaluate the entire conversation (e.g., sentiment scoring, satisfaction), pass a consistent `chatId` across all turns.\n\n#### Adding Custom Data\n\n> **IMPORTANT**: Only send fields you've registered as CustomDataConfigs (Step 5.3). Unregistered fields are stored but **cannot be used in KPIs**.\n\n> **Only send data you'll use in KPIs or for filtering.** Don't duplicate fields already tracked by the platform (session ID, agent ID, user email, timestamps, token count, model, provider — all tracked automatically).\n\nTypeScript:\n```typescript\nolakai(\"event\", \"ai_activity\", {\n prompt: userMessage,\n response: aiResponse,\n userEmail: user.email,\n customData: {\n // Only include fields registered as CustomDataConfigs\n Department: user.department,\n ProjectId: currentProject.id,\n Priority: ticket.priority,\n },\n});\n```\n\n### Step 4: Handle Agentic Workflows\n\nIf your agent makes multiple LLM calls per task, aggregate them into a single event.\n\n> **`taskExecutionId` — Critical for multi-agent workflows.** If multiple agents collaborate on the same task, the orchestrator must generate ONE `taskExecutionId` and pass it to all agents. This is how Olakai correlates cross-agent work as a single logical task.\n\n```typescript\nasync function processDocument(doc: Document): Promise {\n const startTime = Date.now();\n let totalTokens = 0;\n\n // Step 1: Extract\n const extraction = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: `Extract from: ${doc.content}` }],\n });\n totalTokens += extraction.usage?.total_tokens ?? 0;\n\n // Step 2: Analyze\n const analysis = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: `Analyze: ${extraction.choices[0].message.content}` }],\n });\n totalTokens += analysis.usage?.total_tokens ?? 0;\n\n const result = analysis.choices[0].message.content ?? \"\";\n\n // Track the complete workflow as ONE event\n olakai(\"event\", \"ai_activity\", {\n prompt: `Process document: ${doc.title}`,\n response: result,\n tokens: totalTokens,\n requestTime: Date.now() - startTime,\n taskExecutionId: crypto.randomUUID(),\n task: \"Data Processing & Analysis\",\n customData: {\n DocumentType: doc.type,\n StepCount: 2,\n Success: 1,\n },\n });\n\n return result;\n}\n```\n\n### Step 5: Configure Custom KPIs (Essential for Value)\n\n> **This step is required to get real value from Olakai.** Without KPIs, you're only tracking events — not gaining actionable insights.\n\n#### 5.1 Install CLI (if not already)\n```bash\nnpm install -g olakai-cli\nolakai login\n```\n\n#### 5.2 Register Your Agent\n```bash\nolakai agents create \\\n --name \"Document Processor\" \\\n --description \"Processes and summarizes documents\" \\\n --workflow WORKFLOW_ID \\\n --with-api-key\n```\n\n#### 5.2.1 Ensure Agent Has a Workflow\n\n> **Every agent MUST belong to a workflow**, even if it's the only agent.\n\n```bash\n# Check if agent has a workflow\nolakai agents get YOUR_AGENT_ID --json | jq '.workflowId'\n\n# If null, create a workflow and associate:\nolakai workflows create --name \"Your Workflow Name\" --json\nolakai agents update YOUR_AGENT_ID --workflow WORKFLOW_ID\n```\n\n#### 5.3 Create Custom Data Configs FIRST\n\n> **IMPORTANT**: Create configs for ALL fields you send in `customData`. Only registered fields can be used in KPIs. CustomDataConfigs are agent-scoped.\n\n```bash\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"DocumentType\" --type STRING\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"StepCount\" --type NUMBER\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"Success\" --type NUMBER\n\n# Verify all configs exist for this agent\nolakai custom-data list --agent-id YOUR_AGENT_ID\n```\n\n#### 5.4 Create KPIs\n\n```bash\nolakai kpis create \\\n --name \"Documents Processed\" \\\n --agent-id YOUR_AGENT_ID \\\n --calculator-id formula \\\n --formula \"IF(Success = 1, 1, 0)\" \\\n --aggregation SUM\n\nolakai kpis create \\\n --name \"Avg Steps per Document\" \\\n --agent-id YOUR_AGENT_ID \\\n --calculator-id formula \\\n --formula \"StepCount\" \\\n --aggregation AVERAGE\n```\n\n#### 5.5 Update SDK Code to Match\n\nAfter creating configs, ensure your SDK code sends **exactly those field names**:\n\n```typescript\ncustomData: {\n DocumentType: doc.type, // Matches CustomDataConfig \"DocumentType\"\n StepCount: 2, // Matches CustomDataConfig \"StepCount\"\n Success: true ? 1 : 0, // Matches CustomDataConfig \"Success\"\n}\n```\n\n## Framework-Specific Integrations\n\n### Next.js API Routes\n\n```typescript\n// app/api/chat/route.ts\nimport { NextRequest, NextResponse } from \"next/server\";\nimport { olakai } from \"@olakai/sdk\";\nimport { auth } from \"@/auth\";\n\nexport async function POST(req: NextRequest) {\n const session = await auth();\n if (!session?.user) {\n return NextResponse.json({ error: \"Unauthorized\" }, { status: 401 });\n }\n\n const { message } = await req.json();\n\n const response = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: message }],\n });\n\n olakai(\"event\", \"ai_activity\", {\n prompt: message,\n response: response.choices[0].message.content,\n userEmail: session.user.email!,\n task: \"Customer Experience\",\n });\n\n return NextResponse.json({ reply: response.choices[0].message.content });\n}\n```\n\n### FastAPI (Python)\n\n```python\nfrom fastapi import FastAPI, Depends\nfrom olakaisdk import olakai_config, olakai, OlakaiEventParams\n\napp = FastAPI()\n\n@app.on_event(\"startup\")\nasync def startup():\n olakai_config(os.getenv(\"OLAKAI_API_KEY\"))\n\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\n@app.post(\"/chat\")\nasync def chat(message: str, user: User = Depends(get_current_user)):\n response = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\"role\": \"user\", \"content\": message}]\n )\n\n olakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=message,\n response=response.choices[0].message.content,\n userEmail=user.email,\n task=\"Customer Experience\",\n ))\n\n return {\"reply\": response.choices[0].message.content}\n```\n\n## Handling Edge Cases\n\n### Streaming Responses\n\nTrack after the stream completes with the full response:\n\n```typescript\nconst stream = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: userMessage }],\n stream: true,\n});\n\nlet fullResponse = \"\";\nfor await (const chunk of stream) {\n fullResponse += chunk.choices[0]?.delta?.content ?? \"\";\n res.write(chunk.choices[0]?.delta?.content ?? \"\");\n}\n\n// Track after stream completes\nolakai(\"event\", \"ai_activity\", {\n prompt: userMessage,\n response: fullResponse,\n userEmail: user.email,\n});\n```\n\n### Error Handling\n\n```typescript\ntry {\n const response = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages,\n });\n return response.choices[0].message.content;\n} catch (error) {\n // Track the failed attempt\n olakai(\"event\", \"ai_activity\", {\n prompt: messages[messages.length - 1].content,\n response: `Error: ${error instanceof Error ? error.message : \"Unknown\"}`,\n task: \"Software Development\",\n customData: { Success: 0 },\n });\n throw error;\n}\n```\n\n### Non-OpenAI Providers\n\nFor Anthropic, Perplexity, or other providers, use manual tracking:\n\n```typescript\nimport Anthropic from \"@anthropic-ai/sdk\";\n\nconst anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });\n\nasync function callClaude(prompt: string): Promise {\n const startTime = Date.now();\n\n const response = await anthropic.messages.create({\n model: \"claude-sonnet-4-20250514\",\n max_tokens: 1024,\n messages: [{ role: \"user\", content: prompt }],\n });\n\n const content = response.content[0].type === \"text\" ? response.content[0].text : \"\";\n\n olakai(\"event\", \"ai_activity\", {\n prompt,\n response: content,\n tokens: response.usage.input_tokens + response.usage.output_tokens,\n requestTime: Date.now() - startTime,\n task: \"Content Development\",\n });\n\n return content;\n}\n```\n\n## Test-Validate-Iterate Cycle\n\n**Never assume your integration is working. Always validate by generating a test event and inspecting the actual data.**\n\n### Step 1: Generate a Test Event\n\nRun your application to trigger at least one LLM call.\n\n### Step 2: Fetch and Inspect the Event\n\n```bash\nolakai activity list --limit 1 --json\nolakai activity get EVENT_ID --json\n```\n\n### Step 3: Validate Each Component\n\n**Check customData is present:**\n```bash\nolakai activity get EVENT_ID --json | jq '.customData'\n```\n\n**Check KPIs are numeric (not strings or null):**\n```bash\nolakai activity get EVENT_ID --json | jq '.kpiData'\n```\n\n**CORRECT:**\n```json\n{ \"My KPI\": 42 }\n```\n\n**WRONG (formula stored as string):**\n```json\n{ \"My KPI\": \"MyVariable\" }\n```\nFix: `olakai kpis update KPI_ID --formula \"MyVariable\"`\n\n**WRONG (null value):**\n```json\n{ \"My KPI\": null }\n```\nFix by ensuring:\n1. CustomDataConfig exists: `olakai custom-data create --agent-id ID --name \"MyVariable\" --type NUMBER`\n2. Field name case matches exactly (case-sensitive)\n3. SDK actually sends the field in customData\n\n### Validation Flow\n\n```\n1. Trigger LLM call (generate event)\n ↓\n2. Fetch: olakai activity get ID --json\n ↓\n3. Event exists? NO → Check API key, SDK init, debug mode\n ↓\n4. customData correct? NO → Fix SDK customData parameter\n ↓\n5. kpiData numeric? NO → olakai kpis update ID --formula \"X\"\n ↓\n6. kpiData not null? NO → Create CustomDataConfig, check field name case\n ↓\n✅ Integration validated\n```\n\n## KPI Formula Reference\n\n### Supported Operators\n\n| Category | Operators |\n|----------|-----------|\n| Arithmetic | `+`, `-`, `*`, `/` |\n| Comparison | `<`, `<=`, `=`, `<>`, `>=`, `>` |\n| Logical | `AND`, `OR`, `NOT` |\n| Conditional | `IF(condition, true_val, false_val)` |\n| Null handling | `ISNA(value)`, `ISDEFINED(value)` |\n\n### Common Formula Patterns\n\n```bash\n--formula \"StepCount\" # passthrough\n--formula \"SuccessRate * 100\" # percentage conversion\n--formula \"IF(Success = 1, 1, 0)\" # conditional counting\n--formula \"IF(PII detected, 1, 0)\" # built-in variable\n--formula \"IF(ISDEFINED(MyField), MyField, 0)\" # null-safe\n```\n\n### Aggregation Types\n\n| Aggregation | Use For |\n|-------------|---------|\n| `SUM` | Totals, counts |\n| `AVERAGE` | Rates, percentages |\n\n## Quick Reference\n\n```typescript\n// TypeScript — initialize once\nimport { olakaiConfig, olakai } from \"@olakai/sdk\";\nolakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });\n\n// Track any interaction\nolakai(\"event\", \"ai_activity\", {\n prompt: \"input\",\n response: \"output\",\n tokens: 1500,\n requestTime: 5000,\n userEmail: \"user@example.com\",\n chatId: \"conversation-id\",\n taskExecutionId: \"uuid-shared-across-agents\",\n task: \"Data Processing & Analysis\",\n customData: { StepCount: 3, Success: 1 },\n});\n```\n\n```python\n# Python — initialize once\nfrom olakaisdk import olakai_config, olakai, OlakaiEventParams\nolakai_config(os.getenv(\"OLAKAI_API_KEY\"))\n\n# Track any interaction\nolakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=\"input\",\n response=\"output\",\n tokens=1500,\n requestTime=5000,\n userEmail=\"user@example.com\",\n chatId=\"conversation-id\",\n taskExecutionId=\"uuid-shared-across-agents\",\n task=\"Data Processing & Analysis\",\n customData={\"StepCount\": 3, \"Success\": 1},\n))\n```\n","content/olakai/skills/new-project/SKILL.md":"---\nname: new-project\ndescription: \"Build a new AI agent with Olakai monitoring from scratch — project setup, SDK integration, KPI configuration, and end-to-end validation\"\nmetadata:\n revision: 1\n updated-on: \"2026-03-10\"\n source: maintainer\n tags: \"olakai,new-project,agent,monitoring,kpi,governance\"\n---\n\n# Build a New AI Agent Project with Olakai\n\nThis skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.\n\n## Prerequisites\n\nBefore starting, ensure:\n1. Olakai CLI installed: `npm install -g olakai-cli`\n2. CLI authenticated: `olakai login`\n3. API key for SDK (generated per-agent via CLI — see Step 2.2)\n\n## Why Custom KPIs Are Essential\n\nOlakai's core value is **tracking business-specific KPIs for your AI agents**. Without KPIs, you're tracking events without gaining actionable insights.\n\n**What you can measure with KPIs:**\n- Business outcomes (items processed, success rates, revenue impact)\n- Operational data (step counts, retry rates, execution time)\n- Quality indicators (error rates, user satisfaction signals)\n\n**Without KPIs configured:**\n- No dashboard KPIs beyond basic token counts\n- No aggregated performance views\n- No alerting thresholds\n- No ROI calculations\n\n> **Every agent should have 2-4 KPIs that answer: \"How do I know this agent is performing well?\"**\n\n> **KPIs created here belong to this specific agent only.** If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.\n\n## Understanding the customData to KPI Pipeline\n\nBefore diving into implementation, understand how data flows through Olakai:\n\n```\nSDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData\n```\n\n### How It Works\n\n1. **customData** (SDK): Raw JSON you send with each event\n2. **CustomDataConfig** (Platform): Schema defining which fields are processed\n3. **Context Variables**: CustomDataConfig fields become available for formulas\n4. **KPI Formula**: Expression that computes a value (e.g., `SuccessRate * 100`)\n5. **kpiData** (Response): Computed KPI values returned with each event\n\n### Critical Rules\n\n| Rule | Consequence |\n|------|-------------|\n| Only CustomDataConfig fields become variables | Unregistered customData fields are NOT usable in KPIs |\n| Formula evaluation is case-insensitive | `stepCount`, `STEPCOUNT`, `StepCount` all work in formulas |\n| NUMBER configs need numeric values | Don't send `\"5\"` (string), send `5` (number) |\n| KPIs are unique per agent | Each KPI belongs to exactly one agent — create separately for each |\n\n### Built-in Context Variables (Always Available)\n\n| Variable | Type | Description |\n|----------|------|-------------|\n| `Prompt` | string | The prompt text sent to the LLM |\n| `Response` | string | The LLM response text |\n| `Documents count` | number | Number of attached documents |\n| `PII detected` | boolean | Whether PII was detected |\n| `PHI detected` | boolean | Whether PHI was detected |\n| `CODE detected` | boolean | Whether code was detected |\n| `SECRET detected` | boolean | Whether secrets were detected |\n\n## Step 1: Design the Agent Architecture\n\n### 1.1 Determine Agent Type\n\n**Agentic AI** (Multi-step autonomous workflows):\n- Research agents, document processors, data pipelines\n- Track as SINGLE events aggregating all internal LLM calls\n- Focus on workflow-level KPIs (total tokens, total time, success/failure)\n\n**Assistive AI** (Interactive chatbots/copilots):\n- Customer support agents, coding assistants, Q&A systems\n- Track EACH interaction as separate events\n- Focus on conversation-level KPIs (per-message tokens, response quality)\n\n### 1.2 Design Your KPI Schema (CRITICAL)\n\n**Design your KPIs BEFORE writing any SDK code.** This ensures only meaningful data is sent and tracked.\n\n#### Step A: Identify Business Questions\n\nWhat do stakeholders need to know about this agent?\n- \"How many items does it process per run?\"\n- \"What's the success/failure rate?\"\n- \"How efficient is each execution?\"\n\n#### Step B: Map Questions to Data Fields\n\n| Business Question | Field Name | Type | KPI Formula | Aggregation |\n|-------------------|------------|------|-------------|-------------|\n| Throughput | ItemsProcessed | NUMBER | `ItemsProcessed` | SUM |\n| Reliability | SuccessRate | NUMBER | `SuccessRate * 100` | AVERAGE |\n| Error count | SuccessRate | NUMBER | `IF(SuccessRate < 1, 1, 0)` | SUM |\n| Correlation | ExecutionId | STRING | (for filtering only) | - |\n\n#### Step C: Plan Your customData Structure\n\n```typescript\n// ONLY include fields you'll register as CustomDataConfigs\ncustomData: {\n // Business KPIs\n ItemsProcessed: number, // Count of items handled\n SuccessRate: number, // 0-1 success ratio\n\n // Performance KPIs\n StepCount: number, // Number of workflow steps\n\n // Identification (for filtering, not KPIs)\n ExecutionId: string, // Correlation ID\n}\n```\n\n> **IMPORTANT**: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but **cannot be used in KPIs**.\n\n### What NOT to Include in customData\n\nThe Olakai platform automatically tracks these — do NOT duplicate them:\n\n| Already Tracked | Where | Don't Send As customData |\n|-----------------|-------|--------------------------|\n| Session ID | Main payload | `sessionId` |\n| Agent ID | API key association | `agentId` |\n| User email | `userEmail` parameter | `email`, `userEmail` |\n| Timestamp | Event metadata | `timestamp`, `createdAt` |\n| Request time | `requestTime` parameter | `duration`, `latency` |\n| Token count | `tokens` parameter | `tokenCount` |\n| Model | Auto-detected | `model`, `modelName` |\n| Provider | Client config | `provider` |\n\n**customData is ONLY for:**\n1. **KPI variables** — Fields you'll use in formula calculations\n2. **Tagging/filtering** — Fields you'll filter by in queries\n\n## Step 2: Configure Olakai Platform\n\n### 2.1 Create a Workflow (Required)\n\n> **Every agent MUST belong to a workflow**, even if it's the only agent in that workflow.\n\n```bash\nolakai workflows create --name \"Your Workflow Name\" --json\n# Output: { \"id\": \"wfl_xxx...\", \"name\": \"Your Workflow Name\" }\n```\n\n### 2.2 Create the Agent in Olakai\n\n```bash\nolakai agents create \\\n --name \"Your Agent Name\" \\\n --description \"What this agent does\" \\\n --workflow WORKFLOW_ID \\\n --with-api-key \\\n --json\n\n# Returns agent details including apiKey:\n# {\n# \"id\": \"cmkbteqn501kyjy4yu6p6xrrx\",\n# \"name\": \"Your Agent Name\",\n# \"workflowId\": \"wfl_xxx...\",\n# \"apiKey\": \"sk_agent_xxxxx...\" <-- Use this in your SDK\n# }\n```\n\n**Agent-Workflow Hierarchy:**\n```\nWorkflow: \"Customer Support Pipeline\"\n├── Agent: \"Ticket Classifier\"\n├── Agent: \"Response Generator\"\n└── Agent: \"Quality Checker\"\n\nWorkflow: \"Document Processing\"\n└── Agent: \"Document Summarizer\" ← single-agent workflows are valid\n```\n\n### 2.3 Create Custom Data Configurations (BEFORE Writing SDK Code)\n\n> **This step MUST be completed before Step 3 (SDK Integration).** Only fields registered here can be used in KPI formulas.\n\n> **ONLY create configs for data you'll use in KPIs or for filtering.** Don't create configs for data already tracked automatically.\n\n```bash\n# For numeric fields (can be used in KPI calculations)\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"ItemsProcessed\" --type NUMBER\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"SuccessRate\" --type NUMBER\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"StepCount\" --type NUMBER\n\n# For string fields (for filtering/grouping, not calculations)\nolakai custom-data create --agent-id YOUR_AGENT_ID --name \"ExecutionId\" --type STRING\n\n# Verify all configs are created\nolakai custom-data list --agent-id YOUR_AGENT_ID\n```\n\n### 2.4 Create KPI Definitions\n\n#### Quick Start with Templates\n\nInstead of writing formulas from scratch, use predefined classifier templates:\n\n```bash\n# List available templates\nolakai kpis templates\n\n# Create a classifier KPI from a template\nolakai kpis create --name \"User Satisfaction\" \\\n --calculator-id classifier --template-id sentiment_scorer \\\n --scope CHAT --agent-id $AGENT_ID\n\n# Create a time-saved estimator\nolakai kpis create --name \"Time Saved\" \\\n --calculator-id classifier --template-id time_saved_estimator \\\n --scope CHAT --agent-id $AGENT_ID\n```\n\n#### Custom Formula KPIs\n\n```bash\n# Variable passthrough\nolakai kpis create \\\n --name \"Items Processed\" \\\n --agent-id YOUR_AGENT_ID \\\n --calculator-id formula \\\n --formula \"ItemsProcessed\" \\\n --unit \"items\" \\\n --aggregation SUM\n\n# Percentage calculation\nolakai kpis create \\\n --name \"Success Rate\" \\\n --agent-id YOUR_AGENT_ID \\\n --calculator-id formula \\\n --formula \"SuccessRate * 100\" \\\n --unit \"%\" \\\n --aggregation AVERAGE\n\n# Conditional counting\nolakai kpis create \\\n --name \"Error Count\" \\\n --agent-id YOUR_AGENT_ID \\\n --calculator-id formula \\\n --formula \"IF(SuccessRate < 1, 1, 0)\" \\\n --unit \"errors\" \\\n --aggregation SUM\n\n# Validate formulas before creating\nolakai kpis validate --formula \"ItemsProcessed\" --agent-id YOUR_AGENT_ID\n```\n\n## Step 3: Implement SDK Integration\n\n### 3.1 TypeScript Implementation\n\n**Install dependencies:**\n```bash\nnpm install @olakai/sdk openai\n```\n\n**Initialize and track:**\n```typescript\nimport { olakaiConfig, olakai } from \"@olakai/sdk\";\nimport OpenAI from \"openai\";\n\n// Initialize Olakai\nolakaiConfig({\n apiKey: process.env.OLAKAI_API_KEY!,\n debug: process.env.NODE_ENV === \"development\",\n});\n\n// Create LLM client\nconst openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });\n\n// Use wrapped client — monitoring happens automatically\nconst response = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: userPrompt }],\n});\n```\n\n**Agentic workflow with manual event tracking:**\n\n> **`taskExecutionId` — Cross-Agent Task Correlation.** Generate ONE `taskExecutionId` per task and share it across all agents in a multi-agent workflow. This links events from different agents into a single logical task for analytics.\n\n```typescript\nasync function runAgent(input: string): Promise {\n const startTime = Date.now();\n const executionId = crypto.randomUUID();\n const taskExecutionId = crypto.randomUUID();\n let totalTokens = 0;\n let stepCount = 0;\n let itemsProcessed = 0;\n\n try {\n // Step 1: Planning\n stepCount++;\n const plan = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: `Plan: ${input}` }],\n });\n totalTokens += plan.usage?.total_tokens ?? 0;\n\n // Step 2: Process items\n const items = parseItems(plan.choices[0].message.content);\n for (const item of items) {\n stepCount++;\n const result = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: `Process: ${item}` }],\n });\n totalTokens += result.usage?.total_tokens ?? 0;\n itemsProcessed++;\n }\n\n // Step 3: Summarize\n stepCount++;\n const summary = await openai.chat.completions.create({\n model: \"gpt-4o\",\n messages: [{ role: \"user\", content: \"Summarize results\" }],\n });\n totalTokens += summary.usage?.total_tokens ?? 0;\n\n const finalResponse = summary.choices[0].message.content ?? \"\";\n\n // Track the complete workflow as a single event\n // Only send fields that have CustomDataConfigs (from Step 2.3)\n olakai(\"event\", \"ai_activity\", {\n prompt: input,\n response: finalResponse,\n tokens: totalTokens,\n requestTime: Date.now() - startTime,\n taskExecutionId,\n task: \"Data Processing & Analysis\",\n customData: {\n ExecutionId: executionId,\n StepCount: stepCount,\n ItemsProcessed: itemsProcessed,\n SuccessRate: 1.0,\n },\n });\n\n return finalResponse;\n } catch (error) {\n // Track failed execution — same fields, different values\n olakai(\"event\", \"ai_activity\", {\n prompt: input,\n response: `Error: ${error instanceof Error ? error.message : \"Unknown\"}`,\n tokens: totalTokens,\n requestTime: Date.now() - startTime,\n taskExecutionId,\n task: \"Data Processing & Analysis\",\n customData: {\n ExecutionId: executionId,\n StepCount: stepCount,\n ItemsProcessed: itemsProcessed,\n SuccessRate: 0,\n },\n });\n throw error;\n }\n}\n```\n\n### 3.2 Python Implementation\n\n**Install dependencies:**\n```bash\npip install olakai-sdk openai\n```\n\n**Initialize and track:**\n```python\nimport os\nfrom olakaisdk import olakai_config, olakai, OlakaiEventParams\nfrom openai import OpenAI\n\n# Initialize Olakai\nolakai_config(os.getenv(\"OLAKAI_API_KEY\"))\n\n# Create OpenAI client\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n```\n\n**Agentic workflow:**\n```python\nimport time\nimport uuid\n\ndef run_agent(input_text: str) -> str:\n start_time = time.time()\n execution_id = str(uuid.uuid4())\n task_execution_id = str(uuid.uuid4())\n total_tokens = 0\n step_count = 0\n items_processed = 0\n\n try:\n # Your workflow steps here...\n step_count += 1\n response = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\"role\": \"user\", \"content\": input_text}]\n )\n total_tokens += response.usage.total_tokens\n final_response = response.choices[0].message.content\n\n # Track successful execution\n olakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=input_text,\n response=final_response,\n tokens=total_tokens,\n requestTime=int((time.time() - start_time) * 1000),\n taskExecutionId=task_execution_id,\n task=\"Data Processing & Analysis\",\n customData={\n \"ExecutionId\": execution_id,\n \"StepCount\": step_count,\n \"ItemsProcessed\": items_processed,\n \"SuccessRate\": 1.0,\n }\n ))\n\n return final_response\n\n except Exception as e:\n olakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=input_text,\n response=f\"Error: {str(e)}\",\n tokens=total_tokens,\n requestTime=int((time.time() - start_time) * 1000),\n taskExecutionId=task_execution_id,\n task=\"Data Processing & Analysis\",\n customData={\n \"ExecutionId\": execution_id,\n \"StepCount\": step_count,\n \"ItemsProcessed\": items_processed,\n \"SuccessRate\": 0,\n }\n ))\n raise\n```\n\n### 3.3 REST API Direct Integration\n\nFor other languages or custom integrations:\n\n```bash\ncurl -X POST \"https://app.olakai.ai/api/monitoring/prompt\" \\\n -H \"Content-Type: application/json\" \\\n -H \"x-api-key: YOUR_API_KEY\" \\\n -d '{\n \"prompt\": \"User input here\",\n \"response\": \"Agent response here\",\n \"app\": \"your-agent-name\",\n \"task\": \"Data Processing & Analysis\",\n \"tokens\": 1500,\n \"requestTime\": 5000,\n \"customData\": {\n \"ExecutionId\": \"abc-123\",\n \"StepCount\": 5,\n \"ItemsProcessed\": 10,\n \"SuccessRate\": 1.0\n }\n }'\n```\n\n## Step 4: Test-Validate-Iterate Cycle\n\n**Always validate your implementation by running a test and inspecting the actual event data.**\n\n### 4.1 Run Your Agent\n\nExecute your agent with test data to generate at least one event.\n\n### 4.2 Fetch and Inspect the Event\n\n```bash\nolakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json\nolakai activity get EVENT_ID --json\n```\n\n### 4.3 Validate Each Component\n\n**Check customData is present and correct:**\n```bash\nolakai activity get EVENT_ID --json | jq '.customData'\n```\n\n**Check KPIs are numeric (not strings):**\n```bash\nolakai activity get EVENT_ID --json | jq '.kpiData'\n```\n\n**CORRECT** — numeric values:\n```json\n{\n \"Items Processed\": 10,\n \"Success Rate\": 100\n}\n```\n\n**WRONG** — string values (broken formula):\n```json\n{\n \"Items Processed\": \"itemsProcessed\"\n}\n```\nFix: `olakai kpis update KPI_ID --formula \"YourVariable\"`\n\n**WRONG** — null values:\nFix by verifying:\n1. CustomDataConfig exists: `olakai custom-data list --agent-id ID`\n2. Field name case matches exactly (case-sensitive)\n3. SDK actually sends the field in customData\n\n### 4.4 Validation Flow\n\n```\n1. Run agent (generate event)\n ↓\n2. Fetch event: olakai activity get ID --json\n ↓\n3. Check customData present? NO → Fix SDK code\n ↓\n4. Check kpiData numeric? NO → Fix formula\n ↓\n5. Check kpiData not null? NO → Create CustomDataConfig or fix field name\n ↓\n✅ All validations pass — implementation complete\n```\n\n## Step 5: Production Checklist\n\nBefore deploying to production:\n\n- [ ] API key stored securely in environment variables\n- [ ] Error handling wraps all LLM calls\n- [ ] Failed executions still report events (with SuccessRate: 0)\n- [ ] All custom data fields have corresponding CustomDataConfig entries\n- [ ] KPI formulas validated and showing numeric values (not strings)\n- [ ] SDK configured with appropriate retries and timeouts\n- [ ] Sensitive data redaction enabled if needed\n\n## KPI Formula Reference\n\n### Supported Operators\n\n| Category | Operators |\n|----------|-----------|\n| Arithmetic | `+`, `-`, `*`, `/` |\n| Comparison | `<`, `<=`, `=`, `<>`, `>=`, `>` |\n| Logical | `AND`, `OR`, `NOT` |\n| Conditional | `IF(condition, true_val, false_val)`, `MAP(value, match1, out1, default)` |\n| Math | `ABS`, `MAX`, `MIN`, `AVERAGE`, `TRUNC` |\n| Null handling | `ISNA(value)`, `ISDEFINED(value)`, `NA()` |\n\n### Common Formula Patterns\n\n```bash\n--formula \"ItemsProcessed\" # passthrough\n--formula \"SuccessRate * 100\" # percentage (0-1 to 0-100)\n--formula \"IF(SuccessRate < 1, 1, 0)\" # conditional counting\n--formula \"IF(PII detected, 1, 0)\" # built-in variable\n--formula \"IF(ISDEFINED(MyField), MyField, 0)\" # null-safe\n--formula \"IF(AND(StepCount > 5, SuccessRate < 0.9), 1, 0)\" # compound conditions\n```\n\n### Aggregation Types\n\n| Aggregation | Use For | Example |\n|-------------|---------|---------|\n| `SUM` | Totals, counts | Total items processed across all runs |\n| `AVERAGE` | Rates, percentages | Average success rate |\n\n## Task Categories Reference\n\nUse these predefined task categories for the `task` field:\n\n| Category | Example Use |\n|----------|-------------|\n| Research & Intelligence | Competitive intelligence, market research |\n| Data Processing & Analysis | Data extraction, statistical analysis |\n| Content Development | Blog writing, technical documentation |\n| Content Refinement | Editing, proofreading |\n| Customer Experience | Complaint resolution, ticket triage |\n| Software Development | Code generation, code review, debugging |\n| Strategic Planning | Roadmap development, scenario planning |\n\n## Quick Reference\n\n```bash\n# CLI Commands\nolakai login # Authenticate\nolakai workflows create --name \"Name\" --json # Create workflow\nolakai agents create --name \"Name\" --workflow ID --with-api-key # Register agent\nolakai custom-data create --agent-id ID --name X --type NUMBER # Create custom field\nolakai kpis create --formula \"X\" --agent-id ID # Create KPI\nolakai activity list --agent-id ID # View events\n```\n\n```typescript\n// TypeScript SDK\nimport { olakaiConfig, olakai } from \"@olakai/sdk\";\nolakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });\n\nolakai(\"event\", \"ai_activity\", {\n prompt: \"input\",\n response: \"output\",\n tokens: 1500,\n task: \"Data Processing & Analysis\",\n customData: { StepCount: 3, Success: 1 },\n});\n```\n\n```python\n# Python SDK\nfrom olakaisdk import olakai_config, olakai, OlakaiEventParams\nolakai_config(os.getenv(\"OLAKAI_API_KEY\"))\n\nolakai(\"event\", \"ai_activity\", OlakaiEventParams(\n prompt=\"input\",\n response=\"output\",\n tokens=1500,\n task=\"Data Processing & Analysis\",\n customData={\"StepCount\": 3, \"Success\": 1},\n))\n```\n","content/playwright-community/skills/login-flows/SKILL.md":"---\nname: login-flows\ndescription: Common login automation patterns for web apps using Playwright\nmetadata:\n updated-on: \"2026-02-01\"\n source: community\n tags: \"browser,automation,playwright,login,testing\"\n---\n\n# Login Flow Patterns for Playwright\n\nReusable patterns for automating login flows in end-to-end tests.\n\n## Basic Username/Password Login\n\n```typescript\nimport { Page } from '@playwright/test';\n\nasync function login(page: Page, username: string, password: string) {\n await page.goto('/login');\n await page.fill('[name=\"username\"]', username);\n await page.fill('[name=\"password\"]', password);\n await page.click('button[type=\"submit\"]');\n await page.waitForURL('/dashboard');\n}\n```\n\n## OAuth / SSO Login\n\nFor OAuth flows that redirect to an external provider:\n\n```typescript\nasync function loginWithOAuth(page: Page) {\n await page.goto('/login');\n await page.click('text=Sign in with Google');\n\n // Handle the OAuth popup or redirect\n await page.fill('input[type=\"email\"]', process.env.TEST_EMAIL!);\n await page.click('text=Next');\n await page.fill('input[type=\"password\"]', process.env.TEST_PASSWORD!);\n await page.click('text=Next');\n\n // Wait for redirect back to app\n await page.waitForURL('**/dashboard');\n}\n```\n\n## Persisting Auth State (storageState)\n\nAvoid logging in before every test by saving and reusing browser state:\n\n```typescript\n// global-setup.ts — runs once before all tests\nimport { chromium } from '@playwright/test';\n\nasync function globalSetup() {\n const browser = await chromium.launch();\n const page = await browser.newPage();\n\n await page.goto('/login');\n await page.fill('[name=\"username\"]', 'testuser');\n await page.fill('[name=\"password\"]', 'testpass');\n await page.click('button[type=\"submit\"]');\n await page.waitForURL('/dashboard');\n\n // Save signed-in state\n await page.context().storageState({ path: './auth.json' });\n await browser.close();\n}\n\nexport default globalSetup;\n```\n\nThen in `playwright.config.ts`:\n\n```typescript\nexport default defineConfig({\n globalSetup: './global-setup.ts',\n use: {\n storageState: './auth.json',\n },\n});\n```\n\n## MFA / 2FA Handling\n\nFor test environments with TOTP-based 2FA:\n\n```typescript\nimport { authenticator } from 'otplib';\n\nasync function loginWithMFA(page: Page, secret: string) {\n await login(page, 'user', 'pass');\n\n // Generate TOTP code\n const code = authenticator.generate(secret);\n await page.fill('[name=\"totp\"]', code);\n await page.click('button[type=\"submit\"]');\n await page.waitForURL('/dashboard');\n}\n```\n\n## Tips\n\n- Always use `storageState` to avoid repeated logins — it's the single biggest speedup for E2E suites\n- Use environment variables for credentials, never hardcode them\n- For CI, consider a dedicated test user with stable credentials\n- Use `page.waitForURL()` instead of arbitrary `waitForTimeout()` after login\n","content/rails/docs/activerecord/references/associations.md":"# ActiveRecord Associations Reference\n\n## Association Types\n\n### belongs_to\n\nCreates a foreign key column on this model's table.\n\n```ruby\nclass Post < ApplicationRecord\n belongs_to :user # required by default\n belongs_to :category, optional: true # allows nil\n belongs_to :author, class_name: \"User\", foreign_key: \"author_id\"\n belongs_to :parent_post, class_name: \"Post\", optional: true\nend\n```\n\nOptions:\n- `optional: true` — allows nil foreign key\n- `class_name:` — specify model when name differs\n- `foreign_key:` — override foreign key column name\n- `counter_cache: true` — maintain `posts_count` on parent\n- `touch: true` — update parent's `updated_at` on save\n- `inverse_of:` — specify inverse association for bidirectional loading\n\n### has_many\n\n```ruby\nclass User < ApplicationRecord\n has_many :posts, dependent: :destroy\n has_many :draft_posts, -> { where(status: :draft) }, class_name: \"Post\"\n has_many :comments, through: :posts\nend\n```\n\nOptions:\n- `dependent:` — `:destroy`, `:delete_all`, `:nullify`, `:restrict_with_error`\n- `-> { scope }` — default scope on the association\n- `through:` — join through another association\n- `source:` — specify source association name on the through model\n- `inverse_of:` — bidirectional association hint\n\n### has_one\n\n```ruby\nclass User < ApplicationRecord\n has_one :profile, dependent: :destroy\n has_one :latest_post, -> { order(created_at: :desc) }, class_name: \"Post\"\nend\n```\n\n### has_and_belongs_to_many (HABTM)\n\nRequires a join table with no model:\n\n```ruby\n# Join table migration\ncreate_table :posts_tags, id: false do |t|\n t.belongs_to :post\n t.belongs_to :tag\nend\n\nclass Post < ApplicationRecord\n has_and_belongs_to_many :tags\nend\n\nclass Tag < ApplicationRecord\n has_and_belongs_to_many :posts\nend\n```\n\n### has_many :through (preferred over HABTM)\n\nWhen you need attributes on the join or callbacks:\n\n```ruby\nclass User < ApplicationRecord\n has_many :memberships\n has_many :teams, through: :memberships\nend\n\nclass Membership < ApplicationRecord\n belongs_to :user\n belongs_to :team\n # Can have additional attributes: role, joined_at, etc.\nend\n\nclass Team < ApplicationRecord\n has_many :memberships\n has_many :users, through: :memberships\nend\n```\n\n## Polymorphic Associations\n\n```ruby\nclass Comment < ApplicationRecord\n belongs_to :commentable, polymorphic: true\nend\n\nclass Post < ApplicationRecord\n has_many :comments, as: :commentable\nend\n\nclass Photo < ApplicationRecord\n has_many :comments, as: :commentable\nend\n```\n\nMigration:\n\n```ruby\ncreate_table :comments do |t|\n t.text :body\n t.references :commentable, polymorphic: true, index: true\n t.timestamps\nend\n```\n\n## Eager Loading Strategies\n\n```ruby\n# includes: lets Rails choose preload or eager_load\nUser.includes(:posts)\n\n# preload: always separate queries (one per association)\nUser.preload(:posts, :profile)\n\n# eager_load: LEFT OUTER JOIN (single query)\nUser.eager_load(:posts).where(posts: { published: true })\n\n# Nested eager loading\nUser.includes(posts: [:comments, :tags])\n```\n\n## Self-Referential Associations\n\n```ruby\nclass Employee < ApplicationRecord\n belongs_to :manager, class_name: \"Employee\", optional: true\n has_many :reports, class_name: \"Employee\", foreign_key: \"manager_id\"\nend\n```\n\n## Common Patterns\n\n### Dependent options guide\n\n| Option | Behavior |\n|--------|----------|\n| `:destroy` | Call destroy on each associated record (runs callbacks) |\n| `:delete_all` | Direct SQL delete (no callbacks) |\n| `:nullify` | Set foreign key to NULL |\n| `:restrict_with_error` | Add error if records exist, prevent delete |\n| `:restrict_with_exception` | Raise exception if records exist |\n","content/rails/docs/activerecord/references/migrations.md":"# ActiveRecord Migrations Reference\n\n## Generating Migrations\n\n```bash\n# Generate standalone migration\nbin/rails generate migration CreateUsers name:string email:string\nbin/rails generate migration AddRoleToUsers role:string\nbin/rails generate migration RemoveAgeFromUsers age:integer\nbin/rails generate migration AddIndexToUsersEmail\n\n# Generate with model\nbin/rails generate model Post title:string body:text user:references\n```\n\nRails infers migration content from the name:\n- `CreateXxx` → `create_table`\n- `AddXxxToYyy` → `add_column`\n- `RemoveXxxFromYyy` → `remove_column`\n\n## Column Types\n\n```ruby\ncreate_table :examples do |t|\n t.string :name # VARCHAR(255)\n t.text :description # TEXT\n t.integer :count # INTEGER\n t.bigint :large_count # BIGINT\n t.float :score # FLOAT\n t.decimal :price, precision: 10, scale: 2 # DECIMAL(10,2)\n t.boolean :active # BOOLEAN\n t.date :birthday # DATE\n t.datetime :published_at # DATETIME/TIMESTAMP\n t.time :start_time # TIME\n t.json :metadata # JSON (native)\n t.jsonb :settings # JSONB (PostgreSQL)\n t.binary :data # BLOB\n t.uuid :external_id # UUID (PostgreSQL)\n\n t.references :user, foreign_key: true # user_id + index + FK\n t.timestamps # created_at, updated_at\nend\n```\n\n## Column Options\n\n```ruby\nt.string :email, null: false # NOT NULL\nt.string :role, default: \"user\" # DEFAULT value\nt.integer :position, limit: 2 # SMALLINT\nt.string :code, index: true # add index\nt.string :email, index: { unique: true } # unique index\nt.text :comment, comment: \"User comment\" # column comment\n```\n\n## Table Operations\n\n```ruby\n# Create\ncreate_table :users do |t|\n t.string :name\n t.timestamps\nend\n\n# Create with UUID primary key (PostgreSQL)\ncreate_table :users, id: :uuid do |t|\n t.string :name\nend\n\n# Rename\nrename_table :old_name, :new_name\n\n# Drop\ndrop_table :users\n\n# Change table\nchange_table :users do |t|\n t.remove :age\n t.string :role\n t.rename :name, :full_name\nend\n```\n\n## Column Operations\n\n```ruby\n# Add\nadd_column :users, :role, :string, default: \"user\"\n\n# Remove\nremove_column :users, :age, :integer # specify type for reversibility\n\n# Rename\nrename_column :users, :name, :full_name\n\n# Change type\nchange_column :users, :age, :string\nchange_column_null :users, :email, false # set NOT NULL\nchange_column_default :users, :role, \"member\" # change default\n```\n\n## Index Operations\n\n```ruby\n# Single column\nadd_index :users, :email\nadd_index :users, :email, unique: true\n\n# Multi-column\nadd_index :users, [:last_name, :first_name]\n\n# Partial index (PostgreSQL)\nadd_index :users, :email, where: \"active = true\"\n\n# Remove\nremove_index :users, :email\nremove_index :users, column: [:last_name, :first_name]\n```\n\n## Foreign Keys\n\n```ruby\nadd_foreign_key :posts, :users\nadd_foreign_key :posts, :users, column: :author_id\nadd_foreign_key :posts, :users, on_delete: :cascade\n\nremove_foreign_key :posts, :users\n```\n\n## Reversible Migrations\n\n```ruby\nclass MigrateData < ActiveRecord::Migration[8.0]\n def change\n # Reversible block for custom up/down\n reversible do |dir|\n dir.up do\n execute \"UPDATE users SET role = 'member' WHERE role IS NULL\"\n end\n dir.down do\n execute \"UPDATE users SET role = NULL WHERE role = 'member'\"\n end\n end\n end\nend\n```\n\nOr use explicit `up`/`down`:\n\n```ruby\nclass MigrateData < ActiveRecord::Migration[8.0]\n def up\n add_column :users, :role, :string, default: \"member\"\n User.reset_column_information\n User.update_all(role: \"member\")\n end\n\n def down\n remove_column :users, :role\n end\nend\n```\n\n## Running Migrations\n\n```bash\nbin/rails db:migrate # run all pending\nbin/rails db:migrate VERSION=20260317120000 # up to specific version\nbin/rails db:rollback # undo last migration\nbin/rails db:rollback STEP=3 # undo last 3\nbin/rails db:migrate:status # show status of all migrations\nbin/rails db:migrate:redo # rollback then migrate\nbin/rails db:schema:load # load schema.rb (fresh DB)\nbin/rails db:seed # run db/seeds.rb\nbin/rails db:reset # drop + create + schema:load + seed\n```\n\n## Common Patterns\n\n### Safe zero-downtime migration (add column with default)\n\n```ruby\n# Rails 8 handles this in a single step safely:\nadd_column :users, :role, :string, default: \"user\", null: false\n```\n\n### Add index concurrently (PostgreSQL, no table lock)\n\n```ruby\nclass AddIndexToUsersEmail < ActiveRecord::Migration[8.0]\n disable_ddl_transaction!\n\n def change\n add_index :users, :email, unique: true, algorithm: :concurrently\n end\nend\n```\n\n### Data migration with safety\n\n```ruby\nclass BackfillUserRoles < ActiveRecord::Migration[8.0]\n def up\n User.in_batches(of: 1000).update_all(role: \"member\")\n end\n\n def down\n # no-op or reverse\n end\nend\n```\n","content/rails/docs/activerecord/references/queries.md":"# ActiveRecord Queries Reference\n\n## Where Conditions\n\n```ruby\n# Hash conditions (safe, auto-parameterized)\nUser.where(name: \"Alice\")\nUser.where(role: [\"admin\", \"moderator\"]) # IN clause\nUser.where(age: 18..65) # BETWEEN\nUser.where(age: 18..) # >= 18\nUser.where(age: ..65) # <= 65\n\n# String conditions (use ? placeholders)\nUser.where(\"email LIKE ?\", \"%@example.com\")\nUser.where(\"age > ? AND role = ?\", 21, \"user\")\n\n# Named placeholders\nUser.where(\"name = :name AND role = :role\", name: \"Alice\", role: \"admin\")\n\n# NOT\nUser.where.not(role: \"admin\")\nUser.where.not(age: nil)\n\n# OR\nUser.where(role: \"admin\").or(User.where(role: \"moderator\"))\n\n# Missing (NULL foreign key / no associated record)\nPost.where.missing(:comments)\n```\n\n## Ordering and Limiting\n\n```ruby\nUser.order(:name) # ASC\nUser.order(name: :desc)\nUser.order(role: :asc, name: :desc)\nUser.order(Arel.sql(\"LOWER(name)\")) # raw SQL ordering\n\nUser.limit(10)\nUser.offset(20)\nUser.limit(10).offset(20) # pagination\n```\n\n## Select and Pluck\n\n```ruby\n# select: returns AR objects with only specified columns\nUser.select(:id, :name)\n\n# pluck: returns raw arrays, no AR objects (faster)\nUser.pluck(:id) # [1, 2, 3]\nUser.pluck(:id, :name) # [[1, \"Alice\"], [2, \"Bob\"]]\n\n# pick: pluck for single row\nUser.where(email: \"a@b.com\").pick(:id) # 42\n```\n\n## Aggregations\n\n```ruby\nUser.count\nUser.where(active: true).count\nUser.maximum(:age)\nUser.minimum(:age)\nUser.average(:age)\nUser.sum(:balance)\n\n# Group\nUser.group(:role).count # { \"admin\" => 5, \"user\" => 42 }\nUser.group(:role).average(:age)\nUser.group(:role).having(\"COUNT(*) > ?\", 10).count\n```\n\n## Joins\n\n```ruby\n# INNER JOIN\nUser.joins(:posts)\nUser.joins(:posts).where(posts: { published: true })\n\n# Multiple joins\nUser.joins(:posts, :profile)\nUser.joins(posts: :comments) # nested join\n\n# LEFT OUTER JOIN\nUser.left_joins(:posts)\nUser.left_outer_joins(:posts)\n\n# Custom SQL join\nUser.joins(\"INNER JOIN posts ON posts.user_id = users.id AND posts.featured = true\")\n```\n\n## Find Methods\n\n```ruby\nUser.find(1) # raises RecordNotFound\nUser.find([1, 2, 3]) # returns array\nUser.find_by(email: \"a@b.com\") # nil if not found\nUser.find_by!(email: \"a@b.com\") # raises RecordNotFound\n\nUser.first # ORDER BY id ASC LIMIT 1\nUser.last # ORDER BY id DESC LIMIT 1\nUser.take # no ordering, LIMIT 1\n\nUser.find_or_create_by(email: \"a@b.com\") # find or create\nUser.find_or_initialize_by(email: \"a@b.com\") # find or new (unsaved)\n```\n\n## Batch Processing\n\n```ruby\n# find_each: loads in batches of 1000 (configurable)\nUser.where(active: true).find_each(batch_size: 500) do |user|\n user.send_newsletter\nend\n\n# find_in_batches: yields arrays of records\nUser.find_in_batches(batch_size: 1000) do |batch|\n batch.each { |user| process(user) }\nend\n\n# in_batches: yields Relations (for update_all, delete_all)\nUser.in_batches(of: 1000).update_all(notified: true)\n```\n\n## Exists and Presence\n\n```ruby\nUser.exists?(1) # by ID\nUser.exists?(email: \"a@b.com\") # by condition\nUser.where(role: \"admin\").exists? # any admin?\nUser.where(role: \"admin\").any? # same but loads records\nUser.where(role: \"admin\").none? # inverse\nUser.where(role: \"admin\").empty? # no records?\n```\n\n## Raw SQL\n\n```ruby\n# When AR query interface isn't enough\nresults = ActiveRecord::Base.connection.execute(\"SELECT ...\")\nusers = User.find_by_sql(\"SELECT * FROM users WHERE complex_condition\")\n\n# Sanitize user input in raw SQL\nUser.where(\n ActiveRecord::Base.sanitize_sql([\"name = ?\", user_input])\n)\n```\n\n## Query Performance\n\n```ruby\n# Explain query plan\nUser.where(role: \"admin\").explain\n\n# to_sql: inspect generated SQL\nputs User.where(role: \"admin\").order(:name).to_sql\n\n# Avoid N+1: use includes/preload/eager_load\nUser.includes(:posts).where(active: true).each do |user|\n user.posts # no extra query\nend\n\n# strict_loading: raise on lazy load (catch N+1 in dev)\nuser = User.strict_loading.find(1)\nuser.posts # raises ActiveRecord::StrictLoadingViolationError\n```\n","content/rspec/docs/package/references/matchers.md":"# RSpec Matchers Reference\n\n## Equality Matchers\n\n```ruby\nexpect(a).to eq(b) # a == b (most common)\nexpect(a).to eql(b) # a.eql?(b) — value equality\nexpect(a).to equal(b) # a.equal?(b) — identity (same object)\nexpect(a).to be(b) # alias for equal\n```\n\n## Comparison Matchers\n\n```ruby\nexpect(a).to be > 5\nexpect(a).to be >= 5\nexpect(a).to be < 5\nexpect(a).to be <= 5\nexpect(a).to be_between(1, 10).inclusive\nexpect(a).to be_between(1, 10).exclusive\nexpect(a).to be_within(0.5).of(3.14)\n```\n\n## Truthiness Matchers\n\n```ruby\nexpect(value).to be true # === true\nexpect(value).to be false # === false\nexpect(value).to be_truthy # not nil and not false\nexpect(value).to be_falsey # nil or false\nexpect(value).to be_nil # === nil\n```\n\n## Type Matchers\n\n```ruby\nexpect(obj).to be_a(String)\nexpect(obj).to be_an(Array)\nexpect(obj).to be_a_kind_of(Numeric) # alias\nexpect(obj).to be_an_instance_of(Float) # exact class, not subclass\nexpect(obj).to respond_to(:length)\nexpect(obj).to respond_to(:resize).with(2).arguments\n```\n\n## String Matchers\n\n```ruby\nexpect(str).to include(\"sub\")\nexpect(str).to start_with(\"hello\")\nexpect(str).to end_with(\"world\")\nexpect(str).to match(/regex/)\nexpect(str).to match(\"substring\") # also works\n```\n\n## Collection Matchers\n\n```ruby\n# include: contains elements (order doesn't matter, partial match OK)\nexpect([1, 2, 3]).to include(1, 3)\nexpect({ a: 1, b: 2 }).to include(a: 1)\n\n# contain_exactly: exact elements, any order\nexpect([3, 1, 2]).to contain_exactly(1, 2, 3)\n\n# match_array: alias for contain_exactly\nexpect([3, 1, 2]).to match_array([1, 2, 3])\n\n# all: every element matches\nexpect([2, 4, 6]).to all(be_even)\nexpect([\"a\", \"b\"]).to all(be_a(String))\n\n# Size\nexpect(arr).to be_empty\nexpect(arr).to have_attributes(size: 3)\n\n# Nested\nexpect(arr).to include(a_hash_including(name: \"Alice\"))\nexpect(arr).to contain_exactly(\n a_hash_including(name: \"Alice\"),\n a_hash_including(name: \"Bob\")\n)\n```\n\n## Change Matchers\n\n```ruby\nexpect { x += 1 }.to change { x }.by(1)\nexpect { x += 1 }.to change { x }.from(0).to(1)\nexpect { arr.push(1) }.to change(arr, :size).by(1)\nexpect { action }.not_to change { x }\n\n# Compound change\nexpect { action }.to change(User, :count).by(1)\n .and change(AuditLog, :count).by(1)\n```\n\n## Error Matchers\n\n```ruby\nexpect { code }.to raise_error\nexpect { code }.to raise_error(SomeError)\nexpect { code }.to raise_error(SomeError, \"message\")\nexpect { code }.to raise_error(SomeError, /partial/)\nexpect { code }.to raise_error { |e| expect(e.code).to eq(404) }\nexpect { code }.not_to raise_error\n```\n\n## Output Matchers\n\n```ruby\nexpect { print \"hi\" }.to output(\"hi\").to_stdout\nexpect { warn \"oops\" }.to output(/oops/).to_stderr\nexpect { code }.to output.to_stdout\n```\n\n## Predicate Matchers\n\nRSpec auto-generates matchers from predicate methods:\n\n```ruby\n# object.empty? → be_empty\nexpect([]).to be_empty\n\n# object.valid? → be_valid\nexpect(user).to be_valid\n\n# object.has_key?(:name) → have_key(:name)\nexpect(hash).to have_key(:name)\n\n# object.active? → be_active\nexpect(user).to be_active\n```\n\n## Composed Matchers\n\n```ruby\nexpect(value).to be_a(String).and include(\"hello\")\nexpect(value).to start_with(\"a\").or start_with(\"b\")\n\n# Argument matchers in expectations\nexpect(arr).to include(a_value_between(1, 10))\nexpect(hash).to include(name: a_string_starting_with(\"A\"))\nexpect(arr).to all(be_a(Integer).and be > 0)\n```\n\n## Custom Matchers\n\n```ruby\nRSpec::Matchers.define :be_a_multiple_of do |expected|\n match do |actual|\n actual % expected == 0\n end\n\n failure_message do |actual|\n \"expected #{actual} to be a multiple of #{expected}\"\n end\nend\n\nexpect(9).to be_a_multiple_of(3)\n```\n\n## Negation\n\nAll matchers support negation with `not_to` or `to_not`:\n\n```ruby\nexpect(value).not_to eq(other)\nexpect(value).not_to be_nil\nexpect(arr).not_to include(42)\nexpect { code }.not_to raise_error\n```\n","content/rspec/docs/package/references/mocks.md":"# RSpec Mocks Reference\n\n## Doubles (Test Doubles)\n\n```ruby\n# Basic double\nuser = double(\"User\")\nuser = double(\"User\", name: \"Alice\", email: \"a@b.com\")\n\n# Verified double — raises if methods don't exist on the real class\nuser = instance_double(\"User\", name: \"Alice\")\nuser = class_double(\"User\", count: 42)\nuser = object_double(User.new, name: \"Alice\")\n```\n\nPrefer verified doubles (`instance_double`, `class_double`) — they catch method signature mismatches.\n\n## Stubs (allow)\n\n```ruby\n# Stub a method\nallow(user).to receive(:name).and_return(\"Alice\")\nallow(user).to receive(:save).and_return(true)\n\n# Stub with block\nallow(service).to receive(:fetch) { |id| \"item_#{id}\" }\n\n# Stub chain\nallow(User).to receive_message_chain(:active, :recent, :first).and_return(user)\n\n# Stub any instance (use sparingly)\nallow_any_instance_of(User).to receive(:valid?).and_return(true)\n```\n\n## Return Values\n\n```ruby\nallow(obj).to receive(:method).and_return(\"value\")\nallow(obj).to receive(:method).and_return(1, 2, 3) # returns 1, then 2, then 3\nallow(obj).to receive(:method).and_raise(RuntimeError, \"boom\")\nallow(obj).to receive(:method).and_throw(:halt)\nallow(obj).to receive(:method).and_yield(\"block_arg\")\nallow(obj).to receive(:method).and_call_original # call real implementation\nallow(obj).to receive(:method).and_wrap_original do |m, *args|\n m.call(*args) + \" modified\"\nend\n```\n\n## Expectations (expect...to receive)\n\n```ruby\n# Expect a method to be called\nexpect(mailer).to receive(:send_email).with(user.email, anything)\n\n# Expect with specific count\nexpect(logger).to receive(:info).once\nexpect(logger).to receive(:info).twice\nexpect(logger).to receive(:info).exactly(3).times\nexpect(logger).to receive(:info).at_least(:once)\nexpect(logger).to receive(:info).at_most(5).times\n\n# Expect not called\nexpect(mailer).not_to receive(:send_email)\n\n# Ordered expectations\nexpect(obj).to receive(:first).ordered\nexpect(obj).to receive(:second).ordered\n```\n\n## Argument Matchers\n\n```ruby\nexpect(obj).to receive(:method).with(\"exact\")\nexpect(obj).to receive(:method).with(anything)\nexpect(obj).to receive(:method).with(any_args)\nexpect(obj).to receive(:method).with(no_args)\nexpect(obj).to receive(:method).with(hash_including(key: \"val\"))\nexpect(obj).to receive(:method).with(hash_excluding(key: \"val\"))\nexpect(obj).to receive(:method).with(array_including(1, 2))\nexpect(obj).to receive(:method).with(a_string_matching(/pattern/))\nexpect(obj).to receive(:method).with(an_instance_of(String))\nexpect(obj).to receive(:method).with(duck_type(:to_s, :length))\nexpect(obj).to receive(:method).with(boolean) # true or false\n```\n\n## Spying (have_received)\n\nTest after the fact instead of setting expectations before:\n\n```ruby\n# Arrange\nallow(mailer).to receive(:send_email)\n\n# Act\nservice.process(user)\n\n# Assert\nexpect(mailer).to have_received(:send_email).with(user.email)\nexpect(mailer).to have_received(:send_email).once\n```\n\nThis is often cleaner than `expect(...).to receive(...)` because it follows arrange-act-assert.\n\n## Partial Doubles (stubbing real objects)\n\n```ruby\nuser = User.new(name: \"Alice\")\n\n# Stub one method on a real object\nallow(user).to receive(:expensive_calculation).and_return(42)\n\n# The rest of the object works normally\nexpect(user.name).to eq(\"Alice\")\nexpect(user.expensive_calculation).to eq(42)\n```\n\n## Stubbing Constants\n\n```ruby\nstub_const(\"ENV_NAME\", \"test\")\nstub_const(\"Config::MAX_RETRIES\", 5)\nstub_const(\"ExternalService\", class_double(\"ExternalService\", call: \"mocked\"))\n```\n\n## Null Object Doubles\n\n```ruby\n# Returns itself for any undefined method (useful for chaining)\nlogger = double(\"Logger\").as_null_object\nlogger.info(\"test\") # doesn't raise\nlogger.unknown_method # returns the double itself\n```\n\n## Best Practices\n\n- **Prefer verified doubles** (`instance_double`) over plain `double` — catches interface changes.\n- **Prefer spies** (`have_received`) over expectations (`receive`) — follows arrange-act-assert.\n- **Don't mock what you don't own** — wrap third-party APIs in your own adapter, mock the adapter.\n- **Avoid `allow_any_instance_of`** — it's a code smell; refactor to inject dependencies instead.\n- **Mock at boundaries** — mock external services, not internal collaborators.\n","content/snowflake/docs/cortex-code/references/commands.md":"# Commands Reference\n\nFull reference for Cortex Code CLI commands, slash commands, and flags.\n\n## CLI Entry Points\n\n```bash\ncortex # Start interactive REPL\ncortex -p \"query\" # Non-interactive print mode\ncortex resume # Interactive session picker\ncortex -r last # Resume last session\ncortex -r # Resume specific session\n```\n\n## CLI Subcommands\n\n| Command | Description |\n|---------|-------------|\n| `cortex mcp list\\|add\\|remove\\|show` | Manage MCP servers |\n| `cortex skill list\\|add\\|remove` | Manage skills |\n| `cortex resume [id]` | Resume session |\n| `cortex update` | Update CLI |\n| `cortex versions` | List available versions |\n| `cortex worktree list\\|create\\|switch\\|delete` | Git worktrees |\n| `cortex completion install\\|generate` | Shell tab-completion (bash/zsh/fish) |\n| `cortex connections list\\|set` | Snowflake connections |\n| `cortex env detect` | Detect Python environment |\n| `cortex source -- ` | Run command with Snowflake credentials as env vars |\n| `cortex ctx` | Long-term AI memory and task management |\n| `cortex browser` | Browser automation (Playwright MCP) |\n| `cortex search object \"\"` | Search Snowflake objects |\n| `cortex search docs \"\"` | Search Snowflake product docs |\n| `cortex reflect ` | Validate semantic model YAML |\n| `cortex semantic-views list\\|discover\\|describe\\|search\\|ddl\\|query` | Semantic view operations |\n| `cortex analyst query \"\"` | Query via Cortex Analyst |\n| `cortex artifact create notebook\\|file ` | Upload to Snowflake Workspace |\n| `cortex agents discover\\|list\\|describe\\|search` | Cortex Agent operations |\n\n## CLI Flags\n\n| Flag | Short | Description |\n|------|-------|-------------|\n| `--resume ` | `-r` | Resume session |\n| `--print ` | `-p` | Non-interactive mode |\n| `--workdir ` | `-w` | Working directory |\n| `--worktree ` | | Git worktree |\n| `--connection ` | `-c` | Snowflake connection |\n| `--model ` | `-m` | Model override |\n| `--plan` | | Start in plan mode |\n| `--bypass` | | Start in bypass mode |\n| `--config ` | | Custom settings.json |\n| `--no-mcp` | | Disable MCP servers |\n| `--version` | `-V` | Show version |\n\n## Slash Commands — Session\n\n| Command | Description |\n|---------|-------------|\n| `/quit`, `/q` | Exit |\n| `/clear` | Clear screen |\n| `/new` | New session |\n| `/fork [name]` | Fork conversation from a chosen point |\n| `/resume` | Session picker |\n| `/rename ` | Rename session |\n| `/rewind [N]` | Rewind N user messages (or interactive) |\n| `/compact [instructions]` | Summarize context and clear history |\n\n## Slash Commands — Information\n\n| Command | Description |\n|---------|-------------|\n| `/help` | Show help |\n| `/status` | Session status |\n| `/commands` | List all commands |\n\n## Slash Commands — Modes\n\n| Command | Description |\n|---------|-------------|\n| `/plan` | Enable plan mode |\n| `/plan-off` | Disable plan mode |\n| `/bypass` | Enable bypass mode |\n| `/bypass-off` | Disable bypass mode |\n| `/model ` | Switch model |\n\n## Slash Commands — Configuration\n\n| Command | Description |\n|---------|-------------|\n| `/settings` | Interactive settings editor |\n| `/mcp` | MCP server manager |\n| `/skill` | Skill manager |\n| `/hooks` | Hook manager |\n| `/theme` | Theme selector |\n| `/connections` | Connection manager |\n\n## Slash Commands — Development\n\n| Command | Description |\n|---------|-------------|\n| `/add-dir ` | Add working directory |\n| `/sh ` | Execute shell command |\n| `/sql ` | Execute SQL |\n| `/table` | Fullscreen SQL results view |\n| `/diff` | Fullscreen git diff viewer |\n| `/tasks` | Active background tasks |\n| `/worktree` | Worktree manager |\n| `/sandbox` | Sandbox settings |\n\n## Slash Commands — Utilities\n\n| Command | Description |\n|---------|-------------|\n| `/fdbt` | dbt operations |\n| `/lineage ` | Model lineage |\n| `/agents` | Subagent manager |\n| `/setup-jupyter` | Setup Jupyter |\n| `/feedback` | Submit feedback |\n| `/clear-cache` | Clear caches |\n| `/doctor` | Diagnose environment issues |\n| `/update` | Update Cortex Code CLI |\n\n## Custom Slash Commands\n\nDefine custom commands as Markdown files. Loaded in priority order:\n\n1. **Project** — `.cortex/commands/`, `.claude/commands/`\n2. **Global** — `~/.snowflake/cortex/commands/`\n3. **User** — `~/.claude/commands/`\n\nEach `.md` file becomes a `/command` named after the file.\n","content/snowflake/docs/cortex-code/references/configuration.md":"# Configuration Reference\n\nSettings, environment variables, connections, skills, and hooks for Cortex Code.\n\n## Directory Structure\n\n```\n~/.snowflake/cortex/\n├── settings.json # Main configuration\n├── skills.json # Skills configuration\n├── permissions.json # Permission history (auto-generated)\n├── hooks.json # Hook configurations\n├── mcp.json # MCP server configurations\n├── skills/ # Global skills\n│ └── my-skill/\n│ └── SKILL.md\n├── agents/ # Custom agent definitions\n├── commands/ # Custom slash commands\n└── conversations/ # Session history\n```\n\n## settings.json\n\nMain configuration file at `~/.snowflake/cortex/settings.json`:\n\n```json\n{\n \"env\": {\n \"CORTEX_AGENT_MODEL\": \"claude-sonnet-4-5\",\n \"CORTEX_AGENT_ENABLE_SUBAGENTS\": true,\n \"SNOVA_DEBUG\": false,\n \"CORTEX_CHANNEL\": \"stable\"\n },\n \"diffDisplayMode\": \"unified\",\n \"compactMode\": false,\n \"bashDefaultTimeoutMs\": 180000,\n \"theme\": \"dark\"\n}\n```\n\n| Key | Type | Description |\n|-----|------|-------------|\n| `diffDisplayMode` | `\"unified\"` \\| `\"side_by_side\"` | Diff display style |\n| `compactMode` | boolean | Start in compact UI mode |\n| `bashDefaultTimeoutMs` | number | Default bash timeout (ms) |\n| `theme` | `\"dark\"` \\| `\"light\"` \\| `\"pro\"` | UI color theme |\n\n## Environment Variables\n\n### Core\n\n| Variable | Description |\n|----------|-------------|\n| `CORTEX_CODE_STREAMING` | Enable live streaming mode |\n| `CORTEX_ENABLE_MEMORY` | Enable memory tool |\n| `CORTEX_ENABLE_EXPERIMENTAL_SKILLS` | Enable experimental skills |\n| `CORTEX_AGENT_ENABLE_SUBAGENTS` | Enable subagent spawning |\n\n### Snowflake\n\n| Variable | Description |\n|----------|-------------|\n| `SNOWFLAKE_CONNECTION` | Default connection name |\n| `SNOWFLAKE_ACCOUNT` | Snowflake account |\n| `SNOWFLAKE_USER` | Snowflake username |\n| `SNOWFLAKE_WAREHOUSE` | Default warehouse |\n| `SNOWFLAKE_DATABASE` | Default database |\n| `SNOWFLAKE_SCHEMA` | Default schema |\n\n### Priority Order\n\n1. **CLI flags** (highest)\n2. **Environment variables**\n3. **settings.json**\n4. **Built-in defaults** (lowest)\n\n## Skills System\n\nSkills are reusable instruction sets that guide the agent. Invoked with `$skill-name`.\n\n### Skill Locations (Priority Order)\n\n| Priority | Location |\n|----------|----------|\n| 1 | `.cortex/skills/`, `.claude/skills/` (project) |\n| 2 | `~/.snowflake/cortex/skills/` (global) |\n| 3 | Remote (cached from repositories) |\n| 4 | Bundled (shipped with Cortex Code) |\n\n### SKILL.md Format\n\n```markdown\n---\nname: my-skill\ndescription: \"Purpose. Use when: situations. Triggers: keywords.\"\ntools: [\"bash\", \"edit\"]\n---\n\n# My Skill\n\n## Workflow\n1. Step one\n2. Step two\n```\n\n### Remote Skills\n\nConfigure in `~/.snowflake/cortex/skills.json`:\n\n```json\n{\n \"paths\": [\"/path/to/additional/skills\"],\n \"remote\": [\n {\n \"source\": \"https://github.com/org/skills-repo\",\n \"ref\": \"main\",\n \"skills\": [{ \"name\": \"skill-name\" }]\n }\n ]\n}\n```\n\n### Managing Skills\n\n```\n/skill # Interactive skill manager\n```\n\nPress `a` to add, view conflicts, sync project skills to global, or delete.\n\n## Hooks System\n\nHooks run custom code at specific points during execution.\n\n### Hook Events\n\n| Event | When | Use Case |\n|-------|------|----------|\n| `PreToolUse` | Before tool execution | Validate, block, modify tool calls |\n| `PostToolUse` | After tool execution | Log, validate outputs |\n| `UserPromptSubmit` | User submits prompt | Validate or transform input |\n| `Stop` | Agent stops | Verify completion |\n| `SessionStart` | Session begins | Initialize context |\n| `SessionEnd` | Session ends | Cleanup |\n| `PreCompact` | Before compaction | Inject context |\n\n### Configuration\n\nHooks are configured in `~/.snowflake/cortex/hooks.json`:\n\n```json\n{\n \"hooks\": {\n \"PreToolUse\": [\n {\n \"matcher\": \"Bash\",\n \"hooks\": [\n { \"type\": \"command\", \"command\": \"/path/to/validator.sh\", \"timeout\": 30 }\n ]\n }\n ]\n }\n}\n```\n\n### Matchers\n\n| Pattern | Matches |\n|---------|---------|\n| `Bash` | Exact tool name |\n| `Bash\\|Edit` | Bash or Edit |\n| `.*` or `*` | All tools |\n| `snowflake_.*` | All Snowflake tools |\n| `mcp__github__.*` | All GitHub MCP tools |\n\n### Exit Codes\n\n| Code | Meaning |\n|------|---------|\n| 0 | Success, continue |\n| 2 | Block the operation (stderr sent to agent) |\n| Other | Non-blocking error (shown to user) |\n\n### Hook Input\n\nHooks receive JSON on stdin with fields: `session_id`, `transcript_path`, `cwd`, `permission_mode`, `hook_event_name`. Tool events add `tool_name`, `tool_input`.\n\n### Managing Hooks\n\n```\n/hooks # Interactive hook manager\n```\n\n## Theme Configuration\n\n```\n/theme # Interactive theme selector\n/theme dark # Set dark theme\n/theme light # Set light theme\n```\n","content/snowflake/docs/cortex-code/references/mcp-and-agents.md":"# MCP and Agents Reference\n\nExtending Cortex Code with MCP servers and autonomous subagents.\n\n## MCP (Model Context Protocol)\n\nMCP connects Cortex Code to external services, databases, APIs, and tools.\n\n### Managing Servers\n\n```bash\ncortex mcp add # Add server\ncortex mcp list # List all servers\ncortex mcp remove # Remove server\ncortex mcp show # Show server details\n/mcp # Interactive manager\n```\n\n### Configuration\n\nMCP servers are configured in `~/.snowflake/cortex/mcp.json`:\n\n```json\n{\n \"mcpServers\": {\n \"github\": {\n \"command\": \"npx\",\n \"args\": [\"-y\", \"@modelcontextprotocol/server-github\"],\n \"env\": { \"GITHUB_TOKEN\": \"${GITHUB_TOKEN}\" },\n \"transport\": \"stdio\",\n \"timeout\": 60000\n }\n }\n}\n```\n\n### Config Fields\n\n| Field | Description |\n|-------|-------------|\n| `command` | Command to start server |\n| `args` | Command arguments |\n| `env` | Environment variables (`${VAR}` for expansion) |\n| `transport` | `stdio` (default), `sse`, or `http` |\n| `timeout` | Connection timeout in ms |\n| `url` | Server URL (for SSE/HTTP) |\n| `headers` | HTTP headers (for SSE/HTTP) |\n\n### Transport Types\n\n| Type | Use Case | Key Config |\n|------|----------|------------|\n| `stdio` | Local process | `command`, `args` |\n| `sse` | Remote SSE server | `url`, `headers` |\n| `http` | Remote HTTP API | `url`, `headers` |\n\n### Adding Servers\n\n```bash\n# stdio\ncortex mcp add my-server -- npx -y @scope/mcp-server\n\n# With env vars\ncortex mcp add my-server -e API_KEY=secret -- npx -y @scope/mcp-server\n\n# SSE\ncortex mcp add my-server --transport sse https://example.com/sse\n\n# HTTP with auth header\ncortex mcp add my-server --transport http -H \"Authorization: Bearer token\" https://example.com/api\n```\n\n### Common MCP Servers\n\n```bash\n# GitHub\ncortex mcp add github -e GITHUB_TOKEN=$GITHUB_TOKEN -- npx -y @modelcontextprotocol/server-github\n\n# Filesystem\ncortex mcp add filesystem -- npx -y @modelcontextprotocol/server-filesystem ~/Documents\n\n# PostgreSQL\ncortex mcp add postgres -e DATABASE_URL=$DATABASE_URL -- npx -y @modelcontextprotocol/server-postgres\n```\n\n### MCP Tool Naming\n\nTools follow the pattern `mcp____`. Example: `mcp__github__create_issue`.\n\n### Disabling MCP\n\n```bash\ncortex --no-mcp # Start without MCP\n```\n\n## Subagents\n\nCortex Code spawns autonomous child agents for complex, multi-step tasks.\n\n### Built-in Agent Types\n\n| Type | Description | Capabilities |\n|------|-------------|--------------|\n| `general-purpose` | Multi-step tasks, code changes | All tools |\n| `Explore` | Fast codebase exploration | Read-only |\n| `Plan` | Architecture and implementation planning | Read-only |\n| `feedback` | Collect and process user feedback | Ask questions |\n| `dbt-verify` | dbt project verification | All tools |\n\n### Explore Agent Thoroughness\n\nWhen asking for exploration, specify a level:\n- `\"quick\"` — basic file/keyword search\n- `\"medium\"` — moderate exploration across related files\n- `\"very thorough\"` — comprehensive multi-location analysis\n\n### Background Execution\n\nAgents can run in background while you continue working:\n- Use `/agents` to view status and manage running agents\n- Background agents return an `agent_id` for retrieving results later\n- Background agents cannot spawn other background agents\n\n### Custom Agents\n\nDefine custom agents as Markdown files with YAML frontmatter:\n\n```markdown\n---\nname: my-agent\ndescription: \"What this agent does. Triggers: relevant keywords.\"\ntools: [\"Bash\", \"Read\", \"Edit\"]\n---\n\n# My Agent\n\nInstructions for the agent (used as the system prompt).\n```\n\n### Loading Locations (Priority Order)\n\n| Priority | Location |\n|----------|----------|\n| 1 | `~/.snowflake/cortex/agents/` (global) |\n| 2 | `~/.claude/agents/` (global, compatibility) |\n| 3 | `.cortex/agents/` (project-local) |\n| 4 | `.claude/agents/` (project-local, compatibility) |\n\n### Worktree Isolation\n\nFor parallel agent work without file conflicts:\n\n```bash\ncortex worktree create # Create worktree\ncortex worktree list # List worktrees\ncortex --worktree # Start session in worktree\n```\n\n### /agents Command\n\n```\n/agents\n```\n\nOpens a fullscreen tabbed interface showing available agent types, active sessions, and management options.\n","content/snowflake/docs/cortex-code/references/snowflake-tools.md":"# Snowflake-Native Tools Reference\n\nCortex Code integrates directly with Snowflake for SQL execution, object discovery, semantic analysis, and artifact management.\n\n## #TABLE Context Injection\n\nType `#` followed by a fully-qualified 3-part table name to auto-inject metadata into your prompt:\n\n```\nAnalyze the data in #MY_DB.PUBLIC.USERS\n```\n\nThis fetches and injects:\n- Column metadata (name, type, nullable, default, comment)\n- Primary keys\n- Approximate row count\n- Up to 3 sample rows\n\nAutocomplete is available after typing `#`. Names are case-insensitive.\n\n## SQL Execution\n\n### /sql Slash Command\n\n```\n/sql SELECT COUNT(*) FROM my_table\n```\n\nRuns SQL inline. Use `/table` or `Ctrl+T` for a fullscreen table view of results.\n\n### Agent SQL Execution\n\nThe agent executes SQL autonomously when needed via the `snowflake_sql_execute` tool:\n- Runs on the active Snowflake connection\n- Handles large result sets with truncation\n- Auto-refreshes expired tokens\n- Supports `only_compile` mode for validation without execution\n\n## Object Search\n\nSearch for Snowflake objects using semantic (natural language) search:\n\n```bash\ncortex search object \"user activity tables\"\ncortex search object \"revenue\" --types=table,view\n```\n\nSearches across databases, schemas, tables, views, semantic views, and functions.\n\n## Cortex Analyst\n\nConvert natural language questions to SQL using semantic models or semantic views:\n\n```bash\n# Using a local YAML semantic model\ncortex analyst query \"What were total sales last quarter?\" --model=sales_model.yaml\n\n# Using a semantic view\ncortex analyst query \"Top customers by spend\" --view=ANALYTICS.CORE.REVENUE_METRICS\n\n# Validate a semantic model file\ncortex reflect revenue.yaml\ncortex reflect revenue.yaml --target-schema DB.SCHEMA\n```\n\nReturns generated SQL, explanation, and suggested follow-up questions.\n\n## Semantic Views\n\nSemantic views are curated data models with business-friendly definitions built on top of raw tables.\n\n```bash\ncortex semantic-views discover # Find all semantic views in account\ncortex semantic-views list [--limit=N] # List views\ncortex semantic-views describe # Show dimensions, facts, metrics\ncortex semantic-views search \"revenue\" # Search by keyword\ncortex semantic-views ddl # Show SQL DDL definition\ncortex semantic-views query # Execute query against view\n```\n\n**Best practice:** Before writing complex SQL for business analytics, check if a semantic view exists. Semantic views provide verified business definitions more reliable than raw SQL.\n\n## Cortex Agent Discovery\n\nCortex Agents contain curated instructions about which data sources to use:\n\n```bash\ncortex agents discover # Find all agents in account\ncortex agents list [--database DB] # List agents in scope\ncortex agents describe # Show instructions, tools, data sources\ncortex agents search \"revenue\" # Search by domain\n```\n\n## Product Docs Search\n\n```bash\ncortex search docs \"how to create a stored procedure\"\ncortex search docs \"dynamic tables target lag\"\n```\n\nSearches Snowflake product documentation and returns relevant results.\n\n## Artifact Creation\n\nUpload files to Snowflake Workspace:\n\n```bash\ncortex artifact create notebook my_analysis ./analysis.ipynb\ncortex artifact create file my_report ./report.csv\n```\n\nSupports notebooks (`.ipynb`) and generic files. Default workspace: `USER$.PUBLIC.DEFAULT$`.\n\n## Connection Management\n\n### CLI\n\n```bash\ncortex connections list # List configured connections\ncortex connections set # Switch active connection\ncortex --connection # Start with specific connection\n```\n\n### /connections\n\nOpens an interactive fullscreen connection manager.\n\n### connections.toml\n\nConnections are defined in `~/.snowflake/connections.toml`:\n\n```toml\n[default]\naccount = \"myaccount\"\nuser = \"myuser\"\nauthenticator = \"externalbrowser\"\ndatabase = \"MYDB\"\nschema = \"PUBLIC\"\nwarehouse = \"COMPUTE_WH\"\nrole = \"DEVELOPER\"\n\n[prod]\naccount = \"myaccount\"\nuser = \"myuser\"\nauthenticator = \"snowflake_jwt\"\nprivate_key_path = \"~/.snowflake/rsa_key.p8\"\ndatabase = \"PROD_DB\"\nschema = \"PUBLIC\"\nwarehouse = \"PROD_WH\"\nrole = \"ANALYST\"\n```\n\n### Authentication Methods\n\n| Method | Value |\n|--------|-------|\n| Browser SSO | `externalbrowser` |\n| Key-pair | `snowflake_jwt` |\n| Password | `snowflake` |\n| OAuth | `oauth` |\n| Programmatic access token | `PROGRAMMATIC_ACCESS_TOKEN` |\n\n## Source Command\n\nRun a command with Snowflake credentials injected as environment variables:\n\n```bash\ncortex source --map account=SF_ACCOUNT --map user=SF_USER -- python myscript.py\ncortex source myconn --map password=SNOWFLAKE_PASS -- dbt run\n```\n\nAvailable fields: `account`, `user`, `password`, `authenticator`, `warehouse`, `database`, `schema`, `role`, `host`, `token`.\n","content/sourceparts/docs/parts-mcp/python/references/tools.md":"# Parts MCP — Tool Parameter Reference\n\nComplete parameter reference for all parts-mcp tools. Grouped by category.\n\n## Search Tools\n\n### search_parts\n\nSearch for electronic parts across suppliers.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `query` | `str` | required | Search query — part number, description, or keywords |\n| `category` | `str \\| None` | `None` | Category filter (e.g., \"resistor\", \"capacitor\", \"microcontroller\") |\n| `filters` | `dict \\| None` | `None` | Parametric filters (e.g., `{\"resistance\": \"10k\", \"tolerance\": \"1%\"}`) |\n| `limit` | `int` | `20` | Maximum results to return |\n\n**Returns:**\n\n```json\n{\n \"query\": \"STM32F411\",\n \"category\": null,\n \"filters\": {},\n \"results\": [\n {\n \"part_number\": \"STM32F411CEU6\",\n \"manufacturer\": \"STMicroelectronics\",\n \"description\": \"ARM Cortex-M4 MCU, 512KB Flash, 128KB RAM\",\n \"category\": \"microcontroller\"\n }\n ],\n \"total_results\": 15,\n \"success\": true\n}\n```\n\n### search_by_parameters\n\nParametric search within a specific category.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `parameters` | `dict` | required | Search criteria (e.g., `{\"capacitance\": \"100nF\", \"voltage\": \"50V\", \"package\": \"0402\"}`) |\n| `category` | `str` | required | Part category |\n| `limit` | `int` | `20` | Maximum results |\n\n**Returns:**\n\n```json\n{\n \"category\": \"capacitor\",\n \"parameters\": {\"capacitance\": \"100nF\", \"voltage\": \"50V\"},\n \"results\": [...],\n \"total_results\": 42,\n \"success\": true\n}\n```\n\n### get_part_details\n\nGet detailed information about a specific part.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `part_number` | `str` | required | Part number to look up |\n| `manufacturer` | `str \\| None` | `None` | Manufacturer name (disambiguates when multiple manufacturers use same part number) |\n\n**Returns:**\n\n```json\n{\n \"part_number\": \"STM32F411CEU6\",\n \"manufacturer\": \"STMicroelectronics\",\n \"details\": {\n \"description\": \"...\",\n \"specifications\": {...},\n \"datasheet_url\": \"...\",\n \"lifecycle_status\": \"active\"\n },\n \"success\": true\n}\n```\n\n## Sourcing Tools\n\n### compare_prices\n\nCompare prices across multiple suppliers.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `part_number` | `str` | required | Part number to price |\n| `quantity` | `int` | `1` | Quantity needed (affects price breaks) |\n| `suppliers` | `list[str] \\| None` | `None` | Specific suppliers to check (checks all if omitted) |\n\n**Returns:**\n\n```json\n{\n \"part_number\": \"STM32F411CEU6\",\n \"quantity\": 100,\n \"suppliers_checked\": 5,\n \"prices\": [\n {\n \"supplier\": \"LCSC\",\n \"sku\": \"C478234\",\n \"unit_price\": 3.42,\n \"total_price\": 342.00,\n \"stock\": 15000,\n \"lead_time\": \"3-5 days\"\n }\n ],\n \"best_price\": {\"supplier\": \"LCSC\", \"unit_price\": 3.42},\n \"success\": true\n}\n```\n\n### check_availability\n\nCheck stock levels for multiple parts at once.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `part_numbers` | `list[str]` | required | List of part numbers to check |\n| `quantities` | `list[int] \\| None` | `None` | Quantities needed per part (defaults to 1 each) |\n\n**Returns:**\n\n```json\n{\n \"parts\": [\"STM32F411CEU6\", \"LM1117-3.3\"],\n \"quantities\": [100, 100],\n \"availability\": [\n {\n \"part_number\": \"STM32F411CEU6\",\n \"quantity_needed\": 100,\n \"available\": true,\n \"total_stock\": 15000,\n \"in_stock_suppliers\": 3,\n \"manufacturer\": \"STMicroelectronics\",\n \"description\": \"ARM Cortex-M4 MCU\"\n }\n ],\n \"all_available\": true,\n \"success\": true\n}\n```\n\n### find_alternatives\n\nFind drop-in or compatible replacement parts.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `part_number` | `str` | required | Original part number |\n| `parameters` | `dict \\| None` | `None` | Key parameters to match (narrows alternatives) |\n\n**Returns:**\n\n```json\n{\n \"original_part\": \"STM32F411CEU6\",\n \"match_parameters\": {\"core\": \"ARM Cortex-M4\"},\n \"alternatives\": [\n {\n \"part_number\": \"STM32F401CEU6\",\n \"manufacturer\": \"STMicroelectronics\",\n \"compatibility\": \"pin-compatible\",\n \"differences\": [\"256KB Flash vs 512KB\"]\n }\n ],\n \"total_alternatives\": 8,\n \"success\": true\n}\n```\n\n### calculate_bom_cost\n\nCalculate total cost for a complete bill of materials.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `bom` | `list[dict]` | required | BOM items (see format below) |\n| `quantity` | `int` | `1` | Number of boards/assemblies |\n| `preferred_suppliers` | `list[str] \\| None` | `None` | Preferred supplier list |\n\n**BOM item format:**\n\n```json\n{\n \"part_number\": \"STM32F411CEU6\",\n \"quantity\": 1,\n \"reference\": \"U1\",\n \"description\": \"Main MCU\"\n}\n```\n\nFields `part_number` (or `mpn`) and `quantity` are required. `reference` and `description`/`value` are optional.\n\n**Returns:**\n\n```json\n{\n \"bom_items\": 25,\n \"priced_items\": 23,\n \"quantity\": 100,\n \"total_cost\": 1250.00,\n \"cost_breakdown\": [\n {\n \"reference\": \"U1\",\n \"part_number\": \"STM32F411CEU6\",\n \"description\": \"Main MCU\",\n \"quantity\": 100,\n \"unit_price\": 3.42,\n \"line_total\": 342.00,\n \"supplier\": \"LCSC\",\n \"sku\": \"C478234\"\n }\n ],\n \"errors\": [{\"part_number\": \"CUSTOM-PART-1\", \"error\": \"not found\"}],\n \"currency\": \"USD\",\n \"success\": true\n}\n```\n\n### estimate_cost\n\nQuick cost estimate without full BOM processing.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `parts` | `list[dict]` | required | Parts list with `part_number` and `quantity` |\n| `currency` | `str` | `\"USD\"` | Currency code |\n\n## Manufacturing Tools\n\n### upload_bom\n\nUpload a BOM file for processing and part matching. **Local mode only.**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `file_path` | `str` | required | Local path to BOM file |\n\nSupported formats: CSV, XLSX, XLS, JSON, XML. Max size: 50 MB.\n\n**Returns:** `job_id` for polling with `check_bom_status`.\n\n### check_bom_status\n\nPoll BOM processing status. When complete, returns matched and unmatched parts.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `job_id` | `str` | required | Job ID from `upload_bom` |\n\n**Returns (when complete):**\n\n```json\n{\n \"job_id\": \"bom_abc123\",\n \"status\": \"complete\",\n \"bom_id\": \"bom_def456\",\n \"summary\": {\"total_lines\": 25, \"matched\": 23, \"unmatched\": 2},\n \"matched_parts\": [...],\n \"unknown_parts\": [\n {\n \"reference\": \"U3\",\n \"value\": \"CUSTOM-IC\",\n \"footprint\": \"QFP-48\",\n \"manufacturer\": \"\",\n \"mpn\": \"\",\n \"status\": \"unmatched\"\n }\n ],\n \"success\": true\n}\n```\n\n### submit_dfm\n\nQueue a Design for Manufacturability analysis.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_id` | `str` | required | Project ID to analyze |\n| `bom_id` | `str \\| None` | `None` | BOM ID to include |\n| `revision` | `str \\| None` | `None` | Revision identifier |\n| `notes` | `str \\| None` | `None` | Analysis notes |\n| `priority` | `str` | `\"normal\"` | `\"low\"`, `\"normal\"`, or `\"high\"` |\n\n### check_dfm_status\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `job_id` | `str` | required | Job ID from `submit_dfm` |\n\n### quote_fabrication\n\nGet a PCB fabrication quote.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_id` | `str` | required | Project ID |\n| `quantity` | `int` | `5` | Number of boards |\n| `layers` | `int` | `2` | PCB layer count |\n| `thickness` | `float` | `1.6` | Board thickness in mm |\n| `surface_finish` | `str` | `\"HASL\"` | Surface finish (HASL, ENIG, OSP) |\n| `color` | `str` | `\"green\"` | Solder mask color (green, red, blue, black, white, yellow) |\n| `priority` | `str` | `\"normal\"` | `\"low\"`, `\"normal\"`, or `\"high\"` |\n\n### upload_gerbers_for_quote\n\nUpload gerber zip for fab quoting. **Local mode only.** Same parameters as `quote_fabrication` except uses `file_path` instead of `project_id`.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `file_path` | `str` | required | Path to gerber zip file |\n| `quantity` | `int` | `5` | Number of boards |\n| `layers` | `int` | `2` | PCB layer count |\n| `thickness` | `float` | `1.6` | Board thickness in mm |\n| `surface_finish` | `str` | `\"HASL\"` | Surface finish |\n| `color` | `str` | `\"green\"` | Solder mask color |\n| `priority` | `str` | `\"normal\"` | Priority level |\n\n### quote_assembly\n\nCombined fabrication + assembly quote. **Local mode only.** Uploads gerbers for fab and BOM for assembly costing. Polls BOM status internally to get `bom_id`, then calculates COGS.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `gerber_path` | `str` | required | Path to gerber zip |\n| `bom_path` | `str` | required | Path to BOM file |\n| `quantity` | `int` | `5` | Number of assemblies |\n| `layers` | `int` | `2` | PCB layer count |\n| `thickness` | `float` | `1.6` | Board thickness in mm |\n| `surface_finish` | `str` | `\"HASL\"` | Surface finish |\n| `color` | `str` | `\"green\"` | Solder mask color |\n| `priority` | `str` | `\"normal\"` | Priority level |\n\n### check_manufacturing_status\n\nPoll status of any manufacturing job (fab, DFM, AOI, QC).\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `job_id` | `str` | required | Job ID from any manufacturing submission |\n\n## Datasheet Tools\n\n### list_datasheet_sections\n\nLightweight TOC extraction. Returns section titles and page numbers without reading content.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `file_path` | `str \\| None` | `None` | Local PDF path (local mode) |\n| `sku` | `str \\| None` | `None` | Part SKU for cached data |\n\nOne of `file_path` or `sku` is required.\n\n**Returns:**\n\n```json\n{\n \"source\": \"STM32F411CEU6\",\n \"total_pages\": 196,\n \"sections\": [\n {\"title\": \"Features\", \"page\": 1},\n {\"title\": \"Electrical Characteristics\", \"page\": 42},\n {\"title\": \"Absolute Maximum Ratings\", \"page\": 55}\n ],\n \"section_count\": 28,\n \"success\": true\n}\n```\n\n### read_datasheet\n\nRead and chunk a datasheet PDF with optional keyword filtering.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `file_path` | `str \\| None` | `None` | Local PDF path |\n| `sku` | `str \\| None` | `None` | Part SKU for cached chunks |\n| `query` | `str \\| None` | `None` | Keywords to filter chunks (e.g., \"maximum input voltage\") |\n| `chunk_pages` | `int` | `5` | Pages per chunk |\n\nOne of `file_path` or `sku` is required.\n\n**Returns:**\n\n```json\n{\n \"source\": \"STM32F411CEU6\",\n \"total_pages\": 196,\n \"method\": \"cached\",\n \"toc\": [{\"title\": \"Features\", \"page\": 1}],\n \"chunks\": [\n {\n \"text\": \"...chunk content...\",\n \"start_page\": 51,\n \"end_page\": 55\n }\n ],\n \"query\": \"maximum input voltage\",\n \"context_savings\": {\n \"total_chunks\": 40,\n \"returned_chunks\": 3,\n \"total_chars\": 250000,\n \"returned_chars\": 18000,\n \"reduction_pct\": 92.8\n },\n \"success\": true\n}\n```\n\n## KiCad Tools (Local Mode Only)\n\n### find_kicad_projects\n\nDiscover `.kicad_pro` files in configured search paths. No parameters.\n\n**Returns:**\n\n```json\n{\n \"projects\": [\n {\n \"name\": \"my-board\",\n \"path\": \"/home/user/projects/my-board/my-board.kicad_pro\",\n \"modified\": \"2026-03-10T14:30:00\"\n }\n ],\n \"total\": 5,\n \"search_paths\": [\"/home/user/projects\"]\n}\n```\n\n### extract_bom_from_kicad\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_path` | `str` | required | Path to `.kicad_pro` file |\n\n### analyze_kicad_project\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_path` | `str` | required | Path to `.kicad_pro` file |\n\nReturns file counts: total, schematics, PCBs, data files.\n\n### extract_netlist_from_project\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_path` | `str` | required | Path to `.kicad_pro` file |\n\n### match_components_to_parts\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `components` | `list[dict]` | required | Components from KiCad BOM extraction |\n| `auto_search` | `bool` | `True` | Auto-search for matching parts |\n\n### export_parts_to_kicad\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `parts` | `list[dict]` | required | Parts with: `reference`, `value`, `footprint`, `manufacturer`, `part_number`, `supplier`, `quantity`, `unit_price` |\n| `output_path` | `str` | required | Output file path |\n| `format` | `str` | `\"csv\"` | `\"csv\"` or `\"json\"` |\n\n### highlight_net_traces\n\nRender highlighted net traces as vector PDFs.\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_path` | `str` | required | Path to `.kicad_pro` or `.kicad_pcb` |\n| `net_names` | `list[str]` | required | Net names to highlight |\n| `colors` | `dict \\| None` | `None` | Color mapping `{\"net_name\": \"#rrggbb\"}` |\n| `mode` | `str` | `\"both\"` | `\"overlay\"`, `\"traces_only\"`, or `\"both\"` |\n| `output_dir` | `str \\| None` | `None` | Output directory (defaults to project dir) |\n\n### open_in_kicad\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `project_path` | `str` | required | Path to KiCad project file |\n\n## Identification Tools\n\n### identify_pcb\n\nIdentify PCB or component from a photo. **Local mode only.**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `file_path` | `str` | required | Image path (jpg, jpeg, png, gif, heic, webp) |\n| `project_id` | `str \\| None` | `None` | Project to associate |\n| `box_id` | `str \\| None` | `None` | Box/shipment to associate |\n\n### check_identification_status\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `job_id` | `str` | required | Job ID from `identify_pcb` |\n\n### get_identified_item\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `short_code` | `str` | required | Item short code (e.g., `SP-XXXXXX`) |\n\n## Error Patterns\n\nAll tools return `{\"success\": false, \"error\": \"message\"}` on failure. Common errors:\n\n| Error | Cause | Resolution |\n|-------|-------|------------|\n| `\"API key not configured\"` | Missing `SOURCE_PARTS_API_KEY` | Set env var |\n| `\"File not found\"` | Invalid `file_path` | Check path exists |\n| `\"Unsupported file format\"` | Wrong BOM extension | Use CSV, XLSX, XLS, JSON, or XML |\n| `\"File too large\"` | Exceeds 50 MB limit | Split or compress file |\n| `\"Part not found\"` | No match for part number | Try broader search or check spelling |\n| `\"KiCad CLI not found\"` | `kicad-cli` not in PATH | Install KiCad 8+ |\n","content/sourceparts/docs/parts-mcp/python/references/workflows.md":"# Parts MCP — Workflow Recipes\n\nMulti-step workflows for common electronic sourcing tasks.\n\n## End-to-End PCB Sourcing (KiCad to Priced BOM)\n\nComplete workflow from a KiCad project to a fully priced bill of materials.\n\n```\n1. find_kicad_projects()\n → Pick project from results\n\n2. extract_bom_from_kicad(project_path=\"...\")\n → Get component list from the project\n\n3. match_components_to_parts(components=bom_components, auto_search=True)\n → Each component matched to real supplier parts\n\n4. For unmatched components:\n search_parts(query=component_value)\n → Manual search with value/footprint keywords\n\n5. check_availability(part_numbers=[all_matched_mpns], quantities=[all_quantities])\n → Verify everything is in stock\n\n6. For unavailable parts:\n find_alternatives(part_number=unavailable_mpn)\n → Get drop-in replacements\n\n7. calculate_bom_cost(bom=final_parts_list, quantity=100)\n → Total cost at production quantity\n\n8. export_parts_to_kicad(parts=sourced_parts, output_path=\"sourced_bom.csv\")\n → Export back for KiCad integration\n```\n\n## Obsolete Part Replacement\n\nWhen a part is end-of-life or unavailable, find and validate a replacement.\n\n```\n1. get_part_details(part_number=\"OBSOLETE-PART\")\n → Get specifications of the original part\n\n2. find_alternatives(\n part_number=\"OBSOLETE-PART\",\n parameters={\"package\": \"TSSOP-20\", \"voltage\": \"3.3V\"}\n )\n → Find parts matching critical parameters\n\n3. For each alternative:\n get_part_details(part_number=alternative_mpn)\n → Verify specs match requirements\n\n4. compare_prices(part_number=best_alternative, quantity=needed_qty)\n → Check pricing for the replacement\n\n5. check_availability(part_numbers=[best_alternative], quantities=[needed_qty])\n → Confirm stock levels\n```\n\n## Multi-Board Assembly Quoting\n\nGet combined fab + assembly quotes for a multi-board product.\n\n```\nFor each board:\n\n1. quote_assembly(\n gerber_path=\"/path/to/board_gerbers.zip\",\n bom_path=\"/path/to/board_bom.csv\",\n quantity=100,\n layers=4,\n surface_finish=\"ENIG\"\n )\n → Returns fab_job_id, bom_job_id\n\n2. check_manufacturing_status(job_id=fab_job_id)\n → Poll until fab quote ready\n\n3. check_bom_status(job_id=bom_job_id)\n → Poll until BOM processed, note unmatched parts\n\n4. For unmatched parts:\n search_parts(query=part_description)\n → Resolve manually\n\nAggregate COGS across all boards for total assembly cost.\n```\n\n## Parametric Search with Filtering\n\nFind parts matching specific electrical parameters.\n\n```\n1. search_by_parameters(\n parameters={\n \"capacitance\": \"100nF\",\n \"voltage_rating\": \"50V\",\n \"package\": \"0402\",\n \"dielectric\": \"X7R\"\n },\n category=\"capacitor\"\n )\n → Parts matching all criteria\n\n2. For top candidates:\n compare_prices(part_number=candidate, quantity=1000)\n → Price at volume\n\n3. check_availability(part_numbers=top_3_candidates, quantities=[1000, 1000, 1000])\n → Stock check across candidates\n\n4. Pick the best available option by price and stock.\n```\n\n## Datasheet Deep-Dive\n\nExtract specific parameters from a datasheet efficiently.\n\n```\n1. list_datasheet_sections(sku=\"LM1117-3.3\")\n → Get table of contents with page numbers\n → Identify relevant sections (e.g., \"Electrical Characteristics\" on page 5)\n\n2. read_datasheet(\n sku=\"LM1117-3.3\",\n query=\"dropout voltage output current thermal\",\n chunk_pages=5\n )\n → Returns only chunks matching keywords\n → context_savings shows how much context was saved\n\n3. If you need a specific section not returned by query:\n read_datasheet(\n sku=\"LM1117-3.3\",\n query=\"absolute maximum ratings\",\n chunk_pages=3\n )\n → Targeted follow-up read\n```\n\n## DFM and Fabrication Pipeline\n\nFull manufacturing preparation workflow.\n\n```\n1. submit_dfm(project_id=\"proj_123\", bom_id=\"bom_456\", priority=\"high\")\n → Queue DFM analysis\n\n2. check_dfm_status(job_id=dfm_job_id)\n → Poll until complete\n → Review issues and warnings\n\n3. If DFM passes:\n quote_fabrication(\n project_id=\"proj_123\",\n quantity=10,\n layers=2,\n surface_finish=\"HASL\",\n color=\"green\"\n )\n → Get fab pricing\n\n4. check_manufacturing_status(job_id=fab_job_id)\n → Poll until quote ready\n\n5. For local gerber files:\n upload_gerbers_for_quote(\n file_path=\"/path/to/gerbers.zip\",\n quantity=10,\n layers=2\n )\n → Alternative: upload gerbers directly\n```\n\n## PCB Identification and Cataloging\n\nIdentify components on a physical PCB from photos.\n\n```\n1. identify_pcb(\n file_path=\"/path/to/pcb_photo.jpg\",\n project_id=\"proj_123\"\n )\n → Submits image for barcode/QR detection, OCR, component ID\n\n2. check_identification_status(job_id=id_job_id)\n → Poll until identification complete\n\n3. For each identified item:\n get_identified_item(short_code=\"SP-ABC123\")\n → Get full details including barcodes, OCR text, metadata\n\n4. For identified part numbers:\n get_part_details(part_number=identified_mpn)\n → Cross-reference with parts database\n```\n\n## BOM File Upload and Review\n\nUpload a BOM from any supported EDA tool and review matching results.\n\n```\n1. upload_bom(file_path=\"/path/to/altium_bom.xlsx\")\n → Accepts CSV, XLSX, XLS, JSON, XML\n → Supports KiCad, Altium, Eagle, PADS, Fusion 360, Protel 99\n\n2. check_bom_status(job_id=bom_job_id)\n → Poll until \"complete\"\n → Returns matched_parts and unknown_parts\n\n3. For each unknown part:\n search_parts(query=unknown[\"value\"] + \" \" + unknown[\"footprint\"])\n → Try to resolve manually\n\n4. calculate_bom_cost(\n bom=[\n {\"part_number\": p[\"mpn\"], \"quantity\": p[\"quantity\"]}\n for p in all_resolved_parts\n ],\n quantity=50,\n preferred_suppliers=[\"LCSC\", \"Mouser\"]\n )\n → Get total cost with supplier preferences\n```\n","content/sourceparts/skills/electronics-sourcing/SKILL.md":"---\nname: electronics-sourcing\ndescription: \"Guide for AI agents to source electronic components using parts-mcp — tool sequencing, decision patterns, and multi-step workflows\"\nmetadata:\n revision: 1\n updated-on: \"2026-03-15\"\n source: maintainer\n tags: \"electronics,sourcing,bom,components,pcb,manufacturing,workflow\"\n---\n\n# Electronics Sourcing Skill\n\nThis skill teaches you how to source electronic components effectively using parts-mcp tools. It covers tool sequencing, decision patterns, and how to handle common scenarios.\n\n## When to Use Which Search Tool\n\n**`search_parts`** — Use when you have a part number, description, or general keywords. This is your default starting point.\n\n```\n\"Find me a 100nF capacitor\" → search_parts(query=\"100nF capacitor\")\n\"Look up STM32F411\" → search_parts(query=\"STM32F411\")\n```\n\n**`search_by_parameters`** — Use when you need to match specific electrical parameters within a category. More precise than keyword search.\n\n```\n\"I need a 50V 100nF X7R 0402 cap\" → search_by_parameters(\n parameters={\"capacitance\": \"100nF\", \"voltage\": \"50V\", \"dielectric\": \"X7R\", \"package\": \"0402\"},\n category=\"capacitor\"\n)\n```\n\n**Decision rule**: If the user gives you 3+ specific parameters, use `search_by_parameters`. For part numbers or general queries, use `search_parts`.\n\n## The Search → Price → Availability → Alternative Pattern\n\nThis is the core sourcing workflow. Follow this sequence:\n\n```\n1. SEARCH → Find the part\n2. DETAILS → Get full specifications (if needed for validation)\n3. PRICE → Compare across suppliers\n4. AVAILABILITY → Check stock levels\n5. ALTERNATIVE → Find replacements (only if price is too high or stock is insufficient)\n```\n\nDo not skip steps 3 and 4. A part that exists in the database may be out of stock or prohibitively expensive.\n\nWhen the user asks to \"source\" or \"find\" a part, they expect pricing and availability — not just search results. Always follow through to at least step 4.\n\n## BOM Processing Pattern\n\nBOM processing is asynchronous. Always follow this exact sequence:\n\n```\n1. upload_bom(file_path=path) → Get job_id\n2. check_bom_status(job_id=job_id) → Poll until status is \"complete\"\n3. Review matched_parts and unknown_parts\n4. For unmatched: search_parts() to resolve manually\n5. calculate_bom_cost() with all resolved parts\n```\n\n**Polling**: Call `check_bom_status` and check the `status` field. If `\"in_progress\"`, wait and poll again. Do not assume instant completion.\n\n**Handling unmatched parts**: When `unknown_parts` is not empty, try `search_parts` with the `value` and `footprint` fields as the query. If still unmatched, report them to the user — do not silently skip them.\n\n## Datasheet Reading Strategy\n\nDatasheets can be hundreds of pages. Always use this two-step approach:\n\n```\n1. list_datasheet_sections(sku=\"PART-NUMBER\")\n → See the table of contents\n → Identify which sections contain what you need\n\n2. read_datasheet(sku=\"PART-NUMBER\", query=\"relevant keywords\")\n → Read only matching chunks\n → Saves significant context window\n```\n\n**Never** call `read_datasheet` without a `query` parameter unless the datasheet is short (< 20 pages). An unfiltered read of a 200-page datasheet will consume excessive context.\n\n**Keyword tips**: Use specific technical terms. \"maximum input voltage absolute ratings\" is better than \"voltage\". Multiple keywords narrow the results.\n\n## Manufacturing Pipeline\n\nManufacturing operations (DFM, fab quoting, assembly) are all asynchronous:\n\n```\nsubmit_dfm() → check_dfm_status() # DFM analysis\nquote_fabrication() → check_manufacturing_status() # Fab quote\nquote_assembly() → polls internally # Combined fab + assembly\n```\n\n**DFM first**: Always run DFM analysis before requesting a fabrication quote. DFM may reveal issues that affect manufacturability or cost.\n\n**Assembly quotes** combine fabrication and BOM costing in one call. Use `quote_assembly` when the user wants a complete per-unit cost including both PCB fabrication and component assembly.\n\n## Handling Partial Failures\n\nReal-world sourcing often has partial results. Handle gracefully:\n\n**Some parts not found in BOM**:\n- Report the unmatched count clearly\n- Attempt `search_parts` for each unmatched part\n- If still unmatched, list them for the user with their reference designators and values\n- Calculate cost for matched parts, noting the incomplete total\n\n**Some parts unavailable**:\n- Report which parts are out of stock\n- Automatically call `find_alternatives` for each unavailable part\n- Present alternatives with their specifications and pricing\n- Let the user decide on substitutions\n\n**Price comparison with missing suppliers**:\n- Report how many suppliers were checked\n- Note if key suppliers (the user's preferred ones) returned no results\n- Present available pricing data without waiting for all suppliers\n\n## KiCad Project Workflow\n\nWhen working with KiCad projects:\n\n```\n1. find_kicad_projects() → Discover available projects\n2. analyze_kicad_project(project_path=...) → Understand project structure\n3. extract_bom_from_kicad(project_path=...) → Get component list\n4. match_components_to_parts(components=...) → Match to real parts\n5. Follow the BOM processing pattern above for unmatched components\n```\n\n**KiCad CLI requirement**: BOM extraction may invoke `kicad-cli` if no existing BOM file is found in the project. Ensure KiCad 8+ is installed.\n\n**Local mode only**: All KiCad tools require local mode (stdio transport). They are not available in hosted/HTTP mode.\n\n## Cost Optimization Tips\n\nWhen helping users optimize costs:\n\n1. **Check quantity breaks**: Call `compare_prices` with different quantities (1, 10, 100, 1000) to show price break curves\n2. **Suggest alternatives**: If a part is expensive, call `find_alternatives` and compare pricing\n3. **Preferred suppliers**: Use `preferred_suppliers` in `calculate_bom_cost` to bias toward the user's existing supplier relationships\n4. **Consolidation**: When multiple parts come from the same supplier, note the potential for consolidated shipping\n\n## Response Patterns\n\n**When a user asks to \"source a part\"**:\n→ `search_parts` → `get_part_details` → `compare_prices` → `check_availability`\nPresent: part details, best price, stock status, and any concerns.\n\n**When a user provides a BOM file**:\n→ `upload_bom` → poll `check_bom_status` → report matched/unmatched → `calculate_bom_cost`\nPresent: match rate, unmatched items, total cost, per-line breakdown.\n\n**When a user asks \"is this in stock?\"**:\n→ `check_availability` with quantities\nPresent: stock levels, number of suppliers with stock, whether quantity is meetable.\n\n**When a user asks about a datasheet**:\n→ `list_datasheet_sections` → `read_datasheet` with targeted query\nPresent: the specific information requested, citing page numbers.\n\n**When a user asks for a manufacturing quote**:\n→ `submit_dfm` (if not already done) → `quote_fabrication` or `quote_assembly`\nPresent: DFM issues (if any), estimated cost, lead time.\n","content/tavily/skills/tavily-best-practices/SKILL.md":"","content/tavily/skills/tavily-best-practices/references/crawl.md":"","content/tavily/skills/tavily-best-practices/references/extract.md":"","content/tavily/skills/tavily-best-practices/references/integrations.md":"","content/tavily/skills/tavily-best-practices/references/research.md":"","content/tavily/skills/tavily-best-practices/references/sdk.md":"","content/tavily/skills/tavily-best-practices/references/search.md":"","content/vectara/docs/api/python/references/advanced-query.md":"","content/vectara/docs/api/python/references/agents.md":"","content/vectara/docs/api/python/references/data-management.md":"","docs/design.md":"","llms.txt":"# Context Hub (chub)\n\n> CLI tool for searching and fetching LLM-optimized documentation and skills. Install: npm install -g @aisuite/chub\n\n## Content Types\n\n- doc: API/SDK reference documentation. Versioned, language-specific. Entry file: DOC.md\n- skill: Task recipes, automation patterns, coding playbooks. Standalone. Entry file: SKILL.md\n\n## Commands\n\n### chub search [query]\nSearch docs and skills. No query lists all entries.\n\nFlags:\n --tags Filter by tags (comma-separated)\n --json JSON output with { total, results[] }\n\nExamples:\n chub search # list everything\n chub search \"stripe\" # fuzzy search by name/description\n chub search stripe/payments # exact id returns full detail\n chub search --tags automation # filter by tag\n\n### chub get \nFetch one or more docs or skills by ID. Auto-detects type (doc vs skill). Auto-infers language when only one is available.\n\nFlags:\n --lang Language variant (js, py, ts, etc.)\n --version Specific doc version\n --full Fetch all files, not just the entry point\n --file Fetch specific file(s) by path (comma-separated)\n -o, --output Write to file or directory\n --json JSON output with { id, type, content, path, additionalFiles?, annotation? }\n\nExamples:\n chub get stripe/api # single doc (auto-infers lang if only one)\n chub get openai/chat-api --lang py # specific language\n chub get pw-community/login-flows # fetch a skill\n chub get stripe/api openai/chat-api # multiple entries\n chub get acme/widgets --file references/advanced.md # specific reference file\n chub get acme/widgets --full # all files\n chub get stripe/api -o .context/stripe.md # save to file\n\nBehavior:\n - When additional reference files exist beyond the entry point, output includes a footer listing them\n - Use --file to fetch specific reference files without fetching everything\n - If an annotation exists for the entry, it is appended after the content\n\n### chub annotate [id] [note]\nAttach persistent notes to a doc or skill. Notes appear on future `chub get` calls.\n\nFlags:\n --clear Remove annotation for this entry\n --list List all annotations\n --json JSON output\n\nExamples:\n chub annotate stripe/api \"Use idempotency keys for POST requests\" # set note\n chub annotate stripe/api # view current note\n chub annotate stripe/api \"Updated note\" # replaces previous note\n chub annotate stripe/api --clear # remove note\n chub annotate --list # list all annotations\n\nStorage: ~/.chub/annotations/ (local, per-machine)\n\n### chub feedback [id] [rating] [comment]\nRate a doc or skill (up/down). Sent to the registry for maintainers.\n\nFlags:\n --label