{"owner":"vibrantlabsai","repo":"ragas","hasSkills":true,"totalSkillsCount":29,"totalTokensCount":10899,"categories":["cursor-rule","claude-rule","command-prompt","anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":[".cursor/rules/project-structure.mdc","CLAUDE.md",".cursor/rules/docs-structure.mdc",".cursor/rules/docs-diataxis-guidelines.mdc",".cursor/commands/git-pr.md",".cursor/rules/use-uv-cli.mdc",".cursor/rules/update-guide.mdc",".cursor/commands/update-howto-guide.md",".claude/commands/create-pr.md","docs/references/aevaluate.md",".claude/commands/fix-github-issue.md","docs/references/cache.md","docs/references/embeddings.md","docs/references/evaluate.md","docs/references/evaluation_schema.md","docs/references/executor.md","docs/references/generate.md","docs/references/graph.md","docs/references/index.md","docs/references/integrations.md","docs/references/llms.md","docs/references/metrics.md","docs/references/optimizers.md","docs/references/prompt.md","docs/references/run_config.md","docs/references/synthesizers.md","docs/references/testset_schema.md","docs/references/tokenizers.md","docs/references/transforms.md"],"skills":{".cursor/rules/project-structure.mdc":"---\nalwaysApply: true\n---\n# Monorepo Project Structure\n\nThe repository is a monorepo consisting of two primary components:\n\n1. [/](./) – Core evaluation toolkit\n   • Source code lives in [src/](src/)\n   • Tests live in [tests/](tests/)\n   • Build configuration is in [pyproject.toml](pyproject.toml)\n\n2. [examples/](examples/) – Installable examples package\n   • Package: ragas_examples/ containing agent_evals, prompt_evals, rag_eval, workflow_eval, benchmark_llm\n   • Build configuration in [examples/pyproject.toml](examples/pyproject.toml)\n   • Shipped as `ragas-examples` package on PyPI via `ragas[examples]` extra\n   • Local development: `uv pip install -e . -e ./examples`\n   • Usage: `python -m ragas_examples.benchmark_llm.prompt`\n\nShared documentation for all projects is located under [docs/](docs/).\n\nRoot-level [Makefile](Makefile) provides combined commands, while each project directory also contains its own Makefile for project-specific tasks.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n",".cursor/rules/docs-structure.mdc":"---\nglobs: docs/**\n---\n# Documentation Structure & Workflow\n\nFollow these conventions when creating or editing documentation:\n\n1. **Docs live in [docs/](mdc:docs/)**\n   • Use Markdown (`.md`) files.  \n   • Images and other assets go in [docs/_static/](mdc:docs/_static/).\n\n2. **Section Folders mirror MkDocs navigation** (see [mkdocs.yml](mdc:mkdocs.yml)):\n   • 🚀 Get Started → [docs/getstarted/](mdc:docs/getstarted/)  \n   • 📚 Core Concepts → [docs/concepts/](mdc:docs/concepts/)  \n   • 🧪 Experimental → [docs/experimental/](mdc:docs/experimental/)  \n   • 🛠️ How-to Guides → [docs/howtos/](mdc:docs/howtos/)  \n   • 📖 References → [docs/references/](mdc:docs/references/)  \n   • Community → [docs/community/](mdc:docs/community/)\n\n   Place new pages in the appropriate folder **and** update `mkdocs.yml` `nav:` so the page appears in navigation.\n\n3. **Notebook-to-Markdown**\n   • Convert notebooks to Markdown with [docs/ipynb_to_md.py](mdc:docs/ipynb_to_md.py).  \n   • Commit the generated `.md`; notebooks themselves should not live in `docs/`.\n\n4. **Local preview / build**\n   • Run `make build-docs` to build HTML, `make serve-docs` to preview locally (defined in [DEVELOPMENT.md](mdc:DEVELOPMENT.md)).\n\n5. **Style & Assets**\n   • Use relative links (`../`) within docs.  \n   • Reference images via `_static/…` paths so they work in both dev and hosted docs.  \n   • Custom templates/CSS live in [docs/extra/](mdc:docs/extra/) — avoid editing `material` theme defaults directly.\n\n6. **API References (mkdocstrings)**\n   • Always use public API paths in `[ClassName][ragas.module.ClassName]` references.\n   • Check what's exported in `__init__.py` — if a class isn't in `__all__`, mkdocstrings can't link to it.\n   • Example: Use `[BasePrompt][ragas.prompt.BasePrompt]` not `[BasePrompt][ragas.prompt.base.BasePrompt]` or internal module paths.\n\n7. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.\n\n---\n\n# Formatting Guidelines\n\n- When introducing a list with text ending in a colon (e.g., \"This will:\"), always add a blank line before the first list item. \n- In a numbered list, do not add any new line between the items.",".cursor/rules/docs-diataxis-guidelines.mdc":"---\nglobs: docs/**\n---\n# Diátaxis Documentation Guidelines\n\nWhen writing or editing documentation, categorise each page as **one** of the four Diátaxis modes and follow its specific guidance. *Do not mix modes in a single page.*\n\n## 1. Tutorials  🧑‍🏫  (`docs/getstarted/`, `docs/experimental/tutorials/`)\n• Purpose: provide a structured **learning experience** – “Can you teach me to…?”  \n• Form: narrative lesson that leads the reader from zero to a working result.  \n• Include: context, motivation, complete working example.  \n• Avoid: deep technical detail; troubleshooting; exhaustive options.\n\n## 2. How-to Guides  🍳  (`docs/howtos/`)\n• Purpose: help the user **achieve a specific goal** – “How do I…?”  \n• Form: concise series of **step-by-step instructions** focused on the user’s project.  \n• Write from the **user’s perspective**, not the tool’s operations.  \n• Link to reference and explanation pages for background; keep prose minimal.\n• Add any code run outputs as expandable click blocks. Readers should be able to understand the guide without running the code.\n\n## 3. Reference  📑  (`docs/references/`)\n• Purpose: provide **neutral, complete, accurate description** of APIs, commands, options – “What is…?”  \n• Maintain consistent patterns (parameter tables, return types, examples).  \n• Avoid instruction or opinion; instead *link* to how-to or explanation pages.  \n• Examples are welcome if they illustrate usage without drifting into tutorial style.\n\n## 4. Explanation  💡  (`docs/concepts/`, `docs/experimental/core_concepts/`)\n• Purpose: **clarify concepts and rationale** – “Why…?”  \n• Form: discursive article that illuminates design decisions, theory, background.  \n• May link out to tutorials, how-tos, and reference, but does not instruct step-by-step.\n\n### Keep the Borders Sharp\n• Do **not** let content blur between modes (e.g., no instructions inside reference; no lengthy theory in how-tos).  \n• If a page starts serving two modes, split it.\n\n### Filing & Navigation\n• Place the file in the folder matching its mode (above).  \n• Update `mkdocs.yml` `nav:` under the corresponding section.\n\n### Incremental Improvement Cycle (per Diátaxis)\nChoose → Assess → Decide → Do. Focus on small, atomic upgrades rather than grand rewrites.\n\n### Writing Style\n• Use second-person (\"you\") and active voice.\n• Ensure code blocks are **copy-pasteable** and include necessary context (imports, environment).\n• Prefer short sentences; use Markdown admonitions (`!!! note`, `!!! warning`) sparingly for important side-information.\n• Use `??? \"Click to expand\"` collapsible admonitions to contain outputs, long prompts, verbose logs, or any content that would clutter the main article flow. This keeps the primary content scannable while preserving detailed information for readers who need it.\n• **Always add a blank line after text ending with a colon before starting a list.** This ensures proper Markdown rendering in MkDocs. Without the blank line, list items may render as continuation text instead of a proper bulleted/numbered list.\n\n### Cross-linking Between Modes\n• End tutorials with pointers to relevant how-to guides for further exploration.\n• How-to guides should include links to reference/API pages for deeper details.\n• Explanations can reference tutorials and how-tos to illustrate concepts in action.\n\n### Page Metadata & Prerequisites\n• Start each page with a one-sentence purpose statement and a brief list of prerequisites (libraries, data, environment variables).\n• Highlight any external services/configuration required before the reader begins.\n\n### Keep Pages Atomic\n• One page = one task, concept, or API surface. If content grows, **split** rather than creating a mega-guide.\n\n",".cursor/commands/git-pr.md":"Make sure you are on a branch other than main. \nCommit changes. \nRun make format. \nCreate PR using gh cli following .github/pull_request_template.md template\n",".cursor/rules/use-uv-cli.mdc":"---\nalwaysApply: true\n---\n# Use `uv run` for Python CLI commands\n\nThis repository manages its virtual environment and dependencies with **uv**. Therefore, always execute Python or Python-related CLI tools through `uv run`.\n\nExamples:\n- `uv run pytest`\n- `uv run ruff check .`\n- `uv run isort .`\n- `uv run pyright`\n",".cursor/rules/update-guide.mdc":"---\nalwaysApply: false\n---\nWe are writing a how to guide for Ragas docs and Ragas users.\n\nSo after any coding step we complete, or after any succesful runs, always update the guide with what was done. Make sure the content is concise and to the point. \n\nCurrent guide: docs/howtos/applications/evaluate-and-improve-rag.md",".cursor/commands/update-howto-guide.md":"# Update How-to Guide\n\nUpdates the mentioned how-to guide to use src/ragas/metrics/collections API instead of the legacy ragas/metrics API and LLM factory pattern instead of langchainwrapper.\n\n## File Format Decision\n\nIf the source is an `.ipynb` file (or if the `.md` filename starts with `_`, indicating it's derived from a notebook via `docs/ipynb_to_md.py`):\n\n1. **Delete** the `.ipynb` file\n2. **Delete** the corresponding `_xxx.md` file (if it exists)\n3. **Create** a new `.md` file directly (without the `_` prefix)\n\nThis simplifies maintenance by having pure markdown docs instead of notebooks.\n\n\n## Process\n\n### Phase 1: Research (do NOT make changes yet)\n\nRefer pr-description-customizations.md for the list of guides that are already updated. And finally update the doc after you're done. \n\n#### 1.1 Understand the Guide's Purpose\n- Read the target file thoroughly\n- Identify **what the guide is trying to achieve** (e.g., caching, run config, retry handling)\n- Note the specific use case or need the guide addresses\n- Understand what underlying tools/libraries are being used (e.g., instructor, liteLLM, httpx)\n\n#### 1.2 Feasibility Check\nBefore doing anything else, check if the feature works with the new API:\n\n1. **Check `src/ragas/experiment.py`** - Does experiment() support this feature?\n2. **Check `src/ragas/evaluation.py`** - Is this an evaluate()-only feature?\n3. **Check `src/ragas/metrics/collections/`** - Do collections metrics support this?\n4. **Check if simpler alternatives exist** - Does a newer, simpler API make this guide obsolete? (e.g., decorator-based metrics vs subclassing, built-in features vs manual workarounds). Check concept docs and `src/ragas/metrics/` for modern patterns.\n\n**If a simpler approach exists → recommend deletion** instead of migration. See \"When to Recommend Deletion\" section.\n\n**If not supported in new API → STOP immediately:**\n- Keep guide as-is\n- Output this Slack message for the team:\n\n```\n📋 *Doc Update Skipped*: `<guide_path>`\n*Link*: https://docs.ragas.io/en/latest/<guide_path_without_extension>/\n*Reason*: <feature> only works with legacy `evaluate()` API, not yet supported in `experiment()`/collections\n*Action*: Keep as-is until collections API adds support\n```\n\n**If supported → continue to 1.3**\n\n#### 1.3 Present Plan & Wait for Approval\n\n**⏸️ STOP HERE - Do NOT proceed to Phase 2 without explicit user approval.**\n\nPresent a clear summary:\n1. **Current state**: What the guide currently does and how\n2. **Proposed changes**: \n   - Imports to change (from old → new)\n   - LLM/embeddings setup patterns to update\n   - **How the specific use case/feature will be achieved** with the new API\n   - Any restructuring or content changes\n3. **Potential concerns**: Anything uncertain or risky\n4. **Ask**: \"Does this plan look good? Should I proceed?\"\n\n**Wait for user to say \"yes\", \"proceed\", \"go ahead\", or similar before continuing.**\n\n---\n\n### Phase 2: Execute (only after approval)\n\n#### 2.1 Apply Updates\n\n**Keep it Concise**: \n- Remove unnecessary explanations and verbose text\n- Focus on the essential information needed to achieve the goal\n- Use clear, direct language\n- Avoid redundant examples - one good example is better than multiple similar ones\n\n**Import Updates**:\n```python\n# Change from:\nfrom ragas.metrics import MetricName\n\n# To:\nfrom ragas.metrics.collections import MetricName\n```\n\n**LLM Setup**:\n```python\n# Change from:\nfrom langchain_openai import ChatOpenAI\nfrom ragas.llms import LangchainLLMWrapper\nllm = LangchainLLMWrapper(ChatOpenAI(model=\"gpt-4o\"))\n\n# To:\nfrom openai import OpenAI\nfrom ragas.llms import llm_factory\nclient = OpenAI(api_key=\"sk-...\")\nllm = llm_factory(\"gpt-4o\", client=client)\n```\n\n**Embeddings Setup**:\n```python\n# Change from:\nfrom langchain_openai import OpenAIEmbeddings\nfrom ragas.embeddings import LangchainEmbeddingsWrapper\nembeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())\n\n# To:\nfrom openai import OpenAI\nfrom ragas.embeddings.base import embedding_factory  # Use .base to avoid deprecation warning\nclient = OpenAI(api_key=\"sk-...\")\nembeddings = embedding_factory(\"openai\", model=\"text-embedding-3-small\", client=client)\n```\n\n**What to Fix**:\n- Update imports and LLM/embeddings patterns\n- Use `ragas.embeddings.base` import to avoid deprecation warnings\n- **Replace all legacy code with modern approaches** (no need to keep legacy sections)\n- Fix minor issues automatically\n- Don't restructure content unless fixing issues\n\n#### 2.2 Verify Accuracy & Test Code\n\n**Verify with Web Search**:\n- Search for official documentation of any libraries/tools mentioned (instructor, liteLLM, httpx, etc.)\n- Confirm API signatures, parameter names, and usage patterns are correct\n- Verify any claims about library behavior are accurate\n\n**Run the Code**:\n- Install any missing packages first: `uv pip install <package>`\n- Extract ALL Python code blocks from the guide\n- Save as `tests/docs/test_<guide_name>.py` (e.g., `test_run_config.py`)\n- Use `.env` from root for API keys. .env only has openai keys, if you need anything else, let me know\n- Run: `uv run python tests/docs/test_<guide_name>.py`\n- **Verify the original use case/goal is achieved** with the new approach\n\n**If tests fail**:\n1. Check the underlying implementation in `src/` to understand correct usage\n2. Fix the code in the guide based on what you learn from `src/`\n3. Re-run the test\n4. Repeat until tests pass\n5. If stuck after multiple attempts, report the issue with details\n\n**Keep the test file** - excluded from default `pytest` runs via `norecursedirs` in `pyproject.toml`.\n\n**Both verification methods are required** - web search for accuracy, code execution for functionality.\n\n#### 2.3 Check Navigation\n- Verify file is in `mkdocs.yml`\n- Note if location seems wrong or can be put in a more appropriate section\n\n#### 2.4 Summarize Changes\n- List all changes made\n- Mention if anything is not tested due to any reasons (like missing packages, missing API keys, etc.)\n\n## Notes\n- **Two-phase workflow**: Research first, get approval, then execute\n- **Never skip approval**: Always present the plan and wait for explicit \"go ahead\" before making changes\n- **This is not just a straightforward migration** - understand if the original goal is achievable first\n- **Keep guides concise** - remove fluff, focus on essential information\n- **Verify accuracy** - use web search to confirm library APIs and behavior before writing\n- **Test everything** - run all code examples before finalizing\n- Only fix what's broken or outdated\n- Check `src/` before updating to verify APIs exist\n- Don't add legacy sections\n- Use root `.env` for testing\n- **Keep test files** in `tests/docs/` - excluded from default pytest runs\n\n## When to Recommend Deletion\n\nIf a guide teaches **writing custom metrics by subclassing** (`MetricWithLLM`, `SingleTurnMetric`, etc.), it's likely obsolete. The decorator-based approach is simpler:\n\n```python\nfrom ragas.metrics import discrete_metric, numeric_metric, ranking_metric\n\n@discrete_metric(name=\"my_metric\", allowed_values=[\"pass\", \"fail\"])\ndef my_metric(response: str, context: str) -> str:\n    return \"pass\" if condition else \"fail\"\n```\n\nSee `docs/concepts/metrics/overview/index.md` for details. Recommend deletion if decorators cover the use case.\n\n## Reporting Gaps\nIf you identify a gap, use the Slack message template from section 1.2.\n",".claude/commands/create-pr.md":"<!--\nWHITELIST_AFTER_APPROVAL:\n- Bash(git push:*)\n- Bash(gh pr create:*)\n-->\n\nPlease create a pull request with the changes on the current branch. $ARGUMENTS\n\n**IMPORTANT: Use planning mode for this command.**\n\nFollow these steps:\n\n## Planning Phase (Research Only):\n1. Check git status to ensure working tree is clean\n2. Verify current branch is not main/master\n3. Check if current branch tracks a remote branch and is up to date\n4. Run `git log --oneline main..HEAD` to see commits that will be included in PR\n5. Run `git diff main...HEAD` to understand the full scope of changes\n6. Analyze all changes and commits to create a comprehensive PR summary\n7. Draft PR title and description with:\n   - Title: Clear, descriptive title based on the changes\n   - Body: Include \"## Summary\" with bullet points, \"## Test plan\" checklist, and Claude signature\n8. **Present the draft PR title and description using ExitPlanMode tool for user approval**\n\n## Execution Phase (After User Approval):\n9. Push branch to remote with upstream tracking if needed\n10. Create PR using `gh pr create` with the approved title and description\n11. Return the PR URL for easy access\n\nUse the GitHub CLI (`gh`) for all GitHub-related operations. If $ARGUMENTS are provided, incorporate them as hints for the PR title or description.","docs/references/aevaluate.md":"# Async Evaluation\n\n## aevaluate()\n\n::: ragas.evaluation.aevaluate\n\n## Async Usage\n\nRagas provides both synchronous and asynchronous evaluation APIs to accommodate different use cases:\n\n### Using aevaluate() (Recommended for Production)\n\nFor production async applications, use `aevaluate()` to avoid event loop conflicts:\n\n```python\nimport asyncio\nfrom ragas import aevaluate\n\nasync def evaluate_app():\n    result = await aevaluate(dataset, metrics)\n    return result\n\n# In your async application\nresult = await evaluate_app()\n```\n\n### Using evaluate() with Async Control\n\nFor backward compatibility and Jupyter notebook usage, `evaluate()` provides optional control over `nest_asyncio`:\n\n```python\n# Default behavior (Jupyter-compatible)\nresult = evaluate(dataset, metrics)  # allow_nest_asyncio=True\n\n# Production-safe (avoids event loop patching)\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n\n### Migration from nest_asyncio Issues\n\nIf you're experiencing issues with `nest_asyncio` in production:\n\n**Before (problematic):**\n```python\n# This may cause event loop conflicts\nresult = evaluate(dataset, metrics)\n```\n\n**After (fixed):**\n```python\n# Option 1: Use async API\nresult = await aevaluate(dataset, metrics)\n\n# Option 2: Disable nest_asyncio\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n",".claude/commands/fix-github-issue.md":"Please analyze and fix the GitHub issue: $ARGUMENTS.\n\nFollow these steps:\n\n1. Use `gh issue view` to get the issue details\n2. Understand the problem described in the issue\n3. Search the codebase for relevant files\n4. Implement the necessary changes to fix the issue\n5. Write and run tests to verify the fix\n6. Ensure code passes linting and type checking\n7. Create a descriptive commit message\n8. Push and create a PR\n\nRemember to use the GitHub CLI (`gh`) for all GitHub-related tasks.\n","docs/references/cache.md":"::: ragas.cache\n    options:\n        members_order: \"source\"\n","docs/references/embeddings.md":"# References Skill\n\nPath: `docs/references/embeddings.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/embeddings.md)","docs/references/evaluate.md":"# References Skill\n\nPath: `docs/references/evaluate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluate.md)","docs/references/evaluation_schema.md":"# References Skill\n\nPath: `docs/references/evaluation_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluation_schema.md)","docs/references/executor.md":"# References Skill\n\nPath: `docs/references/executor.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/executor.md)","docs/references/generate.md":"# References Skill\n\nPath: `docs/references/generate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/generate.md)","docs/references/graph.md":"# References Skill\n\nPath: `docs/references/graph.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/graph.md)","docs/references/index.md":"# References Skill\n\nPath: `docs/references/index.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/index.md)","docs/references/integrations.md":"# References Skill\n\nPath: `docs/references/integrations.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/integrations.md)","docs/references/llms.md":"# References Skill\n\nPath: `docs/references/llms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/llms.md)","docs/references/metrics.md":"# References Skill\n\nPath: `docs/references/metrics.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/metrics.md)","docs/references/optimizers.md":"# References Skill\n\nPath: `docs/references/optimizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/optimizers.md)","docs/references/prompt.md":"# References Skill\n\nPath: `docs/references/prompt.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/prompt.md)","docs/references/run_config.md":"# References Skill\n\nPath: `docs/references/run_config.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/run_config.md)","docs/references/synthesizers.md":"# References Skill\n\nPath: `docs/references/synthesizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/synthesizers.md)","docs/references/testset_schema.md":"# References Skill\n\nPath: `docs/references/testset_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/testset_schema.md)","docs/references/tokenizers.md":"# References Skill\n\nPath: `docs/references/tokenizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/tokenizers.md)","docs/references/transforms.md":"# References Skill\n\nPath: `docs/references/transforms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/transforms.md)"},"files":{".cursor/rules/project-structure.mdc":"---\nalwaysApply: true\n---\n# Monorepo Project Structure\n\nThe repository is a monorepo consisting of two primary components:\n\n1. [/](./) – Core evaluation toolkit\n   • Source code lives in [src/](src/)\n   • Tests live in [tests/](tests/)\n   • Build configuration is in [pyproject.toml](pyproject.toml)\n\n2. [examples/](examples/) – Installable examples package\n   • Package: ragas_examples/ containing agent_evals, prompt_evals, rag_eval, workflow_eval, benchmark_llm\n   • Build configuration in [examples/pyproject.toml](examples/pyproject.toml)\n   • Shipped as `ragas-examples` package on PyPI via `ragas[examples]` extra\n   • Local development: `uv pip install -e . -e ./examples`\n   • Usage: `python -m ragas_examples.benchmark_llm.prompt`\n\nShared documentation for all projects is located under [docs/](docs/).\n\nRoot-level [Makefile](Makefile) provides combined commands, while each project directory also contains its own Makefile for project-specific tasks.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n",".cursor/rules/docs-structure.mdc":"---\nglobs: docs/**\n---\n# Documentation Structure & Workflow\n\nFollow these conventions when creating or editing documentation:\n\n1. **Docs live in [docs/](mdc:docs/)**\n   • Use Markdown (`.md`) files.  \n   • Images and other assets go in [docs/_static/](mdc:docs/_static/).\n\n2. **Section Folders mirror MkDocs navigation** (see [mkdocs.yml](mdc:mkdocs.yml)):\n   • 🚀 Get Started → [docs/getstarted/](mdc:docs/getstarted/)  \n   • 📚 Core Concepts → [docs/concepts/](mdc:docs/concepts/)  \n   • 🧪 Experimental → [docs/experimental/](mdc:docs/experimental/)  \n   • 🛠️ How-to Guides → [docs/howtos/](mdc:docs/howtos/)  \n   • 📖 References → [docs/references/](mdc:docs/references/)  \n   • Community → [docs/community/](mdc:docs/community/)\n\n   Place new pages in the appropriate folder **and** update `mkdocs.yml` `nav:` so the page appears in navigation.\n\n3. **Notebook-to-Markdown**\n   • Convert notebooks to Markdown with [docs/ipynb_to_md.py](mdc:docs/ipynb_to_md.py).  \n   • Commit the generated `.md`; notebooks themselves should not live in `docs/`.\n\n4. **Local preview / build**\n   • Run `make build-docs` to build HTML, `make serve-docs` to preview locally (defined in [DEVELOPMENT.md](mdc:DEVELOPMENT.md)).\n\n5. **Style & Assets**\n   • Use relative links (`../`) within docs.  \n   • Reference images via `_static/…` paths so they work in both dev and hosted docs.  \n   • Custom templates/CSS live in [docs/extra/](mdc:docs/extra/) — avoid editing `material` theme defaults directly.\n\n6. **API References (mkdocstrings)**\n   • Always use public API paths in `[ClassName][ragas.module.ClassName]` references.\n   • Check what's exported in `__init__.py` — if a class isn't in `__all__`, mkdocstrings can't link to it.\n   • Example: Use `[BasePrompt][ragas.prompt.BasePrompt]` not `[BasePrompt][ragas.prompt.base.BasePrompt]` or internal module paths.\n\n7. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.\n\n---\n\n# Formatting Guidelines\n\n- When introducing a list with text ending in a colon (e.g., \"This will:\"), always add a blank line before the first list item. \n- In a numbered list, do not add any new line between the items.",".cursor/rules/docs-diataxis-guidelines.mdc":"---\nglobs: docs/**\n---\n# Diátaxis Documentation Guidelines\n\nWhen writing or editing documentation, categorise each page as **one** of the four Diátaxis modes and follow its specific guidance. *Do not mix modes in a single page.*\n\n## 1. Tutorials  🧑‍🏫  (`docs/getstarted/`, `docs/experimental/tutorials/`)\n• Purpose: provide a structured **learning experience** – “Can you teach me to…?”  \n• Form: narrative lesson that leads the reader from zero to a working result.  \n• Include: context, motivation, complete working example.  \n• Avoid: deep technical detail; troubleshooting; exhaustive options.\n\n## 2. How-to Guides  🍳  (`docs/howtos/`)\n• Purpose: help the user **achieve a specific goal** – “How do I…?”  \n• Form: concise series of **step-by-step instructions** focused on the user’s project.  \n• Write from the **user’s perspective**, not the tool’s operations.  \n• Link to reference and explanation pages for background; keep prose minimal.\n• Add any code run outputs as expandable click blocks. Readers should be able to understand the guide without running the code.\n\n## 3. Reference  📑  (`docs/references/`)\n• Purpose: provide **neutral, complete, accurate description** of APIs, commands, options – “What is…?”  \n• Maintain consistent patterns (parameter tables, return types, examples).  \n• Avoid instruction or opinion; instead *link* to how-to or explanation pages.  \n• Examples are welcome if they illustrate usage without drifting into tutorial style.\n\n## 4. Explanation  💡  (`docs/concepts/`, `docs/experimental/core_concepts/`)\n• Purpose: **clarify concepts and rationale** – “Why…?”  \n• Form: discursive article that illuminates design decisions, theory, background.  \n• May link out to tutorials, how-tos, and reference, but does not instruct step-by-step.\n\n### Keep the Borders Sharp\n• Do **not** let content blur between modes (e.g., no instructions inside reference; no lengthy theory in how-tos).  \n• If a page starts serving two modes, split it.\n\n### Filing & Navigation\n• Place the file in the folder matching its mode (above).  \n• Update `mkdocs.yml` `nav:` under the corresponding section.\n\n### Incremental Improvement Cycle (per Diátaxis)\nChoose → Assess → Decide → Do. Focus on small, atomic upgrades rather than grand rewrites.\n\n### Writing Style\n• Use second-person (\"you\") and active voice.\n• Ensure code blocks are **copy-pasteable** and include necessary context (imports, environment).\n• Prefer short sentences; use Markdown admonitions (`!!! note`, `!!! warning`) sparingly for important side-information.\n• Use `??? \"Click to expand\"` collapsible admonitions to contain outputs, long prompts, verbose logs, or any content that would clutter the main article flow. This keeps the primary content scannable while preserving detailed information for readers who need it.\n• **Always add a blank line after text ending with a colon before starting a list.** This ensures proper Markdown rendering in MkDocs. Without the blank line, list items may render as continuation text instead of a proper bulleted/numbered list.\n\n### Cross-linking Between Modes\n• End tutorials with pointers to relevant how-to guides for further exploration.\n• How-to guides should include links to reference/API pages for deeper details.\n• Explanations can reference tutorials and how-tos to illustrate concepts in action.\n\n### Page Metadata & Prerequisites\n• Start each page with a one-sentence purpose statement and a brief list of prerequisites (libraries, data, environment variables).\n• Highlight any external services/configuration required before the reader begins.\n\n### Keep Pages Atomic\n• One page = one task, concept, or API surface. If content grows, **split** rather than creating a mega-guide.\n\n",".cursor/commands/git-pr.md":"Make sure you are on a branch other than main. \nCommit changes. \nRun make format. \nCreate PR using gh cli following .github/pull_request_template.md template\n",".cursor/rules/use-uv-cli.mdc":"---\nalwaysApply: true\n---\n# Use `uv run` for Python CLI commands\n\nThis repository manages its virtual environment and dependencies with **uv**. Therefore, always execute Python or Python-related CLI tools through `uv run`.\n\nExamples:\n- `uv run pytest`\n- `uv run ruff check .`\n- `uv run isort .`\n- `uv run pyright`\n",".cursor/rules/update-guide.mdc":"---\nalwaysApply: false\n---\nWe are writing a how to guide for Ragas docs and Ragas users.\n\nSo after any coding step we complete, or after any succesful runs, always update the guide with what was done. Make sure the content is concise and to the point. \n\nCurrent guide: docs/howtos/applications/evaluate-and-improve-rag.md",".cursor/commands/update-howto-guide.md":"# Update How-to Guide\n\nUpdates the mentioned how-to guide to use src/ragas/metrics/collections API instead of the legacy ragas/metrics API and LLM factory pattern instead of langchainwrapper.\n\n## File Format Decision\n\nIf the source is an `.ipynb` file (or if the `.md` filename starts with `_`, indicating it's derived from a notebook via `docs/ipynb_to_md.py`):\n\n1. **Delete** the `.ipynb` file\n2. **Delete** the corresponding `_xxx.md` file (if it exists)\n3. **Create** a new `.md` file directly (without the `_` prefix)\n\nThis simplifies maintenance by having pure markdown docs instead of notebooks.\n\n\n## Process\n\n### Phase 1: Research (do NOT make changes yet)\n\nRefer pr-description-customizations.md for the list of guides that are already updated. And finally update the doc after you're done. \n\n#### 1.1 Understand the Guide's Purpose\n- Read the target file thoroughly\n- Identify **what the guide is trying to achieve** (e.g., caching, run config, retry handling)\n- Note the specific use case or need the guide addresses\n- Understand what underlying tools/libraries are being used (e.g., instructor, liteLLM, httpx)\n\n#### 1.2 Feasibility Check\nBefore doing anything else, check if the feature works with the new API:\n\n1. **Check `src/ragas/experiment.py`** - Does experiment() support this feature?\n2. **Check `src/ragas/evaluation.py`** - Is this an evaluate()-only feature?\n3. **Check `src/ragas/metrics/collections/`** - Do collections metrics support this?\n4. **Check if simpler alternatives exist** - Does a newer, simpler API make this guide obsolete? (e.g., decorator-based metrics vs subclassing, built-in features vs manual workarounds). Check concept docs and `src/ragas/metrics/` for modern patterns.\n\n**If a simpler approach exists → recommend deletion** instead of migration. See \"When to Recommend Deletion\" section.\n\n**If not supported in new API → STOP immediately:**\n- Keep guide as-is\n- Output this Slack message for the team:\n\n```\n📋 *Doc Update Skipped*: `<guide_path>`\n*Link*: https://docs.ragas.io/en/latest/<guide_path_without_extension>/\n*Reason*: <feature> only works with legacy `evaluate()` API, not yet supported in `experiment()`/collections\n*Action*: Keep as-is until collections API adds support\n```\n\n**If supported → continue to 1.3**\n\n#### 1.3 Present Plan & Wait for Approval\n\n**⏸️ STOP HERE - Do NOT proceed to Phase 2 without explicit user approval.**\n\nPresent a clear summary:\n1. **Current state**: What the guide currently does and how\n2. **Proposed changes**: \n   - Imports to change (from old → new)\n   - LLM/embeddings setup patterns to update\n   - **How the specific use case/feature will be achieved** with the new API\n   - Any restructuring or content changes\n3. **Potential concerns**: Anything uncertain or risky\n4. **Ask**: \"Does this plan look good? Should I proceed?\"\n\n**Wait for user to say \"yes\", \"proceed\", \"go ahead\", or similar before continuing.**\n\n---\n\n### Phase 2: Execute (only after approval)\n\n#### 2.1 Apply Updates\n\n**Keep it Concise**: \n- Remove unnecessary explanations and verbose text\n- Focus on the essential information needed to achieve the goal\n- Use clear, direct language\n- Avoid redundant examples - one good example is better than multiple similar ones\n\n**Import Updates**:\n```python\n# Change from:\nfrom ragas.metrics import MetricName\n\n# To:\nfrom ragas.metrics.collections import MetricName\n```\n\n**LLM Setup**:\n```python\n# Change from:\nfrom langchain_openai import ChatOpenAI\nfrom ragas.llms import LangchainLLMWrapper\nllm = LangchainLLMWrapper(ChatOpenAI(model=\"gpt-4o\"))\n\n# To:\nfrom openai import OpenAI\nfrom ragas.llms import llm_factory\nclient = OpenAI(api_key=\"sk-...\")\nllm = llm_factory(\"gpt-4o\", client=client)\n```\n\n**Embeddings Setup**:\n```python\n# Change from:\nfrom langchain_openai import OpenAIEmbeddings\nfrom ragas.embeddings import LangchainEmbeddingsWrapper\nembeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())\n\n# To:\nfrom openai import OpenAI\nfrom ragas.embeddings.base import embedding_factory  # Use .base to avoid deprecation warning\nclient = OpenAI(api_key=\"sk-...\")\nembeddings = embedding_factory(\"openai\", model=\"text-embedding-3-small\", client=client)\n```\n\n**What to Fix**:\n- Update imports and LLM/embeddings patterns\n- Use `ragas.embeddings.base` import to avoid deprecation warnings\n- **Replace all legacy code with modern approaches** (no need to keep legacy sections)\n- Fix minor issues automatically\n- Don't restructure content unless fixing issues\n\n#### 2.2 Verify Accuracy & Test Code\n\n**Verify with Web Search**:\n- Search for official documentation of any libraries/tools mentioned (instructor, liteLLM, httpx, etc.)\n- Confirm API signatures, parameter names, and usage patterns are correct\n- Verify any claims about library behavior are accurate\n\n**Run the Code**:\n- Install any missing packages first: `uv pip install <package>`\n- Extract ALL Python code blocks from the guide\n- Save as `tests/docs/test_<guide_name>.py` (e.g., `test_run_config.py`)\n- Use `.env` from root for API keys. .env only has openai keys, if you need anything else, let me know\n- Run: `uv run python tests/docs/test_<guide_name>.py`\n- **Verify the original use case/goal is achieved** with the new approach\n\n**If tests fail**:\n1. Check the underlying implementation in `src/` to understand correct usage\n2. Fix the code in the guide based on what you learn from `src/`\n3. Re-run the test\n4. Repeat until tests pass\n5. If stuck after multiple attempts, report the issue with details\n\n**Keep the test file** - excluded from default `pytest` runs via `norecursedirs` in `pyproject.toml`.\n\n**Both verification methods are required** - web search for accuracy, code execution for functionality.\n\n#### 2.3 Check Navigation\n- Verify file is in `mkdocs.yml`\n- Note if location seems wrong or can be put in a more appropriate section\n\n#### 2.4 Summarize Changes\n- List all changes made\n- Mention if anything is not tested due to any reasons (like missing packages, missing API keys, etc.)\n\n## Notes\n- **Two-phase workflow**: Research first, get approval, then execute\n- **Never skip approval**: Always present the plan and wait for explicit \"go ahead\" before making changes\n- **This is not just a straightforward migration** - understand if the original goal is achievable first\n- **Keep guides concise** - remove fluff, focus on essential information\n- **Verify accuracy** - use web search to confirm library APIs and behavior before writing\n- **Test everything** - run all code examples before finalizing\n- Only fix what's broken or outdated\n- Check `src/` before updating to verify APIs exist\n- Don't add legacy sections\n- Use root `.env` for testing\n- **Keep test files** in `tests/docs/` - excluded from default pytest runs\n\n## When to Recommend Deletion\n\nIf a guide teaches **writing custom metrics by subclassing** (`MetricWithLLM`, `SingleTurnMetric`, etc.), it's likely obsolete. The decorator-based approach is simpler:\n\n```python\nfrom ragas.metrics import discrete_metric, numeric_metric, ranking_metric\n\n@discrete_metric(name=\"my_metric\", allowed_values=[\"pass\", \"fail\"])\ndef my_metric(response: str, context: str) -> str:\n    return \"pass\" if condition else \"fail\"\n```\n\nSee `docs/concepts/metrics/overview/index.md` for details. Recommend deletion if decorators cover the use case.\n\n## Reporting Gaps\nIf you identify a gap, use the Slack message template from section 1.2.\n",".claude/commands/create-pr.md":"<!--\nWHITELIST_AFTER_APPROVAL:\n- Bash(git push:*)\n- Bash(gh pr create:*)\n-->\n\nPlease create a pull request with the changes on the current branch. $ARGUMENTS\n\n**IMPORTANT: Use planning mode for this command.**\n\nFollow these steps:\n\n## Planning Phase (Research Only):\n1. Check git status to ensure working tree is clean\n2. Verify current branch is not main/master\n3. Check if current branch tracks a remote branch and is up to date\n4. Run `git log --oneline main..HEAD` to see commits that will be included in PR\n5. Run `git diff main...HEAD` to understand the full scope of changes\n6. Analyze all changes and commits to create a comprehensive PR summary\n7. Draft PR title and description with:\n   - Title: Clear, descriptive title based on the changes\n   - Body: Include \"## Summary\" with bullet points, \"## Test plan\" checklist, and Claude signature\n8. **Present the draft PR title and description using ExitPlanMode tool for user approval**\n\n## Execution Phase (After User Approval):\n9. Push branch to remote with upstream tracking if needed\n10. Create PR using `gh pr create` with the approved title and description\n11. Return the PR URL for easy access\n\nUse the GitHub CLI (`gh`) for all GitHub-related operations. If $ARGUMENTS are provided, incorporate them as hints for the PR title or description.","docs/references/aevaluate.md":"# Async Evaluation\n\n## aevaluate()\n\n::: ragas.evaluation.aevaluate\n\n## Async Usage\n\nRagas provides both synchronous and asynchronous evaluation APIs to accommodate different use cases:\n\n### Using aevaluate() (Recommended for Production)\n\nFor production async applications, use `aevaluate()` to avoid event loop conflicts:\n\n```python\nimport asyncio\nfrom ragas import aevaluate\n\nasync def evaluate_app():\n    result = await aevaluate(dataset, metrics)\n    return result\n\n# In your async application\nresult = await evaluate_app()\n```\n\n### Using evaluate() with Async Control\n\nFor backward compatibility and Jupyter notebook usage, `evaluate()` provides optional control over `nest_asyncio`:\n\n```python\n# Default behavior (Jupyter-compatible)\nresult = evaluate(dataset, metrics)  # allow_nest_asyncio=True\n\n# Production-safe (avoids event loop patching)\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n\n### Migration from nest_asyncio Issues\n\nIf you're experiencing issues with `nest_asyncio` in production:\n\n**Before (problematic):**\n```python\n# This may cause event loop conflicts\nresult = evaluate(dataset, metrics)\n```\n\n**After (fixed):**\n```python\n# Option 1: Use async API\nresult = await aevaluate(dataset, metrics)\n\n# Option 2: Disable nest_asyncio\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n",".claude/commands/fix-github-issue.md":"Please analyze and fix the GitHub issue: $ARGUMENTS.\n\nFollow these steps:\n\n1. Use `gh issue view` to get the issue details\n2. Understand the problem described in the issue\n3. Search the codebase for relevant files\n4. Implement the necessary changes to fix the issue\n5. Write and run tests to verify the fix\n6. Ensure code passes linting and type checking\n7. Create a descriptive commit message\n8. Push and create a PR\n\nRemember to use the GitHub CLI (`gh`) for all GitHub-related tasks.\n","docs/references/cache.md":"::: ragas.cache\n    options:\n        members_order: \"source\"\n","docs/references/embeddings.md":"# References Skill\n\nPath: `docs/references/embeddings.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/embeddings.md)","docs/references/evaluate.md":"# References Skill\n\nPath: `docs/references/evaluate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluate.md)","docs/references/evaluation_schema.md":"# References Skill\n\nPath: `docs/references/evaluation_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluation_schema.md)","docs/references/executor.md":"# References Skill\n\nPath: `docs/references/executor.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/executor.md)","docs/references/generate.md":"# References Skill\n\nPath: `docs/references/generate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/generate.md)","docs/references/graph.md":"# References Skill\n\nPath: `docs/references/graph.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/graph.md)","docs/references/index.md":"# References Skill\n\nPath: `docs/references/index.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/index.md)","docs/references/integrations.md":"# References Skill\n\nPath: `docs/references/integrations.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/integrations.md)","docs/references/llms.md":"# References Skill\n\nPath: `docs/references/llms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/llms.md)","docs/references/metrics.md":"# References Skill\n\nPath: `docs/references/metrics.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/metrics.md)","docs/references/optimizers.md":"# References Skill\n\nPath: `docs/references/optimizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/optimizers.md)","docs/references/prompt.md":"# References Skill\n\nPath: `docs/references/prompt.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/prompt.md)","docs/references/run_config.md":"# References Skill\n\nPath: `docs/references/run_config.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/run_config.md)","docs/references/synthesizers.md":"# References Skill\n\nPath: `docs/references/synthesizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/synthesizers.md)","docs/references/testset_schema.md":"# References Skill\n\nPath: `docs/references/testset_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/testset_schema.md)","docs/references/tokenizers.md":"# References Skill\n\nPath: `docs/references/tokenizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/tokenizers.md)","docs/references/transforms.md":"# References Skill\n\nPath: `docs/references/transforms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/transforms.md)"},"items":[{"name":"aevaluate.md","path":"docs/references/aevaluate.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/aevaluate.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Async Evaluation\n\n## aevaluate()\n\n::: ragas.evaluation.aevaluate\n\n## Async Usage\n\nRagas provides both synchronous and asynchronous evaluation APIs to accommodate different use cases:\n\n### Using aevaluate() (Recommended for Production)\n\nFor production async applications, use `aevaluate()` to avoid event loop conflicts:\n\n```python\nimport asyncio\nfrom ragas import aevaluate\n\nasync def evaluate_app():\n    result = await aevaluate(dataset, metrics)\n    return result\n\n# In your async application\nresult = await evaluate_app()\n```\n\n### Using evaluate() with Async Control\n\nFor backward compatibility and Jupyter notebook usage, `evaluate()` provides optional control over `nest_asyncio`:\n\n```python\n# Default behavior (Jupyter-compatible)\nresult = evaluate(dataset, metrics)  # allow_nest_asyncio=True\n\n# Production-safe (avoids event loop patching)\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n\n### Migration from nest_asyncio Issues\n\nIf you're experiencing issues with `nest_asyncio` in production:\n\n**Before (problematic):**\n```python\n# This may cause event loop conflicts\nresult = evaluate(dataset, metrics)\n```\n\n**After (fixed):**\n```python\n# Option 1: Use async API\nresult = await aevaluate(dataset, metrics)\n\n# Option 2: Disable nest_asyncio\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n","isInternal":false,"tokens":294,"sizeBytes":1336},{"name":"cache.md","path":"docs/references/cache.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/cache.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"::: ragas.cache\n    options:\n        members_order: \"source\"\n","isInternal":false,"tokens":16,"sizeBytes":61},{"name":"embeddings.md","path":"docs/references/embeddings.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/embeddings.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/embeddings.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/embeddings.md)","tokens":300,"sizeBytes":177},{"name":"evaluate.md","path":"docs/references/evaluate.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluate.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/evaluate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluate.md)","tokens":300,"sizeBytes":173},{"name":"evaluation_schema.md","path":"docs/references/evaluation_schema.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluation_schema.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/evaluation_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluation_schema.md)","tokens":300,"sizeBytes":191},{"name":"executor.md","path":"docs/references/executor.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/executor.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/executor.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/executor.md)","tokens":300,"sizeBytes":173},{"name":"generate.md","path":"docs/references/generate.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/generate.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/generate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/generate.md)","tokens":300,"sizeBytes":173},{"name":"graph.md","path":"docs/references/graph.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/graph.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/graph.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/graph.md)","tokens":300,"sizeBytes":167},{"name":"index.md","path":"docs/references/index.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/index.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/index.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/index.md)","tokens":300,"sizeBytes":167},{"name":"integrations.md","path":"docs/references/integrations.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/integrations.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/integrations.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/integrations.md)","tokens":300,"sizeBytes":181},{"name":"llms.md","path":"docs/references/llms.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/llms.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/llms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/llms.md)","tokens":300,"sizeBytes":165},{"name":"metrics.md","path":"docs/references/metrics.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/metrics.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/metrics.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/metrics.md)","tokens":300,"sizeBytes":171},{"name":"optimizers.md","path":"docs/references/optimizers.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/optimizers.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/optimizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/optimizers.md)","tokens":300,"sizeBytes":177},{"name":"prompt.md","path":"docs/references/prompt.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/prompt.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/prompt.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/prompt.md)","tokens":300,"sizeBytes":169},{"name":"run_config.md","path":"docs/references/run_config.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/run_config.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/run_config.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/run_config.md)","tokens":300,"sizeBytes":177},{"name":"synthesizers.md","path":"docs/references/synthesizers.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/synthesizers.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/synthesizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/synthesizers.md)","tokens":300,"sizeBytes":181},{"name":"testset_schema.md","path":"docs/references/testset_schema.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/testset_schema.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/testset_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/testset_schema.md)","tokens":300,"sizeBytes":185},{"name":"tokenizers.md","path":"docs/references/tokenizers.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/tokenizers.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/tokenizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/tokenizers.md)","tokens":300,"sizeBytes":177},{"name":"transforms.md","path":"docs/references/transforms.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/transforms.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `docs/references/transforms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/transforms.md)","tokens":300,"sizeBytes":177},{"name":"docs-diataxis-guidelines.mdc","path":".cursor/rules/docs-diataxis-guidelines.mdc","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/rules/docs-diataxis-guidelines.mdc","title":"Cursor Rule: docs-diataxis-guidelines","category":"cursor-rule","format":"mdc","content":"---\nglobs: docs/**\n---\n# Diátaxis Documentation Guidelines\n\nWhen writing or editing documentation, categorise each page as **one** of the four Diátaxis modes and follow its specific guidance. *Do not mix modes in a single page.*\n\n## 1. Tutorials  🧑‍🏫  (`docs/getstarted/`, `docs/experimental/tutorials/`)\n• Purpose: provide a structured **learning experience** – “Can you teach me to…?”  \n• Form: narrative lesson that leads the reader from zero to a working result.  \n• Include: context, motivation, complete working example.  \n• Avoid: deep technical detail; troubleshooting; exhaustive options.\n\n## 2. How-to Guides  🍳  (`docs/howtos/`)\n• Purpose: help the user **achieve a specific goal** – “How do I…?”  \n• Form: concise series of **step-by-step instructions** focused on the user’s project.  \n• Write from the **user’s perspective**, not the tool’s operations.  \n• Link to reference and explanation pages for background; keep prose minimal.\n• Add any code run outputs as expandable click blocks. Readers should be able to understand the guide without running the code.\n\n## 3. Reference  📑  (`docs/references/`)\n• Purpose: provide **neutral, complete, accurate description** of APIs, commands, options – “What is…?”  \n• Maintain consistent patterns (parameter tables, return types, examples).  \n• Avoid instruction or opinion; instead *link* to how-to or explanation pages.  \n• Examples are welcome if they illustrate usage without drifting into tutorial style.\n\n## 4. Explanation  💡  (`docs/concepts/`, `docs/experimental/core_concepts/`)\n• Purpose: **clarify concepts and rationale** – “Why…?”  \n• Form: discursive article that illuminates design decisions, theory, background.  \n• May link out to tutorials, how-tos, and reference, but does not instruct step-by-step.\n\n### Keep the Borders Sharp\n• Do **not** let content blur between modes (e.g., no instructions inside reference; no lengthy theory in how-tos).  \n• If a page starts serving two modes, split it.\n\n### Filing & Navigation\n• Place the file in the folder matching its mode (above).  \n• Update `mkdocs.yml` `nav:` under the corresponding section.\n\n### Incremental Improvement Cycle (per Diátaxis)\nChoose → Assess → Decide → Do. Focus on small, atomic upgrades rather than grand rewrites.\n\n### Writing Style\n• Use second-person (\"you\") and active voice.\n• Ensure code blocks are **copy-pasteable** and include necessary context (imports, environment).\n• Prefer short sentences; use Markdown admonitions (`!!! note`, `!!! warning`) sparingly for important side-information.\n• Use `??? \"Click to expand\"` collapsible admonitions to contain outputs, long prompts, verbose logs, or any content that would clutter the main article flow. This keeps the primary content scannable while preserving detailed information for readers who need it.\n• **Always add a blank line after text ending with a colon before starting a list.** This ensures proper Markdown rendering in MkDocs. Without the blank line, list items may render as continuation text instead of a proper bulleted/numbered list.\n\n### Cross-linking Between Modes\n• End tutorials with pointers to relevant how-to guides for further exploration.\n• How-to guides should include links to reference/API pages for deeper details.\n• Explanations can reference tutorials and how-tos to illustrate concepts in action.\n\n### Page Metadata & Prerequisites\n• Start each page with a one-sentence purpose statement and a brief list of prerequisites (libraries, data, environment variables).\n• Highlight any external services/configuration required before the reader begins.\n\n### Keep Pages Atomic\n• One page = one task, concept, or API surface. If content grows, **split** rather than creating a mega-guide.\n\n","frontmatter":{"globs":"docs/**"},"isInternal":false,"tokens":825,"sizeBytes":3841},{"name":"docs-structure.mdc","path":".cursor/rules/docs-structure.mdc","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/rules/docs-structure.mdc","title":"Cursor Rule: docs-structure","category":"cursor-rule","format":"mdc","content":"---\nglobs: docs/**\n---\n# Documentation Structure & Workflow\n\nFollow these conventions when creating or editing documentation:\n\n1. **Docs live in [docs/](mdc:docs/)**\n   • Use Markdown (`.md`) files.  \n   • Images and other assets go in [docs/_static/](mdc:docs/_static/).\n\n2. **Section Folders mirror MkDocs navigation** (see [mkdocs.yml](mdc:mkdocs.yml)):\n   • 🚀 Get Started → [docs/getstarted/](mdc:docs/getstarted/)  \n   • 📚 Core Concepts → [docs/concepts/](mdc:docs/concepts/)  \n   • 🧪 Experimental → [docs/experimental/](mdc:docs/experimental/)  \n   • 🛠️ How-to Guides → [docs/howtos/](mdc:docs/howtos/)  \n   • 📖 References → [docs/references/](mdc:docs/references/)  \n   • Community → [docs/community/](mdc:docs/community/)\n\n   Place new pages in the appropriate folder **and** update `mkdocs.yml` `nav:` so the page appears in navigation.\n\n3. **Notebook-to-Markdown**\n   • Convert notebooks to Markdown with [docs/ipynb_to_md.py](mdc:docs/ipynb_to_md.py).  \n   • Commit the generated `.md`; notebooks themselves should not live in `docs/`.\n\n4. **Local preview / build**\n   • Run `make build-docs` to build HTML, `make serve-docs` to preview locally (defined in [DEVELOPMENT.md](mdc:DEVELOPMENT.md)).\n\n5. **Style & Assets**\n   • Use relative links (`../`) within docs.  \n   • Reference images via `_static/…` paths so they work in both dev and hosted docs.  \n   • Custom templates/CSS live in [docs/extra/](mdc:docs/extra/) — avoid editing `material` theme defaults directly.\n\n6. **API References (mkdocstrings)**\n   • Always use public API paths in `[ClassName][ragas.module.ClassName]` references.\n   • Check what's exported in `__init__.py` — if a class isn't in `__all__`, mkdocstrings can't link to it.\n   • Example: Use `[BasePrompt][ragas.prompt.BasePrompt]` not `[BasePrompt][ragas.prompt.base.BasePrompt]` or internal module paths.\n\n7. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.\n\n---\n\n# Formatting Guidelines\n\n- When introducing a list with text ending in a colon (e.g., \"This will:\"), always add a blank line before the first list item. \n- In a numbered list, do not add any new line between the items.","frontmatter":{"globs":"docs/**"},"isInternal":false,"tokens":610,"sizeBytes":2272},{"name":"project-structure.mdc","path":".cursor/rules/project-structure.mdc","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/rules/project-structure.mdc","title":"Cursor Rule: project-structure","category":"cursor-rule","format":"mdc","content":"---\nalwaysApply: true\n---\n# Monorepo Project Structure\n\nThe repository is a monorepo consisting of two primary components:\n\n1. [/](./) – Core evaluation toolkit\n   • Source code lives in [src/](src/)\n   • Tests live in [tests/](tests/)\n   • Build configuration is in [pyproject.toml](pyproject.toml)\n\n2. [examples/](examples/) – Installable examples package\n   • Package: ragas_examples/ containing agent_evals, prompt_evals, rag_eval, workflow_eval, benchmark_llm\n   • Build configuration in [examples/pyproject.toml](examples/pyproject.toml)\n   • Shipped as `ragas-examples` package on PyPI via `ragas[examples]` extra\n   • Local development: `uv pip install -e . -e ./examples`\n   • Usage: `python -m ragas_examples.benchmark_llm.prompt`\n\nShared documentation for all projects is located under [docs/](docs/).\n\nRoot-level [Makefile](Makefile) provides combined commands, while each project directory also contains its own Makefile for project-specific tasks.\n","frontmatter":{"alwaysApply":true},"isInternal":false,"tokens":239,"sizeBytes":982},{"name":"update-guide.mdc","path":".cursor/rules/update-guide.mdc","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/rules/update-guide.mdc","title":"Cursor Rule: update-guide","category":"cursor-rule","format":"mdc","content":"---\nalwaysApply: false\n---\nWe are writing a how to guide for Ragas docs and Ragas users.\n\nSo after any coding step we complete, or after any succesful runs, always update the guide with what was done. Make sure the content is concise and to the point. \n\nCurrent guide: docs/howtos/applications/evaluate-and-improve-rag.md","frontmatter":{"alwaysApply":false},"isInternal":false,"tokens":75,"sizeBytes":321},{"name":"use-uv-cli.mdc","path":".cursor/rules/use-uv-cli.mdc","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/rules/use-uv-cli.mdc","title":"Cursor Rule: use-uv-cli","category":"cursor-rule","format":"mdc","content":"---\nalwaysApply: true\n---\n# Use `uv run` for Python CLI commands\n\nThis repository manages its virtual environment and dependencies with **uv**. Therefore, always execute Python or Python-related CLI tools through `uv run`.\n\nExamples:\n- `uv run pytest`\n- `uv run ruff check .`\n- `uv run isort .`\n- `uv run pyright`\n","frontmatter":{"alwaysApply":true},"isInternal":false,"tokens":78,"sizeBytes":314},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n","isInternal":false,"tokens":1425,"sizeBytes":6613},{"name":"create-pr.md","path":".claude/commands/create-pr.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.claude/commands/create-pr.md","title":"Command: /create-pr","category":"command-prompt","format":"markdown","content":"<!--\nWHITELIST_AFTER_APPROVAL:\n- Bash(git push:*)\n- Bash(gh pr create:*)\n-->\n\nPlease create a pull request with the changes on the current branch. $ARGUMENTS\n\n**IMPORTANT: Use planning mode for this command.**\n\nFollow these steps:\n\n## Planning Phase (Research Only):\n1. Check git status to ensure working tree is clean\n2. Verify current branch is not main/master\n3. Check if current branch tracks a remote branch and is up to date\n4. Run `git log --oneline main..HEAD` to see commits that will be included in PR\n5. Run `git diff main...HEAD` to understand the full scope of changes\n6. Analyze all changes and commits to create a comprehensive PR summary\n7. Draft PR title and description with:\n   - Title: Clear, descriptive title based on the changes\n   - Body: Include \"## Summary\" with bullet points, \"## Test plan\" checklist, and Claude signature\n8. **Present the draft PR title and description using ExitPlanMode tool for user approval**\n\n## Execution Phase (After User Approval):\n9. Push branch to remote with upstream tracking if needed\n10. Create PR using `gh pr create` with the approved title and description\n11. Return the PR URL for easy access\n\nUse the GitHub CLI (`gh`) for all GitHub-related operations. If $ARGUMENTS are provided, incorporate them as hints for the PR title or description.","isInternal":false,"tokens":303,"sizeBytes":1305},{"name":"fix-github-issue.md","path":".claude/commands/fix-github-issue.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.claude/commands/fix-github-issue.md","title":"Command: /fix-github-issue","category":"command-prompt","format":"markdown","content":"Please analyze and fix the GitHub issue: $ARGUMENTS.\n\nFollow these steps:\n\n1. Use `gh issue view` to get the issue details\n2. Understand the problem described in the issue\n3. Search the codebase for relevant files\n4. Implement the necessary changes to fix the issue\n5. Write and run tests to verify the fix\n6. Ensure code passes linting and type checking\n7. Create a descriptive commit message\n8. Push and create a PR\n\nRemember to use the GitHub CLI (`gh`) for all GitHub-related tasks.\n","isInternal":false,"tokens":115,"sizeBytes":487},{"name":"git-pr.md","path":".cursor/commands/git-pr.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/commands/git-pr.md","title":"Command: /git-pr","category":"command-prompt","format":"markdown","content":"Make sure you are on a branch other than main. \nCommit changes. \nRun make format. \nCreate PR using gh cli following .github/pull_request_template.md template\n","isInternal":false,"tokens":36,"sizeBytes":158},{"name":"update-howto-guide.md","path":".cursor/commands/update-howto-guide.md","rawUrl":"https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/.cursor/commands/update-howto-guide.md","title":"Command: /update-howto-guide","category":"command-prompt","format":"markdown","content":"# Update How-to Guide\n\nUpdates the mentioned how-to guide to use src/ragas/metrics/collections API instead of the legacy ragas/metrics API and LLM factory pattern instead of langchainwrapper.\n\n## File Format Decision\n\nIf the source is an `.ipynb` file (or if the `.md` filename starts with `_`, indicating it's derived from a notebook via `docs/ipynb_to_md.py`):\n\n1. **Delete** the `.ipynb` file\n2. **Delete** the corresponding `_xxx.md` file (if it exists)\n3. **Create** a new `.md` file directly (without the `_` prefix)\n\nThis simplifies maintenance by having pure markdown docs instead of notebooks.\n\n\n## Process\n\n### Phase 1: Research (do NOT make changes yet)\n\nRefer pr-description-customizations.md for the list of guides that are already updated. And finally update the doc after you're done. \n\n#### 1.1 Understand the Guide's Purpose\n- Read the target file thoroughly\n- Identify **what the guide is trying to achieve** (e.g., caching, run config, retry handling)\n- Note the specific use case or need the guide addresses\n- Understand what underlying tools/libraries are being used (e.g., instructor, liteLLM, httpx)\n\n#### 1.2 Feasibility Check\nBefore doing anything else, check if the feature works with the new API:\n\n1. **Check `src/ragas/experiment.py`** - Does experiment() support this feature?\n2. **Check `src/ragas/evaluation.py`** - Is this an evaluate()-only feature?\n3. **Check `src/ragas/metrics/collections/`** - Do collections metrics support this?\n4. **Check if simpler alternatives exist** - Does a newer, simpler API make this guide obsolete? (e.g., decorator-based metrics vs subclassing, built-in features vs manual workarounds). Check concept docs and `src/ragas/metrics/` for modern patterns.\n\n**If a simpler approach exists → recommend deletion** instead of migration. See \"When to Recommend Deletion\" section.\n\n**If not supported in new API → STOP immediately:**\n- Keep guide as-is\n- Output this Slack message for the team:\n\n```\n📋 *Doc Update Skipped*: `<guide_path>`\n*Link*: https://docs.ragas.io/en/latest/<guide_path_without_extension>/\n*Reason*: <feature> only works with legacy `evaluate()` API, not yet supported in `experiment()`/collections\n*Action*: Keep as-is until collections API adds support\n```\n\n**If supported → continue to 1.3**\n\n#### 1.3 Present Plan & Wait for Approval\n\n**⏸️ STOP HERE - Do NOT proceed to Phase 2 without explicit user approval.**\n\nPresent a clear summary:\n1. **Current state**: What the guide currently does and how\n2. **Proposed changes**: \n   - Imports to change (from old → new)\n   - LLM/embeddings setup patterns to update\n   - **How the specific use case/feature will be achieved** with the new API\n   - Any restructuring or content changes\n3. **Potential concerns**: Anything uncertain or risky\n4. **Ask**: \"Does this plan look good? Should I proceed?\"\n\n**Wait for user to say \"yes\", \"proceed\", \"go ahead\", or similar before continuing.**\n\n---\n\n### Phase 2: Execute (only after approval)\n\n#### 2.1 Apply Updates\n\n**Keep it Concise**: \n- Remove unnecessary explanations and verbose text\n- Focus on the essential information needed to achieve the goal\n- Use clear, direct language\n- Avoid redundant examples - one good example is better than multiple similar ones\n\n**Import Updates**:\n```python\n# Change from:\nfrom ragas.metrics import MetricName\n\n# To:\nfrom ragas.metrics.collections import MetricName\n```\n\n**LLM Setup**:\n```python\n# Change from:\nfrom langchain_openai import ChatOpenAI\nfrom ragas.llms import LangchainLLMWrapper\nllm = LangchainLLMWrapper(ChatOpenAI(model=\"gpt-4o\"))\n\n# To:\nfrom openai import OpenAI\nfrom ragas.llms import llm_factory\nclient = OpenAI(api_key=\"sk-...\")\nllm = llm_factory(\"gpt-4o\", client=client)\n```\n\n**Embeddings Setup**:\n```python\n# Change from:\nfrom langchain_openai import OpenAIEmbeddings\nfrom ragas.embeddings import LangchainEmbeddingsWrapper\nembeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())\n\n# To:\nfrom openai import OpenAI\nfrom ragas.embeddings.base import embedding_factory  # Use .base to avoid deprecation warning\nclient = OpenAI(api_key=\"sk-...\")\nembeddings = embedding_factory(\"openai\", model=\"text-embedding-3-small\", client=client)\n```\n\n**What to Fix**:\n- Update imports and LLM/embeddings patterns\n- Use `ragas.embeddings.base` import to avoid deprecation warnings\n- **Replace all legacy code with modern approaches** (no need to keep legacy sections)\n- Fix minor issues automatically\n- Don't restructure content unless fixing issues\n\n#### 2.2 Verify Accuracy & Test Code\n\n**Verify with Web Search**:\n- Search for official documentation of any libraries/tools mentioned (instructor, liteLLM, httpx, etc.)\n- Confirm API signatures, parameter names, and usage patterns are correct\n- Verify any claims about library behavior are accurate\n\n**Run the Code**:\n- Install any missing packages first: `uv pip install <package>`\n- Extract ALL Python code blocks from the guide\n- Save as `tests/docs/test_<guide_name>.py` (e.g., `test_run_config.py`)\n- Use `.env` from root for API keys. .env only has openai keys, if you need anything else, let me know\n- Run: `uv run python tests/docs/test_<guide_name>.py`\n- **Verify the original use case/goal is achieved** with the new approach\n\n**If tests fail**:\n1. Check the underlying implementation in `src/` to understand correct usage\n2. Fix the code in the guide based on what you learn from `src/`\n3. Re-run the test\n4. Repeat until tests pass\n5. If stuck after multiple attempts, report the issue with details\n\n**Keep the test file** - excluded from default `pytest` runs via `norecursedirs` in `pyproject.toml`.\n\n**Both verification methods are required** - web search for accuracy, code execution for functionality.\n\n#### 2.3 Check Navigation\n- Verify file is in `mkdocs.yml`\n- Note if location seems wrong or can be put in a more appropriate section\n\n#### 2.4 Summarize Changes\n- List all changes made\n- Mention if anything is not tested due to any reasons (like missing packages, missing API keys, etc.)\n\n## Notes\n- **Two-phase workflow**: Research first, get approval, then execute\n- **Never skip approval**: Always present the plan and wait for explicit \"go ahead\" before making changes\n- **This is not just a straightforward migration** - understand if the original goal is achievable first\n- **Keep guides concise** - remove fluff, focus on essential information\n- **Verify accuracy** - use web search to confirm library APIs and behavior before writing\n- **Test everything** - run all code examples before finalizing\n- Only fix what's broken or outdated\n- Check `src/` before updating to verify APIs exist\n- Don't add legacy sections\n- Use root `.env` for testing\n- **Keep test files** in `tests/docs/` - excluded from default pytest runs\n\n## When to Recommend Deletion\n\nIf a guide teaches **writing custom metrics by subclassing** (`MetricWithLLM`, `SingleTurnMetric`, etc.), it's likely obsolete. The decorator-based approach is simpler:\n\n```python\nfrom ragas.metrics import discrete_metric, numeric_metric, ranking_metric\n\n@discrete_metric(name=\"my_metric\", allowed_values=[\"pass\", \"fail\"])\ndef my_metric(response: str, context: str) -> str:\n    return \"pass\" if condition else \"fail\"\n```\n\nSee `docs/concepts/metrics/overview/index.md` for details. Recommend deletion if decorators cover the use case.\n\n## Reporting Gaps\nIf you identify a gap, use the Slack message template from section 1.2.\n","isInternal":false,"tokens":1783,"sizeBytes":7387}],"systemPromptSnippet":"<yakaai_skills repo=\"vibrantlabsai/ragas\">\n<!-- File: docs/references/aevaluate.md (Tokens: ~294 | Category: anthropic-skill) -->\n# Async Evaluation\n\n## aevaluate()\n\n::: ragas.evaluation.aevaluate\n\n## Async Usage\n\nRagas provides both synchronous and asynchronous evaluation APIs to accommodate different use cases:\n\n### Using aevaluate() (Recommended for Production)\n\nFor production async applications, use `aevaluate()` to avoid event loop conflicts:\n\n```python\nimport asyncio\nfrom ragas import aevaluate\n\nasync def evaluate_app():\n    result = await aevaluate(dataset, metrics)\n    return result\n\n# In your async application\nresult = await evaluate_app()\n```\n\n### Using evaluate() with Async Control\n\nFor backward compatibility and Jupyter notebook usage, `evaluate()` provides optional control over `nest_asyncio`:\n\n```python\n# Default behavior (Jupyter-compatible)\nresult = evaluate(dataset, metrics)  # allow_nest_asyncio=True\n\n# Production-safe (avoids event loop patching)\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n\n### Migration from nest_asyncio Issues\n\nIf you're experiencing issues with `nest_asyncio` in production:\n\n**Before (problematic):**\n```python\n# This may cause event loop conflicts\nresult = evaluate(dataset, metrics)\n```\n\n**After (fixed):**\n```python\n# Option 1: Use async API\nresult = await aevaluate(dataset, metrics)\n\n# Option 2: Disable nest_asyncio\nresult = evaluate(dataset, metrics, allow_nest_asyncio=False)\n```\n\n<!-- File: docs/references/cache.md (Tokens: ~16 | Category: anthropic-skill) -->\n::: ragas.cache\n    options:\n        members_order: \"source\"\n\n<!-- File: docs/references/embeddings.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/embeddings.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/embeddings.md)\n\n<!-- File: docs/references/evaluate.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/evaluate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluate.md)\n\n<!-- File: docs/references/evaluation_schema.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/evaluation_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/evaluation_schema.md)\n\n<!-- File: docs/references/executor.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/executor.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/executor.md)\n\n<!-- File: docs/references/generate.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/generate.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/generate.md)\n\n<!-- File: docs/references/graph.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/graph.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/graph.md)\n\n<!-- File: docs/references/index.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/index.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/index.md)\n\n<!-- File: docs/references/integrations.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/integrations.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/integrations.md)\n\n<!-- File: docs/references/llms.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/llms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/llms.md)\n\n<!-- File: docs/references/metrics.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/metrics.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/metrics.md)\n\n<!-- File: docs/references/optimizers.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/optimizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/optimizers.md)\n\n<!-- File: docs/references/prompt.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/prompt.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/prompt.md)\n\n<!-- File: docs/references/run_config.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/run_config.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/run_config.md)\n\n<!-- File: docs/references/synthesizers.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/synthesizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/synthesizers.md)\n\n<!-- File: docs/references/testset_schema.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/testset_schema.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/testset_schema.md)\n\n<!-- File: docs/references/tokenizers.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/tokenizers.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/tokenizers.md)\n\n<!-- File: docs/references/transforms.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `docs/references/transforms.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/docs/references/transforms.md)\n\n<!-- File: .cursor/rules/docs-diataxis-guidelines.mdc (Tokens: ~825 | Category: cursor-rule) -->\n---\nglobs: docs/**\n---\n# Diátaxis Documentation Guidelines\n\nWhen writing or editing documentation, categorise each page as **one** of the four Diátaxis modes and follow its specific guidance. *Do not mix modes in a single page.*\n\n## 1. Tutorials  🧑‍🏫  (`docs/getstarted/`, `docs/experimental/tutorials/`)\n• Purpose: provide a structured **learning experience** – “Can you teach me to…?”  \n• Form: narrative lesson that leads the reader from zero to a working result.  \n• Include: context, motivation, complete working example.  \n• Avoid: deep technical detail; troubleshooting; exhaustive options.\n\n## 2. How-to Guides  🍳  (`docs/howtos/`)\n• Purpose: help the user **achieve a specific goal** – “How do I…?”  \n• Form: concise series of **step-by-step instructions** focused on the user’s project.  \n• Write from the **user’s perspective**, not the tool’s operations.  \n• Link to reference and explanation pages for background; keep prose minimal.\n• Add any code run outputs as expandable click blocks. Readers should be able to understand the guide without running the code.\n\n## 3. Reference  📑  (`docs/references/`)\n• Purpose: provide **neutral, complete, accurate description** of APIs, commands, options – “What is…?”  \n• Maintain consistent patterns (parameter tables, return types, examples).  \n• Avoid instruction or opinion; instead *link* to how-to or explanation pages.  \n• Examples are welcome if they illustrate usage without drifting into tutorial style.\n\n## 4. Explanation  💡  (`docs/concepts/`, `docs/experimental/core_concepts/`)\n• Purpose: **clarify concepts and rationale** – “Why…?”  \n• Form: discursive article that illuminates design decisions, theory, background.  \n• May link out to tutorials, how-tos, and reference, but does not instruct step-by-step.\n\n### Keep the Borders Sharp\n• Do **not** let content blur between modes (e.g., no instructions inside reference; no lengthy theory in how-tos).  \n• If a page starts serving two modes, split it.\n\n### Filing & Navigation\n• Place the file in the folder matching its mode (above).  \n• Update `mkdocs.yml` `nav:` under the corresponding section.\n\n### Incremental Improvement Cycle (per Diátaxis)\nChoose → Assess → Decide → Do. Focus on small, atomic upgrades rather than grand rewrites.\n\n### Writing Style\n• Use second-person (\"you\") and active voice.\n• Ensure code blocks are **copy-pasteable** and include necessary context (imports, environment).\n• Prefer short sentences; use Markdown admonitions (`!!! note`, `!!! warning`) sparingly for important side-information.\n• Use `??? \"Click to expand\"` collapsible admonitions to contain outputs, long prompts, verbose logs, or any content that would clutter the main article flow. This keeps the primary content scannable while preserving detailed information for readers who need it.\n• **Always add a blank line after text ending with a colon before starting a list.** This ensures proper Markdown rendering in MkDocs. Without the blank line, list items may render as continuation text instead of a proper bulleted/numbered list.\n\n### Cross-linking Between Modes\n• End tutorials with pointers to relevant how-to guides for further exploration.\n• How-to guides should include links to reference/API pages for deeper details.\n• Explanations can reference tutorials and how-tos to illustrate concepts in action.\n\n### Page Metadata & Prerequisites\n• Start each page with a one-sentence purpose statement and a brief list of prerequisites (libraries, data, environment variables).\n• Highlight any external services/configuration required before the reader begins.\n\n### Keep Pages Atomic\n• One page = one task, concept, or API surface. If content grows, **split** rather than creating a mega-guide.\n\n<!-- File: .cursor/rules/docs-structure.mdc (Tokens: ~610 | Category: cursor-rule) -->\n---\nglobs: docs/**\n---\n# Documentation Structure & Workflow\n\nFollow these conventions when creating or editing documentation:\n\n1. **Docs live in [docs/](mdc:docs/)**\n   • Use Markdown (`.md`) files.  \n   • Images and other assets go in [docs/_static/](mdc:docs/_static/).\n\n2. **Section Folders mirror MkDocs navigation** (see [mkdocs.yml](mdc:mkdocs.yml)):\n   • 🚀 Get Started → [docs/getstarted/](mdc:docs/getstarted/)  \n   • 📚 Core Concepts → [docs/concepts/](mdc:docs/concepts/)  \n   • 🧪 Experimental → [docs/experimental/](mdc:docs/experimental/)  \n   • 🛠️ How-to Guides → [docs/howtos/](mdc:docs/howtos/)  \n   • 📖 References → [docs/references/](mdc:docs/references/)  \n   • Community → [docs/community/](mdc:docs/community/)\n\n   Place new pages in the appropriate folder **and** update `mkdocs.yml` `nav:` so the page appears in navigation.\n\n3. **Notebook-to-Markdown**\n   • Convert notebooks to Markdown with [docs/ipynb_to_md.py](mdc:docs/ipynb_to_md.py).  \n   • Commit the generated `.md`; notebooks themselves should not live in `docs/`.\n\n4. **Local preview / build**\n   • Run `make build-docs` to build HTML, `make serve-docs` to preview locally (defined in [DEVELOPMENT.md](mdc:DEVELOPMENT.md)).\n\n5. **Style & Assets**\n   • Use relative links (`../`) within docs.  \n   • Reference images via `_static/…` paths so they work in both dev and hosted docs.  \n   • Custom templates/CSS live in [docs/extra/](mdc:docs/extra/) — avoid editing `material` theme defaults directly.\n\n6. **API References (mkdocstrings)**\n   • Always use public API paths in `[ClassName][ragas.module.ClassName]` references.\n   • Check what's exported in `__init__.py` — if a class isn't in `__all__`, mkdocstrings can't link to it.\n   • Example: Use `[BasePrompt][ragas.prompt.BasePrompt]` not `[BasePrompt][ragas.prompt.base.BasePrompt]` or internal module paths.\n\n7. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.\n\n---\n\n# Formatting Guidelines\n\n- When introducing a list with text ending in a colon (e.g., \"This will:\"), always add a blank line before the first list item. \n- In a numbered list, do not add any new line between the items.\n\n<!-- File: .cursor/rules/project-structure.mdc (Tokens: ~239 | Category: cursor-rule) -->\n---\nalwaysApply: true\n---\n# Monorepo Project Structure\n\nThe repository is a monorepo consisting of two primary components:\n\n1. [/](./) – Core evaluation toolkit\n   • Source code lives in [src/](src/)\n   • Tests live in [tests/](tests/)\n   • Build configuration is in [pyproject.toml](pyproject.toml)\n\n2. [examples/](examples/) – Installable examples package\n   • Package: ragas_examples/ containing agent_evals, prompt_evals, rag_eval, workflow_eval, benchmark_llm\n   • Build configuration in [examples/pyproject.toml](examples/pyproject.toml)\n   • Shipped as `ragas-examples` package on PyPI via `ragas[examples]` extra\n   • Local development: `uv pip install -e . -e ./examples`\n   • Usage: `python -m ragas_examples.benchmark_llm.prompt`\n\nShared documentation for all projects is located under [docs/](docs/).\n\nRoot-level [Makefile](Makefile) provides combined commands, while each project directory also contains its own Makefile for project-specific tasks.\n\n<!-- File: .cursor/rules/update-guide.mdc (Tokens: ~75 | Category: cursor-rule) -->\n---\nalwaysApply: false\n---\nWe are writing a how to guide for Ragas docs and Ragas users.\n\nSo after any coding step we complete, or after any succesful runs, always update the guide with what was done. Make sure the content is concise and to the point. \n\nCurrent guide: docs/howtos/applications/evaluate-and-improve-rag.md\n\n<!-- File: .cursor/rules/use-uv-cli.mdc (Tokens: ~78 | Category: cursor-rule) -->\n---\nalwaysApply: true\n---\n# Use `uv run` for Python CLI commands\n\nThis repository manages its virtual environment and dependencies with **uv**. Therefore, always execute Python or Python-related CLI tools through `uv run`.\n\nExamples:\n- `uv run pytest`\n- `uv run ruff check .`\n- `uv run isort .`\n- `uv run pyright`\n\n<!-- File: CLAUDE.md (Tokens: ~1425 | Category: claude-rule) -->\n# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n\n<!-- File: .claude/commands/create-pr.md (Tokens: ~303 | Category: command-prompt) -->\n<!--\nWHITELIST_AFTER_APPROVAL:\n- Bash(git push:*)\n- Bash(gh pr create:*)\n-->\n\nPlease create a pull request with the changes on the current branch. $ARGUMENTS\n\n**IMPORTANT: Use planning mode for this command.**\n\nFollow these steps:\n\n## Planning Phase (Research Only):\n1. Check git status to ensure working tree is clean\n2. Verify current branch is not main/master\n3. Check if current branch tracks a remote branch and is up to date\n4. Run `git log --oneline main..HEAD` to see commits that will be included in PR\n5. Run `git diff main...HEAD` to understand the full scope of changes\n6. Analyze all changes and commits to create a comprehensive PR summary\n7. Draft PR title and description with:\n   - Title: Clear, descriptive title based on the changes\n   - Body: Include \"## Summary\" with bullet points, \"## Test plan\" checklist, and Claude signature\n8. **Present the draft PR title and description using ExitPlanMode tool for user approval**\n\n## Execution Phase (After User Approval):\n9. Push branch to remote with upstream tracking if needed\n10. Create PR using `gh pr create` with the approved title and description\n11. Return the PR URL for easy access\n\nUse the GitHub CLI (`gh`) for all GitHub-related operations. If $ARGUMENTS are provided, incorporate them as hints for the PR title or description.\n\n<!-- File: .claude/commands/fix-github-issue.md (Tokens: ~115 | Category: command-prompt) -->\nPlease analyze and fix the GitHub issue: $ARGUMENTS.\n\nFollow these steps:\n\n1. Use `gh issue view` to get the issue details\n2. Understand the problem described in the issue\n3. Search the codebase for relevant files\n4. Implement the necessary changes to fix the issue\n5. Write and run tests to verify the fix\n6. Ensure code passes linting and type checking\n7. Create a descriptive commit message\n8. Push and create a PR\n\nRemember to use the GitHub CLI (`gh`) for all GitHub-related tasks.\n\n<!-- File: .cursor/commands/git-pr.md (Tokens: ~36 | Category: command-prompt) -->\nMake sure you are on a branch other than main. \nCommit changes. \nRun make format. \nCreate PR using gh cli following .github/pull_request_template.md template\n\n<!-- File: .cursor/commands/update-howto-guide.md (Tokens: ~1783 | Category: command-prompt) -->\n# Update How-to Guide\n\nUpdates the mentioned how-to guide to use src/ragas/metrics/collections API instead of the legacy ragas/metrics API and LLM factory pattern instead of langchainwrapper.\n\n## File Format Decision\n\nIf the source is an `.ipynb` file (or if the `.md` filename starts with `_`, indicating it's derived from a notebook via `docs/ipynb_to_md.py`):\n\n1. **Delete** the `.ipynb` file\n2. **Delete** the corresponding `_xxx.md` file (if it exists)\n3. **Create** a new `.md` file directly (without the `_` prefix)\n\nThis simplifies maintenance by having pure markdown docs instead of notebooks.\n\n\n## Process\n\n### Phase 1: Research (do NOT make changes yet)\n\nRefer pr-description-customizations.md for the list of guides that are already updated. And finally update the doc after you're done. \n\n#### 1.1 Understand the Guide's Purpose\n- Read the target file thoroughly\n- Identify **what the guide is trying to achieve** (e.g., caching, run config, retry handling)\n- Note the specific use case or need the guide addresses\n- Understand what underlying tools/libraries are being used (e.g., instructor, liteLLM, httpx)\n\n#### 1.2 Feasibility Check\nBefore doing anything else, check if the feature works with the new API:\n\n1. **Check `src/ragas/experiment.py`** - Does experiment() support this feature?\n2. **Check `src/ragas/evaluation.py`** - Is this an evaluate()-only feature?\n3. **Check `src/ragas/metrics/collections/`** - Do collections metrics support this?\n4. **Check if simpler alternatives exist** - Does a newer, simpler API make this guide obsolete? (e.g., decorator-based metrics vs subclassing, built-in features vs manual workarounds). Check concept docs and `src/ragas/metrics/` for modern patterns.\n\n**If a simpler approach exists → recommend deletion** instead of migration. See \"When to Recommend Deletion\" section.\n\n**If not supported in new API → STOP immediately:**\n- Keep guide as-is\n- Output this Slack message for the team:\n\n```\n📋 *Doc Update Skipped*: `<guide_path>`\n*Link*: https://docs.ragas.io/en/latest/<guide_path_without_extension>/\n*Reason*: <feature> only works with legacy `evaluate()` API, not yet supported in `experiment()`/collections\n*Action*: Keep as-is until collections API adds support\n```\n\n**If supported → continue to 1.3**\n\n#### 1.3 Present Plan & Wait for Approval\n\n**⏸️ STOP HERE - Do NOT proceed to Phase 2 without explicit user approval.**\n\nPresent a clear summary:\n1. **Current state**: What the guide currently does and how\n2. **Proposed changes**: \n   - Imports to change (from old → new)\n   - LLM/embeddings setup patterns to update\n   - **How the specific use case/feature will be achieved** with the new API\n   - Any restructuring or content changes\n3. **Potential concerns**: Anything uncertain or risky\n4. **Ask**: \"Does this plan look good? Should I proceed?\"\n\n**Wait for user to say \"yes\", \"proceed\", \"go ahead\", or similar before continuing.**\n\n---\n\n### Phase 2: Execute (only after approval)\n\n#### 2.1 Apply Updates\n\n**Keep it Concise**: \n- Remove unnecessary explanations and verbose text\n- Focus on the essential information needed to achieve the goal\n- Use clear, direct language\n- Avoid redundant examples - one good example is better than multiple similar ones\n\n**Import Updates**:\n```python\n# Change from:\nfrom ragas.metrics import MetricName\n\n# To:\nfrom ragas.metrics.collections import MetricName\n```\n\n**LLM Setup**:\n```python\n# Change from:\nfrom langchain_openai import ChatOpenAI\nfrom ragas.llms import LangchainLLMWrapper\nllm = LangchainLLMWrapper(ChatOpenAI(model=\"gpt-4o\"))\n\n# To:\nfrom openai import OpenAI\nfrom ragas.llms import llm_factory\nclient = OpenAI(api_key=\"sk-...\")\nllm = llm_factory(\"gpt-4o\", client=client)\n```\n\n**Embeddings Setup**:\n```python\n# Change from:\nfrom langchain_openai import OpenAIEmbeddings\nfrom ragas.embeddings import LangchainEmbeddingsWrapper\nembeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())\n\n# To:\nfrom openai import OpenAI\nfrom ragas.embeddings.base import embedding_factory  # Use .base to avoid deprecation warning\nclient = OpenAI(api_key=\"sk-...\")\nembeddings = embedding_factory(\"openai\", model=\"text-embedding-3-small\", client=client)\n```\n\n**What to Fix**:\n- Update imports and LLM/embeddings patterns\n- Use `ragas.embeddings.base` import to avoid deprecation warnings\n- **Replace all legacy code with modern approaches** (no need to keep legacy sections)\n- Fix minor issues automatically\n- Don't restructure content unless fixing issues\n\n#### 2.2 Verify Accuracy & Test Code\n\n**Verify with Web Search**:\n- Search for official documentation of any libraries/tools mentioned (instructor, liteLLM, httpx, etc.)\n- Confirm API signatures, parameter names, and usage patterns are correct\n- Verify any claims about library behavior are accurate\n\n**Run the Code**:\n- Install any missing packages first: `uv pip install <package>`\n- Extract ALL Python code blocks from the guide\n- Save as `tests/docs/test_<guide_name>.py` (e.g., `test_run_config.py`)\n- Use `.env` from root for API keys. .env only has openai keys, if you need anything else, let me know\n- Run: `uv run python tests/docs/test_<guide_name>.py`\n- **Verify the original use case/goal is achieved** with the new approach\n\n**If tests fail**:\n1. Check the underlying implementation in `src/` to understand correct usage\n2. Fix the code in the guide based on what you learn from `src/`\n3. Re-run the test\n4. Repeat until tests pass\n5. If stuck after multiple attempts, report the issue with details\n\n**Keep the test file** - excluded from default `pytest` runs via `norecursedirs` in `pyproject.toml`.\n\n**Both verification methods are required** - web search for accuracy, code execution for functionality.\n\n#### 2.3 Check Navigation\n- Verify file is in `mkdocs.yml`\n- Note if location seems wrong or can be put in a more appropriate section\n\n#### 2.4 Summarize Changes\n- List all changes made\n- Mention if anything is not tested due to any reasons (like missing packages, missing API keys, etc.)\n\n## Notes\n- **Two-phase workflow**: Research first, get approval, then execute\n- **Never skip approval**: Always present the plan and wait for explicit \"go ahead\" before making changes\n- **This is not just a straightforward migration** - understand if the original goal is achievable first\n- **Keep guides concise** - remove fluff, focus on essential information\n- **Verify accuracy** - use web search to confirm library APIs and behavior before writing\n- **Test everything** - run all code examples before finalizing\n- Only fix what's broken or outdated\n- Check `src/` before updating to verify APIs exist\n- Don't add legacy sections\n- Use root `.env` for testing\n- **Keep test files** in `tests/docs/` - excluded from default pytest runs\n\n## When to Recommend Deletion\n\nIf a guide teaches **writing custom metrics by subclassing** (`MetricWithLLM`, `SingleTurnMetric`, etc.), it's likely obsolete. The decorator-based approach is simpler:\n\n```python\nfrom ragas.metrics import discrete_metric, numeric_metric, ranking_metric\n\n@discrete_metric(name=\"my_metric\", allowed_values=[\"pass\", \"fail\"])\ndef my_metric(response: str, context: str) -> str:\n    return \"pass\" if condition else \"fail\"\n```\n\nSee `docs/concepts/metrics/overview/index.md` for details. Recommend deletion if decorators cover the use case.\n\n## Reporting Gaps\nIf you identify a gap, use the Slack message template from section 1.2.\n</yakaai_skills>"}