{"owner":"vibrantlabsai","repo":"ragas","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md"],"skills":{"CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n"},"files":{"CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nRagas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.\n\nThe repository contains:\n\n1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)\n   - Core evaluation metrics and test generation\n   - Experimental features available at `ragas.experimental`\n\n## Development Environment Setup\n\n### Installation\n\nChoose the appropriate installation based on your needs:\n\n```bash\n# RECOMMENDED: Minimal dev setup (79 packages - fast)\nmake install-minimal\n\n# FULL: Complete dev environment (383 packages - comprehensive)  \nmake install\n\n# OR manual installation:\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows, use `venv\\Scripts\\activate`\n\n# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)\nuv pip install -e \".[dev-minimal]\"\n\n# Full dev setup (uses [dependency-groups].dev)\nuv sync --group dev\n```\n\n### Installation Methods Explained\n\n- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation\n- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management\n- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches\n\n### Workspace Structure\n\nThe project uses a UV workspace configuration for managing multiple packages:\n\n```bash\n# Install\nuv sync\n\n# Install examples separately\nuv sync --package ragas-examples\n\n# Build specific workspace package\nuv build --package ragas-examples\n```\n\n**Workspace Members:**\n- `ragas` (main package) - Located in `src/ragas/`\n- `ragas-examples` (examples package) - Located in `examples/`\n\nThe workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.\n\n## Common Commands\n\n### Commands (from root directory)\n\n```bash\n# Setup and installation  \nmake install-minimal # Minimal dev setup (79 packages - recommended)\nmake install        # Full dev environment (383 packages - complete)\n\n# Code quality\nmake format         # Format and lint all code\nmake type           # Type check all code\nmake check          # Quick health check (format + type, no tests)\n\n# Testing\nmake test           # Run all unit tests\nmake test-e2e       # Run end-to-end tests\n\n# CI/Build\nmake run-ci         # Run complete CI pipeline\nmake clean          # Clean all generated files\n\n# Documentation\nmake build-docs     # Build all documentation\nmake serve-docs     # Serve documentation locally\n\n# Benchmarks\nmake benchmarks     # Run performance benchmarks\nmake benchmarks-docker # Run benchmarks in Docker\n```\n\n### Testing\n\n```bash\n# Run all tests (from root)\nmake test\n\n# Run specific test (using pytest -k flag)\nmake test k=\"test_name\"\n\n# Run end-to-end tests\nmake test-e2e\n\n# Direct pytest commands for more control\nuv run pytest tests/unit -k \"test_name\"\nuv run pytest tests/unit -v\n```\n\n### Documentation\n\n```bash\n# Build all documentation (from root)\nmake build-docs\n\n# Serve documentation locally\nmake serve-docs\n```\n\n### Benchmarks\n\n```bash\n# Run all benchmarks locally\nmake benchmarks\n\n# Run benchmarks in Docker\nmake benchmarks-docker\n```\n\n## Project Architecture\n\nThe repository has the following structure:\n\n```sh\n/                          # Main ragas project\n├── src/ragas/             # Source code including experimental features\n│   └── experimental/      # Experimental features\n├── tests/                 # All tests (core + experimental)\n│   └── experimental/      # Experimental tests\n├── examples/              # Example code\n├── pyproject.toml         # Build config\n├── docs/                  # Documentation\n├── scripts/               # Build/CI scripts\n├── Makefile               # Build commands\n└── README.md              # Repository overview\n```\n\n### Ragas Core Components\n\nThe Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:\n\n1. **Metrics** - Various metrics for evaluating LLM applications including:\n\n   - AspectCritic\n   - AnswerCorrectness\n   - ContextPrecision\n   - ContextRecall\n   - Faithfulness\n   - and many more\n\n2. **Test Data Generation** - Automatic creation of test datasets for LLM applications\n\n3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools\n\n### Experimental Components\n\nThe experimental features are now integrated into the main ragas package:\n\n1. **Experimental features** are available at `ragas.experimental`\n2. **Dataset and Experiment management** - Enhanced data handling for experiments\n3. **Advanced metrics** - Extended metric capabilities\n4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)\n\nTo use experimental features:\n\n```python\nfrom ragas import Dataset\nfrom ragas import experiment\nfrom ragas.backends import get_registry\n```\n\n## Debugging Logs\n\nTo view debug logs for any module:\n\n```python\nimport logging\n\n# Configure logging for a specific module (example with analytics)\nanalytics_logger = logging.getLogger('ragas._analytics')\nanalytics_logger.setLevel(logging.DEBUG)\n\n# Create a console handler and set its level\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.DEBUG)\n\n# Create a formatter and add it to the handler\nformatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')\nconsole_handler.setFormatter(formatter)\n\n# Add the handler to the logger\nanalytics_logger.addHandler(console_handler)\n```\n\n## Memories\n\n- whenever you create such docs put in in /\\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these\n- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.\n- The project uses two distinct dependency management approaches:\n  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)\n  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)\n- Use `make install-minimal` for most development tasks, `make install` for full ML stack work\n- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.\n","category":"root","tokens":1638}]}