ragas Documentation

Supercharge Your LLM Application Evaluations 🚀

13,849 stars Python Python 1 file · ~12,547 tokens #evaluation#llm#llmops
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy ragas on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name ragas --restart always vibrantlabsai/ragas:latest

File: README.md

text
[Documentation](https://docs.ragas.io/) |
    [Quick start](#fire-quickstart) |
    [Join Discord](https://discord.gg/5djav8GGNZ) |
    [Blog](https://blog.ragas.io/) |
    [NewsLetter](https://newsletter.ragas.io/) |
    [Careers](https://www.ragas.io/careers)

Objective metrics, intelligent test generation, and data-driven insights for LLM apps

Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows.
Don't have a test dataset ready? We also do production-aligned test set generation.

Key Features

  • 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
  • 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
  • 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
  • 📊 Build feedback loops: Leverage production data to continually improve your LLM applications.

:shield: Installation

Pypi:

bash
pip install ragas

Alternatively, from source:

bash
pip install git+https://github.com/vibrantlabsai/ragas

:fire: Quickstart

Clone a Complete Example Project

The fastest way to get started is to use the ragas quickstart command:

bash
# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project

Available templates:

  • rag_eval - Evaluate RAG systems

Coming Soon:

  • agent_evals - Evaluate AI agents
  • benchmark_llm - Benchmark and compare LLMs
  • prompt_evals - Evaluate prompt variations
  • workflow_eval - Evaluate complex workflows

Evaluate your LLM App

ragas comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using DiscreteMetric:

python
import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures key information.

Response: {response}

Answer with only 'accurate' or 'inaccurate'."""
)

# Score your application's output
async def main():
    score = await metric.ascore(
        llm=llm,
        response="The summary of the text is..."
    )
    print(f"Score: {score.value}")  # 'accurate' or 'inaccurate'
    print(f"Reason: {score.reason}")


if __name__ == "__main__":
    asyncio.run(main())

Note: Make sure your OPENAI_API_KEY environment variable is set.

Find the complete Quickstart Guide

Want help in improving your AI application using evals?

In the past 2 years, we have seen and helped improve many AI applications using evals. If you want help with improving and scaling up your AI application using evals.

🔗 Book a slot or drop us a line: [email protected].

🫂 Community

If you want to get more involved with Ragas, check out our discord server. It's a fun community where we geek out about LLM, Retrieval, Production issues, and more.

Contributors

yml
+----------------------------------------------------------------------------+
|     +----------------------------------------------------------------+     |
|     | Developers: Those who built with `ragas`.                      |     |
|     | (You have `import ragas` somewhere in your project)            |     |
|     |     +----------------------------------------------------+     |     |
|     |     | Contributors: Those who make `ragas` better.       |     |     |
|     |     | (You make PR to this repo)                         |     |     |
|     |     +----------------------------------------------------+     |     |
|     +----------------------------------------------------------------+     |
+----------------------------------------------------------------------------+

We welcome contributions from the community! Whether it's bug fixes, feature additions, or documentation improvements, your input is valuable.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

🔍 Open Analytics

At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.

✅ No personal or company-identifying information

✅ Open-source data collection code

✅ Publicly available aggregated data

To opt-out, set the RAGAS_DO_NOT_TRACK environment variable to true.

Cite Us

text
@misc{ragas2024,
  author       = {VibrantLabs},
  title        = {Ragas: Supercharge Your LLM Application Evaluations},
  year         = {2024},
  howpublished = {\url{https://github.com/vibrantlabsai/ragas}},
}

File: docs/community/index.md

❤️ Community

"Alone we can do so little; together we can do so much." - Helen Keller

Our project thrives on the vibrant energy, diverse skills, and shared passion of our community. It's not just about code; it's about people coming together to create something extraordinary. This space celebrates every contribution, big or small, and features the amazing people who make it all happen.

**💬 Discord community**

Join our Discord community here to connect with other developers, share your ideas, and get support.

**🌟  Contributors**

Meet some of our outstanding contributors!

See the full list here.

**📚 Blog & Insights**

Explore insightful articles, tutorials, and stories written by and for our community members.

**📅 Events**

Stay updated with our latest gatherings, meetups, and online webinars.

File: docs/community/pdf_export.md

PDF Export

Purpose

The PDF export feature builds the complete Ragas documentation as a single PDF file using MkDocs with the mkdocs-to-pdf plugin.

Usage

The implementation uses two separate MkDocs configurations:

  • mkdocs.yml for standard HTML builds (no PDF dependencies required)
  • mkdocs-pdf.yml which inherits from the main config and adds the PDF plugin

Build PDF documentation:

bash
make build-docs-pdf

The generated PDF will be available at site/pdf/document.pdf.

Build HTML documentation only:

bash
make build-docs

The make build-docs-pdf command automatically checks for system dependencies before building.

Mermaid diagrams in PDF (offline)

Mermaid diagrams are rendered offline during the PDF build (converted to SVG before WeasyPrint runs). This requires a few additional dependencies besides WeasyPrint.

Required tools

  • Node.js (needed to run Mermaid tooling).
  • Mermaid CLI (mmdc), installed via @mermaid-js/mermaid-cli.
  • A headless browser for Puppeteer (recommended: chrome-headless-shell).

Current Limitations

System Dependencies: WeasyPrint requires OS-specific system libraries (Pango, Cairo) that must be installed separately. If you encounter issues, refer to the WeasyPrint setup instructions and troubleshooting guide.

ReadTheDocs: PDF generation is not currently enabled in the ReadTheDocs build configuration.

File: docs/concepts/components/eval_dataset.md

Evaluation Dataset

An evaluation dataset is a homogeneous collection of data samples designed to assess the performance and capabilities of an AI application. In Ragas, evaluation datasets are represented using the EvaluationDataset class, which provides a structured way to organize and manage data samples for evaluation purposes.

Overview

Structure of an Evaluation Dataset

An evaluation dataset consists of:

  • Samples: A collection of SingleTurnSample or MultiTurnSample instances. Each sample represents a unique interaction or scenario.
  • Consistency: All samples within the dataset should be of the same type (either all single-turn or all multi-turn samples) to maintain consistency in evaluation.

Guidelines for Curating an Effective Evaluation Dataset

  • Define Clear Objectives: Identify the specific aspects of the AI application that you want to evaluate and the scenarios you want to test. Collect data samples that reflect these objectives.

  • Collect Representative Data: Ensure that the dataset covers a diverse range of scenarios, user inputs, and expected responses to provide a comprehensive evaluation of the AI application. This can be achieved by collecting data from various sources or generating synthetic data.

  • Quality and Size: Aim for a dataset that is large enough to provide meaningful insights but not so large that it becomes unwieldy. Ensure that the data is of high quality and accurately reflects the real-world scenarios you want to evaluate.

Creating an Evaluation Dataset from SingleTurnSamples

In this example, we’ll demonstrate how to create an EvaluationDataset using multiple SingleTurnSample instances. We’ll walk through the process step by step, including creating individual samples, assembling them into a dataset, and performing basic operations on the dataset.

Step 1: Import Necessary Classes

First, import the SingleTurnSample and EvaluationDataset classes from your module.

python
from ragas import SingleTurnSample, EvaluationDataset

Step 2: Create Individual Samples

Create several SingleTurnSample instances that represent individual evaluation samples.

python
# Sample 1
sample1 = SingleTurnSample(
    user_input="What is the capital of Germany?",
    retrieved_contexts=["Berlin is the capital and largest city of Germany."],
    response="The capital of Germany is Berlin.",
    reference="Berlin",
)

# Sample 2
sample2 = SingleTurnSample(
    user_input="Who wrote 'Pride and Prejudice'?",
    retrieved_contexts=["'Pride and Prejudice' is a novel by Jane Austen."],
    response="'Pride and Prejudice' was written by Jane Austen.",
    reference="Jane Austen",
)

# Sample 3
sample3 = SingleTurnSample(
    user_input="What's the chemical formula for water?",
    retrieved_contexts=["Water has the chemical formula H2O."],
    response="The chemical formula for water is H2O.",
    reference="H2O",
)

Step 3: Create the EvaluationDataset

Create an EvaluationDataset by passing a list of SingleTurnSample instances.

python
dataset = EvaluationDataset(samples=[sample1, sample2, sample3])

Loading an Evaluation Dataset from Hugging Face Datasets

In practice, you may want to load an evaluation dataset from an existing dataset source, such as the Hugging Face Datasets library. The following example demonstrates how to load an evaluation dataset from a Hugging Face dataset and convert it into an EvaluationDataset instance.

Ensure that the dataset contains the necessary fields for evaluation, such as user inputs, retrieved contexts, responses, and references.

python
from datasets import load_dataset
dataset = load_dataset("vibrantlabsai/amnesty_qa","english_v3")

Load the dataset into a Ragas EvaluationDataset object.

python
from ragas import EvaluationDataset

eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"])

File: docs/concepts/components/eval_sample.md

Evaluation Sample

An evaluation sample is a single structured data instance that is used to assess and measure the performance of your LLM application in specific scenarios. It represents a single unit of interaction or a specific use case that the AI application is expected to handle. In Ragas, evaluation samples are represented using the SingleTurnSample and MultiTurnSample classes.

SingleTurnSample

SingleTurnSample represents a single-turn interaction between a user, LLM, and expected results for evaluation. It is suitable for evaluations that involve a single question and answer pair, possibly with additional context or reference information.

Example

The following example demonstrates how to create a SingleTurnSample instance for evaluating a single-turn interaction in a RAG-based application. In this scenario, a user asks a question, and the AI provides an answer. We’ll create a SingleTurnSample instance to represent this interaction, including any retrieved contexts, reference answers, and evaluation rubrics.

python
from ragas import SingleTurnSample

# User's question
user_input = "What is the capital of France?"

# Retrieved contexts (e.g., from a knowledge base or search engine)
retrieved_contexts = ["Paris is the capital and most populous city of France."]

# AI's response
response = "The capital of France is Paris."

# Reference answer (ground truth)
reference = "Paris"

# Evaluation rubric
rubric = {
    "accuracy": "Correct",
    "completeness": "High",
    "fluency": "Excellent"
}

# Create the SingleTurnSample instance
sample = SingleTurnSample(
    user_input=user_input,
    retrieved_contexts=retrieved_contexts,
    response=response,
    reference=reference,
    rubric=rubric
)

MultiTurnSample

MultiTurnSample represents a multi-turn interaction between Human, AI and optionally a Tool and expected results for evaluation. It is suitable for representing conversational agents in more complex interactions for evaluation. In MultiTurnSample, the user_input attribute represents a sequence of messages that collectively form a multi-turn conversation between a human user and an AI system. These messages are instances of the classes HumanMessage, AIMessage, and ToolMessage

Example

The following example demonstrates how to create a MultiTurnSample instance for evaluating a multi-turn interaction. In this scenario, a user wants to know the current weather in New York City. The AI assistant will use a weather API tool to fetch the information and respond to the user.

python
from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall

# User asks about the weather in New York City
user_message = HumanMessage(content="What's the weather like in New York City today?")

# AI decides to use a weather API tool to fetch the information
ai_initial_response = AIMessage(
    content="Let me check the current weather in New York City for you.",
    tool_calls=[ToolCall(name="WeatherAPI", args={"location": "New York City"})]
)

# Tool provides the weather information
tool_response = ToolMessage(content="It's sunny with a temperature of 75°F in New York City.")

# AI delivers the final response to the user
ai_final_response = AIMessage(content="It's sunny and 75 degrees Fahrenheit in New York City today.")

# Combine all messages into a list to represent the conversation
conversation = [
    user_message,
    ai_initial_response,
    tool_response,
    ai_final_response
]

Now, use the conversation to create a MultiTurnSample object, including any reference responses and evaluation rubrics.

python
from ragas import MultiTurnSample
# Reference response for evaluation purposes
reference_response = "Provide the current weather in New York City to the user."


# Create the MultiTurnSample instance
sample = MultiTurnSample(
    user_input=conversation,
    reference=reference_response,
)

File: docs/concepts/components/index.md

Components Guide

This guide provides an overview of the different components used inside Ragas.

File: docs/concepts/components/prompt.md

Prompt Object

Prompts in Ragas are used inside various metrics and synthetic data generation tasks. In each of these tasks, Ragas also provides a way for the user to modify or replace the default prompt with a custom prompt. This guide provides an overview of the Prompt Object in Ragas.

Components of a Prompt Object

In Ragas, a prompt object is composed of the following key components:

  1. Instruction: A fundamental element of any prompt, the instruction is a natural language directive that clearly describes the task the Language Model (LLM) should perform. This is specified using the instruction variable within the prompt object.

  2. Few-Shot Examples: LLMs are known to perform better when provided with few-shot examples, as they help the model understand the task context and generate more accurate responses. These examples are specified using the examples variable in the prompt object. Each example consists of an input and its corresponding output, which the LLM uses to learn the task.

  3. Input Model: Every prompt expects an input to produce an output. In Ragas, the expected format of this input is defined using the input_model variable. This is a Pydantic model that outlines the structure of the input, enabling validation and parsing of the data provided to the prompt.

  4. Output Model: Upon execution, a prompt generates an output. The format of this output is specified using the output_model variable in the prompt object. Like the input model, the output model is a Pydantic model that defines the structure of the output, facilitating validation and parsing of the data produced by the LLM.

Example

Here's an example of a prompt object that defines a prompt for a text generation task:

python
from ragas.prompt import PydanticPrompt
from pydantic import BaseModel, Field

class MyInput(BaseModel):
    question: str = Field(description="The question to answer")

class MyOutput(BaseModel):
    answer: str = Field(description="The answer to the question")

class MyPrompt(PydanticPrompt[MyInput,MyInput]):
    instruction = "Answer the given question"
    input_model = MyInput
    output_model = MyOutput
    examples = [
        (
            MyInput(question="Who's building the opensource standard for LLM app evals?"),
            MyOutput(answer="Ragas")
        )
    ]

Guidelines for Creating Effective Prompts

When creating prompts in Ragas, consider the following guidelines to ensure that your prompts are effective and aligned with the task requirements:

  1. Clear and Concise Instructions: Provide clear and concise instructions that clearly define the task the LLM should perform. Ambiguity in instructions can lead to inaccurate responses.
  2. Relevant Few-Shot Examples: Include relevant few-shot examples that cover a diverse range of scenarios related to the task (ideally 3-5). These examples help the LLM understand the context and generate accurate responses.
  3. Simple Input and Output Models: Define simple and intuitive input and output models that accurately represent the data format expected by the LLM and the output generated by the LLM. If the models are complex, try to break the task into smaller sub-tasks with separate prompts.

File: docs/concepts/feedback/index.md

Utilizing User Feedback

User feedback can often be noisy and challenging to harness effectively. However, within the feedback, valuable signals exist that can be leveraged to iteratively enhance your LLM and RAG applications. These signals have the potential to be amplified effectively, aiding in the detection of specific issues within the pipeline and preventing recurring errors. Ragas is equipped to assist you in the analysis of user feedback data, enabling the discovery of patterns and making it a valuable resource for continual improvement.

File: docs/concepts/metrics/available_metrics/agents.md

Agentic or Tool use

Agentic or tool use workflows can be evaluated in multiple dimensions. Here are some of the metrics that can be used to evaluate the performance of agents or tools in a given task.

Topic Adherence

AI systems deployed in real-world applications are expected to adhere to domains of interest while interacting with users but LLMs sometimes may answer general queries by ignoring this limitation. The topic adherence metric evaluates the ability of the AI to stay on predefined domains during the interactions. This metric is particularly important in conversational AI systems, where the AI is expected to only provide assistance to queries related to predefined domains.

TopicAdherence requires a predefined set of topics that the AI system is expected to adhere to which is provided using reference_topics along with user_input. The metric can compute precision, recall, and F1 score for topic adherence, defined as

$$
\text{Precision } = {|\text{Queries that are answered and are adheres to any present reference topics}| \over |\text{Queries that are answered and are adheres to any present reference topics}| + |\text{Queries that are answered and do not adheres to any present reference topics}|}
$$

$$
\text{Recall } = {|\text{Queries that are answered and are adheres to any present reference topics}| \over |\text{Queries that are answered and are adheres to any present reference topics}| + |\text{Queries that were refused and should have been answered}|}
$$

$$
\text{F1 Score } = {2 \times \text{Precision} \times \text{Recall} \over \text{Precision} + \text{Recall}}
$$

Example

text
/* Detailed source-code truncated for AI context efficiency. */

Output

text
Topic Adherence (precision): 0.6666666666444444

To change the mode to recall, set the mode parameter to recall.

python
metric = TopicAdherence(llm=llm, mode="recall")

Output

text
0.99999999995

Legacy API (Deprecated)

!!! warning "Deprecation Notice"
The legacy TopicAdherenceScore from ragas.metrics is deprecated and will be removed in v1.0. Please migrate to ragas.metrics.collections.TopicAdherence which provides the same functionality with a modern API.

The legacy API can still be used but requires MultiTurnSample:

python
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall
from ragas.metrics import TopicAdherenceScore  # Legacy import
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))

sample = MultiTurnSample(
    user_input=[...],  # conversation messages
    reference_topics=["science"],
)
scorer = TopicAdherenceScore(llm=evaluator_llm, mode="precision")
score = await scorer.multi_turn_ascore(sample)

Tool call Accuracy

ToolCallAccuracy measures how accurately an LLM agent invokes tools compared to expected tool calls. It evaluates both the sequence of tool calls and the accuracy of their arguments. This metric is particularly useful for validating that agents call the right tools with the right parameters in multi-step workflows.

The metric requires user_input (conversation messages) and reference_tool_calls (expected tool calls). It returns a score between 0 and 1, where higher values indicate better performance.

Key Features

Two Evaluation Modes:

  1. Strict Order (default): Tool calls must match exactly in sequence

    • Use for: Sequential workflows where order matters
    • Example: Must search before filtering results

  2. Flexible Order: Tool calls can be in any order

    • Use for: Parallel operations where order doesn't matter
    • Example: Fetching weather for multiple cities simultaneously

Scoring:

  • Evaluates sequence alignment (correct tools in correct order)
  • Evaluates argument accuracy (correct parameters for each tool)
  • Final score = (argument accuracy) × (sequence aligned ? 1 : 0)

Example: Basic Usage

python
import asyncio
from ragas.metrics.collections import ToolCallAccuracy
from ragas.messages import AIMessage, HumanMessage, ToolCall

async def evaluate_tool_call_accuracy():
    # Define the conversation with tool calls
    user_input = [
        HumanMessage(content="What's the weather like in New York right now?"),
        AIMessage(
            content="The current temperature in New York is 75°F and it's partly cloudy.",
            tool_calls=[ToolCall(name="weather_check", args={"location": "New York"})],
        ),
        HumanMessage(content="Can you translate that to Celsius?"),
        AIMessage(
            content="Let me convert that to Celsius for you.",
            tool_calls=[
                ToolCall(
                    name="temperature_conversion", args={"temperature_fahrenheit": 75}
                )
            ],
        ),
    ]

    # Define expected tool calls
    reference_tool_calls = [
        ToolCall(name="weather_check", args={"location": "New York"}),
        ToolCall(name="temperature_conversion", args={"temperature_fahrenheit": 75}),
    ]

    # Evaluate
    metric = ToolCallAccuracy()
    result = await metric.ascore(
        user_input=user_input,
        reference_tool_calls=reference_tool_calls,
    )
    print(f"Tool Call Accuracy: {result.value}")

if __name__ == "__main__":
    asyncio.run(evaluate_tool_call_accuracy())

Output:

text
Tool Call Accuracy: 1.0

Example: Flexible Order Mode

For scenarios where tool calls can happen in parallel:

python
# Enable flexible order mode
metric = ToolCallAccuracy(strict_order=False)

user_input = [
    HumanMessage(content="Get weather for Paris and London"),
    AIMessage(
        content="Fetching weather data...",
        tool_calls=[
            ToolCall(name="weather_check", args={"location": "London"}),
            ToolCall(name="weather_check", args={"location": "Paris"}),
        ],
    ),
]

reference_tool_calls = [
    ToolCall(name="weather_check", args={"location": "Paris"}),
    ToolCall(name="weather_check", args={"location": "London"}),
]

result = await metric.ascore(
    user_input=user_input,
    reference_tool_calls=reference_tool_calls,
)
print(f"Score: {result.value}")  # 1.0 (order doesn't matter)

Scoring Examples

Perfect match:

python
# All tools called correctly with correct arguments
Expected: [weather_check(location="Paris"), translate(text="hello")]
Got:      [weather_check(location="Paris"), translate(text="hello")]
Score: 1.0

Partial argument match:

python
# Some arguments incorrect
Expected: [search(query="python", limit=10, sort="date")]
Got:      [search(query="python", limit=10, sort="relevance")]
Score: 0.66 (2 out of 3 arguments match)

Wrong order (strict mode):

python
# Correct tools but wrong sequence
Expected: [search(...), filter(...)]
Got:      [filter(...), search(...)]
Score: 0.0 (sequence not aligned)

Use Cases

  1. Agent Validation: Test if agents correctly use tools
  2. Regression Testing: Ensure tool calling doesn't degrade after changes
  3. Multi-Step Workflows: Validate complex sequential operations
  4. Tool Selection: Verify agents pick the right tool from many options

When to Use Different Metrics

Metric Use When
ToolCallAccuracy You care about exact tool sequence and arguments
ToolCallF1 You want precision/recall metrics for tool calling
AgentGoalAccuracy You care about outcome, not the specific tools used

Example: For "Book me a flight to Paris", if you only care that the booking succeeds (not which intermediate tools were called), use AgentGoalAccuracyWithReference instead.

Legacy API (Deprecated)

!!! warning "Deprecation Notice"
The legacy ToolCallAccuracy from ragas.metrics is deprecated and will be removed in v1.0. Please migrate to ragas.metrics.collections.ToolCallAccuracy which provides the same functionality with a modern API.

The legacy API can still be used but requires MultiTurnSample:

python
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import AIMessage, HumanMessage, ToolCall
from ragas.metrics import ToolCallAccuracy  # Legacy import

sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="What's the weather in New York?"),
        AIMessage(
            content="Checking weather...",
            tool_calls=[ToolCall(name="weather_check", args={"location": "New York"})],
        ),
    ],
    reference_tool_calls=[
        ToolCall(name="weather_check", args={"location": "New York"}),
    ],
)

scorer = ToolCallAccuracy()
score = await scorer.multi_turn_ascore(sample)

The legacy version also supported custom argument comparison metrics:

python
from ragas.metrics._string import NonLLMStringSimilarity
from ragas.metrics._tool_call_accuracy import ToolCallAccuracy

metric = ToolCallAccuracy()
metric.arg_comparison_metric = NonLLMStringSimilarity()

Tool Call F1

ToolCallF1 is a metric that return F1-score based on precision and recall of tool calls made by an agent, comparing them to a set of expected calls (reference_tool_calls). While ToolCallAccuracy provides a binary score based on exact order and content match, ToolCallF1 complements it by offering a softer evaluation useful for onboarding and iteration. It helps quantify how close the agent was to the expected behavior even if it over- or under-calls.

Formula

ToolCallF1 is based on classic IR metrics. It uses unordered matching: the order in which the tools are called does not impact the result, only the presence and correctness of tool names and parameters are considered.

$$
\text{Precision} = \frac{\text{tool calls that match both name and parameters}}{\text{tool calls that match both name and parameters} + \text{extra tool calls that were not expected}}
$$

$$
\text{Recall} = \frac{\text{tool calls that match both name and parameters}}{\text{tool calls that match both name and parameters} + \text{expected tool calls that were not made}}
$$

$$
\text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
$$

How is it different from Topic Adherence?

While both ToolCallF1 and TopicAdherenceScore uses precision, recall, and F1-score, they evaluate different aspects:

Metric Evaluates Based on
ToolCallF1 Correctness of tool executions Structured tool call objects
TopicAdherenceScore Whether the conversation stays on-topic Comparison of domain topics

Use ToolCallF1 when you want to track whether the agent correctly executed tools. Use TopicAdherenceScore when evaluating whether the content or intention stays within allowed topics.

Example: Basic Usage

python
import asyncio
from ragas.metrics.collections import ToolCallF1
from ragas.messages import HumanMessage, AIMessage, ToolCall

async def evaluate_tool_call_f1():
    # Define the conversation with tool calls
    user_input = [
        HumanMessage(content="What's the weather like in Paris today?"),
        AIMessage(
            content="Let me check that for you.",
            tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})],
        ),
        HumanMessage(content="And the UV index?"),
        AIMessage(
            content="Sure, here's the UV index for Paris.",
            tool_calls=[ToolCall(name="uv_index_lookup", args={"location": "Paris"})],
        ),
    ]

    # Define expected tool calls
    reference_tool_calls = [
        ToolCall(name="weather_check", args={"location": "Paris"}),
        ToolCall(name="uv_index_lookup", args={"location": "Paris"}),
    ]

    # Evaluate
    metric = ToolCallF1()
    result = await metric.ascore(
        user_input=user_input,
        reference_tool_calls=reference_tool_calls,
    )
    print(f"Tool Call F1: {result.value}")

if __name__ == "__main__":
    asyncio.run(evaluate_tool_call_f1())

Output:

text
Tool Call F1: 1.0

Example: Extra Tool Called

When the agent makes an extra tool call not in the reference:

python
user_input = [
    HumanMessage(content="What's the weather like in Paris today?"),
    AIMessage(
        content="Let me check that for you.",
        tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})],
    ),
    HumanMessage(content="And the UV index?"),
    AIMessage(
        content="Sure, here's the UV index and air quality for Paris.",
        tool_calls=[
            ToolCall(name="uv_index_lookup", args={"location": "Paris"}),
            ToolCall(name="air_quality", args={"location": "Paris"}),  # extra call
        ],
    ),
]

reference_tool_calls = [
    ToolCall(name="weather_check", args={"location": "Paris"}),
    ToolCall(name="uv_index_lookup", args={"location": "Paris"}),
]

result = await metric.ascore(
    user_input=user_input,
    reference_tool_calls=reference_tool_calls,
)
print(f"F1 Score: {result.value}")

Output:

text
F1 Score: 0.67

In this case:

  • TP = 2 (weather_check, uv_index_lookup)
  • FP = 1 (air_quality)
  • FN = 0
  • Precision = 2/3 = 0.67, Recall = 2/2 = 1.0, F1 = 0.67

Scoring Examples

Perfect match:

python
# All tools called correctly
Reference: [weather_check(location="Paris"), uv_index_lookup(location="Paris")]
Got:       [weather_check(location="Paris"), uv_index_lookup(location="Paris")]
F1 Score: 1.0

Missing tool call:

python
# One expected tool not called
Reference: [weather_check(...), uv_index_lookup(...)]
Got:       [weather_check(...)]
F1 Score: 0.67 (TP=1, FP=0, FN=1)

Wrong arguments:

python
# Tool name matches but args differ
Reference: [weather_check(location="Paris")]
Got:       [weather_check(location="London")]
F1 Score: 0.0 (no match, arguments must be exact)

Legacy API (Deprecated)

!!! warning "Deprecation Notice"
The legacy ToolCallF1 from ragas.metrics is deprecated and will be removed in v1.0. Please migrate to ragas.metrics.collections.ToolCallF1 which provides the same functionality with a modern API.

The legacy API can still be used but requires MultiTurnSample:

python
from ragas.metrics import ToolCallF1  # Legacy import
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import HumanMessage, AIMessage, ToolCall

sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="What's the weather like in Paris today?"),
        AIMessage(
            content="Let me check that for you.",
            tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})],
        ),
    ],
    reference_tool_calls=[
        ToolCall(name="weather_check", args={"location": "Paris"}),
    ],
)

scorer = ToolCallF1()
score = await scorer.multi_turn_ascore(sample)

Agent Goal Accuracy

Agent goal accuracy is a metric that can be used to evaluate the performance of the LLM in identifying and achieving the goals of the user. This is a binary metric, with 1 indicating that the AI has achieved the goal and 0 indicating that the AI has not achieved the goal.

With Reference

AgentGoalAccuracyWithReference evaluates whether the agent achieved the user's goal by comparing the workflow's end state against a provided reference outcome. The reference represents the expected/ideal outcome.

text
/* Detailed source-code truncated for AI context efficiency. */

Output

text
Agent Goal Accuracy: 1.0

Without Reference

AgentGoalAccuracyWithoutReference evaluates whether the agent achieved the user's goal without requiring a reference. The metric infers both the user's intended goal and the achieved outcome from the conversation, then compares them.

text
/* Detailed source-code truncated for AI context efficiency. */

Output

text
Agent Goal Accuracy: 1.0

Legacy API (Deprecated)

!!! warning "Deprecation Notice"
The legacy AgentGoalAccuracyWithReference and AgentGoalAccuracyWithoutReference from ragas.metrics are deprecated and will be removed in v1.0. Please migrate to ragas.metrics.collections which provides the same functionality with a modern API.

The legacy API can still be used but requires MultiTurnSample:

python
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage
from ragas.metrics import AgentGoalAccuracyWithReference  # Legacy import
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))

sample = MultiTurnSample(
    user_input=[...],  # conversation messages
    reference="Table booked at one of the chinese restaurants at 8 pm",
)
scorer = AgentGoalAccuracyWithReference(llm=evaluator_llm)
score = await scorer.multi_turn_ascore(sample)

File: docs/concepts/metrics/available_metrics/answer_correctness.md

Answer Correctness

The assessment of Answer Correctness involves gauging the accuracy of the generated answer when compared to the ground truth. This evaluation relies on the ground truth and the answer, with scores ranging from 0 to 1. A higher score indicates a closer alignment between the generated answer and the ground truth, signifying better correctness.

Answer correctness encompasses two critical aspects: semantic similarity between the generated answer and the ground truth, as well as factual similarity. These aspects are combined using a weighted scheme to formulate the answer correctness score. Users also have the option to employ a 'threshold' value to round the resulting score to binary, if desired.

!!! note "Embedding Requirement"
AnswerCorrectness requires embeddings for semantic similarity calculation. When using evaluate() without explicitly providing embeddings, Ragas will automatically match the embedding provider to your LLM provider. For example, if you use Gemini as your LLM, Google embeddings will be used automatically (no OpenAI API key needed). You can also provide embeddings explicitly for full control.

!!! example
Ground truth: Einstein was born in 1879 in Germany.

text
**High answer correctness**: In 1879, Einstein was born in Germany.

**Low answer correctness**: Einstein was born in Spain in 1879.

Example

python
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import AnswerCorrectness

# Setup LLM and embeddings
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)

# Create metric
scorer = AnswerCorrectness(llm=llm, embeddings=embeddings)

# Evaluate
result = await scorer.ascore(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    reference="The first superbowl was held on January 15, 1967"
)
print(f"Answer Correctness Score: {result.value}")

Output:

text
Answer Correctness Score: 0.95

!!! note "Synchronous Usage"
If you prefer synchronous code, you can use the .score() method instead of .ascore():

text
```python
result = scorer.score(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    reference="The first superbowl was held on January 15, 1967"
)
```

Calculation

Let's calculate the answer correctness for the answer with low answer correctness. It is computed as the sum of factual correctness and the semantic similarity between the given answer and the ground truth.

Factual correctness quantifies the factual overlap between the generated answer and the ground truth answer. This is done using the concepts of:

  • TP (True Positive): Facts or statements that are present in both the ground truth and the generated answer.
  • FP (False Positive): Facts or statements that are present in the generated answer but not in the ground truth.
  • FN (False Negative): Facts or statements that are present in the ground truth but not in the generated answer.

In the second example:

  • TP: [Einstein was born in 1879]
  • FP: [Einstein was born in Spain]
  • FN: [Einstein was born in Germany]

Now, we can use the formula for the F1 score to quantify correctness based on the number of statements in each of these lists:

$$
\text{F1 Score} = {|\text{TP} \over {(|\text{TP}| + 0.5 \times (|\text{FP}| + |\text{FN}|))}}
$$

Next, we calculate the semantic similarity between the generated answer and the ground truth. Read more about it here.

Once we have the semantic similarity, we take a weighted average of the semantic similarity and the factual similarity calculated above to arrive at the final score. You can adjust this weightage by modifying the weights parameter.

Legacy Metrics API

The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above.

!!! warning "Deprecation Timeline"
This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above.

Example with Dataset

python
from datasets import Dataset 
from ragas.metrics import answer_correctness
from ragas import evaluate

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset,metrics=[answer_correctness])
score.to_pandas()

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#evaluation Browse evaluation catalog → #llm Browse llm catalog → #llmops Browse llmops catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →