### Index
---
title: "Getting started with Pydantic Logfire"
description: "Logfire is Pydantic's end-to-end AI engineering platform for seeing what your agents, services, applications, and hosts are doing. Pick a quickstart and send your first trace."
---
# Getting started with Pydantic Logfire
Logfire is Pydantic's end-to-end AI engineering platform. It shows you what your agents, services, applications, and hosts are actually doing: every LLM call, request, and query, and how long each one took. Built on OpenTelemetry (OTel), the open standard for traces, metrics, and logs, it spans your whole stack, from AI agents to the databases and servers behind them.
New here? [Create a free account](https://logfire.pydantic.dev/login), then pick a starting point below, or follow a [guided path for your role](get-started/choose-your-path.md).
## Quickstarts
```
/* Detailed source-code truncated for AI context efficiency. */
```bash
logfire auth && python -c 'import logfire; logfire.configure()'
```
/* Detailed source-code truncated for AI context efficiency. */
````
- [__Send your first trace__](first-trace.md)
Install the SDK and watch a trace arrive in the Live view, in about 5 minutes.
- [__Monitor your infrastructure__](guides/web-ui/hosts.md)
Monitor hosts, Kubernetes, and cloud infrastructure alongside your application.
## Explore
- [__Understand agent observability__](ai-observability.md)
See how one trace connects an agent's model calls, tools, and the services behind them.
- [__Watch it live__](guides/web-ui/live.md)
See traces and logs stream in, and drill into any span to read its full detail.
- [__Query with SQL__](guides/web-ui/explore.md)
Slice your data with the SQL you already know, not a proprietary query language.
- [__Dashboards and alerts__](guides/web-ui/dashboards.md)
Chart what matters and get notified in Slack when it changes.
- [__Evaluate and ship AI features__](evaluate/overview.md)
Run evaluations, manage prompts, and route models through the AI gateway.
- [__Metrics explorer__](guides/web-ui/metrics-explorer.md)
Browse the metrics you're sending and break any of them down by dimension, no SQL required.
- [__Integrations__](integrations/index.md)
FastAPI, Django, SQLAlchemy, HTTPX, and 40+ more, each with one line of setup.
## New to observability?
[Core concepts](concepts.md) explains spans, traces, and logs, and how to read them in Logfire. [Read why Logfire exists](why.md).
---
### Comparisons/Arize Phoenix
---
title: "Logfire vs Arize Phoenix"
---
# Logfire vs Arize Phoenix
Arize Phoenix is an ML observability platform focused on model monitoring, drift detection, and LLM tracing. Logfire is an AI-native observability platform built on OpenTelemetry. While both help you observe AI applications, they come from different backgrounds and serve different needs.
## Quick Comparison
| Feature | Pydantic Logfire | Arize Phoenix |
|----------------------|----------------------------------------------------------------|-----------------------------------------------------------------|
| **Primary Focus** | AI observability for agents and apps | ML model monitoring |
| **Strength** | AI + application tracing | Drift detection, model performance |
| **Non-AI Tracing** | Full support | Limited |
| **Language Support** | Python, JS/TS, Rust (SDKs) + any OTel | Python-focused |
| **Evals** | Web UI (datasets, experiments, live monitoring) + code-based via `pydantic-evals` | Web UI + code-based via external library |
| **Pricing** | Per-span ($2/million)* | Usage-based |
| **Setup** | 3 lines of code | OTel-based (several lines of code) |
| **SQL Queries** | Yes (Postgres-compatible) | No. Use `SpanQuery` DSL |
*Logfire Cloud pricing (Team or Growth plans). Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5).
## When to Choose Logfire
- **Application observability:** Scalable AI Observability for LLMs, agents, and apps
- **Polyglot architecture:** You have services in all OTel supported languages, with native SDKs for Python, TypeScript/JavaScript, and Rust
- **Developer experience:** You want the quickest possible setup
- **SQL analysis:** You prefer querying with SQL rather than learning a new querying language
- **Unified tooling:** You don't want separate tools for AI and application monitoring
- **MCP Integration:** Ask questions about your tracing data to your favourite coding assistant
## When to Choose Arize Phoenix
- **ML model monitoring:** You need drift detection and model performance analytics
- **Data science workflows:** Your team thinks in ML metrics, not application traces
- **Existing Arize investment:** You're already using Arize's broader platform
- **Experimentation focus:** You're doing heavy ML experimentation and need those tools
## Key Differences Explained
### Application Observability vs ML Monitoring
**Arize Phoenix** comes from the ML monitoring world. It excels at:
- Model drift detection
- Feature importance analysis
- ML performance metrics
- Evaluation workflows
**Logfire** comes from the application observability world. It excels at:
- AI, LLMs, and agents tracing data
- Code-base evaluation workflows visualisation
- Database, API, and service monitoring
- Real-time debugging with Live View
- SQL-based analysis
- Native connection to MCP to debug your app
### Full-Stack Visibility
When your AI application fails, you need to know why. Was it the model? The data? The infrastructure?
**Arize Phoenix** shows you the AI layer but has limited visibility into your application infrastructure.
**Logfire** is your AI observability tool that can *also* show you everything: user request, database query, LLM call, and API response. One trace, complete context.
### Developer Experience
**Logfire** setup:
```python skip="true" skip-reason="incomplete"
import logfire
logfire.configure()
logfire.instrument_openai()
```
Three lines, and you're observing AI calls with full application context.
**Arize Phoenix** requires more configuration, especially for non-AI instrumentation.
### Query Interface: Essential for Agentic Coding
**Logfire** uses SQL with PostgreSQL-compatible syntax. This is a significant advantage for AI-assisted development:
- **AI assistants excel at SQL**: GPT-5, Claude, and [coding agents write excellent SQL](https://pydantic.dev/articles/logfire-mcp-is-awesome?utm_source=arize_comparison_docs)
- **Agentic workflows**: When coding agents debug your AI application, they can write arbitrary queries
- **Familiar syntax**: No new query language to learn
- **No artificial limitations**: Ask any question, get any answer
When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With proprietary interfaces, it's constrained to anticipated queries.
**Arize Phoenix** has its own query interface optimized for ML metrics but less flexible for ad-hoc analysis.
## Complementary Use
Logfire and Arize Phoenix can work together:
- Use Logfire for AI observability and real-time debugging for your app
- Use Arize for specialized ML model monitoring and drift detection
Both support OpenTelemetry, making integration possible.
## Summary
**Choose Logfire** for AI application observability with excellent full-stack tracing and support.
**Choose Arize Phoenix** for specialized ML model monitoring, drift detection, and data science workflows.
**Consider both** if you need deep ML experimentation alongside first-class AI observability and full-stack monitoring.
---
### Comparisons/Braintrust
---
title: "Logfire vs. Braintrust"
description: "Compare Logfire and Braintrust for AI evaluation, production observability, pricing, querying, prompts, and human review."
---
# Logfire vs. Braintrust
Braintrust is built around LLM evaluation. Logfire gives you serious evaluation workflows without separating them from the application, services, and infrastructure that produced the result.
If you already have a Braintrust evaluation suite, you do not need to rewrite it to try Logfire. Point a verified Python or TypeScript `Eval` flow at Logfire, run it as usual, and review the resulting experiments and traces in the Logfire UI. Follow [Move from Braintrust to Logfire](migrate-from-braintrust.md) for the two-variable switch.
## Quick comparison
| Capability | Logfire | Braintrust |
| --- | --- | --- |
| Primary focus | AI evaluation plus full-stack observability | AI evaluation and LLM observability |
| Existing Braintrust `Eval` code | Compatibility preview for verified Python and TypeScript flows | Native |
| Offline evaluations | Pydantic Evals in code, with datasets, experiments, comparisons, and case review in Logfire | SDK, UI, playground, remote evaluation, and CI workflows |
| Production evaluation | Pydantic AI online evaluators and Live Evaluations | Server-side online scoring rules |
| Evaluator results | Assertions, numeric scores, and categorical labels | Numeric scorers and categorical classifiers |
| Human review | Run annotations and annotation queues | Human review, assignments, labels, corrections, and datasets |
| Production context | AI calls, agents, HTTP services, databases, browser activity, logs, metrics, and infrastructure | AI application traces, scores, and project logs |
| Telemetry | OpenTelemetry-native traces, logs, and metrics | Braintrust spans and scores, with OpenTelemetry ingest |
| Querying | PostgreSQL-compatible SQL across evaluation and application telemetry, plus MCP access | SQL and BTQL over Braintrust resources |
| Prompt iteration | Prompt Playground and Prompt Management | Playgrounds, hosted prompts, and deployment environments |
| Model routing | Logfire AI Gateway | Braintrust gateway and proxy |
| Usage billing | Telemetry records, with no separate fee for evaluator scores | Processed data, evaluator scores, and retention |
## When to choose Logfire
- **You want the evidence behind a failed evaluation.** Follow a case into its model calls, tool calls, retrieval, database work, HTTP requests, logs, and infrastructure telemetry.
- **You want one system for development and production.** Compare offline experiments, monitor live evaluators, annotate real agent runs, and investigate the surrounding application from the same project.
- **You want to query freely.** Use PostgreSQL-compatible SQL for joins, common table expressions, window functions, dashboards, and alerts, or let a coding agent investigate through the Logfire MCP server.
- **You want OpenTelemetry-native observability.** Instrument the application once and keep the data useful beyond a single AI framework or evaluation SDK.
- **You already have Braintrust eval code.** Redirect a compatible suite first, judge the experience with your own data, and move to native Pydantic Evals only when it improves your code.
- **You do not want evaluator coverage to create a separate score bill.** Logfire meters telemetry records rather than charging for every evaluator result.
## When Braintrust may fit better
- **Your workflow depends on Braintrust-hosted resources.** Braintrust provides hosted datasets, prompts, functions, remote evaluations, and server-side scoring rules that Logfire's compatibility endpoint does not translate.
- **You want Braintrust's managed evaluation environment.** Its playground, Loop agent, dataset snapshots, and UI-run workflows are designed around Braintrust resources.
- **Your team wants an AI-centered system.** If application and infrastructure observability belong elsewhere by design, Braintrust's product boundary may match that workflow.
## Why teams move evaluation work to Logfire
### The average score is the start of the investigation
Both products can show that an experiment regressed. The useful next question is why.
In Logfire, an experiment case is a trace in the same system as the work behind it. When that work is instrumented, you can move from a failed evaluator to the prompt, model response, retrieval context, tool failure, slow query, downstream timeout, or service error without reconstructing the request in another observability product.
That same connection works in the other direction. A surprising production run can become a reviewed example, a dataset case, or evidence for the next evaluation cycle.
### More evaluators do not create a new billing dimension
Braintrust meters [scores and processed data separately](https://www.braintrust.dev/pricing). Its public Starter plan includes 10,000 scores and then charges $2.50 per 1,000; Pro includes 50,000 and then charges $1.50 per 1,000.
Logfire does not have a separate score meter. Evaluator results travel with the experiment telemetry. Logfire includes 10 million logs, spans, and metrics each month, then charges [$2 per million additional records](https://pydantic.dev/pricing) on Team and Growth plans.
The units are not identical, so a universal cost conversion would be misleading. The practical difference is simpler: adding another evaluator changes Braintrust score usage, while Logfire does not add a per-score charge.
### SQL gives people and coding agents the same investigative surface
Logfire exposes [PostgreSQL-compatible SQL](../reference/sql.md) over evaluation and application telemetry. You can join related data, use common table expressions and window functions, turn a query into a dashboard or alert, and give a coding agent the same access through the [Logfire MCP server](../how-to-guides/mcp-server.md).
Braintrust provides SQL and Braintrust Query Language (BTQL) over its projects, experiments, datasets, and logs. Its built-in views and Loop agent keep that investigation inside Braintrust's AI development model.
### You can evaluate Logfire with your existing suite
Most migrations ask you to rewrite the workload before you can judge the destination. Logfire's Braintrust compatibility endpoint gives verified Python and TypeScript `Eval` suites a shorter path:
1. Change the Braintrust app URL and API key environment variables.
2. Run the same evaluation command.
3. Review the completed experiment in Logfire.
4. Decide which datasets, evaluators, prompts, and production workflows are worth moving to native Logfire and Pydantic APIs.
The endpoint does not contact Braintrust, import historical data, or pretend to support every hosted Braintrust feature. The [migration guide](migrate-from-braintrust.md) documents the exact supported boundary and the native replacement for each surrounding workflow.
## Summary
**Choose Logfire** when evaluations need to connect to production behavior, SQL analysis, coding-agent workflows, and the rest of your observability data, or when you want to test the move with existing Braintrust `Eval` code.
**Choose Braintrust** when Braintrust-hosted evaluation resources and its managed AI-only workflows are the product you want.
Ready to compare with your own suite? [Send your next Braintrust evaluation to Logfire](migrate-from-braintrust.md).
---
### Comparisons/Datadog
---
title: "Logfire vs Datadog"
---
# Logfire vs Datadog
Datadog is a comprehensive enterprise monitoring platform. Logfire is an AI-native observability platform built on OpenTelemetry. While both offer APM capabilities, their architectures and pricing models are fundamentally different.
## Quick Comparison
| Feature | Logfire | Datadog |
|-------------------------|--------------------------------|---------|
| **Architecture** | OpenTelemetry-native | Proprietary agents |
| **Pricing Model** | Per-span ($2/million)* | Per-host + ingestion + custom metrics |
| **Host Fees** | None | $15-40/host/month |
| **AI/LLM Support** | First-class, one function call | Add-on, separate product |
| **Query Language** | SQL (Postgres-compatible) | Proprietary |
| **Setup Complexity** | 3 lines of code | Agent deployment per host |
| **Autoscaling Impact** | Linear cost increase | High-water-mark billing spikes |
| **Dashboards & alerts** | Included, SQL-based | Included, proprietary query language |
*Logfire Cloud pricing (Team or Growth plans). Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5).
## Pricing Comparison
| Scenario | Datadog | Logfire Cloud* | Savings |
|----------|------------|-------------------|---------|
| Hobby (2 hosts, 5M spans) | $101/mo | $0/mo (free tier) | 100% |
| Startup (10 hosts, 100M spans) | $229/mo | $229/mo | - |
| Scale-up (50-150 hosts, 500M spans) | $9,860/mo | $1,229/mo | 87% |
| High-volume (500 hosts, 2B spans) | $33,550/mo | $4,229/mo | 87% |
*Logfire Cloud pricing (Team or Growth plans -> $2/million spans). Enterprise pricing (self-hosting, SSO, custom retention) available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5).
## When to Choose Logfire
- **AI-native applications:** You're building with LLMs and want first-class AI observability
- **Cost predictability:** You want simple per-span pricing without host fees or metric surcharges
- **OpenTelemetry investment:** You want portable instrumentation, not vendor lock-in
- **Autoscaling/serverless:** Your infrastructure scales dynamically, and you don't want billing surprises
- **SQL querying:** You prefer familiar SQL over learning a proprietary query language
## When to Choose Datadog
- **Existing investment:** You're already deeply integrated with Datadog's ecosystem
- **Enterprise requirements:** You need specific compliance certifications or integrations Datadog offers
## Key Differences Explained
### Pricing Model: Per-Span vs Per-Host
**Datadog** charges per host ($15-40/month) plus data ingestion plus custom metrics. This creates several pain points:
- **Autoscaling penalties:** High-water-mark billing means a traffic spike that adds 10 temporary hosts bills you for all 10 all month
- **Custom metrics explosion:** OpenTelemetry metrics are treated as expensive "custom metrics" with cardinality charges
- **Unpredictable costs:** Multiple billing dimensions make forecasting difficult
**Logfire** includes 10M spans/logs/metrics per month for free. [Paid plans](https://pydantic.dev/pricing/?utm_source=datadog_comparison_docs) start at $49/mo with the same 10M included, then $2 per million after that. That's it. No host fees, no custom metrics fees, no ingestion surprises.
### AI/LLM Support
**Datadog** added LLM observability as a separate product. It works, but AI isn't central to the platform's design.
**Logfire** was built for the AI era. [One function call](https://pydantic.dev/docs/logfire/integrations/?utm_source=datadog_comparison_docs) (`logfire.instrument_openai()`) gives you:
- Token tracking and cost monitoring
- LLM-specific panels for conversations
- Tool call inspection
- Streaming support
- Integration with MCP server to query & debug tracing data
And observability is only half the story: Logfire also ships [evals in the web UI](https://pydantic.dev/docs/logfire/evaluate/evals/?utm_source=datadog_comparison_docs) (datasets, experiments, and live production monitoring), [prompt management](https://pydantic.dev/docs/logfire/prompt-management/?utm_source=datadog_comparison_docs) with a playground, and an AI gateway with key management and spending controls: the full AI engineering loop in one platform, not a bolted-on add-on.
### OpenTelemetry
**Datadog** uses proprietary agents. While they support OTel export, it's not the native path.
**Logfire** is OpenTelemetry-native with [first-class integrations for most technologies](https://pydantic.dev/docs/logfire/integrations/?utm_source=datadog_comparison_docs). Any OTel instrumentation works automatically. Your instrumentation is portable: if you ever want to switch, your code doesn't change.
### Query Language: Essential for Agentic Coding
**Datadog** uses a proprietary query language for dashboards and analysis. This creates limitations:
- Learning curve for humans and AI alike
- Coding agents are constrained to anticipated queries
- Vendor lock-in for your analysis workflows
**Logfire** uses SQL with PostgreSQL-compatible syntax. This is a significant advantage for AI-assisted development:
- **No artificial limitations**: Ask any question, get any answer
- **AI assistants excel at SQL**: GPT-5, Claude, and [coding agents write excellent SQL](https://pydantic.dev/articles/logfire-mcp-is-awesome?utm_source=datadog_comparison_docs)
- **Agentic workflows**: When coding agents debug your AI application, they can write arbitrary queries
- **Familiar syntax**: No new query language to learn
When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With proprietary DSLs, it's constrained to what someone anticipated.
## Migration Path
If you're considering migrating from Datadog:
1. Logfire accepts standard OTel data. Start sending traces without changing your app.
2. Run both in parallel during transition
3. Gradually migrate instrumentation to Logfire SDKs for the best experience
## Summary
**Choose Logfire** for AI-native observability with predictable pricing and OpenTelemetry portability.
**Choose Datadog** if you're deeply invested in their ecosystem or need their specific infrastructure monitoring capabilities.
---
### Comparisons/Grafana
---
title: "Logfire vs Grafana"
---
# Logfire vs Grafana
Grafana is a powerful open-source visualization platform that works with various data sources including Tempo (traces), Loki (logs), and Prometheus (metrics). Logfire is a managed observability platform with integrated visualization. Both can display your observability data, but they take very different approaches.
## Quick Comparison
| Feature | Logfire | Grafana Stack |
|---------|---------|---------------|
| **Setup** | One-line: `logfire.configure()` | Deploy Tempo, Loki, Prometheus, configure each |
| **Configuration** | Minimal | Extensive dashboard building required |
| **AI/LLM Support** | First-class, purpose-built panels | Manual dashboard creation |
| **Live View** | Built-in real-time view | Complex to configure reliably |
| **Query Language** | SQL (Postgres-compatible) | PromQL, LogQL, TraceQL (different for each) |
| **Maintenance** | Zero | Ongoing tuning and management |
## When to Choose Logfire
- **Ready-to-use experience:** You want observability that works immediately
- **AI/LLM applications:** You need purpose-built AI observability features
- **SQL familiarity:** You prefer one familiar query language
- **Minimal ops burden:** You don't want to maintain observability infrastructure
- **Live debugging:** You want real-time visibility that actually works
## When to Choose Grafana
- **Existing investment:** You're already running the Grafana stack
- **Custom dashboards:** You need highly customized visualizations
- **Multi-source aggregation:** You want to combine data from many different sources
- **Full control:** You want to own and customize every aspect
## Key Differences Explained
### Ready-to-Use vs Build-It-Yourself
Here's what a Grafana user told us:
> "The Grafana Tempo and Prometheus, they are just like collectors. They are not doing anything for you. Then you need to set up everything for yourself... no matter how hard we tried and we actually did pour significant developer time into that, I still cannot have reasonable dashboards similar to Logfire front end."
**Grafana** is powerful but requires significant investment:
- Deploy and configure Tempo, Loki, Prometheus
- Learn three different query languages (PromQL, LogQL, TraceQL)
- Build dashboards from scratch
- Tune performance and retention
**Logfire** works immediately:
- Three lines of code to start
- Purpose-built UI that understands your data
- No dashboard building required
- AI-specific panels out of the box
### Live View
> "What I really love about Logfire is the Live view. Having this reasonably robust in Grafana plus Grafana Tempo... I could not get it work."
Real-time observability is table stakes for debugging. Logfire's Live View shows you what's happening right now with "pending spans": requests in flight, operations in progress.
Getting equivalent functionality in Grafana requires significant configuration and often doesn't work as reliably.
### Query Language: Essential for Agentic Coding
**Grafana** requires learning multiple query languages:
- PromQL for metrics
- LogQL for logs
- TraceQL for traces
**Logfire** uses SQL with PostgreSQL-compatible syntax for everything. This is a significant advantage for AI-assisted development:
- **One language**: No context-switching between query syntaxes
- **AI assistants excel at SQL**: GPT-5, Claude, and coding agents write excellent SQL
- **Agentic workflows**: When coding agents debug your AI application, they can write arbitrary queries
- **Familiar syntax**: No new query language to learn
When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With multiple proprietary DSLs, it's constrained to what someone anticipated for each query type.
### AI/LLM Observability
**Grafana** has no native AI support. You'd need to:
- Instrument manually or with OpenLLMetry
- Build custom dashboards for AI metrics
- Create your own panels for conversations, tool calls, etc.
**Logfire** has purpose-built AI features:
- One function call instruments your AI framework
- LLM panels that understand conversations and tool calls
- Token tracking and cost monitoring built in
## Migration Path
If you're running Grafana and want to try Logfire:
1. **Run both:** Logfire can run alongside your existing Grafana setup
2. **Compare experience:** Try the same debugging task in both
3. **Gradual transition:** Start with new projects or AI workloads
Since both support OpenTelemetry, you can even send the same traces to both during evaluation.
## Summary
**Choose Logfire** for observability that works out of the box with purpose-built AI support.
**Choose Grafana** if you're invested in the ecosystem and willing to build and maintain custom dashboards.
---
### Comparisons/Index
---
title: "Logfire vs Alternatives"
---
# Logfire vs Alternatives
## AI Observability That Actually Works
Logfire is AI observability that outperforms because it sees your entire application.
**Here's the problem with AI-only tools:** They've tied one hand behind their back. When debugging AI applications, you need to know:
- What triggered the LLM call?
- What did the AI access (databases, APIs, tools)?
- What happened with the response?
AI-only tools can only show you the middle part. They're missing the context that makes debugging possible.
**Logfire sees everything.** When your AI agent fails, you don't just see "LLM returned error". You see the database timeout that caused it, the API rate limit that preceded it, and the user request that started it all.
This isn't a bonus feature. It's what makes AI debugging actually work.
## Built on Standards, Works Everywhere
Logfire is built on OpenTelemetry, the industry standard for observability. This means:
- **Any OTel instrumentation works:** If your framework has OpenTelemetry support, it works with Logfire. No special integration needed.
- **Any language works:** Python, JavaScript/TypeScript, Rust, Go, Java, .NET. If it speaks OTel, it works.
- **Your instrumentation is portable:** You're never locked in. Even if you use the Logfire SDK for convenience, you can configure it to send data to other backends
- **Works with any OTel-compatible tools:** Integrate with your existing stack
We provide first-party SDKs for Python, JavaScript/TypeScript, and Rust that offer convenient wrappers, but they're optional. Standard OpenTelemetry libraries work perfectly.
## Comparison Guides
### AI Observability Tools
- [Logfire vs Langfuse](langfuse.md) - Open-source LLM observability
- [Logfire vs Arize Phoenix](arize-phoenix.md) - ML observability platform
- [Logfire vs LangSmith](langsmith.md) - LangChain's observability tool
- [Logfire vs Braintrust](braintrust.md) - AI evaluation and observability
- [Move from Braintrust to Logfire](migrate-from-braintrust.md) - keep a verified `Eval` suite, then migrate each workflow at your own pace
### Traditional Observability Platforms
- [Logfire vs Datadog](datadog.md) - Enterprise APM comparison
- [Logfire vs Sentry](sentry.md) - Error monitoring comparison
- [Logfire vs SigNoz](signoz.md) - Open-source observability
- [Logfire vs Grafana](grafana.md) - Grafana stack comparison
## When to Choose Logfire
Logfire is a great fit when you:
- **Want one stack, not two** - AI + infrastructure observability unified, not an agent framework plus a separate APM tool
- **AI enabled workflows** - Use the [Logfire MCP server](https://pydantic.dev/docs/logfire/guides/mcp-server/?utm_source=comparison_docs) to query your app's telemetry data, analyze distributed traces, and fix errors with AI using Logfire's OTel-native API
- **Have polyglot architectures** - Python backend + TypeScript frontend, microservices in different languages
- **Want exceptional Python, TS/JS, Rust support** - While also supporting other languages via OTel
- **Want evals both ways** - [code-first evals with pydantic-evals](https://pydantic.dev/docs/logfire/evaluate/datasets/?utm_source=comparison_docs) that test any Python function, plus a [web UI for datasets, experiments, and live production monitoring](https://pydantic.dev/docs/logfire/evaluate/evals/?utm_source=comparison_docs)
- **Want the full AI engineering loop in one place** - [prompt management](https://pydantic.dev/docs/logfire/prompt-management/?utm_source=comparison_docs), a playground, and an [AI gateway](../reference/advanced/gateway/index.md) with key management and spending controls, next to your traces
- **Prefer SQL-based querying** - Familiar PostgreSQL syntax over proprietary query languages
- **Value simple, predictable pricing** - $2/million spans, scales to billions of spans per month
- **Need enterprise features** - [SOC2, HIPAA, self-hosting](https://pydantic.dev/pricing?utm_source=comparison_docs#enterprise) options
---
### Comparisons/Langfuse
---
title: "Logfire vs Langfuse"
---
# Logfire vs Langfuse
Both Logfire and Langfuse help you observe AI/LLM applications, but they take fundamentally different approaches. Langfuse focuses specifically on LLM tracing, while Logfire is an AI-native full-stack observability platform.
## Quick Comparison
| Feature | Logfire | Langfuse |
|---------|---------|----------|
| **Focus** | Full-stack observability | LLM-specific observability |
| **Language Support** | Python, JS/TS, Rust SDKs + any OTel | Python, JS/TS SDKs |
| **Query Language** | SQL (Postgres-compatible) | Custom UI / API |
| **Self-hosting** | Enterprise option | Open source, self-hostable |
| **Pricing Model** | Per-span ($2/million)* | Per-event + usage-based |
| **Python Support** | First-class (Pydantic team) | Good |
| **Non-AI Tracing** | Full support | Limited |
| **LLM Features** | Token tracking, costs, panels, evals (UI + code), prompt mgmt, playground, AI gateway | Token tracking, costs, evals, prompt mgmt |
| **OpenTelemetry** | Native | Export support |
*Logfire Cloud pricing (Team or Growth plans). Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5).
## When to Choose Logfire
- **Unified observability:** You want AI monitoring and application monitoring in one tool
- **Built for scale:** You need to handle massive data loads, whether SaaS or self-hosted
- **SQL-based analysis:** You prefer querying with familiar SQL (and AI assistants write excellent SQL)
- **Minimal integration effort:** You want to add tracing with minimal code changes
- **Pydantic/FastAPI stack:** You're already using Pydantic or FastAPI
- **Polyglot architecture:** You have services in multiple languages that need unified tracing
## When to Choose Langfuse
- **Lighter workloads:** You have moderate tracing volumes and performance isn't critical
- **Self-hosting priority:** You need open-source self-hosting (Logfire self-hosting is enterprise)
- **LLM-only focus:** You only need LLM tracing and don't want broader observability features
## Key Differences Explained
### Full-Stack vs AI-Only
**Langfuse** focuses specifically on LLM observability: tracing LLM calls, managing prompts, running evaluations.
**Logfire** provides native AI observability with full-stack monitoring support. When your AI agent makes a database call that fails, you see both the LLM trace AND the database error in one view.
This matters because AI applications don't exist in isolation. They call APIs, query databases, process files. Unified observability means faster debugging.
### Query Language: Essential for Agentic Coding
**Langfuse** uses a custom UI and API for querying data.
**Logfire** uses SQL with PostgreSQL-compatible syntax. This is a significant advantage for AI-assisted development:
- **No artificial limitations**: Ask any question, get any answer
- **AI assistants excel at SQL**: GPT-5, Claude, and coding agents write excellent SQL
- **Agentic workflows**: When coding agents debug your AI application, they can write arbitrary queries to [understand and debug production behavior](https://pydantic.dev/articles/logfire-mcp-is-awesome?utm_source=langfure_comparison_webpage)
- **Familiar syntax**: all in SQL, no new query language to learn
When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With custom APIs or DSLs, it's constrained to anticipated queries.
### Pricing Model
**Langfuse Cloud** charges per event plus usage-based pricing.
**Logfire Cloud (Team and Growth)** Free tier includes 10M spans/logs/metrics per month. Plans start at $49/month, then charge $2 per million after the included allowance. Simple and predictable.
## Integration Comparison
Both support major LLM providers and frameworks:
| Integration | Logfire | Langfuse |
|-------------|---------|----------|
| OpenAI | ✅ | ✅ |
| Anthropic | ✅ | ✅ |
| LangChain | ✅ | ✅ (native) |
| LlamaIndex | ✅ | ✅ |
| Pydantic AI | ✅ (native) | ✅ |
| FastAPI | ✅ | ❌ |
| Next.js | ✅ (via JS SDK) | ❌ |
| Express | ✅ (via JS SDK) | ❌ |
| PostgreSQL | ✅ | ❌ |
| Redis | ✅ | ❌ |
Logfire's broader integration set reflects its full-stack, polyglot approach.
## Migration
If you're currently using Langfuse and considering Logfire:
1. Both are OpenTelemetry-compatible, so instrumentation concepts transfer
2. Logfire's Pydantic AI integration is a drop-in replacement
3. Consider running both during transition
## Summary
**Choose Logfire** if you want AI observability with full-stack monitoring across your entire stack with excellent support.
**Choose Langfuse** if you specifically want LLM-focused tooling and prefer open-source self-hosting.
---
### Comparisons/Langsmith
---
title: "Logfire vs LangSmith"
---
# Logfire vs LangSmith
LangSmith is the observability and evaluation platform from the LangChain team. Logfire, combined with Pydantic AI, offers a production-grade alternative built on software engineering fundamentals.
## Quick Comparison
| Feature | Logfire + Pydantic AI | LangSmith + LangChain |
|---------|---------------------|----------------------|
| **Foundation** | Pydantic (500M+ downloads), type-safe | LangChain, flexible/dynamic |
| **Structured Outputs** | Schema-validated responses | String parsing, partial validation |
| **Observability Scope** | Full-stack (AI + systems) | LLM-focused |
| **Standards** | 100% OpenTelemetry GenAI | Proprietary format |
| **Data Retention** | Configurable | 14 days (standard), 400 days (extended) |
| **Query Interface** | SQL (Postgres-compatible) | Custom UI |
| **Framework Lock-in** | Works with any framework | Best with LangChain |
## Pricing Comparison
| Tier | LangSmith | Logfire Cloud* |
|------|-----------|----------------|
| Standard (14 days retention) | ~$500/1M traces | ~$6-10/1M traces |
| Extended (400 days retention) | ~$5,000/1M traces | ~$6-10/1M traces |
*Logfire Cloud pricing. Enterprise pricing available on request.
At scale, Logfire can be 50-100X cheaper than LangSmith. This isn't about Logfire being a "budget option". It's about architectural efficiency. Logfire's OpenTelemetry-native design avoids the overhead of proprietary trace formats, and our query engine (DataFusion) retrieves traces significantly faster.
## When to Choose Logfire + Pydantic AI
- **Production-grade requirements:** You need type safety, validation, and real software engineering practices
- **Full-stack visibility:** You want AI observability AND system observability in one tool
- **Open standards:** You want OTel-compatible instrumentation that isn't locked to one vendor
- **Framework flexibility:** You use multiple AI frameworks or may switch in the future
- **Long-term cost efficiency:** You're scaling and costs matter
- **The full AI engineering loop:** Evals (in code and the [web UI](https://pydantic.dev/docs/logfire/evaluate/evals/?utm_source=comparison_docs)), [prompt management](https://pydantic.dev/docs/logfire/prompt-management/?utm_source=comparison_docs), a playground, and an AI gateway, alongside your observability, not in a separate tool
## When to Choose LangSmith + LangChain
- **LangChain investment:** You're heavily invested in the LangChain ecosystem
- **LangChain-native workflow:** You want prompt iteration tooling built specifically around LangChain primitives (Logfire also ships a [prompt editor](https://pydantic.dev/docs/logfire/prompt-management/?utm_source=comparison_docs) and [playground](https://pydantic.dev/docs/logfire/guides/web-ui/prompt-playground/?utm_source=comparison_docs))
- **Quick prototyping:** You value LangChain's flexibility for rapid experimentation
## Key Differences Explained
### Production-Grade Foundation
The Pydantic stack brings real software engineering practices to AI development:
- **Type safety:** Catch errors at development time, not in production
- **Validated outputs:** Pydantic AI enforces schema-validated responses, not string parsing
- **Battle-tested:** Pydantic has 500M+ monthly downloads, used by OpenAI, Anthropic, and most AI frameworks
### Full-Stack Observability
LangSmith shows you what your LLM did. Logfire shows you what your LLM did AND what happened in your databases, APIs, and services.
When your AI agent fails, you need to know: Was it the LLM reasoning? The tool that returned bad data? The database query that timed out? Only full-stack observability answers these questions.
### Open Standards
Logfire is built on OpenTelemetry with 100% GenAI semantic convention alignment. This means:
- Your instrumentation is portable
- You can use familiar OTel tooling
- You're not locked into any single vendor
Pydantic AI itself works with ANY observability backend that supports OTel. You're not locked into Logfire.
### SQL for Agentic Coding
Logfire uses SQL with PostgreSQL-compatible syntax for querying, not a proprietary UI or API. This is critical for AI-assisted development:
- **Coding agents can query freely**: No limitation to predefined queries
- **AI assistants excel at SQL**: GPT-5, Claude, and coding agents write excellent SQL
- **Arbitrary analysis**: JOINs, aggregations, window functions, CTEs, full analytical power
- **Familiar syntax**: No new query language to learn
When your coding agent needs to debug production issues, it can write whatever query answers the question. With custom UIs or APIs, it's limited to what someone anticipated.
### Evaluations
LangSmith includes built-in evaluation workflows. For Logfire users, [pydantic-evals](https://github.com/pydantic/pydantic-evals) provides a code-first approach to evaluations:
- Evaluate any Python function, not just LLM calls (test tools, data pipelines, entire workflows)
- Define evals as code, version control them
- Run locally or in CI/CD
- Integrate with any testing framework
Different philosophy: UI-managed vs code-first. Choose what fits your workflow.
## Summary
**Choose Logfire + Pydantic AI** for production-grade AI development with type safety, full-stack observability, and open standards.
**Choose LangSmith + LangChain** if you're invested in the LangChain ecosystem and prioritize R&D workflow features.
---
### Comparisons/Migrate From Braintrust
---
title: "Move from Braintrust to Logfire"
description: "Send an existing Braintrust evaluation suite to Logfire, then migrate datasets, evaluators, prompts, and production scoring at your own pace."
---
# Move from Braintrust to Logfire
Keep your existing Braintrust `Eval` code and send its next completed evaluation to Logfire by changing two environment variables. You can then replace Braintrust concepts with native Logfire and Pydantic Evals workflows one at a time.
!!! note "Compatibility preview"
Logfire has verified the normal `Eval` flow from the Python `braintrust` 0.30.1 and TypeScript `braintrust` 3.24.0 packages. The compatibility endpoint does not implement the complete Braintrust API. Start with a staging or continuous integration (CI) run and check the [supported workflow](#know-what-the-compatible-endpoint-covers) before moving production evaluation jobs.
## Send your next evaluation to Logfire
You need a Logfire project and its **write token**, the credential that selects the destination project and lets the Braintrust SDK send evaluation data. Copy one from **Project → Settings → Write tokens** in Logfire.
Set the Braintrust app URL to the compatibility endpoint for your Logfire data region. Put the write token in the variable where the Braintrust SDK expects its API key:
```bash
# US project
export BRAINTRUST_APP_URL="https://logfire-us.pydantic.dev/v1/braintrust"
export BRAINTRUST_API_KEY=""
# Remove overrides that would send part of the run somewhere else.
unset BRAINTRUST_API_URL
unset BRAINTRUST_PROXY_URL
```
For an EU project, use `https://logfire-eu.pydantic.dev/v1/braintrust` instead. Keep the token out of source control.
Run the same Python or TypeScript evaluation command you use today. The compatibility endpoint makes no request to Braintrust. It accepts the SDK protocol, translates the completed run into Logfire experiment data, and returns a result link that opens Logfire.
This change affects future runs only. It does not import Braintrust history, copy hosted resources, or send the run to both services.
## Verify the first run
Let the SDK finish its normal score summary, then use either route:
1. Open the result URL printed by the Braintrust SDK.
2. Or open the destination Logfire project, select **Experiments** in the sidebar, and open the newest run.
Confirm that the experiment contains the expected cases and evaluator results. Case data appears when the SDK requests its final comparison summary, so it does not appear incrementally while the run is still in progress.
One naming difference matters immediately: the write token selects the actual Logfire project. The string passed as the first argument to `Eval(...)`, which Braintrust calls a project, becomes the dataset name used to group experiments inside that Logfire project.
## Translate evaluation concepts
The two systems use similar evaluation shapes, but some nouns have different boundaries.
| If you know this in Braintrust | Use this in Logfire | What changes |
| --- | --- | --- |
| Organization | Organization | The team and access-control boundary remains an organization. |
| Project | Logfire project | A Logfire project holds the telemetry, datasets, experiments, prompts, and other resources for an application or environment. With the compatibility endpoint, the write token selects this project. |
| `Eval("support", ...)` project argument | Dataset name `support` | The compatibility endpoint uses Braintrust's project argument to group related experiments as a Logfire dataset. In native Pydantic Evals, set [`Dataset.name`](../evaluate/evals-in-code.md) directly. |
| Dataset | Code-defined or hosted dataset | Keep cases beside your code in a `pydantic_evals.Dataset`, or [manage a hosted dataset](../evaluate/manage-datasets.md) with your team in Logfire. Braintrust-hosted datasets do not pass through the compatibility endpoint. |
| Dataset row `input` | `Case.inputs` | Both hold the value passed to the task. |
| Dataset row `expected` | `Case.expected_output` | Both hold the value an evaluator can compare with the task output. |
| Task | Evaluation task | Pass the function or agent under test to `Dataset.evaluate()` or `evaluate_sync()`. |
| Scorer | Evaluator | A Pydantic Evals evaluator can return a pass/fail assertion, a numeric score, or a categorical label. See [Evals in code](../evaluate/evals-in-code.md). |
| Classifier | Evaluator that returns a label | The compatibility endpoint maps one classification per name to a Logfire label. It retains additional source data, but it cannot represent several classifications with the same name as one native label. |
| Experiment | Experiment | Each completed run records its cases, outputs, evaluator results, duration, and traces for review. Token use and cost appear when the task reports them. |
| Baseline and experiment comparison | Compare experiment runs | [Review experiments](../evaluate/review-experiments.md) to choose a baseline and inspect improvements, regressions, errors, and coverage changes. The compatibility endpoint supports explicit IDs and named baselines; its automatic choice is the newest prior run, not Braintrust's Git-branch-aware selection. |
| Experiment trace and child spans | Experiment case trace | Open a case trace to see the task, model calls, tools, database work, and other operations behind the result. |
If your current suite passes inline data, tasks, and scorers to `Eval`, you can keep those objects while you test Logfire. Moving to Pydantic Evals later removes the compatibility layer and gives you native typed cases and evaluator results.
## Translate the surrounding workflow
These are workflow replacements, not protocol translations. The Braintrust compatibility endpoint does not move the resources in this table for you.
| Braintrust workflow | Logfire workflow | Migration note |
| --- | --- | --- |
| Logs, traces, and spans | [Live View](../guides/web-ui/live.md) and [Agents](../guides/web-ui/agents.md) | Send application telemetry to Logfire with a native SDK or OpenTelemetry. Evaluation traces then sit beside the production services they exercised. |
| Online scoring | [Pydantic AI online evaluation](https://pydantic.dev/docs/ai/evals/online-evaluation/) and [Live Evaluations](../evaluate/live-evals.md) | Define online evaluators in application code. Braintrust server-side scoring rules are not executed by the compatibility endpoint. |
| Feedback and human review | [Run annotations and annotation queues](../evaluate/human-review.md) | Use Logfire's native review workflow for new judgments. Feedback comments logged through the Braintrust SDK before experiment finalization are retained, but feedback added after finalization is not applied. |
| Hosted prompts | [Prompt Management](../reference/advanced/prompt-management/index.md) | Re-create and test prompt templates in Logfire. The compatibility endpoint does not fetch Braintrust prompts or translate their template syntax. |
| Playground | [Prompt Playground](../guides/web-ui/prompt-playground.md) or [Prompt Management test scenarios](../reference/advanced/prompt-management/scenarios.md) | Use Prompt Playground to rerun a captured agent interaction. Use Prompt Management when the prompt is a versioned runtime resource. |
| Model proxy | [Logfire AI Gateway](../reference/advanced/gateway/index.md) | Configure the model SDK separately. `/v1/braintrust` is an evaluation compatibility endpoint, not a model proxy. |
| BTQL and saved data views | [PostgreSQL-compatible SQL](../reference/sql.md) | Query evaluation and application telemetry together, then save useful queries as dashboards or alerts. |
| Loop | [Logfire MCP server](../how-to-guides/mcp-server.md) | Ask a coding agent to query experiments and traces through Logfire's SQL surface. This is the closest investigation workflow, not a direct import of Loop sessions or actions. |
| Hosted functions, workflows, and remote parameters | Application code, Pydantic AI, and [Managed Variables](../reference/advanced/managed-variables/index.md) | Keep tools, workflows, and evaluators in your application. Use Managed Variables for runtime configuration. Logfire does not translate Braintrust-hosted functions. |
## Choose how far to migrate
You do not need to rewrite the suite before evaluating Logfire:
1. **Redirect one verified eval job.** Change the endpoint and key, then compare the resulting cases, scores, traces, and query workflow.
2. **Move the source of test cases if needed.** Inline and callable data already work. For Braintrust-hosted datasets, export the cases and create a [hosted Logfire dataset](../evaluate/datasets-sdk.md) or put them in a code-defined `pydantic_evals.Dataset`.
3. **Adopt native evaluators when they improve the code.** Replace Braintrust scorers with Pydantic Evals evaluators gradually instead of translating the whole suite at once.
4. **Replace hosted workflows explicitly.** Move prompts, online scoring, model routing, and runtime configuration to their native Logfire equivalents only after testing each one.
## Know what the compatible endpoint covers
The verified Python and TypeScript flows support inline or callable data, local tasks and scorers, structured inputs and outputs, multiple numeric scores, one categorical result per name, metadata, tags, trials, concurrency, task and scorer child spans, and normal summary finalization.
The endpoint does not provide Braintrust-hosted datasets, prompts, functions, remote parameters, attachments, object-storage overflow, arbitrary BTQL, the model proxy, server-side scoring, public sharing, or post-finalization feedback mutation. It also does not project Rust `Eval`, manual `flush()` without comparison, or runs with `summarize_scores=False` because those flows do not request the summary that finalizes the experiment.
## Troubleshoot the switch
**The SDK still contacts Braintrust:** Unset `BRAINTRUST_API_URL` and `BRAINTRUST_PROXY_URL`. Those overrides take precedence over the URLs returned by compatibility login.
**The endpoint returns `401`:** Confirm that `BRAINTRUST_API_KEY` contains a Logfire project write token, not a Logfire user token or a Braintrust API key. Confirm that the endpoint region matches the project.
**The experiment exists but has no cases:** Let the normal SDK summary finish. Do not set `summarize_scores=False`, rely on manual `flush()`, or use the Rust SDK for this compatibility path.
**A hosted resource returns `501`:** The message identifies a Braintrust cloud resource that the compatibility endpoint does not implement. Move that resource using the matching native workflow above rather than retrying the request.
## Next steps
- [Run a native evaluation](../evaluate/evals-in-code.md) when you are ready to replace the compatibility layer.
- [Review experiments](../evaluate/review-experiments.md) to compare the first Logfire run with its next iteration.
- [Compare Logfire and Braintrust](braintrust.md) for the broader product differences.
---
### Comparisons/Sentry
---
title: "Logfire vs Sentry"
---
# Logfire vs Sentry
Sentry is a mature error monitoring platform. Logfire is an AI-native observability platform built on OpenTelemetry with full-stack monitoring capabilities. While Sentry excels at catching and tracking errors, Logfire provides complete visibility into your application's behavior, AI to API.
## Quick Comparison
| Feature | Logfire | Sentry |
|---------------------|----------------------------------------|------------------------------------------|
| **Primary Focus** | Full observability (logs, traces, AI) | Error monitoring |
| **App Tracing** | Core capability | Available, not a core focus |
| **AI/LLM Support** | First-class: auto-instrumentation, evals, prompt management, AI gateway | Generic function tracing only |
| **Logging** | Structured logs with full context | Error-focused |
| **Live View** | Real-time "pending spans" | ❌ |
| **Query Interface** | SQL (Postgres-compatible) | Custom UI |
| **Pricing** | 10M spans/logs/metrics free, then $2/M | Per-event + quotas |
*Logfire Cloud pricing (Team or Growth plans). Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5).
## When to Choose Logfire
- **Full observability:** You want logs, traces, AND error tracking in one tool
- **AI/LLM applications:** You need to observe prompts, responses, token usage
- **Real-time debugging:** You want to see what's happening right now, not just errors
- **Frontend + backend debugging:** Use Logfire's [JavaScript SDK](https://pydantic.dev/docs/logfire/typescript-sdk/) to trace and debug your entire application
- **SQL analysis:** You want to query your data with familiar SQL
- **Unified tooling:** You don't want to juggle Sentry + logging service + APM tool
- **One-click AI-assisted debugging via MCP**
## When to Choose Sentry
- **Error-focused workflow:** Your primary need is catching and triaging errors
- **Frontend error tracking:** You need robust JavaScript error tracking with source maps
- **Issue tracker integration:** You rely on deep integrations with Jira, GitHub Issues, etc.
## Key Differences Explained
### Complete Observability vs Just Errors
**Sentry** captures exceptions and performance issues. It's excellent at what it does, but it only shows you when things go wrong.
**Logfire** provides full observability:
- **Structured logs:** Every log/span/trace with full context, not just errors
- **[Issue alerts](https://pydantic.dev/docs/logfire/guides/web-ui/issues/?utm_source=sentry_comparison_docs):** Automatic exception grouping, fingerprinting, and webhook alerts to Slack
- **Distributed traces:** See requests flow through your entire system
- **Real-time monitoring:** Watch your application in real-time with "pending spans"
- **AI visibility:** Automatic instrumentation for LLM calls, tool invocations, and more
### AI/LLM Support
**Sentry** treats AI calls like any other function. You'll see that an error occurred, but you won't see:
- What prompt was sent
- What the model responded
- Token usage and costs
- Tool calls and their results
**Logfire** was built for AI applications. One function call gives you complete LLM visibility:
```python skip="true" skip-reason="incomplete"
import logfire
logfire.configure()
logfire.instrument_openai() # That's it
```
### Live Debugging
**Sentry** shows you errors after they happen.
**Logfire** has a unique "pending spans" feature that shows you what's happening right now. Watch requests in real-time, see where they're spending time, debug performance issues as they occur.
### SQL-Powered Analytics: Essential for Agentic Coding
**Sentry** uses a custom UI for querying and filtering.
**Logfire** uses SQL with PostgreSQL-compatible syntax. This is a significant advantage for AI-assisted development:
- "Show me all FastAPI requests that called our LLM more than 3 times"
- "What's the average token usage by endpoint?"
- "Which users are hitting rate limits most often?"
AI assistants and coding agents are excellent at writing these queries. When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question without artificial limitations. With custom UIs or APIs, it's constrained to anticipated queries.
## Migration Path
If you're using Sentry and want to add Logfire:
1. **Complementary use:** Many teams run both. Sentry for error alerting, Logfire for observability.
2. **Gradual transition:** Start with Logfire for AI/LLM monitoring, expand from there
3. **Full migration:** Replace Sentry entirely if you want unified tooling
## Summary
**Choose Logfire** for AI/LLM observability including full-stack monitoring support, real-time debugging, and SQL-based analysis.
**Choose Sentry** if your primary need is error monitoring with mature issue tracking integrations.
**Consider both** if you want Sentry's error alerting alongside Logfire's AI and broader observability.
---
### Comparisons/Signoz
---
title: "Logfire vs SigNoz"
---
# Logfire vs SigNoz
SigNoz is an open-source observability platform available as both a self-hosted solution and a managed cloud service. Logfire is an AI-native observability platform, also built on OpenTelemetry, with full-stack monitoring capabilities and (Enterprise) self-hosting options. Both platforms support logs, traces, and metrics, but they serve different needs.
This comparison covers both SigNoz deployment options so you can make an informed choice.
## Quick Comparison
| Aspect | Logfire | SigNoz Cloud | SigNoz Self-Hosted |
|-------------------------|---------------------------------------------------------------------------------------------------------------------|-------------------------------------|----------------------------------------|
| **Hosting** | Managed SaaS (or [Enterprise](https://pydantic.dev/pricing?utm_source=signoz_comparison_docs#enterprise) self-host) | Managed SaaS | You host everything |
| **Scope** | AI-native and full-stack (logs, traces, metrics) | Full-stack (logs, traces, metrics), | Full-stack (logs, traces, metrics) |
| **Setup** | 3 lines of code | OTel configuration | Deploy collectors, ClickHouse, storage |
| **AI/LLM observability** | Native (one function call) + purpose-built UI | Via OpenLLMetry | Via OpenLLMetry |
| **SDKs** | First-class Python, JS/TS, Rust | Standard OTel | No SDKs (standard OTel) |
| **Maintenance** | Zero | Zero | You manage it all |
| **Pricing** | 10M spans/logs/metrics free, then $2/M (plus base plan fee)* | Base plan fee + usage | "Free" but infra + maintenance costs |
| **Query language** | PostgreSQL-compatible SQL | ClickHouse SQL | ClickHouse SQL |
*Logfire Cloud pricing (Team or Growth plans). Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5).
## Logfire vs SigNoz Cloud
If you want managed observability without infrastructure overhead, here's how Logfire and SigNoz Cloud compare.
### What's Similar
Both SaaS platforms handle infrastructure for you: no ClickHouse to manage, no collectors to deploy, no scaling headaches. You instrument your app, send data, and get observability.
### Where Logfire Differs
**AI-native observability.** Logfire was built for AI applications from the ground up. One function call instruments OpenAI, Anthropic, LangChain, PydanticAI, and more. You get purpose-built panels for LLM conversations, tool calls, and automatic token/cost tracking. SigNoz requires custom instrumentation via OpenLLMetry or manual span configuration to achieve similar visibility.
**Developer experience.** Logfire's SDK is designed for simplicity:
```python skip="true" skip-reason="incomplete"
import logfire
logfire.configure()
logfire.instrument_openai()
```
Three lines and you're observing AI calls. SigNoz uses standard OpenTelemetry instrumentation, which requires more configuration and overhead.
**SQL querying for agentic workflows.** Logfire uses PostgreSQL-compatible SQL. This matters more than it might seem. AI coding assistants and agents write excellent PostgreSQL. When you're debugging production issues with Claude or Cursor, your tools can query your observability data without learning tool-specific syntax. SigNoz uses ClickHouse SQL, which is powerful but less familiar to most developers and AI tools.
**SDK flexibility.** The Logfire SDK can send data to any OTel-compatible backend, including SigNoz. If you like our SDK's simplicity but want to use their backend, you can.
**Pricing** includes 10M spans/logs/metrics per month for free. [Paid plans](https://pydantic.dev/pricing/?utm_source=signoz_comparison_docs) start at $49/mo with the same 10M included, then $2 per million after that. That's it. No host fees, no custom metrics fees, no ingestion surprises.
### When to Choose Each
**Choose Logfire** if you're building AI/LLM applications (you get evals, prompt management, a playground, and an AI gateway alongside observability), want the simplest possible setup, work heavily with AI coding tools, use MCP for debugging, and benefit from querying your data with standard SQL.
**Choose SigNoz Cloud** if you need Prometheus-style infrastructure metrics alongside APM, prefer standard OTel instrumentation, or your team is already familiar with ClickHouse.
---
## Logfire vs SigNoz Self-Hosted
If you're considering self-hosting SigNoz, this is a fundamentally different tradeoff: managed convenience vs full control.
### The Self-Hosting Reality
SigNoz self-hosted gives you complete control over your observability stack. It also means:
- Deploying and managing ClickHouse (resource-intensive)
- Running and scaling OTel collectors
- Handling upgrades, patches, and security updates
- Maintaining high availability
- Significant DevOps investment
This is the nature of self-hosted infrastructure. Some organizations need this level of control.
### True Cost Comparison
**SigNoz self-hosted** is open-source, but "free" doesn't include:
- Server costs (ClickHouse needs substantial resources)
- Engineering time for setup, maintenance, and troubleshooting
- On-call burden when your monitoring infrastructure has issues
**Logfire pricing** is transparent: 10M spans/logs/metrics free, then $2/million on Team or Growth plans. Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5). No hidden infrastructure costs.
### When to Choose Each
**Choose Logfire** if you want to focus on your application instead of your monitoring infrastructure, need AI-first observability, don't have dedicated DevOps resources, want to use the MCP server to debug errors, query tracing data with SQL, build dashboards, visualize evals and set up alerts with the same platform.
**Choose SigNoz Self-Hosted** if you have strict data residency requirements that prevent using any cloud service, need complete control over your observability infrastructure, or have a DevOps team with capacity to maintain the stack.
---
## Migration Path
If you're running SigNoz and considering Logfire:
1. Both platforms use OpenTelemetry, so instrumentation concepts transfer directly
2. Try Logfire alongside SigNoz to compare the experience
3. The Logfire SDK works with any OTel backend. You can use our simpler instrumentation while evaluating
---
## Summary
**Logfire** is for teams who want managed or self-hosted (via https://pydantic.dev/pricing?utm_source=signoz_comparison_docs#enterprise) AI-native observability with minimal operational overhead. It's built for developers who'd rather focus on their application than their monitoring infrastructure, especially those without dedicated DevOps resources. Beyond tracing, Logfire lets you query your telemetry data with SQL, build dashboards, visualize evals, and set up alerts all in one platform. You can even debug production errors via MCP server directly from your IDE.
**SigNoz Cloud** is for teams who want managed observability with standard OTel instrumentation and ClickHouse's query capabilities.
**SigNoz Self-Hosted** is for teams who must own their infrastructure (whether for compliance, control, or organizational requirements) and have the resources to maintain it.
---
### Cookbook/Debug A Slow Endpoint
---
title: "Debug a slow endpoint"
description: "A user says /checkout is slow. Instrument a FastAPI app and its PostgreSQL database, find the slow request in Logfire, see the query eating the time, and fix it, end to end in about 15 minutes."
---
# Debug a slow endpoint
A user reports that `/checkout` is slow. You have a hunch it's the database, but you can't prove it, and
you can't tell which query. This walkthrough takes you from "it feels slow" to a named, fixed cause, using
Logfire to look *inside* the request rather than just clocking it from the outside.
You'll instrument a [FastAPI](../integrations/web-frameworks/fastapi.md) app and its
[PostgreSQL](../integrations/databases/psycopg.md) database, watch a slow request arrive in the
[Live view](../guides/web-ui/live.md), expand it to see the SQL query that's eating the time, confirm the
pattern across all your traffic in [Explore](../guides/web-ui/explore.md), and then fix it and watch the
duration drop.
**Who this is for:** backend and site-reliability engineers who own a web service and its database, and
want to know *why* a request is slow, not just *that* it is.
**Time:** about 15 minutes.
Two terms you'll meet throughout, in case they're new: a **span** is one unit of work: a single operation
with a name, a start, and a duration (an HTTP request, a database query). A **trace** is the full journey of
one request, made of those spans nested inside each other. Full-stack tracing is the whole point of this
walkthrough: when the request span *contains* the query span, you can see which part of the request was slow.
## Prerequisites
- **A Logfire project and its write token.** The write token is the credential your app uses to send data to
Logfire. Create a project and copy its token from **Project → Settings → Write tokens** in the
[Logfire web app](https://logfire.pydantic.dev/), and set it as `LOGFIRE_TOKEN` in your environment before
running the app. New to Logfire? [Getting Started](../index.md) walks
through creating a project and linking your machine.
- **A running PostgreSQL database.** If you don't have one handy, start a throwaway one with Docker:
```bash
docker run --rm --name postgres \
-e POSTGRES_USER=user \
-e POSTGRES_PASSWORD=secret \
-e POSTGRES_DB=database \
-p 127.0.0.1:5432:5432 \
-d postgres
```
This gives you a database reachable at `postgres://user:secret@localhost:5432/database`.
- **Python packages.** You need FastAPI, the Psycopg 3 driver, Logfire's instrumentation for each, and a server to run
the app. The `[fastapi,psycopg]` extras ship the instrumentation, not the libraries themselves:
```bash
pip install fastapi 'psycopg[binary]' 'logfire[fastapi,psycopg]' uvicorn
```
## 1. Instrument the app and the database
The trick to seeing *inside* a request is to instrument both layers: FastAPI, so each HTTP request becomes a
span, and Psycopg, so each SQL query becomes a span nested inside it. Three calls do this.
The app below has a deliberately slow `/checkout` endpoint: it looks up a user's cart by scanning an
`order_items` table that has no index on `user_id`, so PostgreSQL reads the whole table on every request.
The setup code seeds enough rows to make the problem visible.
```
/* Detailed source-code truncated for AI context efficiency. */
```
Run it with `python main.py`. The first run seeds 10 million rows, which can take a minute, so wait until
uvicorn logs that it's listening before sending a request to generate a trace:
```bash
curl 'http://localhost:8000/checkout?user_id=42'
```
**What you'll see in Logfire:** open the [Live view](../guides/web-ui/live.md) (the project's home page)
and within a second or two a `GET /checkout` trace appears. See [Instrument FastAPI](../integrations/web-frameworks/fastapi.md)
and [Instrument Psycopg](../integrations/databases/psycopg.md) for everything these two calls capture.
## 2. Find the slow request in the Live view
The [Live view](../guides/web-ui/live.md) streams traces as they arrive. To find slow ones specifically,
click `Search your spans` (keyboard shortcut `/`) to open the search pane. You're writing the `WHERE` clause
of a SQL query over your traces. Filter to slow `/checkout` requests:
```sql
span_name = 'GET /checkout' AND duration > 0.05
```
`duration` is in seconds, so this finds requests slower than 50 ms. Press `Run` (or `cmd+enter`;
`ctrl+enter` on Windows and Linux). Your `curl` request should be right there. A full-table scan over
10 million rows takes 100 ms or more, far longer than a keyed lookup should.
Not comfortable writing SQL? The search pane also takes a plain-language question ("slow checkout requests")
and converts it to SQL for you. See [the Live view guide](../guides/web-ui/live.md#sql-search-pane)
for both aids.
## 3. Expand the trace to find the slow query
Click the trace to open it. The `GET /checkout` span has a `+` beside it, meaning it has child spans.
Expand it. Nested inside the request you'll see the SQL query span from Psycopg, and its duration bar fills
almost the entire width of the request: the query is nearly all of the time.
This is the payoff of instrumenting both layers. Without the query span you'd know only that the request was
slow; with it, you can see *which* operation inside the request is to blame: here, the
`SELECT ... FROM order_items WHERE user_id = ...` statement. Click the query span to read its full SQL and
timing in the details panel.
## 4. Confirm the pattern across all your traffic
One slow request could be a fluke. Before you change anything, confirm it's a pattern. First, generate a
little traffic so there's more than one request to aggregate:
```bash
for i in $(seq 1 20); do curl -s "http://localhost:8000/checkout?user_id=$i" > /dev/null; done
```
Then open [Explore](../guides/web-ui/explore.md), where you write SQL over the `records` table: one row per
span or log.
**Which endpoints are slowest?** This ranks your endpoints by their typical (median) and slow-tail (95th
percentile) durations:
```sql
SELECT
span_name,
count(*) AS requests,
approx_percentile_cont(duration, 0.5) * 1000 AS p50_ms,
approx_percentile_cont(duration, 0.95) * 1000 AS p95_ms
FROM records
WHERE attributes ? 'http.route'
GROUP BY span_name
ORDER BY p95_ms DESC
```
**Which queries are slowest?** Database query spans carry the statement in the `db.statement` attribute.
This ranks them:
```sql
SELECT
attributes->>'db.statement' AS query,
count(*) AS runs,
approx_percentile_cont(duration, 0.95) * 1000 AS p95_ms
FROM records
WHERE attributes ? 'db.statement'
GROUP BY query
ORDER BY p95_ms DESC
```
For each query, set **Time window** (next to **Run**) to a period with real traffic, for example *Last day*,
and raise **Limit** if you have many endpoints. Press **Run** (cmd+enter; ctrl+enter on Windows and Linux). The `order_items` query should sit at
the top, confirming this isn't a one-off.
Prefer to look at it as a ranked service dashboard rather than raw SQL? The
[Services view](../guides/web-ui/services.md) shows every service ranked by request rate, error rate, and
latency (the 95th and 99th percentiles), and its **Top operations** panel surfaces the slowest spans inside a
service: the same finding, one click away.
## 5. Fix it and watch the duration drop
The query scans the whole `order_items` table because there's no index on `user_id`. Add one:
```sql
CREATE INDEX order_items_user_id_idx ON order_items (user_id);
```
Run that against your database (for the Docker setup: `psql postgres://user:secret@localhost:5432/database`),
then send the same request again:
```bash
curl 'http://localhost:8000/checkout?user_id=42'
```
**What you'll see in Logfire:** back in the [Live view](../guides/web-ui/live.md), the new `GET /checkout`
trace arrives, and the query span inside it now takes a fraction of what it did before. PostgreSQL jumps
straight to the matching rows instead of reading the whole table. To confirm it in aggregate, send a fresh
batch of requests (the same `curl` loop from step 4) and re-run the step-4 Explore query over just the last
minute, so it excludes the earlier slow requests: the `order_items` query's p95 has dropped.
## The payoff
You started with "a user says `/checkout` is slow" and finished with a named, fixed cause: a missing index
on `order_items.user_id` that forced a full-table scan on every request. The thing that got you there was
seeing the SQL query *inside* the request span. Full-stack tracing turned "the request is slow" into "this
query is slow," which is the difference between guessing and knowing.
## Troubleshooting
- **`curl` connection refused?** Postgres isn't up yet, or the first run is still seeding 10 million rows. Wait until uvicorn logs that it's listening.
- **No `GET /checkout` trace in the Live view?** Check that `LOGFIRE_TOKEN` is set and that `instrument_fastapi` and `instrument_psycopg` run before the request.
- **Query isn't slow?** The scan only hurts with enough rows. Confirm the seed inserted 10 million rows with `SELECT count(*) FROM order_items`.
## What's next
- **[Instrument FastAPI](../integrations/web-frameworks/fastapi.md)**: everything the request spans capture,
including parsed arguments and validation errors.
- **[Instrument Psycopg](../integrations/databases/psycopg.md)**: options for what to instrument, and adding
context to your query spans.
- **[Debug a slow tool call](debug-a-slow-tool-call.md)**: the same trace-driven technique when the slow
layer is an agent's tool, not the database.
- **[Explore](../guides/web-ui/explore.md)**: the full SQL surface over your telemetry, including metrics.
- **[Alerts](../guides/web-ui/alerts.md)**: turn the "slowest endpoints" query into a scheduled check that
notifies you when latency creeps back up, so you don't have to watch the Live view yourself.
---
### Cookbook/Debug A Slow Tool Call
---
title: "Debug a slow tool call"
description: "Your agent takes seconds to answer. Is that the model, or something it called? Trace a Pydantic AI agent in Logfire, prove a tool, not the LLM, is the bottleneck, and find the slow query inside it."
---
# Debug a slow tool call
Your agent takes a few seconds to answer, and the instinct is to blame the model. Usually it isn't. An
agent's time is mostly spent in the **tools** it calls (a database lookup, a search, an API request),
not in the language model itself. From the outside the two are indistinguishable: all you see is "the
agent is slow." A trace tells them apart.
This walkthrough takes a [Pydantic AI](https://ai.pydantic.dev/) agent that's slow to respond and uses
Logfire to show, in one trace, that the model calls are fast and a single **tool call** is eating the
time, then drills into that tool to find the query behind it.
**Who this is for:** AI engineers building agents who want to know *where* an agent's latency actually
goes, instead of guessing it's "the LLM."
**Time:** about 15 minutes.
Two terms, once: a **span** is one unit of work with a name and a duration; a **trace** is the tree of
nested spans for one request. The whole trick here is that when the agent-run span *contains* the
model-call spans and the tool-call span, you can read their durations side by side and see which one is
slow.
## Prerequisites
- **A Logfire project and its write token.** Create a project and copy its token from
**Project → Settings → Write tokens** in the [Logfire web app](https://logfire.pydantic.dev/). New to
Logfire? See [Getting Started](../index.md).
- **A Google Gemini API key**, from [Google AI Studio](https://aistudio.google.com/apikey), set as
`GEMINI_API_KEY`. The agent uses `google:gemini-flash-lite-latest`; any
[model Pydantic AI supports](https://ai.pydantic.dev/models/) works: swap the model string and set
that provider's key.
- **A PostgreSQL database** for the tool to query. A throwaway one with Docker:
```bash
docker run --rm --name postgres \
-e POSTGRES_USER=user -e POSTGRES_PASSWORD=secret -e POSTGRES_DB=database \
-p 127.0.0.1:5432:5432 -d postgres
```
Then seed a big table with **no index on `user_id`**. The size is what makes the query genuinely
slow later (`psql postgres://user:secret@localhost:5432/database`):
```sql
CREATE TABLE order_items (id serial PRIMARY KEY, user_id integer, product text, price numeric);
INSERT INTO order_items (user_id, product, price)
SELECT mod(i, 5), 'product-' || i, mod(i, 100) FROM generate_series(1, 15000000) i;
```
- **Python packages**: Pydantic AI, the Psycopg 3 driver, and Logfire's instrumentation for both:
```bash
pip install "logfire[psycopg]" "pydantic-ai-slim[google]" "psycopg[binary]"
```
Set your write token and model key so the app can send traces and call the model:
```bash
export LOGFIRE_TOKEN=
export GEMINI_API_KEY=
```
## 1. Build an agent with a tool that hits the database
Here's a small shopping-assistant agent with one tool. The tool answers "what does this customer spend
the most on?" with a `GROUP BY` query over the `order_items` table. Two instrumentation calls do the
work: `instrument_pydantic_ai()` records every agent run, model call, and tool call; `instrument_psycopg()`
records every SQL query as a span nested inside whatever called it.
```python title="agent.py" skip-run="true" skip-reason="external-connection"
import psycopg
from pydantic_ai import Agent, RunContext
import logfire
logfire.configure()
logfire.instrument_pydantic_ai() # every agent run, model call, and tool call becomes a span
logfire.instrument_psycopg() # every SQL query becomes a span, nested inside the tool that ran it
DSN = 'dbname=database user=user password=secret host=localhost port=5432' # database connection string
agent = Agent(
'google:gemini-flash-lite-latest',
system_prompt='You are a shopping assistant. Use the `top_products` tool to answer what a customer spends most on.',
)
@agent.tool
async def top_products(ctx: RunContext[None], user_id: int) -> str:
"""Return the customer's highest-spend products."""
# Async DB access inside an async tool, so the slow query doesn't block the event loop.
async with await psycopg.AsyncConnection.connect(DSN) as conn:
async with conn.cursor() as cur:
await cur.execute(
'SELECT product, sum(price) AS spend FROM order_items '
'WHERE user_id = %s GROUP BY product ORDER BY spend DESC LIMIT 5',
(user_id,),
)
rows = await cur.fetchall()
return ', '.join(f'{product} (${spend})' for product, spend in rows)
if __name__ == '__main__':
result = agent.run_sync('What does customer 3 spend the most on?')
print(result.output)
```
Run it (`python agent.py`). It answers in a few seconds: slow enough to be annoying, not obvious why.
## 2. The agent is slow: model or tool?
Open the [Live view](../guides/web-ui/live.md) and click the agent run to expand its trace. The top
`invoke_agent` span is the whole request; nested under it are the model calls and the tool call, each
with its own duration:
| Span | Duration | What it is |
| ----------------------------- | -------- | --------------------------------- |
| `invoke_agent agent` | ~3.5 s | the whole agent run |
| `chat gemini-flash-lite-latest` | ~0.5 s | the model deciding to call the tool |
| `execute_tool top_products` | **~2.3 s** | **your tool running** |
| `chat gemini-flash-lite-latest` | ~0.7 s | the model writing the final answer |
There's the answer you can't get from the outside: **the model calls are fast (~0.5–0.7 s each); the
`top_products` tool took ~2.3 s.** "The agent is slow" was really "the `top_products` tool is slow, and
the LLM is fine." (Your exact seconds vary by hardware and data size, but the *shape*, a tool
dominating the model calls, is the point.)
This is the trap the trace saves you from: spending an afternoon tuning prompts or switching models to
shave latency that was never in the model.
## 3. Look inside the tool
Click the `execute_tool top_products` span to expand it. Nested inside is a single **SQL query span**
(`SELECT product, sum(price) … GROUP BY product …`) taking ~2.26 s, almost the entire tool call. So the
tool is slow for one concrete reason: that query.
This is the payoff of instrumenting both layers. Without the query span you'd know only that the tool
was slow; with it, you can see it's the `GROUP BY` aggregation over `order_items`, and click the span to
read its full SQL and timing.
One nuance worth knowing, because it's the opposite of the [slow endpoint](debug-a-slow-endpoint.md)
case: **an index on `user_id` barely helps here.** The cost isn't finding customer 3's rows: it's
aggregating millions of them. The real fix for a tool like this is to stop computing it live on every
agent call: precompute the "top products" table, cache the result, or narrow the data first. The
trace's job was to point you at *that* decision (the slow tool, then the slow query inside it) in
seconds instead of guesswork.
## What you've got now
- **A latency you can attribute.** "The agent is slow" became "the `top_products` tool is slow, not the
model," and then "it's this `GROUP BY` query", each step read straight off one trace.
- **The general lesson.** Agent latency usually lives in tools and retrieval, not the model. Instrument
the tools, and a slow agent stops being a mystery.
## Troubleshooting
- **Auth errors?** Set both `LOGFIRE_TOKEN` and `GEMINI_API_KEY`.
- **The SQL span isn't nested under the tool span?** `instrument_psycopg()` must run before the query, and the query must happen inside the tool (the async `AsyncConnection` in the example).
- **Postgres connection fails?** Confirm the throwaway container is running and the `DSN` host and port match it.
## What's next
- [Instrument and evaluate an agent](evaluate-an-agent.md): the full loop of building, tracing, and
scoring the same kind of agent.
- [Debug a slow endpoint](debug-a-slow-endpoint.md): when the slow query *is* fixable with an index,
and how to confirm the pattern across all your traffic in Explore.
- [AI observability](../ai-observability.md): everything Logfire captures from an agent, and why seeing
the whole stack (model *and* tools) matters.
---
### Cookbook/Evaluate An Agent
---
title: "Instrument and evaluate an agent, end to end"
description: "Build a small Pydantic AI support agent, see its trace in Logfire, score its answers with an eval, catch a regression by comparing runs, and send the uncertain cases to a human."
---
# Instrument and evaluate an agent, end to end
This is the full loop an AI engineer runs on a real agent: build it, watch what it actually does, measure whether it's any good, prove a change made it better or worse, and route the answers you're unsure about to a person. You'll do all five here, on one small agent, in one sitting.
**Who this is for:** you're building an LLM-powered agent (an app where a language model reasons, calls your functions, and returns an answer) and you want to stop shipping on vibes. You've got a working feature; you want to know if it's *good*, and to be told when it stops being good.
**What you'll build:** a tiny [Pydantic AI](https://pydantic.dev/docs/ai/overview/) support-question agent, instrumented so every run (the model call, the tokens it used, the cost, and any tool calls) shows up in Logfire. Then a small evaluation (a repeatable quality test, like a test suite for output that has no single right answer) that scores its answers, and a second run that catches a regression.
**Time:** about 20–25 minutes.
Along the way you'll touch three Logfire features, each covered in depth on its own page: [AI observability](../ai-observability.md) (seeing what the agent did), [evaluations](../evaluate/overview.md) (measuring whether it's good), and [human review](../evaluate/human-review.md) (adding a person's judgment). This page stitches them into one journey; follow the links when you want the full API.
A few terms, defined once:
- A **span** is one unit of work: a single operation, with a name, a start, and a duration.
- A **trace** is the full journey of one request or agent run, made of nested spans (the agent run, its model call, each tool call).
- A **token** is the unit language models read and bill by: a few characters of text.
- A **scorer** (also called an **evaluator**) is the thing that judges an output. Each scorer produces evaluator results, such as pass/fail assertions or numeric scores.
## Prerequisites
- **Python 3.10 or newer.**
- **A Logfire project and its write token.** The write token is the credential your app uses to send data to a Logfire project. Create a project and copy the token from **Project → Settings → Write tokens** in the Logfire web app. For local development you can instead run `logfire auth` in your terminal (this opens your browser to link the machine, no token to copy). See [Getting Started](../index.md) if you don't have a project yet.
- **A Google Gemini API key**, from [Google AI Studio](https://aistudio.google.com/apikey), set as the `GEMINI_API_KEY` environment variable. The agent below uses `google:gemini-flash-lite-latest`; any [model Pydantic AI supports](https://pydantic.dev/docs/ai/models/) works the same way: swap the model string and set that provider's key instead.
!!! note "Consequence: this sends data, and some of it costs money"
Running the agent and the eval sends traces, inputs, outputs, and evaluator results to your Logfire project, where they're stored and visible to your team. Every model call (the agent's own answers *and* the LLM-as-a-judge scorer you'll add in step 3) is a real, billed API call to your provider. A dozen cases with one judge is a few dozen calls; keep that in mind before you point this at a 500-case dataset.
Install the packages you need:
```bash
pip install logfire pydantic-evals "pydantic-ai-slim[google]"
```
Logfire needs no extra to instrument Pydantic AI; `pydantic-ai-slim` is Pydantic AI itself and its `[google]` extra adds the Gemini client; `pydantic-evals` is the evaluation library. (Using `uv`? `uv add logfire pydantic-evals "pydantic-ai-slim[google]"`.)
## Step 1: Build the agent
Here's the whole agent. It answers support questions and has one tool that looks up order status, so you'll see a real tool call in the trace, not just a bare model reply.
```python title="agent.py" skip-run="true" skip-reason="external-connection"
from __future__ import annotations
from pydantic_ai import Agent, RunContext
support_agent = Agent(
'google:gemini-flash-lite-latest',
system_prompt=(
'You are a support agent for an online store. Answer the customer concisely. '
'If they ask about an order, use the `order_status` tool to look it up.'
),
)
@support_agent.tool
async def order_status(ctx: RunContext[None], order_id: str) -> str:
"""Look up the current status of an order by its ID."""
# A real app would query your database here; this is a stand-in.
fake_orders = {'A100': 'shipped, arriving Friday', 'A200': 'processing'}
return fake_orders.get(order_id, 'no order found with that ID')
if __name__ == '__main__':
result = support_agent.run_sync('Where is my order A100?')
print(result.output)
```
Run it (`python agent.py`) and you'll get a plain-English answer in your terminal. But you can't yet see *how* it got there, which is the whole problem this page solves.
## Step 2: Instrument it and see the trace
Add three lines. `logfire.configure()` connects the SDK to your project; `logfire.instrument_pydantic_ai()` records every agent run (the model call, the tokens, the cost, and each tool call) with no other code changes.
```python title="agent.py" hl_lines="5 7 8" skip-run="true" skip-reason="external-connection"
from __future__ import annotations
from pydantic_ai import Agent, RunContext
import logfire
logfire.configure() # uses your write token or `logfire auth` credentials
logfire.instrument_pydantic_ai() # records every Pydantic AI agent run
support_agent = Agent(
'google:gemini-flash-lite-latest',
system_prompt=(
'You are a support agent for an online store. Answer the customer concisely. '
'If they ask about an order, use the `order_status` tool to look it up.'
),
)
@support_agent.tool
async def order_status(ctx: RunContext[None], order_id: str) -> str:
"""Look up the current status of an order by its ID."""
fake_orders = {'A100': 'shipped, arriving Friday', 'A200': 'processing'}
return fake_orders.get(order_id, 'no order found with that ID')
if __name__ == '__main__':
result = support_agent.run_sync('Where is my order A100?')
print(result.output)
```
Run it again, then open the [Live view](../guides/web-ui/live.md), the project's home page, where traces stream in as they arrive. Within a second or two you'll see the agent run appear. Click it to expand the trace.
**What you'll see in Logfire:**
- The **agent run** as the top span, with the total time it took.
- Nested under it, the **model call** to `gemini-flash-lite-latest`, with the **input and output tokens** and the **cost** of that call.
- The **`order_status` tool call**: the `order_id` the model passed in, and the string your function returned.
- Opening the model call shows the exchange laid out like a chat transcript (system prompt, user message, the model's reply) in an [LLM panel](../guides/web-ui/llm-panels.md), not a wall of JSON.
This is the point of [AI observability](../ai-observability.md): when an answer is wrong later, you open its trace and see whether the *model* reasoned badly or a *tool* returned bad data. The two look identical from the outside, and the trace is how you tell them apart. For a per-provider, per-model rollup of calls, tokens, and cost, see the [LLMs and providers view](../guides/web-ui/llms.md).
## Step 3: Is it any good? Score it with an eval
Seeing one run tells you whether *that* answer was good. To know whether the agent is good *in general*, and to notice when a change makes it worse, you need an **evaluation**: run the agent over a fixed set of test cases and score every answer.
The pieces (all from [`pydantic_evals`](https://pydantic.dev/docs/ai/evals/evals/), covered fully in the [evals overview](../evaluate/overview.md)):
- A **Dataset** of **Cases**: each an input (a question) and, optionally, the expected answer.
- A **task**: the thing under test. Here it's a thin wrapper around the agent.
- **Scorers**: the judges. You'll attach two: a **code scorer** (fast, free, deterministic) that checks the answer mentions the right fact, and an **`LLMJudge`** (a language model grading another model's output) that rates tone and helpfulness, something code can't check.
```
/* Detailed source-code truncated for AI context efficiency. */
```
Run it (`python eval.py`). `report.print(...)` prints a table to your terminal, one row per case, an averages row, and a few columns: the output, per-case **Metrics** (requests, tokens, cost), a **Duration**, and an **Assertions** column. Each scorer that returns a pass/fail contributes one mark to the Assertions column, in the order you listed the evaluators (here `MentionsFact` then `LLMJudge`), so `✔✔` means both passed.
On the good agent, every case passes both scorers (output columns trimmed here for width; your exact wording, tokens, and cost vary by model and run):
| Case ID | Output | Assertions | Cost |
| ---------------- | ------------------------------------------------------------ | ---------- | --------- |
| order_shipped | Your order A100 has been shipped and is scheduled to arrive this Friday. | ✔✔ | $0.000117 |
| order_processing | Your order A200 is currently processing. | ✔✔ | $0.000105 |
| refund_policy | You may return most items within 30 days of delivery for a full refund… | ✔✔ | $0.000080 |
| **Averages** | | **100.0%** | |
**How to read it, and where a judge can fool you.** The two marks are your two scorers. `MentionsFact` is deterministic: it either found the expected substring or it didn't. The `LLMJudge` is graded by a model, so it measures only what its rubric asks: "polite, concise, helpful." That's exactly where it can pass an answer a human wouldn't: if the agent replied *"Your order A100 shipped and arrives Monday"* when the real answer is Friday, the judge would likely still pass it. The reply *reads* polite and helpful, and the rubric never mentioned the delivery date. The `MentionsFact` scorer, checking for `'Friday'`, would catch it. This is the reason to pair a cheap code scorer with a judge, and to [benchmark your judge against hand-labeled cases](../evaluate/overview.md#best-practices-with-the-reason-attached) before you trust its pass rate.
Because you called `logfire.configure()`, this run also lands on the [Evals: Datasets & Experiments](../evaluate/datasets-and-experiments.md) page as one **experiment** (one full pass of your agent over the dataset), where the next step compares it against a second run. See [Run an evaluation](../evaluate/evals-in-code.md) for the full offline path, and the [evals overview](../evaluate/overview.md) for the concepts.
## Step 4: Catch a regression
Now the payoff. Change something and prove whether it helped or hurt: the thing you can't do by re-reading a few answers.
Make one change that *degrades* quality. Cut the agent off from its tool so it can no longer look up a real order. Disable tools for the run (a reliable switch), rather than only asking the model not to use them in the prompt, which it may not obey:
```python skip="true"
# in agent.py, add model_settings to the Agent so it can't call order_status:
support_agent = Agent(
'google:gemini-flash-lite-latest',
system_prompt='You are a support agent for an online store.',
model_settings={'tool_choice': 'none'}, # the tool stays registered but can't be called
)
```
Then re-run the eval under a different name so it's a separate experiment:
```python skip="true"
# in eval.py, change the run name:
report = await dataset.evaluate(task, name='support-agent-no-tools')
```
Run `python eval.py` again. Now that the agent can't call `order_status`, it answers the order questions from general knowledge (*"I don't have access to your order information"*), so those replies no longer contain the expected fact, and `MentionsFact` fails on both order cases. The averages row drops (in one run, to **66.7%**; the exact number varies by model and run, but the *shape*, both order cases failing, is the point):
| Case ID | Assertions |
| ---------------- | ---------- |
| order_shipped | ✗✔ |
| order_processing | ✗✔ |
| refund_policy | ✔✔ |
| **Averages** | **66.7%** |
Look at the order rows: `✗✔`. `MentionsFact` failed, but the `LLMJudge` still **passed**. "I don't have access to your order information" reads polite and helpful, so the judge waves it through; only the deterministic scorer catches that the answer is useless. That's the judge-fooling failure mode from step 3, caught live.
**What you'll see in Logfire:** open the [Evals: Datasets & Experiments](../evaluate/datasets-and-experiments.md) page and open `support-agent-no-tools` as the candidate. Select **Compare runs**, then choose `support-agent-baseline` as the baseline. Logfire lines up the two runs case by case and highlights the differences. You'll see the pass rate fall and can click straight into a failing case's trace to confirm the agent never called the tool.
That comparison is the whole reason evals exist: "it feels worse" becomes "the pass rate went from 100% to 66.7% on these cases, and here's the trace showing why." Wire this eval into your continuous integration (the automated checks that run on every change), asserting on the pass rate the report gives you, so a regression like this fails the build before it reaches a user.
Now change the prompt back so you're on the good version again.
## Step 5: Send the uncertain cases to a human
Some answers can't be settled by code or by a model: *was this reply actually the right call?* When the judge is unsure, route that agent run to a person. A run annotation records an editable human verdict and supporting evidence. It remains separate from the automated evaluator results in the experiment.
To review one of the support-agent runs:
1. Open **Annotations** under **AI Evaluations** in Logfire.
2. Select the support agent, then select **Proceed to annotate**.
3. Find the uncertain run and select **Add annotation**.
4. Choose **Pass**, **Neutral**, or **Fail**, explain the evidence in **Comment**, and select **Save**.
If your team participates in the Design Partner early-access program, an annotation queue provides a systematic batch of runs to review instead of making reviewers find each run manually. The interactions a reviewer flags as bad are strong candidates for your dataset, so future evals can test them again. See [Human review](../evaluate/human-review.md) to compare the two review modes and check availability.
## What you've got now
Starting from a handful of lines of agent code, you now have:
- **Every agent run traced**: model calls, tokens, cost, and tool calls, readable as a transcript, so a wrong answer is something you can *see* rather than guess at.
- **A repeatable quality measure**: a dataset and two scorers that turn "seems fine" into a pass rate you can defend.
- **A regression caught on purpose**: proof that comparing experiments tells you which direction a change moved quality, before a user finds out.
- **A path to human judgment**: editable run annotations that reviewers can use to identify new regression cases.
The insight the journey produces: observability and evaluation aren't separate tools. The trace tells you *what happened*; the eval measures selected behavior; the human records context that automated checks missed. Together they help you turn a production failure into a repeatable regression case.
## Troubleshooting
- **Nothing appears on the Evals page?** Make sure `logfire.configure()` ran and `LOGFIRE_TOKEN` is set. Without it the eval still runs and prints to your terminal, but the results aren't sent to Logfire.
- **Model auth or rate-limit errors?** Confirm `GEMINI_API_KEY` is set. Free-tier limits are low, so the example uses a lighter model (`gemini-flash-lite-latest`) to stay under them.
## What's next
- [AI observability](../ai-observability.md): the full picture of what Logfire captures from your agent, and why seeing the whole stack (not just the model) matters.
- [Evaluate your AI](../evaluate/overview.md): the concepts behind datasets, evaluators, results, and experiments, plus best practices for LLM judges.
- [Run an evaluation](../evaluate/evals-in-code.md): the offline eval path in full, including hosted datasets shared with your team.
- [Human review](../evaluate/human-review.md): annotate agent runs directly or work through an annotation queue.
- [Live Evaluations](../evaluate/live-evals.md): score real production traffic automatically, then send the uncertain cases here to human review.
---
### Cookbook/Index
---
title: "Cookbook: end-to-end guides"
description: "Runnable, real-world walkthroughs that stitch Logfire's features into one journey: instrument and evaluate an agent, debug a slow endpoint or tool call, roll out a prompt safely, and track LLM cost."
---
# Cookbook
The Cookbook is a set of **end-to-end guides**: each follows a real goal from start to finish, stitching several Logfire features into one journey. Where the feature pages tell you *what each part does*, these show you *how they fit together* to get a job done. Every guide is runnable: copy the code, send the data, and see the result in Logfire.
## Guides
- **[Instrument and evaluate an agent](evaluate-an-agent.md)**: build a Pydantic AI agent, see its traces, tokens, and cost, then measure its quality with evals (repeatable tests of an AI's output quality) and catch a regression. *~25 min · for AI engineers.*
- **[Debug a slow endpoint](debug-a-slow-endpoint.md)**: trace a slow request in a FastAPI + Postgres app down to the exact SQL query behind it, and fix it. *~15 min · for backend and site reliability engineers.*
- **[Debug a slow tool call](debug-a-slow-tool-call.md)**: prove in one trace that a slow agent is stuck in a tool, not the model, and find the slow query inside it. *~15 min · for AI engineers.*
- **[Roll out a prompt safely](roll-out-a-prompt-safely.md)**: change a prompt, test it, promote it with a label, and roll it back, without a redeploy. *~15 min · for AI engineers.*
- **[Track and alert on LLM cost](track-llm-cost.md)**: capture the cost of your LLM calls, chart it on a dashboard, and get alerted when spend spikes. *~15 min · for AI engineers and engineering leads.*
## New to Logfire?
Start with the [quickstart](../index.md) to send your first trace, or pick your [path by role](../get-started/choose-your-path.md).
---
### Cookbook/Roll Out A Prompt Safely
---
title: "Roll out a prompt safely"
description: "Take one prompt from a hardcoded string to a versioned, testable value you can promote to production and roll back in seconds, no redeploy."
---
# Roll out a prompt safely
You have an AI feature whose behavior lives in a prompt: the instruction text you send to a language model. Today, changing one line of that prompt is a code change: edit the string, open a pull request, wait for review, and redeploy the whole application before the new wording reaches users. When the change turns out worse, you do it all again in reverse.
This walkthrough takes a single prompt through the safe way to ship it: you'll move the prompt out of your source code, fetch it by a **label** (a movable pointer like `production`), save a new **version** of it, test that version before it goes live, **promote** it by moving the label (with no redeploy), and **roll it back** the same way if something looks off. At the end, shipping a prompt change is a label move measured in seconds, and every change is recorded.
**Who this is for:** an AI engineer who ships prompt changes and wants to stop coupling them to code deploys.
**Time:** about 15 minutes.
A few terms, defined once, so the steps read cleanly:
- A **prompt** is the instruction text you send to a language model: a system prompt, a template, or a reusable piece of one.
- A **version** is a saved snapshot of a prompt's text. Versions are numbered (v1, v2, v3, …) and never change once saved, so you always have a stable point to promote or roll back to.
- A **label** is a movable pointer such as `production` or `canary`. Your application fetches "whatever `production` points at" rather than a fixed version number, so promoting a change is just moving the label.
For the full model behind these terms, see [Prompt management concepts](../reference/advanced/prompt-management/concepts.md).
## Prerequisites
- **A Logfire project.** If you don't have one, create it in the Logfire web app.
- **A write token**, so your application can send **traces** (the recorded journey of each request, made of nested timed operations called **spans**). Create one in Logfire under **Project → Settings → Write tokens**, and set it in your application's environment as `LOGFIRE_TOKEN`.
- **An API key with the `project:read_variables` scope**, which is what lets your application *fetch* prompts (a write token can't). Create one under **Project → Settings → API keys**, and set it as `LOGFIRE_API_KEY`.
- **An API key for your language model provider**, from that provider's own dashboard. This walkthrough uses an OpenAI-style call, but the prompt-management workflow is the same for any provider.
- **The Logfire SDK with the variables extra**, which installs the prompt-fetching support:
```bash
pip install 'logfire[variables]'
```
## 1. The starting point: a prompt hardcoded in your app
Here's the feature before prompt management, a support assistant whose system prompt is a Python string:
```python skip-run="true" skip-reason="external-connection"
from openai import OpenAI
import logfire
logfire.configure()
logfire.instrument_openai() # so each model call shows up as a trace in Logfire
client = OpenAI()
SUPPORT_PROMPT = """You are a support agent for Acme.
Be concise and friendly. If the customer asks for a refund,
explain the 30-day policy and offer to escalate."""
def answer(question: str) -> str:
response = client.chat.completions.create(
model='gpt-4o',
messages=[
{'role': 'system', 'content': SUPPORT_PROMPT},
{'role': 'user', 'content': question},
],
)
return response.choices[0].message.content
```
Every word of `SUPPORT_PROMPT` is frozen into the deployed code. To change the refund wording, you edit this file, get it reviewed, and redeploy. Let's take that string out of the code.
## 2. Author the prompt in Logfire and pull it in by label
First, create the prompt in Logfire so it lives outside your source tree:
1. In the Logfire web app, open the **Prompts** page and create a prompt named **Support Agent**.
2. Paste your instruction text as its first draft and save it as **v1**.
3. Move the `production` label to v1, so `production` now points at the text you just saved.
Now fetch it from your application instead of hardcoding it. Each prompt is exposed to your code under a name derived from its slug (the lowercase, hyphenated identifier in the prompt's URL): a prompt with slug `support-agent` is fetched as `prompt__support_agent`.
```python skip-run="true" skip-reason="external-connection"
from openai import OpenAI
import logfire
logfire.configure()
logfire.instrument_openai()
client = OpenAI()
# Fetch the prompt Logfire is serving under the `production` label, instead of a
# hardcoded string. The `default` is a known-good fallback used only if the prompt
# can't be resolved (server unreachable, label missing), never an empty prompt.
prompt_var = logfire.var(
name='prompt__support_agent',
type=str,
default=(
'You are a support agent for Acme. Be concise and friendly. If the customer '
'asks for a refund, explain the 30-day policy and offer to escalate.'
),
)
def answer(question: str) -> str:
# Make the model call inside the `with` block so its span records which prompt
# label and version produced the response (that's what makes step 4's audit work).
with prompt_var.get(label='production') as resolved:
response = client.chat.completions.create(
model='gpt-4o',
messages=[
{'role': 'system', 'content': resolved.value},
{'role': 'user', 'content': question},
],
)
return response.choices[0].message.content
```
**What you'll see in Logfire:** your model calls still show up as traces, but now each request is tied to the prompt label and version that produced it, so you can tell later which wording a given response came from.
If your prompt has runtime placeholders (a customer name, a topic) written as `{{...}}`, use `logfire.template_var(...)` instead, which renders those placeholders for you. See [Use prompts in your application](../reference/advanced/prompt-management/application.md) for that variant.
## 3. Make a change: save a new version, then test it before it goes live
Your app is now serving v1 from the `production` label. When you want to change the refund wording, you don't touch the code at all:
1. On the **Prompts** page, open **Support Agent** and edit the draft, say, clearer escalation instructions.
2. Save it as **v2**. Saving a version and putting it live are separate steps: v2 exists now, but `production` still points at v1, so your running app is unaffected.
Before you promote v2, test it. You have two ways, depending on how thorough you want to be:
- **Quick check in the editor.** On the Prompts page, run v2 against saved test cases (Logfire calls these **scenarios**: saved inputs like a representative customer message). This renders the prompt and calls the model so you can read the output. To sweep v2 across many representative cases at once, link a scenario to a dataset for a batch run. See [Test prompts](../reference/advanced/prompt-management/ui.md#scenarios-panel).
!!! warning "This spends real model budget"
Running a prompt against the model calls your provider and costs money, and a batch run calls it once per case. Check the case count and model before running a large batch.
- **Score it with evaluations.** If you want a measured pass/fail rather than an eyeball check (for example, "did the answer mention the 30-day policy?"), run v2 through an evaluation: a dataset of representative inputs scored by a rule or an LLM-as-a-judge. This is how you catch a regression before users see it. See [Evaluate your AI](../evaluate/overview.md) for the full workflow.
## 4. Promote: move the `production` label to v2, no redeploy
Once v2 tests well, put it live by moving the label. Your prompt is backed by a managed variable named `prompt__support_agent` (the same object you fetch in code), and promotion happens there, not in your code:
1. In Logfire, open **Managed Variables** and open `prompt__support_agent`.
2. On the **Values** tab, move the `production` label from v1 to v2.
3. Save the label change.
Your running application fetches `prompt__support_agent` with `label='production'` and picks up v2 on its next resolution. **No redeploy, no code change, no pull request.** The switch is live in seconds.
**What you'll see in Logfire:** new traces now carry v2's label and version, while older traces still show v1, so you can compare response quality, escalation rate, latency, and token cost before and after the switch. See [Promote and roll out prompts](../reference/advanced/prompt-management/promotion-and-rollouts.md).
## 5. Roll back: move the label back (instant, no deploy)
Say v2 looks worse in production. Rolling back is the same move in reverse:
1. Open `prompt__support_agent` under **Managed Variables**.
2. Move the `production` label back to v1.
3. Save.
The next prompt resolution serves v1 again. Because v1 was never deleted (versions are immutable snapshots), you're back to the known-good text instantly, with no deploy. Your traces still record exactly which version was active for each request through the incident, so you can see what happened.
## 6. Optional: roll out to a percentage of traffic first
When you'd rather not flip 100% of traffic to a new version at once, send it to a slice first. Put v2 on a `canary` label, keep `production` on v1, and split traffic between them: for example 10% to `canary`, 90% to `production`. To make this work, fetch the prompt with a stable `targeting_key` (such as a tenant or user ID) instead of pinning `label='production'`, so each user stays on a consistent version during the test:
```python skip="true"
with prompt_var.get(targeting_key='tenant-123') as resolved:
system_prompt = resolved.value
```
Watch the two versions in your traces and evaluations, and when `canary` proves reliable, move `production` to v2. See [Promote and roll out prompts](../reference/advanced/prompt-management/promotion-and-rollouts.md) for weights and targeting rules.
## The payoff
Your prompt is no longer a string frozen into a deploy. It's a versioned value you can:
- **change without shipping code**: save a version, move a label;
- **test before it's live**: in the editor or against a scored evaluation dataset;
- **promote and roll back in seconds**: a label move, not a redeploy;
- **audit**: every trace records which prompt version answered which request.
The prompt tweak that used to mean a code review and a full redeploy is now a label move you can undo instantly.
## Troubleshooting
- **The prompt always resolves to the default text?** Fetching prompts needs `LOGFIRE_API_KEY` (an API key with `project:read_variables` scope), not the write token. Without it, or if the label or slug doesn't exist yet, `logfire.var` silently falls back to `default`.
- **`ModuleNotFoundError` on `logfire.var`?** Install the variables extra: `pip install 'logfire[variables]'`.
- **Traces don't show a prompt label or version?** The model call has to run inside the `with prompt_var.get(...)` block so the span is tagged with what produced it.
## What's next
- [Prompt management concepts](../reference/advanced/prompt-management/concepts.md): the full model of prompts, versions, labels, and fragments.
- [Use prompts in your application](../reference/advanced/prompt-management/application.md): fetching and rendering prompts from your code, including runtime `{{...}}` inputs.
- [Promote and roll out prompts](../reference/advanced/prompt-management/promotion-and-rollouts.md): canary rollouts, targeting rules, and measuring the change.
- [Evaluate your AI](../evaluate/overview.md): score a prompt version before you promote it.
- [Prompt Playground](../guides/web-ui/prompt-playground.md): for one-off prompt experiments against a captured trace, as opposed to prompts your application depends on.
---
### Cookbook/Track Llm Cost
---
title: "Track and alert on LLM cost"
description: "Instrument your model calls once, then watch spend per model, chart it on a dashboard, and get pinged in Slack when a day's bill crosses a threshold."
---
# Track and alert on LLM cost
Large language model calls are metered: every request costs money based on how many **tokens** (the few-character units a model reads and writes, and bills by) go in and come out. This guide takes you from "no idea what we're spending" to a daily cost figure you can watch on a screen and a Slack message the moment a day's spend crosses a line you set.
You'll instrument your model calls, read per-model spend in Logfire's LLMs view, write the SQL that sums cost yourself, put that number on a dashboard, and wire up an alert.
**Who this is for:** an AI engineer or engineering lead who wants to see and cap what their app spends on models, not debug a single call. **Time:** about 15 minutes.
!!! note "Cost is derived from the data you send"
Logfire doesn't read your provider invoice. Every token count and cost figure on these pages is computed from the **spans** (the individual records of each model call) that your app sends to Logfire. If a call isn't instrumented, or its span is dropped before it reaches Logfire (for example by [sampling](../how-to-guides/sampling.md)), it won't show up in any of these numbers. What you see is an estimate of what those calls cost, priced from the open-source [`genai-prices`](https://github.com/pydantic/genai-prices) dataset: close to your bill, but not the invoice itself.
## Prerequisites
You'll need:
- **A Logfire project and its write token**: the credential your app uses to send data to Logfire. Create a project and copy its token from **Project → Settings → Write tokens** in the Logfire web app. New to Logfire? Start with [Getting Started](../index.md).
- **A model provider API key**: for this guide we use OpenAI, whose key comes from your [OpenAI dashboard](https://platform.openai.com/api-keys). The OpenAI SDK reads it from the `OPENAI_API_KEY` environment variable. The same steps work for [Anthropic](../integrations/llms/anthropic.md), [Google Gen AI](../integrations/llms/google-genai.md), [Pydantic AI](../integrations/llms/pydanticai.md), and [other providers](../guides/web-ui/llms.md#supported-instrumentations).
## 1. Instrument your model calls
Add Logfire to the app that makes the model calls. Two lines do it: `logfire.configure()` connects to your project, and `logfire.instrument_openai()` records every OpenAI call your app makes, capturing the token counts and cost of each one automatically.
Install Logfire, the OpenAI SDK, and [`genai-prices`](https://github.com/pydantic/genai-prices), the open-source dataset Logfire prices calls from. `genai-prices` isn't a hard dependency of Logfire, so install it alongside; without it, calls are still traced but `operation.cost` is omitted (it ships with Pydantic AI, so you already have it there):
```bash
pip install logfire openai genai-prices
```
Set your write token and your OpenAI key so the app can send data and make model calls (in local development you can run `logfire projects use ` instead of the token):
```bash
export LOGFIRE_TOKEN=
export OPENAI_API_KEY=
```
Then instrument your calls:
```python skip-run="true" skip-reason="external-connection"
import openai
import logfire
logfire.configure()
logfire.instrument_openai() # records every OpenAI call, with its tokens and cost
client = openai.OpenAI()
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Please write me a limerick about Python logging.'}],
)
print(response.choices[0].message.content)
```
**What you'll see in Logfire:** open your project's [Live view](../guides/web-ui/live.md) and, within a few seconds, a span for the model call. It carries a token-usage badge (a coin icon); hover it to see the model name, input/output/total tokens, and the cost in US dollars. See [LLM panels](../guides/web-ui/llm-panels.md) for the full in-trace view of a single call.
Every supported instrumentation records the same token and cost data: swap in [`logfire.instrument_anthropic()`](../integrations/llms/anthropic.md), [`logfire.instrument_pydantic_ai()`](../integrations/llms/pydanticai.md), or any of the [others](../guides/web-ui/llms.md#supported-instrumentations).
## 2. See per-model spend in the LLMs view
Once calls are flowing, the **LLMs and providers** view (in the project sidebar; the page itself is titled **LLMs**) gives you a per-model breakdown without writing any SQL. Each row is one `(provider, model)` pair (for example `openai` / `gpt-4o`) with its call count, errors, latency, input and output tokens, and total **Cost**. Sort by cost to see which model is spending the most.
This is the fastest way to answer "where is the money going?". When you need to slice spend by day, environment, or user (or put it on a dashboard or an alert), you write the query yourself, which is the rest of this guide. See the [LLMs view guide](../guides/web-ui/llms.md) for every column and the per-model detail page.
## 3. Query cost in Explore
The [Explore](../guides/web-ui/explore.md) page lets you ask any question of your data in SQL. Your model calls live in the `records` table (one row per span) and each span's model-specific detail sits in an `attributes` column, stored as JSON. The token and cost figures follow the OpenTelemetry conventions for GenAI (generative AI) spans:
- `attributes->'gen_ai.request.model'`: the model name (e.g. `gpt-4o-mini`).
- `attributes->'gen_ai.system'`: the provider (e.g. `openai`). Some newer SDKs emit `attributes->'gen_ai.provider.name'` instead.
- `attributes->'gen_ai.usage.input_tokens'` and `attributes->'gen_ai.usage.output_tokens'`: token counts.
- `attributes->'operation.cost'`: the cost in US dollars, attached by the instrumentation for supported providers (OpenAI, Anthropic, and Pydantic AI), priced with the [`genai-prices`](https://github.com/pydantic/genai-prices) dataset (installed alongside Logfire, as above).
This query sums cost and tokens per model per day. Paste it into the query editor, pick a **Time window** (say *Last 7 days*), raise the **Limit** if you have many models, and click **Run** (or press ⌘↵):
```sql
SELECT
date_trunc('day', start_timestamp) AS day,
attributes->>'gen_ai.request.model' AS model,
SUM((attributes->'operation.cost')::numeric) AS cost_usd,
SUM((attributes->'gen_ai.usage.input_tokens')::numeric) AS input_tokens,
SUM((attributes->'gen_ai.usage.output_tokens')::numeric) AS output_tokens
FROM records
WHERE attributes ? 'gen_ai.request.model'
GROUP BY day, model
ORDER BY day DESC, cost_usd DESC
```
A few things to know about the SQL:
- **Cast JSON values before you do math on them.** `SUM(attributes->'...')` errors, because the database doesn't know the JSON value is a number. `SUM((attributes->'...')::numeric)` converts it first. (Use `->` to keep the JSON value for casting, and `->>` to pull it out as plain text, as with `model` above.)
- **`attributes ? 'key'` filters to spans that have an attribute at all.** The `?` operator is true when the span carries that key, so `WHERE attributes ? 'gen_ai.request.model'` keeps only the model-call spans.
- **`operation.cost` is only present when the instrumentation priced the call.** If a span has no `operation.cost` attribute, the Logfire UI still shows a cost for it (computed from the tokens in your browser), but that browser-side figure **cannot be queried in SQL**. This query only sums the costs actually stored on the spans. With `genai-prices` installed (from the setup above), `operation.cost` is attached automatically for the supported providers (OpenAI, Anthropic, Pydantic AI). Keep `genai-prices` up to date for current prices. If you can't rely on `operation.cost`, sum the token columns instead and multiply by your own per-token rates.
**What you'll see in Logfire:** a table with one row per model per day, ordered so today's biggest spender is at the top.
## 4. Put daily spend on a dashboard
A one-off query answers the question once. To keep the answer in front of you, add it to a [dashboard](../guides/web-ui/dashboards.md): a saved screen of charts, each powered by a SQL query, that updates as new data arrives.
Logfire ships a **standard dashboard** for exactly this. Turn on **LLM tokens and costs** (from the dashboards list) for a ready-made breakdown of token usage and cost by model, no query to write. If it's close but not quite what you want, use it as a starting point for a custom chart.
For a custom spend-over-time chart, create a dashboard, add a time-series chart, and use the query below. Dashboards bucket time with the `$resolution` variable, which adjusts to the chart's time range automatically, so each point is spend per bucket, not per calendar day. For fixed one-day totals regardless of range, swap in `time_bucket('1 day', start_timestamp)`:
```sql
SELECT
time_bucket($resolution, start_timestamp) AS x,
attributes->>'gen_ai.request.model' AS model,
SUM((attributes->'operation.cost')::numeric) AS cost_usd
FROM records
WHERE attributes ? 'gen_ai.request.model'
GROUP BY x, model
ORDER BY x
```
See **[Write dashboard queries](../how-to-guides/write-dashboard-queries.md)** for the full walkthrough: creating the dashboard, choosing a chart type, and adding filters so viewers can narrow to one service or environment.
## 5. Alert when spend crosses a threshold
A dashboard shows spend when you look at it. An [alert](../guides/web-ui/alerts.md) tells you without looking: it's a SQL query Logfire runs on a schedule, and when the query returns rows, Logfire sends a notification. Write the query so that **rows mean "spend is too high"**, and you'll hear about a runaway bill instead of finding it later.
In the sidebar, go to **Alerts** under **Notify**, click **New alert**, pick **Custom query**, and enter a query that returns a row only when the spend in your alert window exceeds your threshold (here, 50 US dollars):
```sql
SELECT
SUM((attributes->'operation.cost')::numeric) AS cost_usd
FROM records
WHERE attributes ? 'gen_ai.request.model'
HAVING SUM((attributes->'operation.cost')::numeric) > 50
```
Under **When this alert fires**, set **Fire when** to **the query starts having results** (so you're pinged once when spend crosses the line, not on every check while it stays over), set **Look at rows from** to the window you're capping (the last 24 hours is a rolling window, not a calendar day; for a strict daily reset, filter `start_timestamp` to today in your timezone), and **Check every** to how often to run it. Under **Send notifications to**, add a channel: **Slack**, **Opsgenie**, or a **webhook** of your own. An alert with no channel still records its history in Logfire but won't notify anyone. Click **Create alert**.
**What you'll see in Logfire:** the alert appears on the Alerts list showing its recent status. When a run's total crosses your threshold, the query returns its row and every channel you attached gets pinged.
!!! tip "Cap per-model spend"
To alert on one specific model instead of the total, add `attributes->>'gen_ai.request.model' = 'gpt-4o'` to the `WHERE` clause. To get separate rows per model, use `GROUP BY attributes->>'gen_ai.request.model'` with the `HAVING` threshold: each model over the line becomes its own row, and the alert fires.
## The payoff
You now have a full loop around LLM spend, built from the same spans your app already sends:
- **See it**: per-model cost in the LLMs view, and any single call's tokens and cost in an LLM panel.
- **Slice it**: cost per model per day (or per environment, or per user) with SQL in Explore.
- **Watch it**: a daily-spend chart on a dashboard you glance at.
- **Cap it**: an alert that pings Slack when a day's bill crosses your threshold.
The key insight: because cost is derived from the spans you send, the same instrumentation that makes your model calls debuggable also makes them budgetable. One `instrument_openai()` call feeds all four surfaces.
## Troubleshooting
- **`operation.cost` is empty in Explore?** `genai-prices` isn't installed, so nothing prices the spans. Install it (`pip install genai-prices`). The browser UI still shows a cost, but a span without `operation.cost` can't be summed in SQL.
- **No rows in the LLMs view?** Data hasn't arrived yet, or `LOGFIRE_TOKEN` or your provider key isn't set. Make a call and wait a few seconds.
## What's next
- **[LLMs view](../guides/web-ui/llms.md)**: every per-model column, the detail page, and how the `gen_ai.*` attributes drive it.
- **[LLM panels](../guides/web-ui/llm-panels.md)**: inspect a single call's conversation, tokens, and cost, and how costs are calculated.
- **[Alerts](../guides/web-ui/alerts.md)**: firing modes, delivery channels, and setting up Slack.
- **[Explore](../guides/web-ui/explore.md)** and the **[SQL reference](../reference/sql.md)**: the tables, columns, and functions your queries can use.
---
### Evaluate/Annotate Agent Runs
---
title: "Annotate an agent run"
description: "Review an agent interaction and record a pass, neutral, or fail verdict in Pydantic Logfire."
---
# Annotate an agent run
Review an agent interaction in context, then save a verdict, comment, and tags that explain what it got right or wrong.
An agent **run** is one end-to-end invocation of your agent, including its messages, model calls, and tool calls. Annotate a run when you need a human judgment that automated checks cannot provide, such as whether an answer was helpful, safe, or on-topic. The result is an editable [run annotation](human-review.md) that remains separate from automated evaluator results.
Use this direct workflow when you are investigating one run or a small sample in context. Design Partner customers participating in early access can use an [annotation queue](human-review.md#work-through-an-annotation-queue) for a systematic batch of interactions.
!!! note "Run annotations are in Beta"
Run annotations are available in every project, but the workflow may change while it is in Beta.
!!! note "Review data is stored in Logfire"
Saving an annotation sends its verdict, comment, and tags to your Logfire project, where your team can view them. Do not include sensitive data in a comment.
## Agree on the criterion first
Annotations become comparable only when reviewers use the same rule. Before you start, write one sentence that defines what a pass means. For example, you might define a pass as: "The answer resolves the customer's question without inventing details." Use the **Comment** field to record the evidence that supports your verdict.
## Open the run you want to review
1. In the Logfire web UI, open **Annotations** under **AI Evaluations**.
2. Under **Agents**, select the agent whose runs you want to review.
3. Select **Proceed to annotate**. Logfire takes you to that agent's **Runs** tab, where you can inspect the interaction before recording your judgment.
4. Find the run you want to review. Select the row to inspect its input, final output, model calls, tool calls, and trace. Select **Annotate** to open the annotation panel. To open the panel directly, select **Add annotation** in the run's row.
## Record your judgment
Choose a verdict using the criterion your team agreed on, then add enough context for someone else to understand the decision:
- **Pass**: the run met the criterion you are reviewing.
- **Neutral**: the run is neither clearly good nor clearly bad, or does not have enough information for a pass or fail.
- **Fail**: the run did not meet the criterion you are reviewing.
Use **Comment** to state the evidence you found in the run. Add **Tags** to group related reviews, such as `hallucination`, `tone`, or `tool-error`. Select **Save** to store the annotation, or **Save & next** to continue through the remaining runs.
## Verify the annotation
After you save, the Runs tab's annotated count increases and the run shows its annotation. Return to **Annotations** to see the saved review in **Recent annotations**, where you can filter by verdict.
## Troubleshooting
### I cannot find an agent or run
The agent needs recorded runs in the selected time range. Send or wait for an agent interaction, then adjust the time range on the agent's Runs tab if needed.
### I do not see the annotation controls
Open the interaction from **Agents** > **Runs**. Run annotations are attached to complete agent runs and are not available from the generic Live view.
## Next steps
- [Human review](human-review.md): understand how direct review and annotation queues support evaluation work.
- [Run an evaluation](evals-in-code.md): compare a fixed dataset against your scoring criteria.
- [Live Evaluations](live-evals.md): score production traffic automatically, then use annotations for the cases that need human review.
---
### Evaluate/Datasets And Experiments
---
title: "Datasets and experiments"
description: "Curate repeatable test cases, review experiment results, and compare changes in the Logfire evaluation workspace."
---
# Datasets and experiments
Use the Logfire evaluation workspace to answer two questions: what should you test, and did the latest change help?
A **dataset** is a reusable collection of test cases. Each case has an input and can also have an expected output and metadata. An **experiment** is one run of your AI system over a dataset. Evaluators judge each result and record assertions or scores that Logfire can aggregate and compare.
## Follow the evaluation loop
1. **Curate cases.** Keep cases in a hosted dataset that your team edits in Logfire, or define them with your eval code.
2. **Run an experiment.** Use [pydantic-evals](evals-in-code.md) to run the same task and evaluators over every case.
3. **Read the overview.** Check completion, assertions, task errors, evaluator results, duration, tokens, and cost.
4. **Review case evidence.** Inspect the input, output, evaluator result, and trace for failures or surprising scores.
5. **Compare iterations.** Choose a baseline run and confirm whether a candidate improved the cases that matter.
## Choose where cases live
The **New dataset** flow asks how you want to manage cases:
- **Manage in Logfire** creates a hosted dataset. Your team can add and edit cases in the web UI, define JSON schemas, import cases from code, and add production traces.
- **Manage in code** keeps the source of truth with your eval code. Experiment results still appear in Logfire. You can also sync a copy of the cases to Logfire when you want to browse or edit them there.
Use stable dataset names so successive runs appear together. Names can contain `/`, which lets the Datasets page group related datasets by path prefix.
## Use the workspace
- [Manage datasets](manage-datasets.md) to find, create, import, and edit test cases.
- [Run evals in code](evals-in-code.md) to execute a task and send results to Logfire.
- [Review experiments](review-experiments.md) to diagnose failures and compare a candidate with a baseline.
- Use the [Datasets SDK](datasets-sdk.md) to create, publish, and fetch hosted datasets from Python.
## Consequences to know
- Configuring Logfire during an eval sends its inputs, outputs, evaluator results, and traces to your project.
- Experiments run from code. The web UI helps you prepare datasets and analyze results; it does not execute the task under test.
- Code remains the source of truth for code-defined cases. Syncing imports new cases into the hosted copy and updates cases with matching names; edits in Logfire do not change your code.
- Every experiment is also trace data. Open a case's trace when the result alone does not explain what the agent did.
## Next steps
Start with [Manage datasets](manage-datasets.md) if you need cases. If you already have a dataset and results, go directly to [Review experiments](review-experiments.md).
---
### Evaluate/Datasets Sdk
---
title: "SDK Guide"
description: "Manage evaluation datasets programmatically with the Logfire Python SDK."
---
# SDK Guide
!!! note "Experimental SDK"
The dataset management SDK is under `logfire.experimental.api_client`. The API may change in future releases.
The SDK provides a typed Python client for managing datasets programmatically. This is the recommended approach when you want to define datasets in code, publish them to hosted storage, and later fetch them back for evaluation. You can also manage datasets through the [Web UI](datasets-and-experiments.md).
## Installation
Install the Logfire SDK with the `datasets` extra:
```bash
pip install 'logfire[datasets]'
```
This installs `httpx` and `pydantic-evals` as additional dependencies.
## Creating a Client
```python skip-run="true" skip-reason="external-connection"
from logfire.experimental.api_client import LogfireAPIClient
client = LogfireAPIClient(api_key='your-api-key')
```
The client can also be used as a context manager to ensure the underlying HTTP connection is properly closed:
```python skip="true" skip-reason="external-connection"
with LogfireAPIClient(api_key='your-api-key') as client:
...
```
!!! note "API key scopes"
The API key must have the `project:read_datasets` scope to read datasets, and `project:write_datasets` to create, update, or delete datasets and cases. You can create API keys with these scopes under **Settings > API Keys** in the Logfire UI.
The `base_url` is automatically inferred from the API key. You can override it if needed (e.g., for self-hosters):
```python skip="true" skip-reason="external-connection"
client = LogfireAPIClient(
api_key='your-api-key',
base_url='http://localhost:8000',
)
```
An async client is also available:
```python skip="true" skip-reason="external-connection"
from logfire.experimental.api_client import AsyncLogfireAPIClient
async with AsyncLogfireAPIClient(api_key='your-api-key') as client:
datasets = await client.list_datasets()
```
## Publishing a Local Dataset to Hosted
Define your input, output, and metadata types as dataclasses or Pydantic models, build a local [`pydantic_evals.Dataset`][pydantic_evals.Dataset], and publish it with `push_dataset`. The SDK infers hosted JSON schemas from the dataset's generic types:
```python skip-run="true" skip-reason="external-connection"
from dataclasses import dataclass
from pydantic_evals import Case, Dataset
from logfire.experimental.api_client import LogfireAPIClient
@dataclass
class QuestionInput:
question: str
context: str | None = None
@dataclass
class AnswerOutput:
answer: str
confidence: float
@dataclass
class CaseMetadata:
category: str
difficulty: str
reviewed: bool = False
local_dataset = Dataset[QuestionInput, AnswerOutput, CaseMetadata](
name='qa-golden-set',
cases=[
Case(
name='capital-question',
inputs=QuestionInput(question='What is the capital of France?'),
expected_output=AnswerOutput(answer='Paris', confidence=0.99),
metadata=CaseMetadata(category='geography', difficulty='easy'),
),
Case(
name='math-question',
inputs=QuestionInput(question='What is 15 * 23?'),
expected_output=AnswerOutput(answer='345', confidence=1.0),
metadata=CaseMetadata(category='math', difficulty='easy'),
),
],
)
with LogfireAPIClient(api_key='your-api-key') as client:
dataset = client.push_dataset(
local_dataset,
description='Golden test cases for the Q&A system',
)
print(f"Published dataset: {dataset['name']} (ID: {dataset['id']})")
```
`push_dataset(...)` is designed to be rerunnable:
- it uses `dataset.name` by default, or `name=` if you want a hosted override
- it creates the hosted dataset if it does not exist yet
- it updates the hosted dataset if one already exists with the same name
- it uploads all cases through the existing import/upsert API
- it uses `on_case_conflict='update'` by default, so named cases are updated on repeat pushes
!!! note "Round-tripping evaluators"
`push_dataset(...)` uploads case-level evaluators with their cases, plus dataset-level `evaluators` and `report_evaluators` from the `Dataset` itself. Each push overwrites the hosted values, so removing an evaluator locally and re-pushing also clears it on the server.
To deserialize the hosted values back into typed instances, pass the same custom types you would to `Dataset.from_file(...)`:
```python skip="true" skip-reason="external-connection"
dataset = client.get_dataset(
'qa-golden-set',
QuestionInput,
AnswerOutput,
CaseMetadata,
custom_evaluator_types=[MyEvaluator],
custom_report_evaluator_types=[MyReportEvaluator],
)
```
## Manual Dataset Management
If you need lower-level control, the SDK still exposes `create_dataset(...)`, `add_cases(...)`, `update_dataset(...)`, and the other primitives directly.
Use `create_dataset(...)` when you want to create the hosted dataset record separately from uploading cases:
```python skip="true" skip-reason="external-connection"
dataset = client.create_dataset(
name='qa-golden-set',
description='Golden test cases for the Q&A system',
input_type=QuestionInput,
output_type=AnswerOutput,
metadata_type=CaseMetadata,
)
```
Then use `add_cases(...)` to upload one or more cases:
```python skip="true" skip-reason="external-connection"
client.add_cases(
'qa-golden-set',
cases=local_dataset.cases,
)
```
You can also pass plain dicts instead of `Case` objects:
```python skip="true" skip-reason="external-connection"
client.add_cases(
'qa-golden-set',
cases=[
{'inputs': {'question': 'What color is the sky?'}, 'expected_output': {'answer': 'Blue'}},
],
)
```
!!! tip "Referencing datasets by name or ID"
All dataset operations accept either the dataset's UUID or its name. Names are recommended for readability. If you need the UUID, it's returned in the `create_dataset()` response as `dataset['id']`.
## Listing Cases
```python skip="true" skip-reason="external-connection"
# List all cases in a dataset
cases = client.list_cases('qa-golden-set')
for case in cases:
print(f" {case['name']}: {case['inputs']}")
# Get a specific case
case = client.get_case('qa-golden-set', case_id='some-case-uuid')
```
## Listing and Retrieving Datasets
```python skip="true" skip-reason="external-connection"
# List all datasets in the project
datasets = client.list_datasets()
for ds in datasets:
print(f"{ds['name']}: {ds['case_count']} cases")
# Retrieve only dataset-level metadata for a specific dataset by name or ID
dataset_info = client.get_dataset('qa-golden-set', include_cases=False)
```
To fetch the full hosted dataset back as a typed [`pydantic_evals.Dataset`][pydantic_evals.Dataset], see [Running Evaluations](evals-in-code.md).
## Updating and Deleting
```python skip="true" skip-reason="external-connection"
# Update a dataset's metadata
client.update_dataset('qa-golden-set', description='Updated description')
# Update a specific case
client.update_case(
'qa-golden-set',
case_id='some-case-uuid',
metadata=CaseMetadata(category='geography', difficulty='easy', reviewed=True),
)
# Delete a case
client.delete_case('qa-golden-set', case_id='some-case-uuid')
# Delete an entire dataset and all its cases
client.delete_dataset('qa-golden-set')
```
## What's Next?
- **[Running Evaluations](evals-in-code.md)** --- Fetch your dataset and run evaluations with pydantic-evals.
- **[SDK Reference](../reference/api/datasets.md)** --- Complete method signatures and exception reference.
- **[Web UI Guide](datasets-and-experiments.md)** --- Manage datasets through the Logfire web interface.
---
### Evaluate/Evals In Code
---
title: "Evals in code"
description: "Run evaluations against local or hosted datasets with pydantic-evals, and see a real printed score table."
---
# Evals in code
Run an evaluation from Python and you get a score table you can read at a glance and compare over time. Evals in Logfire are powered by [pydantic-evals](https://pydantic.dev/docs/ai/evals/evals/); this page shows the offline path end to end. For the concepts behind it (scorers, scores, offline vs online), read the [evals overview](overview.md) first.
Your test cases can live in either of two places, and the evaluation step is identical once you have a `Dataset` in hand:
- **Code-defined datasets** are defined in code (or loaded from YAML, a human-readable configuration format) as a [`pydantic_evals.Dataset`][pydantic_evals.Dataset]. No server round-trip. This is the simplest way to start and is all many projects need.
- **Hosted datasets** are stored on Logfire, editable in the [web UI](manage-datasets.md), and fetched back as a typed `Dataset`. Useful when you curate cases from production traces or collaborate with teammates.
## A worked example
Here is a complete, runnable evaluation. The system under test is deterministic so the example is reproducible; in real use, `classify_sentiment` would call your LLM or agent. Two scorers run on every case: a custom `ExactMatch` and the built-in `IsInstance`.
```py title="sentiment_eval.py" skip-run="true" skip-reason="requires-new-pydantic-version"
from dataclasses import dataclass
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance
# The AI system under test. Deterministic here so the example is reproducible;
# in real use this calls your LLM or agent.
def classify_sentiment(text: str) -> str:
lowered = text.lower()
if any(word in lowered for word in ('love', 'great', 'excellent')):
return 'positive'
if any(word in lowered for word in ('hate', 'terrible', 'awful')):
return 'negative'
return 'neutral'
@dataclass
class ExactMatch(Evaluator[str, str]):
def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:
return ctx.output == ctx.expected_output
dataset = Dataset[str, str, None](
name='sentiment-eval',
cases=[
Case(name='clearly_positive', inputs='I love this product', expected_output='positive'),
Case(name='clearly_negative', inputs='This is terrible', expected_output='negative'),
Case(name='neutral_statement', inputs='It is a chair', expected_output='neutral'),
Case(name='subtle_negative', inputs='I would not recommend it', expected_output='negative'),
],
evaluators=[ExactMatch(), IsInstance(type_name='str')],
)
report = dataset.evaluate_sync(classify_sentiment)
report.print(include_input=True, include_output=True)
```
Running it prints a table. Each cell in the **Assertions** column is one mark per scorer, in order (`ExactMatch`, then `IsInstance`):
```text
Evaluation Summary: classify_sentiment
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Case ID ┃ Inputs ┃ Outputs ┃ Assertions ┃ Duration ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ clearly_positive │ I love this product │ positive │ ✔✔ │ 683µs │
│ clearly_negative │ This is terrible │ negative │ ✔✔ │ 689µs │
│ neutral_statement │ It is a chair │ neutral │ ✔✔ │ 657µs │
│ subtle_negative │ I would not recommend it │ neutral │ ✗✔ │ 625µs │
│ Averages │ │ │ 87.5% ✔ │ 663µs │
└───────────────────┴──────────────────────────┴──────────┴────────────┴──────────┘
```
The `subtle_negative` case is the one that matters: the classifier returned `neutral` where `negative` was expected, so `ExactMatch` failed (`✗`) while `IsInstance` still passed (`✔`), and the aggregate lands at 87.5%. That single number is what you compare when you change a prompt or swap a model. To send this run to the **Experiments** tab in Logfire for comparison over time, add `logfire.configure()` before you evaluate.
## Evaluating against your real AI task
Swap the deterministic function for your actual system, and configure Logfire so the run shows up in the UI:
```python skip-run="true" skip-reason="example-ai-task"
from dataclasses import dataclass
from pydantic_evals import Case, Dataset
import logfire
# Configure Logfire so the run appears on the Experiments tab in Logfire.
# Without this, the evaluation still runs but its results stay local.
logfire.configure()
logfire.instrument_pydantic_ai() # optional: traces the AI task under test
@dataclass
class QuestionInput:
question: str
context: str | None = None
@dataclass
class AnswerOutput:
answer: str
confidence: float
dataset = Dataset[QuestionInput, AnswerOutput, None](
name='qa-golden-set',
cases=[
Case(
name='capital_of_france',
inputs=QuestionInput(question='What is the capital of France?'),
expected_output=AnswerOutput(answer='Paris', confidence=1.0),
),
# ... more cases
],
)
async def my_qa_task(inputs: QuestionInput) -> AnswerOutput:
"""The AI system under test: call an LLM, run an agent, hit your API."""
...
async def run_evaluation():
report = await dataset.evaluate(my_qa_task)
report.print()
```
!!! note "Storing the dataset and sending results are independent choices"
Whether you *store* the dataset on Logfire and whether you *send* results to Logfire are two separate decisions. You can fetch a hosted dataset and run a purely local evaluation that only prints to the console by omitting `logfire.configure()`. We recommend configuring Logfire during evaluation so runs show up alongside your dataset in the UI. You can also load local datasets from YAML files, see the [pydantic-evals docs](https://pydantic.dev/docs/ai/evals/evals/).
## Evaluating a hosted dataset
To manage cases on the server (so teammates can edit them in the UI, or you can seed them from production traces), fetch a hosted dataset and use it exactly like a code-defined one. Hosted datasets are typically created in the [web UI](manage-datasets.md) or published from code, see the [Datasets SDK](datasets-sdk.md).
```python skip-run="true" skip-reason="external-connection"
from dataclasses import dataclass
from pydantic_evals import Dataset
import logfire
from logfire.experimental.api_client import LogfireAPIClient
@dataclass
class QuestionInput:
question: str
@dataclass
class AnswerOutput:
answer: str
logfire.configure()
async def my_qa_task(inputs: QuestionInput) -> AnswerOutput:
"""The AI system under test."""
...
async def run_evaluation():
with LogfireAPIClient(api_key='your-api-key') as client:
dataset: Dataset[QuestionInput, AnswerOutput, None] = client.get_dataset(
'qa-golden-set',
input_type=QuestionInput,
output_type=AnswerOutput,
)
report = await dataset.evaluate(my_qa_task)
report.print()
```
`get_dataset` returns a typed `pydantic_evals.Dataset`; pass `custom_evaluator_types=[...]` if your cases carry custom evaluators, or omit the type arguments to get the raw pydantic-evals-compatible dict. See the [Datasets SDK](datasets-sdk.md) for the full API.
## The loop
With Logfire tracing on, every run appears on the [Experiments](review-experiments.md) page, which turns evals into a continuous loop: **observe** production in the live view, **curate** interesting traces into a dataset, **evaluate** with pydantic-evals, **analyze** and compare experiments, then **improve** and repeat.
---
### Evaluate/Human Review
---
title: "Review AI outputs by hand"
description: "Add human judgment to agent runs in Pydantic Logfire by reviewing runs directly or working through an annotation queue."
---
# Review AI outputs by hand
Review an agent run when code and model-based evaluators cannot decide whether the result was helpful, safe, or appropriate.
A **run annotation** is editable reviewer feedback attached to one end-to-end agent invocation. It can record a pass, neutral, or fail verdict, plus a comment and tags that explain the decision. Run annotations are human-review evidence, not automated evaluator results, and they do not roll into experiment or Live Evaluation aggregates.
## When to use human review
- Build a hand-labeled set for checking whether an LLM judge agrees with your reviewers.
- Investigate production interactions where automated signals are missing or ambiguous.
- Identify failures worth preserving as cases in a repeatable [dataset](datasets-and-experiments.md).
- Apply a shared review criterion across a sample of agent runs.
## Choose direct review or a queue
### Review one run
Use direct review when you are investigating one run or a small sample. Open the run, inspect its messages, model calls, tool calls, and output, then save your judgment. Follow [Annotate an agent run](annotate-agent-runs.md) for the complete workflow.
!!! note "Run annotations are in Beta"
Run annotations are available in every project, but the workflow may change while it is in Beta.
### Work through an annotation queue
Use an **annotation queue** when review needs to be systematic. A queue gives reviewers a curated list of agent runs to inspect one after another instead of asking them to find runs manually. Teams can use queues for a random quality sample or for interactions selected by another signal.
!!! note "Annotation queues are a Design Partner feature"
Annotation queues are available to Design Partner customers participating in the early-access program.
## Make the judgment reusable
Agree on one sentence that defines the criterion before reviewing. For example: "The answer resolves the customer's question without inventing details."
For each run:
1. Choose **Pass**, **Neutral**, or **Fail** using that criterion.
2. Add a comment that identifies the evidence behind the verdict.
3. Add tags such as `hallucination`, `tone`, or `tool-error` when they will help reviewers find related cases.
Use the resulting annotations as calibration data for automated evaluators or as leads for new dataset cases. Adding a failing production interaction to a dataset lets future offline experiments check that behavior again.
## Consequences to know
- **Run annotations are stored in Logfire.** A saved verdict, comment, and tags are sent to your Logfire project and remain visible to your team.
- **Comments can contain user data.** Treat comments like other production data. Do not copy sensitive information into them.
## Next steps
- [Annotate an agent run](annotate-agent-runs.md): review one interaction and save a verdict.
- [Manage datasets](manage-datasets.md): preserve reviewed failures as repeatable test cases.
- [Run an evaluation](evals-in-code.md): measure a fixed dataset with automated evaluators.
- [Live Evaluations](live-evals.md): monitor automated evaluator results from production traffic.
---
### Evaluate/Live Evals
---
title: "Monitor live AI quality"
description: "Use Live Evaluations to spot changes in online evaluation results, investigate the underlying traces, and keep retired evaluators out of view."
---
# Monitor live AI quality
Use **Evals: Live Monitoring** to watch the results of [online evaluations](https://pydantic.dev/docs/ai/evals/online-evaluation/) running against real traffic. An online evaluation scores an agent or function after it runs, so it helps you notice a production regression, investigate an unexpected result, or follow a new evaluator rollout.
This page is for monitoring results that already reach Logfire. To add an evaluator to your application, follow the [Pydantic AI online evaluation guide](https://pydantic.dev/docs/ai/evals/online-evaluation/). For a curated test set before deployment, use [offline evaluations](overview.md) instead.
Live Evaluations reads [`gen_ai.evaluation.result`](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md#event-gen_aievaluationresult) OpenTelemetry log events that include a target and evaluation name. The [`@evaluate` decorator](https://pydantic.dev/docs/ai/evals/online-evaluation/#quick-start) for functions and the [`OnlineEvaluation` capability](https://pydantic.dev/docs/ai/evals/online-evaluation/#agent-integration) for Pydantic AI agents emit these events by default.
## Find the signal
Open **Evals: Live Monitoring** from the sidebar. The target list shows agents and functions with recent online-evaluation activity. A target is the agent or function that produced an evaluation result.
1. Choose a time range at the top of the page. Start with **24h** for a normal operating view, then narrow it when investigating a recent deployment or broaden it to compare a longer period.
2. Find the target you want to inspect. Each row shows the target type, its evaluators, the number of events, and when the last event arrived.
3. Read the evaluator summaries. They show a pass rate for pass/fail checks, an average for numeric scores, or one label plus the number of other labels seen. The small activity bars show when results arrived in the selected time range.
The target list is a fast health check, not the whole story. Use the chevron beside a target to expand its evaluator breakdown without leaving the page. Open the target when a summary changes or looks unfamiliar.
A target seen within the last 30 days remains in the list even when it has no events in the selected time range. It appears with zero events and no current score, so narrowing the range does not hide configured evaluators that are currently quiet.
## Investigate an evaluator
Select a target to open its detail page. It brings together the evaluation results for that target in the selected time range.
1. Start with the evaluator cards. They show the current summary, recent activity, and any errors raised while running that evaluator.
2. Review **Recent events** for the individual results and their explanations. Use the evaluator filter to focus on one check.
3. Select the trace link on an event to open it in Live View. A trace is the record of the request that produced the evaluated result. It lets you inspect the evaluation event and any prompt, response, tool-call, or other context that your application recorded.
An evaluator error is different from a failed evaluation: it means the evaluator itself could not produce a result. Open its trace and explanation first, then decide whether the application behavior or the evaluator needs attention.
## Interpret result shapes
The page presents a result according to the value returned by the evaluator:
| Evaluator output | Target-list and detail-page summary |
| --- | --- |
| `bool` | Pass rate, with individual `pass` or `fail` results |
| Number | Average score over the selected time range |
| String | One label, plus the number of other labels seen |
An evaluator that returns multiple named scores appears as one result for each score. If you deploy a new evaluator version, use the detail page to compare the version badges and recent events while both versions are running.
Live Evaluations groups results by target and evaluation name, regardless of the evaluator source or configuration. Give results distinct names when they should appear as separate summaries.
## Hide evaluators without deleting telemetry
Use a hide rule when an old experiment, test evaluator, or retired version makes the monitoring view harder to read. Open an evaluator's overflow menu on the target detail page and choose the matching hide action. You can hide that evaluator for the target or data that arrived before the current time while allowing new events to remain visible. When the selected range contains one recorded evaluator version, the menu also offers an action for that version.
Hide rules affect **Evals: Live Monitoring** only. They do not delete telemetry, change traces, or remove evaluation events from alerts and SQL queries.
Rules created from an evaluator's overflow menu are scoped to that target. On the **Hidden Evaluators** project settings page, you can add, edit, and remove rules, or leave the target empty to match evaluators across the project. Every populated field in one rule must match an event; an event is hidden when any active rule matches it. Removing a rule makes its matching entries visible again.
## Verify your setup
After adding online evaluations to an application and sending traffic through it, you should see:
- a target row for the evaluated agent or function;
- one summary for each evaluator or named score; and
- a recent event that opens the parent trace in Live View.
If all three appear, Logfire is receiving evaluation results and preserving their link to the request that produced them.
## Troubleshoot missing or unexpected results
**No evaluation activity yet:** Confirm the application is configured to send telemetry to the intended Logfire project, then exercise the evaluated code path. Try a wider time range before assuming no events arrived.
**An evaluator shows an error:** Open its recent event and follow the trace link. The event explanation and trace identify whether the evaluator raised or the application result failed the check.
**A result appears under an unfamiliar target:** Check the target name passed by your evaluation integration. The page groups results by that agent or function name.
**A function inside an agent has type `function`:** This is expected for the `@evaluate` decorator, which keeps the decorated function as its target even when the function runs inside a Pydantic AI agent. The `OnlineEvaluation` capability emits results for the agent target.
## Next steps
- [Configure online evaluations in Pydantic AI](https://pydantic.dev/docs/ai/evals/online-evaluation/)
- [Run offline evaluations against a dataset](overview.md)
- [Add human judgment with annotations and review](human-review.md)
---
### Evaluate/Manage Datasets
---
title: "Manage datasets"
description: "Create a hosted or code-defined evaluation dataset, curate its cases, and keep a large dataset list usable."
---
# Manage datasets
Build a dataset, a stable collection of test cases that you can run repeatedly as your agent, model, or prompt changes.
Open a project, then select **AI Evaluations** > **Datasets & experiments**. The **Datasets** tab shows hosted and code-defined datasets in the selected time range.
## Find the dataset you need
Use the controls above the list to narrow a large project:
- Search for a dataset name.
- Filter **Type** to **Hosted** or **Code-defined**.
- Set **Group by** to **Type**, **Path prefix**, or **None**.
- Change the time range if an older dataset is missing.
Choose stable, descriptive names. Use names such as `support/routing` and `support/response-quality` when path-prefix grouping will help your team navigate related datasets.
Each row shows the actions available for that dataset. **Edit cases** is available when cases are hosted in Logfire. **Review experiments** opens the runs associated with that dataset.
If code creates many temporary dataset names, select **Hide for me** on entries you do not need. This changes only your view. Use **Hidden** above the list to restore them later.
## Choose how to manage a new dataset
Select **New dataset**, then choose where the source of truth should live:
- **Manage in Logfire** when teammates need to curate cases in the web UI or you want a shared hosted dataset.
- **Manage in code** when cases belong with the source and review process for your application.
If you manage the dataset in code, choose whether to **Sync cases to Logfire** or **Keep cases in code**. Experiment results can appear in Logfire with either choice when you configure Logfire for the run. Syncing also makes the cases available to browse and edit in Logfire.
## Create a hosted dataset
The hosted-dataset flow has three steps:
1. **Dataset**: enter a stable name and an optional description.
2. **Schemas**: optionally define JSON schemas for inputs, expected outputs, and metadata.
3. **First case**: optionally add an initial input, expected output, and metadata.
You can skip schemas and the first case. Add them later when the shape becomes clear.
After creation, use the **Cases** tab to maintain the dataset:
- **Add case** creates a case in the web UI.
- **Add cases from code** shows the code needed to publish cases.
- **Sync cases from code** imports code-defined cases into the hosted dataset.
- **Export** downloads the cases for reuse outside Logfire.
- **Edit** changes the dataset name, description, or schemas.
If Logfire already discovered a code-defined dataset with the same name, creating its hosted counterpart can import the latest cases instead of starting empty. Review the imported cases before relying on them as a shared test set.
## Add a case from a production trace
Production failures and surprising outputs make useful regression cases:
1. Open [Live view](../guides/web-ui/live.md) and select the span you want to preserve.
2. Select the database icon (**+**) in the span details.
3. Choose an existing hosted dataset or create one.
4. Review the extracted input, expected output, and metadata, then save the case.
The case keeps a link to its source trace, so reviewers can inspect the original behavior.
## Verify the dataset
Before running an experiment, confirm that:
- the dataset has a stable name that future runs will reuse;
- hosted cases have the expected input and optional expected output;
- schemas describe the data you want teammates to enter;
- the dataset appears in the intended path-prefix group;
- **Review experiments** opens the expected run history.
## Troubleshooting
### A code-defined dataset has no browsable cases
This is expected when the cases remain only in code. Use **Sync cases to Logfire**, or create a hosted dataset with the same name and import the latest cases.
### A dataset is missing
Widen the time range and clear the type filter. Also check **Hidden** if you previously hid the dataset for yourself.
### The list contains many one-off datasets
Use **Group by: Path prefix** for consistently named datasets. Hide temporary entries for yourself, then change the code to reuse stable names for future runs.
## Next steps
[Run an eval](evals-in-code.md) against the dataset, then [review the experiment](review-experiments.md) in Logfire.
---
### Evaluate/Overview
---
title: "Evaluate your AI: measure output quality"
description: "Learn how evaluations work in Pydantic Logfire (datasets, evaluators, results, and experiments) and how to measure AI output quality offline and in production."
---
# Evaluate your AI
Know whether your AI system is getting better or worse, with numbers, not vibes. When you change a prompt, swap a model, or refactor an agent, an evaluation tells you if answer quality went up or down before your users find out.
An **evaluation (eval)** is a repeatable test of an AI system's output quality, like a test suite for software, except the output has no single right answer. Instead of `assertEqual`, an eval attaches a **scorer** (also called an **evaluator**: the thing that judges an output) to each answer. An evaluator can produce a pass/fail assertion, a numeric score, or a categorical label. Collect those results over a set of test cases and you can compare versions instead of guessing.
## The stakes
Without evals, "did that prompt change help?" is answered by re-reading a handful of outputs and forming an impression. That impression doesn't survive a teammate, a week, or a model upgrade. A regression that only shows up on 1 in 20 inputs is invisible to spot-checking and obvious to an eval. Evals turn "it feels better" into "the pass rate went from 82% to 91% on our 200-case set."
## The scaffold: dataset → task → evaluators → results → experiment
Every offline eval in Logfire is built from the same five pieces (online evals reuse only the **evaluator** and **evaluator result** pieces, applied to live traffic instead of a dataset). Learn them once:
1. **Dataset**: a collection of test cases (inputs and, optionally, expected outputs) you evaluate against.
2. **Task**: your AI system under test: a function that takes a case's input and returns an output (an LLM call, a Pydantic AI agent, your own API).
3. **Scorers (evaluators)**: one or more judges attached to the task. Each looks at an output and produces one or more evaluator results. A scorer can be plain code (exact match, "contains no personally identifiable information (PII)") or an **LLM-as-a-judge** (using a language model to evaluate another model's output).
4. **Evaluator result**: an assertion, numeric score, or categorical label produced for an output and saved so you can compare and aggregate it appropriately.
5. **Experiment**: one run of your task over the whole dataset, producing evaluator results you can compare across versions.
In prose, the flow is: the **dataset** feeds each case into your **task**; each output is handed to your **scorers**; each scorer emits evaluator results; and the whole pass over the dataset is one **experiment** you can line up against the last one.
## Two ways to hold a dataset
The dataset, the collection of test cases, can live in either of two places, and you can move cases between them:
- **Code-defined**, with [pydantic-evals](evals-in-code.md). You define the cases in Python; when you run an experiment with Logfire configured, its results appear in Logfire automatically. This is the right home when your cases are generated from code or checked into your repo alongside your tests.
- **Hosted on Logfire.** Open **AI Evaluations** > **Datasets & experiments** > **Datasets**, select **New dataset**, and choose **Manage in Logfire**. Your team can edit its cases directly in the web UI, define schemas, and import cases from code.
Reach for a hosted dataset when teammates need to add or edit test cases, or when a team wants one shared set of cases to evaluate against. Reach for a code-defined dataset when the cases belong with your source. See [Manage datasets](manage-datasets.md) for both.
## Offline vs online: a test suite vs monitoring
The two modes use these pieces differently, and the easiest way to hold them apart is an analogy you already know:
- **Offline eval**: running evals against a fixed dataset, like a test suite. You control the inputs, you run it on demand or in continuous integration (CI), and you compare experiments over time. Use this to catch regressions before you ship. See [Run an evaluation](evals-in-code.md).
- **Online (live) eval**: scoring real production traffic as it happens, like monitoring. The inputs are whatever your real users sent; scorers run against live outputs and stream results in. Use this to catch quality drops in production that your test set never anticipated. See [Live Evaluations](live-evals.md).
Offline is your test suite; online is your production monitoring. Most teams need both: the test suite proves a change is safe to ship, monitoring proves it stayed good once real traffic hit it.
## Choose the evaluator result that matches the question
When you design a scorer, choose the result type that matches the judgment:
- **Pass/fail (boolean)**: for checks with a clear yes/no: "did it hallucinate?", "does the output contain PII?", "is it valid JSON?" This is the shape to prefer whenever you can express the judgment as a yes/no.
- **Categorical (a label)**: for a small set of known outcomes: `refused` / `answered` / `deflected`, or `low` / `medium` / `high` severity. Use when the answer is one of a few named buckets.
- **Numeric (a number)**: for genuinely continuous judgments like a relevance rating. The most expressive, but the hardest to keep stable. See the best practices below before reaching for a 1–10 scale.
## Best practices, with the reason attached
!!! tip "Prefer binary or categorical scores over 1–10"
LLM judges are unstable on continuous scales: ask the same judge to rate the same answer "out of 10" twice and you'll often get different numbers, and different judges won't agree on what a 7 means. A yes/no ("is this answer grounded in the source?") or a small set of labels is far more reproducible. Reach for a numeric scale only when the judgment is truly continuous and you've checked the judge is stable on it.
!!! tip "Benchmark your LLM judge before you trust it"
An LLM-as-a-judge is itself an AI system that can be wrong. Before you rely on its scores, hand-label a small set of about 20–100 cases yourself, then check how often the judge agrees with you. If it disagrees with humans a third of the time, its "pass rate" is measuring the judge, not your app. Treat the judge's rubric as code you test and iterate on.
!!! tip "Make the judge explain before it scores"
Ask the judge to write its reasoning first and emit the score second. Models are more accurate when they reason before deciding, and the written reason is what lets *you* audit a surprising score instead of trusting a bare number. In pydantic-evals, `LLMJudge` records that reason alongside the score.
## Consequences to know
- **Sending eval results to Logfire stores them.** With `logfire.configure()`, an eval run (inputs, outputs, and evaluator results) is sent to and stored in your Logfire project, visible to your team. Run without it to keep results local.
- **LLM-as-a-judge scorers cost money.** Each judged output is another model call. A 500-case dataset with two LLM judges is 1,000 extra model calls per experiment.
## Next steps
- **[Run an evaluation](evals-in-code.md)**: the offline path end to end, with a worked example and a real report.
- **[Manage datasets](manage-datasets.md)**: build and curate the test cases you evaluate against.
- **[Evals SDK](datasets-sdk.md)**: manage datasets and evaluations programmatically.
- **[Review experiments](review-experiments.md)**: diagnose failed cases and compare a candidate with a baseline.
- **[Live Evaluations](live-evals.md)**: score real production traffic as it happens.
- **[Human review](human-review.md)**: record editable judgments on agent runs and turn useful failures into regression cases.
---
### Evaluate/Review Experiments
---
title: "Review experiments"
description: "Read an experiment overview, inspect failed cases, and compare a candidate run with a baseline."
---
# Review experiments
Turn an experiment result into a specific decision: keep a change, revise it, or investigate the evaluator.
An experiment is one run of your AI system over a dataset. An evaluator judges each output and records an assertion or score.
Open a project, then select **AI Evaluations** > **Datasets & experiments** > **Experiments**.
## Find the run
The Experiments page lists runs in the selected time range. Search by experiment name, group runs by dataset, and select **Review results** on the run you want to investigate.
Use a descriptive experiment name for the change under test, such as `candidate-routing-prompt-v2`. The dataset identifies what you tested; the experiment name identifies the version or hypothesis you tested.
To start another run, select **New experiment**. Logfire shows setup instructions and example code because the task itself runs in your application or eval suite. The new result appears on this page after the run sends its data to Logfire.
## Read the overview first
The **Overview** tab answers whether the run completed and where to investigate:
- **Results** show how many dataset cases produced results.
- **Assertions** summarize boolean evaluator outcomes.
- **Task errors** count cases where the task under test raised an error.
- **Average task duration** helps identify a latency change.
- **Evaluator analysis** shows each evaluator separately. Assertion evaluators show their pass and fail mix; numeric evaluators show an average and score distribution.
- **Operational metrics** report task duration, tokens, and cost when the run provides them.
Aggregate results are a triage tool, not the final decision. For a numeric evaluator, use the average to understand the center of the run and the histogram to see its spread. An improved average can still hide a low-scoring tail.
## Review case evidence
Open the **Cases** tab, then start with **Needs review**, **Failed**, or **Errors**. Search by case name when you know the scenario.
Before opening a case, scan its input and output preview alongside the selected evaluator values. Use **Metrics** to choose the evaluator columns that matter to the decision, then select **Open case** for the full evidence. Hover an evaluator value to see its name and exact value. Numeric scores also show the run-wide distribution with the current case marked.
Read each case in this order:
1. **Input**: confirm what the task received.
2. **Output**: inspect what the task returned.
3. **Evaluator results**: check which assertion or score changed and read its reason when available.
4. **Technical details**: inspect expected output, metadata, duration, tokens, and cost.
5. **Open trace in Live view**: follow model calls, tool calls, and exceptions when the output does not explain the result.
Use **Prev** and **Next**, or the displayed keyboard shortcuts, to work through the current filtered list without returning to the table.
## Compare a candidate with a baseline
From an experiment result, select **Compare runs**. The current run remains the candidate. Choose a baseline from another run of the same dataset.
Start on **Overview** to compare completion, assertions, task errors, evaluator aggregates, and operational metrics. Assertion bars make changes in the pass and fail mix visible. Numeric histograms show whether an average-score change came from broad improvement, a shifted cluster, or a few outliers.
Then open **Cases**:
- Set **Primary metric** to the evaluator that should determine improved, regressed, and unchanged groups.
- For a numeric primary metric, set **Better score** to **Higher** or **Lower**. Leave it at **Not set** to group cases by raw score movement.
- Add other **Metrics** as supporting columns. The table can show up to seven.
- Set **Group by** to **Outcome** to put regressions and errors ahead of unchanged cases.
- Scan each case's input preview and evaluator deltas, then open it to compare baseline and candidate outputs with the full evidence.
Metric direction matters. A positive raw delta is not always an improvement: higher latency, token use, or cost can be worse. Set **Better score** to match the metric's meaning, then use the outcome group rather than interpreting the sign alone.
## Decide what to change
Use the aggregate view to find a signal, then use cases to explain it:
1. Inspect every regression, task error, and changed evaluator result that matters to the product.
2. Decide whether the task or the evaluator is wrong.
3. Change one meaningful variable, such as a system prompt instruction, tool description, or model.
4. Run the same dataset again and compare the new candidate with the previous run.
Keeping the dataset and evaluators stable makes the comparison interpretable. If you change them, treat coverage changes and newly evaluated cases separately from task-quality changes.
## Verify the conclusion
Before accepting a candidate, confirm that:
- the baseline and candidate use the intended dataset;
- both runs cover the cases and evaluators you expected;
- task errors did not move into a different outcome group;
- important regressions have been inspected at case level;
- duration, token, and cost changes are acceptable;
- the evaluator's direction matches the meaning of its score.
## Troubleshooting
### A run is missing
Widen the time range, clear the search, and confirm that the eval sent data to the same Logfire project you are viewing.
### Completion says `Not reported`
The eval producer did not report that the run finished. You can still review the case results Logfire received, but Logfire cannot confirm that they represent the complete run.
### A case is not comparable
One run may be missing the case or evaluator. Treat this as a coverage change. Do not classify it as a quality regression without inspecting what changed.
### A result looks incomplete
Open its trace in Live view. The trace shows model calls, tool calls, evaluator spans, and exceptions that may not fit in the result summary.
## Next steps
Add newly discovered failures to a [hosted dataset](manage-datasets.md), refine the task or evaluator, and [run the next experiment](evals-in-code.md).
---
### Get Started/Choose Your Path
---
title: "Choose your path"
description: "Pick the starting point that matches your role (AI engineer, backend/SRE, platform/security, product/growth, or solo builder) and follow a short, ordered path to the pages that matter for you."
---
# Choose your path
Logfire does a lot, and the right first pages depend on what you're trying to do. Pick the description below that sounds most like you. Each on-ramp is a short, ordered path (a handful of links with a sentence on why each one matters) that takes you from nothing to seeing real value.
Not sure? If you want the fastest route from zero to seeing your own app, start with the [solo builder](solo-builder.md) path.
## I'm...
### AI engineer
You're building an app or agent on top of a large language model (an LLM, the AI that generates text), and you want to see what it's actually doing: the conversations, the tool calls, the tokens (the units a model bills by), the cost, and whether the answers are any good.
[Start here → For AI engineers](for-ai-engineers.md)
### Backend engineer or SRE
You run services and you want to know when they're slow or failing, watch requests as they happen, and build the dashboards and alerts that tell you about problems before your users do. (SRE stands for site reliability engineer, the person who keeps production healthy.)
[Start here → For backend engineers and SREs](for-backend-and-sre.md)
### Platform or security engineer
You care about where the data lives, who can log in, what sensitive information gets stripped before it leaves your machines, and what it all costs. You might also need to run Logfire on your own infrastructure.
[Start here → For platform and security](for-platform-and-security.md)
### Product or growth
You want to change behavior without shipping code: turn features on and off, run experiments, and understand how people use what you've built.
[Start here → For product and growth](for-product-and-growth.md)
### Solo builder
It's just you and your app. You want the shortest possible path from zero to seeing your app in Logfire, with as little jargon as possible.
[Start here → Solo builder](solo-builder.md)
---