# Technical Documentation: dottxt-ai/outlines

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/dottxt-ai/outlines)

### Overview (`README.md`)

<div align="center" style="margin-bottom: 1em;">

<img src="./docs/assets/images/logo-light-mode.svg#gh-light-mode-only" alt="Outlines Logo" width=300></img>
<img src="./docs/assets/images/logo-dark-mode.svg#gh-dark-mode-only" alt="Outlines Logo" width=300></img>


 🗒️ *Structured outputs for LLMs* 🗒️

Made with ❤👷️ by the team at [.txt](https://dottxt.co)
<br>Trusted by NVIDIA, Cohere, HuggingFace, vLLM, etc.


[![PyPI Version][pypi-version-badge]][pypi]
[![Downloads][downloads-badge]][pypistats]
[![Stars][stars-badge]][stars]


[![Discord][discord-badge]][discord]
[![Blog][dottxt-blog-badge]][dottxt-blog]
[![Twitter][twitter-badge]][twitter]

<br>The .txt API is currently in early access. **[Request access here →](https://h1xbpbfsf0w.typeform.com/to/fwQNWmS8?utm_source=github&utm_medium=organic&utm_campaign=outlines)**


</div>

## 🚀 Building the future of structured generation

We're working with select partners to develop new interfaces to structured generation.

Need XML, FHIR, custom schemas or grammars? Let's talk.

Audit your schema: share one schema, we show you what breaks under generation, the constraints that fix it, and compliance rates before and after. Sign up [here](https://h1xbpbfsf0w.typeform.com/to/rtFUraA2?typeform).

## Table of Contents

- [Why Outlines?](#why-outlines)
- [Quickstart](#quickstart)
- [Real-World Examples](#real-world-examples)
  - [🙋‍♂️ Customer Support Triage](#customer-support-triage)
  - [📦 E-commerce Product Categorization](#e-commerce-product-categorization)
  - [📊 Parse Event Details with Incomplete Data](#parse-event-details-with-incomplete-data)
  - [🗂️ Categorize Documents into Predefined Types](#categorize-documents-into-predefined-types)
  - [📅 Schedule a Meeting with Function Calling](#schedule-a-meeting-with-function-calling)
  - [📝 Dynamically Generate Prompts with Re-usable Templates](#dynamically-generate-prompts-with-re-usable-templates)
- [They Use Outlines](#they-use-outlines)
- [Model Integrations](#model-integrations)
- [Core Features](#core-features)
- [Other Features](#other-features)
- [About .txt](#about-txt)
- [Community](#community)

<div align="center"><img src="./docs/assets/images/install.png" width=300></img></div>

## Why Outlines?

LLMs are powerful but their outputs are unpredictable. Most solutions attempt to fix bad outputs after generation using parsing, regex, or fragile code that breaks easily.

Outlines guarantees structured outputs during generation — directly from any LLM.

- **Works with any model** - Same code runs across OpenAI, Ollama, vLLM, and more
- **Simple integration** - Just pass your desired output type: `model(prompt, output_type)`
- **Guaranteed valid structure** - No more parsing headaches or broken JSON
- **Provider independence** - Switch models without changing code


### The Outlines Philosophy

<div align="center"><img src="./docs/assets/images/use_philosophy.png" width=300></img></div>

Outlines follows a simple pattern that mirrors Python's own type system. Simply specify the desired output type, and Outlines will ensure your data matches that structure exactly:

- For a yes/no response, use `Literal["Yes", "No"]`
- For numerical values, use `int`
- For complex objects, define a structure with a [Pydantic model](https://docs.pydantic.dev/latest/)

## Quickstart

Getting started with outlines is simple:

### 1. Install outlines

``` shell
pip install outlines
```

### 2. Connect to your preferred model

``` python
import outlines
from transformers import AutoTokenizer, AutoModelForCausalLM


MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
    AutoTokenizer.from_pretrained(MODEL_NAME)
)
```

### 3. Start with simple structured outputs

``` python
from typing import Literal
from pydantic import BaseModel


# Simple classification
sentiment = model(
    "Analyze: 'This product completely changed my life!'",
    Literal["Positive", "Negative", "Neutral"]
)
print(sentiment)  # "Positive"

# Extract specific types
temperature = model("What's the boiling point of water in Celsius?", int)
print(temperature)  # 100
```

### 4. Create complex structures

``` python
from pydantic import BaseModel
from enum import Enum

class Rating(Enum):
    poor = 1
    fair = 2
    good = 3
    excellent = 4

class ProductReview(BaseModel):
    rating: Rating
    pros: list[str]
    cons: list[str]
    summary: str

review = model(
    "Review: The XPS 13 has great battery life and a stunning display, but it runs hot and the webcam is poor quality.",
    ProductReview,
    max_new_tokens=200,
)

review = ProductReview.model_validate_json(review)
print(f"Rating: {review.rating.name}")  # "Rating: good"
print(f"Pros: {review.pros}")           # "Pros: ['great battery life', 'stunning display']"
print(f"Summary: {review.summary}")     # "Summary: Good laptop with great display but thermal issues"
```

## Real-world examples

Here are production-ready examples showing how Outlines solves common problems:

<details id="customer-support-triage"><summary><b>🙋‍♂️ Customer Support Triage</b>
<br>This example shows how to convert a free-form customer email into a structured service ticket. By parsing attributes like priority, category, and escalation flags, the code enables automated routing and handling of support issues.
</summary>

``` python
import outlines
from enum import Enum
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
from typing import List


MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
    AutoTokenizer.from_pretrained(MODEL_NAME)
)


def alert_manager(ticket):
    print("Alert!", ticket)


class TicketPriority(str, Enum):
    low = "low"
    medium = "medium"
    high = "high"
    urgent = "urgent"

class ServiceTicket(BaseModel):
    priority: TicketPriority
    category: str
    requires_manager: bool
    summary: str
    action_items: List[str]


customer_email = """
Subject: URGENT - Cannot access my account after payment

I paid for the premium plan 3 hours ago and still can't access any features.
I've tried logging out and back in multiple times. This is unacceptable as I
have a client presentation in an hour and need the analytics dashboard.
Please fix this immediately or refund my payment.
"""

prompt = f"""
<|im_start|>user
Analyze this customer email:

{customer_email}
<|im_end|>
<|im_start|>assistant
"""

ticket = model(
    prompt,
    ServiceTicket,
    max_new_tokens=500
)

# Use structured data to route the ticket
ticket = ServiceTicket.model_validate_json(ticket)
if ticket.priority == "urgent" or ticket.requires_manager:
    alert_manager(ticket)
```
</details>

<details id="e-commerce-product-categorization"><summary><b>📦 E-commerce product categorization</b>
<br>This use case demonstrates how outlines can transform product descriptions into structured categorization data (e.g., main category, sub-category, and attributes) to streamline tasks such as inventory management. Each product description is processed automatically, reducing manual categorization overhead.
</summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```
</details>

<details id="parse-event-details-with-incomplete-data"><summary><b>📊 Parse event details with incomplete data</b>
<br>This example uses outlines to parse event descriptions into structured information (like event name, date, location, type, and topics), even handling cases where the data is incomplete. It leverages union types to return either structured event data or a fallback “I don’t know” answer, ensuring robust extraction in varying scenarios.
</summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```
</details>

<details id="categorize-documents-into-predefined-types"><summary><b>🗂️ Categorize documents into predefined types</b>
<br>In this case, outlines classifies documents into predefined categories (e.g., “Financial Report,” “Legal Contract”) using a literal type specification. The resulting classifications are displayed in both a table format and through a category distribution summary, illustrating how structured outputs can simplify content management.
</summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```
</details>

<details>
<summary id="schedule-a-meeting-with-function-calling"><b>📅 Schedule a meeting from requests with Function Calling</b>
<br>This example demonstrates how outlines can interpret a natural language meeting request and translate it into a structured format matching a predefined function’s parameters. Once the meeting details are extracted (e.g., title, date, duration, attendees), they are used to automatically schedule the meeting.
</summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```
</details>

<details>
<summary id="dynamically-generate-prompts-with-re-usable-templates"><b>📝 Dynamically generate prompts with re-usable templates</b>
<br>Using Jinja-based templates, this example shows how to generate dynamic prompts for tasks like sentiment analysis. It illustrates how to easily re-use and customize prompts—including few-shot learning strategies—for different content types while ensuring the outputs remain structured.
</summary>

```python
import outlines
from typing import List, Literal
from transformers import AutoTokenizer, AutoModelForCausalLM


MODEL_NAME = "microsoft/phi-4"
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
    AutoTokenizer.from_pretrained(MODEL_NAME)
)


# 1. Create a reusable template with Jinja syntax
sentiment_template = outlines.Template.from_string("""
<|im_start>user
Analyze the sentiment of the following {{ content_type }}:

{{ text }}

Provide your analysis as either "Positive", "Negative", or "Neutral".
<|im_end>
<|im_start>assistant
""")

# 2. Generate prompts with different parameters
review = "This restaurant exceeded all my expectations. Fantastic service!"
prompt = sentiment_template(content_type="review", text=review)

# 3. Use the templated prompt with structured generation
result = model(prompt, Literal["Positive", "Negative", "Neutral"])
print(result)  # "Positive"

# Templates can also be loaded from files
example_template = outlines.Template.from_file("templates/few_shot.txt")

# Use with examples for few-shot learning
examples = [
    ("The food was cold", "Negative"),
    ("The staff was friendly", "Positive")
]
few_shot_prompt = example_template(examples=examples, query="Service was slow")
print(few_shot_prompt)
```
</details>

## They use outlines

<div align="center">
<img src="./docs/assets/images/readme-light.png#gh-light-mode-only" alt="Users Logo"></img>
<img src="./docs/assets/images/readme-dark.png#gh-dark-mode-only" alt="Users Logo"></img>
</div>

## Model Integrations

| Model type | Description | Documentation |
|---------|-------------|:-------------:|
| **Server Support** | vLLM and Ollama | [Server Integrations →](https://dottxt-ai.github.io/outlines/latest/features/models/) |
| **Local Model Support** | transformers and llama.cpp | [Model Integrations →](https://dottxt-ai.github.io/outlines/latest/features/models/) |
| **API Support** | OpenAI, Gemini, and [Dottxt](https://h1xbpbfsf0w.typeform.com/to/fwQNWmS8?utm_source=github&utm_medium=organic&utm_campaign=outlines) | [API Integrations →](https://dottxt-ai.github.io/outlines/latest/features/models/) |

## Core Features

| Feature | Description | Documentation |
|---------|-------------|:-------------:|
| **Multiple Choices** | Constrain outputs to predefined options | [Multiple Choices Guide →](https://dottxt-ai.github.io/outlines/latest/features/core/output_types/#multiple-choices) |
| **Function Calls** | Infer structure from function signatures | [Function Guide →](https://dottxt-ai.github.io/outlines/latest/features/core/output_types/#json-schemas) |
| **JSON/Pydantic** | Generate outputs matching JSON schemas | [JSON Guide →](https://dottxt-ai.github.io/outlines/latest/features/core/output_types/#json-schemas) |
| **Regular Expressions** | Generate text following a regex pattern | [Regex Guide →](https://dottxt-ai.github.io/outlines/latest/features/core/output_types/#regex-patterns) |
| **Grammars** | Enforce complex output structures | [Grammar Guide →](https://dottxt-ai.github.io/outlines/latest/features/core/output_types/#context-free-grammars) |

## Other Features

| Feature | Description | Documentation |
|---------|-------------|:-------------:|
| **Prompt templates** | Separate complex prompts from code | [Template Guide →](https://dottxt-ai.github.io/outlines/latest/features/utility/template/) |
| **Custome types** | Intuitive interface to build complex types | [Python Types Guide →](https://dottxt-ai.github.io/outlines/latest/features/core/output_types/#basic-python-types) |
| **Applications** | Encapsulate templates and types into functions | [Application Guide →](https://dottxt-ai.github.io/outlines/latest/features/utility/application/) |

## About .txt

<div align="center">
<img src="./docs/assets/images/dottxt-light.svg#gh-light-mode-only" alt="dottxt logo" width=100></img>
<img src="./docs/assets/images/dottxt-dark.svg#gh-dark-mode-only" alt="dottxt logo" width=100></img>
</div>

Outlines is developed and maintained by [.txt](https://dottxt.co), a company dedicated to making LLMs more reliable for production applications.

Our focus is on advancing structured generation technology through:

- 🧪 **Cutting-edge Research**: We publish our findings on [structured generation](http://blog.dottxt.co/performance-gsm8k.html)
- 🚀 **Enterprise-grade solutions**: You can license [our enterprise-grade libraries](https://docs.dottxt.co).
- 🧩 **Open Source Collaboration**: We believe in building in public and contributing to the community

Follow us on [Twitter](https://twitter.com/dottxtai) or check out our [blog](https://blog.dottxt.co/) to stay updated on our latest work in making LLMs more reliable.

## Community

<div align="center" style="margin-bottom: 1em;">

[![Contributors][contributors-badge]][contributors]
[![Stars][stars-badge]][stars]
[![Downloads][downloads-badge]][pypistats]
[![Discord badge][discord-badge]][discord]

</div>

- 💡 **Have an idea?** Come chat with us on [Discord][discord]
- 🐞 **Found a bug?** Open an [issue](https://github.com/dottxt-ai/outlines/issues)
- 🧩  **Want to contribute?** Consult our [contribution guide](https://dottxt-ai.github.io/outlines/latest/community/contribute/).


## Cite Outlines

```
@article{willard2023efficient,
  title={Efficient Guided Generation for Large Language Models},
  author={Willard, Brandon T and Louf, R{\'e}mi},
  journal={arXiv preprint arXiv:2307.09702},
  year={2023}
}
```

[contributors]: https://github.com/dottxt-ai/outlines/graphs/contributors
[contributors-badge]: https://img.shields.io/github/contributors/dottxt-ai/outlines?style=flat-square&logo=github&logoColor=white&color=ECEFF4
[dottxt-blog]: https://blog.dottxt.co/
[dottxt-blog-badge]: https://img.shields.io/badge/dottxt%20blog-a6b4a3
[dottxt-twitter]: https://twitter.com/dottxtai
[dottxt-twitter-badge]: https://img.shields.io/twitter/follow/dottxtai?style=social
[discord]: https://discord.gg/R9DSu34mGd
[discord-badge]: https://img.shields.io/discord/1182316225284554793?color=ddb8ca&logo=discord&logoColor=white&style=flat-square
[downloads-badge]: https://img.shields.io/pypi/dm/outlines?color=A6B4A3&logo=python&logoColor=white&style=flat-square
[pypistats]: https://pypistats.org/packages/outlines
[pypi-version-badge]: https://img.shields.io/pypi/v/outlines?style=flat-square&logoColor=white&color=ddb8ca
[pypi]: https://pypi.org/project/outlines/
[stars]: https://github.com/dottxt-ai/outlines/stargazers
[stars-badge]: https://img.shields.io/github/stars/dottxt-ai/outlines?style=flat-square&logo=github&color=BD932F&logoColor=white
[twitter-badge]: https://img.shields.io/twitter/follow/dottxtai?style=flat-square&logo=x&logoColor=white&color=bd932f
[twitter]: https://x.com/dottxtai

---

### Overview (`examples/beam-cloud/README.md`)

## Deploy Outlines on Beam

1. Create an account [here](https://beam.cloud) and install the Beam SDK
2. Download the `app.py` file to your computer
3. Deploy it as a serverless API by running: `beam deploy app.py:predict`

---

### Index (`docs/index.md`)

---
title: Welcome to Outlines!
hide:
  - navigation
---

#

<figure markdown>
{ width="500" }
{ width="500" }
</figure>


LLMs are powerful but their outputs are unpredictable. Most solutions attempt to fix bad outputs after generation using parsing, regex, or fragile code that breaks easily.

Outlines guarantees structured outputs during generation — directly from any LLM.

- **Works with any model** - Same code runs across OpenAI, Ollama, vLLM, and more
- **Simple integration** - Just pass your desired output type: `model(prompt, output_type)`
- **Guaranteed valid structure** - No more parsing headaches or broken JSON
- **Provider independence** - Switch models without changing code
- **Rich structure definition** - Use Json Schema, regular expressions or context-free grammars

<figure markdown>
[Get Started](guide/getting_started){ .md-button .md-button--primary }
[View Examples](examples/){ .md-button }
[API Reference](api_reference/){ .md-button }
[GitHub](https://github.com/dottxt-ai/outlines){ .md-button }
</figure>

## 🚀 Building the future of structured generation

We're working with select partners to develop new interfaces to structured generation.

### ⚡ Try the Dottxt API

Want guaranteed structured generation without running your own models? The **Dottxt API** delivers 100% schema-compliant outputs via a simple REST API — no GPU, no setup.

[Request API access →](https://h1xbpbfsf0w.typeform.com/to/fwQNWmS8?utm_source=github&utm_medium=organic&utm_campaign=outlines){ .md-button .md-button--primary }

### 🔍 Audit your schema

Share one schema and we show you what breaks under generation, the constraints that fix it, and compliance rates before and after. Need XML, FHIR, custom schemas or grammars? Let's talk.

[Sign up →](https://h1xbpbfsf0w.typeform.com/to/rtFUraA2?typeform){ .md-button }

## See it in action

```python
from pydantic import BaseModel
from typing import Literal
import outlines
import openai

class Customer(BaseModel):
    name: str
    urgency: Literal["high", "medium", "low"]
    issue: str

client = openai.OpenAI()
model = outlines.from_openai(client, "gpt-4o")

customer = model(
    "Alice needs help with login issues ASAP",
    Customer
)
# ✓ Always returns valid Customer object
# ✓ No parsing, no errors, no retries
```

## Quick install

```shell
pip install outlines
```

## Features

<div class="grid cards" markdown>

- :material-shield-check: **Reliable** - Guaranteed schema compliance -- always valid JSON.
- :material-puzzle: **Feature-rich** - Supports a large proportion of the JSON Schema spec, along with regex and context-free grammars.
- :material-lightning-bolt: **Fast** - Microseconds of overhead vs seconds of retries. Compilation happens once, not every request.
- :material-lightbulb: **Simple** - Outlines is a low-abstraction library. Write code the way you normally do with LLMs. No agent frameworks needed.

</div>

## Supported inference APIs, libraries & servers

- [vLLM](features/models/vllm.md)
- [vLLM offline](features/models/vllm_offline.md)
- [Transformers](features/models/transformers.md)
- [llama.cpp](features/models/llamacpp.md)
- [Ollama](features/models/ollama.md)
- [MLX-LM](features/models/mlxlm.md)
- [SgLang](features/models/sglang.md)
- [TGI](features/models/tgi.md)
- [OpenAI](features/models/openai.md)
- [Anthropic](features/models/anthropic.md)
- [Gemini](features/models/gemini.md)
- [Dottxt](features/models/dottxt.md)

## Who is using Outlines?

Hundreds of organisations and the main LLM serving frameworks ([vLLM][vllm], [TGI][tgi], [LoRAX][lorax], [xinference][xinference], [SGLang][sglang]) use Outlines. Prominent companies and organizations that use Outlines include:

<div class="grid cards" markdown>
  <div class="row"><img src="../logos/amazon.png" width="200"></div>
  <div class="row"><img src="../logos/apple.png" width="200"></div>
  <div class="row"><img src="../logos/best_buy.png" width="200"></div>
  <div class="row"><img src="../logos/canoe.png" width="200"></div>
  <div class="row"><img src="../logos/cisco.png" width="200"></div>
  <div class="row"><img src="../logos/dassault_systems.png" width="200"></div>
  <div class="row"><img src="../logos/databricks.png" width="200"></div>
  <div class="row"><img src="../logos/datadog.png" width="200"></div>
  <div class="row"><img src="../logos/dbt_labs.png" width="200"></div>
  <div class="row"><img src="../assets/images/dottxt.png" width="200"></div>
  <div class="row"><img src="../logos/gladia.jpg" width="200"></div>
  <div class="row"><img src="../logos/harvard.png" width="200"></div>
  <div class="row"><img src="../logos/hf.png" width="200"></div>
  <div class="row"><img src="../logos/johns_hopkins.png" width="200"></div>
  <div class="row"><img src="../logos/meta.png" width="200"></div>
  <div class="row"><img src="../logos/mit.png" width="200"></div>
  <div class="row"><img src="../logos/mount_sinai.png" width="200"></div>
  <div class="row"><img src="../logos/nvidia.png" width="200"></div>
  <div class="row"><img src="../logos/nyu.png" width="200"></div>
  <div class="row"><img src="../logos/safran.png" width="200"></div>
  <div class="row"><img src="../logos/salesforce.png" width="200"></div>
  <div class="row"><img src="../logos/shopify.png" width="200"></div>
  <div class="row"><img src="../logos/smithsonian.png" width="200"></div>
  <div class="row"><img src="../logos/tinder.png" width="200"></div>
  <div class="row"><img src="../logos/upenn.png" width="200"></div>
</div>

Organizations are included either because they use Outlines as a dependency in a public repository, or because of direct communication between members of the Outlines team and employees at these organizations.

Still not convinced, read [what people say about us](community/feedback.md). And make sure to take a look at what the [community is building](community/examples.md)!


## Outlines people

Outlines would not be what it is today without a community of dedicated developers:

<a href="https://github.com/dottxt-ai/outlines/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=dottxt-ai/outlines" />
</a>

## About .txt

Outlines is built with ❤️ by [.txt](https://dottxt.co).

.txt solves the critical problem of reliable structured output generation for large language models. Our [commercially-licensed libraries][dottxt-doc] ensure 100% compliance with JSON Schema, regular expressions and context-free grammars while adding only microseconds of latency. Unlike open-source alternatives, we offer superior reliability, performance, and enterprise support.


## Acknowledgements

<div class="grid" markdown>

<figure markdown>
  <a href="https://www.normalcomputing.ai">
  { width="150" }
  </a>
</figure>

</div>

Outlines was originally developed at [@NormalComputing](https://twitter.com/NormalComputing) by [@remilouf](https://twitter.com/remilouf) and [@BrandonTWillard](https://twitter.com/BrandonTWillard). It is now maintained by [.txt](https://dottxt.co).

[discord]: https://discord.gg/R9DSu34mGd
[aesara]: https://github.com/aesara-devs
[blackjax]: https://github.com/blackjax-devs/blackjax
[pythological]: https://github.com/pythological
[hy]: https://hylang.org/
[.txt]: https://dottxt.co
[vllm]: https://github.com/vllm-project/vllm
[tgi]: https://github.com/huggingface/text-generation-inference
[lorax]: https://github.com/predibase/lorax
[xinference]: https://github.com/xorbitsai/inference
[sglang]: https://github.com/sgl-project/sglang/
[dottxt-doc]: https://docs.dottxt.co

---

### Community Index (`docs/community/index.md`)

# Community

Outlines exists for a community of users who believe software doesn't need to be complicated. Who share the same passion for Large Language Models but don't want to compromise on robustness. Together, we are bringing these powerful models back to the world of software.

## Connect on Discord

The Outlines community lives on our Discord server. There you can ask questions, share ideas or just chat with people like you. Don't be a stranger and [join us][discord].

[discord]: https://discord.gg/UppQmhEpe8

---

### Examples Index (`docs/examples/index.md`)

# Examples

This part of the documentation provides a few cookbooks that you can browse to get acquainted with the library and get some inspiration about what you could do with structured generation. Remember that you can easily change the model that is being used!

- [Classification](classification.md): Classify customer requests.
- [Named Entity Extraction](extraction.md): Extract information from pizza orders.
- [Dating Profiles](dating_profiles.md): Build dating profiles from descriptions using prompt templating and JSON-structured generation.
- [Chain Of Density](chain_of_density.md): Summarize documents using chain of density prompting and JSON-structured generation.
- [Playing Chess](models_playing_chess.md): Make Phi-3 Mini play chess against itself using regex-structured generation.
- [SimToM](simtom.md): Improve LLMs' Theory of Mind capabilities with perspective-taking prompting and JSON-structured generation.
- [Q&A with Citations](qa-with-citations.md): Answer questions and provide citations using JSON-structured generation.
- [Knowledge Graph Generation](knowledge_graph_extraction.md): Generate a Knowledge Graph from unstructured text using JSON-structured generation.
- [Structured Generation Workflow](structured_generation_workflow.md):
- [Chain Of Thought (CoT)](chain_of_thought.md): Generate a series of intermediate reasoning steps using regex-structured generation.
- [ReAct Agent](react_agent.md): Build an agent with open weights models using regex-structured generation.
- [Structured Generation from PDFs](read-pdfs.md): Use Outlines with vision-language models to read PDFs and produce structured output.
- [Earnings reports to CSV](earnings-reports.md): Extract data from earnings reports to CSV using regex-structured generation.
- [Receipt Digitization](receipt-digitization.md): Extract information from a picture of a receipt using structured generation.
- [Extract Events Details](extract_event_details.md):

Run Outlines on the cloud:

- [BentoML](deploy-using-bentoml.md)
- [Cerebrium](deploy-using-cerebrium.md)
- [Modal](deploy-using-modal.md)

---

### Features Index (`docs/features/index.md`)

# Features

This section presents in details the different features of Outlines.

## Core Concepts

- [Models](./models/index.md)
- [Model Inputs](./core/inputs.md)
- [Output Types](./core/output_types.md)
- [Generators](./core/generator.md)
- [Error Handling](./core/exceptions.md)

## Utilities

- [Applications](./utility/application.md)
- [Templates](./utility/template.md)
- [Regex DSL](./utility/regex_dsl.md)

## Advanced

- [Logits Processors](./advanced/logits_processors.md)

---

### Features Models Index (`docs/features/models/index.md`)

---
title: Models
---

# Models

## Overview

Outlines models are objects that wrap an inference client or engine. Models provide a standardized interface to generate structured text.

All Outlines model classes have an associated loader function to facilitate initializing a model instance. The name of this function is `from_` plus the name of the model in lower-case letters. For instance, Outlines has a `Transformers` model and an associated `from_transformers` loader function. The parameters to load a model are specific to each provider, please consult the documentation of the model you want to use for more information.

After having created a model instance, you can either directly call it to generate text or first create a reusable generator that you would then call.

The input you must provide to a model to generate text can be a simple text prompt or a vision or chat input for models that support them. See the [model inputs section](../core/inputs.md) for more information on model inputs formats.

In all cases, you can provide an `output_type` to constrain the format of the generation output. See the [output types section](../core/output_types.md) for more information on constrained generation.

For instance:

```python
from outlines import from_transformers, Generator
import transformers

# Create a model
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)

# Call it directly
response = model("How many countries are there in the world", max_new_tokens=20)
print(response) # 'There are 200 countries in the world.'

# Call it directly with an output_type
response = model("How many countries are there in the world", int, max_new_tokens=20)
print(response) # '200'

# Create a generator first and then call it
generator = Generator(model, int)
response = generator("How many countries are there in the world")
print(response) # '200'
```

Some models support streaming through a `stream` method. It takes the same argument as the `__call__` method, but returns an iterator instead of a string.

For instance:

```python
from outlines import from_openai, Generator
import openai

# Create the model
model = from_openai(
    openai.OpenAI(),
    "gpt-4o"
)

# Stream the response
for chunk in model.stream("Tell a short story about a cat.", max_tokens=50):
    print(chunk) # 'This...'
```

Additionally, some models support batch processing through a `batch` method. It's similar to the `__call__` method, but takes a list of prompts instead of a single prompt and returns a list of strings.

For instance:

```python
from outlines import from_transformers, Generator
import transformers

# Create a model
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)

# Call it directly
response = model.batch(["What's the capital of Latvia?", "What's the capital of Estonia?"], max_new_tokens=20)
print(response) # ['Riga', 'Tallinn']
```

## Features Matrix

In alphabetical order:

| | [Anthropic](anthropic.md) | [Dottxt](dottxt.md) | [Gemini](gemini.md) | [LlamaCpp](llamacpp.md) | [LM Studio](lmstudio.md) | [MLXLM](mlxlm.md) | [Mistral](mistral.md) | [Ollama](ollama.md) | [OpenAI](openai.md) | [SGLang](sglang.md) | [TGI](tgi.md) | [Transformers](transformers.md) | [Transformers MultiModal](transformers_multimodal.md) | [VLLM](vllm.md) | [VLLMOffline](vllm_offline.md) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **Output Types** | | | | | | | | | | | | | | | |
| Simple Types | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| JSON Schema | ❌ | ✅ | 🟠 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Multiple Choice | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Regex | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Grammar | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | 🟠 | ❌ | ✅ | ✅ | ✅ | ✅ |
| **Generation Features** | | | | | | | | | | | | | | | |
| Async | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
| Streaming | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
| Vision | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Batching | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |

## Model Types

Models can be divided into two categories: local models and server-based models.

In the case of local models, the text generation happens within the inference library object used to instantiate the model. This gives Outlines direct access to the generation process (through a logits processor) and means all structured generation output types are available.

The local models available are the following:

- LlamaCpp
- MLXLM
- Transformers
- TransformersMultiModal
- VLLMOffline

In the case of server-based models, the model is initialized with a client that sends a request to a server that is in charge of the actual text generation. As a result, we have limited control over text generation and some output types are not supported. The server on which the text generation happens can either be remote (with OpenAI or Anthopic for instance) or local (with SGLang for instance).

The server-based models available are the following:

- Anthropic
- Dottxt
- Gemini
- LM Studio
- Mistral
- Ollama
- OpenAI
- SgLang
- TGI
- VLLM

Some models have an async version. To use them, just pass the async version of the provider object to their loading function. It will then return a `Async<ModelName>` instance with the same methods and features as the regular sync instance.

For instance:

```python
from outlines import from_tgi
from huggingface_hub import AsyncInferenceClient

model = from_tgi(
    AsyncInferenceClient("http://localhost:8000/v1")
)
print(type(model)) # outlines.models.tgi.AsyncTGI
```

The models that have an async version are the following:

- LM Studio
- Mistral
- Ollama
- OpenAI
- SgLang
- TGI
- VLLM

---

### Core Concepts (`docs/core_concepts.md`)

---
title: Core concepts
---

# Core concepts

Coming soon. This will document various concepts at a high level, so users can understand Outlines before diving into specific implementations.

1. Constrained decoding, tokens, and the basics of logit biasing
2. Different ways to define output structure (regex, JSON schema, Pydantic models, context-free grammars)
3. How finite state machines are used to guarantee output structure
4. `Generator`, `Application`, `Template`,
5. Prompt engineering vs. structured generation

---

### Community Contribute (`docs/community/contribute.md`)

---
title: Contribute
---

--8<-- "CONTRIBUTING.md"

---

### Community Examples (`docs/community/examples.md`)

# Community projects and articles

Publishing examples and articles about Outlines are a meaningful way to contribute to the community. Here is a list of projects we are aware of. Drop us a line if we forgot yours!

[MMSG](https://github.com/leloykun/mmsg) is a Python library for generating interleaved text and image content in a structured format you can directly pass to downstream APIs.

[Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report](https://arxiv.org/abs/2406.11403) shows that Structured Generation can outperform finetuning, and maybe even multimodality, in document-image understanding tasks as part of CVPR's 2nd MMFM Challenge.

[Chess LLM Arena](https://huggingface.co/spaces/mlabonne/chessllm) is a HuggingFace Space where you can make LLMs compete in a chess match.

[LLM Data Gen](https://huggingface.co/spaces/lhoestq/LLM_DataGen) is a HuggingFace Space that generates synthetic dataset files in JSONLines format.

[Fast, High-Fidelity LLM Decoding with Regex Constraints ](https://vivien000.github.io/blog/journal/llm-decoding-with-regex-constraints.html) presents an efficient alternative to Outlines's structured generation.

[gigax](https://github.com/GigaxGames/gigax) is an Open-Source library that allows to create real-time LLM-powered NPCs for video games.

[Improving Prompt Consistency with Structured Generations](https://huggingface.co/blog/evaluation-structured-outputs) shows how structured generation can improve consistency of evaluation runs by reducing sensitivity to changes in prompt format.

[AskNews](https://asknews.app) is a news curation service processing 300k news articles per day in a structured way, with Outlines.

---

### Community Feedback (`docs/community/feedback.md`)

---
title: Feedback
---

# Feedback

If Outlines has been helpful to you, let us know on [Discord][discord] or give us a shoutout on [Twitter][twitter]! It's always heartwarming ❤️


<head>
  
  
  
  <style>
    .tweet-masonry {
      column-count: 2;
      column-gap: 20px;
      padding: 20px;
    }

    .twitter-tweet {
      display: inline-block;
      width: 100%;
      margin-bottom: 20px;
      margin-top: 0px !important;
      break-inside: avoid;
    }

    @media (max-width: 600px) {
      .tweet-masonry {
        column-count: 1;
      }
    }
  </style>
</head>
<body>

<div class="tweet-masonry">

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">I am once again reminding you that structured extraction using LLMs is going to transform every single industry in the next 10 years <a href="https://t.co/xQ3tcWnrZ8">https://t.co/xQ3tcWnrZ8</a></p>&mdash; Sam Hogan (@0xSamHogan) <a href="https://twitter.com/0xSamHogan/status/1780637917737816323?ref_src=twsrc%5Etfw">April 17, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">outline&#39;s growth is insane, using is an understatement! <a href="https://t.co/rHCNWhZdCs">https://t.co/rHCNWhZdCs</a></p>&mdash; jason liu (@jxnlco) <a href="https://twitter.com/jxnlco/status/1780618454040797554?ref_src=twsrc%5Etfw">April 17, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Outlines is an amazing lib and more popular than <a href="https://twitter.com/remilouf?ref_src=twsrc%5Etfw">@remilouf</a>’s modesty will admit. <a href="https://t.co/DfHbMPIlX1">https://t.co/DfHbMPIlX1</a> <a href="https://t.co/mDHIWJrD0C">https://t.co/mDHIWJrD0C</a></p>&mdash; Delip Rao e/σ (@deliprao) <a href="https://twitter.com/deliprao/status/1780780217180598377?ref_src=twsrc%5Etfw">April 18, 2024</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Impressive implementation of a true regex / json / grammar guided text generation <a href="https://t.co/RX5RVYaVIx">pic.twitter.com/RX5RVYaVIx</a></p>&mdash; Rohan Paul (@rohanpaul_ai) <a href="https://twitter.com/rohanpaul_ai/status/1741099984299135403?ref_src=twsrc%5Etfw">December 30, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Most underrated Github Repo in AI + LLM JSON guided Generation: <a href="https://t.co/lSB8KIet1H">https://t.co/lSB8KIet1H</a></p>&mdash; 🎙Jean-Louis Queguiner (@JiliJeanlouis) <a href="https://twitter.com/JiliJeanlouis/status/1736857292581093706?ref_src=twsrc%5Etfw">December 18, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Nice and useful. <a href="https://t.co/LX72AE0lgt">https://t.co/LX72AE0lgt</a></p>&mdash; Dan Roy (@roydanroy) <a href="https://twitter.com/roydanroy/status/1691556956941525458?ref_src=twsrc%5Etfw">August 15, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">HUGE dub for open source AI <a href="https://t.co/bYKuiEUZ1j">https://t.co/bYKuiEUZ1j</a></p>&mdash; kenneth 🖇 (@k3nnethfrancis) <a href="https://twitter.com/k3nnethfrancis/status/1691304781732843521?ref_src=twsrc%5Etfw">August 15, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">This is amazing - glad to see more outp guidance modules! <br><br>Will try this out soon I&#39;m wondering how they translate from regex automatons to token boundaries<br><br>Also why Open Source will succeed. Even today I don&#39;t see any guided output functionality from the big providers. <a href="https://t.co/Ity2H25Klf">https://t.co/Ity2H25Klf</a></p>&mdash; Hrishi (@hrishioa) <a href="https://twitter.com/hrishioa/status/1691181499671080960?ref_src=twsrc%5Etfw">August 14, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Outlines - a library to help LLM developers guide text generation in a fast and reliable way.<br><br>&quot;Provides generation methods that guarantee that the output will match a regular expressions, or follow a JSON schema.&quot;<br><br>Need to check this out. Reliable JSON output is a common use… <a href="https://t.co/Bkbh8vKogN">pic.twitter.com/Bkbh8vKogN</a></p>&mdash; elvis (@omarsar0) <a href="https://twitter.com/omarsar0/status/1691179888214966273?ref_src=twsrc%5Etfw">August 14, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Woah this is cool! Makes open source models more usable.<br><br>Give any LLM Function Call capability (and more) with Outlines: <a href="https://t.co/PtPykR5ZGR">https://t.co/PtPykR5ZGR</a> <a href="https://t.co/RRQjWHnIxv">https://t.co/RRQjWHnIxv</a> <a href="https://t.co/BwNnH8SMwv">pic.twitter.com/BwNnH8SMwv</a></p>&mdash; Yohei (@yoheinakajima) <a href="https://twitter.com/yoheinakajima/status/1691231912466223104?ref_src=twsrc%5Etfw">August 14, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">This is awesome! Being able to guarantee the output&#39;s structure unblocks so many applications. This is a great milestone and a fundamental building block for more advanced AI apps. <a href="https://t.co/WdwMOc7hE8">https://t.co/WdwMOc7hE8</a></p>&mdash; Guilherme Castro (@skastr052) <a href="https://twitter.com/skastr052/status/1691239359494619136?ref_src=twsrc%5Etfw">August 15, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Juggling with the unpredictable outputs of ChatGPT API lately while building my product. 😓 <br><br>Tried prompt engineering to channel its wisdom into a neat JSON, but it&#39;s like asking a cat to fetch. 🐱<br><br>Luckily, stumbled upon &quot;Outlines&quot; – looks like a promising way to tame the LLM… <a href="https://t.co/oYQ6q8exAS">pic.twitter.com/oYQ6q8exAS</a></p>&mdash; Charlie (@14435635Sun) <a href="https://twitter.com/14435635Sun/status/1691439342689095680?ref_src=twsrc%5Etfw">August 15, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">A complex system of LLM input-outputs interacting with non-LLM agents and models benefits immeasurably from structured outputs. The outlines package saves so much time, <a href="https://t.co/NhVQ6NpKDR">https://t.co/NhVQ6NpKDR</a></p>&mdash; Amir Sani (@amirsani) <a href="https://twitter.com/amirsani/status/1728734266568376433?ref_src=twsrc%5Etfw">November 26, 2023</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div>
</body>
</html>

# Let us know!

We highly value the insights of our users, and we would love to hear from you. If you are using Outlines for your projects and would like to share your experience with us, let's connect:

- What are you building with it?
- What do you like about it?
- What challenges are you facing?
- What do you think could be improved?

To schedule an appointment follow [this link](https://cal.com/dottxt/outlines). This is exclusively intended to share your experience, please go on [Discord][discord] or [GitHub](https://github.com/dottxt-ai/outlines/discussions) for support.

[discord]: https://discord.gg/UppQmhEpe8
[twitter]: https://twitter.com/dottxtai

---

### Community Versioning (`docs/community/versioning.md`)

---
title: Versioning Guide
---

# Versioning Guide


The Outlines project follows a structured versioning scheme designed to provide clarity and minimize risk for downstream dependents.

Each part of the version number (`major.minor.patch`) conveys information about the nature and impact of the changes included in the release.

- **Major Releases** includes compatibility-breaking changes to core interfaces, such as `LogitsProcessor`s and `Guides`.
- **Minor Releases** introduce changes of substance to internal or unexposed functionality. These changes are well tested and intended to maintain compatibility with existing use of core interfaces.
- **Patch Releases** address bug fixes and incorporate low-risk changes to improve stability and performance.

!!! note "Breaking Changes"

    Outlines v1.0 introduced several breaking changes to the core interface. See [the migration guide](../guide/migration.md) for more details.

## Releases

Releases along with release notes can be found on the [Outlines Releases GitHub Page](https://github.com/dottxt-ai/outlines/releases).

## Version Pinning Recommendations

Here are our recommendations for managing dependencies on the Outlines package:

**Small, Risk-Tolerant Projects:** Pin to a specific major version.

**Large, Conservative Projects:** Pin to a specific minor version.

---

### Examples Chain Of Density (`docs/examples/chain_of_density.md`)

# Summarize documents using Chain of Density prompting

A good summary should be informative, concise and clear. While large language models are generally good at summarizing documents, their summaries tend to be long and contain redundant information; their information density tends to be on the lower end. This is where [chain of Density](https://arxiv.org/abs/2309.04269), a new prompting technique, comes in. In this example we will show how one can implement chain of density with a few lines of code using Outlines, leveraging both Outline's prompt templating and its structured generation capabilities.

The article we will try to summarize is the first three paragraphs of the [Alan Turing page on Wikipedia](https://en.wikipedia.org/wiki/Alan_Turing):

```
/* Detailed source-code truncated for AI context efficiency. */
```

## How Chain Of Density works

Chain Of Density starts with asking the model to generate a first long and non-specific summary. Then it asks the model to generate 4 extra summaries by proceeding in the following way:

1. Identify 1-3 entities missing in the previous summary;
2. Add all entities marked as missing in the previous step, while not dropping entities;
3. Make the summary more concise;

The prompt also asks the model to return a list of JSON objects that contain the missing entities and the new summary. This is where structured generation will come in handy :) The paper provides the prompt and an example:



We can now implement the prompt provided in the paper. We stored the prompt template in a text file, and we can load it using the `Template` class:

```python
from outlines import Template

chain_of_density = Template.from_file("prompt_templates/chain_of_density.txt")
```

??? Note

    Note that we modified the prompt slightly so it returns a JSON object that contains the summaries, instead of a list of summaries.


## Outlines implementation

We will use Outline's JSON-structured generation to ensure that the model's output is consistent with the format specified in the prompt. We start with defining the JSON objects that the model is asked to return using Pydantic. One JSON object that contains a list of `Summary` objects that contain the missing entities and new summary:

```python
from pydantic import BaseModel, conlist

class Summary(BaseModel):
    missing_entities: str
    denser_summary: str

class Summaries(BaseModel):
    summaries: conlist(Summary, max_length=5, min_length=5)
```

We now generate the prompt by passing the article we want to summarize to the prompt template previously loaded. We load a quantized version of Mistral-7B using the AutoAWQ library, and then use the `Summaries` schema to generate the summaries with structured generation:

```python
import outlines
import transformers

MODEL_NAME = "TheBloke/Mistral-7B-OpenOrca-AWQ"

model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(MODEL_NAME),
    transformers.AutoTokenizer.from_pretrained(MODEL_NAME)
)
prompt = chain_of_density(article=article)
result = model(prompt, Summaries, max_new_tokens=2000)
```

We can now check the results:

```python
print(result)
# {'summaries': [
#     {
#       'missing_entities': 'English mathematician, cryptanalyst, philosopher',
#       'denser_summary': 'Alan Mathison Turing was an English mathematician, cryptanalyst, philosopher.'
#     },
#     {
#       'missing_entities': '',
#       'denser_summary': "Alan Mathison Turing was an English mathematician who was a crucial figure in WW2's Bletchley Park codebreaking centre and designed one of the first computers."
#     },
#     {
#       'missing_entities': 'cryptanalyst, studied, biology, father',
#       'denser_summary': 'Alan Mathison Turing was an English cryptanalyst, studied theoretical computer science, and contributed to mathematical biology.'
#     },
#     {
#       'missing_entities': 'biology, morphogenesis, chemical',
#       'denser_summary': 'Alan Mathison Turing was an English cryptanalyst, studied theoretical computer science, and predicted chemical reactions in morphogenesis.
#     '},
#     {
#       'missing_entities': '',
#       'denser_summary': 'Alan Mathison Turing was an English cryptanalyst, developed computer science, and made strides in mathematical biology research.'
#       }
# ]}
```

Not bad, considering we used a smallish model to generate the summary! Chain of Density seems to be a very effective prompting technique to generate dense summaries, even with small quantized models. Its implementation in Outlines is also very short.

Note that this is the first article I tried and it worked out of the box. Try it out on other articles, and please share the results on Twitter, or by opening [a new discussion](https://github.com/dottxt-ai/outlines/discussions/categories/show-and-tell) on the Outlines repository!

---

### Examples Chain Of Thought (`docs/examples/chain_of_thought.md`)

# Chain of thought


Chain of thought is a prompting technique introduced in the paper ["Chain-of-Thought Prompting Elicits Reasoning in Large Language Models"](https://arxiv.org/abs/2201.11903) where throught prompting the authors generate a series of intermediate reasoning steps which improves the ability of LLMs to perform complex reasoning.

In this guide, we use [outlines](https://dottxt-ai.github.io/outlines/) to apply chain of thought through structured output.

We use [llama.cpp](https://github.com/ggerganov/llama.cpp) using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) library. Outlines supports llama-cpp-python, but we need to install it ourselves:

```shell
pip install llama-cpp-python
```

To create an outlines `LlamaCpp` model, you first need to create a `Llama` object from the `llama-cpp-python` library. Then you can create the outlines model by calling `models.from_llamacpp` with the `Llama` object instance as argument. To create the `Llama` object, you need to provide the model weights by passing the name of the repository on the HuggingFace Hub, and the filenames or glob pattern (it will automatically download the weights from the hub):

```python
import llama_cpp
import outlines

llm = llama_cpp.Llama(
    "NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF",
    tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
        "NousResearch/Hermes-2-Pro-Llama-3-8B"
    ),
    n_gpu_layers=-1,
    flash_attn=True,
    n_ctx=8192,
    verbose=False
)
model = outlines.from_llamacpp(llm)
```

??? note "(Optional) Store the model weights in a custom folder"

    By default the model weights are downloaded to the hub cache but if we want so store the weights in a custom folder, we pull a quantized GGUF model [Hermes-2-Pro-Llama-3-8B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF) by [NousResearch](https://nousresearch.com/) from [HuggingFace](https://huggingface.co/):

    ```shell
    wget https://hf.co/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF/resolve/main/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf
    ```

    We initialize the model:

    ```python
    from llama_cpp import Llama

    llm = Llama("/path/to/model/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf", ...)
    ```

## Chain of thought

We first define our Pydantic class for a reasoning step:

```python
from pydantic import BaseModel, Field

class Reasoning_Step(BaseModel):
    reasoning_step: str = Field(..., description="Reasoning step")
```

We then define the Pydantic class for reasoning which will consist on a list of reasoning steps and a conclusion, and we get its JSON schema:

```python
from typing import List

class Reasoning(BaseModel):
    reasoning: List[Reasoning_Step] = Field(..., description="List of reasoning steps")
    conclusion: str = Field(..., description="Conclusion")

json_schema = Reasoning.model_json_schema()
```

We then need to adapt our prompt to the [Hermes prompt format for JSON schema](https://github.com/NousResearch/Hermes-Function-Calling?tab=readme-ov-file#prompt-format-for-json-mode--structured-outputs):

```python
from outlines import Template

generate_hermes_prompt = Template.from_string(
    """
    <|im_start|>system
    You are a world class AI model who answers questions in JSON
    Here's the json schema you must adhere to:
    <schema>
    {{ json_schema }}
    </schema>
    <|im_end|>
    <|im_start|>user
    {{ user_prompt }}
    <|im_end|>
    <|im_start|>assistant
    <schema>
    """
)
```

For a given user prompt:

```python
user_prompt = "9.11 and 9.9 -- which is bigger?"
```

We can use `outlines.Generator` with the Pydantic class we previously defined, and call the generator with the Hermes prompt:

```python
generator = outlines.Generator(model, regex_str)
prompt = generate_hermes_prompt(json_schema=json_schema, user_prompt=user_prompt)
response = generator(prompt, max_tokens=1024, temperature=0, seed=42)
```

We obtain a series of intermediate reasoning steps as well as the conclusion:

```python
import json

json_response = json.loads(response)

print(json_response["reasoning"])
print(json_response["conclusion"])
# [{'reasoning_step': 'Both 9.11 and 9.9 are decimal numbers.'},
#  {'reasoning_step': 'When comparing decimal numbers, we look at the numbers after the decimal point.'},
#  {'reasoning_step': 'In this case, 9.11 has the number 1 after the decimal point, while 9.9 has the number 9.'},
#  {'reasoning_step': 'Since 1 is greater than 9, 9.11 is greater than 9.9.'}]
# '9.11 is bigger.'
```

We notice that the 4th reasoning step is wrong ``Since 1 is greater than 9, 9.11 is greater than 9.9.'', so we should probably give the model some examples for this particular task.

This example was originally contributed by [Alonso Silva](https://github.com/alonsosilvaallende).

---

### Examples Classification (`docs/examples/classification.md`)

# Classification

Classification is a classic problem in NLP and finds many applications: spam detection, sentiment analysis, triaging of incoming requests, etc. We will use the example of a company that wants to sort support requests between those that require immediate attention (`URGENT`), those that can wait a little (`STANDARD`). You could easily extend the example by adding new labels.


This tutorial shows how one can implement multi-label classification using Outlines.

As always, we start with initializing the model. Since we are GPU poor we will be using a quantized version of Mistal-7B-v0.1:

```python
import outlines
import transformers

MODEL_NAME = "TheBloke/Mistral-7B-OpenOrca-AWQ"

model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(MODEL_NAME),
    transformers.AutoTokenizer.from_pretrained(MODEL_NAME)
)
```

We will use a prompt template stored in a text file:

```python
from outlines import Template

customer_support = Template.from_file("prompt_templates/classification.txt")
```

## Choosing between multiple choices

Outlines provides a convenient way to do multi-label classification, passing a Literal type hint to the `outlines.Generator` object:

```python
from typing import Literal
import outlines

generator = outlines.Generator(model, Literal["URGENT", "STANDARD"])

```
Outlines supports batched requests, so we will pass two requests to the model:

```python
requests = [
    "My hair is one fire! Please help me!!!",
    "Just wanted to say hi"
]

prompts = [customer_support(request=request) for request in requests]
```

We can now ask the model to classify the requests:

```python
labels = generator(prompts)
print(labels)
# ['URGENT', 'STANDARD']
```

## Using JSON-structured generation

Another (convoluted) way to do multi-label classification is to JSON-structured generation in Outlines. We first need to define our Pydantic schema that contains the labels:

```python
from enum import Enum
from pydantic import BaseModel


class Label(str, Enum):
    urgent = "URGENT"
    standard = "STANDARD"


class Classification(BaseModel):
    label: Label
```

We can then create a generator with the Pydantic model we just defined and call it:

```python
generator = outlines.Generator(model, Classification)
labels = generator(prompts)
print(labels)
# ['{"label":"URGENT"}', '{ "label": "STANDARD" }']
```

---

### Examples Dating Profiles (`docs/examples/dating_profiles.md`)

# Generate a synthetic dating profile from a description

In this example we will see how we can use Outlines to generate synthetic data for a dating application. This example was originally contributed by [Vibhor Kumar](https://github.com/veezbo).

```python
import json
from dataclasses import dataclass
from enum import Enum

import torch
import transformers
from pydantic import BaseModel, conlist, constr

import outlines
```

## Defining the profile with Pydantic

Here a dating profile will consist in a biography, a job, a list of interests and two question-answer pairs. The questions are written in advance by the team, and the users are asked to provide an answer:

```python
class QuestionChoice(str, Enum):
    A = "The key to my heart is"
    B = "The first item on my bucket list is"
    C = "Perks of dating me"
    D = "Message me if you also love"
    E = "People would describe me as"
    F = "I can beat you in a game of"

@dataclass
class QuestionAnswer:
    question: QuestionChoice
    answer: str
```

Users need to provide a short biography, with a minimum of 10 and a maximum of 300 characters. The application also limits job descriptions to 50 characters. In addition to the question-answer pairs, the user is required to provide a list of between 1 and 5 interests:

```python
class DatingProfile(BaseModel):
    bio: constr(min_length=10, max_length=300)
    job: constr(max_length=50)
    interests: conlist(str, min_length=1, max_length=5)  # type: ignore
    qna1: QuestionAnswer
    qna2: QuestionAnswer
```

## Prompt template and examples

We will ask the model to generate profiles from a high-level description:

```python
@dataclass
class Example:
    description: str
    profile: DatingProfile
```

We will use Outlines' prompt templating abilities to generate the prompt for us. This help clearly separate the general prompting logic from what is specific to an example.

```python
from outlines import Template

dating_profile_prompt = Template.from_string(
    """
    You are a world-renowned matchmaker who understands the modern dating
    market. Your job is to generate dating app profiles for male clients
    interested in women based on a provided description. The profiles should be
    authentic, show off their strengths, and maximize their likelihood of
    getting matches on dating apps.  Here are some examples of past clients that
    you have successfully created profiles for:

    {% for example in examples %}
    Description:
    {{ example.description }}
    Profile:
    {{ example.profile }}
    {% endfor %}

    Here is the new client who you need to create a profile for:
    Description: {{ description }}
    Profile:
    """
)
```

We will provide the model with several few-shot examples:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Load the model

We will use Mosaic's MPT-7B model (requires 13GB of GPU memory) which can fit on a single GPU with a reasonable context window. We initialize it with Outlines:

```python
MODEL_NAME = "mosaicml/mpt-7b-8k-instruct"

config = transformers.AutoConfig.from_pretrained(
    MODEL_NAME, trust_remote_code=True
)
config.init_device = "meta"
model_kwargs = {
    "config": config,
    "trust_remote_code": True,
    "torch_dtype": torch.bfloat16,
    "device_map": "cuda",
}
tf_model = transformers.AutoModelForCausalLM.from_pretrained(MODEL_NAME, **model_kwargs)
tf_tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_NAME)
model = outlines.from_transformers(tf_model, tokenizer=tf_tokenizer)
```

## JSON-structured generation of profiles

We will now generate a dating profile from a textual description of oneself:

``` python
new_description = """I'm a laid-back lawyer who spends a lot of his free-time
gaming. I work in a corporate office, but ended up here after the start-up  I
cofounded got acquired, so still play ping pong with my cool coworkers every
day.  I have a bar at home where I make cocktails, which is great for
entertaining  friends. I secretly like to wear suits and get a new one tailored
every few  months. I also like weddings because I get to wear those suits, and
it's  a good excuse for a date. I watch the latest series because I'm paying,
with my hard-earned money, for every streaming service."""

prompt = dating_profile_prompt(description=new_description, examples=samples)
profile = model(prompt, DatingProfile)
parsed_profile = DatingProfile.model_validate_json(json.loads(profile))
```

## Results

Here are a couple of results:

```json
{
    "bio": """I'm an ambitious lawyer with a casual and fashionable style. I love
    games and sports, but my true passion is preparing refreshing cocktails at
    home and dressing to the nines at weddings. I'm currently looking for a woman
    to show a good time to and get a kiss on the opulent suit I just had made.
    Send resume to this inbox.""",
    "job": "Lawyer",
    "interests":
    [
        "Stylish guys",
        "Gaming",
        "Ping pong",
        "Cocktails",
        "Weddings"
    ],
    "qna1":
    {
        "question": "The first item on my bucket list is",
        "answer": "be married and have a family."
    },
    "qna2":
    {
        "question": "People would describe me as",
        "answer": "charming, stylish, and funny."
    }
}
```

```json
{
    "bio": """I’m a sexy lawyer with time on my hands. I love to game and
    play ping pong, but the real reason you should swipe to the right
    is because I look great in a suit. Who doesn’t love a man in a
    suit? Just saying. Send me a message if you think it’s time to take
    your dating life to the next level.""",
    "job": "Lawyer",
    "interests":
    [
        "Gaming",
        "Ping Pong",
        "Tailored Suits",
        "Weddings",
        "Streaming Services"
    ],
    "qna1":
    {
        "question": "The first item on my bucket list is",
        "answer": "simulate space but stay alive for as long as possible"
    },
    "qna2":
    {
        "question": "People would describe me as",
        "answer": "easy-going, a little nerdy but with a mature essence"
    }
}
```

---

### Examples Deploy Using Bentoml (`docs/examples/deploy-using-bentoml.md`)

# Run Outlines using BentoML

[BentoML](https://github.com/bentoml/BentoML) is an open-source model serving library for building performant and scalable AI applications with Python. It comes with tools that you need for serving optimization, model packaging, and production deployment.

In this guide, we will show you how to use BentoML to run programs written with Outlines on GPU locally and in [BentoCloud](https://www.bentoml.com/), an AI Inference Platform for enterprise AI teams. The example source code in this guide is also available in the [examples/bentoml/](https://github.com/dottxt-ai/outlines/blob/main/examples/bentoml/) directory.

## Import a model

First we need to download an LLM (Mistral-7B-v0.1 in this example and you can use any other LLM) and import the model into BentoML's [Model Store](https://docs.bentoml.com/en/latest/guides/model-store.html). Let's install BentoML and other dependencies from PyPi (preferably in a virtual environment):

```shell
pip install -r requirements.txt
```

Then save the code snippet below as `import_model.py` and run `python import_model.py`.

**Note**: You need to accept related conditions on [Hugging Face](https://huggingface.co/mistralai/Mistral-7B-v0.1) first to gain access to Mistral-7B-v0.1.

```python
import bentoml

MODEL_ID = "mistralai/Mistral-7B-v0.1"
BENTO_MODEL_TAG = MODEL_ID.lower().replace("/", "--")

def import_model(model_id, bento_model_tag):

    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer

    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID,
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
    )

    with bentoml.models.create(bento_model_tag) as bento_model_ref:
        tokenizer.save_pretrained(bento_model_ref.path)
        model.save_pretrained(bento_model_ref.path)


if __name__ == "__main__":
    import_model(MODEL_ID, BENTO_MODEL_TAG)
```

You can verify the download is successful by running:

```shell
$ bentoml models list

Tag                                          Module  Size        Creation Time
mistralai--mistral-7b-v0.1:m7lmf5ac2cmubnnz          13.49 GiB   2024-04-25 06:52:39
```

## Define a BentoML Service

As the model is ready, we can define a [BentoML Service](https://docs.bentoml.com/en/latest/guides/services.html) to wrap the capabilities of the model.

We will run the JSON-structured generation example [in the README](https://github.com/dottxt-ai/outlines?tab=readme-ov-file#efficient-json-generation-following-a-json-schema), with the following schema:

```python
DEFAULT_SCHEMA = """{
    "title": "Character",
    "type": "object",
    "properties": {
        "name": {
            "title": "Name",
            "maxLength": 10,
            "type": "string"
        },
        "age": {
            "title": "Age",
            "type": "integer"
        },
        "armor": {"$ref": "#/definitions/Armor"},
        "weapon": {"$ref": "#/definitions/Weapon"},
        "strength": {
            "title": "Strength",
            "type": "integer"
        }
    },
    "required": ["name", "age", "armor", "weapon", "strength"],
    "definitions": {
        "Armor": {
            "title": "Armor",
            "description": "An enumeration.",
            "enum": ["leather", "chainmail", "plate"],
            "type": "string"
        },
        "Weapon": {
            "title": "Weapon",
            "description": "An enumeration.",
            "enum": ["sword", "axe", "mace", "spear", "bow", "crossbow"],
            "type": "string"
        }
    }
}"""
```

First, we need to define a BentoML service by decorating an ordinary class (`Outlines` here) with `@bentoml.service` decorator. We pass to this decorator some configuration and GPU on which we want this service to run in BentoCloud (here an L4 with 24GB memory):

```python
import typing as t
import bentoml

from import_model import BENTO_MODEL_TAG

@bentoml.service(
    traffic={
        "timeout": 300,
    },
    resources={
        "gpu": 1,
        "gpu_type": "nvidia-l4",
    },
)
class Outlines:

    bento_model_ref = bentoml.models.get(BENTO_MODEL_TAG)

    def __init__(self) -> None:
        import outlines
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer

        # Load tokenizer and model from the BentoML model reference path
        hf_tokenizer = AutoTokenizer.from_pretrained(self.bento_model_ref.path)
        hf_model = AutoModelForCausalLM.from_pretrained(
            self.bento_model_ref.path,
            torch_dtype=torch.float16,
            low_cpu_mem_usage=True,
            device_map="cuda"
        )

        # Then use the loaded model with Outlines
        self.model = outlines.from_transformers(hf_model, hf_tokenizer)

    ...
```

We then need to define an HTTP endpoint using `@bentoml.api` to decorate the method `generate` of `Outlines` class:

```python
    ...

    @bentoml.api
    async def generate(
        self,
        prompt: str = "Give me a character description.",
        json_schema: t.Optional[str] = DEFAULT_SCHEMA,
    ) -> t.Dict[str, t.Any]:
        import json
        import outlines
        from outlines.types import JsonSchema

        generator = outlines.Generator(self.model, JsonSchema(json_schema))
        character = generator(prompt)

        return json.loads(character)
```

Here `@bentoml.api` decorator defines `generate` as an HTTP endpoint that accepts a JSON request body with two fields: `prompt` and `json_schema` (optional, which allows HTTP clients to provide their own JSON schema). The type hints in the function signature will be used to validate incoming JSON requests. You can define as many HTTP endpoints as you want by using `@bentoml.api` to decorate other methods of `Outlines` class.

Now you can save the above code to `service.py` (or use [this implementation](https://github.com/dottxt-ai/outlines/blob/main/examples/bentoml/)), and run the code using the BentoML CLI.

## Run locally for testing and debugging

Then you can run a server locally by:

```shell
bentoml serve .
```

The server is now active at <http://localhost:3000>. You can interact with it using the Swagger UI or in other different ways:

<details>

<summary>CURL</summary>

```shell
curl -X 'POST' \
  'http://localhost:3000/generate' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "Give me a character description."
}'
```

</details>

<details>

<summary>Python client</summary>

```python
import bentoml

with bentoml.SyncHTTPClient("http://localhost:3000") as client:
    response = client.generate(
        prompt="Give me a character description"
    )
    print(response)
```

</details>

Expected output:

```shell
{
  "name": "Aura",
  "age": 15,
  "armor": "plate",
  "weapon": "sword",
  "strength": 20
}
```

## Deploy to BentoCloud

After the Service is ready, you can deploy it to [BentoCloud](https://docs.bentoml.com/en/latest/bentocloud/get-started.html) for better management and scalability. [Sign up](https://cloud.bentoml.com/signup) if you haven't got a BentoCloud account.

Make sure you have [logged in to BentoCloud](https://docs.bentoml.com/en/latest/bentocloud/how-tos/manage-access-token.html), then run the following command to deploy it.

```shell
bentoml deploy .
```

Once the application is up and running on BentoCloud, you can access it via the exposed URL.

**Note**: For custom deployment in your own infrastructure, use [BentoML to generate an OCI-compliant image](https://docs.bentoml.com/en/latest/guides/containerization.html).

---

### Examples Deploy Using Cerebrium (`docs/examples/deploy-using-cerebrium.md`)

# Run Outlines using Cerebrium

[Cerebrium](https://www.cerebrium.ai/) is a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications. They offer Serverless GPU's with low cold start times with over 12 varieties of GPU chips that auto scale and you only pay for the compute you use.

In this guide we will show you how you can use Cerebrium to run programs written with Outlines on GPUs in the cloud.

# Setup Cerebrium

First, we install Cerebrium and login to get authenticated.

```shell
pip install cerebrium
cerebrium login
```

Then let us create our first project

```shell
cerebrium init outlines-project
```

## Setup Environment and Hardware

You set up your environment and hardware in the cerebrium.toml file that was created using the init function above.

```toml
[cerebrium.deployment]
docker_base_image_url = "nvidia/cuda:12.1.1-runtime-ubuntu22.04"

[cerebrium.hardware]
cpu = 2
memory = 14.0
gpu = "AMPERE A10"
gpu_count = 1
provider = "aws"
region = "us-east-1"

[cerebrium.dependencies.pip]
outline = "==1.0.0"
transformers = "==4.38.2"
datasets = "==2.18.0"
accelerate = "==0.27.2"
```

## Setup inference

Running code in Cerebrium is like writing normal python with no special syntax. In a `main.py` file specify the following:

```python
import outlines
import transformers
from outlines.types import JsonSchema


model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

schema = """{
    "title": "Character",
    "type": "object",
    "properties": {
        "name": {
            "title": "Name",
            "maxLength": 10,
            "type": "string"
        },
        "age": {
            "title": "Age",
            "type": "integer"
        },
        "armor": {"$ref": "#/definitions/Armor"},
        "weapon": {"$ref": "#/definitions/Weapon"},
        "strength": {
            "title": "Strength",
            "type": "integer"
        }
    },
    "required": ["name", "age", "armor", "weapon", "strength"],
    "definitions": {
        "Armor": {
            "title": "Armor",
            "description": "An enumeration.",
            "enum": ["leather", "chainmail", "plate"],
            "type": "string"
        },
        "Weapon": {
            "title": "Weapon",
            "description": "An enumeration.",
            "enum": ["sword", "axe", "mace", "spear", "bow", "crossbow"],
            "type": "string"
        }
    }
}"""

generator = outlines.Generator(model, JsonSchema(schema))
```

On first deploy, it will download the model and store it on disk therefore for subsequent calls it will load the model from disk.

Every function in Cerebrium is callable through an API endpoint. Code at the top most layer (ie: not in a function) is instantiated only when the container is spun up the first time so for subsequent calls, it will simply run the code defined in the function you call.

To deploy an API that creates a new character when called with a prompt you can add the following code to `main.py`:

```python
def generate(
    prompt: str = "Amiri, a 53 year old warrior woman with a sword and leather armor.",
):

    character = generator(
        f"<s>[INST]Give me a character description. Describe {prompt}.[/INST]"
    )

    return character
```


## Run on the cloud

```shell
cerebrium deploy
```

You will see your application deploy, install pip packages and download the model. Once completed it will output a CURL request you can use to call your endpoint. Just remember to end
the url with the function you would like to call - in this case /generate. You should see your response returned!

---

### Examples Deploy Using Modal (`docs/examples/deploy-using-modal.md`)

# Run Outlines using Modal

[Modal](https://modal.com/) is a serverless platform that allows you to easily run code on the cloud, including GPUs. It can come very handy for those of us who don't have a monster GPU at home and want to be able to quickly and easily provision, configure and orchestrate cloud infrastructure.

In this guide we will show you how you can use Modal to run programs written with Outlines on GPU in the cloud.

## Requirements

We recommend installing `modal` and `outlines` in a virtual environment. You can create one with:

```shell
python -m venv venv
source venv/bin/activate
```

Then install the required packages:

```shell
pip install modal outlines
```

## Build the image

First we need to define our container image. If you need to access a gated model, you will need to provide an [access token](https://huggingface.co/settings/tokens). See the `.env` call below for how to provide a HuggingFace token.

Setting a token is best done by setting an environment variable `HF_TOKEN` with your token. If you do not wish to do this, we provide a commented-out line in the code to set the token directly in the code.

```python
from modal import Image, App, gpu
import os

# This creates a modal App object. Here we set the name to "outlines-app".
# There are other optional parameters like modal secrets, schedules, etc.
# See the documentation here: https://modal.com/docs/reference/modal.App
app = App(name="outlines-app")

# Specify a language model to use.
# Another good model to use is "NousResearch/Hermes-2-Pro-Mistral-7B"
language_model = "mistral-community/Mistral-7B-v0.2"

# Please set an environment variable HF_TOKEN with your Hugging Face API token.
# The code below (the .env({...}) part) will copy the token from your local
# environment to the container.
# More info on Image here: https://modal.com/docs/reference/modal.Image
outlines_image = Image.debian_slim(python_version="3.11").pip_install(
    "outlines",
    "transformers",
    "datasets",
    "accelerate",
    "sentencepiece",
).env({
    # This will pull in your HF_TOKEN environment variable if you have one.
    'HF_TOKEN':os.environ['HF_TOKEN']

    # To set the token directly in the code, uncomment the line below and replace
    # 'YOUR_TOKEN' with the HuggingFace access token.
    # 'HF_TOKEN':'YOUR_TOKEN'
})
```

## Setting the container up

When running longer Modal apps, it's recommended to download your language model when the container starts, rather than when the function is called. This will cache the model for future runs.

```python
# This function imports the model from Hugging Face. The modal container
# will call this function when it starts up. This is useful for
# downloading models, setting up environment variables, etc.
def import_model():
    import outlines
    import transformers

    outlines.from_transformers(
        transformers.AutoModelForCausalLM.from_pretrained(language_model),
        transformers.AutoTokenizer.from_pretrained(language_model)
    )

# This line tells the container to run the import_model function when it starts.
outlines_image = outlines_image.run_function(import_model)
```

## Define a schema

We will run the JSON-structured generation example [in the README](https://github.com/dottxt-ai/outlines?tab=readme-ov-file#efficient-json-generation-following-a-json-schema), with the following schema:

```python
# Specify a schema for the character description. In this case,
# we want to generate a character with a name, age, armor, weapon, and strength.
schema = """{
    "title": "Character",
    "type": "object",
    "properties": {
        "name": {
            "title": "Name",
            "maxLength": 10,
            "type": "string"
        },
        "age": {
            "title": "Age",
            "type": "integer"
        },
        "armor": {"$ref": "#/definitions/Armor"},
        "weapon": {"$ref": "#/definitions/Weapon"},
        "strength": {
            "title": "Strength",
            "type": "integer"
        }
    },
    "required": ["name", "age", "armor", "weapon", "strength"],
    "definitions": {
        "Armor": {
            "title": "Armor",
            "description": "An enumeration.",
            "enum": ["leather", "chainmail", "plate"],
            "type": "string"
        },
        "Weapon": {
            "title": "Weapon",
            "description": "An enumeration.",
            "enum": ["sword", "axe", "mace", "spear", "bow", "crossbow"],
            "type": "string"
        }
    }
}"""
```

To make the inference work on Modal we need to wrap the corresponding function in a `@app.function` decorator. We pass to this decorator the image and GPU on which we want this function to run.

Let's choose an A100 with 80GB memory. Valid GPUs can be found [here](https://modal.com/docs/reference/modal.gpu).

```
/* Detailed source-code truncated for AI context efficiency. */
```

We then need to define a `local_entrypoint` to call our function `generate` remotely.

```python
@app.local_entrypoint()
def main(
    prompt: str = "Amiri, a 53 year old warrior woman with a sword and leather armor.",
):
    # We use the "generate" function defined above -- note too that we are calling
    # .remote() on the function. This tells modal to run the function in our cloud
    # machine. If you want to run the function locally, you can call .local() instead,
    # though this will require additional setup.
    generate.remote(prompt)
```

Here `@app.local_entrypoint()` decorator defines `main` as the function to start from locally when using the Modal CLI. You can save above code to `example.py` (or use [this implementation](https://github.com/dottxt-ai/outlines/blob/main/examples/modal_example.py)). Let's now see how to run the code on the cloud using the Modal CLI.

## Run on the cloud

First install the Modal client from PyPi, if you have not already:

```shell
pip install modal
```

You then need to obtain a token from Modal. Run the following command:

```shell
modal setup
```

Once that is set you can run inference on the cloud using:

```shell
modal run example.py
```

You should see the Modal app initialize, and soon after see the result of the `print` function in your terminal. That's it!

---

### Examples Earnings Reports (`docs/examples/earnings-reports.md`)

# Extracting financial data from earnings reports

A common task in finance is to extract financial data from earnings reports. Earnings reports are infamously poorly formatted, as the SEC does not have requirements for producing machine-readable documents.

Earnings reports are often provided as HTML documents, which can be difficult to parse. Investors often use complicated parsing systems or manual review to extract data. Entire companies are built around automating this task.

This cookbook is a proof of concept about how we can use LLMs to extract financial data directly into CSV. Comma-separated values are well-structured and can be defined by a regular expression, which Outlines can use to guide the LLM's output.

The example is a smaller subset of a full demo found [here](https://github.com/dottxt-ai/demos/tree/main/earnings-reports). The demo contains the full set of pre-processing steps needed to convert raw HTML into a structured CSV file, and tests the results across three company's 10k reports.

## Setup

Install outlines and required dependencies:

```shell
# Later versions of torch can have difficulty with certain CUDA drivers.
# We recommend using 2.4.0 for now, but you may wish to experiment with
# other versions.
pip install outlines pandas transformers torch==2.4.0 accelerate
```

## Load the model

Choose your language model. We'll use Phi-3 mini, which is small enough to run on reasonably small machines.

```python
import outlines
import torch
import transformers

model_name = 'microsoft/Phi-3-mini-4k-instruct'
tf_model = transformers.AutoModelForCausalLM.from_pretrained(
    model_name, device_map="cuda", torch_dtype=torch.bfloat16
)
tf_tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = outlines.from_transformers(tf_model, tf_tokenizer)
```

## Set up the data

For brevity, we've attached the markdown version of Nvidia's 10k report. The [full demonstration](https://github.com/dottxt-ai/demos/tree/main/earnings-reports) processes the raw HTML version of the report to these markdown tables. Pages are filtered by whether they seem to contain income statements, and then compacted into the string you see below.

```
/* Detailed source-code truncated for AI context efficiency. */
```

The markdown tables extracted from the earnings reports can vary widely in row names, column counts, data types, etc. The advantage of LLMs here is that we can define the data we want in terms of the data types, and the LLM will output the data in the desired format.

For comparison, here is how the income statement looks in the original HTML:



## Define the data we want

Outlines is often used for JSON output, but it can also be used for CSV. We know the columns we want to extract, and we know the data types of the columns. Year for example is always a four-digit number, revenue is a number with commas, and so on.

We can define a regex pattern for each column type:

```python
# Define the column type regex patterns
column_types = {
    # Year is always a four-digit number
    "year": r"\d{4}",

    # Revenue, operating income, and net income are always numbers with commas.
    # This regex permits integers that may begin with a minus sign, and may have
    # commas separating the thousands, millions, etc.
    "integer_comma": r"((-?\d+),?\d+|(-?\d+))",
    # Number is currently not used, but it represents a number with up to two decimal places.
    "number": r"(-?\d+(?:\.\d{1,2})?)",
}
```

Next, let's choose the columns we want to extract. We want

- Year, always a four-digit number
- Revenue, a number with commas
- Operating income, a number with commas
- Net income, a number with commas

```python
# Define the columns to extract, and their data types.
columns_to_extract = {
    "year": "year",
    "revenue": "integer_comma",
    "operating_income": "integer_comma",
    "net_income": "integer_comma",
}
```

You can modify `column_type_regex` to match the data types of the columns you want to extract.  Adding a new financial metric to extract is as simple as adding a new key/value pair to `columns_to_extract`:

```python
columns_to_extract["diluted_earnings_per_share"] = "number"
```

Additional columns are not well tested for accuracy, so use with caution.

## Create the regex describing the data we want


```python
# Create the header line. This is the requested column names
# separated by commas, i.e. "year,revenue,..."
header = ",".join(columns_to_extract.keys())

# Create the data capture patterns. These are the regex patterns
# that will be used to capture the data in each column
data_patterns = [column_types[dtype] for dtype in columns_to_extract.values()]
data_line = ",".join(data_patterns)

# Our final regex pattern.
max_rows = 3 # We expect 3 rows of data, firms usually report 3 years of income statements
csv_regex = f"{header}(\n{data_line}){{,{max_rows}}}\n\n"

print(csv_regex)
```

which gives us

```
year,revenue,operating_income,net_income,basic_earnings_per_share(
\d{4},((-?\d+),?\d+|(-?\d+)),((-?\d+),?\d+|(-?\d+)),((-?\d+),?\d+|(-?\d+)),(-?\d+(?:\.\d{1,2})?)){,3}
```

Pretty hairy, right? Thankfully, we have a simple function to construct this regex for you. The regex defines a header line, followed by a data line that repeats for each row of data we want to extract. Passing the regex to `outlines.Generator` will produce a function that will __always__ produce a CSV string that is consistent with the regex.

## Prompting the model

Outlines does not add system or instruction tokens by default, so we need to use `transformers.AutoTokenizer` to add them for whatever model we're using.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(model_name)

def add_instruction(prompt):
    return tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)

print(add_instruction("Howdy"))
```
```
<|user|>
Howdy<|end|>
<|assistant|>
```

Our prompt roughly describes the task we want the model to perform, and a few pieces of information it may need to know about income statements.

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Running the model

Now that we have our prompt and regular expression, we can run the model.

Construct our regex extractor function.

```python
from outlines.types import Regex

csv_extractor = outlines.Generator(model, Regex(csv_regex))
```

Provide the prompt to the model and run it:

```python
csv_data = csv_extractor(
    extract_financial_data_prompt(columns_to_extract, income_statement),
    max_new_tokens=1024,
)

print(csv_data)
```
```
year,revenue,operating_income,net_income
2024,60922,32972,29760
2023,26974,4224,4368
2022,26914,10041,9752
```

Voila! We've extracted the financial data from the income statement, and it's correct upon inspection.

You can even load this into a `pandas` DataFrame for further analysis:

```python
import pandas as pd
from io import StringIO

df = pd.read_csv(StringIO(csv_data))
print(df)
```
```
   year  revenue  operating_income  net_income
0  2024    60922             32972       29760
1  2023    26974              4224        4368
2  2022    26914             10041        9752
```

---

### Examples Extract Event Details (`docs/examples/extract_event_details.md`)

This recipe demonstrates how to use the `outlines` library to extract structured event details from a text message.
We will extract the title, location, and start date and time from messages like the following:

```plaintext
Hello Kitty, my grandmother will be here, I think it's better to postpone
our appointment to review math lessons to next Monday at 2pm at the same
place, 3 avenue des tanneurs, one hour will be enough see you 😘
```

Let see how to extract the event details from the message with the MLX
library dedicated to Apple Silicon processor (M series).

```python
--8<-- "docs/cookbook/extract_event_details.py"
```

The output will be:

```plaintext
Today: Saturday 16 November 2024 and it's 10:55
```

and the extracted event information will be:

```json
{
  "title":"Math Review",
  "location":"3 avenue des tanneurs",
  "start":"2024-11-22T14:00:00Z"
}
```


To find out more about this use case, we recommend the project developped by [Joseph Rudoler](https://x.com/JRudoler) the [ICS Generator](https://github.com/jrudoler/ics-generator)

---

### Examples Extraction (`docs/examples/extraction.md`)

# Named entity extraction

Named Entity Extraction is a fundamental problem in NLP. It involves identifying and categorizing named entities within a document: people, organization, dates, places, etc. It is usually the first step in a more complex NLP worklow. Here we will use the example of a pizza restaurant that receives orders via their website and need to identify the number and types of pizzas that are being ordered.

Getting LLMs to output the extracted entities in a structured format can be challenging. In this tutorial we will see how we can use Outlines' JSON-structured generation to extract entities from a document and return them in a valid JSON data structure 100% of the time.

As always, we start with initializing the model. We will be using a quantized version of Mistal-7B-v0.1 (we're GPU poor):

```python
import transformers
import outlines

model_name = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda"),
    transformers.AutoTokenizer.from_pretrained(model_name),
)
```

And we will be using the following prompt template:

```python
from outlines import Template

take_order = Template.from_string(
    """You are the owner of a pizza parlor. Customers \
    send you orders from which you need to extract:

    1. The pizza that is ordered
    2. The number of pizzas

    # EXAMPLE

    ORDER: I would like one Margherita pizza
    RESULT: {"pizza": "Margherita", "number": 1}

    # OUTPUT INSTRUCTIONS

    Answer in valid JSON. Here are the different objects relevant for the output:

    Order:
        pizza (str): name of the pizza
        number (int): number of pizzas

    Return a valid JSON of type "Order"

    # OUTPUT

    ORDER: {{ order }}
    RESULT: """
)
```

We now define our data model using Pydantic:

```python
from enum import Enum
from pydantic import BaseModel

class Pizza(str, Enum):
    margherita = "Margherita"
    pepperonni = "Pepperoni"
    calzone = "Calzone"

class Order(BaseModel):
    pizza: Pizza
    number: int
```

We can now define our generator and call it on several incoming orders:

```python
orders = [
    "Hi! I would like to order two pepperonni pizzas and would like them in 30mins.",
    "Is it possible to get 12 margheritas?"
]
prompts = [take_order(order=order) for order in orders]

generator = outlines.Generator(model, Order)

results = generator(prompts)
print(results)
# ['{"pizza": "Pepperoni", "number": 2}',
# '{"pizza": "Margherita", "number": 12}']
```

There are several ways you could improve this example:

- Clients may order several types of pizzas.
- Clients may order drinks as well.
- If the pizza place has a delivery service we need to extract the client's address and phone number
- Clients may specify the time for which they want the pizza. We could then check against a queuing system and reply to them with the estimated delivery time.

How would you change the Pydantic model to account for these use cases?

---

### Examples Knowledge Graph Extraction (`docs/examples/knowledge_graph_extraction.md`)

# Knowledge Graph Extraction

In this guide, we use [outlines](https://dottxt-ai.github.io/outlines/) to extract a knowledge graph from unstructured text.

We will use [llama.cpp](https://github.com/ggerganov/llama.cpp) using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) library. Outlines supports llama-cpp-python, but we need to install it ourselves:

```shell
pip install llama-cpp-python
```

To create an outlines `LlamaCpp` model, you first need to create a `Llama` object from the `llama-cpp-python` library. Then you can create the outlines model by calling `models.from_llamacpp` with the `Llama` object instance as argument. To create the `Llama` object, you need to provide the model weights by passing the name of the repository on the HuggingFace Hub, and the filenames or glob pattern (it will automatically download the weights from the hub):

```python
import llama_cpp
import outlines

llm = llama_cpp.Llama(
    "NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF",
    tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
        "NousResearch/Hermes-2-Pro-Llama-3-8B"
    ),
    n_gpu_layers=-1,
    flash_attn=True,
    n_ctx=8192,
    verbose=False
)
model = outlines.from_llamacpp(llm)
```

??? note "(Optional) Store the model weights in a custom folder"

    By default the model weights are downloaded to the hub cache but if we want so store the weights in a custom folder, we pull a quantized GGUF model [Hermes-2-Pro-Llama-3-8B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF) by [NousResearch](https://nousresearch.com/) from [HuggingFace](https://huggingface.co/):

    ```shell
    wget https://hf.co/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF/resolve/main/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf
    ```

    We initialize the model:

    ```python
    from llama_cpp import Llama

    llm = Llama("/path/to/model/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf", ...)
    ```

## Knowledge Graph Extraction

We first need to define our Pydantic class for each node and each edge of the knowledge graph:

```python
from pydantic import BaseModel, Field

class Node(BaseModel):
    """Node of the Knowledge Graph"""

    id: int = Field(..., description="Unique identifier of the node")
    label: str = Field(..., description="Label of the node")
    property: str = Field(..., description="Property of the node")


class Edge(BaseModel):
    """Edge of the Knowledge Graph"""

    source: int = Field(..., description="Unique source of the edge")
    target: int = Field(..., description="Unique target of the edge")
    label: str = Field(..., description="Label of the edge")
    property: str = Field(..., description="Property of the edge")
```

We then define the Pydantic class for the knowledge graph and get its JSON schema:

```python
from typing import List

class KnowledgeGraph(BaseModel):
    """Generated Knowledge Graph"""

    nodes: List[Node] = Field(..., description="List of nodes of the knowledge graph")
    edges: List[Edge] = Field(..., description="List of edges of the knowledge graph")

schema = KnowledgeGraph.model_json_schema()
```

We then need to adapt our prompt to the [Hermes prompt format for JSON schema](https://github.com/NousResearch/Hermes-Function-Calling?tab=readme-ov-file#prompt-format-for-json-mode--structured-outputs):

```python
from outlines import Template

generate_hermes_prompt = Template.from_string(
    """
    <|im_start|>system
    You are a world class AI model who answers questions in JSON
    Here's the json schema you must adhere to:
    <schema>
    {{ schema }}
    </schema>
    <|im_end|>
    <|im_start|>user
    {{ user_prompt }}
    <|im_end|>
    <|im_start|>assistant
    <schema>
    """
)
```

For a given user prompt, for example:

```python
user_prompt = "Alice loves Bob and she hates Charlie."
```

We can use `outlines.Generator` by passing the Pydantic class we previously defined, and call the generator with the Hermes prompt:

```python
from outlines import Generator

generator = Generator(model, KnowledgeGraph)
prompt = generate_hermes_prompt(schema=schema, user_prompt=user_prompt)
response = generator(prompt, max_tokens=1024, temperature=0, seed=42)
```

We obtain the nodes and edges of the knowledge graph:

```python
print(response)
# {"nodes":[{"id":1,"label":"Alice","property":"loves,hates"},
# {"id":2,"label":"Bob","property":"loved_by"},
# {"id":3,"label":"Charlie","property":"hated_by"}],
# "edges":[{"source":1,"target":2,"label":"loves","property":"love"},
# {"source":1,"target":3,"label":"hates","property":"hate"}]}

```

## (Optional) Visualizing the Knowledge Graph

We can use the [Graphviz library](https://graphviz.readthedocs.io/en/stable/) to visualize the generated knowledge graph. For detailed installation instructions, see [here](https://graphviz.readthedocs.io/en/stable/#installation).

```python
from graphviz import Digraph

dot = Digraph()
for node in response["nodes"]:
    dot.node(str(node["id"]), node["label"], shape='circle', width='1', height='1')
for edge in response["edges"]:
    dot.edge(str(edge["source"]), str(edge["target"]), label=edge["label"])

dot.render('knowledge-graph.gv', view=True)
```



This example was originally contributed by [Alonso Silva](https://github.com/alonsosilvaallende).

---

### Examples Models Playing Chess (`docs/examples/models_playing_chess.md`)

# Large language models playing chess

In this example we will make a Phi-3 model play chess against itself. On its own the model easily generates invalid moves, so we will give it a little help. At each step we will generate a regex that only matches valid move, and use it to help the model only generating valid moves.

## The chessboard

The game will be played on a standard checkboard. We will use the `chess` [library](https://github.com/niklasf/python-chess) to track the opponents' moves, and check that the moves are valid.

```python
%pip install outlines -q
%pip install chess -q
%pip install transformers accelerate einops -q

import chess

board = chess.Board("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1")
```

## The opponents

Phi-3 will be playing against itself:

```python
import transformers
import outlines

model_name = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(model_name),
    transformers.AutoTokenizer.from_pretrained(model_name),
)
```

## A little help for the language model

To make sure Phi-3 generates valid chess moves we will use Outline's regex-structured generation. We define a function that takes the current state of the board and returns a regex that matches all possible legal moves:

```python
import re
from outlines.types.dsl import either, String

def legal_moves_regex(board):
    """Build a regex that only matches valid moves."""
    legal_moves = list(board.legal_moves)
    legal_modes_str = [board.san(move) for move in legal_moves]
    legal_modes_str = [re.sub(r"[+#]", "", move) for move in legal_modes_str]
    regex_pattern = either(*[String(move) for move in legal_modes_str])
    return regex_pattern
```

## Prompting the language model

The prompt corresponds to the current state of the board, so we start with:

```python
prompt = "Let's play Chess. Moves: "

```

We update the prompt at each step so it reflects the state of the board after the previous move.

## Let's play

```python
board_state = " "
turn_number = 0
while not board.is_game_over():
    regex_pattern = legal_moves_regex(board)
    structured = model(prompt + board_state, regex_pattern)
    move = board.parse_san(structured)

    if turn_number % 2 == 0 :  # It's White's turn
        board_state += board.san(move) + " "
    else:
        board_state += board.san(move) + " " + str(turn_number) + "."

    turn_number += 1

    board.push(move)

    print(board_state)
```

Interestingly enough, Phi-3 hates capturing.

```pgn
 e4 e5 1.Nf3 Ne7 3.b4 Nf5 5.Nc3 Ne7 7.Bb5 a6 9.Na4 b6 11.c3 Nec6 13.c4 a5 15.d4 Qg5 17.Nd2 Bb7 19.dxe5
```

*This example was originally authored by [@903124S](https://x.com/903124S) in [this gist](https://gist.github.com/903124/cfbefa24da95e2316e0d5e8ef8ed360d).*

---

### Examples Qa With Citations (`docs/examples/qa-with-citations.md`)

# Generate Synthetic Data and Q&A with Citations

This tutorial is adapted from the [instructor-ollama notebook](https://github.com/alonsosilvaallende/Hermes-Function-Calling/blob/main/examples/instructor_ollama.ipynb). We start with a simple example to generate synthetic data and then we approach the problem of question answering by providing citations.

We will use [llama.cpp](https://github.com/ggerganov/llama.cpp) using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) library. Outlines supports llama-cpp-python, but we need to install it ourselves:

```shell
pip install llama-cpp-python
```

We download the model weights by passing the name of the repository on the HuggingFace Hub, and the filenames (or glob pattern):
```python
import llama_cpp
import outlines

llm = llama_cpp.Llama(
    "NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF",
    tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
        "NousResearch/Hermes-2-Pro-Llama-3-8B"
    ),
    n_gpu_layers=-1,
    flash_attn=True,
    n_ctx=8192,
    verbose=False
)
model = outlines.from_llamacpp(llm)
```

??? note "(Optional) Store the model weights in a custom folder"

    By default the model weights are downloaded to the hub cache but if we want so store the weights in a custom folder, we pull a quantized GGUF model [Hermes-2-Pro-Llama-3-8B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF) by [NousResearch](https://nousresearch.com/) from [HuggingFace](https://huggingface.co/):

    ```shell
    wget https://hf.co/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF/resolve/main/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf
    ```

    We initialize the model:

    ```python
    from llama_cpp import Llama

    llm = Llama("/path/to/model/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf", ...)
    ```

## Generate Synthetic Data

We first need to define our Pydantic class for a user:

```python
from pydantic import BaseModel, Field

class UserDetail(BaseModel):
    id: int = Field(..., description="Unique identifier") # so the model keeps track of the number of users
    first_name: str
    last_name: str
    age: int
```

We then define a Pydantic class for a list of users:

```python
from typing import List

class Users(BaseModel):
    users: List[UserDetail]
```

We can use a `outlines.Generator` by passing this Pydantic class we just defined, and call the generator:

```python
import json

generator = outlines.Generator(model, Users)
response = generator("Create 5 fake users", max_tokens=1024, temperature=0, seed=42)
response = json.loads(response)
print(response['users'])
# [{'id': 1, 'first_name': 'John', 'last_name': 'Doe', 'age': 25},
# {'id': 2, 'first_name': 'Jane', 'last_name': 'Doe', 'age': 30},
# {'id': 3, 'first_name': 'Bob', 'last_name': 'Smith', 'age': 40},
# {'id': 4, 'first_name': 'Alice', 'last_name': 'Smith', 'age': 35},
# {'id': 5, 'first_name': 'John', 'last_name': 'Smith', 'age': 20}]
```

```python
for user in response['users']:
    print(user['first_name'])
    print(user['last_name'])
    print(user['age'])
    print("#####")
# John
# Doe
# 25
# #####
# Jane
# Doe
# 30
# #####
# Bob
# Smith
# 40
# #####
# Alice
# Smith
# 35
# #####
# John
# Smith
# 20
# #####
```

## QA with Citations

We first need to define our Pydantic class for QA with citations:

```python
from typing import List
from pydantic import BaseModel

class QuestionAnswer(BaseModel):
    question: str
    answer: str
    citations: List[str]

schema = QuestionAnswer.model_json_schema()
```

We then need to adapt our prompt to the [Hermes prompt format for JSON schema](https://github.com/NousResearch/Hermes-Function-Calling?tab=readme-ov-file#prompt-format-for-json-mode--structured-outputs):

```python
from outlines import Template

hermes_prompt = Template.from_string(
    """
    <|im_start|>system
    You are a world class AI model who answers questions in JSON with correct and exact citations
    extracted from the `Context`.
    Here's the json schema you must adhere to:
    <schema>
    {{ schema }}
    </schema>
    <|im_end|>
    <|im_start|>user
    `Context`:
    {{ context }}
    `Question`:
    {{ question }}
    <|im_end|>
    <|im_start|>assistant
    """
)
```

We can use `outlines.Generator` by passing the Pydantic class we previously defined, and call the generator with Hermes prompt:

```python
question = "What did the author do during college?"
context = """
My name is Jason Liu, and I grew up in Toronto Canada but I was born in China.
I went to an arts high school but in university I studied Computational Mathematics and physics.
As part of coop I worked at many companies including Stitchfix, Facebook.
I also started the Data Science club at the University of Waterloo and I was the president of the club for 2 years.
"""
generator = outlines.Generator(model, QuestionAnswer)
prompt = hermes_prompt(question=question, context=context, schema=schema)
response = generator(prompt, max_tokens=1024, temperature=0, seed=42)
print(response)
# {"question": "What did the author do during college?", "answer": "The author studied Computational Mathematics and physics in university and was also involved in starting the Data Science club, serving as its president for 2 years.", "citations": ["I went to an arts high school but in university I studied Computational Mathematics and physics.", "I also started the Data Science club at the University of Waterloo and I was the president of the club for 2 years."]}
```

We can do the same for a list of question-context pairs:

```
/* Detailed source-code truncated for AI context efficiency. */
```

This example was originally contributed by [Alonso Silva](https://github.com/alonsosilvaallende).

---

### Examples React Agent (`docs/examples/react_agent.md`)

# ReAct Agent

This example shows how to use [outlines](https://dottxt-ai.github.io/outlines/) to build your own agent with open weights local models and structured outputs. It is inspired by the blog post [A simple Python implementation of the ReAct pattern for LLMs](https://til.simonwillison.net/llms/python-react-pattern) by [Simon Willison](https://simonwillison.net/).

The ReAct pattern (for Reason+Act) is described in the paper [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629). It's a pattern where you implement additional actions that an LLM can take - searching Wikipedia or running calculations for example - and then teach it how to request the execution of those actions, and then feed their results back into the LLM.

Additionally, we give the LLM the possibility of using a scratchpad described in the paper [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://arxiv.org/abs/2112.00114) which improves the ability of LLMs to perform multi-step computations.

We use [llama.cpp](https://github.com/ggerganov/llama.cpp) using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) library. Outlines supports llama-cpp-python, but we need to install it ourselves:

```shell
pip install llama-cpp-python
```

We download the model weights by passing the name of the repository on the HuggingFace Hub, and the filenames (or glob pattern):
```python
import llama_cpp
import outlines

llm = llama_cpp.Llama(
    "NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF",
    tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
        "NousResearch/Hermes-2-Pro-Llama-3-8B"
    ),
    n_gpu_layers=-1,
    flash_attn=True,
    n_ctx=8192,
    verbose=False
)
model = outlines.from_llamacpp(llm)
```

??? note "(Optional) Store the model weights in a custom folder"

    By default the model weights are downloaded to the hub cache but if we want so store the weights in a custom folder, we pull a quantized GGUF model [Hermes-2-Pro-Llama-3-8B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF) by [NousResearch](https://nousresearch.com/) from [HuggingFace](https://huggingface.co/):

    ```shell
    wget https://hf.co/NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF/resolve/main/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf
    ```

    We initialize the model:

    ```python
    from llama_cpp import Llama

    llm = Llama("/path/to/model/Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf", ...)
    ```

## Build a ReAct agent

In this example, we use two tools:

- wikipedia: \<search term\> - search Wikipedia and returns the snippet of the first result
- calculate: \<expression\> - evaluate an expression using Python's eval() function

```python
import httpx

def wikipedia(q):
    return httpx.get("https://en.wikipedia.org/w/api.php", params={
        "action": "query",
        "list": "search",
        "srsearch": q,
        "format": "json"
    }).json()["query"]["search"][0]["snippet"]


def calculate(numexp):
    return eval(numexp)
```

We define the logic of the agent through a Pydantic class. First, we want the LLM to decide only between the two previously defined tools:

```python
from enum import Enum

class Action(str, Enum):
    wikipedia = "wikipedia"
    calculate = "calculate"
```

Our agent will loop through Thought and Action. We explicitly give the Action Input field so it doesn't forget to add the arguments of the Action. We also add a scratchpad (optional).

```python
from pydantic import BaseModel, Field

class Reason_and_Act(BaseModel):
    Scratchpad: str = Field(..., description="Information from the Observation useful to answer the question")
    Thought: str = Field(..., description="It describes your thoughts about the question you have been asked")
    Action: Action
    Action_Input: str = Field(..., description="The arguments of the Action.")
```

Our agent will reach a Final Answer. We also add a scratchpad (optional).

```python
class Final_Answer(BaseModel):
    Scratchpad: str = Field(..., description="Information from the Observation useful to answer the question")
    Final_Answer: str = Field(..., description="Answer to the question grounded on the Observation")
```

Our agent will decide when it has reached a Final Answer and therefore to stop the loop of Thought and Action.

```python
from typing import Union

class Decision(BaseModel):
    Decision: Union[Reason_and_Act, Final_Answer]

json_schema = Decision.model_json_schema()
```

We then need to adapt our prompt to the [Hermes prompt format for JSON schema](https://github.com/NousResearch/Hermes-Function-Calling?tab=readme-ov-file#prompt-format-for-json-mode--structured-outputs) and explain the agent logic. We can load a template from a file for that:

```python
from outlines import Template

hermes_prompt = Template.from_file("prompt_templates/react_agent.txt")
```

We define a ChatBot class

```python
class ChatBot:
    def __init__(self, prompt=""):
        self.prompt = prompt

    def __call__(self, user_prompt):
        self.prompt += user_prompt
        result = self.execute()
        return result

    def execute(self):
        generator = outlines.Generator(model, Decision)
        result = generator(self.prompt, max_tokens=1024, temperature=0, seed=42)
        return result
```

We define a query function:

```
/* Detailed source-code truncated for AI context efficiency. */
```

We can now test our ReAct agent:

```python
print(query("What's 2 to the power of 10?"))
# Scratchpad:
# Thought: I need to perform a mathematical calculation to find the result of 2 to the power of 10.
#  -- running calculate: 2**10
#
# Observation: 1024
#
# Scratchpad: 2 to the power of 10 is 1024.
# Final Answer: 2 to the power of 10 is 1024.
# 2 to the power of 10 is 1024.
```

```python
print(query("What does England share borders with?"))
# Scratchpad:
# Thought: To answer this question, I will use the 'wikipedia' action to gather information about England's geographical location and its borders.
#  -- running wikipedia: England borders
#
# Observation: Anglo-Scottish <span class="searchmatch">border</span> (Scottish Gaelic: Crìochan Anglo-Albannach) is an internal <span class="searchmatch">border</span> of the United Kingdom separating Scotland and <span class="searchmatch">England</span> which runs for
#
# Scratchpad: Anglo-Scottish border (Scottish Gaelic: Crìochan Anglo-Albannach) is an internal border of the United Kingdom separating Scotland and England which runs for
# Final Answer: England shares a border with Scotland.
# England shares a border with Scotland.
```

As mentioned in Simon's blog post, this is not a very robust implementation at all and there's a ton of room for improvement. But it is lovely how simple it is with a few lines of Python to make these extra capabilities available to the LLM. And now you can run it locally with an open weights LLM.

This example was originally contributed by [Alonso Silva](https://github.com/alonsosilvaallende).

---

### Examples Read Pdfs (`docs/examples/read-pdfs.md`)

# PDF to structured output with vision language models

A common task with language models is to ask language models questions about a PDF file.

Typically, the output is unstructured text, i.e. "talking" to your PDF.

In some cases, you may wish to extract structured information from the PDF, like tables, lists, citations, etc.

PDFs are difficult to machine read. However, you can simply convert the PDF to images, and then use a vision language model to extract structured information from the images.

This cookbook demonstrates how to

1. Convert a PDF to a list of images
2. Use a vision language model to extract structured information from the images

## Dependencies

You'll need to install these dependencies:

```shell
pip install outlines pillow transformers torch==2.4.0 pdf2image

# Optional, but makes the output look nicer
pip install rich
```

## Import the necessary libraries

```python
from PIL import Image
import outlines
import torch
from transformers import AutoProcessor
from pydantic import BaseModel
from typing import List, Optional
from pdf2image import convert_from_path
import os
from rich import print
import requests
```

## Choose a model

We've tested this example with [Pixtral 12b](https://huggingface.co/mistral-community/pixtral-12b) and [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct).

To use Pixtral:

```python
from transformers import LlavaForConditionalGeneration, LlavaProcessor
model_name="mistral-community/pixtral-12b"
model_class=LlavaForConditionalGeneration
processor_class = LlavaProcessor
```

To use Qwen-2-VL:

```python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
model_name = "Qwen/Qwen2-VL-7B-Instruct"
model_class = Qwen2VLForConditionalGeneration
processor_class = AutoProcessor
```

You can load your model into memory with:

```python
# This loads the model into memory. On your first run,
# it will have to download the model, which might take a while.
model_kwargs={"device_map": "auto", "torch_dtype": torch.bfloat16}
processor_kwargs={"device_map": "cpu"}
tf_model = model_class.from_pretrained(model_name, **model_kwargs)
tf_processor = processor_class.from_pretrained(model_name, **processor_kwargs)

model = outlines.from_transformers(tf_model, tf_processor)
```

## Convert the PDF to images

We'll use the `pdf2image` library to convert each page of the PDF to an image.

`convert_pdf_to_images` is a convenience function that converts each page of the PDF to an image, and optionally saves the images to disk when `output_dir` is provided.

Note: the `dpi` argument is important. It controls the resolution of the images. High DPI images are higher quality and may yield better results,
but they are also larger, slower to process, and require more memory.

```python
from pdf2image import convert_from_path
from PIL import Image
import os
from typing import List, Optional

def convert_pdf_to_images(
    pdf_path: str,
    output_dir: Optional[str] = None,
    dpi: int = 120,
    fmt: str = 'PNG'
) -> List[Image.Image]:
    """
    Convert a PDF file to a list of PIL Image objects.

    Args:
        pdf_path: Path to the PDF file
        output_dir: Optional directory to save the images
        dpi: Resolution for the conversion. High DPI is high quality, but also slow and memory intensive.
        fmt: Output format (PNG recommended for quality)

    Returns:
        List of PIL Image objects
    """
    # Convert PDF to list of images
    images = convert_from_path(
        pdf_path,
        dpi=dpi,
        fmt=fmt
    )

    # Optionally save images
    if output_dir:
        os.makedirs(output_dir, exist_ok=True)
        for i, image in enumerate(images):
            image.save(os.path.join(output_dir, f'page_{i+1}.{fmt.lower()}'))

    return images
```

We're going to use the [Louf & Willard paper](https://arxiv.org/pdf/2307.09702) that described the method that Outlines uses for structured generation.

To download the PDF, run:

```python
# Download the PDF file
pdf_url = "https://arxiv.org/pdf/2307.09702"
response = requests.get(pdf_url)

# Save the PDF locally
with open("louf-willard.pdf", "wb") as f:
    f.write(response.content)
```

Now, we can convert the PDF to a list of images:

```python
# Load the pdf
images = convert_pdf_to_images(
    "louf-willard.pdf",
    dpi=120,
    output_dir="output_images"
)
```

## Extract structured information from the images

The structured output you can extract is exactly the same as everywhere else in Outlines -- you can use regular expressions, JSON schemas, selecting from a list of options, etc.

### Extracting data into JSON

Suppose you wished to go through each page of the PDF, and extract the page description, key takeaways, and page number.

You can do this by defining a JSON schema, and then using `outlines.Generator` to extract the data.

First, define the structure you want to extract:

```python
class PageSummary(BaseModel):
    description: str
    key_takeaways: List[str]
    page_number: int
```

Second, we need to set up the prompt. Adding special tokens can be tricky, so we use the transformers processor to apply the special tokens for us. To do so, we specify a list of messages, where each message is a dictionary with a `role` and `content` key.

Images are denoted with `type: "image"`, and text is denoted with `type: "text"`.

```python
messages = [
    {
        "role": "user",
        "content": [
            # The text you're passing to the model --
            # this is where you do your standard prompting.
            {"type": "text", "text": f"""
                Describe the page in a way that is easy for a PhD student to understand.

                Return the information in the following JSON schema:
                {PageSummary.model_json_schema()}

                Here is the page:
                """
            },

            # This a placeholder, the actual image is passed in when
            # we call the generator function down below.
            {"type": "image", "image": ""},
        ],
    }
]

# Convert the messages to the final prompt
prompt = tf_processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
```

Now we iterate through each image, and extract the structured information:

```python
# Page summarizer function
page_summary_generator = outlines.Generator(model, PageSummary)

for image in images:
    result = page_summary_generator({"text": prompt, "images": image})
    print(result)
```

### Regular expressions to extract the arxiv paper identifier

The [arXiv paper identifier](https://info.arxiv.org/help/arxiv_identifier.html) is a unique identifier for each paper. These identifiers have the format `arXiv:YYMM.NNNNN` (five end digits) or `arXiv:YYMM.NNNN` (four end digits). arXiv identifiers are typically watermarked on papers uploaded to arXiv.

arXiv identifiers are optionally followed by a version number, i.e. `arXiv:YYMM.NNNNNvX`.

We can use a regular expression to define this patter:

```python
from outlines.types import Regex

paper_regex = Regex(r'arXiv:\d{2}[01]\d\.\d{4,5}(v\d)?')
```

We can build an extractor function from the regex:

```python
id_extractor = outlines.Generator(model, paper_regex)
```

Now, we can extract the arxiv paper identifier from the first image:

```python
arxiv_instruction = tf_processor.apply_chat_template(
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": f"""
                Extract the arxiv paper identifier from the page.

                Here is the page:
                """},
                {"type": "image", "image": ""},
            ],
        }
    ],
    tokenize=False,
    add_generation_prompt=True
)

# Extract the arxiv paper identifier
paper_id = id_extractor({"text": arxiv_instruction, "images": images[0]})
```

As of the time of this writing, the arxiv paper identifier is

```
arXiv:2307.09702v4
```

Your version number may be different, but the part before `vX` should match.

### Categorize the paper into one of several categories

`outlines.Generator` also allows the model to select one of several options by providing a Literal type hint with the categories.

Suppose we wanted to categorize the paper into being about "language models", "cell biology", or "other". We would then define the output type as `Literal["llms", "cell biology", "other"]`.

Let's define a few categories we might be interested in:

```python
categories = [
    "llms",
    "cell biology",
    "other"
]
```

Now we can construct the prompt:

```python
categorization_instruction = tf_processor.apply_chat_template(
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": f"""
                Please choose one of the following categories
                that best describes the paper.

                {categories}

                Here is the paper:
                """},

                {"type": "image", "image": ""},
            ],
        }
    ],
    tokenize=False,
    add_generation_prompt=True
)
```

Now we can show the model the first page and extract the category:

```python
from typing import Literal

# Build the choice extractor
categorizer = outlines.Generator(model, Literal["llms", "cell biology", "other"])

# Categorize the paper
category = categorizer({"text": categorization_instruction, "images": images[0]})
print(category)
```

Which should return:

```
llms
```

## Additional notes

You can provide multiple images to the model by

1. Adding additional image messages
2. Providing a list of images to the generator

For example, to have two images, you can do:

```python
two_image_prompt = tf_processor.apply_chat_template(
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "are both of these images of hot dogs?"},

                # Tell the model there are two images
                {"type": "image", "image": ""},
                {"type": "image", "image": ""},
            ],
        }
    ],
    tokenize=False,
    add_generation_prompt=True
)

# Pass two images to the model
generator = outlines.Generator(model, Literal["hot dog", "not hot dog"])

result = generator({"text": two_image_prompt, "images": [images[0], images[1]]})
print(result)
```

Using the first to pages of the paper (they are not images of hot dogs), we should get

```
not hot dog
```

---

### Examples Receipt Digitization (`docs/examples/receipt-digitization.md`)

# Receipt Data Extraction with VLMs

## Setup

You'll need to install the dependencies:

```shell
pip install outlines torch==2.4.0 transformers accelerate pillow rich
```

## Import libraries

Load all the necessary libraries:

```python
# LLM stuff
import outlines
import torch
from transformers import AutoProcessor
from pydantic import BaseModel, Field
from typing import Literal, Optional, List

# Image stuff
from PIL import Image
import requests

# Rich for pretty printing
from rich import print
```

## Choose a model

This example has been tested with `mistral-community/pixtral-12b` ([HF link](https://huggingface.co/mistral-community/pixtral-12b)) and `Qwen/Qwen2-VL-7B-Instruct` ([HF link](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)).

We recommend Qwen-2-VL as we have found it to be more accurate than Pixtral.

If you want to use Qwen-2-VL, you can do the following:

```python
# To use Qwen-2-VL:
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
model_name = "Qwen/Qwen2-VL-7B-Instruct"
model_class = Qwen2VLForConditionalGeneration
processor_class = AutoProcessor
```

If you want to use Pixtral, you can do the following:

```python
# To use Pixtral:
from transformers import LlavaForConditionalGeneration, LlavaProcessor
model_name="mistral-community/pixtral-12b"
model_class=LlavaForConditionalGeneration
processor_class = LlavaProcessor
```

## Load the model

Load the model into memory:

```python
model_kwargs={"device_map": "auto", "torch_dtype": torch.bfloat16}
processor_kwargs={"device_map": "cuda"}
tf_model = model_class.from_pretrained(model_name, **model_kwargs)
tf_processor = processor_class.from_pretrained(model_name, **processor_kwargs)

model = outlines.from_transformers(tf_model, tf_processor)
```

## Image processing

Images can be quite large. In GPU-poor environments, you may need to resize the image to a smaller size.

Here's a helper function to do that:

```python
def load_and_resize_image(image_path, max_size=1024):
    """
    Load and resize an image while maintaining aspect ratio

    Args:
        image_path: Path to the image file
        max_size: Maximum dimension (width or height) of the output image

    Returns:
        PIL Image: Resized image
    """
    image = Image.open(image_path)

    # Get current dimensions
    width, height = image.size

    # Calculate scaling factor
    scale = min(max_size / width, max_size / height)

    # Only resize if image is larger than max_size
    if scale < 1:
        new_width = int(width * scale)
        new_height = int(height * scale)
        image = image.resize((new_width, new_height), Image.Resampling.LANCZOS)

    return image
```

You can change the resolution of the image by changing the `max_size` argument. Small max sizes will make the image more blurry, but processing will be faster and require less memory.

## Load an image

Load an image and resize it. We've provided a sample image of a Trader Joe's receipt, but you can use any image you'd like.

Here's what the image looks like:



```python
# Path to the image
image_path = "https://raw.githubusercontent.com/dottxt-ai/outlines/refs/heads/main/docs/cookbook/images/trader-joes-receipt.jpg"

# Download the image
response = requests.get(image_path)
with open("receipt.png", "wb") as f:
    f.write(response.content)

# Load + resize the image
image = load_and_resize_image("receipt.png")
```

## Define the output structure

We'll define a Pydantic model to describe the data we want to extract from the image.

In our case, we want to extract the following information:

- The store name
- The store address
- The store number
- A list of items, including the name, quantity, price per unit, and total price
- The tax
- The total
- The date
- The payment method

Most fields are optional, as not all receipts contain all information.

```python
class Item(BaseModel):
    name: str
    quantity: Optional[int]
    price_per_unit: Optional[float]
    total_price: Optional[float]

class ReceiptSummary(BaseModel):
    store_name: str
    store_address: str
    store_number: Optional[int]
    items: List[Item]
    tax: Optional[float]
    total: Optional[float]
    # Date is in the format YYYY-MM-DD. We can apply a regex pattern to ensure it's formatted correctly.
    date: Optional[str] = Field(pattern=r'\d{4}-\d{2}-\d{2}', description="Date in the format YYYY-MM-DD")
    payment_method: Literal["cash", "credit", "debit", "check", "other"]
```

## Prepare the prompt

We'll use the `tf_processor` to convert the image and the text prompt into a format that the model can understand. Practically,
this is the code that adds user, system, assistant, and image tokens to the prompt.

```python
# Set up the content you want to send to the model
messages = [
    {
        "role": "user",
        "content": [
            {
                # The image is provided as a PIL Image object
                "type": "image",
                "image": image,
            },
            {
                "type": "text",
                "text": f"""You are an expert at extracting information from receipts.
                Please extract the information from the receipt. Be as detailed as possible --
                missing or misreporting information is a crime.

                Return the information in the following JSON schema:
                {ReceiptSummary.model_json_schema()}
            """},
        ],
    }
]

# Convert the messages to the final prompt
prompt = tf_processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
```

If you are curious, the final prompt that is sent to the model looks (roughly) like this:

```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>
You are an expert at extracting information from receipts.
Please extract the information from the receipt. Be as detailed as
possible -- missing or misreporting information is a crime.

Return the information in the following JSON schema:

<JSON SCHEMA OMITTED>
<|im_end|>
<|im_start|>assistant
```

## Run the model

```python
# Prepare a function to process receipts
receipt_summary_generator = outlines.Generator(model, ReceiptSummary)

# Generate the receipt summary
result = receipt_summary_generator(
    {"text": prompt, "images": image},
    max_new_tokens=1024
)
print(result)
```

## Output

The output should look like this:

```
/* Detailed source-code truncated for AI context efficiency. */
```

Voila! You've successfully extracted information from a receipt using an LLM.

## Bonus: roasting the user for their receipt

You can roast the user for their receipt by adding a `roast` field to the end of the  `ReceiptSummary` model.

```python
class ReceiptSummary(BaseModel):
    ...
    roast: str
```

which gives you a result like

```
{
    ...
    "roast": "You must be a fan of Trader Joe's because you bought enough
    items to fill a small grocery bag and still had to pay for a bag fee.
    Maybe you should start using reusable bags to save some money and the
    environment."
}
```

Qwen is not particularly funny, but worth a shot.

---

### Examples Simtom (`docs/examples/simtom.md`)

# Build perspective-taking agents with SimToM

Prompting strategies like Chain-of-Thought (CoT) can improve LLMs' reasoning capabilities. However, they underwhelm in tasks that require keeping track of inconsistent world states. [SimToM](https://arxiv.org/abs/2311.10227) proposes a simple, two-stage prompting framework for LLMs inspired by Simulation Theory. The authors showed that this approach outperforms zero-shot prompting and CoT on ToMI and BigToM, two benchmarks with Theory of Mind questions.

In this example, we will implement SimToM with a few lines of code using Outlines' prompt templating and structured generation capabilities.

## How SimToM works

SimToM calls an LLM with two consecutive prompts:

1. **Perspective-taking**: The first prompt receives a `story` and a `character`. The goal is to understand the situation based on the character's point of view and filter out the rest of the story.
2. **Question-Answering**: The second prompt receives the character's point of view from the previous step and tasks the LLM to answer a question using that context.



## Outlines implementation

To implement SimToM with Outlines, we will need to:

1. Write the prompts with [prompt templates](https://dottxt-ai.github.io/outlines/latest/reference/prompting/).
2. Define the JSON object each prompt will return using Pydantic.
3. Generate responses with a Mistral model using the [transformers integration](https://dottxt-ai.github.io/outlines/latest/reference/models/transformers/).

Let's dive into it!

### Using Prompt Templates

The authors have shared their code, prompts and data in [this GitHub repository](https://github.com/shawnsihyunlee/simulatedtom). Below, we define in Outlines the prompts they used for the ToMI dataset:

```python
from outlines import Template

perspective_taking = Template.from_file("prompt_templates/simtom_prospective_taking.txt")
simulation = Template.from_file("prompt_templates/simtom_simulation.txt")
```

### JSON Structured Generation

Outlines guarantees that the LLM will return a valid JSON object, which we can specify as a Pydantic model.

We will need two Pydantic models for SimToM, one for each prompt:

```python
from pydantic import BaseModel, Field
from typing import List

class PerspectiveTaking(BaseModel):
    """This is for the first prompt."""
    character: str = Field(description="The character we extract the events for.")
    events: List[str] = Field(description="All events that the character knows about.")

class Simulation(BaseModel):
    """This is for the second prompt."""
    answer: str
```

### Calling an LLM

Let's try SimToM with an example from the ToMI dataset:

```python
story = """
1 Aria entered the front_yard.
2 Aiden entered the front_yard.
3 The grapefruit is in the green_bucket.
4 Aria moved the grapefruit to the blue_container.
5 Aiden exited the front_yard.
6 Noah entered the playroom.
"""
question = "7 Where was the grapefruit at the beginning?"
character = "Aria"
```

We load `Mistral-7B-Instruct-v0.3`, create the prompt using the template we defined earlier, and generate a structured response. As a reminder, the goal of the first call is to get all the events a character, `Aria`, knows about.

```python
import transformers
import outlines
# Load an LLM from Hugging Face
MODEL_NAME = "mistral-community/Mistral-7B-Instruct-v0.3"
model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(MODEL_NAME),
    transformers.AutoTokenizer.from_pretrained(MODEL_NAME),
)

perspective_prompt = perspective_taking(story=story, character=character)

# Call Mistral 7B with the first prompt
generator = outlines.Generator(model, PerspectiveTaking)
perspective = generator(perspective_prompt, max_new_tokens=1024)

print(perspective)
# {'character': 'Aria', 'events': ['1 Aria entered the front_yard.', '3 The grapefruit is in the green_bucket.', '4 Aria moved the grapefruit to the blue_container.']}
```

Not bad! We will now generate the second prompt with those events.

```python
import json

sim_prompt = simulation(events=json.loads(perspective)["events"], name=character, question=question)

# Call Mistral 7B with the second prompt
generator = outlines.Generator(model, Simulation)
result = generator(sim_prompt, max_new_tokens=1024)

print(result)
# {'answer': 'green_bucket'}
```

And this is it! SimToM could be useful in agentic workflows, where agents must act based on what they know, not all available information. One caveat of SimToM is that the perspective-taking step may remove important information, leading to wrong results. As the authors note in their paper, it can feature as a simple and effective baseline for evaluating LLMs on Theory of Mind reasoning tasks.

---

### Examples Structured Generation Workflow (`docs/examples/structured_generation_workflow.md`)

# Structured Generation Workflow: Generating Synthetic Phone Numbers

This is a condensed version of [Coding for Structured Generation with LLMs](https://blog.dottxt.co/coding-for-structured-generation.html).

For this example we're going to be building an LLM program to generate **synthetic data** in the form of realistic looking phone numbers for Washington State. Using an LLM for this task *is a bit overkill* since we could just as easily accomplish this with a tool like [Faker](https://fakerjs.dev/), but this example still serves as a useful way to demonstrate a workflow for using structured generation.

## Unstructured approach

Before diving into how to use structure generation for this task let's start with an unstructured example. We begin by loading our model:

```python
import outlines
import transformers

model_name = 'microsoft/Phi-3-mini-4k-instruct'
model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(model_name),
    transformers.AutoTokenizer.from_pretrained(model_name)
)
```

Next we need a prompt for this model. Since we're focusing on structured generation, we won't be engaging in any form of "prompt hacking" and will be leaving this prompt untouched for the rest of this example.

```python
prompt_phone = """
    Please generate a realistic phone number for Washington State in the following format
    (555) 555-5555
"""
```

With our prompt ready we can now generate 10 example phone numbers

```python
phone_generator_unstruct = outlines.Generator(model)
for _ in range(3):
    print(phone_generator_unstruct(prompt_phone, max_new_tokens=12))
```

> I'd be happy to help you generate a realistic phone\
I cannot generate a real phone number as I'm just\
I'm an AI and don't have the ability\
Sure! Here is a randomly generated phone number in the format\
Here's a phone number that fits the format for a\
In Washington State, phone numbers typically have a three-dig\
Here are a few examples of phone numbers that could be considered\
I'd be happy to help generate a realistic phone number\
I'd be happy to help you generate a random phone\
Based on the format you provided, a realistic phone number for\

As we can see, none of these outputs are even phone numbers!

Let's see  if we can improve this using structured generation.

## The Structured Generation Workflow

In order to solve this problem we're going to introduce a *Structured Generation Workflow* outlined in this image:



Let's step through this:

### Real example

We start with a real example phone number, in this case for the Seattle Public Library, that we can use to verify the structure we are creating.

```python
phone_number = "(206) 386-4636"
```

For a simple example like this, we'll just be using a single phone number, for more complex examples it can be helpful to have more examples.

### Draft Structure

The next step in the process is for use to define a simple regex that we feel correctly models our real data.

```python
from outlines.types import Regex

phone_regex_1 = Regex(r'\([0-9]{3}\) [0-9]{3}-[0-9]{4}')
```

Next we need to validate this regex against our real data.

### Validate by matching examples

Whenever writing non-trivial code with structured generation it is *essential* that you first validate the code against your real data example(s).

We'll start with a simple method of validation: just checking that our regex matches the data.

```
import re

re.match(phone_regex_1.pattern, phone_number)
# <re.Match object; span=(0, 14), match='(206) 386-4636'>

```

Now that we have a match, we can move on to generating structured output!

### Generate Structure

We're ready to see if structured generation can make an improvement over our initial unstructured approach:

```python
phone_generator_v1 = outlines.Generator(model, phone_regex_1)

for _ in range(3):
    print(phone_generator_v1(prompt_phone))
```
> (206) 555-1234\
(206) 555-1234\
(206) 555-1234\
(206) 555-1234\
(206) 555-1234\
(206) 555-1234\
(206) 123-4567\
(206) 555-1234\
(206) 555-1234\
(206) 555-1234

At least we have phone numbers! But I think we can do better!

### Inspect output

In this case the model *did* create phone numbers and, impressively, got the area code correct. So using structured generation did improve things. However these numbers are pretty boring. Let's improve that structure!

## Iteration

We've walked through the loop once, so we can go quickly now through each iteration.

We start by improving our structure:

```python
phone_regex_2 = Regex(r'\([0-9]{3}\) [2-46-9]{3}-[02-9]{4}')
```

Before rushing to another round of generation, let's validate this new regex. We'll add just a bit more sophistication over our last check:

```python
re.match(phone_regex_2.pattern, phone_number)[0] == phone_number
# True
```
Now that we've validated, let's generate with this new regex!

```python
phone_generator_v2 = outlines.Generator(model, phone_regex_2)

for _ in range(3):
    print(phone_generator_v2(prompt_phone))
```

> (206) 867-5309\
(206) 666-7777\
(206) 444-3333\
(206) 444-3333\
(206) 943-2222\
(206) 323-6789\
(206) 444-3333\
(206) 867-5309\
(206) 466-2255\
(206) 222-3333

Better, but I don't like those repeated sequences. Like good software developers, let's iterate again!

## Reiteration - with debugging

Here's a fancier regex that should give us more interesting results:

```python
phone_regex_3_error = r'\([0-9]{3}\) [2-4][7-9][4-6]-[3-6][2-8][1-4]'
```

This looks good to me, but there's a subtle bug, that's why we *always* need to validate our structure against real data. This time we'll make our validator do a bit more work to verify the correct string is matched:

```python
if not re.match(phone_regex_3_error, phone_number):
    print("Regex fails match")
else:
    matched_string = re.match(phone_regex_3_error, phone_number)[0]
    if matched_string == phone_number:
    print("Successful match")
    else:
    print(f"Error {matched_string} != {phone_number}")
```
This prints out:
>  Error (206) 386-463 != (206) 386-4636

Ah! We were missing the last digit, let's fix that and regenerate:

```python
phone_regex_3_fixed = Regex(r'\([0-9]{3}\) [2-4][7-9][4-6]-[3-6][2-8][1-4][6-9]')
phone_generator_v3 = outlines.Generator(model, phone_regex_3_fixed)

for _ in range(3):
    print(phone_generator_v3(prompt_phone))
```

>(206) 494-3216\
(206) 374-6218\
(206) 494-3337\
(206) 476-3216\
(206) 484-3548\
(206) 495-3218\
(206) 494-5517\
(206) 375-4636\
(206) 384-6216\
(206) 385-6218

Much better!

Now you've seen a quick example of the structured generation workflow that can be used at the basis for building and iteration on much larger structured generation tasks!

---

### Guide Architecture (`docs/guide/architecture.md`)

# Architecture Overview

This guide explains how Outlines is organized so you can navigate the codebase, debug issues, and extend the library.

## How Structured Generation Works

When you ask an LLM to output JSON or follow a specific format, traditional approaches generate text freely and hope it matches. Outlines takes a different approach: it constrains the model at generation time by masking invalid tokens, making it impossible for the model to produce invalid output.

## Core Abstractions

Outlines has three main abstractions: **Model**, **Generator**, and **Type System**.

### Model and ModelTypeAdapter

The `Model` class (`outlines/models/base.py`) is the abstract base class for all LLM integrations. There are two categories based on how structured generation is implemented:

**Steerable models** (`SteerableModel`): Models where Outlines directly applies a logits processor during generation. This includes:
- `LlamaCpp` - llama.cpp bindings
- `MLXLM` - Apple MLX models
- `Transformers` - HuggingFace Transformers

**Black-box models** (`BlackBoxModel`): Models where Outlines delegates structured generation to the provider's API rather than applying logits processors directly. This includes:
- `OpenAI`, `Anthropic`, `Gemini`, `Mistral` - Cloud API providers
- `VLLM`, `VLLMOffline`, `SGLang`, `TGI`, `Ollama` - Inference servers with built-in structured generation
- `Dottxt` - Dottxt API

Note: Some black-box models (like vLLM or Ollama) could technically expose logits, but they implement structured generation server-side, so Outlines delegates to their APIs instead of building processors locally.

**The Model interface:**

Every model subclass must implement these methods:

| Method | Purpose |
|--------|---------|
| `generate(model_input, output_type, **kwargs)` | Generate a single response (internal, receives logits processor or output type) |
| `generate_batch(model_input, output_type, **kwargs)` | Generate responses for multiple prompts |
| `generate_stream(model_input, output_type, **kwargs)` | Stream a response token by token |

The base `Model` class provides these convenience methods that create a `Generator` internally:

| Method | Purpose |
|--------|---------|
| `__call__(model_input, output_type, backend, **kwargs)` | Generate a single response |
| `batch(model_input, output_type, backend, **kwargs)` | Generate batch responses |
| `stream(model_input, output_type, backend, **kwargs)` | Stream a response |

**ModelTypeAdapter - Bridging formats:**

Each model has a `type_adapter` attribute that handles format conversion between Outlines and the specific model provider:

```python
class ModelTypeAdapter(ABC):
    @abstractmethod
    def format_input(self, model_input) -> Any:
        """Convert user input to model-specific format.

        For API models: creates the `messages` argument
        For local models: may apply chat templates, convert str to list, etc.
        """
        ...

    @abstractmethod
    def format_output_type(self, output_type) -> Any:
        """Convert output type to model-specific format.

        For black-box models: creates `response_format` argument
        For steerable models: formats the logits processor for the model
        """
        ...
```

### Generator - Unifying the Generation Interface

The `Generator` (`outlines/generator.py`) is a factory function that returns the appropriate generator class based on the model type.

**Why Generator exists:**

Without Generator, users would need different code for different model types:

```python
# Without Generator - user needs to know model internals
if isinstance(model, SteerableModel):
    processor = build_logits_processor(output_type)
    result = model.generate(prompt, processor)
else:
    result = model.generate(prompt, output_type)
```

With Generator, the complexity is hidden:

```python
# With Generator - same code works for any model
generator = Generator(model, output_type)
result = generator(prompt)
```

**Generator classes:**

| Class | Used For | How It Works |
|-------|----------|--------------|
| `SteerableGenerator` | Local models (`LlamaCpp`, `MLXLM`, `Transformers`) | Builds and caches a logits processor from the output type, resets and passes it to the model on each call |
| `BlackBoxGenerator` | Sync API models | Passes output type directly to model's generate method |
| `AsyncBlackBoxGenerator` | Async API models | Async version of BlackBoxGenerator |

**SteerableGenerator internals:**

When you create a `SteerableGenerator` with an output type, it:

1. Converts the Python type to a `Term` using `python_types_to_terms()`
2. Based on the Term type, builds the appropriate logits processor:
   - `CFG` → calls `get_cfg_logits_processor()`
   - `JsonSchema` → calls `get_json_schema_logits_processor()`
   - Other terms → converts to regex via `to_regex()`, then calls `get_regex_logits_processor()`
3. Caches the processor for reuse
4. On each call, resets processor state and passes it to the model

### Type System - From Python Types to Constraints

The type system (`outlines/types/dsl.py`) converts Python types into constraints that can be enforced during generation.

**The conversion pipeline:**

```
Python Type → Term (via python_types_to_terms)
                    ↓
            ┌───────┴───────┐
            ↓               ↓
    CFG or JsonSchema    Other Terms
            ↓               ↓
    Direct to backend   to_regex() → Regex string
            ↓               ↓
            └───────┬───────┘
                    ↓
            Logits Processor (via backend)
```

**Term classes:**

`Term` is the base class for Outlines' constraint DSL. Key subclasses:

| Term | Purpose | Example |
|------|---------|---------|
| `Regex` | Match a regex pattern | `Regex("[0-9]+")` |
| `JsonSchema` | Match valid JSON for a schema | `JsonSchema(MyPydanticModel)` |
| `CFG` | Match a context-free grammar | `CFG(grammar_string)` |
| `String` | Match a literal string | `String("hello")` |
| `Sequence` | Concatenate terms | `String("[") + item + String("]")` |
| `Alternatives` | Match any of several terms | `term1 \| term2` |
| `KleeneStar` | Zero or more repetitions | `zero_or_more(term)` |
| `KleenePlus` | One or more repetitions | `one_or_more(term)` |
| `Optional` | Zero or one occurrence | `optional(term)` |

**python_types_to_terms:**

This function converts Python types to Term instances:

```python
def python_types_to_terms(ptype) -> Term:
    # Already a Term - return as-is
    if isinstance(ptype, Term):
        return ptype

    # Basic types - return predefined regex patterns
    if is_int(ptype):
        return types.integer
    if is_float(ptype):
        return types.number
    if is_str(ptype):
        return types.string
    if is_bool(ptype):
        return types.boolean

    # Structured types - convert to JsonSchema
    if is_pydantic_model(ptype) or is_dataclass(ptype) or is_typed_dict(ptype):
        return JsonSchema(ptype)

    # Enum - create alternatives from members
    if is_enum(ptype):
        return Alternatives([...])

    # Union, Literal, List, Tuple, Dict - handle recursively
    ...
```

## Data Flow

Here's how a structured generation request flows through the system:

```
1. User calls: model("What is 2+2?", int)

2. Model.__call__ creates Generator:
   Generator(model, int)

3. Generator factory checks model type:
   - SteerableModel → SteerableGenerator
   - BlackBoxModel → BlackBoxGenerator

4. For SteerableGenerator:
   a. python_types_to_terms(int) → Regex("-?[0-9]+")
   b. to_regex(term) → regex string
   c. get_regex_logits_processor(backend, model, regex) → LogitsProcessor

5. Generator.__call__(prompt):
   a. processor.reset()  # Reset state for new generation
   b. model.generate(prompt, processor)

6. During generation (steerable models only):
   - Model computes logits for all tokens
   - LogitsProcessor masks invalid tokens (set to -inf)
   - Model samples from remaining valid tokens

7. Result returned to user
```

## File Organization

```
outlines/
├── __init__.py              # Public API exports
├── generator.py             # Generator factory and classes
├── models/
│   ├── base.py              # Model, AsyncModel, ModelTypeAdapter base classes
│   ├── transformers.py      # HuggingFace Transformers
│   ├── llamacpp.py          # llama.cpp bindings
│   ├── mlxlm.py             # Apple MLX models
│   ├── openai.py            # OpenAI API
│   ├── anthropic.py         # Anthropic API
│   ├── vllm.py              # vLLM server
│   ├── vllm_offline.py      # vLLM offline mode
│   └── ...                  # Other providers
├── types/
│   ├── __init__.py          # Predefined types: integer, number, date, etc.
│   ├── dsl.py               # Term classes, python_types_to_terms, to_regex
│   └── utils.py             # Type checking utilities
├── backends/
│   ├── __init__.py          # get_*_logits_processor functions
│   ├── base.py              # LogitsProcessorType protocol
│   ├── outlines_core.py     # Default backend using outlines-core
│   ├── llguidance.py        # Microsoft llguidance backend
│   └── xgrammar.py          # xgrammar backend
├── processors/
│   ├── base_logits_processor.py  # Base processor implementation
│   └── tensor_adapters/     # Tensor library adapters
├── grammars/                # Predefined grammar files
└── templates.py             # Prompt template utilities
```

## Backends

Backends are responsible for converting constraints (regex, JSON schema, CFG) into logits processors that can be applied during generation. They only apply to steerable models.

**Available backends:**

| Backend | Default For | Description |
|---------|-------------|-------------|
| `outlines_core` | Regex, JSON Schema | The default backend, built on the `outlines-core` Rust library. Compiles constraints into finite state machines. |
| `llguidance` | CFG | Microsoft's llguidance library. Supports context-free grammars and is the only backend that handles CFG constraints. |
| `xgrammar` | - | Alternative backend using the xgrammar library. |

**How backends are selected:**

1. If the user specifies a backend via the `backend` parameter, that backend is used
2. Otherwise, the default backend for the constraint type is used:
   - Regex → `outlines_core`
   - JSON Schema → `outlines_core`
   - CFG → `llguidance`

**Backend interface:**

All backends inherit from `BaseBackend` and implement three methods:

```python
class BaseBackend(ABC):
    @abstractmethod
    def get_json_schema_logits_processor(self, json_schema: str) -> LogitsProcessorType:
        ...

    @abstractmethod
    def get_regex_logits_processor(self, regex: str) -> LogitsProcessorType:
        ...

    @abstractmethod
    def get_cfg_logits_processor(self, grammar: str) -> LogitsProcessorType:
        ...
```

**Specifying a backend:**

```python
from outlines import from_transformers, Generator

model = from_transformers("microsoft/Phi-3-mini-4k-instruct")

# Use xgrammar instead of the default outlines_core
generator = Generator(model, int, backend="xgrammar")
```

## Extension Points

### Adding a New Model Provider

1. Create a new file in `outlines/models/` (e.g., `mymodel.py`)
2. Implement a `ModelTypeAdapter` subclass with `format_input()` and `format_output_type()`
3. Implement a `Model` subclass with `generate()`, `generate_batch()`, and `generate_stream()`
4. Add a factory function (e.g., `from_mymodel()`)
5. Export from `outlines/models/__init__.py`
6. Add to `SteerableModel` or `BlackBoxModel` type alias as appropriate

---

### Guide Chat Templating (`docs/guide/chat_templating.md`)

# Chat templating

Instruction-tuned language models use "special tokens" to indicate different parts of text, such as the system prompt, the user prompt, any images, and the assistant's response. A [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating) is how different types of input are composited together into a single, machine-readable string.

Outlines supports chat templating throught the `Chat` model input class. It contains a list of messages similar in format to the chat history you would use with API models such as OpenAI or Anthropic and to the expected arguments of the `apply_chat_template` method of transformers tokenizers. You can find detailed information on the interface of this object in the [model inputs documentation](../features/core/inputs.md).

---

### Guide Core Concepts (`docs/guide/core_concepts.md`)

---
title: Core concepts
---

# Core concepts

Coming soon. This will document various concepts at a high level, so users can understand Outlines before diving into specific implementations.

1. Constrained decoding, tokens, and the basics of logit biasing
2. Different ways to define output structure (regex, JSON schema, Pydantic models, context-free grammars)
3. How finite state machines are used to guarantee output structure
4. `Generator`, `Application`, `Template`,
5. Prompt engineering vs. structured generation

---

### Guide Fastapi Vllm Deployment (`docs/guide/fastapi_vllm_deployment.md`)

---
title: Deploying with FastAPI
---

# Deploying with FastAPI

This guide demonstrates how to build a FastAPI application that leverages Outlines' async integration with vLLM. We create a customer support API that can intelligently categorize tickets and generate structured responses.

## Prerequisites

Before starting, ensure you have a vLLM server running (locally or remotely) and the following packages installed:

```shell
pip install fastapi uvicorn outlines openai pydantic
```

## Building the Application

### Step 1: Define Data Models

First, let's define our Pydantic models for structured outputs:

```python
# models.py
from enum import Enum
from typing import List
from pydantic import BaseModel, Field

class TicketCategory(str, Enum):
    BILLING = "billing"
    TECHNICAL = "technical"
    ACCOUNT = "account"
    PRODUCT = "product"
    OTHER = "other"

class TicketPriority(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    URGENT = "urgent"

class TicketAnalysis(BaseModel):
    category: TicketCategory
    priority: TicketPriority
    summary: str = Field(description="Brief summary of the issue")
    customer_sentiment: str = Field(description="Customer emotional state")
    key_issues: List[str] = Field(description="List of main problems")
    requires_human: bool = Field(description="Whether this needs human intervention")

class SupportResponse(BaseModel):
    greeting: str
    acknowledgment: str = Field(description="Acknowledge the customer's issue")
    solution_steps: List[str] = Field(description="Steps to resolve the issue")
    closing: str
```

### Step 2: Define the prompts

Let us now write the prompts that we will be using in our application, using Jinja 2's templating language. We separate them from the application implementation so they are easier to modify and version.

```ascii
{# prompts/categorize.txt #}
Analyze this customer support ticket:

Customer ID: {{ customer_id }}
Message: {{ message }}

Extract the category, priority, and other relevant information.
```

```ascii
{# prompts/respond.txt #}
Generate a professional customer support response.

Customer Message: {{ message }}
Category: {{ category }}
Priority: {{  priority }}
Customer Sentiment: {{ customer_sentiment }}

Create a helpful, empathetic response that addresses their concerns.
```

### Step 3: Create the FastAPI Application

Now let's create our FastAPI application with async vLLM integration:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Running the Application

### Step 1: Start your vLLM server

```shell
vllm serve Qwen/Qwen2.5-VL-7B-Instruct
```

### Step 2: Run the FastAPI application

```shell
uvicorn main:app --reload --host 0.0.0.0 --port 8080
```

## Testing the API

### Example 1: Analyze a support ticket

```shell
curl -X POST "http://localhost:8080/analyze-ticket" \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": "CUST123",
    "message": "I have been charged twice for my subscription this month. This is unacceptable and I want a refund immediately!"
  }'
```

Expected response:

```json
{
  "category": "billing",
  "priority": "high",
  "summary": "Customer charged twice for subscription, requesting refund",
  "customer_sentiment": "angry",
  "key_issues": ["duplicate charge", "subscription billing", "refund request"],
  "requires_human": false
}
```

### Example 2: Generate a support response

```shell
# First, get the analysis
ANALYSIS=$(curl -s -X POST "http://localhost:8080/analyze-ticket" \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": "CUST456",
    "message": "My app keeps crashing when I try to upload photos."
  }')

# Then generate a response
curl -X POST "http://localhost:8080/generate-response" \
  -H "Content-Type: application/json" \
  -d "{
    \"request\": {
      \"customer_id\": \"CUST456\",
      \"message\": \"My app keeps crashing when I try to upload photos.\"
    },
    \"analysis\": $ANALYSIS
  }"
```

By combining FastAPI's async capabilities with Outlines' structured generation, you can build robust APIs that leverage large language models.

## Using Alternative Backends: SGLang and TGI

One of the key advantages of Outlines is its unified API across different inference backends. You can easily switch from vLLM to SGLang or TGI with minimal code changes - just modify the model initialization in the `lifespan` function.

### Using SGLang Instead of vLLM

To use SGLang, simply change the client initialization:

```python
@asynccontextmanager
async def lifespan(app: FastAPI):
    """Initialize the async SGLang model on startup."""
    global async_model

    client = openai.AsyncOpenAI(
        base_url="http://localhost:30000/v1",  # SGLang server URL
        api_key="dummy"
    )
    async_model = models.from_sglang(client)

    yield

    async_model = None
```

Start your SGLang server with:

```shell
python -m sglang.launch_server \
    --model-path meta-llama/Llama-2-7b-chat-hf \
    --port 30000
```

### Using TGI Instead of vLLM

For TGI (Text Generation Inference), use the Hugging Face client:

```python
import huggingface_hub

@asynccontextmanager
async def lifespan(app: FastAPI):
    """Initialize the async TGI model on startup."""
    global async_model

    client = huggingface_hub.AsyncInferenceClient(
        "http://localhost:8080"  # TGI server URL
    )
    async_model = models.from_tgi(client)

    yield

    async_model = None
```

Start your TGI server with:

```shell
docker run --gpus all -p 8080:80 \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-2-7b-chat-hf
```

The rest of your FastAPI application - all the endpoints, error handling, and business logic - remains completely unchanged. This flexibility allows you to test different inference engines without rewriting your application.

---

### Guide Getting Started (`docs/guide/getting_started.md`)

---
title: Getting Started
---

# Getting Started

## Installation

We recommend using `uv` to install Outlines. You can find `uv` installation instructions [here](https://github.com/astral-sh/uv).

```shell
uv pip install 'outlines[transformers]'
```

or the classic `pip`:

```shell
pip install 'outlines[transformers]'
```

For more information, see the [installation guide](installation.md).

## Creating a Model

Outlines contains a variety of models that wrap LLM inference engines/clients. For each of them, you need to install the model's associated library as described in the [installation guide](installation.md).

The full list of available models along with detailed explanation on how to use them can be found in the [models page](../features/models/index.md) of the Features section of the documentation.

For a quick start, you can find below an example of how to initialize all supported models in Outlines:

=== "vLLM"

    ```python
    import outlines
    from openai import OpenAI

    # You must have a separate vLLM server running
    # Create an OpenAI client with the base URL of the VLLM server
    openai_client = OpenAI(base_url="http://localhost:11434/v1")

    # Create an Outlines model
    model = outlines.from_vllm(openai_client, "microsoft/Phi-3-mini-4k-instruct")
    ```

=== "Ollama"

    ```python
    import outlines
    from ollama import Client

    # Create an Ollama client
    ollama_client = Client()

    # Create an Outlines model, the model must be available on your system
    model = outlines.from_ollama(ollama_client, "tinyllama")
    ```

=== "OpenAI"

    ```python
    import outlines
    from openai import OpenAI

    # Create an OpenAI client instance
    openai_client = OpenAI()

    # Create an Outlines model
    model = outlines.from_openai(openai_client, "gpt-4o")
    ```

=== "Transformers"

    ```python
    import outlines
    from transformers import AutoModelForCausalLM, AutoTokenizer

    # Define the model you want to use
    model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"

    # Create a HuggingFace model and tokenizer
    hf_model = AutoModelForCausalLM.from_pretrained(model_name)
    hf_tokenizer = AutoTokenizer.from_pretrained(model_name)

    # Create an Outlines model
    model = outlines.from_transformers(hf_model, hf_tokenizer)
    ```


=== "llama.cpp"

    ```python
    import outlines
    from llama_cpp import Llama

    # Model to use, it will be downloaded from the HuggingFace hub
    repo_id = "TheBloke/Llama-2-13B-chat-GGUF"
    file_name = "llama-2-13b-chat.Q4_K_M.gguf"

    # Create a Llama.cpp model
    llama_cpp_model = Llama.from_pretrained(repo_id, file_name)

    # Create an Outlines model
    model = outlines.from_llamacpp(llama_cpp_model)
    ```

=== "Gemini"

    ```python
    import outlines
    from google.generativeai import GenerativeModel

    # Create a Gemini client
    gemini_client = GenerativeModel()

    # Create an Outlines model
    model = outlines.from_gemini(gemini_client, "gemini-1-5-flash")
    ```

=== "mlx-lm"

    ```python
    import outlines
    import mlx_lm

    # Create an MLXLM model with the output of mlx_lm.load
    # The model will be downloaded from the HuggingFace hub
    model = outlines.from_mlxlm(
        *mlx_lm.load("mlx-community/SmolLM-135M-Instruct-4bit")
    )
    ```

=== "SgLang"

    ```python
    import outlines
    from openai import OpenAI

    # You must have a separate SgLang server running
    # Create an OpenAI client with the base URL of the SgLang server
    openai_client = OpenAI(base_url="http://localhost:11434/v1")

    # Create an Outlines model
    model = outlines.from_sglang(openai_client)
    ```

=== "TGI"

    ```python
    # SgLang

    import outlines
    from huggingface_hub import InferenceClient

    # You must have a separate TGI server running
    # Create an InferenceClient client with the base URL of the TGI server
    tgi_client = InferenceClient("http://localhost:8080")

    # Create an Outlines model
    model = outlines.from_tgi(tgi_client)
    ```

=== "vLLM (offline)"

    ```python
    import outlines
    from vllm import LLM

    # Create a vLLM model
    vllm_model = LLM("microsoft/Phi-3-mini-4k-instruct")

    # Create an Outlines model
    model = outlines.from_vllm_offline(vllm_model)
    ```


## Generating Text

Once you have created the Outlines model for your inference engine/client, you are already all set to generate text! Models are callable such that you can simply call them with a text prompt. For instance:

```python
model = <your_model_as_defined_above>

# Call the model to generate text
result = model("Write a short story about a cat.")
print(result) # 'In a quiet village where the cobblestones hummed softly beneath the morning mist...'
```

Most models also support streaming through the use of a `streaming` method. You can directly use with a prompt just like regular text generation. For instance:

```python
model = <your_model_as_defined_above>

# Stream text
for chunk in model.streaming("Write a short story about a cat.")
    print(chunk) # 'In ...'
```

## Structured Generation

Outlines follows a simple pattern that mirrors Python's own type system for structured outputs. Simply specify the desired output type as you would when using type hinting with a function, and Outlines will ensure your data matches that structure exactly.

Supported output types can be organized in 5 categories:

- [Basic Types](../features/core/output_types.md#basic-python-types): `int`, `float`, `bool`...
- [Multiple Choices](../features/core/output_types.md#multiple-choices): using `Literal` or `Enum`
- [JSON Schemas](../features/core/output_types.md#json-schemas): using a wide range of possible objects including Pydantic models and dataclasses
- [Regex](../features/core/output_types.md#regex-patterns): through the Outlines's `Regex` object
- [Context-free Grammars](../features/core/output_types.md#context-free-grammars): through the Outlines's `CFG` object

Consult the section on [Output Types](../features/core/output_types.md) in the features documentation for more detailed information on all supported types for each output type category.

In the meantime, you can find below examples of using each of the five output type categories:

=== "Basic Types"

    ```python
    model = <your_model_as_defined_above>

    # Generate an integer
    result = model("How many countries are there in the world?", int)
    print(result) # '200'
    ```

=== "Multiple Choice"

    ```python
    from enum import Enum

    # Define our multiple choice output type
    class PizzaOrBurger(Enum):
        pizza = "pizza"
        burger = "burger"

    model = <your_model_as_defined_above>

    # Generate text corresponding to either of the choices defined above
    result = model("What do you want to eat, a pizza or a burger?", PizzaOrBurger)
    print(result) # 'pizza'
    ```

=== "JSON Schemas"

    ```python
    from datetime import date
    from typing import Dict, List, Union
    from pydantic import BaseModel

    model = <your_model_as_defined_above>

    # Define the class we will use as an output type
    class Character(BaseModel):
        name: str
        birth_date: date
        skills: Union[Dict, List[str]]

    # Generate a character
    result = model("Create a character", Character)
    print(result) # '{"name": "Aurora", "birth_date": "1990-06-15", "skills": ["Stealth", "Diplomacy"]}'
    print(Character.model_validate_json(result)) # name=Aurora birth_date=datetime.date(1990, 6, 15) skills=['Stealth', 'Diplomacy']
    ```

=== "Regex"

    ```python
    from outlines.types import Regex

    model = <your_model_as_defined_above>

    # Define our regex for a 3 digit number
    output_type = Regex(r"[0-9]{3}")

    # Generate the number
    result = model("Write a 3 digit number", output_type)
    print(result) # '236'
    ```

=== "Context-free Grammars"

    ```python
    from outlines.types import CFG

    model = <your_model_as_defined_above>

    # Define your Lark grammar as string
    arithmetic_grammar = """
        ?start: sum

        ?sum: product
            | sum "+" product   -> add
            | sum "-" product   -> sub

        ?product: atom
            | product "*" atom  -> mul
            | product "/" atom  -> div

        ?atom: NUMBER           -> number
            | "-" atom         -> neg
            | "(" sum ")"

        %import common.NUMBER
        %import common.WS_INLINE

        %ignore WS_INLINE
    """

    # Generate an arithmetic operation
    result = model("Write an arithmetic operation", CFG(grammar_string))
    print(result) # '2 + 3'
    ```

It's important to note that not all output types are available for all models due to limitations in the underlying inference engines. The [Models](../features/models/index.md) section of the features documentation includes a features matrix that summarize the availability of output types.

## Generators

Generators are an important type of objects in Outlines that are used to encapsulate a model and an output type. After having created a generator, you can call it using a similar interface to a model and it will generate text conforming to the output type you initially provided.

This feature is useful if you want to generate text several times for given model and output type. Not only does it prevent having to include the same output type at each call, but it also allows us to compile the output type only once instead of doing it at each generation (which is important for local models as this operation can be expensive).

For instance:

```python
from typing import Literal
from outlines import Generator

model = <your_model_as_defined_above>

# Create a generator
generator = Generator(model, Literal["pizza", "burger"])

# Call it as you would call a model
result = generator("What do you want to eat, a pizza or a burger?")
print(result) # pizza
```

You can find more information on generators in the dedicated page on [Generators](../features/core/generator.md) in the features documentation.

## Other features

On top of more detailed explanation on the concepts already discussed here, the [Features](../features/index.md) section of the documentation contains information on additional Outlines features such as applications, prompt templates, the regex DSL...

---

### Guide Installation (`docs/guide/installation.md`)

---
title: Installation
---

# Installation

## Dependency Management

We recommend using modern Python packaging tools such as `uv` for managing python dependencies.

### uv (Recommended)

```shell
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a virtual environment and install Outlines
uv venv
source .venv/bin/activate
uv pip install outlines
```

or with pip:

```shell
pip install outlines
```





## Optional Dependencies

To use Outlines models, you need to install the Python libraries for the associated inference engines/clients. Such libraries are not part of the general installation as you should only install the libraries needed for the specific models you want to use.

Outlines models with the installation of their associated additional depencies:

- [Anthropic](../features/models/anthropic.md): `pip install anthropic`
- [Dottxt](../features/models/dottxt.md): `pip install dottxt`
- [Gemini](../features/models/gemini.md): `pip install google-generativeai`
- [Llamacpp](../features/models/llamacpp.md): `pip install llama-cpp-python`
- [Mlx-lm](../features/models/mlxlm.md): `pip install mlx mlx-lm`
- [Ollama](../features/models/ollama.md): `pip install ollama` (after having downloaded Ollama in your system)
- [OpenAI](../features/models/openai.md): `pip install openai`
- [SGLang](../features/models/sglang.md): `pip install openai`
- [TGI](../features/models/tgi.md): `pip install huggingface_hub`
- [Transformers](../features/models/transformers.md): `pip install transformers`
- [TransformersMultiModal](../features/models/transformers_multimodal.md): `pip install transformers`
- [vLLM (online server)](../features/models/vllm.md): `pip install openai`
- [vLLM (offline)](../features/models/vllm_offline.md): `pip install vllm`

If you encounter any problems using Outlines with these libraries, take a look at their installation instructions. The installation of `openai` and `transformers` should be straightforward, but other libraries have specific hardware requirements.

!!! warning "Hardware Requirements"

    If you are using a local model, your model may require specific hardware. Please check the documentation for these libraries.

    Some libraries like `vllm` and `llama-cpp-python` require specific hardware, such as a compatible GPU. `mlx-lm` on its side is designed for Apple Silicon, so it may not be appropriate for your use case if you are on a different platform.

## Bleeding Edge

You can install the latest version of Outlines from the repository's `main` branch:

```sh
pip install git+https://github.com/dottxt-ai/outlines.git@main
```

This can be useful, for instance, when a fix has been merged but not yet released.

## Installing for Development

See the [contributing documentation](../community/contribute.md) for instructions on how to install Outlines for development, including an example using the `dot-install` method for one of the backends.

---

### Guide Migration (`docs/guide/migration.md`)

# Outlines 1.0 migration guide

Outlines 1.0 introduces some breaking changes that affect the way you use the library. You are likely concerned by all of the following sections, so please read this document carefully until the end.

This guide will help you migrate your code to the new version.

All previous functionalities will be supported until Outlines version 1.1.0, but a warning message will be displayed to remind you to migrate your code and provide instructions to help you do so. Please migrate your code to the v1 as soon as possible.

## Removed or modified features
- [Generate functions](#generate-functions)
- [Models](#models)
- [Samplers](#samplers)
- [Functions](#functions)
- [Text generation return types](#text-generation-return-types)
- [Inference arguments](#inference-arguments)

### Generate functions

The whole `generate` module has been removed. That includes the functions `generate.cfg`, `generate.choice`, `generate.format`,`generate.fsm`, `generate.json`, `generate.regex` and `generate.text`.

You should replace these functions by the [`Generator`](../features/core/generator.md) object along with the right output type as an argument (on top of the model). The output type can either be a python type or be an object from the `outlines.types` module. You can find more information about the output types in the [Output Types](../features/core/output_types.md) section of the features documentation.

Associated v1 output types for each deprecated function:
- `generate.cfg` -> `outlines.types.CFG`
- `generate.choice` -> `typing.Literal` or `typing.Union`
- `generate.format` -> native python types (`str`, `int` etc.)
- `generate.fsm` -> `outlines.types.FSM`
- `generate.json` -> `pydantic.BaseModel`, `typing.TypedDict`, `dataclasses.dataclass`, `genson.schema.SchemaBuilder` or `outlines.types.JsonSchema`
- `generate.regex` -> `outlines.types.Regex`
- `generate.text` -> no output type (`None`)

For instance, instead of:

```python
from outlines import generate

model = ...
generator = generate.choice(model, ["foo", "bar"])
```

You should now use:

```python
from typing import Literal
from outlines import Generator

model = ...
generator = Generator(model, Literal["foo", "bar"])
```

### Models

The model classes found in the `outlines.models` module are maintained but there are a few important changes to be aware of.

The functions used to created a model have been replaced by equivalent functions named with a `from_` prefix. The function `outlines.models.transformers` has been replaced by `outlines.from_transformers` for instance. On top of this change of name, the arguments have been modified. You should refer to the [models documentation](../features/models/index.md) for more details, but the overall idea is that you now need to provide a model/client instance from the inference library the Outlines model is wrapping.

For instance, instead of:

```python
from outlines import models

model = models.llamacpp(
    repo_id="M4-ai/TinyMistral-248M-v2-Instruct-GGUF",
    filename="TinyMistral-248M-v2-Instruct.Q4_K_M.gguf",
    chat_format="qwen",
)
```

You should now do:

```python
from llama_cpp import Llama
from outlines import from_llamacpp

llamacpp_model = Llama.from_pretrained(
    repo_id="M4-ai/TinyMistral-248M-v2-Instruct-GGUF",
    filename="TinyMistral-248M-v2-Instruct.Q4_K_M.gguf",
    chat_format="qwen",
)
model = from_llamacpp(llamacpp_model)
```

The `load_lora` methods that are present on the `VLLM` and `LlamaCpp` models have been removed. You should now handle lora loading through the `Llama` instance in the case of the `LlamaCpp` model or provide it as a keyword argument when calling the model in the case of the `VLLM` model.

For instance, instead of:

```python
from outlines import from_vllm
from vllm import LLM

model = from_vllm(
    LLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
)
model.load_lora("path/to/lora/file")

response = model("foo")
```

You should now do:

```python
from outlines import from_vllm
from vllm import LLM
from vllm.lora.request import LoRARequest

model = from_vllm(
    LLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
)
lora_request = LoRARequest("path/to/lora/file", 1, "path/to/lora/file")

response = model("foo", lora_request=lora_request)
```

The `ExLlamav2` model has been removed without replacement. This inference library is not fully compatible with Outlines, so we decided to remove it. You can still use it until final deprecation, but we recommend you to migrate to a different inference library right now.

### Samplers

The `outlines.samplers` module has been removed without replacement. You should now use the arguments of the inference library model to control the sampling. Depending on the model you use, this could be done at initialization or when calling the model to generate text (so when calling the outlines model or a generator).

For instance, instead of:

```python
from outlines import generate

model = <transformers_model>

generator = generate.text(model, samplers.beam_search(2))
response = generator("foo")
```

You should now do:

```python
from outlines import Generator

model = <transformers_model>

generator = Generator(model)
response = generator("foo", num_beams=2)
```

### Functions

The `outlines.function` module has been removed. It is replaced by the `outlines.applications` module. An [`Application`](../features/utility/application.md) serves a similar purpose as a `Function`: it encapsulates a prompt template and an output type. A difference is that can `Application` is not instantiated with a model name. Instead, you should provide a model instance along with the prompt when calling it.

For instance, instead of:

```python
from outlines import Function

prompt_template = ...
output_type = ...

fn = Function(
    prompt_template,
    output_type,
    "hf-internal-testing/tiny-random-GPTJForCausalLM",
)

result = fn("foo")
```

You should now do:

```python
from outlines import Application

prompt_template = ...
output_type = ...

application = Application(
    prompt_template,
    output_type,
)

model = ...

result = application(model, "foo")
```

### Text generation return types

In the previous version of Outlines, the return type of the generators depended on the output type provided. For instance, if you passed a Pydantic model to the `generate.json` function, the return type was a Pydantic model instance. In the v1, the return type of a generator is always a `str`, the raw text generated by the model. You are responsible for parsing the text into the desired format.

For instance, instead of:

```python
from pydantic import BaseModel
from outlines import generate

class Foo(BaseModel):
    bar: str

model = ...

generator = generate.json(model, Foo)

result = generator("foo")
print(result.bar)
```

You should now do:

```python
from pydantic import BaseModel
from outlines import Generator

class Foo(BaseModel):
    bar: str

model = ...

generator = Generator(model, Foo)

result = generator("foo")
result = Foo.model_validate_json(result) # parse the text into the Pydantic model instance
print(result.bar)
```

The [Output Types](../features/core/output_types.md) section of the features documentation includes extensive details on available output types.

### Inference arguments

In the previous version of Outlines, some of the inference arguments were standardized across the models and were provided as positional arguments to the generator or through the sampling params dictionary. Additionally, various default values were added by outlines to the inference library models. This is no longer the case. You should refer to the documentation of the inference library you use to find the right arguments for your use case and pass them as keyword arguments to the outlines generator when calling it.

For instance, instead of:

```python
from outlines import generate

model = <transformers_model>

generator = generate.text(model)

result = generator("foo", 256, ".", 10) # 256 tokens, stop at "." and seed 10
```

You should now do:

```python
from outlines import Generator

model = <transformers_model>

generator = Generator(model)

result = generator("foo", max_new_tokens=256, stop_strings=".", seed=10)
```

---

### Guide Selecting An Inference Backend (`docs/guide/selecting_an_inference_backend.md`)

This guide should provide a general overview of the available models in the [API reference](/api/models/).

## Models

-  [Anthropic](/api/models/anthropic)

---

### Guide Vlm (`docs/guide/vlm.md`)

# Vision-Language Models with Outlines

This guide demonstrates how to use Outlines with vision-language models. Vision-language models can process both text and images, allowing for tasks like image captioning, visual question answering, and more.

We will be using the Pixtral-12B model from Mistral to take advantage of some of its visual reasoning capabilities and a workflow to generate a multistage atomic caption.

## Setup
First, we need to install the necessary dependencies. In addition to Outlines, we"ll need to install the transformers library and any specific requirements for the vision-language model we"ll be using.

```shell
pip install outlines transformers torch pillow
```

### Initializing the Model
We"ll use the `outlines.from_transformers` function to initialize our vision-language model. For this function to return a vision multi-modal model we need to pass in a transformers model and a transformers processor that can handle both text and image inputs. Today we"ll be using the Pixtral model with the AutoProcessor.

```python
import outlines
import torch
from transformers import (
    AutoProcessor,
    LlavaForConditionalGeneration
)

model_name="mistral-community/pixtral-12b" # original magnet model is able to be loaded without issue
model_class=LlavaForConditionalGeneration
processor_class=AutoProcessor

def get_vision_model(model_name: str, model_class, processor_class):
    model_kwargs = {
        "torch_dtype": torch.bfloat16,
        "attn_implementation": "flash_attention_2",
        "device_map": "auto",
    }
    processor_kwargs = {
        "device": "cuda",
    }

    model = outlines.from_transformers(
        model_class.from_pretrained(model_name, **model_kwargs),
        processor_class.from_pretrained(model_name, **processor_kwargs),
    )
    return model
model = get_vision_model(model_name, model_class, processor_class)
```

### Defining the Schema
Next, we will define a schema for the output we expect from our vision multi-modal model. This schema will help structure the model's responses. We use the `outlines.Generator` object to create a generator for our schema that will then be called with our prompt and images.

```python
from enum import Enum
from pydantic import BaseModel, Field, confloat, constr
from pydantic.types import StringConstraints, PositiveFloat
from typing import List
from typing_extensions import Annotated

class TagType(Enum):
    ENTITY = "Entity"
    RELATIONSHIP = "Relationship"
    STYLE = "Style"
    ATTRIBUTE = "Attribute"
    COMPOSITION = "Composition"
    CONTEXTUAL = "Contextual"
    TECHNICAL = "Technical"
    SEMANTIC = "Semantic"

class ImageTag(BaseModel):
    tag: Annotated[
        constr(min_length=1, max_length=30),
        Field(
            description=(
                "Descriptive keyword or phrase representing the tag."
            )
        )
    ]
    category: TagType
    confidence: Annotated[
        confloat(le=1.0),
        Field(
            description=(
                "Confidence score for the tag, between 0 (exclusive) and 1 (inclusive)."
            )
        )
    ]

class ImageData(BaseModel):
    tags_list: List[ImageTag] = Field(..., min_items=8, max_items=20)
    short_caption: Annotated[str, StringConstraints(min_length=10, max_length=150)]
    dense_caption: Annotated[str, StringConstraints(min_length=100, max_length=2048)]

image_data_generator = outlines.Generator(model, ImageData)
```

This schema defines the structure for image tags, including categories like Entity, Relationship, Style, etc., as well as short and dense captions.

### Preparing the Prompt

We'll create a prompt that instructs the model on how to analyze the image and generate the structured output:

```
/* Detailed source-code truncated for AI context efficiency. */
```

This prompt provides detailed instructions to the model on how to generate comprehensive tag lists, captions, and dense captions for image analysis. Because of the ordering of the instructions the original tag generation serves as a sort of visual grounding for the captioning task, reducing the amount of manual post processing required. It is essential to include the <image> tag in the prompt at the location where the image will be inserted.

### Generating Structured Output
Now we can use our model to generate structured output based on an input image:

```python
from io import BytesIO
from urllib.request import urlopen
from PIL import Image

def img_from_url(url):
    img_byte_stream = BytesIO(urlopen(url).read())
    return Image.open(img_byte_stream).convert("RGB")

image_url="https://upload.wikimedia.org/wikipedia/commons/9/98/Aldrin_Apollo_11_original.jpg"
image= img_from_url(image_url)
result = image_data_generator({
    "text": pixtral_instruction,
    "images": image
})
print(result)
```

This code loads an image from a URL, passes it to our vision multi-modal model along with the instruction prompt, and generates a structured output based on the defined schema. We end up with an output like this, ready to be used for the next stage in your pipeline:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Conclusion

This guide demonstrated how Outlines enables structured output generation with vision-language models. With the techniques shown above, you can build:

- **Content Management Systems**: Automatically tag and categorize visual content with structured metadata that can be directly stored in databases, enabling powerful search and filtering capabilities
- **Accessibility Tools**: Generate rich, structured descriptions of images that can be adapted for different contexts - from brief alt-text to detailed scene descriptions for screen readers
- **Quality Assurance Pipelines**: Validate visual content against specific criteria by extracting structured attributes and checking them against business rules

---

### Examples Prompt Templates Chain Of Density (`docs/examples/prompt_templates/chain_of_density.txt`)

Article: {{ article }}

You will generate increasingly concise, entity-dense summaries of the above Article.

Repeat the following 2 steps 5 times.

Step 1. Identify 1-3 informative Entities ("; " delimited) from the Article which are missing from the previously generated summary.
Step 2. Write a new, denser summary of identical length which covers every entity and detail from the previous summary plus the Missing Entities.

A Missing Entity is:
- Relevant: to the main story.
- Specific: descriptive yet concise (5 words or fewer).
- Novel: not in the previous summary.
- Faithful: present in the Article.
- Anywhere: located anywhere in the Article.

Guidelines:
- The first summary should be long (4-5 sentences, ~80 words) yet highly non-specific, containing little information beyond the entities marked as missing. Use overly verbose language and fillers (e.g., "this article discusses") to reach ~80 words.
- Make every word count: rewrite the previous summary to improve flow and make space for additional entities.
- Make space with fusion, compression, and removal of uninformative phrases like "the article discusses".
- The summaries should become highly dense and concise yet self-contained, e.g., easily understood without the Article.
- Missing entities can appear anywhere in the new summary.
- Never drop entities from the previous summary. If space cannot be made, add fewer new entities.

Remember, use the exact same number of words for each summary.

Answer in JSON. The JSON should be a a dictionary with key "summaries" that contains a list (length 5) of dictionaries whose keys are "Missing_Entities" and "Denser_Summary".

---

### Examples Prompt Templates Classification (`docs/examples/prompt_templates/classification.txt`)

You are an experienced customer success manager.

Given a request from a client, you need to determine when the
request is urgent using the label "URGENT" or when it can wait
a little with the label "STANDARD".

# Examples

Request: "How are you?"
Label: STANDARD

Request: "I need this fixed immediately!"
Label: URGENT

# TASK

Request: {{ request }}
Label:

---

### Examples Prompt Templates React Agent (`docs/examples/prompt_templates/react_agent.txt`)

<|im_start|>system
You are a world class AI model who answers questions in JSON with correct Pydantic schema.
Here's the json schema you must adhere to:
<schema>
{{ schema }}
</schema>
Today is {{ today }}
You run in a loop of Scratchpad, Thought, Action, Action Input, PAUSE, Observation.
At the end of the loop you output a Final Answer.
Use Scratchpad to store the information from the Observation useful to answer the question
Use Thought to describe your thoughts about the question you have been asked and reflect carefully about the Observation if it exists.
Use Action to run one of the actions available to you.
Use Action Input to input the arguments of the selected action - then return PAUSE.
Observation will be the result of running those actions.
Your available actions are:
calculate:
e.g. calulate: 4**2 / 3
Runs a calculation and returns the number - uses Python so be sure to use floating point syntax if necessary
wikipedia:
e.g. wikipedia: Django
Returns a summary from searching Wikipedia
DO NOT TRY TO GUESS THE ANSWER. Begin!
<|im_end|>
<|im_start|>user
{{ question }}
<|im_end|>
<|im_start|>assistant

---

### Examples Prompt Templates Simtom Prospective Taking (`docs/examples/prompt_templates/simtom_prospective_taking.txt`)

<s>[INST] The following is a sequence of events about some characters, that takes place in multiple locations.
Your job is to output only the events that the specified character, {{character}}, knows about.

Here are a few rules:
1. A character knows about all events that they do.
2. If a character is in a certain room/location, that character knows about all other events that happens in the room. This includes other characters leaving or exiting the location, the locations of objects in that location, and whether somebody moves an object to another place.
3. If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location.

Story: {{story}}
What events does {{character}} know about? Only output the events according to the above rules, do not provide an explanation. [/INST]

---

### Examples Prompt Templates Simtom Simulation (`docs/examples/prompt_templates/simtom_simulation.txt`)

<s>[INST] {% for event in events %}
{{event}}
{% endfor %}
You are {{name}}.
Based on the above information, answer the following question:
{{question}}
You must choose one of the above choices, do not say there is not enough information. Answer with a single word, do not output anything else. [/INST]

---

### Features Advanced Backends (`docs/features/advanced/backends.md`)

---
title: Structured Generation Backends
---

# Structured Generation Backends

Outlines relies on a structured generation backend to control text generation for steerable models such thah they conform to the output type provided. One of those backends is of course `outlines-core`, but you also have access to two other libraries that fulfill the same purpose: `llguidance` and `xgrammar`.

## Overview

To select the backend to use for your generation, provide a value for the `backend` argument when calling a model or a generator.

For instance:

```python
from typing import Literal
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

output_type = Literal["Paris", "London", "Rome", "Berlin"]

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

result = model("What is the capital of France?", output_type, backend="llguidance")
print(result) # 'Paris'

generator = outlines.Generaor(model, output_type)
result = generator("What is the capital of France?", backend="xgrammar")
print(result) # 'Paris'
```

If you do not provide a value for the `backend` argument, the default value will be used. The default value depends on the type of output type:

- JSON schema: `outlines_core`
- Regex: `outlines_core`
- Context-free grammar: `llguidance`

## Features matrix

As mentioned previously, selecting the structured generation backend is only applicable to steerable models, so `Transformers`, `LlmaCpp` and `MLXLM`. Additionaly, some backends do not support some models within those or some output types.

| | outlines_core | llguidance | xgrammar |
|---|---|---|---|
| **Models** | | | |
| Transformers | ✅ | ✅ | ✅ |
| LlamaCpp | ✅ | ✅ | ❌ |
| MLXLM | ✅ | ✅ | ✅ |
| **Output Types** | | | |
| JSON Schema | ✅ | ✅ | ✅ |
| Regex | ✅ | ✅ | ✅ |
| Grammar | ❌ | ✅ | ✅ |

---

### Features Advanced Logits Processors (`docs/features/advanced/logits_processors.md`)

---
title: Logits Processors
---

# Logits Processors

Logits processors are objects that control text generation by modifying the probability distribution of possible next tokens. They do this by adjusting the logits (raw model outputs) at each generation step, effectively biasing the model's token selection.

Processors can be used to:

1. Generate structured output (e.g., JSON that follows a specific schema)
2. Prevent the model from generating specific words or tokens
3. Implement custom token sampling strategies

## Overview

Outlines uses logits processors with steerable models — models that run locally and allow fine-grained control over the generation process. When using such models in Outlines, the output type provided is turned into a logits processor that is then passed to the inference engine.

There are three models that support logits processors:

- LlamaCpp
- MLXLM
- Transformers

Instead of providing an output type that will be turned into a logits processor, it is possible to directly provide a logits processor. To do so, you must create a `Generator` instance using the `processor` keyword argument. You cannot directly call the model with a logits processor.

For instance:

```python
import transformers
from outlines import Generator, from_transformers
from outlines.processors import RegexLogitsProcessor

# Create a model
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("NousResearch/Hermes-2-Pro-Llama-3-8B"),
    transformers.AutoTokenizer.from_pretrained("NousResearch/Hermes-2-Pro-Llama-3-8B")
)

# Create a regex logits processor that only returns hex unicode notations
logits_processor = RegexLogitsProcessor(r"U\+[0-9A-Fa-f]{4,6}", model.tokenizer, model.tensor_library_name)

# Create a generator with the logits processor and use it to generate text
generator = Generator(model, processor=logits_processor)
response = generator("What's the unicode for the hugging face emoji")

print(response) # U+1F917
```

## Creating Custom Logits Processors

You can create your own logits processor by subclassing the `OutlinesLogitsProcessor` class. This allows you to implement specific logic to modify logits as needed.
Your logits processor needs to implement the `process_logits` method to modify the logits.
`process_logits` accepts:
- `input_ids`: the ids of the tokens of the existing sequences in a 2D tensor.
- `logits`: the logits for the current generation step in a 2D tensor.

In the example below, we create a custom logits processor to force the model to provide a response using only binary representation (so only the tokens for 0 and 1 are allowed):

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Features Core Exceptions (`docs/features/core/exceptions.md`)

---
title: Error Handling
---

# Error Handling

For server-based models, Outlines provides a common exception hierarchy under `OutlinesError`.

For local models, native runtime exceptions are preserved (see the list of local and server-based models at [Model Types](../models/index.md#model-types)).

Exception normalization covers both sync and async modes, and both `generate` and `generate_stream()` (or equivalent) methods.

## Exception Hierarchy

The common exception classes live in `outlines.exceptions`.

```
OutlinesError
└── APIError
    ├── AuthenticationError
    ├── PermissionDeniedError
    ├── NotFoundError
    ├── RateLimitError
    ├── BadRequestError
    ├── ServerError
    ├── APITimeoutError
    ├── APIConnectionError
    ├── ProviderResponseError
    └── GenerationError
```

## Quick Reference

| Class                   | Category            | HTTP status(es)               | Suggested user action                                                                       |
| ----------------------- | ------------------- | ----------------------------- | ------------------------------------------------------------------------------------------- |
| `AuthenticationError`   | Client error        | 401                           | Fix input: Check API key.                                                                    |
| `PermissionDeniedError` | Client error        | 403                           | Fix input: Check permissions for API key.                                                    |
| `NotFoundError`         | Client error        | 404                           | Fix input: Confirm model name, endpoint, etc.                                                |
| `RateLimitError`        | Client error        | 429                           | Retry: Slow down and retry, or reduce request frequency, batch size, etc.                   |
| `BadRequestError`       | Client error        | 400, 409, 413, 422, other 4xx | Fix input: Check prompt length, schema, unsupported parameters, etc.                        |
| `ServerError`           | Server error        | 5xx, 529 (Anthropic overload) | Retry: Perhaps retry after a short wait.                                                     |
| `APITimeoutError`       | Network/transport   | — (no HTTP status)            | Retry: Provider may be overloaded.                                                           |
| `APIConnectionError`    | Network/transport   | — (no HTTP status)            | Retry: Could not reach provider, check connection.                                           |
| `ProviderResponseError` | Response/generation | — (no HTTP status)            | Provider bug: May be a temporary issue or schema/format mismatch.                           |
| `GenerationError`       | Response/generation | — (no HTTP status)            | Inspect output: Output was refused, hit a content filter, reached the model's max-token limit, or similar. |

## Usage

Normalized `APIError` subclasses are raised by Outlines model wrappers. If you call a provider SDK directly, expect provider-native exceptions instead.

```python
from outlines.exceptions import (
    APIError,
    AuthenticationError,
    RateLimitError,
    ServerError,
    APITimeoutError,
    APIConnectionError,
)

try:
    result = model("Hello")
except AuthenticationError:
    print("Check your API key")
except RateLimitError as e:
    print(f"Rate limited by {e.provider}, back off and retry")
except APIError as e:
    if e.retryable:
        print(f"Transient error from {e.provider} (status={e.status_code}), retry")
    else:
        raise
```

## `retryable` Flag

Four exception types set `retryable = True`: `RateLimitError`, `ServerError`, `APITimeoutError`, and `APIConnectionError`. You can branch on this attribute without hardcoding a list of types:

```python
except APIError as e:
    if e.retryable:
        time.sleep(backoff)
        retry()
    else:
        raise
```

## Available Attributes

Every `APIError` instance exposes:

| Attribute            | Type            | Description                                   |
| -------------------- | --------------- | --------------------------------------------- |
| `provider`           | `str \| None`   | Provider name, e.g. `"mistral"`, `"openai"` |
| `original_exception` | `Exception \| None` | The raw SDK exception                     |
| `status_code`        | `int \| None`   | HTTP status code, if available                |
| `request_id`         | `str \| None`   | Provider request ID, if available             |
| `retryable`          | `bool`          | Whether the error is worth retrying           |
| `hint`               | `str`           | Actionable suggestion shown in tracebacks                                 |

---

### Features Core Generator (`docs/features/core/generator.md`)

---
title: Generator API
---

# Generator


The `Generator` class is the core component of Outlines v1. `Generator` accepts a [model](../models/index.md) and an optional [output type](../core/output_types.md). If no output type is provided, the `Generator` will return unstructured text.

!!! note

    `Generator` is new as of Outlines v1, and replaces previous generator constructors:

    - `generate.cfg`
    - `generate.choice`
    - `generate.format`
    - `generate.fsm`
    - `generate.json`
    - `generate.regex`
    - `generate.text`

## Methods

Generators implement the same methods as models:

- `__call__`
- `batch`
- `stream`

All of them take a single positional argument: the [model input](../core/inputs.md) from which text is generated. Contrarily to the equivalent methods of models, you do not need to provide an output type as it has already been defined when initializing the generator.

## Basic Usage

```python
from outlines import Generator, from_transformers
import transformers

# Initialize a model
model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(model_name),
    transformers.AutoTokenizer.from_pretrained(model_name),
)

# Create a generator for plain text
generator = Generator(model)
result = generator("Write a short poem about AI.")

# Print the result
print(result)
```

## Structured Generation

```python
from pydantic import BaseModel
from outlines import Generator, from_transformers
import transformers

# Define a Pydantic model for structured output
class BookRecommendation(BaseModel):
    title: str
    author: str
    year: int

# Initialize a model
model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained(model_name),
    transformers.AutoTokenizer.from_pretrained(model_name),
)

# Create a generator for JSON output
generator = Generator(model, BookRecommendation)

# Generate a book recommendation
result = generator("Recommend a science fiction book.")

# Parse the JSON result into a Pydantic model
book = BookRecommendation.model_validate_json(result)
print(f"{book.title} by {book.author} ({book.year})")
```

## Parameters

- `model`: The language model to use for generation
- `output_type`: Optional. The type of output to generate

## Generation Parameters

When calling the generator, you can pass additional parameters to control the generation process. These parameters are passed through to the underlying model, so they depend on the specific model being used.

Common parameters for most models include:
- `max_new_tokens`: Maximum number of tokens to generate
- `temperature`: Controls randomness (higher values = more random)
- `top_p`: Controls diversity via nucleus sampling
- `stop_strings`: String or list of strings at which to stop generation

Example:
```python
result = generator(
    "Write a short story.",
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9,
    stop_strings=["THE END", "###"]
)
```

## Return Value

The generator always returns a raw string containing the generated text. When generating structured outputs, you need to parse this string into the desired format.

Unlike in Outlines v0, where the return type could be a parsed object, in v1 you are responsible for parsing the output when needed:

```python
# Outlines v1 approach
from pydantic import BaseModel
from outlines import Generator

class Person(BaseModel):
    name: str
    age: int

generator = Generator(model, Person)
result = generator("Generate a person:")

# Parse the result yourself
person = Person.model_validate_json(result)
```

::: outlines.generator.Generator

---

### Features Core Inputs (`docs/features/core/inputs.md`)

---
title: Model Inputs
---

# Model Inputs

Outlines models accept various types of inputs to generate text. The input format depends on the capabilities of the underlying model and the type of task you want to perform. The most basic type of input is a single string prompt, it's accepted by all models.

## Overview

The model input is the first argument of the `__call__`, `stream` and `batch` methods of both models and generators.

There are 3 types of model inputs:

- **Text prompts** - Simple strings
- **Multimodal inputs** - List containning a string prompt along with assets
- **Chat inputs** - `Chat` instances containing messages

## Text Prompts

The simplest form of input is a plain text string. This works with all models and is suitable for standard text generation tasks.

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

# Create a model
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)

# Simple text prompt
response = model("What's the capital of France?", max_new_tokens=20)
print(response)  # 'Paris'
```

## Multimodal Inputs (Vision)

For models that support them, you can provide a list containing a text prompt and one or more assets.

There are 3 types of assets defined in Outlines:

- `Image`: contains a PIL Image
- `Video`: contains any object (you must choose a format that is supported by your model)
- `Audio`: contains any object (you must choose a format that is supported by your model)

Among those, `Image` is by far the most important as multiple models support vision inputs.

For instance with vision input:

```python
import io
import requests
import PIL
import outlines
import openai
from outlines.inputs import Image

# Create the model
model = outlines.from_openai(
    openai.OpenAI(),
    "gpt-4o"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

## Chat Inputs

For conversational models, you can use the `Chat` class to provide a conversation history with multiple messages.

A `Chat` instance is instantiated with an optional list of messages. Each message must be a dictionary containing two mandatory keys:
- `role`: must be one of `system`, `assistant` or `user`
- `content`: must be either a string or a multimodal input (if the model supports it)

For instance:

```python
import io
import requests
import PIL
from outlines.inputs import Chat, Image

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])
print(prompt)
# {'role': 'system', 'content': 'You are a helpful assistant.'}
# {'role': 'user', 'content': ['Describe the image', Image(image=<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x300 at 0x7FFA526CCC40>)]}
```

After having created a `Chat` instance, you can add one or several messages thanks to the `append` and `extend` methods. You can also remove the last message of the Chat with the `pop` method.

For instance:

```python
from outlines.inputs import Chat

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
])

# Add a message
prompt.append({"role": "user", "content": "How are you doing today?"})
print(prompt)
# {'role': 'system', 'content': 'You are a helpful assistant.'}
# {'role': 'user', 'content': 'How are you doing today?'}

# Remove the last messsage
last_message = prompt.pop()
print(last_message)
# {'role': 'user', 'content': 'How are you doing today?'}
print(prompt)
# {'role': 'system', 'content': 'You are a helpful assistant.'}

# RAdd several messages
prompt.extend([
    {"role": "user", "content": "How are you doing today?"},
    {"role": "assistant", "content": "Excellent, thanks!"}
])
print(prompt)
# {'role': 'system', 'content': 'You are a helpful assistant.'}
# {'role': 'user', 'content': 'How are you doing today?'}
# {'role': 'assistant', 'content': 'Excellent, thanks!'}
```

Finally, there are three convenience method to easily add a message:

- add_system_message
- add_user_message
- add_assistant_message

As the role is already set, you only need to provide the content.

For instance:

```python
from outlines.inputs import Chat

# Create the chat input
prompt = Chat()

prompt.add_system_message("You are a helpful assistant.")
prompt.add_system_message("How are you doing today?")
prompt.add_system_message("Excellent, thanks!")

print(prompt)
# {'role': 'system', 'content': 'You are a helpful assistant.'}
# {'role': 'user', 'content': 'How are you doing today?'}
# {'role': 'assistant', 'content': 'Excellent, thanks!'}
```

## Batching

In the case of batching, for models that support it, you just have to provide several instances of the model inputs described above in a list.

For instance:

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

# Create model
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

# Create a list of prompts that will be used in a single batch
prompts = [
    "What's the capital of Lithuania?",
    "What's the capital of Latvia?",
    "What's the capital of Estonia?"
]

# Call it to generate text
result = model.batch(prompts, max_new_tokens=20)
print(result) # ['Vilnius', 'Riga', 'Tallinn']
```

---

### Features Core Output Types (`docs/features/core/output_types.md`)

---
title: Output Types
---

# Output Types

Outlines provides a simple and intuitive way of defining the output structure of text generation. Possible output formats include basic Python types, multiple-choices, JSON schemas, regular expressions and context-free grammars.

## Overview

Outlines models accept a __prompt__ and an __output type__ when they are invoked, as well as additional inference keyword arguments that are forwarded on to the underlying model.

Output types can be from the general Python ecosystem, including:
- Most native Python types, such as `int` or `str`
- Types from the `typing` module, such as `Literal`, `List`, `Dict`, `Enum`, etc
- Types from popular third party libraries such as Pydantic or GenSON.

Outlines also provides special classes for certain output structures (more details below):
- Multiple choices with `Choice`
- JSON schemas with `JsonSchema`
- Regular expressions with `Regex`
- Context-free grammars with `CFG`

The general idea is that you should provide as an output type what you would give as the type hint of the return type of a function.

Consider the following functions for instance:

```python
from datetime import date
from typing import Dict, List, Literal, Union
from pydantic import BaseModel

class Character(BaseModel):
    name: str
    birth_date: date
    skills: Union[Dict, List[str]]

def give_int() -> int:
    ...

def pizza_or_burger() -> Literal["pizza", "burger"]:
    ...

def create_character() -> Character:
    ...
```

With an Outlines model, you can generate text that respects the type hints above by providing those as the output type:

```python
model("How many minutes are there in one hour", int) # "60"
model("Pizza or burger", Literal["pizza", "burger"]) # "pizza"
model("Create a character", Character, max_new_tokens=100) # '{"name": "James", "birth_date": "1980-05-10)", "skills": ["archery", "negotiation"]}'
```

An important difference with function type hints though is that an Outlines generator always returns a string.
You have to cast the response into the type you want yourself.

For instance:

```python
result = model("Create a character", Character, max_new_tokens=100)
casted_result = Character.model_validate_json(result)
print(result) # '{"name": "Aurora", "birth_date": "1990-06-15", "skills": ["Stealth", "Diplomacy"]}'
print(casted_result) # name=Aurora birth_date=datetime.date(1990, 6, 15) skills=['Stealth', 'Diplomacy']
```

## Output Type Categories

We can group possible output types in several categories based on the use case they correspond to. While most of those types are native python or types coming from well-known third-party libraries, there are three Outlines-specific types: `JsonSchema`, `Regex` and `CFG`. Their use is explained below.

### Basic Python Types

The most straightforward form of structured generation is to return an answer that conforms to a given basic type such as an int or a python list. You can use the basic Python types and the types from the `typing` library. For instance:

```python
from typing import Dict

output_type = float # example of valid value: "0.05"
output_type = bool # example of valid value: "True"
output_type = Dict[int, str] # example of valid value: "{1: 'hello', 2: 'there'}"
```

You can combine types to create more complex response formats by relying on collection types and types such as `Union` and `Optional`. Let's consider for instance the output type below used to represent semi-structured data:

```python
from typing import Dict, List, Optional, Tuple, Union

output_type = Dict[str, Union[int, str, List[Tuple[str, Optional[float]]]]]
```

Values created with this output type would be dictionaries with string as keys and values made of either an integer, a string or a list of two elements tuples: a string and either a float or None. Example of a valid response for text generated with this output type (it would be contained in a string):

```json
{
    "name": "Alice",
    "age": 30,
    "metrics": [("engagement", 0.85), ("satisfaction", None)]
}
```

### Multiple Choices

Outlines supports multiple choice classification by using the `Literal` or `Enum` output types. For instance:

```python
from enum import Enum
from typing import Literal

class PizzaOrBurger(Enum):
    pizza = "pizza"
    burger = "burger"

# Equivalent multiple-choice output types
output_type = Literal["pizza", "burger"]
output_type = PizzaOrBurger
```

Additionally, you can use the Outlines-specific type `Choice` that takes a `list` as an argument. This type is useful in situations in which the list of choices is dynamic.

For instance:

```python
from outlines.types import Choice

def get_multiple_choices() -> list:
    # we could have something complex here
    return ["pizza", "burger"]

output_type = Choice(get_multiple_choices())
```

### JSON Schemas

Multiple different common Python types are often used to store information equivalent to a JSON schema. The following can be used in Outlines to generate text that respects a JSON schema:

- A Pydantic class
- A Dataclass
- A TypedDict
- A [GenSON](https://github.com/wolverdude/GenSON) `SchemaBuilder`
- A Callable (the parameters are turned into the keys and the type hinting is used to define the types of the values)

For instance:

```python
from dataclasses import dataclass

@dataclass
class Character:
    name: str
    age: int

output_type = Character

def character(name: str, age: int):
    return None

output_type = character
```

There are two other JSON schema formats that require Outlines-specific classes: JSON schema strings and dictionaries.

As those are contained in regular Python strings or dictionaries, the associated output format would be ambiguous if they were to be provided directly. As a result, Outlines requires them to be wrapped in a `outlines.types.JsonSchema` object. For instance:

```python
from outlines.types import JsonSchema

schema_string = '{"type": "object", "properties": {"answer": {"type": "number"}}}'
output_type = JsonSchema(schema_string)

schema_dict = {
    "type": "object",
    "properties": {
        "answer": {"type": "number"}
    }
}
output_type = JsonSchema(schema_dict)
```

`JsonSchema` accepts two optional parameters:
- `whitespace_pattern` (defaults to `None`): specifies the pattern to use for JSON syntactic whitespace. If none is provided, the default permissive JSON whitespace rules are used. It is applied by the `outlines_core` backend; the `llguidance` and `xgrammar` backends do not support it and raise an error if one is set.
- `ensure_ascii` (defaults to `True`): defines the value to use for the argument `ensure_ascii` of the `json.dumps` method. If false, non-ASCII characters will be turned into unicodes.

### Regex Patterns

Outlines provides support for text generation constrained by regular expressions. Since regular expressions are expressed as simple raw string literals, regex strings must wrapped in an `outlines.types.Regex` object.

```python
from outlines.types import Regex

regex = r"[0-9]{3}"
output_type = Regex(regex)
```

The `outlines.types` module contains a few common regex patterns stored in variables you can import and directly use as output types. Common patterns include a sentence, an email address and an [ISBN reference](https://en.wikipedia.org/wiki/ISBN). For instance:

```python
from outlines.types import sentence

print(type(sentence)) # outlines.types.dsl.Regex
print(sentence.pattern) # [A-Z].*\s*[.!?]
```

To help you create complex regex patterns yourself, you can use the Outlines [regex DSL](../utility/regex_dsl.md).

### Context-Free Grammars

Outlines allows you to generate text that respects the syntax of a context-free grammar. Context-free grammars are defined using [Lark](https://lark-parser.readthedocs.io/en/latest/index.html), a grammar language. Since grammars are expressed as a string, Large CFG strings should be be wrapped in an `outlines.types.CFG` object. For instance:

```python
from outlines.types import CFG

grammar_string = """
    start: expr
    expr: "{" expr "}" | "[" expr "]" |
"""
output_type = CFG(grammar_string)
```

You can find a few Lark grammar examples in the [grammars module](../../api_reference/grammars.md).

## Output type availability

The output types presented above are not available for all models as some have only limited support for structured outputs. Please refer to the documentation of the specific model you wish to use to know what output types it supports.

---

### Features Models Anthropic (`docs/features/models/anthropic.md`)

---
title: Anthropic
---

# Anthropic

!!! Installation

    You need to install the `anthropic` library to be able to use the Anthropic API in Outlines. Install all optional dependencies of the `Anthropic` model with: `pip install "outlines[anthropic]"`.

    You also need to have an Anthropic API key. This API key must either be set as an environment variable called `ANTHROPIC_API_KEY` or be provided to the `anthropic.Anthropic` class when instantiating it.

## Model Initialization

To create an Anthropic model instance, you can use the `from_anthropic` function. It takes 2 arguments:

- `client`: an `anthropic.Anthropic` instance
- `model_name`: the name of the model you want to use in subsequent model calls (optional)

For instance:

```python
from anthropic import Anthropic
import outlines

# Create the Anthropic client
client = Anthropic()

# Create the model
model = outlines.from_anthropic(
    client,
    "claude-3-5-sonnet-latest"
)
```

Check the [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models) for an up-to-date list of available models.

## Text Generation

Once you've created your Outlines `Anthropic` model instance, you're all set to generate text with this provider. You can simply call the model with a text prompt.

For instance:

```python
from anthropic import Anthropic
import outlines

# Create the model
model = outlines.from_anthropic(
    Anthropic(),
    "claude-3-5-sonnet-latest"
)

# Call it to generate text
response = model("What's the capital of Latvia?", max_tokens=20)
print(response) # 'Riga'
```

#### Vision

Some Anthropic models support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

For instance:

```python
import io
import requests
import PIL
from anthropic import Anthropic
from outlines import from_anthropic
from outlines.inputs import Image

# Create the model
model = from_anthropic(
    Anthropic(),
    "claude-3-5-sonnet-latest"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Chat

You can also use chat inputs with the `Anthropic` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
from anthropic import Anthropic
from outlines import from_anthropic
from outlines.inputs import Chat, Image

# Create the model
model = from_anthropic(
    Anthropic(),
    "claude-3-5-sonnet-latest"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "user", "content": "You are a helpful assistant that helps me described pictures."},
    {"role": "assistant", "content": "I'd be happy to help you describe pictures! Please go ahead and share an image"},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `Anthropic` model supports streaming through the `stream` method.

For instance:

```python
from anthropic import Anthropic
import outlines

# Create the model
model = outlines.from_anthropic(
    Anthropic(),
    "claude-3-5-sonnet-latest"
)

# Stream the response
for chunk in model.stream("Tell me a short story about a cat.", max_tokens=50):
    print(chunk) # 'Once...'
```

## Inference arguments

When calling the model or streaming, you can provide keyword arguments that will be passed down to the Anthropic client. Make sure to include all the arguments you need to configure the client's behavior to your expected behavior. Some of the most common arguments include `max_tokens`, `temperature`, `stop_sequences` and `top_k`.

See the [Anthropic API documentation](https://docs.anthropic.com/en/api/messages) for the full list of available arguments.

!!! Warning

    You must set a value for `max_tokens` with Anthropic models.

---

### Features Models Dottxt (`docs/features/models/dottxt.md`)

---
title: Dottxt
---

# Dottxt

!!! Installation

    You need to install the `dottxt` python sdk to be able to use the Dottxt API in Outlines. Install all optional dependencies of the `Dottxt` model with: `pip install "outlines[dottxt]"`.

    You also need to have a Dottxt API key. Request one [here](https://h1xbpbfsf0w.typeform.com/to/fwQNWmS8?utm_source=github&utm_medium=organic&utm_campaign=outlines). Once obtained, the key must either be set as an environment variable called `DOTTXT_API_KEY` or be provided to the `dottxt.client.DotTxt` class when instantiating it.

## Model Initialization

### Synchronous

Use `from_dottxt` to create a model instance from a `dottxt.DotTxt` client. The `model` argument is the identifier of the model to use. It can be set at initialization time or passed per call via the `model=` keyword argument — but it must be provided somewhere or a `ValueError` is raised.

```python
from dottxt.client import DotTxt
import outlines

client = DotTxt(api_key="...")
model = outlines.from_dottxt(client, "dottxt/dottxt-v1-alpha")
```

Use `client.models.list()` to get the list of available model identifiers for your account.

### Asynchronous

Pass a `dottxt.AsyncDotTxt` client to the same `from_dottxt` function — it detects the client type and returns an `AsyncDottxt` instance:

```python
from dottxt.client import AsyncDotTxt
import outlines

client = AsyncDotTxt(api_key="...")
model = outlines.from_dottxt(client, "dottxt/dottxt-v1-alpha")
```

## Text Generation

Dottxt only supports constrained generation with JSON schema output types. You must always provide a value for the `output_type` parameter as unconstrained generation is not available.

### Synchronous

```python
from typing import List
from pydantic import BaseModel
from dottxt.client import DotTxt
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

model = outlines.from_dottxt(DotTxt(), "dottxt/dottxt-v1-alpha")

result = model("Create a character", Character)
print(result)  # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result))  # name=Evelyn, age=34, skills=[...]
```

### Asynchronous

```python
from typing import List
from pydantic import BaseModel
from dottxt.client import AsyncDotTxt
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

model = outlines.from_dottxt(AsyncDotTxt(), "dottxt/dottxt-v1-alpha")

result = await model("Create a character", Character)
print(result)  # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
```

## Inference arguments

Optional parameters supported by the `dottxt` SDK's `generate` method can be passed at generation time as keyword arguments: `temperature`, `max_tokens`, `seed`, or any additional OpenAI-compatible chat completion parameter.

```python
result = model("Create a character", Character, temperature=0.8, max_tokens=256)
```

The `model` identifier can also be overridden per call this way:

```python
result = model("Create a character", Character, model="dottxt/other-model")
```

Consult the [dottxt python sdk GitHub repository](https://github.com/dottxt-ai/dottxt-python) for the full list of parameters.

---

### Features Models Gemini (`docs/features/models/gemini.md`)

# Gemini

!!! Installation

    You need to install the `google.genai` libray to be able to use the Gemini API in Outlines. Install all optional dependencies of the `Gemini` model with: `pip install "outlines[gemini]"`.

    You also need to have a Gemini API key. This API key must either be set as an environment variable called `GEMINI_API_KEY` or be provided to the `google.genai.Client` class when instantiating it.

## Model Initialization

To create a Gemini model instance, you can use the `from_gemini` function. It takes 2 arguments:

- `client`: a `google.genai.Client` instance
- `model_name`: the name of the model you want to use in subsequent model calls (optional)

For instance:

```python
import outlines
from google import genai

# Create the client
client = genai.Client()

# Create the model
model = outlines.from_gemini(
    client,
    "gemini-1.5-flash-latest"
)
```

Check the [Gemini documentation](https://github.com/googleapis/python-genai) for an up-to-date list of available models.

## Text Generation

Once you've created your Outlines `Gemini` model instance, you're all set to generate text with this provider. You can simply call the model with a prompt.

For instance:

```python
import outlines
from google.genai import Client

# Create the model
model = outlines.from_gemini(
    Client(),
    "gemini-1.5-flash-latest"
)

# Call it to generate text
result = model("What's the capital of Latvia?", max_output_tokens=20)
print(result) # 'Riga'
```

#### Vision

Some Gemini models support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

For instance:

```python
import io
import requests
import PIL
import outlines
from google.genai import Client
from outlines.inputs import Image

# Create the model
model = outlines.from_gemini(
    Client(),
    "gemini-1.5-flash-latest"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_output_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Chat

You can also use chat inputs with the `Gemini` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import outlines
from google.genai import Client
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_gemini(
    Client(),
    "gemini-1.5-flash-latest"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "user", "content": "You are a helpful assistant that helps me described pictures."},
    {"role": "assistant", "content": "I'd be happy to help you describe pictures! Please go ahead and share an image"},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt, max_output_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `Gemini` model supports streaming through the `stream` method.

For instance:

```python
import outlines
from google.genai import Client

# Create the model
model = outlines.from_gemini(
    Client(),
    "gemini-1.5-flash-latest"
)

# Stream text
for chunk in model.stream("Write a short story about a cat.", max_output_tokens=20):
    print(chunk) # 'In...'
```

## Structured Generation

Gemini provides supports for some forms of structured output: multiple choice, JSON schema (with caveats) and lists of structured objects. To use it, call the model with an `output_type` on top of your prompt.

#### Multiple Choice

```python
import outlines
from google import genai
from enum import Enum

class PizzaOrBurger(Enum):
    pizza = "pizza"
    burger = "burger"

# Create the model
model = outlines.from_gemini(genai.Client(), "gemini-1.5-flash-latest")

# Call it with the ouput type to generate structured text
result = model("Pizza or burger?", PizzaOrBurger, max_output_tokens=20)
print(result) # 'pizza'
```

#### JSON Schema

Gemini supports only three types of objects used to define a JSON Schema:

- Pydantic classes
- Dataclasses
- TypedDicts

```python
from typing import List
from pydantic import BaseModel
from google import genai
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_gemini(genai.Client(), "gemini-1.5-flash-latest")

# Call it with the ouput type to generate structured text
result = model("Create a character", Character)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

#### Lists of Structured Objects

A specificity of Gemini is that, despite not supporting regex, it does support a list of structured objects as an output type. To use it, put any of three available types described above in the typing `List` class

```python
from dataclasses import dataclass
from google import genai
import outlines

@dataclass
class Character:
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_gemini(genai.Client(), "gemini-1.5-flash-latest")

# Call it with the ouput type to generate structured text
result = model("Create a character", list[Character])
print(result) # '[{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}, {["name":...'
```

!!! Attention

    The structured objects must be in a built-in `list`, not a `List` from the `typing` library

## Inference arguments

You can provide the same optional parameters you would pass to the `google.genai.Client` client both during the initialization of the Gemini model and when generating text. Some of the common inference arguments include `max_output_tokens`, `temperature`, and other generation parameters.

Consult the [Google Generative AI documentation](https://github.com/googleapis/python-genai) for the full list of parameters.

---

### Features Models Llamacpp (`docs/features/models/llamacpp.md`)

---
title: llama.cpp
---

# llama.cpp

Outlines provides an integration with [Llama.cpp](https://github.com/ggerganov/llama.cpp) using the [llama-cpp-python library](https://github.com/abetlen/llama-cpp-python). Llamacpp allows to run quantized models on machines with limited compute.

!!! Installation

    You need to install the `llama-cpp-python` library to use the llama.cpp integration. Install all optional dependencies of the `LlamaCpp` model with: `"pip install "outlines[llamacpp]"`.

    See the [llama-cpp-python Github page](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#supported-backends) for instructions on installing with CUDA, Metal, ROCm and other backends.


## Model Initialization

To load the model, you can use the `from_llamacpp` function. The first argument of the function is a `Llama` model instance from the `llama_cpp` library. Consult the [Llama class API reference](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama) for detailed information on how to create a model instance and on the various available parameters.

You can also pass a `chat_mode` argument to `from_llamacpp`. If `True` (default), the model will regard all `str` inputs as user messages in a chat conversation. If `False`, the model will regard all `str` inputs as plain text prompts.

For instance:

```python
import outlines
from llama_cpp import Llama

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)
```

You can also disable chat mode:

```python
import outlines
from llama_cpp import Llama

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    ),
    chat_mode=False,
)
```

## Text Generation

To generate text, you can simply call the model with a prompt.

For instance:

```python
import outlines
from llama_cpp import Llama

# Create the model
model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

# Call it to generate text
result = model("What's the capital of Latvia?", max_tokens=20)
print(result) # 'Riga'
```

#### Chat

You can also use chat inputs with the `LlamaCpp` model. To do so, call the model with a `Chat` instance.

For instance:

```python
import outlines
from llama_cpp import Llama
from outlines.inputs import Chat

# Create the model
model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

# Create the prompt containing the text and the image
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "assistant", "content": "What's the capital of Latvia?"},
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'Riga.'
```

#### Streaming

The `LlamaCpp` model also supports streaming.

For instance:

```python
import outlines
from llama_cpp import Llama

# Create the model
model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

# Stream text
for chunk in model.stream("Write a short story about a cat.", max_tokens=100):
    print(chunk) # 'In...'
```

## Structured Generation

The `LlamaCpp` model supports all output types available in Outlines. Simply provide an `output_type` after the prompt when calling the model.

### Basic Type

```python
import outlines
from llama_cpp import Llama

output_type = int

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

result = model("How many countries are there in the world?", output_type)
print(result) # '200'
```

### JSON Schema

```python
from typing import List
from pydantic import BaseModel
import outlines
from llama_cpp import Llama

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

result = model("Create a character.", output_type=Character, max_tokens=200)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

### Multiple Choice

```python
from typing import Literal
import outlines
from llama_cpp import Llama

output_type = Literal["Paris", "London", "Rome", "Berlin"]

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

result = model("What is the capital of France?", output_type)
print(result) # 'Paris'
```

### Regex

```python
from outlines.types import Regex
import outlines
from llama_cpp import Llama

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

result = model("Generate a fake social security number.", output_type)
print(result) # '782-32-3789'
```

### Context-free grammar

```python
from outlines.types import CFG
import outlines
from llama_cpp import Llama

output_type = CFG("""
root ::= answer
answer ::= "yes" | "no"
""")

model = outlines.from_llamacpp(
    Llama.from_pretrained(
        repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
        filename="mistral-7b-instruct-v0.2.Q5_K_M.gguf",
    )
)

result = model("Are you feeling good today?", output_type)
print(result) # 'yes'
```

## Inference Arguments

When calling the model, you can provide optional inference parameters on top of the prompt and the output type. These parameters will be passed on to the `__call__` method of the `llama_cpp.Llama` model. Some common inference arguments include `max_tokens`, `temperature`, `frequency_penalty` and `top_p`.

See the [llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__call__) for more information on inference parameters.

---

### Features Models Lmstudio (`docs/features/models/lmstudio.md`)

---
title: LM Studio
---

# LM Studio

!!! Installation

    To be able to use LM Studio in Outlines, you must install both LM Studio and the optional dependency libraries of the model.

    - To download LM Studio: https://lmstudio.ai/download
    - To install the lmstudio python sdk: `pip install "outlines[lmstudio]"`

    Consult the [`lmstudio-python` documentation](https://lmstudio.ai/docs/python) for detailed information on installation and client initialization.

## Model Initialization

To create an LM Studio model instance, you can use the `from_lmstudio` function. It takes 2 arguments:

- `client`: an `lmstudio.Client` or `lmstudio.AsyncClient` instance
- `model_name`: the name of the model you want to use (optional; if not provided, the default model loaded in LM Studio is used)

Based on whether the inference client instance is synchronous or asynchronous, you will receive an `LMStudio` or an `AsyncLMStudio` model instance.

For instance:

```python
import lmstudio
import outlines

# Create the client or async client
client = lmstudio.Client()
async_client = lmstudio.AsyncClient()

# Create a sync model
model = outlines.from_lmstudio(
    client,
    "openai/gpt-oss-20b",
)

# Create an async model
model = outlines.from_lmstudio(
    async_client,
    "openai/gpt-oss-20b",
)
```

You can find the list of available models in the [LM Studio model catalog](https://lmstudio.ai/models). The model must be downloaded in LM Studio (through the app or with `lms get <model_name>`) before you can use it.

## Text Generation

Once you've created your Outlines `LMStudio` model instance, you're all set to generate text with this provider. You can simply call the model with a prompt.

For instance:

```python
import lmstudio
import outlines

# Create the model
model = outlines.from_lmstudio(lmstudio.Client(), "openai/gpt-oss-20b")

# Call it to generate text
response = model("What's the capital of Latvia?")
print(response) # 'Riga'
```

#### Vision

Some LM Studio models support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

```python
import io
import requests
import PIL
import lmstudio
import outlines
from outlines.inputs import Image

# Create the model
model = outlines.from_lmstudio(
    lmstudio.Client(),
    "qwen/qwen2.5-vl-7b"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Generate text
response = model(prompt)
print(response) # The image shows a black puppy with a curious and attentive expression.
```

#### Chat

You can also use chat inputs with the `LMStudio` model. To do so, call the model with a `Chat` instance. The content of messages within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import lmstudio
import outlines
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_lmstudio(
    lmstudio.Client(),
    "qwen/qwen2.5-vl-7b"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `LMStudio` model supports streaming through the `stream` method.

```python
import lmstudio
import outlines

# Create the model
model = outlines.from_lmstudio(lmstudio.Client(), "openai/gpt-oss-20b")

# Stream text
for chunk in model.stream("Write a short story about a cat"):
    print(chunk) # 'In...'
```

## Asynchronous Calls

LM Studio supports asynchronous operations by passing an `AsyncClient` instead of a regular `Client`. This returns an `AsyncLMStudio` model instance that supports async/await patterns.

### Basic Async Generation

```python
import asyncio
import outlines
import lmstudio

async def generate_text():
    # Create an async model
    async_client = lmstudio.AsyncClient()
    async_model = outlines.from_lmstudio(async_client, "openai/gpt-oss-20b")

    result = await async_model("Write a haiku about Python.")
    print(result)

    # Release the client's resources once you're done
    await async_model.close()

asyncio.run(generate_text())
```

The `AsyncLMStudio` model manages the lifecycle of its underlying async client: the client session is entered on the first call to the model. Once you're done using the model, call its `close` method to release the resources.

### Async Streaming

The async model also supports streaming with async iteration:

```python
import asyncio
import outlines
import lmstudio

async def stream_text():
    async_client = lmstudio.AsyncClient()
    async_model = outlines.from_lmstudio(async_client, "openai/gpt-oss-20b")

    async for chunk in async_model.stream("Tell me a story about a robot."):
        print(chunk, end="")

    await async_model.close()

asyncio.run(stream_text())
```

### Concurrent Async Requests

One of the main benefits of async calls is the ability to make multiple concurrent requests:

```python
import asyncio
import outlines
import lmstudio

async def generate_multiple():
    async_client = lmstudio.AsyncClient()
    async_model = outlines.from_lmstudio(async_client, "openai/gpt-oss-20b")

    # Define multiple prompts
    prompts = [
        "Write a tagline for a coffee shop.",
        "Write a tagline for a bookstore.",
        "Write a tagline for a gym."
    ]

    tasks = [async_model(prompt) for prompt in prompts]
    results = await asyncio.gather(*tasks)

    for prompt, result in zip(prompts, results):
        print(f"{prompt}\n{result}\n")

    await async_model.close()

asyncio.run(generate_multiple())
```

## Structured Generation

LM Studio only provides support for structured generation based on a JSON schema. To use it, call the model with a JSON schema object as an `output_type` on top of your prompt.

For instance:

```python
from typing import List
from pydantic import BaseModel
import lmstudio
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_lmstudio(lmstudio.Client(), "openai/gpt-oss-20b")

# Call it with the output type to generate structured text
result = model("Create a character", Character)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

## Inference arguments

When calling the model, you can provide keyword arguments that will be passed down to the `respond` method of the LM Studio model handle. For instance, you can control sampling with the `config` argument:

```python
import lmstudio
import outlines

model = outlines.from_lmstudio(lmstudio.Client(), "openai/gpt-oss-20b")

response = model(
    "What's the capital of Latvia?",
    config={"temperature": 0.5, "maxTokens": 100},
)
```

Consult the [LM Studio prediction parameters documentation](https://lmstudio.ai/docs/python/llm-prediction/parameters) for the full list of inference parameters.

---

### Features Models Mistral (`docs/features/models/mistral.md`)

# Mistral

!!! Installation

    You need to install the `mistralai` library to be able to use the Mistral API in Outlines. Install all optional dependencies of the `Mistral` model with: `pip install "outlines[mistral]"`.

    You also need to have an Mistral API key. This API key must either be set as an environment variable called `MISTRAL_API_KEY` or be provided to the `mistralai.Mistral` class when instantiating it.

## Model Initialization

To create an `Mistral` or `AsyncMistral` model instance, you can use the `from_mistral` function. It takes 3 arguments:

- `client`: a `mistralai.Mistral` instance
- `model_name` (optional): the name of the model you want to use
- `async_client` (optional): whether it should create a sync or an async model

As the `mistralai` library uses a single class to handle both sync and async requests, you must set the `async_client` argument to True to get an `AsyncMistral` model.

For instance:

```python
import mistralai
import outlines

# Create the Mistral client
client = mistral.Mistral()

# Create a sync model
model = outlines.from_mistral(
    client,
    "mistral-large-latest"
)

# Create aa async model
model = outlines.from_mistral(
    client,
    "mistral-large-latest",
    True
)
```

The mistralai python SDK provides methods to query the API for a list of [all available models](https://docs.mistral.ai/getting-started/models/models_overview/#api-versioning),
including paid endpoints for [premium models](https://docs.mistral.ai/getting-started/models/models_overview/) in addition to open weights.

## Text Generation

Once you've created your Outlines `Mistral` model instance, you're all set to generate text with this provider. You can simply call the model with a prompt.

For instance:

```python
import mistralai
import outlines

# Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest"
)

# Call it to generate text
response = model("What's the capital of Latvia?", max_tokens=20)
print(response) # 'Riga'
```

#### Vision

Some Mistral models support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

For instance:

```python
import io
import requests
import PIL
import outlines
import mistralai
from outlines.inputs import Image

# Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Chat

You can also use chat inputs with the `Mistral` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import mistralai
import outlines
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `Mistral` model supports streaming through the `stream` method.

For instance:

```python
import mistralai
import outlines

# Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest"
)

# Stream the response
for chunk in model.stream("Tell me a short story about a cat.", max_tokens=50):
    print(chunk) # 'Once...'
```

## Structured Generation

Mistral provides supports for some forms of structured output: JSON schemas and JSON syntax. To use it, call the model with an `output_type` on top of your prompt.

#### JSON Schema

```python
from typing import List
from pydantic import BaseModel
import mistralai
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest"
)

# Call it with the output type to generate structured text
result = model("Create a character, use the json format.", Character, top_p=0.1)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

#### JSON Syntax

What we mean by JSON syntax is what is sometimes called JSON mode, meaning that the model will return a valid JSON, but you do not get to specify its structure. To use this JSON mode, provide the `dict` type as an output type.

```python
import mistralai
import outlines

## Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest"
)


# Call it with the output type to generate structured text
result = model("Create a character, use the json format.", dict, temperature=0.5)
print(result) # '{"first_name": "Henri", "last_name": "Smith", "height": "170"}'
```

## Asynchronous Calls

All features presented above for the sync model are also available for the async model.

For instance:

```python
import asyncio
import mistralai
import outlines
from pydantic import BaseModel
from typing import List

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_mistral(
    mistralai.Mistral(),
    "mistral-large-latest",
    True
)

async def text_generation():
    # Regular generation
    response = await model("What's the capital of Latvia?", max_tokens=20)
    print(response) # 'Riga'

    # Streaming
    async for chunk in  model.stream("Tell me a short story about a cat.", max_tokens=50):
        print(chunk, end="") # 'Once...'

    # Structured generation
    result = await model("Create a character, use the json format.", Character, top_p=0.1)
    print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
    print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']

asyncio.run(text_generation())
```

## Inference arguments

When calling the model, you can provide keyword arguments that will be passed down to the `chat.complete` method of the Mistral client and its async and streaming equivalents. Some of the most common arguments include `max_tokens`, `temperature`, `stop` and `top_p`.

Another keyword argument of interest is `n`. If set with an integer value superior to 1, Mistral will generate several sample responses and you will receive a list of strings as a response to your model call.

See the [Mistral API documentation](https://docs.mistral.ai/api/#tag/chat) for the full list of available arguments.


## Troubleshooting

- **ImportError: No module named 'mistralai'**
  → Run `pip install mistralai`.

- **Authentication Error**
  → Verify `MISTRAL_API_KEY` is set and valid. Test with the [Mistral Playground](https://chat.mistral.ai).

- **Schema Error (e.g., "Mistral does not support your schema")**
  → Ensure no `pattern` fields in Pydantic (Outlines sets `additionalProperties: false`); try a simpler schema or a different Outlines model (local models in particular).

- **Model Not Found Error**
  → Confirm the model name (e.g., `"mistral-small-latest"`) and your subscription tier. Check [docs](https://docs.mistral.ai/getting-started/models/).

- **Rate Limits or Quotas**
  → Monitor usage in the Mistral console; upgrade your plan for higher limits.

- **Input Validation Errors**
  → Ensure Chat messages use valid roles (`system`, `user`, `assistant`); list inputs start with strings.


*Last updated: October 2, 2025*

---

### Features Models Mlxlm (`docs/features/models/mlxlm.md`)

---
title: mlx-lm
---

# mlx-lm

Outlines provides an integration with [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms), allowing models to be run quickly on Apple Silicon via the [mlx](https://ml-explore.github.io/mlx/build/html/index.html) library.

!!! Note "Installation"

    You need a device that [supports Metal](https://support.apple.com/en-us/102894) to use the mlx-lm integration.

    You need to install the `mlx` and `mlx-lm` libraries to be able to use mlx in Outlines. Install all optional dependencies of the `MLXLM` model with: `pip install "outlines[mlxlm]"`.

## Model Initialization

To create a MLXLM model instance, you can use the `from_mlxlm` function. It takes 2 arguments:

- `model`: an `mlx.nn.Module` instance
- `tokenizer`: a `transformers.PreTrainedTokenizer` instance

However, we recommend you simply pass on the output of the `mlx_lm.load` function (it takes a model name as an argument).

For instance:

```python
import outlines
import mlx_lm

# Create the model
model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)
```

## Text Generation

To generate text, you can simply call the model with a prompt.

For instance:

```python
import outlines
import mlx_lm

# Load the model
model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

# Call it to generate text
result = model("What's the capital of Latvia?", max_tokens=20)
print(result) # 'Riga'
```

#### Chat

You can use chat inputs with the `MLXLM` model. To do so, call the model with a `Chat` instance.

For instance:

```python
import outlines
import mlx_lm
from outlines.inputs import Chat

# Load the model
model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

# Create the prompt containing the text and the image
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "assistant", "content": "What's the capital of Latvia?"},
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'Riga.'
```

#### Streaming

The `MLXLM` model also supports streaming. For instance:

```python
import outlines
import mlx_lm

# Load the model
model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

# Stream text
for chunk in model.stream("Write a short story about a cat.", max_tokens=100):
    print(chunk) # 'In...'
```

#### Batch Generation

The `MLXLM` model supports generating text in batches. To do so, use the `batch` method and provide a list of strings as a model input. However, constrained generation is not supported with batching, so you cannot provide an `output_type`. For instance:

```python
import outlines
import mlx_lm

# Load the model
model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

# Generate text in batches
result = model.batch(["What's the capital of Lithuania?", "What's the capital of Latvia?"], max_tokens=20)
print(result) # ['Vilnius', 'Riga']
```

## Structured Generation

As a local model, `MLXLM` supports all forms of structured generation available in Outlines.

#### Basic Type

```python
import outlines
import mlx_lm

output_type = int

model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

result = model("How many countries are there in the world?", output_type)
print(result) # '200'
```

#### JSON Schema

```python
from pydantic import BaseModel
from typing import List
import outlines
import mlx_lm

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

result = model("Create a character.", output_type=Character)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

#### Multiple Choice

```python
from typing import Literal
import outlines
import mlx_lm

output_type = Literal["Paris", "London", "Rome", "Berlin"]

model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

result = model("What is the capital of France?", output_type)
print(result) # 'Paris'
```

#### Regex

```python
from outlines.types import Regex
import outlines
import mlx_lm

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

result = model("Generate a fake social security number.", output_type)
print(result) # '782-32-3789'
```

#### Context-Free Grammar

```python
from outlines.types import CFG
import outlines
import mlx_lm

arithmetic_grammar = """
?start: sum

?sum: product
| sum "+" product   -> add
| sum "-" product   -> sub

?product: atom
| product "*" atom  -> mul
| product "/" atom  -> div

?atom: NUMBER           -> number
| "-" atom         -> neg
| "(" sum ")"

%import common.NUMBER
%import common.WS_INLINE

%ignore WS_INLINE
"""
output_type = CFG(arithmetic_grammar)

model = outlines.from_mlxlm(
    *mlx_lm.load("mlx-community/TinyLlama-1.1B-Chat-v1.0-4bit")
)

result = model("Write an addition.", output_type, max_tokens=20)
print(result) # '23 + 48'
```

## Inference Arguments

When calling the model, you can provide optional inference parameters on top of the prompt and the output type. These parameters will be passed on to the `mlx_lm.generate` function used to generate text.

See the [MLXLM documentation](https://github.com/ml-explore/mlx-lm) for more information on inference parameters.

---

### Features Models Ollama (`docs/features/models/ollama.md`)

---
title: Ollama
---

# Ollama

!!! Installation

    To be able to use Ollama in Outlines, you must install both Ollama and the optional dependency libraries of the model.

    - To download Ollama: https://ollama.com/download
    - To install the ollama python sdk: `pip install "outlines[ollama]"`

    Consult the [`ollama` documentation](https://github.com/ollama/ollama-python) for detailed information on installation and client initialization.

## Model Initialization

To create an Ollama model instance, you can use the `from_ollama` function. It takes 2 arguments:

- `client`: an `ollama.Client` or `ollama.AsyncClient` instance
- `model_name`: the name of the model you want to use

Based on whether the inference client instance is synchronous or asynchronous, you will receive an `Ollama` or an `AsyncOllama` model instance.

For instance:

```python
import ollama
import outlines

# Create the client or async client
client = ollama.Client()
async_client = ollama.AsyncClient()

# Create a sync model
model = outlines.from_ollama(
    client,
    "qwen2.5vl:3b",
)

# Create an async model
model = outlines.from_ollama(
    async_client,
    "qwen2.5vl:3b",
)
```

You can find the list of available models on the [Ollama library](https://ollama.com/library).

## Text Generation

Once you've created your Outlines `Ollama` model instance, you're all set to generate text with this provider. You can simply call the model with a prompt.

For instance:

```python
import ollama
import outlines

# Create the model
model = outlines.from_ollama(ollama.Client(), "qwen2.5vl:3b")

# Call it to generate text
response = model("What's the capital of Latvia?")
print(response) # 'Riga'
```

#### Vision

Some Ollama models support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

```python
import io
import requests
import PIL
import ollama
import outlines
from outlines.inputs import Image

# Create the model
model = outlines.from_ollama(
    ollama.Client(),
    "qwen2.5vl:3b"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Generate text
response = model(prompt)
print(response) # The image shows a black puppy with a curious and attentive expression.
```

#### Chat

You can also use chat inputs with the `Ollama` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import ollama
import outlines
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_ollama(
    ollama.Client(),
    "qwen2.5vl:3b"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `Anthropic` model supports streaming through the `stream` method.

```python
import ollama
import outlines

# Create the model
model = outlines.from_ollama(ollama.Client(), "qwen2.5vl:3b")

# Stream text
for chunk in model.stream("Write a short story about a cat"):
    print(chunk) # 'In...'
```

## Asynchronous Calls

Ollama supports asynchronous operations by passing an `AsyncClient` instead of a regular `Client`. This returns an `AsyncOllama` model instance that supports async/await patterns.

### Basic Async Generation

```python
import asyncio
import outlines
import ollama

async def generate_text():
    # Create an async model
    async_client = ollama.AsyncClient()
    async_model = outlines.from_ollama(async_client, "qwen2.5vl:3b")

    result = await async_model("Write a haiku about Python.")
    print(result)

asyncio.run(generate_text())
```

### Async Streaming

The async model also supports streaming with async iteration:

```python
import asyncio
import outlines
import ollama

async def stream_text():
    async_client = ollama.AsyncClient()
    async_model = outlines.from_ollama(async_client, "qwen2.5vl:3b")

    async for chunk in async_model.stream("Tell me a story about a robot."):
        print(chunk, end="")

asyncio.run(stream_text())
```

### Concurrent Async Requests

One of the main benefits of async calls is the ability to make multiple concurrent requests:

```python
import asyncio
import outlines
import ollama

async def generate_multiple():
    async_client = ollama.AsyncClient()
    async_model = outlines.from_ollama(async_client, "qwen2.5vl:3b")

    # Define multiple prompts
    prompts = [
        "Write a tagline for a coffee shop.",
        "Write a tagline for a bookstore.",
        "Write a tagline for a gym."
    ]

    tasks = [async_model(prompt) for prompt in prompts]
    results = await asyncio.gather(*tasks)

    for prompt, result in zip(prompts, results):
        print(f"{prompt}\n{result}\n")

asyncio.run(generate_multiple())
```

## Structured Generation

Ollama only provides support for structured generation based on a JSON schema. To use it, call the model with a JSON schema object as an `output_type` on top of your prompt.

For instance:

```python
from typing import List
from pydantic import BaseModel
import ollama
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_ollama(ollama.Client(), "tinyllama")

# Call it with the output type to generate structured text
result = model("Create a character", Character)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

## Inference arguments

When calling the model, you can provide keyword arguments that will be passed down to the `generate` method of the Ollama client.

Consult the [Ollama REST API documentation](https://github.com/ollama/ollama/blob/main/docs/api#generate-a-completion) for the full list of inference parameters.

---

### Features Models Openai (`docs/features/models/openai.md`)

# OpenAI

!!! Installation

    You need to install the `openai` library to be able to use the OpenAI API in Outlines. Install all optional dependencies of the `OpenAI` model with: `pip install "outlines[openai]"`.

    You also need to have an OpenAI API key. This API key must either be set as an environment variable called `OPENAI_API_KEY` or be provided to the `openai.OpenAI` class when instantiating it.

## Model Initialization

To create an OpenAI model instance, you can use the `from_openai` function. It takes 2 arguments:

- `client`: an `openai.OpenAI`, `openai.AzureOpenAI`, `openai.AsyncOpenAI` or `openai.AsyncAzureOpenAI` instance
- `model_name`: the name of the model you want to use

Based on whether the inference client instance is synchronous or asynchronous, you will receive an `OpenAI` or an `AsyncOpenAI` model instance.

For instance:

```python
import outlines
import openai

# Create the client or async client
client = openai.OpenAI()
async_client = openai.AsyncOpenAI()

# Create a sync model
model = outlines.from_openai(
    client,
    "gpt-4o"
)

# Create aa async model
model = outlines.from_openai(
    async_client,
    "gpt-4o"
)
```

Check the [OpenAI documentation](https://platform.openai.com/docs/models) for an up-to-date list of available models. As shown above, you can use Azure OpenAI in Outlines the same way you would use OpenAI, just provide an `openai.AzureOpenAI` instance to the Outlines model class.

## Text Generation

Once you've created your Outlines `OpenAI` model instance, you're all set to generate text with this provider. You can simply call the model with a prompt.

For instance:

```python
import openai
import outlines

# Create the model
model = outlines.from_openai(
    openai.OpenAI(),
    "gpt-4o"
)

# Call it to generate text
response = model("What's the capital of Latvia?", max_tokens=20)
print(response) # 'Riga'
```

#### Vision

Some OpenAI models support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

For instance:

```python
import io
import requests
import PIL
import outlines
import openai
from outlines.inputs import Image

# Create the model
model = outlines.from_openai(
    openai.OpenAI(),
    "gpt-4o"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Chat

You can also use chat inputs with the `OpenAI` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import openai
import outlines
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_openai(
    openai.OpenAI(),
    "gpt-4o"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `OpenAI` model supports streaming through the `stream` method.

For instance:

```python
import openai
import outlines

# Create the model
model = outlines.from_openai(
    openai.OpenAI(),
    "gpt-4o"
)

# Stream the response
for chunk in model.stream("Tell me a short story about a cat.", max_tokens=50):
    print(chunk) # 'Once...'
```

## Structured Generation

OpenAI provides supports for some forms of structured output: JSON schemas and JSON syntax. To use it, call the model with an `output_type` on top of your prompt.

#### JSON Schema

```python
from typing import List
from pydantic import BaseModel
import openai
import outlines

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_openai(openai.OpenAI(), "gpt-4o")

# Call it with the output type to generate structured text
result = model("Create a character, use the json format.", Character, top_p=0.1)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

#### JSON Syntax

What we mean by JSON syntax is what is sometimes called JSON mode, meaning that the model will return a valid JSON, but you do not get to specify its structure. To use this JSON mode, provide the `dict` type as an output type.

```python
import openai
import outlines

# Create the model
model = outlines.from_openai(openai.OpenAI(), "gpt-4o")

# Call it with the output type to generate structured text
result = model("Create a character, use the json format.", dict, temperature=0.5)
print(result) # '{"first_name": "Henri", "last_name": "Smith", "height": "170"}'
```

## Asynchronous Calls

All features presented above for the sync model are also available for the async model.

For instance:

```python
import asyncio
import openai
import outlines
from pydantic import BaseModel
from typing import List

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

# Create the model
model = outlines.from_openai(
    openai.AsyncOpenAI(),
    "gpt-4o"
)

async def text_generation():
    # Regular generation
    response = await model("What's the capital of Latvia?", max_tokens=20)
    print(response) # 'Riga'

    # Streaming
    async for chunk in  model.stream("Tell me a short story about a cat.", max_tokens=50):
        print(chunk, end="") # 'Once...'

    # Structured generation
    result = await model("Create a character, use the json format.", Character, top_p=0.1)
    print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
    print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']

asyncio.run(text_generation())
```

## Inference arguments

When calling the model, you can provide keyword arguments that will be passed down to the `chat.completions.create` method of the OpenAI client. Some of the most common arguments include `max_tokens`, `temperature`, `stop` and `top_p`.

Another keyword argument of interest is `n`. If set with an integer value superior to 1, OpenAI will generate several sample responses and you will receive a list of strings as a response to your model call.

See the [OpenAI API documentation](http://platform.openai.com/docs/api-reference/chat/create) for the full list of available arguments.

---

### Features Models Openai Compatible (`docs/features/models/openai_compatible.md`)

# OpenAI-Compatible APIs

Many inference providers offer OpenAI-compatible APIs, allowing you to use the familiar OpenAI SDK while connecting to different backends. Outlines allows you can leverage various providers while maintaining consistent code.

## What are OpenAI-Compatible APIs?

OpenAI-compatible APIs implement the same REST endpoints and request/response formats as OpenAI's API, but serve different models or run on different infrastructure. This allows you to use the `openai` Python library with any compatible provider by simply changing the `base_url`.

!!! Installation

    You need to install the `openai` library to be able to use the OpenAI-compatible APIs in Outlines. Install all optional dependencies of the `OpenAI` model with: `pip install "outlines[openai]"`.

## General Usage Pattern

The standard approach is to use the OpenAI SDK with a custom base URL:

```python
import openai
import outlines

# Point to your OpenAI-compatible endpoint
client = openai.OpenAI(
    base_url="https://your-provider.com/v1",  # Custom endpoint
    api_key="your-api-key"
)

# Use with Outlines
model = outlines.from_openai(client, "model-name")
```

## Important: Provider-Specific Parameters

!!! Warning "API-Specific Parameters"

    Some providers require additional parameters in the API request for structured generation to work properly. These are typically passed as extra arguments when calling the model.

    For example, some providers may need special parameters in the request body to enable guided generation or specify constraints. Always consult your provider's documentation for structured generation requirements.

## Popular OpenAI-Compatible Providers

Many providers offer OpenAI-compatible endpoints:

- **Groq**
- **Together AI**
- **Anyscale**
- **Fireworks AI**
- **Perplexity**
- **Local servers** (LocalAI, etc.)

## Configuration Examples

### Basic Setup
```python
import openai
import outlines

# Generic OpenAI-compatible setup
client = openai.OpenAI(
    base_url="https://api.your-provider.com/v1",
    api_key="your-api-key"
)

model = outlines.from_openai(client, "provider-model-name")
```

### With Authentication Headers
```python
import openai
import outlines

# Some providers need custom headers
client = openai.OpenAI(
    base_url="https://api.your-provider.com/v1",
    api_key="your-api-key",
    default_headers={"Custom-Header": "value"}
)

model = outlines.from_openai(client, "provider-model-name")
```

## Related Documentation

For specific implementations that use OpenAI-compatible APIs:

- [SGLang](sglang.md): Local inference server with OpenAI-compatible endpoints
- [vLLM](vllm.md): High-performance inference with OpenAI-compatible API
- [OpenAI](openai.md): The original OpenAI API implementation

---

### Features Models Openrouter (`docs/features/models/openrouter.md`)

# Openrouter

!!! Installation

    [OpenRouter](https://openrouter.ai/docs/api-reference/overview) uses the same API as OpenAI, so both services are [interoperable](./openai_compatible.md) using the `openai` library. Install all optional dependencies of the `OpenAI` model with: `pip install "outlines[openai]"`.

    You also need to have an Openrouter API key. This API key must either be set as an environment variable called `OPENAI_API_KEY` or be provided to the `openai.OpenAI` class when instantiating it.

## Model Initialization

To create a model instance, you can use the `from_openai` function. It takes 2 arguments:

- `client`: an `openai.OpenAI` instance
- `model_name`: the name of the model you want to use, defined as `provider/model`

For instance:

```python
import outlines
import openai

# Create the client
client = openai.OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="OPENAI_API_KEY",
)

# Create the model
model = outlines.from_openai(
    client,
    "x-ai/grok-4"
)
```

Leaving an empty string in the model name field will lead OpenRouter to use your default model defined in [settings](https://openrouter.ai/settings/preferences).

The [OpenRouter](https://openrouter.ai/models) website lists available models. Keep in mind that some models do not support `json_schema` response formats and may return a 400 error code as a result.

## Related Documentation

For specific implementations that use OpenAI-compatible APIs:

- [OpenAI](./openai.md): The original OpenAI API implementation
- [OpenAI compatible API](./openai_compatible.md): Details on how to use OpenAI-compatible APIs

---

### Features Models Orcarouter (`docs/features/models/orcarouter.md`)

# OrcaRouter

!!! Installation

    [OrcaRouter](https://www.orcarouter.ai) uses the same API as OpenAI, so both services are [interoperable](./openai_compatible.md) using the `openai` library. Install all optional dependencies of the `OpenAI` model with: `pip install "outlines[openai]"`.

    You also need to have an OrcaRouter API key. This API key must either be set as an environment variable called `OPENAI_API_KEY` or be provided to the `openai.OpenAI` class when instantiating it.

## Model Initialization

To create a model instance, you can use the `from_openai` function. It takes 2 arguments:

- `client`: an `openai.OpenAI` instance
- `model_name`: the name of the model you want to use, defined as `provider/model`

For instance:

```python
import os
import outlines
import openai

# Create the client
client = openai.OpenAI(
    base_url="https://api.orcarouter.ai/v1",
    api_key=os.environ["OPENAI_API_KEY"],
)

# Create the model
model = outlines.from_openai(
    client,
    "openai/gpt-5.5"
)
```

You can also pass `"orcarouter/auto"` as the model name to let OrcaRouter pick an upstream for each request according to the routing strategy you configured in the [routing console](https://www.orcarouter.ai/console/routing).

The [OrcaRouter](https://www.orcarouter.ai/models) website lists available models. Keep in mind that some models do not support `json_schema` response formats and may return a 400 error code as a result.

## Related Documentation

For specific implementations that use OpenAI-compatible APIs:

- [OpenAI](./openai.md): The original OpenAI API implementation
- [OpenAI compatible API](./openai_compatible.md): Details on how to use OpenAI-compatible APIs

---

### Features Models Requesty (`docs/features/models/requesty.md`)

# Requesty

!!! Installation

    [Requesty](https://docs.requesty.ai) uses the same API as OpenAI, so both services are [interoperable](./openai_compatible.md) using the `openai` library. Install all optional dependencies of the `OpenAI` model with: `pip install "outlines[openai]"`.

    You also need to have a Requesty API key. You can create one at [app.requesty.ai/api-keys](https://app.requesty.ai/api-keys). This API key must either be set as an environment variable called `OPENAI_API_KEY` or be provided to the `openai.OpenAI` class when instantiating it.

## Model Initialization

To create a model instance, you can use the `from_openai` function. It takes 2 arguments:

- `client`: an `openai.OpenAI` instance
- `model_name`: the name of the model you want to use, defined as `provider/model`

For instance:

```python
import os
import outlines
import openai

# Create the client
client = openai.OpenAI(
    base_url="https://router.requesty.ai/v1",
    api_key=os.environ["OPENAI_API_KEY"],
)

# Create the model
model = outlines.from_openai(
    client,
    "openai/gpt-4o-mini"
)
```

The [Requesty](https://app.requesty.ai/router/list) website lists available models. Keep in mind that some models do not support `json_schema` response formats and may return a 400 error code as a result.

## Related Documentation

For specific implementations that use OpenAI-compatible APIs:

- [OpenAI](./openai.md): The original OpenAI API implementation
- [OpenAI compatible API](./openai_compatible.md): Details on how to use OpenAI-compatible APIs

---

### Features Models Sglang (`docs/features/models/sglang.md`)

---
title: SGLang
---

# SGLang

## Prerequisites

The Outlines `SGLang` model is intended to be used along with an SGLang instance running on a separate server (can be local or remote). Make sure you have a SGLang server running and accessible before using the `SGLang` model. For instance by running:

```shell
pip install "sglang[all]"

python -m sglang.launch_server \
  --model-path NousResearch/Meta-Llama-3-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000
```

Follow the [Installation instructions](https://docs.sglang.ai/start/install.html) for more information on how to set up a SGLang server for your particular setup.

As the SGLang client relies on the `openai` python sdk, you need to have the `openai` package installed. Install all optional dependencies of the `SGLang` model with: `pip install "outlines[sglang]"`.

When launching your SGLang server, you can specify the backend engine to use for structured generation through the `grammar-backend` cli argument. Add `--grammar-backend outlines` to your command to use Outlines instead of the default engine.

## Model Initialization

To load the model, you can use the `from_sglang` function. The argument of the function is either an `OpenAI` or `AsyncOpenAI` instance from the `openai` library. Make sure the value of the `base_url` argument of the `OpenAI` client points to your running SGLang server. Consult the [SGLang documentation](https://docs.sglang.ai/backend/send_request.html) on using an OpenAI client with an SGLang server for more information.

Based on whether the `openai` client instance is synchronous or asynchronous, you will receive a `SGLang` or `AsyncSGLang` model instance.

For instance:

```python
import openai
import outlines

# Create the OpenAI client
sync_openai_client = openai.OpenAI(base_url="http://localhost:11434")
async_openai_client = openai.AsyncOpenAI(base_url="http://localhost:11434")

# Create a sync model
sync_model = outlines.from_sglang(sync_openai_client)
print(type(sync_model)) # <class 'outlines.models.sglang.SGLang'>

# Create an async model
async_model = outlines.from_sglang(async_openai_client)
print(type(async_model)) # <class 'outlines.models.sglang.AsyncSGLang'>
```

## Text Generation

To generate text, you can simply call the model with a prompt.

For instance:

```python
import openai
import outlines

# Create the model
model = outlines.from_openai(openai.OpenAI(base_url="http://localhost:11434"))

# Call it to generate text
response = model("What's the capital of Latvia?", max_tokens=20)
print(response) # 'Riga'
```

#### Vision

Some models you can run with SGLang support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

For instance:

```python
import io
import requests
import PIL
import outlines
import openai
from outlines.inputs import Image

# Create the model
model = outlines.from_openai(openai.OpenAI(base_url="http://localhost:11434"))

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Chat

You can also use chat inputs with the `SGLang` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import openai
import outlines
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_openai(openai.OpenAI(base_url="http://localhost:11434"))

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Streaming

Finally, the `SGLang` model supports streaming through the `stream` method.

For instance:

```python
import openai
import outlines

# Create the model
model = outlines.from_openai(openai.OpenAI(base_url="http://localhost:11434"))

# Stream the response
for chunk in model.stream("Tell me a short story about a cat.", max_tokens=50):
    print(chunk) # 'Once...'
```

## Structured Generation

SGLang supports all output types available in Outlines (context-free grammars with caveats though, see the subsection below for more details). Simply provide an `output_type` after the prompt when calling the model. All structured generation features work with both synchronous and asynchronous models.

### Simple Type

```python
import openai
import outlines

output_type = int

openai_client = openai.OpenAI(base_url="http://localhost:11434")
model = outlines.from_sglang(openai_client)

result = model("How many countries are there in the world?", output_type)
print(result) # '200'
```

### JSON Schema

```python
import openai
import outlines
from pydantic import BaseModel

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

openai_client = openai.OpenAI(base_url="http://localhost:11434")
model = outlines.from_sglang(openai_client)

result = model("Create a character.", Character, frequency_penalty=1.5)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

### Multiple Choice

```python
from typing import Literal
import openai
import outlines

output_type = Literal["Paris", "London", "Rome", "Berlin"]

openai_client = openai.OpenAI(base_url="http://localhost:11434")
model = outlines.from_sglang(openai_client)

result = model("What is the capital of France?", output_type, temperature=0)
print(result) # 'Paris'
```

### Regex

```python
import openai
import outlines
from outlines.types import Regex

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

openai_client = openai.OpenAI(base_url="http://localhost:11434")
model = outlines.from_sglang(openai_client)

result = model("Generate a fake social security number.", output_type, top_p=0.1)
print(result) # '782-32-3789'
```

### Context-Free Grammar

SGLang supports grammars, but expects an EBNF format instead of the Lark format Outlines uses. Thus, to use a context-free grammar with SGLang, provide a string using the EBNF syntax to the Outlines `CFG` object.

```python
import openai
import outlines
from outlines.types import CFG

ebnf_grammar = """
root ::= answer
answer ::= "yes" | "no"
"""
output_type = CFG(ebnf_grammar)

openai_client = openai.OpenAI(base_url="http://localhost:11434")
model = outlines.from_sglang(openai_client)

result = model("Is the weather good today?", output_type)
print(result) # 'yes'
```

### Async Structured Generation

All structured generation features work seamlessly with async models:

```python
import asyncio
import openai
import outlines
from typing import List
from pydantic import BaseModel

class User(BaseModel):
    name: str
    email: str
    age: int

async def generate_user():
    async_client = openai.AsyncOpenAI(base_url="http://localhost:11434")
    async_model = outlines.from_sglang(async_client)

    result = await async_model("Generate a random user profile.", output_type=User)
    user = User.model_validate_json(result)
    print(f"Name: {user.name}, Email: {user.email}, Age: {user.age}")

asyncio.run(generate_user())
```

## Inference Arguments

When calling the model, you can provide optional parameters on top of the prompt and the output type. Those will be passed on to the `chat.completions.create` method of the OpenAI client.

An optional parameter of particular interest is `extra_body`, which is a dictionary containing arguments that are specific to SGLang and are not part of the standard `openai` interface.

See the [SGLang documentation](https://docs.sglang.ai/backend/openai_api_completions.html) on parameters for the OpenAI-compatible server for more information on inference parameters.

---

### Features Models Tgi (`docs/features/models/tgi.md`)

---
title: TGI
---

# TGI

## Prerequisites

The Outlines `TGI` model is intended to be used along with a HuggingFace `Text Generation Inference` server (running locally or remotely). Make sure you have a TGI server running before using the `TGI` model. For instance running:

```shell
docker run \
  --gpus all \
  --shm-size 1g \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:3.3.4 \
  --model-id NousResearch/Meta-Llama-3-8B-Instruct
```

Please consult the [installation guide](https://huggingface.co/docs/text-generation-inference/en/quicktour) for more information about how to run TGI with your particular setup.
As the TGI client relies on the `huggingface_hub` python package, you need to have it installed. Install all optional dependencoes of the `TGI` model with: `pip install "outlines[tgi]"`

## Model Initialization

To load the model, you can use the `from_tgi` function. The argument of the function is either an `InferenceClient` or `AsyncInferenceClient` instance from the `huggingface_hub` library. Consult the [HuggingFace documentation](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client) for more information on their inference client.

Based on whether the inference client instance is synchronous or asynchronous, you will receive a `TGI` or an `AsyncTGI` model instance.

For instance:

```python
import outlines
import huggingface_hub

# Create the inference client
client = huggingface_hub.InferenceClient("http://localhost:11434")
async_client = huggingface_hub.AsyncInferenceClient("http://localhost:11434")

# Create a sync model
sync_model = outlines.from_tgi(client)
print(type(sync_model))  # <class 'outlines.models.tgi.TGI'>

# Create an async model
async_model = outlines.from_tgi(async_client)
print(type(async_model))  # <class 'outlines.models.tgi.AsyncTGI'>
```

## Text Generation

To generate text, you can simply call the model with a prompt.

For instance:

```python
import outlines
import huggingface_hub

# Create the model
client = huggingface_hub.InferenceClient("http://localhost:11434")
model = outlines.from_tgi(client)

# Call it to generate text
result = model("Write a short story about a cat.", stop_sequences=["."])
print(result) # 'In a quiet village where the cobblestones hummed softly beneath the morning mist...'
```

The `TGI` model supports streaming. For instance:

```python
import outlines
import huggingface_hub

# Create the model
client = huggingface_hub.InferenceClient("http://localhost:11434")
model = outlines.from_tgi(client)

# Stream text
for chunk in model.stream("Write a short story about a cat.", stop_sequences=["."]):
    print(chunk) # 'In ...'
```

## Asynchronous Calls

TGI supports asynchronous operations by passing an `AsyncInferenceClient` instead of a regular `InferenceClient`. This returns an `AsyncTGI` model instance that supports async/await patterns.

### Basic Async Generation

```python
import asyncio
import outlines
import huggingface_hub

async def generate_text():
    # Create an async model
    async_client = huggingface_hub.AsyncInferenceClient("http://localhost:11434")
    async_model = outlines.from_tgi(async_client)

    result = await async_model("Write a haiku about Python.", max_new_tokens=50)
    print(result)

asyncio.run(generate_text())
```

### Async Streaming

The async model also supports streaming with async iteration:

```python
import asyncio
import outlines
import huggingface_hub

async def stream_text():
    async_client = huggingface_hub.AsyncInferenceClient("http://localhost:11434")
    async_model = outlines.from_tgi(async_client)

    async for chunk in async_model.stream("Tell me a story about a robot.", max_new_tokens=100):
        print(chunk, end="")

asyncio.run(stream_text())
```

### Concurrent Async Requests

One of the main benefits of async calls is the ability to make multiple concurrent requests:

```python
import asyncio
import outlines
import huggingface_hub

async def generate_multiple():
    async_client = huggingface_hub.AsyncInferenceClient("http://localhost:11434")
    async_model = outlines.from_tgi(async_client)

    # Define multiple prompts
    prompts = [
        "Write a tagline for a coffee shop.",
        "Write a tagline for a bookstore.",
        "Write a tagline for a gym."
    ]

    tasks = [async_model(prompt, max_new_tokens=30) for prompt in prompts]
    results = await asyncio.gather(*tasks)

    for prompt, result in zip(prompts, results):
        print(f"{prompt}\n{result}\n")

asyncio.run(generate_multiple())
```

## Structured Generation

TGI supports all output types available in Outlines except for context-free grammars. Simply provide an `output_type` after the prompt when calling the model. All structured generation features work with both synchronous and asynchronous models.

### Simple Type

```python
import outlines
import huggingface_hub

output_type = int

tgi_client = huggingface_hub.InferenceClient("http://localhost:8080")
model = outlines.from_tgi(tgi_client)

result = model("How many countries are there in the world?", output_type)
print(result) # '200'
```### JSON Schema

```python
import outlines
import huggingface_hub
from typing import List
from pydantic import BaseModel

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

tgi_client = huggingface_hub.InferenceClient("http://localhost:8080")
model = outlines.from_tgi(tgi_client)

result = model("Create a character.", output_type=Character, frequency_penalty=1.5)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```### Multiple Choice

```python
import outlines
import huggingface_hub
from typing import Literal

output_type = Literal["Paris", "London", "Rome", "Berlin"]

tgi_client = huggingface_hub.InferenceClient("http://localhost:8080")
model = outlines.from_tgi(tgi_client)

result = model("What is the capital of France?", output_type, temperature=0)
print(result) # 'Paris'
```### Regex

```python
import outlines
import huggingface_hub
from outlines.types import Regex

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

tgi_client = huggingface_hub.InferenceClient("http://localhost:8080")
model = outlines.from_tgi(tgi_client)

result = model("Generate a fake social security number.", output_type, top_p=0.1)
print(result) # '782-32-3789'
```

### Async Structured Generation

All structured generation features work seamlessly with async models:

```python
import asyncio
import outlines
import huggingface_hub
from pydantic import BaseModel

class User(BaseModel):
    name: str
    email: str
    age: int

async def generate_user():
    async_client = huggingface_hub.AsyncInferenceClient("http://localhost:11434")
    async_model = outlines.from_tgi(async_client)

    result = await async_model("Generate a random user profile.", output_type=User)
    user = User.model_validate_json(result)
    print(f"Name: {user.name}, Email: {user.email}, Age: {user.age}")

asyncio.run(generate_user())
```

## Inference parameters

When calling the model, you can provide optional parameters on top of the prompt and the output type. Those will be passed on to the `text_generation` method of the TGI client.

Common parameters include `max_new_tokens`, `stop_sequences`, `temperature`, `top_k`, `top_p`, and others as specified in the [TGI inference client documentation](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient).

---

### Features Models Transformers (`docs/features/models/transformers.md`)

---
title: Transformers
---

# Transformers

!!! Installation

    You need to install the `transformers` library to be able to use the Transformers in Outlines. Install all optional dependencies of the `Transformers` model with: `pip install "outlines[transformers]"`.

    See the [HuggingFace documentation](https://huggingface.co/docs/transformers/en/installation) for more information on installing `transformers` with CPU, GPU...

## Model Initialization

To load the model, you can use the `from_transformers` function. It takes 3 arguments:

- `model`: a `transformers` model (created with `AutoModelForCausalLM` for instance)
- `tokenizer_or_processor`: a `transformers` tokenizer (created with `AutoTokenizer` for instance, it must be an instance of either `PreTrainedTokenizer` or `PreTrainedTokenizerFast`)
- `device_dtype` (optional): the tensor dtype to use for inference. If not provided, the model will use the default dtype.

For instance:

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

# Create the transformers model and tokenizer
hf_model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
hf_tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

# Create the Outlines model
model = outlines.from_transformers(hf_model, hf_tokenizer)
```

If you provide a processor instead of a tokenizer for the second argument of the `from_transformers` function, you would get a `TransformersMultiModal` instance. See the [TransformersMultiModal model documentation](./transformers_multimodal.md) for more information on using multimodal models in Outlines.

## Text Generation

To generate text, you can simply call the model with a prompt.

For instance:

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

# Create model
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

# Call it to generate text
result = model("What's the capital of Latvia?", max_new_tokens=20)
print(result) # 'Riga'
```

#### Chat

You can also use chat inputs with the `Transformers` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import outlines
from outlines.inputs import Chat
from transformers import AutoModelForCausalLM, AutoTokenizer

# Create the model
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of Latvia?"},
])

# Call the model to generate a response
response = model(prompt, max_new_tokens=50)
print(response) # 'This is a picture of a black dog.'
```

#### Batching

Finally, the `Transformers` model supports batching through the `batch` method. To use it, provide a list of prompts (using the formats described above) to the `batch` method. You will receive as a result a list of completions.

For instance:

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

# Create model
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

# Create a list of prompts that will be used in a single batch
prompts = [
    "What's the capital of Lithuania?",
    "What's the capital of Latvia?",
    "What's the capital of Estonia?"
]

# Call it to generate text
result = model.batch(prompts, max_new_tokens=20)
print(result) # ['Vilnius', 'Riga', 'Tallinn']
```

## Structured Generation

As a local model, `Transformers` supports all output types available in Outlines. Simply provide an `output_type` after the prompt when calling the model.

### Simple Type

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

output_type = int

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

result = model("How many countries are there in the world?", output_type, max_new_tokens=5)
print(result) # '200'
```

### JSON Schema

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
from pydantic import BaseModel
from typing import List

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

result = model("Create a character.", output_type=Character, max_new_tokens=200, repetition_penalty=0.5)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

### Multiple Choice

```python
from typing import Literal
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

output_type = Literal["Paris", "London", "Rome", "Berlin"]

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

result = model("What is the capital of France?", output_type, max_new_tokens=10, temperature=0)
print(result) # 'Paris'
```

### Regex

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
from outlines.types import Regex

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

result = model("Generate a fake social security number.", output_type, max_new_tokens=20, top_p=0.5)
print(result) # '782-32-3789'
```

### Context-Free Grammar

```python
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
from outlines.types import CFG

arithmetic_grammar = """
?start: sum

?sum: product
| sum "+" product   -> add
| sum "-" product   -> sub

?product: atom
| product "*" atom  -> mul
| product "/" atom  -> div

?atom: NUMBER           -> number
| "-" atom         -> neg
| "(" sum ")"

%import common.NUMBER
%import common.WS_INLINE

%ignore WS_INLINE
"""
output_type = CFG(arithmetic_grammar)

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

result = model("Write an addition.", output_type, max_new_tokens=100)
print(result) # '23 + 48'
```

## Inference Arguments

When calling the model, you can provide optional inference parameters on top of the prompt and the output type. These parameters will be passed on to the `generate` method of the `transformers` model. Some common inference arguments include `max_new_tokens`, `temperature`, `repetition_penalty` and `top_p`.

See the [transformers documentation](https://huggingface.co/docs/transformers/en/main_classes/text_generation) for more information on inference parameters.

!!! Warning

    The `max_new_tokens` inference parameter has a default value of 20. This is insufficient for most tasks and will result in the generation output not respecting the output type (because the response is truncated). We recommend you always provide a value for this argument.

---

### Features Models Transformers Multimodal (`docs/features/models/transformers_multimodal.md`)

---
title: Transformers MultiModal
---

# Transformers MultiModal

The Outlines `TransformersMultiModal` model inherits from `Transformers` and shares most of its interface. Please start by reading the [Transformers documentation](./transformers.md) as this document only focuses on the specificities of `TransformersMultiModal` compared to `Transformers`.

## Model Initialization

To load the model, you can use the `from_transformers` function. It takes 2 arguments:

- `model`: a `transformers` model (created with `AutoModelForImageTextToText` for instance)
- `tokenizer_or_processor`: a `transformers` processor (created with `AutoProcessor` for instance, it must be an instance of `ProcessorMixin`)
- `device_dtype` (optional): the tensor dtype to use for inference. If not provided, the model will use the default dtype.

For instance:

```python
import outlines
from transformers import AutoModelForImageTextToText, AutoProcessor

# Create the transformers model and processor
hf_model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
hf_processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

# Create the Outlines model
model = outlines.from_transformers(hf_model, hf_processor)
```

## Model Input

As with other multimodal models, you should provide a list containing a text prompt and assets (`Image`, `Audio` or `Video` instances) as the model input. The type of asset to provide depends on the capabilities of the `transformers` model you are running.

Here's an example of using a vision multimodal model:

```python
from io import BytesIO
from urllib.request import urlopen

from PIL import Image as PILImage
from pydantic import BaseModel
from transformers import (
    LlavaForConditionalGeneration,
    AutoProcessor,
)

import outlines
from outlines.inputs import Image

TEST_MODEL = "trl-internal-testing/tiny-LlavaForConditionalGeneration"
IMAGE_URL = "https://upload.wikimedia.org/wikipedia/commons/2/25/Siam_lilacpoint.jpg"

class Animal(BaseModel):
    specie: str
    color: str
    weight: int

def get_image_from_url(image_url):
    img_byte_stream = BytesIO(urlopen(image_url).read())
    image = PILImage.open(img_byte_stream).convert("RGB")
    image.format = "PNG"
    return image

# Create a model
model = outlines.from_transformers(
    LlavaForConditionalGeneration.from_pretrained(TEST_MODEL),
    AutoProcessor.from_pretrained(TEST_MODEL),
)

# Call it with a model input dict containing a text prompt and an image + an output type
result = model(
    ["<image>Describe this animal.", Image(get_image_from_url(IMAGE_URL))],
    Animal,
    max_new_tokens=100
)
print(result) # '{"specie": "cat", "color": "white", "weight": 4}'
print(Animal.model_validate_json(result)) # specie=cat, color=white, weight=4
```
!!! Warning

    Make sure your prompt contains the tags expected by your processor to correctly inject the assets in the prompt. For some vision multimodal models for instance, you need to add as many `<image>` tags in your prompt as there are image assets included in your model input. `Chat` method, instead, does not require this step.


### Chat
The `Chat` interface offers a more convenient way to work with multimodal inputs. You don't need to manually add asset tags like `<image>`. The model's HF processor handles the chat templating and asset placement for you automatically.
To do so, call the model with a `Chat` instance using a multimodal chat format. Assets must be pre-processed as `outlines.inputs.{Image, Audio, Video}` format, and only `image`, `video`, and `audio` types are supported.

For instance:

```python
import outlines
from outlines.inputs import Chat, Image
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image as PILImage
from io import BytesIO
from urllib.request import urlopen
import torch

model_kwargs = {
        "torch_dtype": torch.bfloat16,
        "attn_implementation": "flash_attention_2",
        "device_map": "auto",
    }

def get_image_from_url(image_url):
    img_byte_stream = BytesIO(urlopen(image_url).read())
    image = PILImage.open(img_byte_stream).convert("RGB")
    image.format = "PNG"
    return image

# Create the model
model = outlines.from_transformers(
    AutoModelForImageTextToText.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", **model_kwargs),
    AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", **model_kwargs)
)

IMAGE_URL = "https://upload.wikimedia.org/wikipedia/commons/2/25/Siam_lilacpoint.jpg"

# Create the chat mutimodal input
prompt = Chat([
    {
        "role": "user",
        "content": [
            {"type": "image", "image": Image(get_image_from_url(IMAGE_URL))},
            {"type": "text", "text": "Describe the image in few words."}
        ],
    }
])

# Call the model to generate a response
response = model(prompt, max_new_tokens=50)
print(response) # 'A Siamese cat with blue eyes is sitting on a cat tree, looking alert and curious.'
```

Or using a list containing text and assets:

```
/* Detailed source-code truncated for AI context efficiency. */
```


### Batching
The `TransformersMultiModal` model supports batching through the `batch` method. To use it, provide a list of prompts (using the formats described above) to the `batch` method. You will receive as a result a list of completions.

An example using the Chat format:

```
/* Detailed source-code truncated for AI context efficiency. */
```


An example using a list of lists with tag assets:

```python
from io import BytesIO
from urllib.request import urlopen

from PIL import Image as PILImage
from transformers import (
    LlavaForConditionalGeneration,
    AutoProcessor,
)

import outlines
from outlines.inputs import Image

TEST_MODEL = "trl-internal-testing/tiny-LlavaForConditionalGeneration"
IMAGE_URL = "https://upload.wikimedia.org/wikipedia/commons/2/25/Siam_lilacpoint.jpg"
IMAGE_URL_2 ="https://upload.wikimedia.org/wikipedia/commons/9/98/Aldrin_Apollo_11_original.jpg"

def get_image_from_url(image_url):
    img_byte_stream = BytesIO(urlopen(image_url).read())
    image = PILImage.open(img_byte_stream).convert("RGB")
    image.format = "PNG"
    return image

# Create a model
model = outlines.from_transformers(
    LlavaForConditionalGeneration.from_pretrained(TEST_MODEL),
    AutoProcessor.from_pretrained(TEST_MODEL),
)

# Call the batch method with a list of model input dicts
result = model.batch(
    [
        ["<image>Describe the image.", Image(get_image_from_url(IMAGE_URL))],
        ["<image>Describe the image.", Image(get_image_from_url(IMAGE_URL_2))],
    ]
)
print(result) # ['The image shows a cat', 'The image shows an astronaut']
```

---

### Features Models Vllm (`docs/features/models/vllm.md`)

---
title: vLLM
---

# vLLM

## Prerequisites

The Outlines `VLLM` model is intended to be used along with a vLLM instance running on a separate server (can be local or remote). Make sure you have a vLLM server running and accessible before using the `VLLM` model. For instance by running:

```shell
pip install vllm

vllm serve microsoft/Phi-3-mini-4k-instruct \
  --dtype auto \
  --api-key token-abc123
```

Follow the [Installation instructions](https://docs.vllm.ai/en/latest/getting_started/installation/index.html) for more information on how to set up a vLLM server for your particular setup.

!!! warning "vLLM server version"

    Structured output requires a vLLM server **>= 0.12**. Older servers will silently ignore the constrained generation arguments and return **unconstrained** output.

As the vLLM client relies on the `openai` python sdk, you need to have the `openai` package installed. Install all optional dependencies for the `VLLM` model with: `pip install openai`.

If you want to use the vllm offline inference mode instead of the server mode, please refer to the [VLLMOffline](./vllm_offline.md) model documentation.

## Model Initialization

To load the model, you can use the `from_vllm` function. The argument of the function is either an `OpenAI` or `AsyncOpenAI` instance from the `openai` library. Make sure the value of the `base_url` argument of the `OpenAI` client points to your running vLLM server. Consult the [vLLM documentation](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) on using an OpenAI client with a vLLM server for more information.

Based on whether the `openai` client instance is synchronous or asynchronous, you will receive a `VLLM` or `AsyncVLLM` model instance.

For instance:

```python
import openai
import outlines

# Create the OpenAI client
sync_openai_client = openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
async_openai_client = openai.AsyncOpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")

# Create a sync model
sync_model = outlines.from_vllm(sync_openai_client, "microsoft/Phi-3-mini-4k-instruct")
print(type(sync_model)) # <class 'outlines.models.vllm.VLLM'>

# Create an async model
async_model = outlines.from_vllm(async_openai_client, "microsoft/Phi-3-mini-4k-instruct")
print(type(async_model)) # <class 'outlines.models.vllm.AsyncVLLM'>
```

## Text Generation

To generate text, you can simply call the model with a prompt.

For instance:

```python
import openai
import outlines

# Create the model
model = outlines.from_vllm(openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123"), "microsoft/Phi-3-mini-4k-instruct")

# Call it to generate text
response = model("What's the capital of Latvia?", max_tokens=20)
print(response) # 'The capital of Latvia is Riga.'
```

#### Vision

Some models you can run with VLLM support vision input. To use this feature, provide a list containing a text prompt and `Image` instances.

For instance:

```python
import io
import requests
import PIL
import outlines
import openai
from outlines.inputs import Image

# Create the model
model = outlines.from_vllm(
    openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123"),
    "Qwen/Qwen2.5-VL-3B-Instruct"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the prompt containing the text and the image
prompt = [
    "Describe the image",
    Image(get_image("https://picsum.photos/id/237/400/300"))
]

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'The image shows a black puppy lying on a wooden surface...'
```

#### Chat

You can also use chat inputs with the `VLLM` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import io
import requests
import PIL
import openai
import outlines
from outlines.inputs import Chat, Image

# Create the model
model = outlines.from_vllm(
    openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123"),
    "Qwen/Qwen2.5-VL-3B-Instruct"
)

# Function to get an image
def get_image(url):
    r = requests.get(url)
    return PIL.Image.open(io.BytesIO(r.content))

# Create the chat input
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": ["Describe the image", Image(get_image("https://picsum.photos/id/237/400/300"))]
    },
])

# Call the model to generate a response
response = model(prompt, max_tokens=50)
print(response) # 'The image shows a black puppy lying on a wooden surface...'
```

#### Streaming

Finally, the `VLLM` model supports streaming through the `stream` method.

For instance:

```python
import openai
import outlines

# Create the model
model = outlines.from_vllm(
    openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123"),
    "microsoft/Phi-3-mini-4k-instruct"
)

# Stream the response
for chunk in model.stream("Tell me a short story about a cat.", max_tokens=50):
    print(chunk, end="") # 'Once upon a time...'
print()
```

## Asynchronous Calls

vLLM supports asynchronous operations by passing an `AsyncOpenAI` client instead of a regular `OpenAI` client. This returns an `AsyncVLLM` model instance that supports async/await patterns.

### Basic Async Generation

```python
import asyncio
import openai
import outlines

async def generate_text():
    async_client = openai.AsyncOpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
    async_model = outlines.from_vllm(async_client, "microsoft/Phi-3-mini-4k-instruct")

    result = await async_model("Write a haiku about Python.", max_tokens=50)
    print(result)

asyncio.run(generate_text())
```

### Async Streaming

The async model also supports streaming with async iteration:

```python
import asyncio
import openai
import outlines

async def stream_text():
    async_client = openai.AsyncOpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
    async_model = outlines.from_vllm(async_client, "microsoft/Phi-3-mini-4k-instruct")

    async for chunk in async_model.stream("Tell me a story about a robot.", max_tokens=100):
        print(chunk, end="")

asyncio.run(stream_text())
```

### Concurrent Async Requests

One of the main benefits of async calls is the ability to make multiple concurrent requests:

```python
import asyncio
import openai
import outlines

async def generate_multiple():
    async_client = openai.AsyncOpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
    async_model = outlines.from_vllm(async_client, "microsoft/Phi-3-mini-4k-instruct")

    prompts = [
        "Write a tagline for a coffee shop.",
        "Write a tagline for a bookstore.",
        "Write a tagline for a gym."
    ]

    tasks = [async_model(prompt, max_tokens=30) for prompt in prompts]
    results = await asyncio.gather(*tasks)

    for prompt, result in zip(prompts, results):
        print(f"{prompt}\n{result}\n")

asyncio.run(generate_multiple())
```

## Structured Generation

vLLM supports all output types available in Outlines. Simply provide an `output_type` after the prompt when calling the model. All structured generation features work with both synchronous and asynchronous models.

### Simple Type

```python
import openai
import outlines

output_type = int

openai_client = openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
model = outlines.from_vllm(openai_client, "microsoft/Phi-3-mini-4k-instruct")

result = model("How many countries are there in the world?", output_type)
print(result) # '200'
```

### JSON Schema

```python
import openai
import outlines
from typing import List
from pydantic import BaseModel

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

openai_client = openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
model = outlines.from_vllm(openai_client, "microsoft/Phi-3-mini-4k-instruct")

result = model("Create a character.", output_type=Character, frequency_penalty=1.5)
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

### Multiple Choice

```python
from typing import Literal
import openai
import outlines

output_type = Literal["Paris", "London", "Rome", "Berlin"]

openai_client = openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
model = outlines.from_vllm(openai_client, "microsoft/Phi-3-mini-4k-instruct")

result = model("What is the capital of France?", output_type, temperature=0)
print(result) # 'Paris'
```

### Regex

```python
import openai
import outlines
from outlines.types import Regex

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

openai_client = openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
model = outlines.from_vllm(openai_client, "microsoft/Phi-3-mini-4k-instruct")

result = model("Generate a fake social security number.", output_type, top_p=0.1)
print(result) # '782-32-3789'
```

### Context-Free Grammar

```python
import openai
import outlines
from outlines.types import CFG

arithmetic_grammar = """
?start: sum

?sum: product
| sum "+" product   -> add
| sum "-" product   -> sub

?product: atom
| product "*" atom  -> mul
| product "/" atom  -> div

?atom: NUMBER           -> number
| "-" atom         -> neg
| "(" sum ")"

%import common.NUMBER
%import common.WS_INLINE

%ignore WS_INLINE
"""
output_type = CFG(arithmetic_grammar)

openai_client = openai.OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
model = outlines.from_vllm(openai_client, "microsoft/Phi-3-mini-4k-instruct")

result = model("Write an addition.", output_type, extra_body={"guided_decoding_backend": "outlines"})
print(result) # '23 + 48'
```

### Async Structured Generation

All structured generation features work seamlessly with async models:

```python
import asyncio
import openai
import outlines
from pydantic import BaseModel

class User(BaseModel):
    name: str
    email: str
    age: int

async def generate_user():
    async_client = openai.AsyncOpenAI(base_url="http://0.0.0.0:8000/v1", api_key="token-abc123")
    async_model = outlines.from_vllm(async_client, "microsoft/Phi-3-mini-4k-instruct")

    result = await async_model("Generate a random user profile.", output_type=User)
    user = User.model_validate_json(result)
    print(f"Name: {user.name}, Email: {user.email}, Age: {user.age}")

asyncio.run(generate_user())
```

## Inference Arguments

When calling the model, you can provide optional parameters on top of the prompt and the output type. Those will be passed on to the `chat.completions.create` method of the OpenAI client.

An optional parameter of particular interest is `extra_body`, which is a dictionary containing arguments that are specific to vLLM and are not part of the standard `openai` interface. Among those, `guided_decoding_backend` allows you to select the library used by the vLLM server to control structured generation. You can use the value `outlines` to generated structured text with Outlines.

See the [vLLM documentation](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#extra-parameters) on extra parameters for the OpenAI-compatible server for more information on inference parameters.

---

### Features Models Vllm Offline (`docs/features/models/vllm_offline.md`)

---
title: vLLM Offline
---

# vLLM Offline

Outlines provides an integration with [vLLM](https://docs.vllm.ai/en/latest/) using the [vllm library](https://github.com/vllm-project/vllm). This model allows you to use vLLM in the "Offline Inference" mode, meaning that text generation happens within the model, there is no separate server. If you want to use vLLM with a server, see the [VLLM model documentation](./vllm.md).

!!! Note "Installation"

    You need to install the `vllm` library to be able to use the `VLLMOffline` model: `pip install vllm`. Due to a library version conflict between outlines and vllm, you MUST install `vllm` before installing `outlines`.

    When installing `outlines` (after having first installed `vllm`), you may encounter the following error: `ERROR: pip's dependency resolver does not currently take into account all the packages that are installed`. You can safely ignore it.

    See the [vLLM documentation](https://docs.vllm.ai/en/latest/getting_started/installation/index.html) for instructions on how to install vLLM for CPU, ROCm...

## Model Initialization

To load the model, you can use the `from_vllm_offline` function. The single argument of the function is a `LLM` model instance from the `vllm` library. You will then receive a `VLLMOffline` model instance you can use to generate text.

Consult the [LLM class API reference](https://docs.vllm.ai/en/latest/api/vllm/index.html#vllm.LLM) for detailed information on how to create an `LLM` instance and on the various available parameters.

For instance:

```python
import outlines
from vllm import LLM

# Create the model
model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)
```

!!! Note

    When initializing the `vllm.LLM` object, you can specify a `guided_decoding_backend` to choose what library will be used by vLLM to constrain the generation. Consult the [vLLM documentation](https://docs.vllm.ai/en/v0.8.2/features/structured_outputs.html) on structured output for the list of possible values.

## Text Generation

Once you've created your Outlines `VLLMOffline` model instance, you're all set to generate text with this provider. You can simply call the model with a prompt.

For instance:

```python
import outlines
from vllm import LLM, SamplingParams

# Create the model
model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

# Call it to generate text
response = model("What's the capital of Latvia?", sampling_params=SamplingParams(max_tokens=20))
print(response) # 'Riga'
```

#### Chat

You can also use chat inputs with the `VLLMOffline` model. To do so, call the model with a `Chat` instance. The content of messsage within the chat can be vision inputs as described above.

For instance:

```python
import outlines
from vllm import LLM, SamplingParams
from outlines.inputs import Chat

# Create the model
model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

# Create the chat prompt
prompt = Chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of Latvia?"},
])

# Call the model to generate a response
response = model(prompt, sampling_params=SamplingParams(max_tokens=50))
print(response) # 'Riga'
```

#### Streaming

The `VLLMOffline` model supports streaming through the `stream` method.

For instance:

```python
import outlines
from vllm import LLM, SamplingParams

# Create the model
model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

# Stream the response
for chunk in model.stream("Tell me a short story about a cat.", sampling_params=SamplingParams(max_tokens=50)):
    print(chunk) # 'Once...'
```

#### Batching

Finally, the `VLLMOffline` model also supports batching through the `batch` method. To use it, provide a list of prompts (using the formats described above) to the `batch` method. You will receive as a result a list of completions.

For instance:

```python
import outlines
from vllm import LLM

# Create the model
model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

# Create a list of prompts that will be used in a single batch
prompts = [
    "What's the capital of Lithuania?",
    "What's the capital of Latvia?",
    "What's the capital of Estonia?"
]

# Call it to generate text
result = model.batch(prompts, max_new_tokens=20)
print(result) # ['Vilnius', 'Riga', 'Tallinn']
```

## Structured Generation

The `VLLMOffline` model supports all output types available in Outlines. Simply provide an `output_type` after the prompt when calling the model.

### Simple Type

```python
import outlines
from vllm import LLM

output_type = int

model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

result = model("How many countries are there in the world?", output_type)
print(result) # '200'
```

### JSON Schema

```python
import outlines
from vllm import LLM, SamplingParams
from typing import List
from pydantic import BaseModel

class Character(BaseModel):
    name: str
    age: int
    skills: List[str]

model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

result = model("Create a character.", output_type=Character, sampling_params=SamplingParams(frequency_penalty=1.5, max_tokens=200))
print(result) # '{"name": "Evelyn", "age": 34, "skills": ["archery", "stealth", "alchemy"]}'
print(Character.model_validate_json(result)) # name=Evelyn, age=34, skills=['archery', 'stealth', 'alchemy']
```

### Multiple Choice

```python
from typing import Literal
import outlines
from vllm import LLM, SamplingParams

output_type = Literal["Paris", "London", "Rome", "Berlin"]

model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

result = model("What is the capital of France?", output_type, sampling_params=SamplingParams(temperature=0))
print(result) # 'Paris'
```

### Regex

```python
import outlines
from vllm import LLM, SamplingParams
from outlines.types import Regex

output_type = Regex(r"\d{3}-\d{2}-\d{4}")

model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

result = model("Generate a fake social security number.", output_type, sampling_params=SamplingParams(top_p=0.1))
print(result) # '782-32-3789'
```

### Context-Free Grammar

```python
import outlines
from vllm import LLM, SamplingParams
from outlines.types import CFG

arithmetic_grammar = """
?start: sum

?sum: product
| sum "+" product   -> add
| sum "-" product   -> sub

?product: atom
| product "*" atom  -> mul
| product "/" atom  -> div

?atom: NUMBER           -> number
| "-" atom         -> neg
| "(" sum ")"

%import common.NUMBER
%import common.WS_INLINE

%ignore WS_INLINE
"""
output_type = CFG(arithmetic_grammar)

model = outlines.from_vllm_offline(
    LLM("microsoft/Phi-3-mini-4k-instruct")
)

result = model("Write an addition.", output_type)
print(result) # '23 + 48'
```

## Inference Arguments

When calling the model, you can provide optional parameters on top of the prompt and the output type. Those will be passed on to the `generate` method of the `LLM` model instance. An argument of particular interest is `sampling_params`. It takes as a value a `vllm.SamplingParams` instance containing parameters such as max_tokens or temperature.

See the [vLLM documentation](https://docs.vllm.ai/en/latest/api/vllm/sampling_params.html#vllm.sampling_params.SamplingParams) on sampling parameters for more information on inference parameters.

---

### Features Utility Application (`docs/features/utility/application.md`)

---
title: Application
---

# Application

The `Application` class enables you to encapsulate a prompt template and an output type into a reusable component.

## Overview

An `Application` combines a prompt template with an output type, creating a reusable component that can be applied to different models.

Applications are useful for simplifying repeated tasks where you have a well-defined `Template` and a fixed output type, such as classification tasks or data extraction.

To create an `Application` instance, initialize the class with a prompt template and an output type. You can then call the application with a model and the variables defined in your template in a dictionary.

For instance:

```python
from typing import Literal
import transformers
from outlines import Application, Template, from_transformers

# Create a template
template_str = "Is {{ name }} a boy or a girl name?"
template = Template.from_string(template_str)

# Create a model
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

# Create the application and call it to generate text
application = Application(template, Literal["boy", "girl"])
response = application(model, {"name": "Alice"}, max_new_tokens=10)

print(response) # "girl"
```

Instead of providing an Outlines `Template` instance, you can provide a `Callable` that returns a string. The parameters of the callable are used as the variables of the template such that you must provide values for them in the dictionary when calling the application.

For instance, we can create the same example as above using a a function instead of a template:

```python
from typing import Literal
import transformers
from outlines import Application, from_transformers

# Create a function that will be used as a template
def template_func(name: str) -> str:
    return f"Is {name} a boy or a girl name?"

# Create a model
model = from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
    transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
)

# Create the application with the function template and call it to generate text
application = Application(template_func, Literal["boy", "girl"])
response = application(model, {"name": "Alice"}, max_new_tokens=10)

print(response) # "girl"
```

---

### Features Utility Regex Dsl (`docs/features/utility/regex_dsl.md`)

---
title: Regex DSL
---

# Regex DSL

This library provides a Domain-Specific Language (DSL) to construct regular expressions in a more intuitive and modular way. It allows you to create complex regexes using simple building blocks that represent literal strings, patterns, and various quantifiers. Additionally, these custom regex types can be used directly as types in [Pydantic](https://pydantic-docs.helpmanual.io/) schemas to enforce pattern constraints during text generation.

---

## Why Use This DSL?

1. **Modularity & Readability**: Instead of writing cryptic regular expression strings, you compose a regex as a tree of objects.
2. **Enhanced Debugging**: Each expression can be visualized as an ASCII tree, making it easier to understand and debug complex regexes.
3. **Pydantic Integration**: Use your DSL-defined regex as types in Pydantic models. The DSL seamlessly converts to JSON Schema with proper pattern constraints.
4. **Extensibility**: Easily add or modify quantifiers and other regex components by extending the provided classes.

---

## Building Blocks


Every regex component in this DSL is a **Term**. Here are two primary types:

- **`String`**: Represents a literal string. It escapes the characters that have a special meaning in regular expressions.
- **`Regex`**: Represents an existing regex pattern string.

```python
from outlines.types import String, Regex

# A literal string "hello"
literal = String("hello")   # Internally represents "hello"

# A regex pattern to match one or more digits
digit = Regex(r"[0-9]+")     # Internally represents the pattern [0-9]+

# Converting to standard regex strings:
from outlines.types.dsl import to_regex

print(to_regex(literal))  # Output: hello
print(to_regex(digit))    # Output: [0-9]+
```

---

## Early Introduction to Quantifiers & Combining Terms

The DSL supports common regex quantifiers as methods on every `Term`. These methods allow you to specify how many times a pattern should be matched. They include:

- **`exactly(count)`**: Matches the term exactly `count` times.
- **`optional()`**: Matches the term zero or one time.
- **`one_or_more()`**: Matches the term one or more times (Kleene Plus).
- **`zero_or_more()`**: Matches the term zero or more times (Kleene Star).
- **`between(min_count, max_count)`**: Matches the term between `min_count` and `max_count` times (inclusive).
- **`at_least(count)`**: Matches the term at least `count` times.
- **`at_most(count)`**: Matches the term up to `count` times.

These quantifiers can also be used as functions that take the `Term` as an argument. If the term is a plain string, it will be automatically converted to a `String` object. Thus `String("foo").optional()` is equivalent to `optional("foo")`.

Let's see these quantifiers side by side with examples.

### Quantifiers in Action

#### `exactly(count)`

This method restricts the term to appear exactly `count` times.

```python
# Example: exactly 5 digits
five_digits = Regex(r"\d").exactly(5)
print(to_regex(five_digits))  # Output: (\d){5}
```

You can also use the `exactly` function:

```python
from outlines.types import exactly

# Example: exactly 5 digits
five_digits = exactly(Regex(r"\d"), 5)
print(to_regex(five_digits))  # Output: (\d){5}
```

#### `optional()`

This method makes a term optional, meaning it may occur zero or one time.

```python
# Example: an optional "s" at the end of a word
maybe_s = String("s").optional()
print(to_regex(maybe_s))  # Output: (s)?
```

You can also use the `optional` function:

```python
from outlines.types import optional

# Example: an optional "s" at the end of a word
maybe_s = optional("s")
print(to_regex(maybe_s))  # Output: (s)?
```

#### `one_or_more()`

This method indicates that the term must appear at least once.

```python
# Example: one or more alphabetic characters
letters = Regex(r"[A-Za-z]").one_or_more()
print(to_regex(letters))  # Output: ([A-Za-z])+
```

You can also use the `one_or_more` function:

```python
from outlines.types import one_or_more

# Example: one or more alphabetic characters
letters = one_or_more(Regex(r"[A-Za-z]"))
print(to_regex(letters))  # Output: ([A-Za-z])+

```

#### `zero_or_more()`

This method indicates that the term can occur zero or more times.

```python
# Example: zero or more spaces
spaces = String(" ").zero_or_more()
print(to_regex(spaces))  # Output: ( )*
```

You can also use the `zero_or_more` function:

```python
from outlines.types import zero_or_more

# Example: zero or more spaces
spaces = zero_or_more(" ")
print(to_regex(spaces))  # Output: ( )*
```

#### `between(min_count, max_count)`

This method indicates that the term can appear any number of times between `min_count` and `max_count` (inclusive).

```python
# Example: Between 2 and 4 word characters
word_chars = Regex(r"\w").between(2, 4)
print(to_regex(word_chars))  # Output: (\w){2,4}
```

You can also use the `between` function:

```python
from outlines.types import between

# Example: Between 2 and 4 word characters
word_chars = between(Regex(r"\w"), 2, 4)
print(to_regex(word_chars))  # Output: (\w){2,4}
```

#### `at_least(count)`

This method indicates that the term must appear at least `count` times.

```python
# Example: At least 3 digits
at_least_three = Regex(r"\d").at_least(3)
print(to_regex(at_least_three))  # Output: (\d){3,}
```

You can also use the `at_least` function:

```python
from outlines.types import at_least

# Example: At least 3 digits
at_least_three = at_least(Regex(r"\d"), 3)
print(to_regex(at_least_three))  # Output: (\d){3,}
```

#### `at_most(count)`

This method indicates that the term can appear at most `count` times.

```python
# Example: At most 3 digits
up_to_three = Regex(r"\d").at_most(3)
print(to_regex(up_to_three))  # Output: (\d){0,3}
```

You can also use the `at_most` function:

```python
from outlines.types import at_most

# Example: At most 3 digits
up_to_three = at_most(Regex(r"\d"), 3)
print(to_regex(up_to_three))  # Output: (\d){0,3}
```

---

## Combining Terms

The DSL allows you to combine basic terms into more complex patterns using concatenation and alternation.

### Concatenation (`+`)

The `+` operator (and its reflected variant) concatenates terms, meaning that the terms are matched in sequence.

```python
# Example: Match "hello world"
pattern = String("hello") + " " + Regex(r"\w+")
print(to_regex(pattern))  # Output: hello\ (\w+)
```

### Alternation (`either()`)

The `either()` function creates alternatives, allowing a match for one of several patterns. You can provide as many terms as you want.

```python
# Example: Match either "cat" or "dog" or "mouse"
animal = either(String("cat"), "dog", "mouse")
print(to_regex(animal))  # Output: (cat|dog|mouse)
```

*Note:* When using `either()` with plain strings (such as `"dog"`), the DSL automatically wraps them in a `String` object that escapes the characters that have a special meaning in regular expressions, just like with quantifier functions.

---

## Custom types

The DSL comes "batteries included" with types that represent common text constructs:

- `integer` represents an integer number as recognized by `int`
- `boolean` represents a boolean, "True" or "False" as recognized by `bool`
- `number` represents a floating-point number recognize by Python's `float`
- `date` represents a date as understood by `datetime.date`
- `time` represents a time as understood by `datetime.time`
- `datetime` represents a time as understood by `datetime.datetime`
- `digit` represents a single digit
- `char` represents a single character
- `newline` represents a new line character
- `whitespace` represents a white space
- `hex_str` represents a hexadecimal string, optionally prefixed with "0x"
- `uuid4` represents a UUID version 4 string in the format "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx"
- `ipv4` represents an IPv4 address in the format "xxx.xxx.xxx.xxx" where each octet is between 0 and 255
- `sentence` represents a sentence
- `paragraph` represents a paragraph (one or more sentences separated by one or more line breaks)

For instance you can describe the answers in the GSM8K dataset using the following pattern:

```python
from outlines.types import sentence, digit

answer = "A: " + sentence.between(2,4) + " So the answer is: " + digit.between(1,4)
```

---

## Practical Examples

### Example 1: Matching a Custom ID Format

Suppose you want to create a regex that matches an ID format like "ID-12345", where:

- The literal "ID-" must be at the start.
- Followed by exactly 5 digits.

```python
id_pattern = "ID-" + Regex(r"\d").exactly(5)
print(to_regex(id_pattern))  # Output: ID-(\d){5}
```

### Example 2: Email Validation with Pydantic

You can define a regex for email validation and use it as a type in a Pydantic model.

```python
from pydantic import BaseModel, ValidationError

# Define an email regex term (this is a simplified version)
email_regex = Regex(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+")

class User(BaseModel):
    name: str
    email: email_regex  # Use our DSL regex as a field type

# Valid input
user = User(name="Alice", email="alice@example.com")
print(user)

# Invalid input (raises a ValidationError)
try:
    User(name="Bob", email="not-an-email")
except ValidationError as e:
    print(e)
```

When used in a Pydantic model, the email field is automatically validated against the regex pattern and its JSON Schema includes the `pattern` constraint.

### Example 3: Building a Complex Pattern

Consider a pattern to match a simple date format: `YYYY-MM-DD`.

```python
year = Regex(r"\d").exactly(4)         # Four digits for the year
month = Regex(r"\d").exactly(2)        # Two digits for the month
day = Regex(r"\d").exactly(2)          # Two digits for the day

# Combine with literal hyphens
date_pattern = year + "-" + month + "-" + day
print(to_regex(date_pattern))
# Output: (\d){4}\-(\d){2}\-(\d){2}
```

---

## Visualizing Your Pattern

One of the unique features of this DSL is that each term can print its underlying structure as an ASCII tree. This visualization can be particularly helpful when dealing with complex expressions.

```python
# A composite pattern using concatenation and quantifiers
pattern = "a" + String("b").one_or_more() + "c"
print(pattern)
```

*Expected Output:*

```ascii
└── Sequence
    ├── String('a')
    ├── KleenePlus(+)
    │   └── String('b')
    └── String('c')
```

This tree representation makes it easy to see the hierarchy and order of operations in your regular expression.

---

## Final Words

This DSL is designed to simplify the creation and management of regular expressions—whether you're validating inputs in a web API, constraining the output of an LLM, or just experimenting with regex patterns. With intuitive methods for common quantifiers and operators, clear visual feedback, and built-in integration with Pydantic, you can build robust and maintainable regex-based validations with ease.

Feel free to explore the library further and adapt the examples to your use cases. Happy regexing!

---

### Features Utility Template (`docs/features/utility/template.md`)

---
title: Template
---

# Template

Outlines templates provide a way of creating reusable prompt structures with placeholders for dynamic content.

## Overview

To create a `Template` instance, you can use two class methods:
- `from_string`: Creates a template from a string containing a Jinja2 template
- `from_file`: Creates a template from a file containing a Jinja2 template

After creating a template, you can call it with the variables required by the template as keyword arguments.

For instance:

```python
from outlines import Template

# Create a template from a string
template_str = """
Hello, {{ name }}!
The weather today is {{ weather }}.
"""
template = Template.from_string(template_str)

# Create a template from a file, assuming the content of template_str is put into a file
template = Template.from_file("path_to/my_file.txt")

# Call the template to render the prompt
prompt: str = template(name="Alice", weather="sunny")
print(prompt)  # "Hello, Alice!\nThe weather today is sunny."
```

## Composite Templates

Templates can be nested and composed to create complex prompt structures:

```python
from outlines import Template

# Create component templates
user_template = Template.from_string("User: {{ query }}")
system_template = Template.from_string("System: {{ instruction }}")

# Create a composite template
chat_template = Template.from_string("""
{{ system }}
{{ user }}
""")

# Fill in nested templates
prompt = chat_template(
    system=system_template(instruction="You are a helpful assistant."),
    user=user_template(query="What is machine learning?")
)

print(prompt)
# System: You are a helpful assistant.
#
# User: What is machine learning?
```

## Custom Filters

You can add custom filters to your Outlines template to extend the templating functionality. To do so, provide as second argument a dictionary with filter names as keys and filter functions as values. The filter can then be used in your jinja2 template following the regular syntax. When rendering a prompt, the function will be applied to the associated variable.

For instance:

```python
from outlines import Template

def uppercase(text: str) -> str:
    return text.upper()

# Add custom filter when creating template
template = Template.from_string(
    "Hello {{ name | uppercase }}!",
    filters={"uppercase": uppercase}
)
prompt = template(name="alice")
print(prompt)  # "Hello ALICE!"
```

---

--- METRICS ---
- Files Extracted: 72
- Estimated Token Budget: ~88694 tokens
- Canonical Reference: https://codewiki.google/github.com/dottxt-ai/outlines