Technical Documentation: dottxt-ai/outlines
ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference
Overview (`README.md`)
🚀 Building the future of structured generation
We're working with select partners to develop new interfaces to structured generation.
Need XML, FHIR, custom schemas or grammars? Let's talk.
Audit your schema: share one schema, we show you what breaks under generation, the constraints that fix it, and compliance rates before and after. Sign up here.
Table of Contents
- Why Outlines?
- Quickstart
- Real-World Examples
- They Use Outlines
- Model Integrations
- Core Features
- Other Features
- About .txt
- Community
Why Outlines?
LLMs are powerful but their outputs are unpredictable. Most solutions attempt to fix bad outputs after generation using parsing, regex, or fragile code that breaks easily.
Outlines guarantees structured outputs during generation — directly from any LLM.
- Works with any model - Same code runs across OpenAI, Ollama, vLLM, and more
- Simple integration - Just pass your desired output type:
model(prompt, output_type) - Guaranteed valid structure - No more parsing headaches or broken JSON
- Provider independence - Switch models without changing code
The Outlines Philosophy
Outlines follows a simple pattern that mirrors Python's own type system. Simply specify the desired output type, and Outlines will ensure your data matches that structure exactly:
- For a yes/no response, use
Literal["Yes", "No"] - For numerical values, use
int - For complex objects, define a structure with a Pydantic model
Quickstart
Getting started with outlines is simple:
1. Install outlines
pip install outlines2. Connect to your preferred model
import outlines
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME)
)3. Start with simple structured outputs
from typing import Literal
from pydantic import BaseModel
# Simple classification
sentiment = model(
"Analyze: 'This product completely changed my life!'",
Literal["Positive", "Negative", "Neutral"]
)
print(sentiment) # "Positive"
# Extract specific types
temperature = model("What's the boiling point of water in Celsius?", int)
print(temperature) # 1004. Create complex structures
from pydantic import BaseModel
from enum import Enum
class Rating(Enum):
poor = 1
fair = 2
good = 3
excellent = 4
class ProductReview(BaseModel):
rating: Rating
pros: list[str]
cons: list[str]
summary: str
review = model(
"Review: The XPS 13 has great battery life and a stunning display, but it runs hot and the webcam is poor quality.",
ProductReview,
max_new_tokens=200,
)
review = ProductReview.model_validate_json(review)
print(f"Rating: {review.rating.name}") # "Rating: good"
print(f"Pros: {review.pros}") # "Pros: ['great battery life', 'stunning display']"
print(f"Summary: {review.summary}") # "Summary: Good laptop with great display but thermal issues"Real-world examples
Here are production-ready examples showing how Outlines solves common problems:
🙋♂️ Customer Support Triage
This example shows how to convert a free-form customer email into a structured service ticket. By parsing attributes like priority, category, and escalation flags, the code enables automated routing and handling of support issues.
import outlines
from enum import Enum
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
from typing import List
MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME)
)
def alert_manager(ticket):
print("Alert!", ticket)
class TicketPriority(str, Enum):
low = "low"
medium = "medium"
high = "high"
urgent = "urgent"
class ServiceTicket(BaseModel):
priority: TicketPriority
category: str
requires_manager: bool
summary: str
action_items: List[str]
customer_email = """
Subject: URGENT - Cannot access my account after payment
I paid for the premium plan 3 hours ago and still can't access any features.
I've tried logging out and back in multiple times. This is unacceptable as I
have a client presentation in an hour and need the analytics dashboard.
Please fix this immediately or refund my payment.
"""
prompt = f"""
<|im_start|>user
Analyze this customer email:
{customer_email}
<|im_end|>
<|im_start|>assistant
"""
ticket = model(
prompt,
ServiceTicket,
max_new_tokens=500
)
# Use structured data to route the ticket
ticket = ServiceTicket.model_validate_json(ticket)
if ticket.priority == "urgent" or ticket.requires_manager:
alert_manager(ticket)📦 E-commerce product categorization
This use case demonstrates how outlines can transform product descriptions into structured categorization data (e.g., main category, sub-category, and attributes) to streamline tasks such as inventory management. Each product description is processed automatically, reducing manual categorization overhead.
/* Detailed source-code truncated for AI context efficiency. */📊 Parse event details with incomplete data
This example uses outlines to parse event descriptions into structured information (like event name, date, location, type, and topics), even handling cases where the data is incomplete. It leverages union types to return either structured event data or a fallback “I don’t know” answer, ensuring robust extraction in varying scenarios.
/* Detailed source-code truncated for AI context efficiency. */🗂️ Categorize documents into predefined types
In this case, outlines classifies documents into predefined categories (e.g., “Financial Report,” “Legal Contract”) using a literal type specification. The resulting classifications are displayed in both a table format and through a category distribution summary, illustrating how structured outputs can simplify content management.
/* Detailed source-code truncated for AI context efficiency. */📅 Schedule a meeting from requests with Function Calling
This example demonstrates how outlines can interpret a natural language meeting request and translate it into a structured format matching a predefined function’s parameters. Once the meeting details are extracted (e.g., title, date, duration, attendees), they are used to automatically schedule the meeting.
/* Detailed source-code truncated for AI context efficiency. */📝 Dynamically generate prompts with re-usable templates
Using Jinja-based templates, this example shows how to generate dynamic prompts for tasks like sentiment analysis. It illustrates how to easily re-use and customize prompts—including few-shot learning strategies—for different content types while ensuring the outputs remain structured.
import outlines
from typing import List, Literal
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_NAME = "microsoft/phi-4"
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME)
)
# 1. Create a reusable template with Jinja syntax
sentiment_template = outlines.Template.from_string("""
<|im_start>user
Analyze the sentiment of the following {{ content_type }}:
{{ text }}
Provide your analysis as either "Positive", "Negative", or "Neutral".
<|im_end>
<|im_start>assistant
""")
# 2. Generate prompts with different parameters
review = "This restaurant exceeded all my expectations. Fantastic service!"
prompt = sentiment_template(content_type="review", text=review)
# 3. Use the templated prompt with structured generation
result = model(prompt, Literal["Positive", "Negative", "Neutral"])
print(result) # "Positive"
# Templates can also be loaded from files
example_template = outlines.Template.from_file("templates/few_shot.txt")
# Use with examples for few-shot learning
examples = [
("The food was cold", "Negative"),
("The staff was friendly", "Positive")
]
few_shot_prompt = example_template(examples=examples, query="Service was slow")
print(few_shot_prompt)They use outlines
Model Integrations
| Model type | Description | Documentation |
|---|---|---|
| Server Support | vLLM and Ollama | Server Integrations → |
| Local Model Support | transformers and llama.cpp | Model Integrations → |
| API Support | OpenAI, Gemini, and Dottxt | API Integrations → |
Core Features
| Feature | Description | Documentation |
|---|---|---|
| Multiple Choices | Constrain outputs to predefined options | Multiple Choices Guide → |
| Function Calls | Infer structure from function signatures | Function Guide → |
| JSON/Pydantic | Generate outputs matching JSON schemas | JSON Guide → |
| Regular Expressions | Generate text following a regex pattern | Regex Guide → |
| Grammars | Enforce complex output structures | Grammar Guide → |
Other Features
| Feature | Description | Documentation |
|---|---|---|
| Prompt templates | Separate complex prompts from code | Template Guide → |
| Custome types | Intuitive interface to build complex types | Python Types Guide → |
| Applications | Encapsulate templates and types into functions | Application Guide → |
About .txt
Outlines is developed and maintained by .txt, a company dedicated to making LLMs more reliable for production applications.
Our focus is on advancing structured generation technology through:
- 🧪 Cutting-edge Research: We publish our findings on structured generation
- 🚀 Enterprise-grade solutions: You can license our enterprise-grade libraries.
- 🧩 Open Source Collaboration: We believe in building in public and contributing to the community
Follow us on Twitter or check out our blog to stay updated on our latest work in making LLMs more reliable.
Community
- 💡 Have an idea? Come chat with us on Discord
- 🐞 Found a bug? Open an issue
- 🧩 Want to contribute? Consult our contribution guide.
Cite Outlines
@article{willard2023efficient,
title={Efficient Guided Generation for Large Language Models},
author={Willard, Brandon T and Louf, R{\'e}mi},
journal={arXiv preprint arXiv:2307.09702},
year={2023}
}Overview (`examples/beam-cloud/README.md`)
Deploy Outlines on Beam
- Create an account here and install the Beam SDK
- Download the
app.pyfile to your computer - Deploy it as a serverless API by running:
beam deploy app.py:predict
Index (`docs/index.md`)
title: Welcome to Outlines!
hide:
- navigation
LLMs are powerful but their outputs are unpredictable. Most solutions attempt to fix bad outputs after generation using parsing, regex, or fragile code that breaks easily.
Outlines guarantees structured outputs during generation — directly from any LLM.
- Works with any model - Same code runs across OpenAI, Ollama, vLLM, and more
- Simple integration - Just pass your desired output type:
model(prompt, output_type) - Guaranteed valid structure - No more parsing headaches or broken JSON
- Provider independence - Switch models without changing code
- Rich structure definition - Use Json Schema, regular expressions or context-free grammars
🚀 Building the future of structured generation
We're working with select partners to develop new interfaces to structured generation.
⚡ Try the Dottxt API
Want guaranteed structured generation without running your own models? The Dottxt API delivers 100% schema-compliant outputs via a simple REST API — no GPU, no setup.
Request API access →{ .md-button .md-button--primary }
🔍 Audit your schema
Share one schema and we show you what breaks under generation, the constraints that fix it, and compliance rates before and after. Need XML, FHIR, custom schemas or grammars? Let's talk.
Sign up →{ .md-button }
See it in action
from pydantic import BaseModel
from typing import Literal
import outlines
import openai
class Customer(BaseModel):
name: str
urgency: Literal["high", "medium", "low"]
issue: str
client = openai.OpenAI()
model = outlines.from_openai(client, "gpt-4o")
customer = model(
"Alice needs help with login issues ASAP",
Customer
)
# ✓ Always returns valid Customer object
# ✓ No parsing, no errors, no retriesQuick install
pip install outlinesFeatures
- :material-shield-check: Reliable - Guaranteed schema compliance -- always valid JSON.
- :material-puzzle: Feature-rich - Supports a large proportion of the JSON Schema spec, along with regex and context-free grammars.
- :material-lightning-bolt: Fast - Microseconds of overhead vs seconds of retries. Compilation happens once, not every request.
- :material-lightbulb: Simple - Outlines is a low-abstraction library. Write code the way you normally do with LLMs. No agent frameworks needed.
Supported inference APIs, libraries & servers
Who is using Outlines?
Hundreds of organisations and the main LLM serving frameworks (vLLM, TGI, LoRAX, xinference, SGLang) use Outlines. Prominent companies and organizations that use Outlines include:
Organizations are included either because they use Outlines as a dependency in a public repository, or because of direct communication between members of the Outlines team and employees at these organizations.
Still not convinced, read what people say about us. And make sure to take a look at what the community is building!
Outlines people
Outlines would not be what it is today without a community of dedicated developers:
About .txt
Outlines is built with ❤️ by .txt.
.txt solves the critical problem of reliable structured output generation for large language models. Our commercially-licensed libraries ensure 100% compliance with JSON Schema, regular expressions and context-free grammars while adding only microseconds of latency. Unlike open-source alternatives, we offer superior reliability, performance, and enterprise support.
Acknowledgements
Outlines was originally developed at @NormalComputing by @remilouf and @BrandonTWillard. It is now maintained by .txt.
Community Index (`docs/community/index.md`)
Community
Outlines exists for a community of users who believe software doesn't need to be complicated. Who share the same passion for Large Language Models but don't want to compromise on robustness. Together, we are bringing these powerful models back to the world of software.
Connect on Discord
The Outlines community lives on our Discord server. There you can ask questions, share ideas or just chat with people like you. Don't be a stranger and join us.
Examples Index (`docs/examples/index.md`)
Examples
This part of the documentation provides a few cookbooks that you can browse to get acquainted with the library and get some inspiration about what you could do with structured generation. Remember that you can easily change the model that is being used!
- Classification: Classify customer requests.
- Named Entity Extraction: Extract information from pizza orders.
- Dating Profiles: Build dating profiles from descriptions using prompt templating and JSON-structured generation.
- Chain Of Density: Summarize documents using chain of density prompting and JSON-structured generation.
- Playing Chess: Make Phi-3 Mini play chess against itself using regex-structured generation.
- SimToM: Improve LLMs' Theory of Mind capabilities with perspective-taking prompting and JSON-structured generation.
- Q&A with Citations: Answer questions and provide citations using JSON-structured generation.
- Knowledge Graph Generation: Generate a Knowledge Graph from unstructured text using JSON-structured generation.
- Structured Generation Workflow:
- Chain Of Thought (CoT): Generate a series of intermediate reasoning steps using regex-structured generation.
- ReAct Agent: Build an agent with open weights models using regex-structured generation.
- Structured Generation from PDFs: Use Outlines with vision-language models to read PDFs and produce structured output.
- Earnings reports to CSV: Extract data from earnings reports to CSV using regex-structured generation.
- Receipt Digitization: Extract information from a picture of a receipt using structured generation.
- Extract Events Details:
Run Outlines on the cloud:
Features Index (`docs/features/index.md`)
Features
This section presents in details the different features of Outlines.
Core Concepts
Utilities
Advanced
Features Models Index (`docs/features/models/index.md`)
title: Models
Models
Overview
Outlines models are objects that wrap an inference client or engine. Models provide a standardized interface to generate structured text.
All Outlines model classes have an associated loader function to facilitate initializing a model instance. The name of this function is from_ plus the name of the model in lower-case letters. For instance, Outlines has a Transformers model and an associated from_transformers loader function. The parameters to load a model are specific to each provider, please consult the documentation of the model you want to use for more information.
After having created a model instance, you can either directly call it to generate text or first create a reusable generator that you would then call.
The input you must provide to a model to generate text can be a simple text prompt or a vision or chat input for models that support them. See the model inputs section for more information on model inputs formats.
In all cases, you can provide an output_type to constrain the format of the generation output. See the output types section for more information on constrained generation.
For instance:
from outlines import from_transformers, Generator
import transformers
# Create a model
model = from_transformers(
transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)
# Call it directly
response = model("How many countries are there in the world", max_new_tokens=20)
print(response) # 'There are 200 countries in the world.'
# Call it directly with an output_type
response = model("How many countries are there in the world", int, max_new_tokens=20)
print(response) # '200'
# Create a generator first and then call it
generator = Generator(model, int)
response = generator("How many countries are there in the world")
print(response) # '200'Some models support streaming through a stream method. It takes the same argument as the __call__ method, but returns an iterator instead of a string.
For instance:
from outlines import from_openai, Generator
import openai
# Create the model
model = from_openai(
openai.OpenAI(),
"gpt-4o"
)
# Stream the response
for chunk in model.stream("Tell a short story about a cat.", max_tokens=50):
print(chunk) # 'This...'Additionally, some models support batch processing through a batch method. It's similar to the __call__ method, but takes a list of prompts instead of a single prompt and returns a list of strings.
For instance:
from outlines import from_transformers, Generator
import transformers
# Create a model
model = from_transformers(
transformers.AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
transformers.AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)
# Call it directly
response = model.batch(["What's the capital of Latvia?", "What's the capital of Estonia?"], max_new_tokens=20)
print(response) # ['Riga', 'Tallinn']Features Matrix
In alphabetical order:
| Anthropic | Dottxt | Gemini | LlamaCpp | LM Studio | MLXLM | Mistral | Ollama | OpenAI | SGLang | TGI | Transformers | Transformers MultiModal | VLLM | VLLMOffline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Output Types | |||||||||||||||
| Simple Types | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| JSON Schema | ❌ | ✅ | 🟠 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Multiple Choice | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Regex | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Grammar | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | 🟠 | ❌ | ✅ | ✅ | ✅ | ✅ |
| Generation Features | |||||||||||||||
| Async | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
| Streaming | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
| Vision | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Batching | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
Model Types
Models can be divided into two categories: local models and server-based models.
In the case of local models, the text generation happens within the inference library object used to instantiate the model. This gives Outlines direct access to the generation process (through a logits processor) and means all structured generation output types are available.
The local models available are the following:
- LlamaCpp
- MLXLM
- Transformers
- TransformersMultiModal
- VLLMOffline
In the case of server-based models, the model is initialized with a client that sends a request to a server that is in charge of the actual text generation. As a result, we have limited control over text generation and some output types are not supported. The server on which the text generation happens can either be remote (with OpenAI or Anthopic for instance) or local (with SGLang for instance).
The server-based models available are the following:
- Anthropic
- Dottxt
- Gemini
- LM Studio
- Mistral
- Ollama
- OpenAI
- SgLang
- TGI
- VLLM
Some models have an async version. To use them, just pass the async version of the provider object to their loading function. It will then return a Async<ModelName> instance with the same methods and features as the regular sync instance.
For instance:
from outlines import from_tgi
from huggingface_hub import AsyncInferenceClient
model = from_tgi(
AsyncInferenceClient("http://localhost:8000/v1")
)
print(type(model)) # outlines.models.tgi.AsyncTGIThe models that have an async version are the following:
- LM Studio
- Mistral
- Ollama
- OpenAI
- SgLang
- TGI
- VLLM
Core Concepts (`docs/core_concepts.md`)
title: Core concepts
Core concepts
Coming soon. This will document various concepts at a high level, so users can understand Outlines before diving into specific implementations.
- Constrained decoding, tokens, and the basics of logit biasing
- Different ways to define output structure (regex, JSON schema, Pydantic models, context-free grammars)
- How finite state machines are used to guarantee output structure
Generator,Application,Template,- Prompt engineering vs. structured generation
Community Contribute (`docs/community/contribute.md`)
title: Contribute
--8<-- "CONTRIBUTING.md"
Community Examples (`docs/community/examples.md`)
Community projects and articles
Publishing examples and articles about Outlines are a meaningful way to contribute to the community. Here is a list of projects we are aware of. Drop us a line if we forgot yours!
MMSG is a Python library for generating interleaved text and image content in a structured format you can directly pass to downstream APIs.
Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report shows that Structured Generation can outperform finetuning, and maybe even multimodality, in document-image understanding tasks as part of CVPR's 2nd MMFM Challenge.
Chess LLM Arena is a HuggingFace Space where you can make LLMs compete in a chess match.
LLM Data Gen is a HuggingFace Space that generates synthetic dataset files in JSONLines format.
Fast, High-Fidelity LLM Decoding with Regex Constraints presents an efficient alternative to Outlines's structured generation.
gigax is an Open-Source library that allows to create real-time LLM-powered NPCs for video games.
Improving Prompt Consistency with Structured Generations shows how structured generation can improve consistency of evaluation runs by reducing sensitivity to changes in prompt format.
AskNews is a news curation service processing 300k news articles per day in a structured way, with Outlines.
Community Feedback (`docs/community/feedback.md`)
title: Feedback
Feedback
If Outlines has been helpful to you, let us know on Discord or give us a shoutout on Twitter! It's always heartwarming ❤️
I am once again reminding you that structured extraction using LLMs is going to transform every single industry in the next 10 years https://t.co/xQ3tcWnrZ8— Sam Hogan (@0xSamHogan) April 17, 2024
outline's growth is insane, using is an understatement! https://t.co/rHCNWhZdCs— jason liu (@jxnlco) April 17, 2024
Outlines is an amazing lib and more popular than @remilouf’s modesty will admit. https://t.co/DfHbMPIlX1 https://t.co/mDHIWJrD0C— Delip Rao e/σ (@deliprao) April 18, 2024
Impressive implementation of a true regex / json / grammar guided text generation pic.twitter.com/RX5RVYaVIx— Rohan Paul (@rohanpaul_ai) December 30, 2023
Most underrated Github Repo in AI + LLM JSON guided Generation: https://t.co/lSB8KIet1H— 🎙Jean-Louis Queguiner (@JiliJeanlouis) December 18, 2023
Nice and useful. https://t.co/LX72AE0lgt— Dan Roy (@roydanroy) August 15, 2023
HUGE dub for open source AI https://t.co/bYKuiEUZ1j— kenneth 🖇 (@k3nnethfrancis) August 15, 2023
This is amazing - glad to see more outp guidance modules! Will try this out soon I'm wondering how they translate from regex automatons to token boundariesAlso why Open Source will succeed. Even today I don't see any guided output functionality from the big providers. https://t.co/Ity2H25Klf— Hrishi (@hrishioa) August 14, 2023
Outlines - a library to help LLM developers guide text generation in a fast and reliable way."Provides generation methods that guarantee that the output will match a regular expressions, or follow a JSON schema."Need to check this out. Reliable JSON output is a common use… pic.twitter.com/Bkbh8vKogN— elvis (@omarsar0) August 14, 2023
Woah this is cool! Makes open source models more usable.Give any LLM Function Call capability (and more) with Outlines: https://t.co/PtPykR5ZGR https://t.co/RRQjWHnIxv pic.twitter.com/BwNnH8SMwv— Yohei (@yoheinakajima) August 14, 2023
This is awesome! Being able to guarantee the output's structure unblocks so many applications. This is a great milestone and a fundamental building block for more advanced AI apps. https://t.co/WdwMOc7hE8— Guilherme Castro (@skastr052) August 15, 2023
Juggling with the unpredictable outputs of ChatGPT API lately while building my product. 😓 Tried prompt engineering to channel its wisdom into a neat JSON, but it's like asking a cat to fetch. 🐱Luckily, stumbled upon "Outlines" – looks like a promising way to tame the LLM… pic.twitter.com/oYQ6q8exAS— Charlie (@14435635Sun) August 15, 2023
A complex system of LLM input-outputs interacting with non-LLM agents and models benefits immeasurably from structured outputs. The outlines package saves so much time, https://t.co/NhVQ6NpKDR— Amir Sani (@amirsani) November 26, 2023