README Copy Section

Additional documentation

Consult the main README for general information about the project.
These are advanced topics that are not necessary for a basic deployment.

- Deploying:
- Troubleshooting deployment
- Debugging the app on App Service
- Deploying with azd: deep dive and CI/CD
- Deploying with existing Azure resources
- Deploying from a free account
- Enabling optional features
- All features
- Login and access control
- Multimodal
- Private endpoints
- Agentic retrieval
- Sharing deployment environments
- Local development
- Customizing the app
- App architecture
- HTTP Protocol
- Data ingestion
- Evaluation
- Safety evaluation
- Monitoring with Application Insights
- Productionizing
- Alternative RAG chat samples

---

Agentic Retrieval Copy Section

RAG chat: Using agentic retrieval

This repository includes an optional feature that uses agentic retrieval from Azure AI Search to find the most relevant content given a user's conversation history. The agentic retrieval feature uses a LLM to analyze the conversation and generate multiple search queries to find relevant content. This can improve the quality of the responses, especially for complex or multi-faceted questions.

Deployment Copy Chapter

1. Enable agentic retrieval:

Set the azd environment variable to enable the agentic retrieval feature:

azd env set USE_AGENTIC_KNOWLEDGEBASE true

2. (Optional) Customize the agentic retrieval model

You can configure which model agentic retrieval uses. By default, gpt-5.4 is used.

To change the model, set the following environment variables appropriately:

azd env set AZURE_OPENAI_KNOWLEDGEBASE_DEPLOYMENT knowledgebase
   azd env set AZURE_OPENAI_KNOWLEDGEBASE_MODEL gpt-5.4
   azd env set AZURE_OPENAI_KNOWLEDGEBASE_MODEL_VERSION 2026-03-05

You can only change it to one of the supported models.

3. (Optional) Choose the default retrieval reasoning effort

Agentic retrieval can run in minimal, low, or medium reasoning modes. The default is minimal, which keeps token usage and latency low. Because minimal disables the Azure AI Search LLM query-planning features (query expansion and knowledge-source selection), the app first rewrites the conversation into a single search intent and requests extractiveData.

Override the default by setting the following environment variable:

azd env set AZURE_SEARCH_KNOWLEDGEBASE_RETRIEVAL_REASONING_EFFORT low

Use minimal for the app's single-intent retrieval flow, low for Azure AI Search query planning and expansion, or medium for the most exhaustive (and most expensive) retrieval plans. Explicit deployment and Developer settings overrides continue to take precedence over the default.

4. (Optional) Enable web or SharePoint knowledge sources

By default, agentic retrieval only searches the documents in your search index. You can optionally enable additional knowledge sources:

Web source: Enables searching the public web for information.

azd env set USE_WEB_SOURCE true
   azd env set AZURE_SEARCH_KNOWLEDGEBASE_RETRIEVAL_REASONING_EFFORT low

> [!IMPORTANT]
> Web Knowledge Source and answer synthesis are not supported with minimal, which requires extractiveData. When USE_WEB_SOURCE=true, explicitly set AZURE_SEARCH_KNOWLEDGEBASE_RETRIEVAL_REASONING_EFFORT to low or medium.
>
> Web source requires the agent to use answer synthesis mode, which disables certain UI customizations including streaming, follow-up questions, and LLM parameter options.
> ⚠️ The Microsoft Data Protection Addendum doesn't apply to data sent to Web Knowledge Source. Learn more in the Web Knowledge source documentation

SharePoint source: Enables searching SharePoint documents. Requires authentication to be enabled and uses the logged-in user's token via on-behalf-of flow.

azd env set USE_SHAREPOINT_SOURCE true

> [!NOTE]
> SharePoint source requires that users have a Microsoft Copilot license.
> See licensing requirements for the Sharepoint knowledge source.

These sources can be used independently or together. When enabled, the agentic retrieval agent will search all configured sources and merge results based on the configured merge strategy.

5. Update the infrastructure and application:

Execute azd up to provision the infrastructure changes (only the new model, if you ran up previously) and deploy the application code with the updated environment variables. The post-provision script will configure Azure AI Search with a Knowledge agent pointing at the search index.

6. Try out the feature:

Open the web app and start a new chat. Agentic retrieval will be used to find all sources.

7. Review the query plan

Agentic retrieval uses additional billed tokens behind the scenes for the planning process.
To see the token usage, select the lightbulb icon on a chat answer. This will open the "Thought process" tab, which shows the amount of tokens used by and the queries produced by the planning process

---

Appservice Copy Section

RAG chat: Debugging the app on App Service

When you run azd up or azd deploy, it deploys your application to App Service,
and displays the deployed endpoint in the console.

If you encounter an error with that deployed app, you can debug the deployment using the tips below.

- Debugging failed Azure App Service deployments
- Checking the deployment logs for errors
- Checking the app logs for errors
- Checking Azure Monitor for errors
- Configuring log levels

Debugging failed Azure App Service deployments Copy Chapter

If you see a 500 error upon visiting your app after deployment, something went wrong
during either the deployment or the server start script.

We recommend always waiting 10 minutes, to give the server time to properly startup.

If you still see a 500 error after 10 minutes:

1. Check the deployment logs
2. Look for errors in the app logs
3. Look for errors in Azure Monitor

Checking the deployment logs for errors Copy Chapter

In the Azure portal, navigate to your App Service.

Select _Deployment Center_ from the side navigation menu, then select _Logs_.
You should see a timestamped list of recent deploys:

Check whether the status of the most recent deploy is "Success (Active)" or "Failed". If it's success, the deployment logs might still reveal issues, and if it's failed, the logs should certainly reveal the issue.

Click the commit ID to open the logs for the most recent deploy. First scroll down to see if any errors or warnings are reported at the end. This is what you'll hopefully see if all went well:

Now scroll back up to find the timestamp with the label "Running oryx build".
Oryx is the open source tool that builds apps for App Service, Functions, and other platforms, across all the supported MS languages. Click the _Show logs_ link next to that label. That will pop open detailed logs at the bottom. Scroll down.

<details>
<summary>Expand to see the logs for a successful Oryx build for the application.</summary>

/ Detailed source-code truncated for AI context efficiency. /

</details>

Look for these important steps in the Oryx build:

- _Detected following platforms: python: 3.11.7_
That should match your runtime in the App Service configuration.
- _Running pip install..._
That should install all the requirements in your requirements.txt - if it didn't find your requirements.txt, then you won't see the packages installed.

If you see all those steps in the Oryx build, then that's a good sign that the build went well, and you can move on to checking the App Service logs.

Checking the app logs for errors Copy Chapter

Select _Advanced Tools_ from the side nav:

Select _Go_ to open the Kudu website.

When the Kudu website loads, find the _Current Docker Logs_ link and select _Download as zip_ next to it:

In the downloaded zip file, find the filename that starts with the most recent date and ends with "_default_docker.log":

Open that file to see the full logs, with the most recent logs at the bottom.

<details>
<summary>Here are the full logs for the app successfully starting:</summary>

/ Detailed source-code truncated for AI context efficiency. /

</details>

A few notable logs:

- 2024-02-08T19:30:33.441385332Z Site's appCommandLine: python3 -m gunicorn main:app
This log indicates that App Service was correctly configured with a custom startup command to run the app.
- [2024-02-08 19:31:11 +0000] [75] [INFO] Starting gunicorn 20.1.0
That's the start of the gunicorn server serving the app.
- 2024-02-08T19:32:20.726942614Z [2024-02-08 19:32:20 +0000] [77] [INFO] Application startup complete.
At this point, the app has started successfully.

If you do not see any errors in those logs, then the app should be running successfully. If you do see errors, then try looking in Azure Monitor.

Checking Azure Monitor for errors Copy Chapter

By default, deployed apps use Application Insights to trace and log errors. (If you explicitly opted out of Application Insights, then you won't have this feature.)

In the Azure Portal, navigate to the Application Insights for your app.

To see any exceptions and server errors, navigate to the _Investigate -> Failures_ blade and browse through the exceptions.

Configuring log levels Copy Chapter

By default, the deployed app only logs messages from packages with a level of WARNING or higher,
but logs all messages from the app with a level of INFO or higher.

These lines of code in app/backend/app.py configure the logging level:

Set root level to WARNING to avoid seeing overly verbose logs from SDKS

logging.basicConfig(level=logging.WARNING)
Set the app logger level to INFO by default

default_level = "INFO"
app.logger.setLevel(os.getenv("APP_LOG_LEVEL", default_level))

To change the default level, either change default_level or set the APP_LOG_LEVEL environment variable
to one of the allowed log levels:
DEBUG, INFO, WARNING, ERROR, CRITICAL.

If you need to log in a route handler, use the the global variable current_app's logger:

async def chat():
    current_app.logger.info("Received /chat request")

Otherwise, use the logging module's root logger:

logging.info("System message: %s", system_message)

If you're having troubles finding the logs in App Service, read the section above on checking app logs or watch this video about viewing App Service logs.

---

Architecture Copy Section

RAG Chat: Application Architecture

This document provides a detailed architectural overview of this application, a Retrieval Augmented Generation (RAG) application that creates a ChatGPT-like experience over your own documents. It combines Azure OpenAI Service for AI capabilities with Azure AI Search for document indexing and retrieval.

For getting started with the application, see the main README.

Architecture Diagram Copy Chapter

The following diagram illustrates the complete architecture including user interaction flow, application components, and Azure services:

/ Detailed source-code truncated for AI context efficiency. /

Chat Query Flow Copy Chapter

The following sequence diagram shows how a user query is processed:

sequenceDiagram
    participant U as User
    participant F as Frontend
    participant B as Backend API
    participant S as Azure AI Search
    participant O as Azure OpenAI
    participant Bl as Blob Storage
    U->>F: Enter question
    F->>B: POST /chat with query
    B->>S: Search for relevant documents
    S-->>B: Return search results with citations
    B->>O: Send query + context to GPT model
    O-->>B: Return AI response
    B->>Bl: Log interaction (optional)
    B-->>F: Return response with citations
    F-->>U: Display answer with sources

Document Ingestion Flow Copy Chapter

The following diagram shows how documents are processed and indexed:

sequenceDiagram
    participant D as Documents
    participant Bl as Blob Storage
    participant P as PrepDocs Script
    participant DI as Document Intelligence
    participant O as Azure OpenAI
    participant S as Azure AI Search
    D->>Bl: Upload documents
    P->>Bl: Read documents
    P->>DI: Extract text and layout
    DI-->>P: Return extracted content
    P->>P: Split into chunks
    P->>O: Generate embeddings
    O-->>P: Return vector embeddings
    P->>S: Index documents with embeddings
    S-->>P: Confirm indexing complete

Key Components Copy Chapter

Frontend (React/TypeScript) Copy Section

- Chat Interface: Main conversational UI
- Settings Panel: Configuration options for AI behavior
- Citation Display: Shows sources and references
- Authentication: Optional user login integration

Backend (Python) Copy Section

- API Layer: RESTful endpoints for chat, search, and configuration. See HTTP Protocol for detailed API documentation.
- Approach Patterns: Different strategies for processing queries
- ChatReadRetrieveRead: Multi-turn conversation with retrieval
- Authentication: Optional integration with Azure Active Directory

Azure Services Integration Copy Section

- Azure OpenAI: Powers the conversational AI capabilities
- Azure AI Search: Provides semantic and vector search over documents
- Azure Blob Storage: Stores original documents and processed content
- Application Insights: Provides monitoring and telemetry

Optional Features Copy Chapter

The architecture supports several optional features that can be enabled. For detailed configuration instructions, see the optional features guide:

- GPT-4 with Vision: Process image-heavy documents
- Speech Services: Voice input/output capabilities
- Chat History: Persistent conversation storage in Cosmos DB
- Authentication: User login and access control
- Private Endpoints: Network isolation for enhanced security

Deployment Options Copy Chapter

The application can be deployed using:

- Azure Container Apps (default): Serverless container hosting
- Azure App Service: Traditional PaaS hosting option. See the App Service hosting guide for detailed instructions.

Both options support the same feature set and can be configured through the Azure Developer CLI (azd).

---

Azd Copy Section

RAG chat: Deploying with the Azure Developer CLI

This guide includes advanced topics that are not necessary for a basic deployment. If you are new to the project, please consult the main README for steps on deploying the project.

📺 Watch: Deployment of your chat app

* How does azd up work?
* Configuring continuous deployment
* GitHub actions
* Azure DevOps

How does azd up work? azd up work?')" title="Copy chapter prompt for LLMs"> Copy Chapter

The azd up command comes from the Azure Developer CLI, and takes care of both provisioning the Azure resources and deploying code to the selected Azure hosts.

The azd up command uses the azure.yaml file combined with the infrastructure-as-code .bicep files in the infra/ folder. The azure.yaml file for this project declares several "hooks" for the prepackage step and postprovision steps. The up command first runs the prepackage hook which installs Node dependencies and builds the React.JS-based JavaScript files. It then packages all the code (both frontend and backend) into a zip file which it will deploy later.

Next, it provisions the resources based on main.bicep and main.parameters.json. At that point, since there is no default value for the OpenAI resource location, it asks you to pick a location from a short list of available regions. Then it will send requests to Azure to provision all the required resources. With everything provisioned, it runs the postprovision hook to process the local data and add it to an Azure AI Search index.

Finally, it looks at azure.yaml to determine the Azure host and uploads the zip to Azure App Service. The azd up command is now complete, but it may take another 5-10 minutes for the App Service app to be fully available and working, especially for the initial deploy.

Related commands are azd provision for just provisioning (if infra files change) and azd deploy for just deploying updated app code.

Configuring continuous deployment Copy Chapter

This repository includes both a GitHub Actions workflow and an Azure DevOps pipeline for continuous deployment with every push to main. The GitHub Actions workflow is the default, but you can switch to Azure DevOps if you prefer.

More details are available in Learn.com: Configure a pipeline and push updates

GitHub actions Copy Section

After you have deployed the app once with azd up, you can enable continuous deployment with GitHub Actions.

Run this command to set up a Service Principal account for CI deployment and to store your azd environment variables in GitHub Actions secrets:

azd pipeline config

You can trigger the "Deploy" workflow manually from your GitHub actions, or wait for the next push to main.

If you change your azd environment variables at any time (via azd env set or as a result of provisioning), re-run that command in order to update the GitHub Actions secrets.

Azure DevOps Copy Section

After you have deployed the app once with azd up, you can enable continuous deployment with Azure DevOps.

Run this command to set up a Service Principal account for CI deployment and to store your azd environment variables in GitHub Actions secrets:

azd pipeline config --provider azdo

If you change your azd environment variables at any time (via azd env set or as a result of provisioning), re-run that command in order to update the GitHub Actions secrets.

---

Azure App Service Copy Section

RAG chat: Deploying on Azure App Service

Due to a limitation of the Azure Developer CLI (azd), there can be only one host option in the azure.yaml file.
By default, host: containerapp is used and host: appservice is commented out.

To deploy to Azure App Service, please follow the following steps:

1. Comment out host: containerapp and uncomment host: appservice in the azure.yaml file.

2. Login to your Azure account:

azd auth login

3. Create a new azd environment to store the deployment parameters:

azd env new

Enter a name that will be used for the resource group.
This will create a new folder in the .azure folder, and set it as the active environment for any calls to azd going forward.

4. Set the deployment target to appservice:

azd env set DEPLOYMENT_TARGET appservice

5. (Optional) This is the point where you can customize the deployment by setting other azd environment variables, in order to use existing resources, enable optional features (such as auth or vision), or deploy to free tiers.
6. Provision the resources and deploy the code:

azd up

This will provision Azure resources and deploy this sample to those resources, including building the search index based on the files found in the ./data folder.

Important: Beware that the resources created by this command will incur immediate costs, primarily from the AI Search resource. These resources may accrue costs even if you interrupt the command before it is fully executed. You can run azd down or delete the resources manually to avoid unnecessary spending.

---

Azure Container Apps Copy Section

RAG chat: Deploying on Azure Container Apps

Due to a limitation of the Azure Developer CLI (azd), there can be only one host option in the azure.yaml file.
By default, host: containerapp is used and host: appservice is commented out.

However, if you have an older version of the repo, you may need to follow these steps to deploy to Container Apps instead, or you can stick with Azure App Service.

To deploy to Azure Container Apps, please follow the following steps:

1. Comment out host: appservice and uncomment host: containerapp in the azure.yaml file.

2. Login to your Azure account:

azd auth login

3. Create a new azd environment to store the deployment parameters:

azd env new

Enter a name that will be used for the resource group.
This will create a new folder in the .azure folder, and set it as the active environment for any calls to azd going forward.

4. Set the deployment target to containerapps:

azd env set DEPLOYMENT_TARGET containerapps

5. (Optional) This is the point where you can customize the deployment by setting other azd1 environment variables, in order to use existing resources, enable optional features (such as auth or vision), or deploy to free tiers.
6. Provision the resources and deploy the code:

azd up

This will provision Azure resources and deploy this sample to those resources, including building the search index based on the files found in the ./data folder.

Important: Beware that the resources created by this command will incur immediate costs, primarily from the AI Search resource. These resources may accrue costs even if you interrupt the command before it is fully executed. You can run azd down or delete the resources manually to avoid unnecessary spending.

Customizing Workload Profile Copy Chapter

The default workload profile is Consumption. If you want to use a dedicated workload profile like D4, please run:

azd env set AZURE_CONTAINER_APPS_WORKLOAD_PROFILE D4

For a full list of workload profiles, please check the workload profile documentation.
Please note dedicated workload profiles have a different billing model than Consumption plan. Please check the billing documentation for details.

Private endpoints Copy Chapter

Private endpoints is still in private preview for Azure Container Apps and not supported for now.

---

Customization Copy Section

RAG chat: Customizing the chat app

📺 Watch: (RAG Deep Dive series) Customizing the app

Tip: We recommend using GitHub Copilot Agent mode when adding new features or making code changes. This project includes an AGENTS.md file that guides Copilot to generate code following project conventions.

This guide provides more details for customizing the RAG chat app.

- Using your own data
- Customizing the UI
- Customizing the backend
- Chat approach
- Improving answer quality
- Identify the problem point
- Improving OpenAI Responses API results
- Improving Azure AI Search results
- Evaluating answer quality

Using your own data Copy Chapter

The Chat App is designed to work with any PDF documents. The sample data is provided to help you get started quickly, but you can easily replace it with your own data. You'll want to first remove all the existing data, then add your own. See the data ingestion guide for more details.

Customizing the UI Copy Chapter

The frontend is built using React and Fluent UI components. The frontend components are stored in the app/frontend/src folder. To modify the page title, header text, example questions, and other UI elements, you can customize the app/frontend/src/locales/{en/es/fr/jp/it}/translation.json file for different languages(English is the default). The primary strings and labels used throughout the application are defined within these files.

Customizing the backend Copy Chapter

The backend is built using Quart, a Python framework for asynchronous web applications. The backend code is stored in the app/backend folder. The frontend and backend communicate over HTTP using JSON or streamed NDJSON responses. Learn more in the HTTP Protocol guide.

Chat approach Copy Section

Typically, the primary backend code you'll want to customize is the app/backend/approaches folder, which contains the code and prompts powering the RAG flow.

The RAG flow is implemented in chatreadretrieveread.py.

1. Query rewriting: It calls the OpenAI Responses API to turn the user question into a good search query, using the prompt from query_rewrite.system.jinja2 and tools from chat_query_rewrite_tools.json.
2. Search: It queries Azure AI Search for search results for that query (optionally using the vector embeddings for that query).
3. Answering: It then calls the OpenAI Responses API to answer the question based on the sources, using the prompts from chat_answer.system.jinja2 and chat_answer.user.jinja2. That call includes the past message history as well (or as many messages fit inside the model's token limit).

The prompts are currently tailored to the sample data since they start with "Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook." Modify the query_rewrite.system.jinja2, chat_answer.system.jinja2, and chat_answer.user.jinja2 prompts to match your data.

#### Chat with multimodal feature

If you followed the instructions in the multimodal guide to enable multimodal RAG,
there are several differences in the chat approach:

1. Query rewriting: Unchanged.
2. Search: For this step, it calculates a vector embedding for the user question using the Azure AI Vision vectorize text API, and passes that to the Azure AI Search to compare against the image embedding fields in the indexed documents. For each matching document, it downloads each associated image from Azure Blob Storage and converts it to a base 64 encoding.
3. Answering: When it combines the search results and user question, it includes the base 64 encoded images, and sends along both the text and images to the multimodal LLM. The model generates a response that includes citations to the images, and the UI renders the images when a citation is clicked.

The settings can be customized to disable calculating the image vector embeddings or to disable sending image inputs to the LLM, if desired.

#### Making settings overrides permanent

The UI provides a "Developer Settings" menu for customizing the approaches, like disabling semantic ranker or using vector search.
Those settings are passed in the "context" field of the request to the backend, and are not saved permanently.
However, if you find a setting that you do want to make permanent, there are two approaches:

1. Change the defaults in the frontend. You'll find the defaults in Chat.tsx. For example, this line of code sets the default retrieval mode to Hybrid:

const [retrievalMode, setRetrievalMode] = useState<RetrievalMode>(RetrievalMode.Hybrid);

You can change the default to Text by changing the code to:

const [retrievalMode, setRetrievalMode] = useState<RetrievalMode>(RetrievalMode.Text);

2. Change the overrides in the backend. Each of the approaches has a run method that takes a context parameter, and the first line of code extracts the overrides from that context. That's where you can override any of the settings. For example, to change the retrieval mode to text:

overrides = context.get("overrides", {})
    overrides["retrieval_mode"] = "text"

By changing the setting on the backend, you can safely remove the Developer Settings UI from the frontend, if you don't wish to expose that to your users.

Improving answer quality Copy Chapter

Once you are running the chat app on your own data and with your own tailored system prompt,
the next step is to test the app with questions and note the quality of the answers.
If you notice any answers that aren't as good as you'd like, here's a process for improving them.

Identify the problem point Copy Section

The first step is to identify where the problem is occurring. For example, if using the Chat tab, the problem could be:

1. OpenAI Responses API is not generating a good search query based on the user question
2. Azure AI Search is not returning good search results for the query
3. OpenAI Responses API is not generating a good answer based on the search results and user question

You can look at the "Thought process" tab in the chat app to see each of those steps,
and determine which one is the problem.

Improving OpenAI Responses API results Copy Section

If the problem is with the Responses API calls (steps 1 or 3 above), you can try changing the relevant prompt.

Once you've changed the prompt, make sure you ask the same question multiple times to see if the overall quality has improved, and run an evaluation when you're satisfied with the changes. The Responses API can yield different results every time, even for a temperature of 0.0, but especially for a higher temperature than that (like our default of 0.3 for step 3).

You can also try changing the Responses API parameters, like temperature, to see if that improves results for your domain.

Improving Azure AI Search results Copy Section

If the problem is with Azure AI Search (step 2 above), the first step is to check what search parameters you're using. Generally, the best results are found with hybrid search (text + vectors) plus the additional semantic re-ranking step, and that's what we've enabled by default. There may be some domains where that combination isn't optimal, however. Check out this blog post which evaluates AI search strategies for a better understanding of the differences, or watch this RAG Deep Dive video on AI Search.

#### Configuring parameters in the app

You can change many of the search parameters in the "Developer settings" in the frontend and see if results improve for your queries. The most relevant options:

#### Configuring parameters in the Azure Portal

You may find it easier to experiment with search options with the index explorer in the Azure Portal.
Open up the Azure AI Search resource, select the Indexes tab, and select the index there.

Then use the JSON view of the search explorer, and make sure you specify the same options you're using in the app. For example, this query represents a search with semantic ranker configured:

{
  "search": "eye exams",
  "queryType": "semantic",
  "semanticConfiguration": "default",
  "queryLanguage": "en-us",
  "speller": "lexicon",
  "top": 3
}

You can also use the highlight parameter to see what text is being matched in the content field in the search results.

{
    "search": "eye exams",
    "highlight": "content"
    ...
}

The search explorer works well for testing text, but is harder to use with vectors, since you'd also need to compute the vector embedding and send it in. It is probably easier to use the app frontend for testing vectors/hybrid search.

#### Other approaches to improve search results

Here are additional ways for improving the search results:

- Adding additional metadata to the "content" field, like the document title, so that it can be matched in the search results. Modify searchmanager.py to include more text in the content field.
- Making additional fields searchable by the full text search step. For example, the "sourcepage" field is not currently searchable, but you could make that into a SearchableField with searchable=True in searchmanager.py. A change like that requires re-building the index.
- Using function calling to search by particular fields, like searching by the filename. See this blog post on function calling for structured retrieval.
- Using a different splitting strategy for the documents, or modifying the existing ones, to improve the chunks that are indexed. You can find the currently available splitters in textsplitter.py.

Evaluating answer quality Copy Section

Once you've made changes to the prompts or settings, you'll want to rigorously evaluate the results to see if they've improved. Follow the evaluation guide to learn how to run evaluations, review results, and compare answers across runs.

---

Data Ingestion Copy Section

RAG chat: Data ingestion

The azure-search-openai-demo project can set up a full RAG chat app on Azure AI Search and OpenAI so that you can chat on custom data, like internal enterprise data or domain-specific knowledge sets. For full instructions on setting up the project, consult the main README, and then return here for detailed instructions on the data ingestion component.

The chat app provides two ways to ingest data: manual ingestion and cloud ingestion. Both approaches use the same code for processing the data, but the manual ingestion runs locally while cloud ingestion runs in Azure Functions as Azure AI Search custom skills.

- Supported document formats
- Ingestion stages
- Document extraction
- Figure processing
- Text processing
- Local ingestion
- Categorizing data for enhanced search
- Indexing additional documents
- Removing documents
- Cloud ingestion
- Enabling cloud ingestion
- Indexer architecture
- Indexing of additional documents
- Removal of documents
- Scheduled indexing
- Debugging tips

Supported document formats Copy Chapter

In order to ingest a document format, we need a tool that can turn it into text. By default, the manual indexing uses Azure Document Intelligence (DI in the table below), but we also have local parsers for several formats. The local parsers are not as sophisticated as Azure Document Intelligence, but they can be used to decrease charges.

| Format | Manual indexing | Integrated Vectorization |
| ------ | ------------------------------------ | ------------------------ |
| PDF | Yes (DI or local with PyPDF) | Yes |
| HTML | Yes (DI or local with BeautifulSoup) | Yes |
| DOCX, PPTX, XLSX | Yes (DI) | Yes |
| Images (JPG, PNG, BPM, TIFF, HEIFF)| Yes (DI) | Yes |
| TXT | Yes (Local) | Yes |
| JSON | Yes (Local) | Yes |
| CSV | Yes (Local) | Yes |

Ingestion stages Copy Chapter

The ingestion pipeline consists of three main stages that transform raw documents into searchable content in Azure AI Search. These stages apply to both local ingestion and cloud ingestion.

Document extraction Copy Section

The first stage extracts text and structured content from source documents using parsers tailored to each file format. For PDF, HTML, DOCX, PPTX, XLSX, and image files, the pipeline defaults to using Azure Document Intelligence to extract text, tables, and figures with layout information. Alternatively, local parsers like PyPDF and BeautifulSoup can be used to reduce costs for simpler documents. For TXT, JSON, and CSV files, lightweight local parsers extract the content directly.

During extraction, tables are converted to HTML markup to preserve their structure, and figures (when multimodal is enabled) are identified with bounding boxes and placeholders.

The output from this stage is a list of pages, each containing the extracted text with embedded table HTML and figure placeholders like <figure id="fig1"></figure>.

Figure processing Copy Section

This stage is optional and only applies when the multimodal feature is enabled and the document itself has figures. See multimodal feature documentation for more details.

When multimodal support is enabled, figures extracted in the previous stage are enriched with descriptions and embeddings. Each figure is:

1. Cropped and saved: The figure image is cropped from the PDF using its bounding box coordinates and saved as a PNG file.
2. Described: A text description is generated using either Azure OpenAI's GPT-4 Vision model or Azure AI Content Understanding, depending on configuration.
3. Uploaded: The figure image is uploaded to Azure Blob Storage and assigned a URL.
4. Embedded (optional): If image embeddings are enabled, a vector embedding is computed for the figure using Azure AI Vision.

The output from this stage is enriched figure metadata, including the description text, storage URL, and optional embedding vector.

Text processing Copy Section

The final stage combines the extracted text with figure descriptions, splits the content into searchable chunks, and computes embeddings.

Figure merging Copy Section

First, figure placeholders in the page text are replaced with full HTML markup that includes the figure caption and generated description, creating a cohesive text narrative that incorporates visual content.

#### Chunking

Next, the combined text is split into chunks using a sentence-aware splitter that respects semantic boundaries. The default chunk size is approximately 1000 characters (roughly 400-500 tokens for English), with a 10% overlap between consecutive chunks to preserve context across boundaries. The splitter uses a sliding window approach, ensuring that sentences ending one chunk also start the next, which reduces the risk of losing important context at chunk boundaries.

Why chunk documents? While Azure AI Search can index full documents, chunking is essential for the RAG pattern because it limits the amount of information sent to OpenAI, which has token limits for context windows. By breaking content into focused chunks, the system can retrieve and inject only the most relevant pieces of text into the LLM prompt, improving both response quality and cost efficiency.

If needed, you can modify the chunking algorithm in app/backend/prepdocslib/textsplitter.py. For a deeper, diagram-rich explanation of how the splitter works (figures, recursion, merge heuristics, guarantees, and examples), see the text splitter documentation.

#### Embedding

Finally, if vector search is enabled, text embeddings are computed for each chunk using Azure OpenAI's embedding models (text-embedding-ada-002, text-embedding-3-small, or text-embedding-3-large). These embeddings are generated in batches for efficiency, with retry logic to handle rate limits.

Indexing Copy Section

The final step is to index the chunks into Azure AI Search. Each chunk is stored as a separate document in the search index, with metadata linking it back to the source file and page number. If vector search is enabled, the computed embeddings are also stored alongside the text, enabling efficient similarity searches during query time.

Here's an example of what a final indexed chunk document looks like:

{
    "id": "file-Northwind_Health_Plus_Benefits_Details_pdf-4E6F72746877696E645F4865616C74685F506C75735F42656E65666974735F44657461696C732E706466-page-0",
    "content": "# Zava\n\nNorthwind Health Plus Plan\n...",
    "category": null,
    "sourcepage": "Northwind_Health_Plus_Benefits_Details.pdf#page=1",
    "sourcefile": "Northwind_Health_Plus_Benefits_Details.pdf",
    "storageUrl": "https://std4gfbajn3e3yu.blob.core.windows.net/content/Northwind_Health_Plus_Benefits_Details.pdf",
    "embedding": [0.0123, -0.0456, ...]
}

If multimodal is enabled, that document will also include an "images" field and figure descriptions in the "content" field.

Local ingestion Copy Chapter

The prepdocs.py script is responsible for both uploading and indexing documents. The typical usage is to call it using scripts/prepdocs.sh (Mac/Linux) or scripts/prepdocs.ps1 (Windows), as these scripts will set up a Python virtual environment and pass in the required parameters based on the current azd environment. You can pass additional arguments directly to the script, for example scripts/prepdocs.ps1 --removeall. Whenever azd up or azd provision is run, the script is called automatically.

The script uses the following steps to index documents:

1. If it doesn't yet exist, create a new index in Azure AI Search.
2. Upload the PDFs to Azure Blob Storage.
3. Split the PDFs into chunks of text.
4. Upload the chunks to Azure AI Search. If using vectors (the default), also compute the embeddings and upload those alongside the text.

Enhancing search functionality with data categorization Copy Section

To enhance search functionality, categorize data during the ingestion process with the --category argument, for example scripts/prepdocs.ps1 --category ExampleCategoryName. This argument specifies the category to which the data belongs, enabling you to filter search results based on these categories.

After running the script with the desired category, ensure these categories are added to the 'Include Category' dropdown list. This can be found in the developer settings in Settings.tsx. The default option for this dropdown is "All". By including specific categories, you can refine your search results more effectively.

Indexing additional documents Copy Section

To upload more PDFs, put them in the data/ folder and run ./scripts/prepdocs.sh or ./scripts/prepdocs.ps1.

The prepdocs script writes an .md5 file with an MD5 hash of each file that gets uploaded. Whenever the prepdocs script is re-run, that hash is checked against the current hash and the file is skipped if it hasn't changed.

Removing documents Copy Section

You may want to remove documents from the index. For example, if you're using the sample data, you may want to remove the documents that are already in the index before adding your own.

To remove all documents, use ./scripts/prepdocs.sh --removeall or ./scripts/prepdocs.ps1 --removeall.

You can also remove individual documents by using the --remove flag. Open either scripts/prepdocs.sh or scripts/prepdocs.ps1 and replace /data/* with /data/YOUR-DOCUMENT-FILENAME-GOES-HERE.pdf. Then run scripts/prepdocs.sh --remove or scripts/prepdocs.ps1 --remove.

Cloud ingestion Copy Chapter

This project includes an optional feature to perform data ingestion in the cloud using Azure Functions as custom skills for Azure AI Search indexers. This approach offloads the ingestion workload from your local machine to the cloud, allowing for more scalable and efficient processing of large datasets.

Enabling cloud ingestion Copy Section

1. If you've previously deployed, delete the existing search index or create a new index. This feature cannot be used on existing index. In the newly created index schema, a new field 'parent_id' is added. This is used internally by the indexer to manage life cycle of chunks. Run this command to set a new index name:

azd env set AZURE_SEARCH_INDEX cloudindex

2. Run this command:

azd env set USE_CLOUD_INGESTION true

3. (Recommended) Increase the capacity for the embedding model to the maximum quota allowed for your region/subscription, so that the Azure Functions can generate embeddings without hitting rate limits:

azd env set AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY 400

4. Provision the new Azure Functions resources, deploy the function apps, and update the search indexer with:

azd up

5. That will upload the documents in the data/ folder to the Blob storage container, create the indexer and skillset, and run the indexer to ingest the data. You can monitor the indexer status from the portal.

6. When you have new documents to ingest, you can upload documents to the Blob storage container and run the indexer from the Azure Portal to ingest new documents.

Indexer architecture Copy Section

The cloud ingestion pipeline uses four Azure Functions as custom skills within an Azure AI Search indexer. Each function corresponds to a stage in the ingestion process. Here's how it works:

1. User uploads documents to Azure Blob Storage (content container)
2. Azure AI Search Indexer monitors the blob container and orchestrates processing
3. Custom skills process documents through three stages:
- Document Extractor (Skill #1): Extracts text and figure metadata from source documents
- Figure Processor (Skill #2): Enriches figures with descriptions and embeddings
- Shaper Skill (Skill #3): Built-in Azure AI Search skill that consolidates enriched data
- Text Processor (Skill #4): Combines text with enriched figures, chunks content, and generates embeddings
4. Azure AI Search Index receives the final processed chunks with embeddings

The functions are defined in the app/functions/ directory, and the custom skillset is configured in the app/backend/setup_cloud_ingestion.py script.

#### Document Extractor Function

- Implements the document extraction stage
- Emits markdown text with <figure id="..."> placeholders and figure metadata

#### Figure Processor Function

- Implements the figure processing stage
- Emits enriched figure metadata with descriptions, URLs, and embeddings

#### Shaper Skill

- Consolidates enrichments from the figure processor back into the main document context
- Required because Azure AI Search's enrichment tree isolates data by context
- The Shaper explicitly combines:
- Original pages array from document_extractor
- Enriched figures array with descriptions, URLs, and embeddings from figure_processor
- File metadata (file_name, storageUrl)
- Creates a consolidated_document object that the text processor can consume

#### Text Processor Function

- Implements the text processing stage (figure merging, chunking, embedding)
- Receives the consolidated document with enriched figures from the Shaper skill
- Emits search-ready chunks with figure references and embeddings

Indexing of additional documents Copy Section

To add additional documents to the index, first upload them to your data source (Blob storage, by default).
Then navigate to the Azure portal and run the indexer. The Azure AI Search indexer will identify the new documents and ingest them into the index.

Removal of documents Copy Section

To remove documents from the index, remove them from your data source (Blob storage, by default).
Then navigate to the Azure portal and run the indexer. The Azure AI Search indexer will take care of removing those documents from the index.

Scheduled indexing Copy Section

If you would like the indexer to run automatically, you can set it up to run on a schedule.

Debugging tips Copy Chapter

If you are not sure if a file successfully uploaded, you can query the index from the Azure Portal or from the REST API. Open the index and paste the queries below into the search bar.

To see all the filenames uploaded to the index:

{
  "search": "*",
  "count": true,
  "top": 1,
  "facets": ["sourcefile"]
}

To search for specific filenames:

{
  "search": "*",
  "count": true,
  "top": 1,
  "filter": "sourcefile eq 'employee_handbook.pdf'",
  "facets": ["sourcefile"]
}

---

Deploy Existing Copy Section

RAG chat: Deploying with existing Azure resources

If you already have existing Azure resources, or if you want to specify the exact name of new Azure Resource, you can do so by setting azd environment values.
You should set these values before running azd up. Once you've set them, return to the deployment steps.

* Resource group
* OpenAI resource
* Azure AI Search resource
* Azure App Service Plan and App Service resources
* Azure AI Vision resources
* Azure Document Intelligence resource
* Azure Speech resource
* Azure Storage Account

1. Run azd env set AZURE_RESOURCE_GROUP {Name of existing resource group}
1. Run azd env set AZURE_LOCATION {Location of existing resource group}

OpenAI resource Copy Chapter

Azure OpenAI Copy Section

Foundry account (kind: 'AIServices' with project management enabled) plus a Foundry

project, and deploys the models on that account. If you set AZURE_OPENAI_SERVICE below to

reuse an existing account (with OPENAI_HOST left as azure), the deployment still targets

to a Foundry (AIServices) account with project management enabled, and creates a Foundry

project inside it. If the existing account is a classic kind: 'OpenAI' account, this is

the fully bring-your-own path instead (OPENAI_HOST=azure_custom with AZURE_OPENAI_CUSTOM_URL),

1. Run azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}
1. Run azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}
1. Run azd env set AZURE_OPENAI_LOCATION {Location of existing OpenAI service}
1. Run azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}. Only needed if your chat deployment name is not the default 'gpt-5.4-mini'.
1. Run azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}. Only needed if your chat model is not the default 'gpt-5.4-mini'.
1. Run azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION {Version string for existing chat deployment}. Only needed if your chat deployment model version is not the default '2026-03-17'. You definitely need to change this if you changed the model.
1. Run azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_SKU {Name of SKU for existing chat deployment}. Only needed if your chat deployment SKU is not the default 'GlobalStandard'.
1. Run azd env set AZURE_OPENAI_EMB_DEPLOYMENT {Name of existing embedding deployment}. Only needed if your embeddings deployment is not the default 'embedding'.
1. Run azd env set AZURE_OPENAI_EMB_MODEL_NAME {Model name of existing embedding deployment}. Only needed if your embeddings model is not the default 'text-embedding-3-large'.
1. Run azd env set AZURE_OPENAI_EMB_DIMENSIONS {Dimensions for existing embedding deployment}. Only needed if your embeddings model is not the default 'text-embedding-3-large'.
1. Run azd env set AZURE_OPENAI_EMB_DEPLOYMENT_VERSION {Version string for existing embedding deployment}. If your embeddings deployment is one of the 'text-embedding-3' models, set this to the number 1.
1. This project does not use keys when authenticating to Azure OpenAI. However, if your Azure OpenAI service must have key access enabled for some reason (like for use by other projects), then run azd env set AZURE_OPENAI_DISABLE_KEYS false. The default value is true so you should only run the command if you need key access.

When you run azd up after and are prompted to select a value for openAiResourceGroupLocation, make sure to select the same location as the existing OpenAI resource group.

Openai.com OpenAI Copy Section

1. Run azd env set OPENAI_HOST openai
2. Run azd env set OPENAI_ORGANIZATION {Your OpenAI organization}
3. Run azd env set OPENAI_API_KEY {Your OpenAI API key}
4. Run azd up

You can retrieve your OpenAI key by checking your user page and your organization by navigating to your organization page.
Learn more about creating an OpenAI free trial at this link.
Do not check your key into source control.

When you run azd up after and are prompted to select a value for openAiResourceGroupLocation, you can select any location as it will not be used.

Azure AI Search resource Copy Chapter

1. Run azd env set AZURE_SEARCH_SERVICE {Name of existing Azure AI Search service}
1. Run azd env set AZURE_SEARCH_SERVICE_RESOURCE_GROUP {Name of existing resource group with ACS service}
1. If that resource group is in a different location than the one you'll pick for the azd up step,
then run azd env set AZURE_SEARCH_SERVICE_LOCATION {Location of existing service}
1. If the search service's SKU is not standard, then run azd env set AZURE_SEARCH_SERVICE_SKU {Name of SKU}. If you specify the free tier, then your app will no longer be able to use semantic ranker. You can switch between Basic, S1, S2, and S3 tiers, but you can't switch to or from Free, S3HD, L1, or L2. (See other possible SKU values)
1. If you have an existing index that is set up with all the expected fields, then run azd env set AZURE_SEARCH_INDEX {Name of existing index}. Otherwise, the azd up command will create a new index.

You can also customize the search service (new or existing) for non-English searches:

1. To configure the language of the search query to a value other than "en-US", run azd env set AZURE_SEARCH_QUERY_LANGUAGE {Name of query language}. (See other possible values)
1. To turn off the spell checker, run azd env set AZURE_SEARCH_QUERY_SPELLER none. Consult this table to determine if spell checker is supported for your query language.
1. To configure the name of the analyzer to use for a searchable text field to a value other than "en.microsoft", run azd env set AZURE_SEARCH_ANALYZER_NAME {Name of analyzer name}. (See other possible values)

> * Backend identity: Search Index Data Reader, Search Index Data Contributor

* Signed-in user (principalId): Search Index Data Reader, Search Index Data Contributor, Search Service Contributor

Azure App Service Plan and App Service resources Copy Chapter

1. Run azd env set AZURE_APP_SERVICE_PLAN {Name of existing Azure App Service Plan}
1. Run azd env set AZURE_APP_SERVICE {Name of existing Azure App Service}.
1. Run azd env set AZURE_APP_SERVICE_SKU {SKU of Azure App Service, defaults to B1}.

Azure AI Vision resources Copy Chapter

1. Run azd env set AZURE_VISION_SERVICE {Name of existing Azure AI Vision Service Name}
1. Run azd env set AZURE_VISION_RESOURCE_GROUP {Name of existing Azure AI Vision Resource Group Name}
1. Run azd env set AZURE_VISION_LOCATION {Name of existing Azure AI Vision Location}
1. Run azd env set AZURE_VISION_SKU {SKU of Azure AI Vision service, defaults to F0}

Azure Document Intelligence resource Copy Chapter

In order to support analysis of many document formats, this repository uses a preview version of Azure Document Intelligence (formerly Form Recognizer) that is only available in limited regions.
If your existing resource is in one of those regions, then you can re-use it by setting the following environment variables:

1. Run azd env set AZURE_DOCUMENTINTELLIGENCE_SERVICE {Name of existing Azure AI Document Intelligence service}
1. Run azd env set AZURE_DOCUMENTINTELLIGENCE_LOCATION {Location of existing service}
1. Run azd env set AZURE_DOCUMENTINTELLIGENCE_RESOURCE_GROUP {Name of resource group with existing service, defaults to main resource group}
1. Run azd env set AZURE_DOCUMENTINTELLIGENCE_SKU {SKU of existing service, defaults to S0}

Azure Speech resource Copy Chapter

1. Run azd env set AZURE_SPEECH_SERVICE {Name of existing Azure Speech service}
1. Run azd env set AZURE_SPEECH_SERVICE_RESOURCE_GROUP {Name of existing resource group with speech service}
1. If that resource group is in a different location than the one you'll pick for the azd up step,
then run azd env set AZURE_SPEECH_SERVICE_LOCATION {Location of existing service}
1. If the speech service's SKU is not "S0", then run azd env set AZURE_SPEECH_SERVICE_SKU {Name of SKU}.

Azure Storage Account Copy Chapter

1. Run azd env set AZURE_STORAGE_ACCOUNT {Name of existing Azure Storage Account}
1. Run azd env set AZURE_STORAGE_RESOURCE_GROUP {Name of existing resource group with storage account}
1. If that resource group is in a different location than the one you'll pick for the azd up step,
then run azd env set AZURE_STORAGE_ACCOUNT_LOCATION {Location of existing storage account}
1. To change the storage SKU from the default Standard_LRS, run azd env set AZURE_STORAGE_SKU {Name of SKU}. For production, we recommend Standard_ZRS for improved resiliency.

---

Deploy Features Copy Section

RAG chat: Enabling optional features

This document covers optional features that can be enabled in the deployed Azure resources.
You should typically enable these features before running azd up. Once you've set them, return to the deployment steps.

* Using different chat models
* Using reasoning models
* Using different embedding models
* Enabling multimodal embeddings and answering
* Enabling media description with Azure Content Understanding
* Enabling cloud data ingestion
* Enabling client-side chat history
* Enabling persistent chat history with Azure Cosmos DB
* Enabling language picker
* Enabling speech input/output
* Enabling authentication
* Enabling login and document level access control
* Enabling user document upload
* Enabling CORS for an alternate frontend
* Enabling query rewriting
* Adding an OpenAI load balancer
* Deploying with private endpoints
* Using local parsers

Using different chat models Copy Chapter

As of June 2026, the default chat model is gpt-5.4-mini. If you deployed this sample before that date, the default model may be gpt-4.1-mini, gpt-3.5-turbo, or gpt-4o-mini. You can change the chat model to any Azure OpenAI model that's available in your Azure OpenAI resource region by following these steps:

1. To set the name of the deployment, run this command with a unique name in your Azure OpenAI account. You can use any deployment name, as long as it's unique in your Azure OpenAI account. For convenience, many developers use the same deployment name as the model name, but this is not required.

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT <your-deployment-name>

For example:

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT gpt-5.4-mini

1. To set the GPT model to a different available model, run this command with the appropriate model name. A few examples are below.

For gpt-5.4-mini(default):

azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-5.4-mini

For gpt-5.2:

azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-5.2

1. To set the Azure OpenAI model version from the available versions, run this command with the appropriate version string.

For gpt-5.4-mini (default)

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2026-03-17

For gpt-5.2:

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2025-12-11

1. To set the Azure OpenAI deployment SKU name, run this command with the desired SKU name.

For GlobalStandard (default):

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_SKU GlobalStandard

For Standard:

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_SKU Standard

1. To set the Azure OpenAI deployment capacity (TPM, measured in thousands of tokens per minute), run this command with the desired capacity. This is not necessary if you are using the default capacity of 30.

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 20

1. To update the deployment with the new parameters, run this command.

azd up

This process does not delete your previous model deployment. If you want to delete previous deployments, go to your Azure OpenAI resource in Azure AI Foundry and delete it there.

Using reasoning models Copy Chapter

The default model (gpt-5.4-mini) is a reasoning model. These models spend more time processing and understanding the user's request, leading to higher quality responses.
To learn more about supported reasoning models and configuring reasoning effort, see the reasoning models guide.

Using agentic retrieval Copy Chapter

This feature allows you to use agentic retrieval in place of the Search API. To enable agentic retrieval, follow the steps in the agentic retrieval guide

Using different embedding models Copy Chapter

By default, the deployed Azure web app uses the text-embedding-3-large embedding model. If you want to use a different embedding model, you can do so by following these steps:

1. Run one of the following commands to set the desired model:

azd env set AZURE_OPENAI_EMB_MODEL_NAME text-embedding-ada-002

azd env set AZURE_OPENAI_EMB_MODEL_NAME text-embedding-3-small

azd env set AZURE_OPENAI_EMB_MODEL_NAME text-embedding-3-large

2. Specify the desired dimensions of the model: (from 256-3072, model dependent)

Default dimensions for text-embedding-ada-002

azd env set AZURE_OPENAI_EMB_DIMENSIONS 1536

Default dimensions for text-embedding-3-small

azd env set AZURE_OPENAI_EMB_DIMENSIONS 1536

Default dimensions for text-embedding-3-large

azd env set AZURE_OPENAI_EMB_DIMENSIONS 3072

3. Set the model version, depending on the model you are using:

For text-embedding-ada-002:

azd env set AZURE_OPENAI_EMB_DEPLOYMENT_VERSION 2

For text-embedding-3-small and text-embedding-3-large:

azd env set AZURE_OPENAI_EMB_DEPLOYMENT_VERSION 1

4. To set the embedding model deployment SKU name, run this command with the desired SKU name.

For GlobalStandard:

azd env set AZURE_OPENAI_EMB_DEPLOYMENT_SKU GlobalStandard

For Standard:

azd env set AZURE_OPENAI_EMB_DEPLOYMENT_SKU Standard

5. When prompted during azd up, make sure to select a region for the OpenAI resource group location that supports the desired embedding model and deployment SKU. There are limited regions available.

If you have already deployed:

* You'll need to change the deployment name by running the appropriate commands for the model above.
* You'll need to create a new index, and re-index all of the data using the new model. You can either delete the current index in the Azure Portal, or create an index with a different name by running azd env set AZURE_SEARCH_INDEX new-index-name. When you next run azd up, the new index will be created. See the data ingestion guide for more details.

Enabling multimodal embeddings and answering Copy Chapter

When your documents include images, you can optionally enable this feature that can
use image embeddings when searching and also use images when answering questions.

Learn more in the multimodal guide.

Enabling media description with Azure Content Understanding Copy Chapter

⚠️ This feature is compatible with the multimodal feature, but this feature enables only a subset of multimodal capabilities,
so you may want to enable the multimodal feature instead or as well.

By default, if your documents contain image-like figures, the data ingestion process will ignore those figures,
so users will not be able to ask questions about them.

You can optionably enable the description of media content using Azure Content Understanding. When enabled, the data ingestion process will send figures to Azure Content Understanding and replace the figure with the description in the indexed document.

To enable media description with Azure Content Understanding, run:

azd env set USE_MEDIA_DESCRIBER_AZURE_CU true

If you have already run azd up, you will need to run azd provision to create the new Content Understanding service.
If you have already indexed your documents and want to re-index them with the media descriptions,
first remove the existing documents and then re-ingest the data.

⚠️ This feature does not yet support DOCX, PPTX, or XLSX formats. If you have figures in those formats, they will be ignored.
Convert them first to PDF or image formats to enable media description.

Enabling cloud data ingestion Copy Chapter

By default, this project runs a local script in order to ingest data. Once you move beyond the sample documents, you may want to enable cloud ingestion, which uses Azure AI Search indexers and custom Azure AI Search skills based off the same code used by the local ingestion. That approach scales better to larger amounts of data.

Learn more in the cloud ingestion guide.

Enabling client-side chat history Copy Chapter

📺 Watch: (RAG Deep Dive series) Storing chat history

This feature allows users to view the chat history of their conversation, stored in the browser using IndexedDB. That means the chat history will be available only on the device where the chat was initiated. To enable browser-stored chat history, run:

azd env set USE_CHAT_HISTORY_BROWSER true

Enabling persistent chat history with Azure Cosmos DB Copy Chapter

📺 Watch: (RAG Deep Dive series) Storing chat history

This feature allows authenticated users to view the chat history of their conversations, stored in the server-side storage using Azure Cosmos DB.This option requires that authentication be enabled. The chat history will be persistent and accessible from any device where the user logs in with the same account. To enable server-stored chat history, run:

azd env set USE_CHAT_HISTORY_COSMOS true

When both the browser-stored and Cosmos DB options are enabled, Cosmos DB will take precedence over browser-stored chat history.

⚠️ Re-deploying over older versions: If you previously deployed this template before the chat history container was migrated to MultiHash partition keys (/entra_oid + /session_id), re-deploying may fail with a PropertyChangeNotAllowed error. Cosmos DB partition keys are immutable. To resolve this, delete the chat-history-v2 container in Azure Portal or via CLI, then re-deploy. Existing chat history in that container will be lost. See Troubleshooting deployment for more details.

Enabling language picker Copy Chapter

You can optionally enable the language picker to allow users to switch between different languages. Currently, it supports English, Spanish, French, Japanese, Danish, Dutch, Brasilian Portugese, Turkish, Italian and Polish.

To add support for additional languages, create new locale files and update app/frontend/src/i18n/config.ts accordingly. To enable language picker, run:

azd env set ENABLE_LANGUAGE_PICKER true

Enabling speech input/output Copy Chapter

📺 Watch a short video of speech input/output

You can optionally enable speech input/output by setting the azd environment variables.

Speech Input Copy Section

The speech input feature uses the browser's built-in Speech Recognition API. It may not work in all browser/OS combinations. To enable speech input, run:

azd env set USE_SPEECH_INPUT_BROWSER true

Speech Output Copy Section

The speech output feature uses Azure Speech Service for speech-to-text. Additional costs will be incurred for using the Azure Speech Service. See pricing. To enable speech output, run:

azd env set USE_SPEECH_OUTPUT_AZURE true

To set the voice for the speech output, run:

azd env set AZURE_SPEECH_SERVICE_VOICE en-US-AndrewMultilingualNeural

Alternatively you can use the browser's built-in Speech Synthesis API. It may not work in all browser/OS combinations. To enable speech output, run:

azd env set USE_SPEECH_OUTPUT_BROWSER true

Enabling authentication Copy Chapter

By default, the deployed Azure web app will have no authentication or access restrictions enabled, meaning anyone with routable network access to the web app can chat with your indexed data. If you'd like to automatically setup authentication and user login as part of the azd up process, see this guide.

Alternatively, you can manually require authentication to your Azure Active Directory by following the Add app authentication tutorial and set it up against the deployed web app.

To then limit access to a specific set of users or groups, you can follow the steps from Restrict your Microsoft Entra app to a set of users by changing "Assignment Required?" option under the Enterprise Application, and then assigning users/groups access. Users not granted explicit access will receive the error message -AADSTS50105: Your administrator has configured the application <app_name> to block users unless they are specifically granted ('assigned') access to the application.-

Enabling login and document level access control Copy Chapter

By default, the deployed Azure web app allows users to chat with all your indexed data. You can enable an optional login system using Azure Active Directory to restrict access to indexed data based on the logged in user. Enable the optional login and document level access control system by following this guide.

Enabling user document upload Copy Chapter

You can enable an optional user document upload system to allow users to upload their own documents and chat with them. This feature requires you to first enable login and document level access control. Then you can enable the optional user document upload system by setting an azd environment variable:

azd env set USE_USER_UPLOAD true

Then you'll need to run azd up to provision an Azure Data Lake Storage Gen2 account for storing the user-uploaded documents.
When the user uploads a document, it will be stored in a directory in that account with the same name as the user's Entra object id,
and will have ACLs associated with that directory. When the ingester runs, it will also set the oids of the indexed chunks to the user's Entra object id. Whenever any content is retrieved or added to the directory, the "owner" property will be checked to ensure that the user is the owner of the directory, and thus has access to the content.

If you are enabling this feature on an existing index, you should also update your index to have the new storageUrl field:

python ./scripts/manageacl.py  -v --acl-action enable_acls

And then update existing search documents with the storage URL of the main Blob container:

python ./scripts/manageacl.py  -v --acl-action update_storage_urls --url <https://YOUR-MAIN-STORAGE-ACCOUNT.blob.core.windows.net/content/>

Going forward, all uploaded documents will have their storageUrl set in the search index.
This is necessary to disambiguate user-uploaded documents from admin-uploaded documents.

Enabling CORS for an alternate frontend Copy Chapter

By default, the deployed Azure web app will only allow requests from the same origin. To enable CORS for a frontend hosted on a different origin, run:

1. Run azd env set ALLOWED_ORIGIN https://<your-domain.com>
2. Run azd up

For the frontend code, change BACKEND_URI in api.ts to point at the deployed backend URL, so that all fetch requests will be sent to the deployed backend.

For an alternate frontend that's written in Web Components and deployed to Static Web Apps, check out
azure-search-openai-javascript and its guide
on using a different backend.
Both these repositories adhere to the same HTTP protocol for AI chat apps.

Enabling query rewriting Copy Chapter

By default, the query rewriting feature from the Azure AI Search service is not enabled. Note that the search service query rewriting feature is different from the query rewriting step that is used by the Chat tab in the codebase. The in-repo query rewriting step also incorporates conversation history, while the search service query rewriting feature only considers the query itself. To enable search service query rewriting, set the following environment variables:

1. Check that your Azure AI Search service is using one of the supported regions for query rewriting.
1. Ensure semantic ranker is enabled. Query rewriting may only be used with semantic ranker. Run azd env set AZURE_SEARCH_SEMANTIC_RANKER free or azd env set AZURE_SEARCH_SEMANTIC_RANKER standard depending on your desired semantic ranker tier.
1. Enable query rewriting. Run azd env set AZURE_SEARCH_QUERY_REWRITING true. An option in developer settings will appear allowing you to toggle query rewriting on and off. It will be on by default.

Adding an OpenAI load balancer Copy Chapter

As discussed in more details in our productionizing guide, you may want to consider implementing a load balancer between OpenAI instances if you are consistently going over the TPM limit.
Fortunately, this repository is designed for easy integration with other repositories that create load balancers for OpenAI instances. For seamless integration instructions with this sample, please check:

* Scale Azure OpenAI for Python with Azure API Management
* Scale Azure OpenAI for Python chat using RAG with Azure Container Apps

Deploying with private endpoints Copy Chapter

It is possible to deploy this app with public access disabled, using Azure private endpoints and private DNS Zones. For more details, read the private deployment guide. That requires a multi-stage provisioning, so you will need to do more than just azd up after setting the environment variables.

Using local parsers Copy Chapter

If you want to decrease the charges by using local parsers instead of Azure Document Intelligence, you can set environment variables before running the data ingestion script. Note that local parsers will generally be not as sophisticated.

1. Run azd env set USE_LOCAL_PDF_PARSER true to use the local PDF parser.
1. Run azd env set USE_LOCAL_HTML_PARSER true to use the local HTML parser.

The local parsers will be used the next time you run the data ingestion script. To use these parsers for the user document upload system, you'll need to run azd provision to update the web app to use the local parsers.

---

Deploy Freetrial Copy Section

RAG chat: Deploying with a free trial account

If you have just created an Azure free trial account and are using the free trial credits,
there are several modifications you need to make, due to restrictions on the free trial account.

Follow these instructions before you run azd up.

Accomodate for low OpenAI quotas Copy Chapter

The free trial accounts currently get a max of 1K TPM (tokens-per-minute), whereas our Bicep templates try to allocate 30K TPM.

To reduce the TPM allocation, run these commands:

azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 1
azd env set AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY 1

Alternatively, if you have an OpenAI.com account, you can use that instead:

azd env set OPENAI_HOST openai
azd env set OPENAI_ORGANIZATION {Your OpenAI organization}
azd env set OPENAI_API_KEY {Your OpenAI API key}

Accomodate for Azure Container Apps restrictions Copy Chapter

By default, this project deploys to Azure Container Apps, using a remote build process that builds the Docker image in the cloud.
Unfortunately, free trial accounts cannot use that remote build process.

You have two options:

1. Comment out or delete remoteBuild: true in azure.yaml, and make sure you have Docker installed in your environment.

2. Deploy using App Service instead:

* Comment out host: containerapp and uncomment host: appservice in the azure.yaml file.
* Set the deployment target to appservice:

azd env set DEPLOYMENT_TARGET appservice

---

Deploy Lowcost Copy Section

RAG chat: Deploying with minimal costs

This AI RAG chat application is designed to be easily deployed using the Azure Developer CLI, which provisions the infrastructure according to the Bicep files in the infra folder. Those files describe each of the Azure resources needed, and configures their SKU (pricing tier) and other parameters. Many Azure services offer a free tier, but the infrastructure files in this project do not default to the free tier as there are often limitations in that tier.

However, if your goal is to minimize costs while prototyping your application, follow the steps below before running azd up. Once you've gone through these steps, return to the deployment steps.

📺 Live stream: Deploying from a free account

1. Log in to your Azure account using the Azure Developer CLI:

azd auth login

1. Create a new azd environment for the free resource group:

azd env new

Enter a name that will be used for the resource group.
This will create a new folder in the .azure folder, and set it as the active environment for any calls to azd going forward.

1. Switch from Azure Container Apps to the free tier of Azure App Service:

Azure Container Apps has a consumption-based pricing model that is very low cost, but it is not free, plus Azure Container Registry costs a small amount each month.

To deploy to App Service instead:

* Comment out host: containerapp and uncomment host: appservice in the azure.yaml file.
* Set the deployment target to appservice:

azd env set DEPLOYMENT_TARGET appservice

* Set the App Service SKU to the free tier:

azd env set AZURE_APP_SERVICE_SKU F1

Limitation: You are only allowed a certain number of free App Service instances per region. If you have exceeded your limit in a region, you will get an error during the provisioning stage. If that happens, you can run azd down, then azd env new to create a new environment with a new region.

1. Use the free tier of Azure AI Search:

azd env set AZURE_SEARCH_SERVICE_SKU free

Limitations:
1. You are only allowed one free search service across all regions.
If you have one already, either delete that service or follow instructions to
reuse your existing search service.
2. The free tier does not support semantic ranker, so the app UI will no longer display
the option to use the semantic ranker. Note that will generally result in decreased search relevance.
3. The free tier does not support managed identities. As a result, cloud ingestion and multimodal/vector features that require role assignments to the search service principal will have those role assignments skipped during provisioning. If you need those permissions, use a non-free tier (for example, Basic/B1 or Standard).

1. Use the free tier of Azure Document Intelligence (used in analyzing files):

azd env set AZURE_DOCUMENTINTELLIGENCE_SKU F0

Limitation for PDF files:

The free tier will only scan the first two pages of each PDF.
In our sample documents, those first two pages are just title pages,
so you won't be able to get answers from the documents.
You can either use your own documents that are only 2-pages long,
or you can use a local Python package for PDF parsing by setting:

azd env set USE_LOCAL_PDF_PARSER true

Limitation for HTML files:

The free tier will only scan the first two pages of each HTML file.
So, you might not get very accurate answers from the files.
You can either use your own files that are only 2-pages long,
or you can use a local Python package for HTML parsing by setting:

azd env set USE_LOCAL_HTML_PARSER true

1. Use the free tier of Azure Cosmos DB:

azd env set AZURE_COSMOSDB_SKU free

Limitation: You can have only one free Cosmos DB account. To keep your account free of charge, ensure that you do not exceed the free tier limits. For more information, see the Azure Cosmos DB lifetime free tier.

1. ⚠️ This step is currently only possible if you're deploying to App Service (see issue 2281):

Turn off Azure Monitor (Application Insights):

azd env set AZURE_USE_APPLICATION_INSIGHTS false

Application Insights is quite inexpensive already, so turning this off may not be worth the costs saved,
but it is an option for those who want to minimize costs.

1. Use OpenAI.com instead of Azure OpenAI: This should not be necessary, as the costs are same for both services, but you may need this step if your account does not have access to Azure OpenAI for some reason.

azd env set OPENAI_HOST openai
    azd env set OPENAI_ORGANIZATION {Your OpenAI organization}
    azd env set OPENAI_API_KEY {Your OpenAI API key}

Both Azure OpenAI and openai.com OpenAI accounts will incur costs, based on tokens used,
but the costs are fairly low for the amount of sample data (less than $10).

1. Disable vector search:

azd env set USE_VECTORS false

By default, the application computes vector embeddings for documents during the data ingestion phase,
and then computes a vector embedding for user questions asked in the application.
Those computations require an embedding model, which incurs costs per tokens used. The costs are fairly low,
so the benefits of vector search would typically outweigh the costs, but it is possible to disable vector support.
If you do so, the application will fall back to a keyword search, which is less accurate.

1. Once you've made the desired customizations, follow the steps in the README to run azd up. We recommend using "eastus" as the region, for availability reasons.

Reducing costs locally Copy Chapter

To save costs for local development, you could use an OpenAI-compatible model.
Follow steps in local development guide.

---

Deploy Private Copy Section

RAG chat: Deploying with private access

📺 Watch: (RAG Deep Dive series) Private network deployment

The azure-search-openai-demo project can set up a full RAG chat app on Azure AI Search and OpenAI so that you can chat on custom data, like internal enterprise data or domain-specific knowledge sets. For full instructions on setting up the project, consult the main README, and then return here for detailed instructions on configuring private endpoints.

If you want to disable public access for the application so that it can only be access from a private network, follow this guide.

Before you begin Copy Chapter

Deploying with private networking adds additional cost to your deployment. Please see pricing for the following products:

* Azure Container Registry: Premium tier is used when virtual network is added (required for private links), which incurs additional costs.
* Azure Container Apps: Workload profiles environment is used when virtual network is added (required for private links), which incurs additional costs. Additionally, min replica count is set to 1, so you will be charged for at least one instance.
* VPN Gateway: VpnGw2 SKU. Pricing includes a base monthly cost plus an hourly cost based on the number of connections.
* Virtual Network: Pay-as-you-go tier. Costs based on data processed.

The pricing for the following features depends on the optional features used. Most deployments will have at least 5 private endpoints (Azure OpenAI, Azure Cognitive Services, Azure AI Search, Azure Blob Storage, and either Azure App Service or Azure Container Apps).

* Azure Private Endpoints: Pricing is per hour per endpoint.
* Private DNS Zones: Pricing is per month and zones.
* Azure Private DNS Resolver: Pricing is per month and zones.

⚠️ To avoid unnecessary costs, remember to take down your app if it's no longer in use,
either by deleting the resource group in the Portal or running azd down.
You might also decide to delete the VPN Gateway when not in use.

Deployment steps for private access Copy Chapter

1. Configure the azd environment variables to use private endpoints and a VPN gateway, with public network access disabled. This will allow you to connect to the chat app from inside the virtual network, but not from the public Internet.

azd env set AZURE_USE_PRIVATE_ENDPOINT true
    azd env set AZURE_USE_VPN_GATEWAY true
    azd env set AZURE_PUBLIC_NETWORK_ACCESS Disabled
    azd up

2. Provision all the Azure resources:

azd provision

3. Once provisioning is complete, you will see an error when it tries to run the data ingestion script, because you are not yet connected to the VPN. That message should provide a URL for the VPN configuration file download. If you don't see that URL, run this command:

azd env get-value AZURE_VPN_CONFIG_DOWNLOAD_LINK

Open that link in your browser. Select "Download VPN client" to download a ZIP file containing the VPN configuration.

4. Open AzureVPN/azurevpnconfig.xml, and replace the <clientconfig> empty tag with the following:

<clientconfig>
        <dnsservers>
          <dnsserver>10.0.11.4</dnsserver>
        </dnsservers>
      </clientconfig>

> Note: We use the IP address 10.0.11.4 since it is the first available IP in the dns-resolver-subnet(10.0.11.0/28) from the provisioned virtual network, as Azure reserves the first four IP addresses in each subnet. Adding this DNS server allows your VPN client to resolve private DNS names for Azure services accessed through private endpoints. See the network configuration in network-isolation.bicep for details.

5. Install the Azure VPN Client.

6. Open the Azure VPN Client and select "Import" button. Select the azurevpnconfig.xml file you just downloaded and modified.

7. Select "Connect" and the new VPN connection. You will be prompted to select your Microsoft account and login.

8. Once you're successfully connected to VPN, you can run the data ingestion script:

azd hooks run postprovision

9. Finally, you can deploy the app:

azd deploy

Environment variables controlling private access Copy Chapter

1. AZURE_PUBLIC_NETWORK_ACCESS: Controls the value of public network access on supported Azure resources. Valid values are 'Enabled' or 'Disabled'.
1. When public network access is 'Enabled', Azure resources are open to the internet.
1. When public network access is 'Disabled', Azure resources are only accessible over a virtual network.
1. AZURE_USE_PRIVATE_ENDPOINT: Controls deployment of private endpoints which connect Azure resources to the virtual network.
1. When set to 'true', ensures private endpoints are deployed for connectivity even when AZURE_PUBLIC_NETWORK_ACCESS is 'Disabled'.
1. Note that private endpoints do not make the chat app accessible from the internet. Connections must be initiated from inside the virtual network.
1. AZURE_USE_VPN_GATEWAY: Controls deployment of a VPN gateway for the virtual network. If you do not use this and public access is disabled, you will need a different way to connect to the virtual network.

Compatibility with other features Copy Chapter

* GitHub Actions / Azure DevOps: The private access deployment is not compatible with the built-in CI/CD pipelines, as it requires a VPN connection to deploy the app. You could modify the pipeline to only do provisioning, and set up a different deployment strategy for the app.

---

Deploy Troubleshooting Copy Section

RAG chat: Troubleshooting deployment

If you are experiencing an error when deploying the RAG chat solution using the deployment steps, this guide will help you troubleshoot common issues.

1. You're attempting to create resources in regions not enabled for Azure OpenAI (e.g. East US 2 instead of East US), or where the model you're trying to use isn't enabled. See this matrix of model availability.

1. You've exceeded a quota, most often number of resources per region. See this article on quotas and limits.

1. You're getting "same resource name not allowed" conflicts. That's likely because you've run the sample multiple times and deleted the resources you've been creating each time, but are forgetting to purge them. Azure keeps resources for 48 hours unless you purge from soft delete. See this article on purging resources.

1. You see CERTIFICATE_VERIFY_FAILED when the prepdocs.py script runs. That's typically due to incorrect SSL certificates setup on your machine. Try the suggestions in this StackOverflow answer.

1. After running azd up and visiting the website, you see a '404 Not Found' in the browser. Wait 10 minutes and try again, as it might be still starting up. Then try running azd deploy and wait again. If you still encounter errors with the deployed app and are deploying to App Service, consult the guide on debugging App Service deployments. Please file an issue if the logs don't help you resolve the error.

1. You see a RoleAssignmentExists error (HTTP 409) when re-deploying after switching between local development and CI/CD pipelines (or vice versa). This happens because the role assignment GUID changes when the principalType changes between User and ServicePrincipal. Running azd up again should resolve the issue, as the template now generates separate role assignments for each principal type.

1. You see a PropertyChangeNotAllowed error referencing Cosmos DB partition keys when re-deploying over an older version of this template. Cosmos DB partition keys are immutable and cannot be changed after container creation. To resolve this, delete the chat-history-v2 container in Azure Portal or via CLI (az cosmosdb sql container delete), then re-deploy. The container will be recreated with the correct MultiHash partition key scheme. Note: existing chat history in that container will be lost.

1. You see a Conflict error (HTTP 409) about Cognitive Services resources when re-deploying after a previous azd down. Azure soft-deletes Cognitive Services resources for 48 days, blocking re-creation with the same name. To resolve this, set the RESTORE_COGNITIVE_SERVICES environment variable to true before re-deploying:

azd env set RESTORE_COGNITIVE_SERVICES true
   azd up

After the resources are restored, set it back to false to avoid issues on subsequent deployments:

azd env set RESTORE_COGNITIVE_SERVICES false

Alternatively, you can manually purge the soft-deleted resources via the Azure CLI and re-deploy without the flag.

---

Evaluation Copy Section

Evaluating the RAG answer quality

📺 Watch: (RAG Deep Dive series) Evaluating RAG answer quality

Follow these steps to evaluate the quality of the answers generated by the RAG flow.

* Deploy an evaluation model
* Setup the evaluation environment
* Generate ground truth data
* Run bulk evaluation
* Review the evaluation results

Deploy an evaluation model Copy Chapter

1. Run this command to tell azd to deploy a reasoning model for evaluation:

azd env set USE_EVAL true

2. Set both the evaluation model and chat model capacities to the highest possible values to ensure that the evaluation runs relatively quickly. The evaluation model grades responses, while the chat model generates target answers and query rewrites. Either deployment can be rate limited during a bulk run.

azd env set AZURE_OPENAI_EVAL_DEPLOYMENT_CAPACITY 100
    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 100

By default, the evaluation deployment uses gpt-5.4 version 2026-03-05. To change those settings, set the azd environment variables AZURE_OPENAI_EVAL_MODEL and AZURE_OPENAI_EVAL_MODEL_VERSION to the desired values.

3. Then, run the following command to provision the model:

azd provision

Setup the evaluation environment Copy Chapter

Make a new Python virtual environment and activate it. This is currently required due to incompatibilities between the dependencies of the evaluation script and the main project.

python -m venv .evalenv

source .evalenv/bin/activate

Install all the dependencies for the evaluation script by running the following command:

pip install -r evals/requirements.txt

Generate ground truth data Copy Chapter

Generate ground truth data by running the following command:

python evals/generate_ground_truth.py --numquestions=200 --numsearchdocs=1000

The options are:

* numquestions: The number of questions to generate. We suggest at least 200.
* numsearchdocs: The number of documents (chunks) to retrieve from your search index. You can leave off the option to fetch all documents, but that will significantly increase time it takes to generate ground truth data. You may want to at least start with a subset.
* kgfile: An existing RAGAS knowledge base JSON file, which is usually ground_truth_kg.json. You may want to specify this if you already created a knowledge base and just want to tweak the question generation steps.
* groundtruthfile: The file to write the generated ground truth answwers. By default, this is evals/ground_truth.jsonl.

🕰️ This may take a long time, possibly several hours, depending on the size of the search index.

Review the generated data in evals/ground_truth.jsonl after running that script, removing any question/answer pairs that don't seem like realistic user input.

Run bulk evaluation Copy Chapter

Review the configuration in evals/evaluate_config.json to ensure that everything is correctly setup. You may want to adjust the metrics used. See the ai-rag-chat-evaluator README for more information on the available metrics.

By default, the evaluation script will evaluate every question in the ground truth data.
Run the evaluation script by running the following command:

python evals/run_evaluate.py

The options are:

* numquestions: The number of questions to evaluate. By default, this is all questions in the ground truth data.
* resultsdir: The directory to write the evaluation results. By default, this is a timestamped folder in evals/results. This option can also be specified in evaluate_config.json.
* targeturl: The URL of the running application to evaluate. By default, this is http://localhost:50505. This option can also be specified in evaluate_config.json.

🕰️ This may take a long time, possibly several hours, depending on the number of ground truth questions, the TPM capacity of the evaluation model, and the number of LLM-based metrics requested.

Check for hidden rate limiting Copy Section

The application SDKs automatically retry HTTP 429 responses, so an evaluation request can eventually succeed while spending significant time waiting for capacity. The evaluation client only sees the final successful response.

When Application Insights is enabled for the target deployment, query its Logs after each run:

dependencies
| where timestamp between (datetime(<RUN_START_UTC>) .. datetime(<RUN_END_UTC>))
| where resultCode == "429"
| summarize attempts=count(), affectedRequests=dcount(operation_Id) by target, name

Any returned rows mean the latency results were affected by throttling. Increase the relevant deployment capacity, run azd provision, and repeat the evaluation before comparing latency.

Review the evaluation results Copy Chapter

The evaluation script will output a summary of the evaluation results, inside the evals/results directory.

You can see a summary of results across all evaluation runs by running the following command:

cd evals
python -m evaltools summary results

Compare answers to the ground truth by running the following command,
replacing RUNHERE with the name of a run directory inside evals/results:

cd evals
python -m evaltools diff results/RUNHERE

Compare answers across two runs by running the following command,
replacing FIRSTRUNHERE and SECONDRUNHERE with run directory names inside evals/results:

cd evals
python -m evaltools diff results/FIRSTRUNHERE results/SECONDRUNHERE

Evaluate multimodal RAG answers Copy Chapter

The repository also includes an evaluate_config_multimodal.json file specifically for evaluating multimodal RAG answers. This configuration uses a different ground truth file, ground_truth_multimodal.jsonl, which includes questions based off the sample data that require both text and image sources to answer.

Note that the "groundedness" evaluator is not reliable for multimodal RAG, since it does not currently incorporate the image sources. We still include it in the metrics, but the more reliable metrics are "relevance" and "citations matched".

---

Http Protocol Copy Section

RAG Chat: HTTP Protocol

The frontend and backend of this RAG chat application exchange messages over HTTP, using both regular JSON for single responses and streaming newline-delimited JSON (NDJSON) for streamed responses.

The HTTP protocol is inspired by the OpenAI Responses API, but contains additional fields required for the chat application.

Table of contents:

* HTTP requests to chat app endpoints
* Request context properties
* HTTP responses from RAG chat app endpoints
* Non-streaming response
* Successful response
* Error response
* Streaming response
* Successful streamed response
* Error in streamed response
* Answer formatting
* Response context properties

HTTP requests to chat app endpoints Copy Chapter

All requests use the POST method, with the following headers:

* Content-Type: application/json
* Authorization: Bearer <ID token>: _Optional._ For authentication, if the app is deployed with user login enabled

The path is chat for a non-streaming request and chat/stream for a streaming request.

The body of the request contains these properties, in JSON format:

* "messages": A list of messages, each containing "content" and "role", where "role" may be "assistant" or "user". When triggered from the "Ask" tab (single-turn RAG), the list will contain a single message, whereas requests from the "Chat" tab (multi-turn RAG) may contain multiple messages.
* "session_state": _Optional_. An object containing the "memory" for the chat app, such as the session ID for chat history storage.
* "context": _Optional_. An object containing any additional options for the request, such as the temperature to use for the LLM. See below for supported options.

Usage example Copy Section

The example belows represents a valid and compliant request body to the chat app endpoints:

{
    "messages": [
        {
            "content": "What is included in my Northwind Health Plus plan that is not in standard?",
            "role": "user"
        }
    ],
    "context": {},
    "session_state": null
}

Request context properties Copy Section

These are the currently supported properties in the context object:

* "overrides": An object containing settings for the chat application.
* "temperature": The temperature to use for the LLM for the question-answering response call.
* "top": The number of results to return from Azure AI Search.
* "retrieval_mode": The mode to use for the Azure AI Search step. Can be "hybrid", "vectors", or "text".
* "semantic_ranker": Whether to use the semantic ranker for the Azure AI Search step.
* "semantic_captions": Whether to use semantic captions for the Azure AI Search step.
* "suggest_followup_questions": Whether to suggest follow-up questions for the chat app.
* "use_oid_security_filter": Whether to use the OID security filter for the Azure AI Search step.
* "use_groups_security_filter": Whether to use the groups security filter for the Azure AI Search step.
* "vector_fields": Which embedding fields to use for the Azure AI Search step. This is either textEmbeddingOnly, imageEmbeddingOnly, or textAndImageEmbeddings. The default is textEmbeddingOnly, but if you have multimodal embeddings enabled, it defaults to textAndImageEmbeddings.
* "use_multimodal_answering": Whether to send both text and images to the LLM for answering questions.

Example of the overrides object:

"overrides": {
    "top": 3,
    "retrieval_mode": "text",
    "semantic_ranker": false,
    "semantic_captions": false,
    "suggest_followup_questions": false,
    "use_oid_security_filter": false,
    "use_groups_security_filter": false,
    "vector_fields": "textEmbeddingOnly",
    "use_multimodal_answering": false,
}

HTTP responses from RAG chat app endpoints Copy Chapter

The HTTP response is JSON for a non-streaming response, or newline-delimited JSON ("NDJSON"/"jsonlines") for a streaming response.

Non-streaming response Copy Section

The response contains this header:

* Content-Type: application/json

#### Successful response

A successful response has a status code of 200, and the body contains a JSON object with the following properties:

* "output_text": A string containing the actual content of the response. See Answer formatting.
* "session_state": _Optional_. An object containing the "memory" for the chat app, such as the session ID for chat history storage.
* "context": _Optional_. An object containing additional details needed for the chat app, used for citation display and the thought process tab. See response context properties.

Here's an example JSON response:

/ Detailed source-code truncated for AI context efficiency. /

#### Error response

An error response has a status code of 400 or 500, and the body contains a JSON object with the following properties:

* "error": A string describing the error.

Here's an example JSON response for a 500-level error:

{
    "error": "The app encountered an error processing your request.\nIf you are an administrator of the app, view the full error in the logs."
}

Here's an example JSON response for a 400-level error:

{
    "error": "Your message contains content that was flagged by the OpenAI content filter."
}

Streaming response Copy Section

The response contains these headers:

* Content-Type: application/json-lines
* Transfer-Encoding: chunked

#### Successful streamed response

A successful response has a status code of 200.
The body of the response contains a sequence of JSON objects, each representing a chunk of the response.
The first chunk contains the context property, since that is available before the answer, and subsequent chunks contain parts of the answer to the question.

Each JSON object contains the following properties:

* "type": A string indicating the event type. Either "response.context" for context events or "response.output_text.delta" for text content chunks. _The response.output_text.delta type matches the OpenAI Responses API streaming event._
* "delta": _(For response.output_text.delta events only)_ A string containing a text chunk of the answer.
* "context": _(For response.context events only)_ An object containing additional details needed for the chat app. See response context properties.
* "session_state": _Optional_. An object containing the "memory" for the chat app, such as a user ID.

Here's an example of the first three JSON objects in a streaming response:

/ Detailed source-code truncated for AI context efficiency. /

#### Error in streamed response

If an error is encountered before the stream begins, then the response may look like a non-streaming error response. However, if an error is encountered during the stream, then the server will have already sent a 200 response, and will send a chunk with an error object. Typically that would be the last chunk, but it may not be.

Here's an example of an error chunk:

{
    "error": "The app encountered an error processing your request.\nIf you are an administrator of the app, view the full error in the logs."
}

Answer formatting Copy Section

To support the display of citations, the answer from the LLM should contain source information in square brackets, such as [info1.txt].

Here's a full example of an answer with citation:

There is no specific information provided about what is included in the Northwind Health Plus plan that is not in the standard plan. It is recommended to read the plan details carefully and ask questions to understand the specific benefits of the Northwind Health Plus plan [Northwind_Standard_Benefits_Details.pdf#page=91].

Response context properties Copy Section

The response context object can contain the following properties:

* "followup_questions": A list of follow-up questions to ask the user.

Example:

"followup_questions": [
        "What types of prescription drugs are covered?",
        "Which services have lower out-of-pocket costs?"
    ]

When the app sees this property in the response and the user has requested follow-up questions (in the settings), the app prompts the user with clickable versions of the questions. See image

* "data_points": An object containing text and/or image data chunks, a list in the "text" or "images" properties.

Example:

"data_points": {
        "text": [
            "Northwind_Standard_Benefits_Details.pdf#page=91:    Tips for Avoiding Intentionally False Or Misleading Statements:   When it comes to understanding a health plan, it is important to be aware of any intentionally false or misleading statements that the plan provider may make...(truncated)",
            "Northwind_Standard_Benefits_Details.pdf#page=91:  It is important to research the providers and services offered in the Northwind Standard plan in order to  determine if the providers and services offered are sufficient for the employee's needs...(truncated)",
            "Northwind_Standard_Benefits_Details.pdf#page=17:  Employees should keep track of their claims and follow up with  Northwind Health if a claim is not processed in a timely manner...(truncated)"
        ]
    },

Example with images:

"data_points": {
        "images": [
            {
                "detail": "auto",
                "url": "data:image/png;base64,iVBOR1BORw0KGgoAAAANSUhEUgAAAAEAAAABAQAAAAA3bvkkAAAACklEQVR4nGMAAQAABQABDQ0tuhsAAAAASUVORK5CYII="
            }
        ],
        "text": [
            "Financial Market Analysis Report 2023-6.png: 3</td><td>1</td></tr></table> Financial markets are interconnected, with movements in one segment often influencing other...(truncated)"
        ]
    },

The app turns the data points into clickable citations and the "Supporting content" tab. See image

* "thoughts": A list describing each step of the backend. Each step should contain:
* "title": A string describing the step.
* "description": A string or list of strings describing the step.
* "props"`: _Optional_. An object containing additional properties for the step.

Example:

/ Detailed source-code truncated for AI context efficiency. /

The app displays these thoughts in the "Thought process" tab, available by selecting the lightbulb icon on each answer. See image

---