azure-search-openai-demo (Agent Skills)

# Instructions for Coding Agents

This file contains instructions for developers working on the Azure Search and OpenAI demo application. It covers the overall code layout, how to add new data, how to add new azd environment variables, how to add new developer settings, and how to add tests for new features.

Always keep this file up to date with any changes to the codebase or development process.
If necessary, edit this file to ensure it accurately reflects the current state of the project.

## Overall code layout

* app: Contains the main application code, including frontend and backend.
  * app/backend: Contains the Python backend code, written with Quart framework.
    * app/backend/approaches: Contains the different approaches
      * app/backend/approaches/approach.py: Base class for all approaches
      * app/backend/approaches/chatreadretrieveread.py: Chat approach, includes query rewriting step first
      * app/backend/approaches/promptmanager.py: Manages loading and rendering of Jinja2 prompt templates
      * app/backend/approaches/prompts/query_rewrite.system.jinja2: Jinja2 template used to rewrite the query based off search history into a better search query
      * app/backend/approaches/prompts/chat_query_rewrite_tools.json: Tools used by the query rewriting prompt
      * app/backend/approaches/prompts/chat_answer.system.jinja2: Jinja2 template for the system message used by the Chat approach to answer questions
      * app/backend/approaches/prompts/chat_answer.user.jinja2: Jinja2 template for the user message used by the Chat approach, including sources
    * app/backend/prepdocslib: Contains the document ingestion library used by both local and cloud ingestion
      * app/backend/prepdocslib/blobmanager.py: Manages uploads to Azure Blob Storage
      * app/backend/prepdocslib/cloudingestionstrategy.py: Builds the Azure AI Search indexer and skillset for the cloud ingestion pipeline
      * app/backend/prepdocslib/csvparser.py: Parses CSV files
      * app/backend/prepdocslib/embeddings.py: Generates embeddings for text and images using Azure OpenAI
      * app/backend/prepdocslib/figureprocessor.py: Generates figure descriptions for both local ingestion and the cloud figure-processor skill
      * app/backend/prepdocslib/fileprocessor.py: Orchestrates parsing and chunking of individual files
      * app/backend/prepdocslib/filestrategy.py: Strategy for uploading and indexing files (local ingestion)
      * app/backend/prepdocslib/htmlparser.py: Parses HTML files
      * app/backend/prepdocslib/integratedvectorizerstrategy.py: Strategy using Azure AI Search integrated vectorization
      * app/backend/prepdocslib/jsonparser.py: Parses JSON files
      * app/backend/prepdocslib/listfilestrategy.py: Lists files from local filesystem or Azure Data Lake
      * app/backend/prepdocslib/mediadescriber.py: Interfaces for describing images (Azure OpenAI GPT-4o, Content Understanding)
      * app/backend/prepdocslib/page.py: Data classes for pages, images, and chunks
      * app/backend/prepdocslib/parser.py: Base parser interface
      * app/backend/prepdocslib/pdfparser.py: Parses PDFs using Azure Document Intelligence or local parser
      * app/backend/prepdocslib/searchmanager.py: Manages Azure AI Search index creation and updates
      * app/backend/prepdocslib/servicesetup.py: Shared service setup helpers for OpenAI, embeddings, blob storage, etc.
      * app/backend/prepdocslib/strategy.py: Base strategy interface for document ingestion
      * app/backend/prepdocslib/textparser.py: Parses plain text and markdown files
      * app/backend/prepdocslib/textprocessor.py: Processes text chunks for cloud ingestion (merges figures, generates embeddings)
      * app/backend/prepdocslib/textsplitter.py: Splits text into chunks using different strategies
    * app/backend/app.py: The main entry point for the backend application.
  * app/functions: Azure Functions used for cloud ingestion custom skills (document extraction, figure processing, text processing). Each function bundles a synchronized copy of `prepdocslib`; run `python scripts/copy_prepdocslib.py` to refresh the local copies if you modify the library.
  * app/frontend: Contains the React frontend code, built with TypeScript, built with vite.
    * app/frontend/src/api: Contains the API client code for communicating with the backend.
    * app/frontend/src/components: Contains the React components for the frontend.
    * app/frontend/src/locales: Contains the translation files for internationalization.
      * app/frontend/src/locales/da/translation.json: Danish translations
      * app/frontend/src/locales/en/translation.json: English translations
      * app/frontend/src/locales/es/translation.json: Spanish translations
      * app/frontend/src/locales/fr/translation.json: French translations
      * app/frontend/src/locales/it/translation.json: Italian translations
      * app/frontend/src/locales/ja/translation.json: Japanese translations
      * app/frontend/src/locales/nl/translation.json: Dutch translations
      * app/frontend/src/locales/ptBR/translation.json: Portuguese translations
      * app/frontend/src/locales/tr/translation.json: Turkish translations
    * app/frontend/src/pages: Contains the main pages of the application
* infra: Contains the Bicep templates for provisioning Azure resources.
* evals: Contains evaluation configs, datasets, and results.
  * evals/results: Contains raw per-run eval output folders. Use descriptive setup-based names for repeated runs, such as `gpt54-low-top5-run1`.
  * evals/results_summaries: Contains derived grouped summaries such as `baseline.json` and `baseline.md`.
  * evals/results_comparisons: Reserved for derived candidate-vs-baseline comparison artifacts.
  * evals/eval_compare.py: Compares eval result folders and reports averages, confidence intervals, and paired significance tests.
* tests: Contains the test code, including e2e tests, app integration tests, and unit tests.

## Adding new data

New files should be added to the `data` folder, and then either run scripts/prepdocs.sh or scripts/prepdocs.ps1 to ingest the data.

## Adding a new azd environment variable

An azd environment variable is stored by the azd CLI for each environment. It is passed to the "azd up" command and can configure both provisioning options and application settings.
When adding new azd environment variables, update:

1. infra/main.parameters.json : Add the new parameter with a Bicep-friendly variable name and map to the new environment variable
1. infra/main.bicep: Add the new Bicep parameter at the top, and add it to the `appEnvVariables` object
1. .azdo/pipelines/azure-dev.yml: Add the new environment variable under `env` section
1. .github/workflows/azure-dev.yml: Add the new environment variable under `env` section

You may also need to update:

1. app/backend/prepdocs.py: If the variable is used in the ingestion script, retrieve it from environment variables here. Not always needed.
1. app/backend/app.py: If the variable is used in the backend application, retrieve it from environment variables in setup_clients() function. Not always needed.

## Adding a new setting to "Developer Settings" in RAG app

When adding a new developer setting, update:

* frontend:
  * app/frontend/src/api/models.ts : Add to ChatAppRequestOverrides
  * app/frontend/src/components/Settings.tsx : Add a UI element for the setting
  * app/frontend/src/locales/*/translations.json: Add a translation for the setting label/tooltip for all languages
  * app/frontend/src/pages/chat/Chat.tsx: Add the setting to the component, pass it to Settings

* backend:
  * app/backend/approaches/chatreadretrieveread.py :  Retrieve from overrides parameter
  * app/backend/app.py: Some settings may need to be sent down in the /config route.

## When adding tests for a new feature

All tests are in the `tests` folder and use the pytest framework.
There are three styles of tests:

* e2e tests: These use playwright to run the app in a browser and test the UI end-to-end. They are in e2e.py and they mock the backend using the snapshots from the app tests. (Before running e2e tests, make sure to run `npm run build` in app/frontend first to build the frontend code.)
* app integration tests: Mostly in test_app.py, these test the app's API endpoints and use mocks for services like Azure OpenAI and Azure Search.
* unit tests: The rest of the tests are unit tests that test individual functions and methods. They are in test_*.py files.

When adding a new feature, add tests for it in the appropriate file.
If the feature is a UI element, add an e2e test for it.
If it is an API endpoint, add an app integration test for it.
If it is a function or method, add a unit test for it.
Use mocks from tests/conftest.py to mock external services. Prefer mocking at the HTTP/requests level when possible.

When you're running tests, make sure you activate the .venv virtual environment first:

```shell
source .venv/bin/activate
```

To check for coverage, run the following command:

```shell
pytest --cov --cov-report=annotate:cov_annotate
```

Open the cov_annotate directory to view the annotated source code. There will be one file per source file. If a file has 100% source coverage, it means all lines are covered by tests, so you do not need to open the file.

For each file that has less than 100% test coverage, find the matching file in cov_annotate and review the file.

If a line starts with a ! (exclamation mark), it means that the line is not covered by tests. Add tests to cover the missing lines.

## Sending pull requests

When sending pull requests, make sure to follow the PULL_REQUEST_TEMPLATE.md format.

## Upgrading dependencies

### Python backend dependencies

To upgrade a particular package in the backend, use the following command, replacing `<package-name>` with the name of the package you want to upgrade:

```shell
cd app/backend && uv pip compile requirements.in -o requirements.txt --python-version 3.10 --upgrade-package <package-name>
```

After upgrading, run tests to verify compatibility:

```shell
source .venv/bin/activate
pytest tests/
```

### npm frontend dependencies

To upgrade a particular package in the frontend:

1. **Navigate to the frontend directory**:

   ```shell
   cd app/frontend
   ```

2. **Upgrade the package** (replace `<package-name>` with the package you want to upgrade):

   ```shell
   npm install <package-name>@latest
   ```

3. **Build the frontend** to verify the upgrade works:

   ```shell
   npm run build
   ```

4. **Run all tests** to ensure nothing broke:

   ```shell
   # Run e2e tests from the root directory
   cd ../..
   source .venv/bin/activate
   pytest tests/e2e.py
   ```

5. **Commit changes** if the upgrade is successful:

   ```shell
   git add package.json package-lock.json
   git commit -m "chore: upgrade <package-name> to <version>"
   ```

**Important notes for frontend upgrades**:

* When upgrading React or related core packages, you may need to upgrade multiple packages together (e.g., `react`, `react-dom`, `@types/react`, `@types/react-dom`)
* Some upgrades may require code changes for API compatibility - check the package's changelog
* For major version upgrades of UI libraries like Fluent UI or MSAL, review breaking changes carefully. Manual tests are required for any MSAL changes since the E2E tests do not cover authentication flows.
* If npm reports peer dependency conflicts, the `.npmrc` file has `legacy-peer-deps=true` which allows the install to proceed. This is currently needed because `react-helmet-async` declares peer dependencies on React 17/18, but works fine with React 19.

## Manual test plan for authentication changes (msal, msal-browser, authentication.py)

The unit tests mock `msal` at the client level, so any change to `msal`, `@azure/msal-browser`, `cryptography`, or `app/backend/core/authentication.py` needs a live deploy against Entra to confirm the on-behalf-of (OBO) flow, token cache, and Container Apps Easy Auth integration still work.

Use a dedicated azd env with login + access control enabled so the OBO code path is actually exercised.

1. **Create the env and enable auth:** (any env name works — the steps below use `<AUTH_ENV_NAME>` as a placeholder)

   ```shell
   azd env new <AUTH_ENV_NAME> --subscription <SUB_ID> --location <REGION>
   azd env set AZURE_USE_AUTHENTICATION true
   azd env set AZURE_ENFORCE_ACCESS_CONTROL true
   azd env set AZURE_ENABLE_UNAUTHENTICATED_ACCESS false
   azd env set AZURE_AUTH_TENANT_ID <YOUR_TENANT_ID>
   azd env set AZURE_TENANT_ID <YOUR_TENANT_ID>
   # If the chosen region is capacity-constrained for Azure AI Search,
   # override just the Search location:
   azd env set AZURE_SEARCH_SERVICE_LOCATION <OTHER_REGION>
   ```

2. **Provision + deploy:**

   ```shell
   ./scripts/auth_init.sh   # creates the client + server Entra app registrations
   azd up -e <AUTH_ENV_NAME>
   ```

   `azd up` runs `auth_update.sh` as a postprovision hook to update the client app's redirect URIs.

3. **Ingest data for this env.** The `.md5` marker files under `data/` are shared across envs and cause `prepdocs` to skip everything on a fresh env, leaving the search index empty. Remove them first, then re-ingest:

   ```shell
   rm data/*.md5
   ./scripts/prepdocs.sh
   ```

4. **Apply an ACL to at least one document for your user oid** (find your oid via `az ad signed-in-user show --query id -o tsv`):

   ```shell
   python scripts/manageacl.py -v --acl-action add --acl-type oids \
     --acl <YOUR_OID> \
     --url "https://<storage-account>.blob.core.windows.net/content/Northwind_Standard_Benefits_Details.pdf"
   ```

5. **Verify auth enforcement with curl** (both should return `HTTP 401` because Container Apps Easy Auth blocks unauthenticated traffic before the app sees the request):

   ```shell
   BASE=https://<your-backend-fqdn>
   curl -sS -o /dev/null -w "%{http_code}\n" "$BASE/auth_setup"
   curl -sS -o /dev/null -w "%{http_code}\n" -X POST "$BASE/chat" \
     -H "Content-Type: application/json" \
     -d '{"messages":[{"role":"user","content":"hi"}]}'
   ```

6. **Verify the OBO flow in the browser:**
   * Load the app URL — you should be redirected to Entra sign-in.
   * Sign in with the tenant user whose oid you ACL'd.
   * Ask a question that only the ACL'd document can answer (e.g. "What is included in the Northwind Standard plan?"). You should get an answer with citations only from that document.
   * Ask a question about a document that is NOT ACL'd to you (e.g. "What is included in the Northwind Health Plus plan?"). You should get "I don't know" / no citations. This confirms the OBO token was issued by `msal.ConfidentialClientApplication.acquire_token_on_behalf_of` and correctly passed to Azure AI Search as the access-control filter.

7. **Verify the popup login flow works in local dev.** The deployed Container Apps site uses Easy Auth redirect for login; the local dev server uses the MSAL popup flow, which is a completely different code path. Do NOT skip this — the msal-browser 5.x popup requires `/redirect` to serve the dedicated [`app/frontend/redirect.html`](app/frontend/redirect.html) bridge page (in dev the vite server owns it via a `/redirect` → `/redirect.html` middleware rewrite; in prod the Quart `/redirect` route serves the built `redirect.html` from `static/`), and previous upgrades have silently broken this.

   ```shell
   # Terminal 1: backend, pointed at the deployed login env's config
   azd env select <AUTH_ENV_NAME>
   PORT=50505 ./app/start.sh

   # Terminal 2: frontend dev server
   cd app/frontend && BACKEND_PORT=50505 npm run dev
   ```

   Open `http://localhost:5173/`, click **Login**, complete the Entra popup, and confirm:
   * The popup closes automatically after auth (does NOT leave the popup stuck on `http://localhost:5173/redirect#code=...`).
   * The Login button in the top bar switches to your username.
   * Ask a question and confirm ACL-filtered citations come back the same as in step 6.
   * Click **Logout**, then **Login** again. The popup should complete cleanly a second time.

   If the popup gets stuck on the redirect URL, check that `/redirect` serves the dedicated [`app/frontend/redirect.html`](app/frontend/redirect.html) page (built from [`app/frontend/src/redirect.ts`](app/frontend/src/redirect.ts)) which runs `broadcastResponseToMainFrame()` from `@azure/msal-browser/redirect-bridge`. Per MSAL best practice this must be a minimal HTML page with ONLY the bridge script — no routing, no other application code. msal-browser 5.x uses a `BroadcastChannel` handshake and a truly blank redirect page no longer works.

8. **Optional: log out and reload on the deployed site.** Confirms Easy Auth logout works and re-auth kicks in cleanly.

If any step fails, check container logs — `AuthError` from `app/backend/core/authentication.py` usually indicates the OBO token exchange or token validation broke.

## Manual test plan for msgraph-sdk / microsoft-kiota-* changes

`msgraph-sdk` (and its transitive `microsoft-kiota-*` deps) is only imported by `scripts/auth_init.py` and `scripts/auth_update.py`, which register Entra client + server apps for the login-enabled deploy. The unit tests in `tests/test_auth_init.py` mock `GraphServiceClient` end-to-end, so a bump can pass CI while breaking a real Graph call (usually due to renamed request-body classes or new required fields).

Validate against a live tenant. If you don't already have a login-enabled deploy, follow steps 1–2 of the msal test plan above to create + provision an auth-enabled azd env (any env name works — the steps below use `$AZURE_ENV_NAME` from your current azd env).

1. **First run — exercises PATCH + query paths only.** If the client and server app registrations from a prior run already exist, `auth_init.sh` short-circuits the `applications.post()` create path:

   ```shell
   ./scripts/auth_init.sh
   ```

   Look for `Application already exists, not creating new one` — that means the POST paths were skipped. The PATCH paths (`applications.by_application_id().patch()`) for permissions + known-client-apps *did* run, plus `oauth2_permission_grants` queries.

2. **Force the create path** to exercise `applications.post()`, `service_principals.post()`, and `applications.by_application_id().add_password.post()` (where major SDK bumps most often break):

   ```shell
   # Grab the existing app IDs from the azd env
   CLIENT_APP=$(azd env get-value AZURE_CLIENT_APP_ID)
   SERVER_APP=$(azd env get-value AZURE_SERVER_APP_ID)

   # Delete both Entra app registrations (safe on a test env)
   az ad app delete --id $CLIENT_APP
   az ad app delete --id $SERVER_APP

   # Clear the cached IDs and secrets so auth_init recreates from scratch
   azd env set AZURE_CLIENT_APP_ID ""
   azd env set AZURE_CLIENT_APP_SECRET ""
   azd env set AZURE_SERVER_APP_ID ""
   azd env set AZURE_SERVER_APP_SECRET ""

   ./scripts/auth_init.sh
   ```

   You should see `Creating application registration` (twice — once for server, once for client), followed by `Granted admin consent for ...` messages. Every msgraph SDK code path in `auth_init.py` is now exercised.

3. **Test the update path** by running the postprovision hook that updates client-app redirect URIs:

   ```shell
   ./scripts/auth_update.sh
   ```

   Look for `Application update for client app id ... complete.`

4. **Re-run the msal test plan** end-to-end (browser sign-in, ACL'd doc question, non-ACL'd doc question) since the fresh apps will have new client IDs. `azd deploy backend` after `auth_init.sh` picks up the new IDs.

If POST fails with a serialization / model-class error (e.g. `AttributeError` on a model instance, or a 400 from Graph complaining about a missing property), the bumped SDK likely renamed or relocated a request-body class. Check the msgraph-sdk release notes and update the corresponding import in `scripts/auth_init.py`.

## Checking Python type hints

To check Python type hints, use the following command:

```shell
ty check
```

Note that we do not currently enforce type hints in the tests folder, as it would require adding a lot of `# type: ignore` comments to the existing tests.
We only enforce type hints in the main application code and scripts.

## Python code style

Do not use single underscores in front of "private" methods or variables in Python code. We do not follow that convention in this codebase, since this is an application and not a library.

## Starting the app locally

The simplest way to start the app is with `./app/start.sh` (or `./app/start.ps1` on Windows). This builds the frontend and starts the backend — no separate frontend process needed unless you want hot reloading.

To avoid port conflicts (e.g. if another instance is already running), pick a random port:

```shell
PORT=50506 ./app/start.sh
```

On Windows (PowerShell):

```powershell
$env:PORT = 50506
./app/start.ps1
```

If you also need the frontend dev server with hot reloading (for UI changes), run it separately with the matching `BACKEND_PORT`:

```shell
# Terminal 1: backend
PORT=50506 ./app/start.sh

# Terminal 2: frontend with HMR
cd app/frontend && BACKEND_PORT=50506 npm run dev
```

**Tips for coding agents**: Always specify your own random port via `PORT` to avoid colliding with a developer's running instance or other parallel agents. The start scripts will detect if the port is already in use and tell you to pick a different one.

## Deploying the application

To deploy the application, use the `azd` CLI tool. Make sure you have the latest version of the `azd` CLI installed. Then, run the following command from the root of the repository:

```shell
azd up
```

That command will BOTH provision the Azure resources AND deploy the application code.

If you only changed the Bicep templates and want to re-provision the Azure resources, run:

```shell
azd provision
```

If you only changed the application code and want to re-deploy the code, run:

```shell
azd deploy
```

If you are using cloud ingestion and only want to deploy individual functions, run the necessary deploy commands, for example:

```shell
azd deploy document-extractor
azd deploy figure-processor
azd deploy text-processor
```