## 1. Project Overview & Quickstart (nlweb-ai/NLWeb) ## File: README.md # What is NLWeb? **NLWeb** simplifies the process of building conversational interfaces for websites. It natively supports MCP (Model Context Protocol), allowing the same natural language APIs to serve both humans and AI agents. Schema.org and related semi-structured formats like RSS — used by over 100 million websites — have become not just de facto syndication mechanisms, but also a semantic layer for the web. NLWeb leverages these to enable natural language interfaces more easily. NLWeb is a collection of open protocols and associated open source tools. Its main focus is establishing a foundational layer for the AI Web — much like HTML revolutionized document sharing. To make this vision reality, NLWeb provides practical implementation code—not as the definitive solution, but as proof-of-concept demonstrations showing one possible approach. We expect and encourage the community to develop diverse, innovative implementations that surpass our examples. This mirrors the web's own evolution, from the humble 'htdocs' folder in NCSA's http server to today's massive data center infrastructures—all unified by shared protocols that enable seamless communication. AI has the potential to enhance every web interaction. Realizing this requires a collaborative spirit reminiscent of the Web's early "barn raising" days. Shared protocols, sample implementations, and community participation are all essential. NLWeb brings together protocols, Schema.org formats, and sample code to help sites quickly implement conversational endpoints — benefitting both users through natural interfaces and agents through structured interaction. Join us in building this connected web of agents. ## How It Works NLWeb has two primary components: 1. **A simple protocol** to interact with a site using natural language. It returns responses in JSON using Schema.org. See the [NLWeb spec](https://nlweb.ai/spec) for details. 2. **A straightforward implementation** that uses existing markup on sites with structured lists (e.g., products, recipes, attractions, reviews). Combined with UI widgets, this enables conversational interfaces to be added with ease. ## NLWeb and MCP/A2A MCP and A2A are emerging standards for enabling chatbots and AI assistants to interact with tools and each other. Every NLWeb instance also acts as an MCP server (and soon A2A) and supports a core method, `ask`, which allows a natural language question to be posed to a website. The response returned uses Schema.org — a widely adopted vocabulary for describing web data. **In short, NLWeb is to MCP/A2A what HTML is to HTTP.** ## Platform Compatibility NLWeb is platform-agnostic and supports: * **Operating systems**: Windows, macOS, Linux * **Vector stores**: [Qdrant](docs/setup-qdrant.md), [Snowflake](docs/setup-snowflake.md), [Milvus](docs/setup-milvus.md), [Azure AI Search](docs/setup-azure.md), [Elasticsearch](docs/setup-elasticsearch.md), [Postgres](docs/setup-postgres.md), [Cloudflare AutoRAG](docs/setup-cloudflare-autorag.md) * **LLMs**: OpenAI, DeepSeek, Gemini, Anthropic, Inception, [HuggingFace](docs/setup-huggingface.md) It is designed to be lightweight and scalable — capable of running on everything from data center clusters to laptops and, soon, mobile devices. ## Repository Structure This repository is organized into the following modules: * **[AskAgent](AskAgent/)** — The core NLWeb query agent. Handles natural language queries against websites using Schema.org structured data, with connectors for popular LLMs and vector databases, data ingestion tools, and a sample web UI. * **[AgentFinder](AgentFinder/)** — Agent discovery service for finding and routing to NLWeb agents across the web. * **[DataFinder](DataFinder/)** — Natural language to SQL translator for enterprise data sources (HubSpot, Dynamics 365, Jira) using schema.org-based ontology mappings. * **[ModelRouter](ModelRouter/)** — LLM model routing and scoring, selecting cost-effective models that meet quality thresholds. * **[NLWebScorer](NLWebScorer/)** — Neural scorer models for ranking and evaluating search result quality. Supporting directories: * **[config](config/)** — YAML configuration files for LLM providers, embedding models, retrieval backends, and web server settings. * **[static](static/)** — Frontend web UI assets (HTML, CSS, JavaScript) served by the web server. * **[demo](demo/)** — Demo scripts and example data sources for getting started. * **[scripts](scripts/)** — CLI utilities and helper scripts. * **[docs](docs/)** — Full documentation. Most production deployments will: * Use their own user interface * Integrate NLWeb directly into their application environment * Connect NLWeb to live databases instead of duplicating content (to avoid freshness issues) ## Documentation ### Getting Started * [Hello world on your laptop](docs/nlweb-hello-world.md) * [Running it on Azure](docs/setup-azure.md) * Running on GCP — *coming soon* * Running on AWS — *coming soon* ### NLWeb Details * [Modifying Prompts](docs/nlweb-prompts.md) * [Changing Control Flow](docs/nlweb-control-flow.md) * [Modifying the User Interface](docs/nlweb-user-interface.md) * [REST API](docs/nlweb-rest-api.md) * [Adding Memory](docs/nlweb-memory.md) * [Using the Check Connectivity Script to Test your Configuration](docs/nlweb-check-connectivity.md) ## License NLWeb uses the [MIT License](LICENSE). ## Deployment (CI/CD) CI/CD pipelines are not yet included. Contributions to add automated testing or deployment workflows are welcome. ## Access For questions about this GitHub project, please contact [NLWeb Support](mailto:NLWebSup@microsoft.com). ## Contributing See [Contribution Guidance](CONTRIBUTING.md) for more details. ## Contributor Wall of Fame [](https://github.com/microsoft/nlweb/graphs/contributors) ## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies. --- ## File: demo/README.md # Demos [Import data from an RSS feed](#import-data-from-an-rss-feed) [Agent-enable your Github data](#agent-enable-your-github-data) [Ask questions of clinical trial data](#ask-questions-of-clinical-trial-data) ## Import data from an RSS feed With NLWeb, you can easily import data from an RSS feed for querying over in natural language using a script. First, navigate to the NLWeb --> AskAgent --> python directory. Before you run this command, ensure that the database you want to write to is set as the `preferred_endpoint` in your config_retrieval.yaml file in the config directory (or use the --database switch). In this example, I am using qdrant_local. The format for this command is the following. Replace with an RSS feed, and choose a descriptive site name for that data. ```sh # Run from the AskAgent/python folder python -m data_loading.db_load ``` As an example, here is the RSS feed for Kevin Scott's podcast "Behind the Tech". This command will extract the data from the RSS feed, create embeddings, and store those embeddings in the vector database specified in the config_retrieval.yaml file. ```sh # Run from the AskAgent/python folder python -m data_loading.db_load https://feeds.libsyn.com/121695/rss behindthetech ``` Now, using our debug tool, you can easily ask questions about your Github data in natural language. Start your web server by running `python app-file.py` from the **AskAgent/python** directory. Then in a web browser, navigate to http://localhost:8000/static/str_chat.html. Select "behindthetech" from the site dropdown and your retrieval provider from the database dropdown (I am using "Qdrant Local"). Then ask questions in natural language. If you have created a new site name, you will need to add this to the list of site options in the dropdown-interface.js file in the static directory for it to appear in the tool above. NOTE: to remove the "behindthetech" data from your vector database, run this: ```sh # Run from the AskAgent/python folder python -m data_loading.db_load --only-delete delete-site behindthetech ``` ## Agent-enable your Github data Let's create an agent that utilizes NLWeb over GitHub data. First, follow these instructions to get a fine-grained personal access token from GitHub: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token Copy the .env.example file into a new file called .env. In the .env file, update the value of GITHUB_TOKEN to the value of the token you generated. First, we will run a script to get your Github data into a format that NLWeb can consume. It will output a json file. From the **demo** folder, run: ```sh # Run from the demo folder python extract_github_data.py ``` Then, you can extract this data, create embeddings, and import them into your retrieval provider with this script. From the **AskAgent/python** folder, run the db_load tool, pointing to the json file that you just created and giving it a site name like "github": ```sh # Run from the AskAgent/python folder python -m data_loading.db_load ../../demo/ghrepoinfo.json github ``` Finally, you can repeat the same process as above. Start the web server with the below command. ```sh python app-file.py ``` Then in a web browser, navigate to http://localhost:8000/static/str_chat.html. You can now ask questions of your data. ## Ask questions of clinical trial data In this demonstration, we will import data on clinical trials that is available on the website https://clinicaltrials.gov. First, search the website and download the latest clinical trial data on any topic. For example, you can search for "cancer" with this query: https://clinicaltrials.gov/expert-search?term=Cancer Then, on the search results page, click the "Download" button. This will present a screen like the following. Choose to download JSON (and check the box to put each study into a separate file and download them as a zip archive), the number of results you would like, and all available data fields. If you encounter any issues, there is more information on how to download the clinical trial data at https://clinicaltrials.gov/data-api/how-download-study-records. Once you have downloaded the zip file, extract all files from the zip into a directory and note the name of that directory. In the sample code, we have downloaded to `C:\Data\ctg-studies`. In the demo folder, open the file called `import_clinical_trials.py` and set the `json_dir` variable near the top to the value of the directory where your extracted json files are. Now, we will run two commands to process the data. These commands should be run from the "AskAgent/python" directory. In the second command, replace 'C:\Data\ctg-studies' with the value of the directory where you extracted the files (**but note that 'processed' should remain appended to the end**). ``` python ..\..\demo\import_clinical_trials.py python -m data_loading.db_load C:\Data\ctg-studies\processed\ CancerClinicalTrials --directory ``` Finally, you can query the data that you have imported. Still in the AskAgent/python directory, start the service by running: ``` python app-file.py ``` Open a web browser and navigate to http://localhost:8000/static/fp_chat.html. You should be able to ask questions of this data. --- ## File: docs/release_notes/2025-06-01-release.md # Release Notes Summary - Microsoft/NLWeb Updates since June 24, 2025 ## 🔒 Security Improvements - **Fix path traversal vulnerability in static file handler (#231)** - Added security checks to prevent directory traversal attacks, file extension whitelist, and logging for security events. - **Checking URL in a more secure way in streaming.js (#236)** - Improved URL validation to prevent domain spoofing attacks by using proper URL parsing instead of unsafe `startsWith` checks. ## 🔧 Code Quality & Refactoring - **Gemini Developer API now fixed (#232)** - Updated Gemini integration to use the Developer API. (Note that Vertex API is not supported currently as its API follows a different authentication pattern than we are using today.) This fixes issues #138, #107, and #112. - **Modular Code Refactor (#244)** - There is a new folder called 'refactored' with the new architecture. With this refactor, there are fewer file changes required to add new providers or tools, which we going forward will refer to as 'methods' to avoid confusion with the term 'tools' as used in other contexts such as MCP. These accordingly appear in the new 'methods' folder. We plan to cut the repo over to using this new architecture in the coming couple of weeks after we get things tested & stabilized. ## ✨ New Features - **Elasticsearch retrieval (#229)** - Added Elasticsearch retrieval functionality with [documentation](https://github.com/microsoft/NLWeb/blob/main/docs/setup-elasticsearch.md). - **Stats query handler (#226)** - New statistics query handling capability for visualizing data using Data Commons. You can learn to use this tool via the [Statistics Tool Documentation](https://github.com/microsoft/NLWeb/blob/main/docs/tools-statistics.md) provided, including sample query patterns. ## 📝 Documentation & Maintenance - **Documentation name update (#240)** - Some files had been added in different PRs that didn't match the standard naming conventions. These were updated to the standard format and fixed links throughout repo. --- ## File: docs/release_notes/2025-06-23-release.md # NLWeb Release Notes - June 23 2025 This release introduces significant enhancements to NLWeb's retrieval system, adds powerful tool calling capabilities, and provides new configuration options for response headers. ## Major Features ### 1. Multi-Backend Retrieval System (PR #214) NLWeb now supports concurrent querying across multiple retrieval backends, providing improved performance and redundancy. **Key improvements:** - Multiple backends can be simultaneously active (previously only one at a time) - Parallel querying across all enabled backends with automatic deduplication - Configurable write endpoint while reading from multiple sources - Support for Azure AI Search, Qdrant, Milvus, OpenSearch, and Snowflake backends [Learn more about the retrieval system →](../nlweb-retrieval.md) ### 2. Tool Calling Framework (PRs #219, #208, #217) A new extensible tool system enables specialized handling of different query types beyond simple search. **Available tools:** - **Search**: Traditional keyword and semantic search (default) - **Details**: Retrieve specific information about named items (PR #217) - **Compare**: Side-by-side comparison of two items - **Ensemble**: Create cohesive sets of related items (PR #219) - **Recipe Tools**: Ingredient substitutions and accompaniment suggestions (PR #208) **Example ensemble queries:** - "Give me an appetizer, main and dessert for an Italian dinner" - "I'm visiting Seattle for a day - suggest museums and nearby restaurants" - "What should I wear for hiking in Colorado in winter?" - "Plan a romantic date night with dinner and entertainment" - "Create a workout routine with warmup, main exercises, and cooldown" [Explore the tools system →](../tools.md) ### 3. Configurable Response Headers (PR #205) NLWeb instances can now define custom headers that are sent as messages at the beginning of each response, enabling: - License specification (e.g., MIT License with link to terms) - Data retention policies (e.g., "may be retained for up to 1 day") - UI component specifications for rendering results - Custom metadata for your deployment [Configure response headers →](../nlweb-headers.md) ### 4. Enhanced Debug Panel The web interface debug panel has been significantly improved: - Real-time display of tool selection process - Streaming updates for multi-stage operations - Clear visualization of backend queries - Performance metrics for each operation phase ### 5. Comprehensive Testing Framework (PR #167) New testing infrastructure ensures reliability: - End-to-end query testing with configurable test suites - Multi-backend retrieval verification - Tool selection accuracy tests - Performance benchmarking utilities - Database operation testing ## Configuration Changes ### Retrieval Configuration ```yaml # New format in config_retrieval.yaml write_endpoint: qdrant_local endpoints: backend_name: enabled: true # Enable/disable without removing config db_type: azure_ai_search # ... other settings ``` ### Tool Selection - Enable/disable tool selection via `tool_selection_enabled` in config_nlweb.yaml - Tools defined in XML for easy customization - Per-type tool inheritance following schema.org hierarchy ## Migration Notes ### From Single to Multi-Backend 1. Update `config_retrieval.yaml` to new format 2. Set `enabled: true` for existing backend 3. Ensure `write_endpoint` points to primary backend 4. Test thoroughly before enabling additional backends ### Tool System Adoption - Tool selection is enabled by default - Set `tool_selection_enabled: false` to maintain previous behavior - Queries with `generate_mode` set to "summarize" or "generate" skip tool selection ## Performance Improvements - Parallel backend queries reduce latency for multi-source deployments - Async tool evaluation speeds up request routing - Optimized deduplication algorithm for large result sets - Streaming responses for better perceived performance ## Developer Features - Simplified handler base class for creating custom tools - Improved logging system with configurable loggers per module - Better error handling with development/production modes - Comprehensive type hints and documentation ## Bug Fixes - Fixed CONFIG UnboundLocalError in decontextualize module (PR #206) - Resolved ensemble tool ranking errors with tuple handling (PR #219) - Corrected multi-database connectivity issues (PR #214) - Fixed FastTrack abort conditions for non-search tools - Improved decontextualization with URL-based retrieval support (PR #206) ## Coming Soon - Remote tool support for distributed NLWeb instances - MCP (Model Context Protocol) server integration - Dynamic tool loading from external sources - Enhanced ensemble strategies for complex queries - GraphQL API endpoint option ## Upgrading 1. Back up your configuration files 2. Update retrieval backend configuration to new format 3. Review and adjust tool selection settings 4. Test with your existing queries 5. Enable new features incrementally For detailed upgrade instructions, see the documentation for each major feature area. --- *For questions or issues, please refer to our [GitHub repository](https://github.com/microsoft/NLWeb) or documentation.* --- ## File: docs/release_notes/2025-07-29-release.md # NLWeb Release Notes Updates from July 9 - July 29, 2025 ### Major Features & Enhancements - **Ollama Support Added** (#184) - Added comprehensive support for Ollama as an LLM provider - **Clinical Trials Demo** (#247) - New demonstration showcasing clinical trials data upload with comprehensive documentation ### Codebase Cleanup - Removed legacy old_code and old_static directories (#269) - Code refactoring improvements and critical path fixes (#267) ### Documentation Updates - Updated README with PostgreSQL information (#294) - General README improvements and clarifications (#278) ### Bug Fixes & Improvements - Added requirement for item type specification in URL parameters (#297) - Fixed state management naming issues for ToolSelector step completion (#296) - Resolved post-refactoring issues with Docker-based installations (#295) - Fixed Gemini embedding bug (#291) - Fixed MCP test interface to support both standard and stream requests (#285) - Fixed issue where NLWS message types were not displayed properly (#283) - Fixed Elasticsearch post-refactoring (#281) - Improved get_param function to properly handle list parameters (#273) - Added CodeQL Fork Analysis Workflow for automated security scanning (#241) --- *Total PRs merged: 15* *Contributors: 11 developers* --- ## File: docs/life-of-a-chat-query.md # Life of a Chat Query NLWeb aims to make it very simple to provide conversational interfaces to websites (or more generally, collections of content) which can be abstracted as 'lists of items'. Items maybe recipes, events, products, books, movies, etc. NLWeb leverages the fact that most such websites already make their data available in a structured form, in a common vocabulary, namely that provided by Schema.org. Given the widespread prevalence of schema.org based markup, it is not surprising that most LLMs seem to understand schema.org markup very well. We exploit this to make it easy to create conversational interfaces. This document has a brief description of the processing that is done when a User submits a query to an NLWeb instance. At a high level, the flow is very similar to the processing of a query in modern Web search (where the results are not just a list of 10 links, but could involve more sophisticated 'tools'). The main difference is that in 'traditional' (i.e., pre-llm) search engines, there would be specialized algorithms or special purpose models for many of the tasks involved in the query processing. This both made it very expensive to develop robust search tools and also made them somewhat limited. Here, we rely on LLMs to perform these tasks. Depending on the request parameters, control of the results that are returned can stay with 'traditional' code, which affords greater control over the returned results. In particular, the result can be a list of items, each of which includes the data item corresponding to that result, preventing hallucination of items. So, results can be less than most relevant, but a result will not be 'made up'. 1. User submits next query in the conversation 2. Multiple parallel calls are made for checking relevancy, decontextualizing query based on conversation history, determining if there are items that should be remembered in memory, etc. Each of these is implemented as a call to an LLM, though alternate implementations are possible. At the end of this, we have a decontextualized query, which we know is relevant to the site, that we have all the information required to answer the query, etc. In some cases, this step might result in the query being broken down into multiple smaller queries. In some cases, the system may return a response (e.g., when more information is required for further processing) and not go any further. - 2b. Fast Track: We expect that most conversations, especially early on, will resemble search and will involve a query that is relevant, doesn't require decontextualization, etc. So, it is very likely that step 2 will not make any changes to what follows. Consequently, after a light weight check to see if this condition might hold, a 'fast track' path to (3) is launched, in parallel to (2). Results from (4) are blocked from being sent to the user until the results of the analysis from (2) are completed. In some cases, the results from the fast track channel may be entirely dropped. 3. Tool Selection & Execution: Based on the manifest in tools.xml, LLM calls are made to determine which tool is most appropriate for the query. The LLM also extracts the parameters required for that tool, which is then invoked. Tools may search the underlying vector database and make calls to LLMs as needed. The selected tool is executed with the extracted parameters. To give a flavor for how tools may process the request, we describe how three tools work: **Search Tool**: This loosely follows the traditional search flow: - The (decontextualized) query is sent to a database service to retrieve potential answers - Typically uses a vector database with retrieval based on tfidf scores on embeddings and structured data constraints - Results are returned as json objects encoded in schema.org schema - Results are scored using specific LLM calls, which may also generate appropriate snippets - Top N results above a threshold are collected with score, snippet, and database object **Item Details Tool**: Retrieves specific information about items: - Items are specified by name, description, or contextual reference (e.g., "ingredients in the olive oil cake recipe") - First queries the vector database for candidate items - Scores items using an LLM to match candidates against the item description - Extracts requested details either within the scoring call or via a separate LLM call **Ensemble Queries Tool**: Combines multiple items of different types: - Handles queries like "appetizer, entree and dessert, Asian fusion themed for a summer party" - Extracts separate queries for each type of item from the prompt - Makes independent vector database queries to retrieve candidates - Ranks candidates using LLM calls for appropriateness - Selects top 2-3 from each query and sends all to an LLM to create ensembles 4. The results returned from the database are scored. This is again done with a set of very specific calls to an LLM. The LLM may also be asked to generate a 'snippet' that is appropriate for the query. The top N results that have a score above some threshold, together with the score, snippet and the associated database object are collected. 4a. Optional: If the user has requested post processing, this is done and the results from (4), together with the results from post processing are returned to the user. Post processing may for example summarize the results in (4) or go a step further and try to use the results from (4) to answer the query. 5. The results are then returned to the user in the specified format. ## Notes - Processing a single query might involve over 50 LLM API calls. The calls tend to be very narrow and specific. Different kinds of calls may be to different models. The prompts can be specialized, declaratively, for particular object types (with the default type hierarchy from schema.org), e.g., Recipe vs Real Estate and further for specific sites. - Tools may be domain-specific and bring in additional knowledge based on the task. For example, a recipe-specific tool might handle substitutions for dietary restrictions or missing ingredients. - Since the items that are returned each come from the database, the user can be assured that none of the results are 'made up'. There is of course the possibility that the results are not the best, but there will not be results returned that are not in the database. Post processing may degrade this, however, so be sure to test any you add carefully. - The system is designed to be extensible, with plans to enable calling other NLWeb/MCP servers in the future, allowing for distributed tool execution across different services. ## 2. Official Technical Reference & Guides (nlweb-ai/website) ## File: README.md # NLWeb.ai Website The official documentation website for NLWeb - a protocol that simplifies building conversational interfaces for websites and enables natural language interactions with AI agents. ## Overview This is a Next.js-based documentation website that provides comprehensive information about: - **NLWeb Protocol**: A standardized interface for natural language interactions with agents - **Agent Finder**: Software that helps agents discover and integrate with NLWeb-compatible websites - **Implementation Guides**: Practical examples and quickstart guides ## Tech Stack - **Framework**: Next.js 16.1.1 with React 19 - **Documentation**: MDX (Markdown + JSX) using Nextra - **Styling**: Tailwind CSS 4 - **UI Components**: Radix UI primitives - **Syntax Highlighting**: react-syntax-highlighter ## Getting Started ### Prerequisites - Node.js (v20 or higher recommended) - pnpm ### Installation ```bash pnpm install ``` ### Development Run the development server: ```bash pnpm dev ``` Open [http://localhost:3000](http://localhost:3000) to view the website. ### Build Build the production-ready site: ```bash pnpm build ``` ### Start Production Server ```bash pnpm start ``` ## Project Structure ``` nlweb-internal-website/ ├── app/ # Next.js app directory │ ├── docs/ # Documentation pages │ │ ├── intro/ # Introduction pages │ │ ├── about/ # About and FAQ │ │ ├── nlweb-core/ # NLWeb Core software docs │ │ └── agent-finder/ # Agent Finder docs │ │ ├── overview/ # Agent Finder overview │ │ ├── motivation/ # Why Agent Finder exists │ │ ├── quickstart/ # Getting started guide │ │ └── api/ # API documentation │ └── specification/ # Protocol specification │ ├── protocol/ # Core protocol docs │ │ ├── overview/ # Protocol overview │ │ ├── ask/ # Ask query structure │ │ ├── response/ # Response structure │ │ ├── actions/ # Actions specification │ │ └── binding/ # HTTP transport binding │ └── appendix/ # Additional examples │ ├── http/ # HTTP payload examples │ └── mcp/ # MCP payload examples ├── components/ # Reusable React components ├── NLWEBSPEC.md # Source specification document ├── AGENTFINDER.md # Agent Finder design document └── CLAUDE.md # Project instructions for Claude AI ``` ## Documentation ### How Documentation Works This website uses MDX files for documentation. MDX allows you to use JSX components directly in Markdown, making it powerful for creating interactive documentation. ### Route Structure Routes are automatically generated from the folder structure in the `app/` directory. The file system path directly maps to the URL: - `app/docs/intro/page.mdx` → `/docs/intro` - `app/specification/protocol/overview/page.mdx` → `/specification/protocol/overview` - `app/docs/agent-finder/motivation/page.mdx` → `/docs/agent-finder/motivation` This means adding new pages is as simple as creating a new folder with a `page.mdx` file inside. ### Modifying Existing Documentation #### 1. Find the MDX File Documentation pages are located in the `app/` directory. The file structure mirrors the URL structure: - `/docs/intro` → `app/docs/intro/page.mdx` - `/specification/protocol/overview` → `app/specification/protocol/overview/page.mdx` - `/docs/agent-finder/motivation` → `app/docs/agent-finder/motivation/page.mdx` #### 2. Edit the MDX File Open the relevant `page.mdx` file and edit it. MDX supports: **Standard Markdown:** ```mdx # Heading 1 ## Heading 2 ### Heading 3 Regular text with **bold** and *italic*. - Bullet points - More points 1. Numbered lists 2. Continue... ``` **Code Blocks:** ````mdx ```json { "query": { "text": "example" } } ``` ```` **React Components:** ```mdx ``` #### 3. Test Your Changes Run the development server to see your changes live: ```bash pnpm dev ``` ### Adding New Documentation Pages #### 1. Create the MDX File Create a new `page.mdx` file in the appropriate directory. Follow the existing structure: ``` app/ └── docs/ └── your-section/ └── your-page/ └── page.mdx ``` #### 2. Add Content Start with a heading and write your content: ```mdx # Your Page Title Your content goes here. Use Markdown syntax and JSX components as needed. ## Section 1 More content... ## Section 2 Even more content... ``` #### 3. Verify Your Page - Navigate to the URL that matches your folder path (e.g., `/docs/your-section/your-page`) - Test internal links to and from your page - Verify the page renders correctly with all formatting ### Documentation Style Guidelines When writing or modifying documentation, follow these conventions (based on existing pages): 1. **Use Clear Headings**: Start with an H1 (`#`) for the page title, use H2 (`##`) for main sections 2. **Code Examples**: Include practical code examples in appropriate language blocks 3. **Consistent Structure**: Look at `app/specification/protocol/overview/page.mdx` as a reference 4. **Schema.org References**: When applicable, reference Schema.org vocabulary 5. **Links**: Use relative links for internal pages, absolute for external ### Source Documents - **NLWEBSPEC.md**: The authoritative NLWeb protocol specification - **AGENTFINDER.md**: Design philosophy and specification for Agent Finder - **CLAUDE.md**: Instructions for AI-assisted development (project guidelines) When updating protocol documentation, consult `NLWEBSPEC.md` as the source of truth. ### Common Documentation Patterns #### Protocol Specification Pages Follow the pattern in existing spec pages: - Overview section explaining the concept - Detailed attributes/structure definitions - Code examples with syntax highlighting - Cross-references to related pages #### Software Documentation For software docs (NLWeb Core, Agent Finder): - Overview/motivation section - Quickstart guide with installation steps - API reference with examples - Use cases and best practices ## Components Reusable components are located in the `components/` directory. The site primarily uses: - Radix UI primitives for interactive elements - Custom components for documentation-specific needs - React Syntax Highlighter for code blocks When creating new pages, reuse existing components rather than creating new ones. ## Deployment This site can be deployed to any platform that supports Next.js: - **Vercel**: Automatic deployment with git integration - **Netlify**: Similar to Vercel - **Self-hosted**: Use `pnpm build && pnpm start` ## Contributing When contributing documentation: 1. Read the source specification documents (`NLWEBSPEC.md`, `AGENTFINDER.md`) 2. Follow existing page structure and styling conventions 3. Create new folders and `page.mdx` files for new pages 4. Test locally before submitting changes 5. Ensure all internal links work correctly ## License Please refer to the repository license file for licensing information. ## Resources - **NLWeb Protocol Spec**: Documented in this site at `/specification/protocol/overview` - **GitHub Repository**: [nlweb-ai organization](https://github.com/nlweb-ai) - **NLWeb Core**: Implementation software for websites - **Agent Finder**: Integration software for AI agents ## Support For questions or issues: - File an issue in the GitHub repository - Check the FAQ at `/docs/about/faq` - Review the specification documentation --- ## File: app/docs/intro/page.mdx # What is NLWeb? NLWeb is an open protocol for building conversational interfaces to websites and applications. It provides a standardized way for both humans and AI agents to interact with any site using natural language. The protocol builds on widely adopted, standard formats — Schema.org vocabularies used by over 100 million websites, JSON-LD for extensibility, and Server-Sent Events for streaming. NLWeb natively supports agentic protocols like MCP and A2A, and every NLWeb endpoint is also natively a ChatGPT app, making it instantly accessible to hundreds of millions of users. NLWeb has two primary components: 1. A [protocol specification](/docs/specification) that defines how to interact with any application using natural language — including query structure, response formats, streaming, and support for long-running tasks. 2. A [reference implementation](https://github.com/nlweb-ai/NLWeb) that makes it easy to add a conversational interface to any site with structured content (products, recipes, articles, reviews, and more). ## Implementations Beyond the reference implementation, NLWeb is being adopted by a growing number of platforms and companies, including [Cloudflare](https://www.cloudflare.com), [Tollbit](https://www.tollbit.com), [Wix](https://www.wix.com), and others — making it easy for millions of sites to become conversational. To see NLWeb in action, try [Microsoft News](https://news.microsoft.com/source) — a live example of the protocol powering a conversational interface on a major website. ## Get Involved NLWeb is fully open source. We invite the community to build on the protocol, create new implementations, and help shape the conversational web. **[Read the specification](/docs/specification)** or **[explore the code](https://github.com/nlweb-ai/NLWeb)**. --- ## File: app/docs/about/faq/page.mdx # Frequently asked questions ## Quick Links - [NLWeb vs MCP vs A2A?](#0) - [What's the difference between 'ask' and 'await'?](#1) - [How does NLWeb handle context across multiple queries?](#2) - [What response types can an NLWeb agent return?](#3) - [What's the deal with this site design?](#4) ---
### Q: NLWeb vs MCP vs A2A? The major difference is that NLWeb is designed for text-in, text-out natural language interactions. You don't need to understand how to call specific tools, define parameters, or handle complex tool schemas. Just ask in natural language and get structured responses back. In contrast, MCP and A2A are protocols where you need to understand available tools, their schemas, and how to invoke them correctly. NLWeb abstracts away this complexity by providing a unified 'ask' interface. NLWeb is also transport-protocol agnostic and can work over HTTP, WebSockets, JSON-RPC, or within agentic protocols like MCP and A2A, making it complementary to these protocols rather than competing with them.
---
### Q: What's the difference between 'ask' and 'await'? 'ask' is the primary API for querying an NLWeb agent with a natural language request. It can return immediate answers, elicitations (requests for more information), promises (for long-running tasks), or failures. 'await' is a helper API used to check the status of or cancel a long-running task that returned a promise. You use 'ask' to make your initial request, and if you receive a promise token back, you use 'await' with that token to check on the task's progress or get the final result.
---
### Q: How does NLWeb handle context across multiple queries? NLWeb treats context as a first-class object in both requests and responses. The request can include a 'context' section with conversation history (prev queries), free-form contextual text, and persistent user preferences or memory. The response includes a 'session_context' in the meta section that should be included in subsequent requests, similar to HTTP cookies. This enables agents to maintain conversational state, understand references to previous queries, and personalize responses based on accumulated user information.
---
### Q: What response types can an NLWeb agent return? An NLWeb agent can return four types of responses: 1. **Answer** - provides the requested information or confirms task completion with structured results 2. **Elicitation** - requests additional information from the user when the query is ambiguous or incomplete 3. **Promise** - returns a token for long-running operations that can be checked later using the 'await' API 4. **Failure** - indicates an error occurred with an error code and message The response type is specified in the '_meta.response_type' field.
---
### Q: What's the deal with this site design? This CSS has a lineage. It was originally created for the [Sitemaps protocol](https://www.sitemaps.org/) site, then adopted by [Schema.org](https://schema.org/), and now lives on at [NLWeb.ai](https://nlweb.ai). Three generations of web standards, one stylesheet. Thanks [Shiva Shivakumar](https://en.wikipedia.org/wiki/Narayanan_Shivakumar).
--- ## File: app/docs/schema/security/page.mdx # Security Considerations ## Content Authenticity The structured data provided in Schema Feeds MUST be consistent with the schema.org markup present on the corresponding HTML pages. Publishers SHOULD NOT include data in feeds that differs from what appears on their pages. Some consumers may not treat Schema Feed data with the same level of trust as markup extracted directly from HTML pages. Publishers should be aware that providing accurate, consistent data across both feeds and pages is essential for broad adoption. ## Data Validation Consumers SHOULD validate that the structured data in feeds conforms to schema.org definitions and SHOULD handle malformed data gracefully. --- ## File: app/docs/schema/schema-map/page.mdx # Schema Map Format ## Overview A Schema Map is an XML file that follows the sitemap protocol format. Each entry in the Schema Map points to a Schema Feed file and specifies its content type. ## XML Schema ```xml https://example.com/feeds/products.jsonl 2026-01-15 structuredData/schema.org https://example.com/feeds/articles.rss 2026-01-14 structuredData/rss ``` ## Elements ### `` The root element. MUST include the sitemaps namespace and SHOULD include the Schema Feeds namespace. ### `` A container for information about a single Schema Feed file. ### `` REQUIRED. The URL of the Schema Feed file. MUST be an absolute URL. ### `` OPTIONAL. The date the Schema Feed was last modified, in W3C Datetime format. Consumers MAY use this to avoid re-fetching unchanged feeds. ### `` REQUIRED. Specifies the format of the Schema Feed file. Defined values: | Value | Description | |-------|-------------| | `structuredData/schema.org` | JSON Lines file containing schema.org JSON-LD objects | | `structuredData/rss` | RSS 2.0 feed | Additional content types MAY be defined in future versions of this specification. ## Schema Map Index For large sites, Schema Maps MAY be organized using an index file, following the sitemap index pattern: ```xml https://example.com/schemamap-products.xml 2026-01-15 https://example.com/schemamap-articles.xml 2026-01-14 ``` --- ## File: app/docs/schema/introduction/page.mdx # Introduction **Version:** 0.1 (Draft) **Date:** January 2026 **Status:** Proposal --- ## Abstract This specification defines Schema Feeds, a mechanism for websites to provide aggregated structured data in schema.org format through a small number of files, rather than requiring consumers to crawl individual pages. Schema Feeds builds upon the familiar patterns of sitemaps and robots.txt to enable efficient discovery and retrieval of a site's complete structured data. --- ## Background Approximately fifteen years ago, the major search engines—Google, Bing, and Yahoo—recognized a fundamental opportunity: many websites are powered by structured databases, yet this structure is lost when content is rendered as HTML. If search engines could access the underlying structured data, they could provide richer, more accurate search results. However, different websites use different internal schemas. Reconciling these diverse schemas at web scale would be impractical. To address this, the search engines collaborated to create schema.org—a shared vocabulary of schemas that websites could use to expose their structured data in a common format. Today, tens of millions of websites publish schema.org markup, making it one of the most successful standards on the web. ## Problem Statement Despite the success of schema.org, a significant inefficiency remains: structured data is embedded within individual HTML pages. To collect all the structured data from a website, a consumer must crawl every page on the site, parse the HTML, and extract the embedded markup. This approach has several drawbacks: - **Inefficiency:** Crawling millions of pages to extract structured data is computationally expensive for both the crawler and the website. - **Latency:** Changes to structured data are only discovered when pages are re-crawled, which may take days or weeks. - **Incompleteness:** Crawlers may miss pages, resulting in incomplete data collection. - **Overhead:** The structured data is a small fraction of the page content, yet the entire page must be retrieved. ## Solution Overview Schema Feeds allows a website to publish all of its structured data in a small number of aggregated files. Consumers can retrieve these files directly, without crawling individual pages. This approach is analogous to how sitemaps allow search engines to discover URLs without crawling, and how RSS feeds allow aggregators to discover content without polling individual pages. --- ## File: app/docs/schema/implementation/page.mdx # Implementation Guidelines ## For Publishers 1. **Generate feeds from your database:** Rather than extracting markup from rendered pages, generate Schema Feeds directly from your content database. 2. **Organize by type:** Consider creating separate feeds for different content types (products, articles, events, etc.). 3. **Keep feeds updated:** Regenerate feeds when content changes. The `lastmod` element helps consumers know when to re-fetch. 4. **Use compression:** Large feeds SHOULD be gzip-compressed to reduce bandwidth. 5. **Consider incremental feeds:** For frequently-updated sites, consider providing both a complete feed and a "recent changes" feed. --- ## For Consumers 1. **Respect robots.txt:** Honor any crawl restrictions in robots.txt. 2. **Use conditional requests:** Use `If-Modified-Since` headers to avoid re-downloading unchanged feeds. 3. **Handle large files gracefully:** Feeds may contain millions of items. Stream-process JSONL files rather than loading entirely into memory. 4. **Validate content types:** Verify that feed content matches the declared `contentType`. --- ## Relationship to Existing Standards | Standard | Relationship | |----------|--------------| | schema.org | Schema Feeds uses schema.org vocabulary for structured data | | Sitemaps | Schema Maps follow the sitemap XML format | | robots.txt | Discovery uses a new directive in robots.txt | | JSON-LD | The primary structured data format | | RSS | Supported as an alternative feed format | --- ## Future Considerations - **Delta feeds:** A mechanism for publishing only changes since a given timestamp. - **Webhooks:** Push-based notification when feeds are updated. - **Additional content types:** Support for CSV, N-Triples, or other formats. - **Feed signing:** Cryptographic signatures to verify feed authenticity. --- ## File: app/docs/schema/feed-formats/page.mdx # Schema Feed Formats ## JSON Lines (structuredData/schema.org) ### Format Files with content type `structuredData/schema.org` MUST be formatted as JSON Lines (JSONL): one JSON object per line, with lines separated by newline characters (`\n`). Each line MUST contain a valid JSON-LD object using schema.org vocabulary. ### Requirements - Each JSON object MUST include an `@context` property set to `"https://schema.org"` or include the context implicitly. - Each JSON object MUST include an `@type` property specifying the schema.org type. - Each JSON object SHOULD include an `@id` or `url` property to identify the canonical source. - Files MUST be encoded as UTF-8. - Files SHOULD be compressed using gzip and served with the `.jsonl.gz` extension. ### Example ```json {"@context":"https://schema.org","@type":"Product","@id":"https://example.com/products/123","name":"Widget Pro","description":"A professional-grade widget","price":49.99,"priceCurrency":"USD"} {"@context":"https://schema.org","@type":"Product","@id":"https://example.com/products/124","name":"Widget Basic","description":"An entry-level widget","price":19.99,"priceCurrency":"USD"} {"@context":"https://schema.org","@type":"Product","@id":"https://example.com/products/125","name":"Widget Enterprise","description":"Enterprise widget solution","price":199.99,"priceCurrency":"USD"} ``` --- ## RSS (structuredData/rss) Files with content type `structuredData/rss` MUST be valid RSS 2.0 feeds as defined by the RSS 2.0 specification. RSS feeds are particularly appropriate for time-ordered content such as articles, blog posts, and news items. --- ## File: app/docs/schema/discovery/page.mdx # Discovery ## Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. | Term | Definition | |------|------------| | **Schema Feed** | A file containing aggregated structured data from a website. | | **Schema Map** | An index file that lists the locations of Schema Feed files, analogous to a sitemap index. | | **Consumer** | Any agent that retrieves and processes Schema Feeds, such as a search engine, AI agent, or data aggregator. | --- ## Discovery via robots.txt ### The schemamap Directive Websites advertise their Schema Maps by adding one or more `schemamap` directives to their robots.txt file. This follows the established pattern used by the `sitemap` directive. **Syntax:** ``` schemamap: ``` Where `` is the absolute URL of a Schema Map file. ### Example ``` User-agent: * Disallow: /private/ Sitemap: https://example.com/sitemap.xml schemamap: https://example.com/schemamap.xml ``` ### Requirements - The `schemamap` directive is case-insensitive. - A robots.txt file MAY contain multiple `schemamap` directives, each pointing to a different Schema Map. - The URL MUST be an absolute URL. - The URL SHOULD use HTTPS. - The Schema Map file SHOULD be accessible without authentication. --- ## File: app/docs/schema/appendix/page.mdx # Appendix ## Complete Example ### robots.txt ``` User-agent: * Disallow: /admin/ Sitemap: https://shop.example.com/sitemap.xml schemamap: https://shop.example.com/schemamap.xml ``` ### schemamap.xml ```xml https://shop.example.com/feeds/products.jsonl.gz 2026-01-15T08:00:00Z structuredData/schema.org https://shop.example.com/feeds/reviews.jsonl.gz 2026-01-15T06:00:00Z structuredData/schema.org https://shop.example.com/feeds/blog.rss 2026-01-14T12:00:00Z structuredData/rss ``` ### products.jsonl (excerpt, uncompressed) ```json {"@context":"https://schema.org","@type":"Product","@id":"https://shop.example.com/p/SKU001","name":"Ergonomic Keyboard","description":"Split mechanical keyboard with Cherry MX switches","brand":{"@type":"Brand","name":"TypeWell"},"offers":{"@type":"Offer","price":149.00,"priceCurrency":"USD","availability":"https://schema.org/InStock"}} {"@context":"https://schema.org","@type":"Product","@id":"https://shop.example.com/p/SKU002","name":"Wireless Mouse","description":"Precision wireless mouse with 30-day battery","brand":{"@type":"Brand","name":"ClickPro"},"offers":{"@type":"Offer","price":79.00,"priceCurrency":"USD","availability":"https://schema.org/InStock"}} ``` --- ## MIME Types | Content Type | Recommended MIME Type | File Extension | |--------------|----------------------|----------------| | structuredData/schema.org | application/x-jsonlines | .jsonl | | structuredData/schema.org (compressed) | application/gzip | .jsonl.gz | | structuredData/rss | application/rss+xml | .rss | --- ## References - [schema.org](https://schema.org) - [Sitemaps Protocol](https://www.sitemaps.org/protocol.html) - [robots.txt Specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) - [JSON-LD 1.1](https://www.w3.org/TR/json-ld11/) - [JSON Lines](https://jsonlines.org/) - [RSS 2.0 Specification](https://www.rssboard.org/rss-specification)