## 1. Project Overview & Quickstart (nlweb-ai/LICENSE) # LICENSE Open-source repository nlweb-ai/LICENSE ### Repository Details - **Repository:** [nlweb-ai/LICENSE](https://github.com/nlweb-ai/LICENSE) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (nlweb-ai/website) ## File: README.md # NLWeb.ai Website The official documentation website for NLWeb - a protocol that simplifies building conversational interfaces for websites and enables natural language interactions with AI agents. ## Overview This is a Next.js-based documentation website that provides comprehensive information about: - **NLWeb Protocol**: A standardized interface for natural language interactions with agents - **Agent Finder**: Software that helps agents discover and integrate with NLWeb-compatible websites - **Implementation Guides**: Practical examples and quickstart guides ## Tech Stack - **Framework**: Next.js 16.1.1 with React 19 - **Documentation**: MDX (Markdown + JSX) using Nextra - **Styling**: Tailwind CSS 4 - **UI Components**: Radix UI primitives - **Syntax Highlighting**: react-syntax-highlighter ## Getting Started ### Prerequisites - Node.js (v20 or higher recommended) - pnpm ### Installation ```bash pnpm install ``` ### Development Run the development server: ```bash pnpm dev ``` Open [http://localhost:3000](http://localhost:3000) to view the website. ### Build Build the production-ready site: ```bash pnpm build ``` ### Start Production Server ```bash pnpm start ``` ## Project Structure ``` nlweb-internal-website/ ├── app/ # Next.js app directory │ ├── docs/ # Documentation pages │ │ ├── intro/ # Introduction pages │ │ ├── about/ # About and FAQ │ │ ├── nlweb-core/ # NLWeb Core software docs │ │ └── agent-finder/ # Agent Finder docs │ │ ├── overview/ # Agent Finder overview │ │ ├── motivation/ # Why Agent Finder exists │ │ ├── quickstart/ # Getting started guide │ │ └── api/ # API documentation │ └── specification/ # Protocol specification │ ├── protocol/ # Core protocol docs │ │ ├── overview/ # Protocol overview │ │ ├── ask/ # Ask query structure │ │ ├── response/ # Response structure │ │ ├── actions/ # Actions specification │ │ └── binding/ # HTTP transport binding │ └── appendix/ # Additional examples │ ├── http/ # HTTP payload examples │ └── mcp/ # MCP payload examples ├── components/ # Reusable React components ├── NLWEBSPEC.md # Source specification document ├── AGENTFINDER.md # Agent Finder design document └── CLAUDE.md # Project instructions for Claude AI ``` ## Documentation ### How Documentation Works This website uses MDX files for documentation. MDX allows you to use JSX components directly in Markdown, making it powerful for creating interactive documentation. ### Route Structure Routes are automatically generated from the folder structure in the `app/` directory. The file system path directly maps to the URL: - `app/docs/intro/page.mdx` → `/docs/intro` - `app/specification/protocol/overview/page.mdx` → `/specification/protocol/overview` - `app/docs/agent-finder/motivation/page.mdx` → `/docs/agent-finder/motivation` This means adding new pages is as simple as creating a new folder with a `page.mdx` file inside. ### Modifying Existing Documentation #### 1. Find the MDX File Documentation pages are located in the `app/` directory. The file structure mirrors the URL structure: - `/docs/intro` → `app/docs/intro/page.mdx` - `/specification/protocol/overview` → `app/specification/protocol/overview/page.mdx` - `/docs/agent-finder/motivation` → `app/docs/agent-finder/motivation/page.mdx` #### 2. Edit the MDX File Open the relevant `page.mdx` file and edit it. MDX supports: **Standard Markdown:** ```mdx # Heading 1 ## Heading 2 ### Heading 3 Regular text with **bold** and *italic*. - Bullet points - More points 1. Numbered lists 2. Continue... ``` **Code Blocks:** ````mdx ```json { "query": { "text": "example" } } ``` ```` **React Components:** ```mdx ``` #### 3. Test Your Changes Run the development server to see your changes live: ```bash pnpm dev ``` ### Adding New Documentation Pages #### 1. Create the MDX File Create a new `page.mdx` file in the appropriate directory. Follow the existing structure: ``` app/ └── docs/ └── your-section/ └── your-page/ └── page.mdx ``` #### 2. Add Content Start with a heading and write your content: ```mdx # Your Page Title Your content goes here. Use Markdown syntax and JSX components as needed. ## Section 1 More content... ## Section 2 Even more content... ``` #### 3. Verify Your Page - Navigate to the URL that matches your folder path (e.g., `/docs/your-section/your-page`) - Test internal links to and from your page - Verify the page renders correctly with all formatting ### Documentation Style Guidelines When writing or modifying documentation, follow these conventions (based on existing pages): 1. **Use Clear Headings**: Start with an H1 (`#`) for the page title, use H2 (`##`) for main sections 2. **Code Examples**: Include practical code examples in appropriate language blocks 3. **Consistent Structure**: Look at `app/specification/protocol/overview/page.mdx` as a reference 4. **Schema.org References**: When applicable, reference Schema.org vocabulary 5. **Links**: Use relative links for internal pages, absolute for external ### Source Documents - **NLWEBSPEC.md**: The authoritative NLWeb protocol specification - **AGENTFINDER.md**: Design philosophy and specification for Agent Finder - **CLAUDE.md**: Instructions for AI-assisted development (project guidelines) When updating protocol documentation, consult `NLWEBSPEC.md` as the source of truth. ### Common Documentation Patterns #### Protocol Specification Pages Follow the pattern in existing spec pages: - Overview section explaining the concept - Detailed attributes/structure definitions - Code examples with syntax highlighting - Cross-references to related pages #### Software Documentation For software docs (NLWeb Core, Agent Finder): - Overview/motivation section - Quickstart guide with installation steps - API reference with examples - Use cases and best practices ## Components Reusable components are located in the `components/` directory. The site primarily uses: - Radix UI primitives for interactive elements - Custom components for documentation-specific needs - React Syntax Highlighter for code blocks When creating new pages, reuse existing components rather than creating new ones. ## Deployment This site can be deployed to any platform that supports Next.js: - **Vercel**: Automatic deployment with git integration - **Netlify**: Similar to Vercel - **Self-hosted**: Use `pnpm build && pnpm start` ## Contributing When contributing documentation: 1. Read the source specification documents (`NLWEBSPEC.md`, `AGENTFINDER.md`) 2. Follow existing page structure and styling conventions 3. Create new folders and `page.mdx` files for new pages 4. Test locally before submitting changes 5. Ensure all internal links work correctly ## License Please refer to the repository license file for licensing information. ## Resources - **NLWeb Protocol Spec**: Documented in this site at `/specification/protocol/overview` - **GitHub Repository**: [nlweb-ai organization](https://github.com/nlweb-ai) - **NLWeb Core**: Implementation software for websites - **Agent Finder**: Integration software for AI agents ## Support For questions or issues: - File an issue in the GitHub repository - Check the FAQ at `/docs/about/faq` - Review the specification documentation --- ## File: app/docs/intro/page.mdx # What is NLWeb? NLWeb is an open protocol for building conversational interfaces to websites and applications. It provides a standardized way for both humans and AI agents to interact with any site using natural language. The protocol builds on widely adopted, standard formats — Schema.org vocabularies used by over 100 million websites, JSON-LD for extensibility, and Server-Sent Events for streaming. NLWeb natively supports agentic protocols like MCP and A2A, and every NLWeb endpoint is also natively a ChatGPT app, making it instantly accessible to hundreds of millions of users. NLWeb has two primary components: 1. A [protocol specification](/docs/specification) that defines how to interact with any application using natural language — including query structure, response formats, streaming, and support for long-running tasks. 2. A [reference implementation](https://github.com/nlweb-ai/NLWeb) that makes it easy to add a conversational interface to any site with structured content (products, recipes, articles, reviews, and more). ## Implementations Beyond the reference implementation, NLWeb is being adopted by a growing number of platforms and companies, including [Cloudflare](https://www.cloudflare.com), [Tollbit](https://www.tollbit.com), [Wix](https://www.wix.com), and others — making it easy for millions of sites to become conversational. To see NLWeb in action, try [Microsoft News](https://news.microsoft.com/source) — a live example of the protocol powering a conversational interface on a major website. ## Get Involved NLWeb is fully open source. We invite the community to build on the protocol, create new implementations, and help shape the conversational web. **[Read the specification](/docs/specification)** or **[explore the code](https://github.com/nlweb-ai/NLWeb)**. --- ## File: app/docs/about/faq/page.mdx # Frequently asked questions ## Quick Links - [NLWeb vs MCP vs A2A?](#0) - [What's the difference between 'ask' and 'await'?](#1) - [How does NLWeb handle context across multiple queries?](#2) - [What response types can an NLWeb agent return?](#3) - [What's the deal with this site design?](#4) ---
### Q: NLWeb vs MCP vs A2A? The major difference is that NLWeb is designed for text-in, text-out natural language interactions. You don't need to understand how to call specific tools, define parameters, or handle complex tool schemas. Just ask in natural language and get structured responses back. In contrast, MCP and A2A are protocols where you need to understand available tools, their schemas, and how to invoke them correctly. NLWeb abstracts away this complexity by providing a unified 'ask' interface. NLWeb is also transport-protocol agnostic and can work over HTTP, WebSockets, JSON-RPC, or within agentic protocols like MCP and A2A, making it complementary to these protocols rather than competing with them.
---
### Q: What's the difference between 'ask' and 'await'? 'ask' is the primary API for querying an NLWeb agent with a natural language request. It can return immediate answers, elicitations (requests for more information), promises (for long-running tasks), or failures. 'await' is a helper API used to check the status of or cancel a long-running task that returned a promise. You use 'ask' to make your initial request, and if you receive a promise token back, you use 'await' with that token to check on the task's progress or get the final result.
---
### Q: How does NLWeb handle context across multiple queries? NLWeb treats context as a first-class object in both requests and responses. The request can include a 'context' section with conversation history (prev queries), free-form contextual text, and persistent user preferences or memory. The response includes a 'session_context' in the meta section that should be included in subsequent requests, similar to HTTP cookies. This enables agents to maintain conversational state, understand references to previous queries, and personalize responses based on accumulated user information.
---
### Q: What response types can an NLWeb agent return? An NLWeb agent can return four types of responses: 1. **Answer** - provides the requested information or confirms task completion with structured results 2. **Elicitation** - requests additional information from the user when the query is ambiguous or incomplete 3. **Promise** - returns a token for long-running operations that can be checked later using the 'await' API 4. **Failure** - indicates an error occurred with an error code and message The response type is specified in the '_meta.response_type' field.
---
### Q: What's the deal with this site design? This CSS has a lineage. It was originally created for the [Sitemaps protocol](https://www.sitemaps.org/) site, then adopted by [Schema.org](https://schema.org/), and now lives on at [NLWeb.ai](https://nlweb.ai). Three generations of web standards, one stylesheet. Thanks [Shiva Shivakumar](https://en.wikipedia.org/wiki/Narayanan_Shivakumar).
--- ## File: app/docs/schema/security/page.mdx # Security Considerations ## Content Authenticity The structured data provided in Schema Feeds MUST be consistent with the schema.org markup present on the corresponding HTML pages. Publishers SHOULD NOT include data in feeds that differs from what appears on their pages. Some consumers may not treat Schema Feed data with the same level of trust as markup extracted directly from HTML pages. Publishers should be aware that providing accurate, consistent data across both feeds and pages is essential for broad adoption. ## Data Validation Consumers SHOULD validate that the structured data in feeds conforms to schema.org definitions and SHOULD handle malformed data gracefully. --- ## File: app/docs/schema/schema-map/page.mdx # Schema Map Format ## Overview A Schema Map is an XML file that follows the sitemap protocol format. Each entry in the Schema Map points to a Schema Feed file and specifies its content type. ## XML Schema ```xml https://example.com/feeds/products.jsonl 2026-01-15 structuredData/schema.org https://example.com/feeds/articles.rss 2026-01-14 structuredData/rss ``` ## Elements ### `` The root element. MUST include the sitemaps namespace and SHOULD include the Schema Feeds namespace. ### `` A container for information about a single Schema Feed file. ### `` REQUIRED. The URL of the Schema Feed file. MUST be an absolute URL. ### `` OPTIONAL. The date the Schema Feed was last modified, in W3C Datetime format. Consumers MAY use this to avoid re-fetching unchanged feeds. ### `` REQUIRED. Specifies the format of the Schema Feed file. Defined values: | Value | Description | |-------|-------------| | `structuredData/schema.org` | JSON Lines file containing schema.org JSON-LD objects | | `structuredData/rss` | RSS 2.0 feed | Additional content types MAY be defined in future versions of this specification. ## Schema Map Index For large sites, Schema Maps MAY be organized using an index file, following the sitemap index pattern: ```xml https://example.com/schemamap-products.xml 2026-01-15 https://example.com/schemamap-articles.xml 2026-01-14 ``` --- ## File: app/docs/schema/introduction/page.mdx # Introduction **Version:** 0.1 (Draft) **Date:** January 2026 **Status:** Proposal --- ## Abstract This specification defines Schema Feeds, a mechanism for websites to provide aggregated structured data in schema.org format through a small number of files, rather than requiring consumers to crawl individual pages. Schema Feeds builds upon the familiar patterns of sitemaps and robots.txt to enable efficient discovery and retrieval of a site's complete structured data. --- ## Background Approximately fifteen years ago, the major search engines—Google, Bing, and Yahoo—recognized a fundamental opportunity: many websites are powered by structured databases, yet this structure is lost when content is rendered as HTML. If search engines could access the underlying structured data, they could provide richer, more accurate search results. However, different websites use different internal schemas. Reconciling these diverse schemas at web scale would be impractical. To address this, the search engines collaborated to create schema.org—a shared vocabulary of schemas that websites could use to expose their structured data in a common format. Today, tens of millions of websites publish schema.org markup, making it one of the most successful standards on the web. ## Problem Statement Despite the success of schema.org, a significant inefficiency remains: structured data is embedded within individual HTML pages. To collect all the structured data from a website, a consumer must crawl every page on the site, parse the HTML, and extract the embedded markup. This approach has several drawbacks: - **Inefficiency:** Crawling millions of pages to extract structured data is computationally expensive for both the crawler and the website. - **Latency:** Changes to structured data are only discovered when pages are re-crawled, which may take days or weeks. - **Incompleteness:** Crawlers may miss pages, resulting in incomplete data collection. - **Overhead:** The structured data is a small fraction of the page content, yet the entire page must be retrieved. ## Solution Overview Schema Feeds allows a website to publish all of its structured data in a small number of aggregated files. Consumers can retrieve these files directly, without crawling individual pages. This approach is analogous to how sitemaps allow search engines to discover URLs without crawling, and how RSS feeds allow aggregators to discover content without polling individual pages. --- ## File: app/docs/schema/implementation/page.mdx # Implementation Guidelines ## For Publishers 1. **Generate feeds from your database:** Rather than extracting markup from rendered pages, generate Schema Feeds directly from your content database. 2. **Organize by type:** Consider creating separate feeds for different content types (products, articles, events, etc.). 3. **Keep feeds updated:** Regenerate feeds when content changes. The `lastmod` element helps consumers know when to re-fetch. 4. **Use compression:** Large feeds SHOULD be gzip-compressed to reduce bandwidth. 5. **Consider incremental feeds:** For frequently-updated sites, consider providing both a complete feed and a "recent changes" feed. --- ## For Consumers 1. **Respect robots.txt:** Honor any crawl restrictions in robots.txt. 2. **Use conditional requests:** Use `If-Modified-Since` headers to avoid re-downloading unchanged feeds. 3. **Handle large files gracefully:** Feeds may contain millions of items. Stream-process JSONL files rather than loading entirely into memory. 4. **Validate content types:** Verify that feed content matches the declared `contentType`. --- ## Relationship to Existing Standards | Standard | Relationship | |----------|--------------| | schema.org | Schema Feeds uses schema.org vocabulary for structured data | | Sitemaps | Schema Maps follow the sitemap XML format | | robots.txt | Discovery uses a new directive in robots.txt | | JSON-LD | The primary structured data format | | RSS | Supported as an alternative feed format | --- ## Future Considerations - **Delta feeds:** A mechanism for publishing only changes since a given timestamp. - **Webhooks:** Push-based notification when feeds are updated. - **Additional content types:** Support for CSV, N-Triples, or other formats. - **Feed signing:** Cryptographic signatures to verify feed authenticity. --- ## File: app/docs/schema/feed-formats/page.mdx # Schema Feed Formats ## JSON Lines (structuredData/schema.org) ### Format Files with content type `structuredData/schema.org` MUST be formatted as JSON Lines (JSONL): one JSON object per line, with lines separated by newline characters (`\n`). Each line MUST contain a valid JSON-LD object using schema.org vocabulary. ### Requirements - Each JSON object MUST include an `@context` property set to `"https://schema.org"` or include the context implicitly. - Each JSON object MUST include an `@type` property specifying the schema.org type. - Each JSON object SHOULD include an `@id` or `url` property to identify the canonical source. - Files MUST be encoded as UTF-8. - Files SHOULD be compressed using gzip and served with the `.jsonl.gz` extension. ### Example ```json {"@context":"https://schema.org","@type":"Product","@id":"https://example.com/products/123","name":"Widget Pro","description":"A professional-grade widget","price":49.99,"priceCurrency":"USD"} {"@context":"https://schema.org","@type":"Product","@id":"https://example.com/products/124","name":"Widget Basic","description":"An entry-level widget","price":19.99,"priceCurrency":"USD"} {"@context":"https://schema.org","@type":"Product","@id":"https://example.com/products/125","name":"Widget Enterprise","description":"Enterprise widget solution","price":199.99,"priceCurrency":"USD"} ``` --- ## RSS (structuredData/rss) Files with content type `structuredData/rss` MUST be valid RSS 2.0 feeds as defined by the RSS 2.0 specification. RSS feeds are particularly appropriate for time-ordered content such as articles, blog posts, and news items. --- ## File: app/docs/schema/discovery/page.mdx # Discovery ## Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. | Term | Definition | |------|------------| | **Schema Feed** | A file containing aggregated structured data from a website. | | **Schema Map** | An index file that lists the locations of Schema Feed files, analogous to a sitemap index. | | **Consumer** | Any agent that retrieves and processes Schema Feeds, such as a search engine, AI agent, or data aggregator. | --- ## Discovery via robots.txt ### The schemamap Directive Websites advertise their Schema Maps by adding one or more `schemamap` directives to their robots.txt file. This follows the established pattern used by the `sitemap` directive. **Syntax:** ``` schemamap: ``` Where `` is the absolute URL of a Schema Map file. ### Example ``` User-agent: * Disallow: /private/ Sitemap: https://example.com/sitemap.xml schemamap: https://example.com/schemamap.xml ``` ### Requirements - The `schemamap` directive is case-insensitive. - A robots.txt file MAY contain multiple `schemamap` directives, each pointing to a different Schema Map. - The URL MUST be an absolute URL. - The URL SHOULD use HTTPS. - The Schema Map file SHOULD be accessible without authentication. --- ## File: app/docs/schema/appendix/page.mdx # Appendix ## Complete Example ### robots.txt ``` User-agent: * Disallow: /admin/ Sitemap: https://shop.example.com/sitemap.xml schemamap: https://shop.example.com/schemamap.xml ``` ### schemamap.xml ```xml https://shop.example.com/feeds/products.jsonl.gz 2026-01-15T08:00:00Z structuredData/schema.org https://shop.example.com/feeds/reviews.jsonl.gz 2026-01-15T06:00:00Z structuredData/schema.org https://shop.example.com/feeds/blog.rss 2026-01-14T12:00:00Z structuredData/rss ``` ### products.jsonl (excerpt, uncompressed) ```json {"@context":"https://schema.org","@type":"Product","@id":"https://shop.example.com/p/SKU001","name":"Ergonomic Keyboard","description":"Split mechanical keyboard with Cherry MX switches","brand":{"@type":"Brand","name":"TypeWell"},"offers":{"@type":"Offer","price":149.00,"priceCurrency":"USD","availability":"https://schema.org/InStock"}} {"@context":"https://schema.org","@type":"Product","@id":"https://shop.example.com/p/SKU002","name":"Wireless Mouse","description":"Precision wireless mouse with 30-day battery","brand":{"@type":"Brand","name":"ClickPro"},"offers":{"@type":"Offer","price":79.00,"priceCurrency":"USD","availability":"https://schema.org/InStock"}} ``` --- ## MIME Types | Content Type | Recommended MIME Type | File Extension | |--------------|----------------------|----------------| | structuredData/schema.org | application/x-jsonlines | .jsonl | | structuredData/schema.org (compressed) | application/gzip | .jsonl.gz | | structuredData/rss | application/rss+xml | .rss | --- ## References - [schema.org](https://schema.org) - [Sitemaps Protocol](https://www.sitemaps.org/protocol.html) - [robots.txt Specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) - [JSON-LD 1.1](https://www.w3.org/TR/json-ld11/) - [JSON Lines](https://jsonlines.org/) - [RSS 2.0 Specification](https://www.rssboard.org/rss-specification)