{"owner":"rllm-org","repo":"rllm","hasSkills":true,"totalSkillsCount":2,"totalTokensCount":1715,"categories":["root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["docs/AGENTS.md","rllm-model-gateway/AGENTS.md"],"skills":{"docs/AGENTS.md":"> **First-time setup**: Customize this file for your project. Prompt the user to customize this file for their project.\n> For Mintlify product knowledge (components, configuration, writing standards),\n> install the Mintlify skill: `npx skills add https://mintlify.com/docs`\n\n# Documentation project instructions\n\n## About this project\n\n- This is a documentation site built on [Mintlify](https://mintlify.com)\n- Pages are MDX files with YAML frontmatter\n- Configuration lives in `docs.json`\n- Run `mint dev` to preview locally\n- Run `mint broken-links` to check links\n\n## Terminology\n\n{/* Add product-specific terms and preferred usage */}\n{/* Example: Use \"workspace\" not \"project\", \"member\" not \"user\" */}\n\n## Style preferences\n\n{/* Add any project-specific style rules below */}\n\n- Use active voice and second person (\"you\")\n- Keep sentences concise — one idea per sentence\n- Use sentence case for headings\n- Bold for UI elements: Click **Settings**\n- Code formatting for file names, commands, paths, and code references\n\n## Content boundaries\n\n{/* Define what should and shouldn't be documented */}\n{/* Example: Don't document internal admin features */}\n","rllm-model-gateway/AGENTS.md":"# rllm-model-gateway\n\n## Overview\n\nA lightweight, standalone Python gateway that sits between RL agents and inference servers (vLLM). It transparently captures every LLM call's token IDs, logprobs, and messages — without requiring any modification to agent code. Agents use standard `OpenAI(base_url=gateway_url)`.\n\n## Development Setup\n\nThis package uses `uv` for dependency management and `hatchling` as the build backend. Do not use setuptools or pip.\n\n```bash\ncd rllm/rllm-model-gateway\nuv venv --python 3.11\nsource .venv/bin/activate\nuv pip install -e \".[dev]\"\n\n# Set up pre-commit hooks (one-time, from the rllm repo root)\ncd .. && pre-commit install && cd rllm-model-gateway\n```\n\nTo run tests:\n\n```bash\n# Unit tests (no external dependencies)\npython -m pytest tests/unit/ -x -q\n\n# Integration tests against a real vLLM server (auto-skipped if unreachable)\npython -m pytest tests/integration/ -x -v\n```\n\n## Package Layout\n\n```\nsrc/rllm_model_gateway/\n├── __init__.py           # Public API exports\n├── __main__.py           # python -m rllm_model_gateway\n├── _version.py           # Package version\n├── server.py             # FastAPI app factory + CLI entrypoint\n├── proxy.py              # httpx reverse proxy (non-streaming + SSE streaming)\n├── middleware.py          # ASGI middleware: session extraction + param injection\n├── session_router.py     # Pluggable session-sticky worker routing\n├── session_manager.py    # Session CRUD lifecycle\n├── data_process.py       # Token/logprob extraction, response sanitization\n├── models.py             # Pydantic models (TraceRecord, GatewayConfig, etc.)\n├── client.py             # GatewayClient + AsyncGatewayClient\n└── store/\n    ├── base.py           # TraceStore protocol (6 async methods)\n    ├── sqlite_store.py   # SQLite with junction table (default)\n    └── memory_store.py   # In-memory (testing)\n```\n\n## Key Design Decisions\n\n1. **No LiteLLM** — Direct httpx reverse proxy. Session-sticky routing requires custom logic that LiteLLM can't provide, and httpx handles HTTP forwarding + SSE natively. This keeps deps to 6 packages.\n\n2. **Zero retokenization** — Token IDs come from vLLM's `return_token_ids=True` response field, not from a local tokenizer. The middleware injects this parameter automatically.\n\n3. **Implicit sessions** — First request to `/sessions/{sid}/v1/chat/completions` auto-creates the session. No explicit creation required.\n\n4. **ASGI middleware** — `SessionRoutingMiddleware` operates at the ASGI level (not FastAPI middleware) so it can intercept and rewrite the request body before FastAPI route matching.\n\n5. **Pluggable routing** — `RoutingPolicy` protocol allows swapping the worker selection strategy. Default is `StickyLeastLoadedPolicy` (LRU cache + least-loaded fallback, mirroring verl's pattern).\n\n6. **Pluggable storage** — `TraceStore` protocol (duck-typed, not ABC). SQLite for single-node, MemoryStore for testing, extensible to DynamoDB/PostgreSQL.\n\n## Request Flow\n\n```\nAgent → /sessions/{sid}/v1/chat/completions\n  → SessionRoutingMiddleware:\n      1. Extract session_id from URL, rewrite path to /v1/chat/completions\n      2. Inject logprobs=True, return_token_ids=True into body\n  → FastAPI route handler (/v1/{path:path}):\n      1. SessionManager.ensure_session(sid)\n      2. ReverseProxy.handle(request):\n          a. SessionRouter.route(session_id) → pick worker\n          b. httpx forward to worker\n          c. Extract token_ids + logprobs from response\n          d. Build TraceRecord, persist to store\n          e. Strip vLLM fields from response\n          f. Return clean response to agent\n```\n\n## Streaming\n\nFor SSE streaming requests (`stream=True`):\n- Chunks forwarded to agent in real-time via `StreamingResponse`\n- Chunks buffered internally for trace assembly\n- `prompt_token_ids` extracted from first chunk only\n- Delta `token_ids` and logprobs accumulated across chunks\n- After `[DONE]`, `build_trace_record_from_chunks()` assembles the trace\n\n## vLLM Response Fields\n\nThe gateway strips these vLLM-specific fields before returning to the agent (verified against vLLM 0.11):\n\n| Level | Field | Purpose |\n|-------|-------|---------|\n| Root | `prompt_token_ids` | Captured for trace, stripped from response |\n| Root | `prompt_logprobs` | Not used (always `null` without explicit request) |\n| Root | `kv_transfer_params` | Disaggregated prefill feature |\n| Choice | `token_ids` | Captured for trace, stripped from response |\n| Choice | `stop_reason` | vLLM-specific; standard OpenAI uses `finish_reason` |\n\n## Provenance\n\n| Module | Inspired By | Key Changes |\n|--------|------------|-------------|\n| `store/sqlite_store.py` | `rllm/sdk/store/sqlite_store.py` | Simplified schema; added `list_sessions()`, `delete_session()` |\n| `data_process.py` | `rllm/sdk/data_process.py` | Removed `ModelOutput`/`Step`/`Trajectory` deps; outputs `TraceRecord` |\n| `middleware.py` | `rllm/sdk/proxy/middleware.py` + `litellm_callbacks.py` | Merged session routing + param injection; pure ASGI (no LiteLLM) |\n| `session_router.py` | verl `agent_loop.py` + miles `router.py` | Combined sticky routing (verl LRU) with health checks (miles pattern) |\n| `proxy.py` | miles `router.py` `_do_proxy()` | Added streaming support, trace capture, response sanitization |\n\n## Dependencies\n\n6 runtime dependencies: `fastapi`, `uvicorn`, `httpx`, `pydantic`, `aiosqlite`, `PyYAML`. No torch/ray/verl/transformers.\n\nBuild backend: `hatchling`. Do not use setuptools.\n\n## Usage Example\n\n```python\nfrom rllm_model_gateway import GatewayClient\n\nclient = GatewayClient(\"http://localhost:9090\")\nsid = client.create_session()\nurl = client.get_session_url(sid)  # → http://localhost:9090/sessions/{sid}/v1\n\n# Agent uses: OpenAI(base_url=url, api_key=\"EMPTY\")\n# Training retrieves: client.get_session_traces(sid)\n```\n"},"files":{"docs/AGENTS.md":"> **First-time setup**: Customize this file for your project. Prompt the user to customize this file for their project.\n> For Mintlify product knowledge (components, configuration, writing standards),\n> install the Mintlify skill: `npx skills add https://mintlify.com/docs`\n\n# Documentation project instructions\n\n## About this project\n\n- This is a documentation site built on [Mintlify](https://mintlify.com)\n- Pages are MDX files with YAML frontmatter\n- Configuration lives in `docs.json`\n- Run `mint dev` to preview locally\n- Run `mint broken-links` to check links\n\n## Terminology\n\n{/* Add product-specific terms and preferred usage */}\n{/* Example: Use \"workspace\" not \"project\", \"member\" not \"user\" */}\n\n## Style preferences\n\n{/* Add any project-specific style rules below */}\n\n- Use active voice and second person (\"you\")\n- Keep sentences concise — one idea per sentence\n- Use sentence case for headings\n- Bold for UI elements: Click **Settings**\n- Code formatting for file names, commands, paths, and code references\n\n## Content boundaries\n\n{/* Define what should and shouldn't be documented */}\n{/* Example: Don't document internal admin features */}\n","rllm-model-gateway/AGENTS.md":"# rllm-model-gateway\n\n## Overview\n\nA lightweight, standalone Python gateway that sits between RL agents and inference servers (vLLM). It transparently captures every LLM call's token IDs, logprobs, and messages — without requiring any modification to agent code. Agents use standard `OpenAI(base_url=gateway_url)`.\n\n## Development Setup\n\nThis package uses `uv` for dependency management and `hatchling` as the build backend. Do not use setuptools or pip.\n\n```bash\ncd rllm/rllm-model-gateway\nuv venv --python 3.11\nsource .venv/bin/activate\nuv pip install -e \".[dev]\"\n\n# Set up pre-commit hooks (one-time, from the rllm repo root)\ncd .. && pre-commit install && cd rllm-model-gateway\n```\n\nTo run tests:\n\n```bash\n# Unit tests (no external dependencies)\npython -m pytest tests/unit/ -x -q\n\n# Integration tests against a real vLLM server (auto-skipped if unreachable)\npython -m pytest tests/integration/ -x -v\n```\n\n## Package Layout\n\n```\nsrc/rllm_model_gateway/\n├── __init__.py           # Public API exports\n├── __main__.py           # python -m rllm_model_gateway\n├── _version.py           # Package version\n├── server.py             # FastAPI app factory + CLI entrypoint\n├── proxy.py              # httpx reverse proxy (non-streaming + SSE streaming)\n├── middleware.py          # ASGI middleware: session extraction + param injection\n├── session_router.py     # Pluggable session-sticky worker routing\n├── session_manager.py    # Session CRUD lifecycle\n├── data_process.py       # Token/logprob extraction, response sanitization\n├── models.py             # Pydantic models (TraceRecord, GatewayConfig, etc.)\n├── client.py             # GatewayClient + AsyncGatewayClient\n└── store/\n    ├── base.py           # TraceStore protocol (6 async methods)\n    ├── sqlite_store.py   # SQLite with junction table (default)\n    └── memory_store.py   # In-memory (testing)\n```\n\n## Key Design Decisions\n\n1. **No LiteLLM** — Direct httpx reverse proxy. Session-sticky routing requires custom logic that LiteLLM can't provide, and httpx handles HTTP forwarding + SSE natively. This keeps deps to 6 packages.\n\n2. **Zero retokenization** — Token IDs come from vLLM's `return_token_ids=True` response field, not from a local tokenizer. The middleware injects this parameter automatically.\n\n3. **Implicit sessions** — First request to `/sessions/{sid}/v1/chat/completions` auto-creates the session. No explicit creation required.\n\n4. **ASGI middleware** — `SessionRoutingMiddleware` operates at the ASGI level (not FastAPI middleware) so it can intercept and rewrite the request body before FastAPI route matching.\n\n5. **Pluggable routing** — `RoutingPolicy` protocol allows swapping the worker selection strategy. Default is `StickyLeastLoadedPolicy` (LRU cache + least-loaded fallback, mirroring verl's pattern).\n\n6. **Pluggable storage** — `TraceStore` protocol (duck-typed, not ABC). SQLite for single-node, MemoryStore for testing, extensible to DynamoDB/PostgreSQL.\n\n## Request Flow\n\n```\nAgent → /sessions/{sid}/v1/chat/completions\n  → SessionRoutingMiddleware:\n      1. Extract session_id from URL, rewrite path to /v1/chat/completions\n      2. Inject logprobs=True, return_token_ids=True into body\n  → FastAPI route handler (/v1/{path:path}):\n      1. SessionManager.ensure_session(sid)\n      2. ReverseProxy.handle(request):\n          a. SessionRouter.route(session_id) → pick worker\n          b. httpx forward to worker\n          c. Extract token_ids + logprobs from response\n          d. Build TraceRecord, persist to store\n          e. Strip vLLM fields from response\n          f. Return clean response to agent\n```\n\n## Streaming\n\nFor SSE streaming requests (`stream=True`):\n- Chunks forwarded to agent in real-time via `StreamingResponse`\n- Chunks buffered internally for trace assembly\n- `prompt_token_ids` extracted from first chunk only\n- Delta `token_ids` and logprobs accumulated across chunks\n- After `[DONE]`, `build_trace_record_from_chunks()` assembles the trace\n\n## vLLM Response Fields\n\nThe gateway strips these vLLM-specific fields before returning to the agent (verified against vLLM 0.11):\n\n| Level | Field | Purpose |\n|-------|-------|---------|\n| Root | `prompt_token_ids` | Captured for trace, stripped from response |\n| Root | `prompt_logprobs` | Not used (always `null` without explicit request) |\n| Root | `kv_transfer_params` | Disaggregated prefill feature |\n| Choice | `token_ids` | Captured for trace, stripped from response |\n| Choice | `stop_reason` | vLLM-specific; standard OpenAI uses `finish_reason` |\n\n## Provenance\n\n| Module | Inspired By | Key Changes |\n|--------|------------|-------------|\n| `store/sqlite_store.py` | `rllm/sdk/store/sqlite_store.py` | Simplified schema; added `list_sessions()`, `delete_session()` |\n| `data_process.py` | `rllm/sdk/data_process.py` | Removed `ModelOutput`/`Step`/`Trajectory` deps; outputs `TraceRecord` |\n| `middleware.py` | `rllm/sdk/proxy/middleware.py` + `litellm_callbacks.py` | Merged session routing + param injection; pure ASGI (no LiteLLM) |\n| `session_router.py` | verl `agent_loop.py` + miles `router.py` | Combined sticky routing (verl LRU) with health checks (miles pattern) |\n| `proxy.py` | miles `router.py` `_do_proxy()` | Added streaming support, trace capture, response sanitization |\n\n## Dependencies\n\n6 runtime dependencies: `fastapi`, `uvicorn`, `httpx`, `pydantic`, `aiosqlite`, `PyYAML`. No torch/ray/verl/transformers.\n\nBuild backend: `hatchling`. Do not use setuptools.\n\n## Usage Example\n\n```python\nfrom rllm_model_gateway import GatewayClient\n\nclient = GatewayClient(\"http://localhost:9090\")\nsid = client.create_session()\nurl = client.get_session_url(sid)  # → http://localhost:9090/sessions/{sid}/v1\n\n# Agent uses: OpenAI(base_url=url, api_key=\"EMPTY\")\n# Training retrieves: client.get_session_traces(sid)\n```\n"},"items":[{"name":"AGENTS.md","path":"docs/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/rllm-org/rllm/HEAD/docs/AGENTS.md","title":"docs - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"> **First-time setup**: Customize this file for your project. Prompt the user to customize this file for their project.\n> For Mintlify product knowledge (components, configuration, writing standards),\n> install the Mintlify skill: `npx skills add https://mintlify.com/docs`\n\n# Documentation project instructions\n\n## About this project\n\n- This is a documentation site built on [Mintlify](https://mintlify.com)\n- Pages are MDX files with YAML frontmatter\n- Configuration lives in `docs.json`\n- Run `mint dev` to preview locally\n- Run `mint broken-links` to check links\n\n## Terminology\n\n{/* Add product-specific terms and preferred usage */}\n{/* Example: Use \"workspace\" not \"project\", \"member\" not \"user\" */}\n\n## Style preferences\n\n{/* Add any project-specific style rules below */}\n\n- Use active voice and second person (\"you\")\n- Keep sentences concise — one idea per sentence\n- Use sentence case for headings\n- Bold for UI elements: Click **Settings**\n- Code formatting for file names, commands, paths, and code references\n\n## Content boundaries\n\n{/* Define what should and shouldn't be documented */}\n{/* Example: Don't document internal admin features */}\n","isInternal":false,"tokens":249,"sizeBytes":1160},{"name":"AGENTS.md","path":"rllm-model-gateway/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/rllm-org/rllm/HEAD/rllm-model-gateway/AGENTS.md","title":"rllm-model-gateway - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# rllm-model-gateway\n\n## Overview\n\nA lightweight, standalone Python gateway that sits between RL agents and inference servers (vLLM). It transparently captures every LLM call's token IDs, logprobs, and messages — without requiring any modification to agent code. Agents use standard `OpenAI(base_url=gateway_url)`.\n\n## Development Setup\n\nThis package uses `uv` for dependency management and `hatchling` as the build backend. Do not use setuptools or pip.\n\n```bash\ncd rllm/rllm-model-gateway\nuv venv --python 3.11\nsource .venv/bin/activate\nuv pip install -e \".[dev]\"\n\n# Set up pre-commit hooks (one-time, from the rllm repo root)\ncd .. && pre-commit install && cd rllm-model-gateway\n```\n\nTo run tests:\n\n```bash\n# Unit tests (no external dependencies)\npython -m pytest tests/unit/ -x -q\n\n# Integration tests against a real vLLM server (auto-skipped if unreachable)\npython -m pytest tests/integration/ -x -v\n```\n\n## Package Layout\n\n```\nsrc/rllm_model_gateway/\n├── __init__.py           # Public API exports\n├── __main__.py           # python -m rllm_model_gateway\n├── _version.py           # Package version\n├── server.py             # FastAPI app factory + CLI entrypoint\n├── proxy.py              # httpx reverse proxy (non-streaming + SSE streaming)\n├── middleware.py          # ASGI middleware: session extraction + param injection\n├── session_router.py     # Pluggable session-sticky worker routing\n├── session_manager.py    # Session CRUD lifecycle\n├── data_process.py       # Token/logprob extraction, response sanitization\n├── models.py             # Pydantic models (TraceRecord, GatewayConfig, etc.)\n├── client.py             # GatewayClient + AsyncGatewayClient\n└── store/\n    ├── base.py           # TraceStore protocol (6 async methods)\n    ├── sqlite_store.py   # SQLite with junction table (default)\n    └── memory_store.py   # In-memory (testing)\n```\n\n## Key Design Decisions\n\n1. **No LiteLLM** — Direct httpx reverse proxy. Session-sticky routing requires custom logic that LiteLLM can't provide, and httpx handles HTTP forwarding + SSE natively. This keeps deps to 6 packages.\n\n2. **Zero retokenization** — Token IDs come from vLLM's `return_token_ids=True` response field, not from a local tokenizer. The middleware injects this parameter automatically.\n\n3. **Implicit sessions** — First request to `/sessions/{sid}/v1/chat/completions` auto-creates the session. No explicit creation required.\n\n4. **ASGI middleware** — `SessionRoutingMiddleware` operates at the ASGI level (not FastAPI middleware) so it can intercept and rewrite the request body before FastAPI route matching.\n\n5. **Pluggable routing** — `RoutingPolicy` protocol allows swapping the worker selection strategy. Default is `StickyLeastLoadedPolicy` (LRU cache + least-loaded fallback, mirroring verl's pattern).\n\n6. **Pluggable storage** — `TraceStore` protocol (duck-typed, not ABC). SQLite for single-node, MemoryStore for testing, extensible to DynamoDB/PostgreSQL.\n\n## Request Flow\n\n```\nAgent → /sessions/{sid}/v1/chat/completions\n  → SessionRoutingMiddleware:\n      1. Extract session_id from URL, rewrite path to /v1/chat/completions\n      2. Inject logprobs=True, return_token_ids=True into body\n  → FastAPI route handler (/v1/{path:path}):\n      1. SessionManager.ensure_session(sid)\n      2. ReverseProxy.handle(request):\n          a. SessionRouter.route(session_id) → pick worker\n          b. httpx forward to worker\n          c. Extract token_ids + logprobs from response\n          d. Build TraceRecord, persist to store\n          e. Strip vLLM fields from response\n          f. Return clean response to agent\n```\n\n## Streaming\n\nFor SSE streaming requests (`stream=True`):\n- Chunks forwarded to agent in real-time via `StreamingResponse`\n- Chunks buffered internally for trace assembly\n- `prompt_token_ids` extracted from first chunk only\n- Delta `token_ids` and logprobs accumulated across chunks\n- After `[DONE]`, `build_trace_record_from_chunks()` assembles the trace\n\n## vLLM Response Fields\n\nThe gateway strips these vLLM-specific fields before returning to the agent (verified against vLLM 0.11):\n\n| Level | Field | Purpose |\n|-------|-------|---------|\n| Root | `prompt_token_ids` | Captured for trace, stripped from response |\n| Root | `prompt_logprobs` | Not used (always `null` without explicit request) |\n| Root | `kv_transfer_params` | Disaggregated prefill feature |\n| Choice | `token_ids` | Captured for trace, stripped from response |\n| Choice | `stop_reason` | vLLM-specific; standard OpenAI uses `finish_reason` |\n\n## Provenance\n\n| Module | Inspired By | Key Changes |\n|--------|------------|-------------|\n| `store/sqlite_store.py` | `rllm/sdk/store/sqlite_store.py` | Simplified schema; added `list_sessions()`, `delete_session()` |\n| `data_process.py` | `rllm/sdk/data_process.py` | Removed `ModelOutput`/`Step`/`Trajectory` deps; outputs `TraceRecord` |\n| `middleware.py` | `rllm/sdk/proxy/middleware.py` + `litellm_callbacks.py` | Merged session routing + param injection; pure ASGI (no LiteLLM) |\n| `session_router.py` | verl `agent_loop.py` + miles `router.py` | Combined sticky routing (verl LRU) with health checks (miles pattern) |\n| `proxy.py` | miles `router.py` `_do_proxy()` | Added streaming support, trace capture, response sanitization |\n\n## Dependencies\n\n6 runtime dependencies: `fastapi`, `uvicorn`, `httpx`, `pydantic`, `aiosqlite`, `PyYAML`. No torch/ray/verl/transformers.\n\nBuild backend: `hatchling`. Do not use setuptools.\n\n## Usage Example\n\n```python\nfrom rllm_model_gateway import GatewayClient\n\nclient = GatewayClient(\"http://localhost:9090\")\nsid = client.create_session()\nurl = client.get_session_url(sid)  # → http://localhost:9090/sessions/{sid}/v1\n\n# Agent uses: OpenAI(base_url=url, api_key=\"EMPTY\")\n# Training retrieves: client.get_session_traces(sid)\n```\n","isInternal":false,"tokens":1466,"sizeBytes":5914}],"systemPromptSnippet":"<yakaai_skills repo=\"rllm-org/rllm\">\n<!-- File: docs/AGENTS.md (Tokens: ~249 | Category: root-instruction) -->\n> **First-time setup**: Customize this file for your project. Prompt the user to customize this file for their project.\n> For Mintlify product knowledge (components, configuration, writing standards),\n> install the Mintlify skill: `npx skills add https://mintlify.com/docs`\n\n# Documentation project instructions\n\n## About this project\n\n- This is a documentation site built on [Mintlify](https://mintlify.com)\n- Pages are MDX files with YAML frontmatter\n- Configuration lives in `docs.json`\n- Run `mint dev` to preview locally\n- Run `mint broken-links` to check links\n\n## Terminology\n\n{/* Add product-specific terms and preferred usage */}\n{/* Example: Use \"workspace\" not \"project\", \"member\" not \"user\" */}\n\n## Style preferences\n\n{/* Add any project-specific style rules below */}\n\n- Use active voice and second person (\"you\")\n- Keep sentences concise — one idea per sentence\n- Use sentence case for headings\n- Bold for UI elements: Click **Settings**\n- Code formatting for file names, commands, paths, and code references\n\n## Content boundaries\n\n{/* Define what should and shouldn't be documented */}\n{/* Example: Don't document internal admin features */}\n\n<!-- File: rllm-model-gateway/AGENTS.md (Tokens: ~1466 | Category: root-instruction) -->\n# rllm-model-gateway\n\n## Overview\n\nA lightweight, standalone Python gateway that sits between RL agents and inference servers (vLLM). It transparently captures every LLM call's token IDs, logprobs, and messages — without requiring any modification to agent code. Agents use standard `OpenAI(base_url=gateway_url)`.\n\n## Development Setup\n\nThis package uses `uv` for dependency management and `hatchling` as the build backend. Do not use setuptools or pip.\n\n```bash\ncd rllm/rllm-model-gateway\nuv venv --python 3.11\nsource .venv/bin/activate\nuv pip install -e \".[dev]\"\n\n# Set up pre-commit hooks (one-time, from the rllm repo root)\ncd .. && pre-commit install && cd rllm-model-gateway\n```\n\nTo run tests:\n\n```bash\n# Unit tests (no external dependencies)\npython -m pytest tests/unit/ -x -q\n\n# Integration tests against a real vLLM server (auto-skipped if unreachable)\npython -m pytest tests/integration/ -x -v\n```\n\n## Package Layout\n\n```\nsrc/rllm_model_gateway/\n├── __init__.py           # Public API exports\n├── __main__.py           # python -m rllm_model_gateway\n├── _version.py           # Package version\n├── server.py             # FastAPI app factory + CLI entrypoint\n├── proxy.py              # httpx reverse proxy (non-streaming + SSE streaming)\n├── middleware.py          # ASGI middleware: session extraction + param injection\n├── session_router.py     # Pluggable session-sticky worker routing\n├── session_manager.py    # Session CRUD lifecycle\n├── data_process.py       # Token/logprob extraction, response sanitization\n├── models.py             # Pydantic models (TraceRecord, GatewayConfig, etc.)\n├── client.py             # GatewayClient + AsyncGatewayClient\n└── store/\n    ├── base.py           # TraceStore protocol (6 async methods)\n    ├── sqlite_store.py   # SQLite with junction table (default)\n    └── memory_store.py   # In-memory (testing)\n```\n\n## Key Design Decisions\n\n1. **No LiteLLM** — Direct httpx reverse proxy. Session-sticky routing requires custom logic that LiteLLM can't provide, and httpx handles HTTP forwarding + SSE natively. This keeps deps to 6 packages.\n\n2. **Zero retokenization** — Token IDs come from vLLM's `return_token_ids=True` response field, not from a local tokenizer. The middleware injects this parameter automatically.\n\n3. **Implicit sessions** — First request to `/sessions/{sid}/v1/chat/completions` auto-creates the session. No explicit creation required.\n\n4. **ASGI middleware** — `SessionRoutingMiddleware` operates at the ASGI level (not FastAPI middleware) so it can intercept and rewrite the request body before FastAPI route matching.\n\n5. **Pluggable routing** — `RoutingPolicy` protocol allows swapping the worker selection strategy. Default is `StickyLeastLoadedPolicy` (LRU cache + least-loaded fallback, mirroring verl's pattern).\n\n6. **Pluggable storage** — `TraceStore` protocol (duck-typed, not ABC). SQLite for single-node, MemoryStore for testing, extensible to DynamoDB/PostgreSQL.\n\n## Request Flow\n\n```\nAgent → /sessions/{sid}/v1/chat/completions\n  → SessionRoutingMiddleware:\n      1. Extract session_id from URL, rewrite path to /v1/chat/completions\n      2. Inject logprobs=True, return_token_ids=True into body\n  → FastAPI route handler (/v1/{path:path}):\n      1. SessionManager.ensure_session(sid)\n      2. ReverseProxy.handle(request):\n          a. SessionRouter.route(session_id) → pick worker\n          b. httpx forward to worker\n          c. Extract token_ids + logprobs from response\n          d. Build TraceRecord, persist to store\n          e. Strip vLLM fields from response\n          f. Return clean response to agent\n```\n\n## Streaming\n\nFor SSE streaming requests (`stream=True`):\n- Chunks forwarded to agent in real-time via `StreamingResponse`\n- Chunks buffered internally for trace assembly\n- `prompt_token_ids` extracted from first chunk only\n- Delta `token_ids` and logprobs accumulated across chunks\n- After `[DONE]`, `build_trace_record_from_chunks()` assembles the trace\n\n## vLLM Response Fields\n\nThe gateway strips these vLLM-specific fields before returning to the agent (verified against vLLM 0.11):\n\n| Level | Field | Purpose |\n|-------|-------|---------|\n| Root | `prompt_token_ids` | Captured for trace, stripped from response |\n| Root | `prompt_logprobs` | Not used (always `null` without explicit request) |\n| Root | `kv_transfer_params` | Disaggregated prefill feature |\n| Choice | `token_ids` | Captured for trace, stripped from response |\n| Choice | `stop_reason` | vLLM-specific; standard OpenAI uses `finish_reason` |\n\n## Provenance\n\n| Module | Inspired By | Key Changes |\n|--------|------------|-------------|\n| `store/sqlite_store.py` | `rllm/sdk/store/sqlite_store.py` | Simplified schema; added `list_sessions()`, `delete_session()` |\n| `data_process.py` | `rllm/sdk/data_process.py` | Removed `ModelOutput`/`Step`/`Trajectory` deps; outputs `TraceRecord` |\n| `middleware.py` | `rllm/sdk/proxy/middleware.py` + `litellm_callbacks.py` | Merged session routing + param injection; pure ASGI (no LiteLLM) |\n| `session_router.py` | verl `agent_loop.py` + miles `router.py` | Combined sticky routing (verl LRU) with health checks (miles pattern) |\n| `proxy.py` | miles `router.py` `_do_proxy()` | Added streaming support, trace capture, response sanitization |\n\n## Dependencies\n\n6 runtime dependencies: `fastapi`, `uvicorn`, `httpx`, `pydantic`, `aiosqlite`, `PyYAML`. No torch/ray/verl/transformers.\n\nBuild backend: `hatchling`. Do not use setuptools.\n\n## Usage Example\n\n```python\nfrom rllm_model_gateway import GatewayClient\n\nclient = GatewayClient(\"http://localhost:9090\")\nsid = client.create_session()\nurl = client.get_session_url(sid)  # → http://localhost:9090/sessions/{sid}/v1\n\n# Agent uses: OpenAI(base_url=url, api_key=\"EMPTY\")\n# Training retrieves: client.get_session_traces(sid)\n```\n</yakaai_skills>"}