llm_wiki Documentation

File: README.md Copy Chapter

LLM Wiki

Features Copy Chapter

• Two-Step Chain-of-Thought Ingest — LLM analyzes first, then generates wiki pages with source traceability and incremental cache
• Multimodal Image Ingestion — extract embedded images from PDFs, generate factual captions with a vision LLM, surface them in image-aware search results with lightbox preview and jump-to-source
• Multi-format Document Parsing — ingest PDF, Office documents, EPUB/MOBI, Org mode, images, media, web clips, and batches of URLs, with built-in, cloud, or local MinerU PDF processing
• Flexible Model Configuration — configure models per project, route Chat and Ingest independently, and manage custom providers, headers, and streaming output
• Source-grounded Retrieval — use Read Sources Only mode to answer exclusively from original imported material
• Project Management & Migration — export and import complete project archives across devices, and rebuild the Wiki index from existing pages
• 4-Signal Knowledge Graph — relevance model with direct links, source overlap, Adamic-Adar, and type affinity
• Louvain Community Detection — automatic knowledge cluster discovery with cohesion scoring
• Graph Insights — surprising connections and knowledge gaps with one-click Deep Research
• Vector Semantic Search — optional embedding-based retrieval via LanceDB, supports any OpenAI-compatible endpoint
• Persistent Ingest Queue — serial processing with crash recovery, cancel, retry, and progress visualization
• Folder Import — recursive folder import preserving directory structure, folder context as LLM classification hint
• Source Folder Auto-Watch — detects external changes in raw/sources/ and keeps ingest/delete cleanup in sync
• Deep Research — LLM-optimized search topics, multi-query web search via Tavily, SerpApi, or SearXNG, auto-ingest results into wiki
• Rust Backend Chat Agent — tool-using chat runtime with wiki/source/graph/web retrieval, workspace file generation, shell approval, cancellation, and streaming tool events
• Agent Skills — scan and enable local SKILL.md folders, select skills with /skill, and let the Agent read skill instructions on demand
• Generated Outputs Preview — Agent-created Markdown, HTML, images, and other workspace files appear as outputs with preview and quick folder access
• Mermaid Diagram Rendering — render Mermaid code blocks directly in chat and preview, with compact syntax-error cards instead of raw parser output
• Async Review System — LLM flags items for human judgment, predefined actions, pre-generated search queries
• Chrome Web Clipper — one-click web page capture with auto-ingest into knowledge base
• Local HTTP API + MCP Server + AI Agent Skill — built-in 127.0.0.1:19828 JSON API and bundled MCP server for hybrid search, file read, graph traversal, and source rescan; ready-made agent skill installs into Claude Code / Codex with one command (npx skills add …)

What is this? Copy Chapter

LLM Wiki is a cross-platform desktop application that turns your documents into an organized, interlinked knowledge base — automatically. Instead of traditional RAG (retrieve-and-answer from scratch every time), the LLM incrementally builds and maintains a persistent wiki from your sources. Knowledge is compiled once and kept current, not re-derived on every query.

This project is based on Karpathy's LLM Wiki pattern — a methodology for building personal knowledge bases using LLMs. llm_wiki is created and maintained by nash_su, who implemented the core ideas as a full desktop application with significant enhancements.

Credits Copy Chapter

The foundational methodology comes from Andrej Karpathy's llm-wiki.md, which describes the pattern of using LLMs to incrementally build and maintain a personal wiki. The original document is an abstract design pattern; this project is a concrete implementation with substantial extensions.

What We Kept from the Original Copy Chapter

The core architecture follows Karpathy's design faithfully:

• Three-layer architecture: Raw Sources (immutable) → Wiki (LLM-generated) → Schema (rules & config)
• Three core operations: Ingest, Query, Lint
• index.md as the content catalog and LLM navigation entry point
• log.md as the chronological operation record with parseable format
• [[wikilink]] syntax for cross-references
• YAML frontmatter on every wiki page
• Obsidian compatibility — the wiki directory works as an Obsidian vault
• Human curates, LLM maintains — the fundamental role division

What We Changed & Added Copy Chapter

1. From CLI to Desktop Application Copy Section

The original is an abstract pattern document designed to be copy-pasted to an LLM agent. We built it into a full cross-platform desktop application with:

• Three-column layout: Knowledge Tree / File Tree (left) + Chat (center) + Preview (right)
• Icon sidebar for switching between Wiki, Sources, Search, Graph, Lint, Review, Deep Research, Settings
• Custom resizable panels — drag-to-resize left and right panels with min/max constraints
• Activity panel — real-time processing status showing file-by-file ingest progress
• All state persisted — conversations, settings, review items, project config survive restarts
• Scenario templates — Research, Reading, Personal Growth, Business, General — each pre-configures purpose.md and schema.md

2. Purpose.md — The Wiki's Soul Copy Section

The original has Schema (how the wiki works) but no formal place for why the wiki exists. We added purpose.md:

• Defines goals, key questions, research scope, evolving thesis
• LLM reads it during every ingest and query for context
• LLM can suggest updates based on usage patterns
• Different from schema — schema is structural rules, purpose is directional intent

3. Two-Step Chain-of-Thought Ingest Copy Section

The original describes a single-step ingest where the LLM reads and writes simultaneously. We split it into two sequential LLM calls for significantly better quality:

Step 1 (Analysis): LLM reads source → structured analysis
  - Key entities, concepts, arguments
  - Connections to existing wiki content
  - Contradictions & tensions with existing knowledge
  - Recommendations for wiki structure

Step 2 (Generation): LLM takes analysis → generates wiki files
  - Source summary with frontmatter (type, title, sources[])
  - Entity pages, concept pages with cross-references
  - Updated index.md, log.md, overview.md
  - Review items for human judgment
  - Search queries for Deep Research

Additional ingest enhancements beyond the original:

• SHA256 incremental cache — source file content is hashed before ingest; unchanged files are skipped automatically, saving LLM tokens and time
• Persistent ingest queue — serial processing prevents concurrent LLM calls; queue persisted to disk, survives app restart; failed tasks auto-retry up to 3 times
• Folder import — recursive folder import preserving directory structure; folder path passed to LLM as classification context (e.g., "papers > energy" helps categorize content)
• Source folder auto-watch — files added, edited, or deleted in raw/sources/ outside the app are picked up automatically and reuse the same ingest/delete lifecycle as in-app actions
• Queue visualization — Activity Panel shows progress bar, pending/processing/failed tasks with cancel and retry buttons
• Auto-embedding — when vector search is enabled, new pages are automatically embedded after ingest
• Source traceability — every generated wiki page includes a sources: [] field in YAML frontmatter, linking back to the raw source files that contributed to it
• overview.md auto-update — global summary page regenerated on every ingest to reflect the latest state of the wiki
• Guaranteed source summary — fallback ensures a source summary page is always created, even if the LLM omits it
• Language-aware generation — LLM responds in the user's configured language (English or Chinese)
• Progressive Sources view — large source folders render progressively while scrolling, keeping big source collections responsive

4. Knowledge Graph with Relevance Model Copy Section

The original mentions [[wikilinks]] for cross-references but has no graph analysis. We built a full knowledge graph visualization and relevance engine:

4-Signal Relevance Model:

Graph Visualization (sigma.js + graphology + ForceAtlas2):

• Node colors by page type or community, sizes scaled by link count (√ scaling)
• Edge thickness and color by relevance weight (green=strong, gray=weak)
• Hover interaction: neighbors stay visible, non-neighbors dim, edges highlight with relevance score label
• Zoom controls (ZoomIn, ZoomOut, Fit-to-screen)
• Position caching prevents layout jumps when data updates
• Legend switches between type counts and community info based on coloring mode

5. Louvain Community Detection Copy Section

Not in the original. Automatic discovery of knowledge clusters using the Louvain algorithm (graphology-communities-louvain):

• Auto-clustering — discovers which pages naturally group together based on link topology, independent of predefined page types
• Type / Community toggle — switch between coloring nodes by page type (entity, concept, source...) or by discovered knowledge cluster
• Cohesion scoring — each community scored by intra-edge density (actual edges / possible edges); low-cohesion clusters (< 0.15) flagged with warning
• 12-color palette — distinct visual separation between clusters
• Community legend — shows top node label, member count, and cohesion per cluster

6. Graph Insights — Surprising Connections & Knowledge Gaps Copy Section

Not in the original. The system automatically analyzes graph structure to surface actionable insights:

Surprising Connections:

• Detects unexpected relationships: cross-community edges, cross-type links, peripheral↔hub couplings
• Composite surprise score ranks the most noteworthy connections
• Dismissable — mark connections as reviewed so they don't reappear

Knowledge Gaps:

• Isolated pages (degree ≤ 1) — pages with few or no connections to the rest of the wiki
• Sparse communities (cohesion < 0.15, ≥ 3 pages) — knowledge areas with weak internal cross-references
• Bridge nodes (connecting 3+ clusters) — critical junction pages that hold multiple knowledge areas together

Interactive:

• Click any insight card to highlight corresponding nodes and edges in the graph; click again to deselect
• Knowledge gaps and bridge nodes have a Deep Research button — triggers LLM-optimized research with domain-aware topics (reads overview.md + purpose.md for context)
• Research topic shown in editable confirmation dialog before starting — user can refine topic and search queries

7. Optimized Query Retrieval Pipeline Copy Section

The original describes a simple query where the LLM reads relevant pages. We built a multi-phase retrieval pipeline with optional vector search and budget control:

Phase 1: Tokenized Search
  - English: word splitting + stop word removal
  - Chinese: CJK bigram tokenization (每个 → [每个, 个…])
  - Title match bonus (+10 score)
  - Searches both wiki/ and raw/sources/

Phase 1.5: Vector Semantic Search (optional)
  - Embedding via any OpenAI-compatible /v1/embeddings endpoint
  - Stored in LanceDB (Rust backend) for fast ANN retrieval
  - Cosine similarity finds semantically related pages even without keyword overlap
  - Results merged into search: boosts existing matches + adds new discoveries

Phase 2: Graph Expansion
  - Top search results used as seed nodes
  - 4-signal relevance model finds related pages
  - 2-hop traversal with decay for deeper connections

Phase 3: Budget Control
  - Configurable context window: 4K → 1M tokens
  - Proportional allocation: 60% wiki pages, 20% chat history, 5% index, 15% system
  - Pages prioritized by combined search + graph relevance score

Phase 4: Context Assembly
  - Numbered pages with full content (not just summaries)
  - System prompt includes: purpose.md, language rules, citation format, index.md
  - LLM instructed to cite pages by number: [1], [2], etc.

Vector Search is fully optional — disabled by default, enabled in Settings with independent endpoint, API key, and model configuration. When disabled, the pipeline falls back to tokenized search + graph expansion. Benchmark: overall recall improved from 58.2% to 71.4% with vector search enabled.

8. Multi-Conversation Chat with Persistence Copy Section

The original has a single query interface. We built full multi-conversation support:

• Independent chat sessions — create, rename, delete conversations
• Conversation sidebar — quick switching between topics
• Per-conversation persistence — each conversation saved to .llm-wiki/chats/{id}.json
• Configurable history depth — limit how many messages are sent as context (default: 10)
• Cited references panel — collapsible section on each response showing which wiki pages were used, grouped by type with icons
• Reference persistence — cited pages stored directly in message data, stable across restarts
• Regenerate — re-generate the last response with one click (removes last assistant + user message pair, re-sends)
• Save to Wiki — archive valuable answers to wiki/queries/, then auto-ingest to extract entities/concepts into the knowledge network

9. Rust Backend Chat Agent & Skills Copy Section

Not in the original. Chat now runs through a Rust backend Agent runtime rather than a browser-only TypeScript loop:

• Tool-using Agent — can choose wiki search, source search, graph search, web search, AnyTXT, workspace file tools, approved shell commands, and skill file reads
• Skill management — scan project and user skill folders, enable or disable skills, and pick a skill per conversation with /skill completion
• Generated workspace outputs — files produced by Agent tools are kept under agent-workspace/, shown as generated outputs, and can be previewed or opened from the chat
• User interaction forms — skills can ask for structured user input such as single choice, multiple choice, or free text without hardcoding skill-specific UI
• Safer execution model — project workspace commands can continue smoothly, while external shell commands still require explicit approval

10. Thinking / Reasoning Display Copy Section

Not in the original. For LLMs that emit `` blocks (DeepSeek, QwQ, etc.):

• Streaming thinking — rolling 5-line display with opacity fade during generation
• Collapsed by default — thinking blocks hidden after completion, click to expand
• Visual separation — thinking content shown in distinct style, separate from the main response

11. Markdown Rendering: KaTeX Math & Mermaid Diagrams Copy Section

Not in the original. Rich Markdown rendering across chat and preview:

• KaTeX rendering — inline $...$ and block $$...$$ formulas rendered via remark-math + rehype-katex
• Milkdown math plugin — preview editor renders math natively via @milkdown/plugin-math
• Auto-detection — bare \begin{aligned} and other LaTeX environments automatically wrapped with $$ delimiters
• Unicode fallback — 100+ symbol mappings (α, ∑, →, ≤, etc.) for simple inline notation outside math blocks
• Mermaid code blocks — fenced mermaid diagrams render directly as flowcharts, sequence diagrams, and other Mermaid-supported visuals
• Compact Mermaid errors — syntax failures are captured inside a small error card instead of spilling raw parser output into the chat

12. Review System (Async Human-in-the-Loop) Copy Section

The original suggests staying involved during ingest. We added an asynchronous review queue:

• LLM flags items needing human judgment during ingest
• Predefined action types: Create Page, Deep Research, Skip — constrained to prevent LLM hallucination of arbitrary actions
• Search queries generated at ingest time — LLM pre-generates optimized web search queries for each review item
• User handles reviews at their convenience — doesn't block ingest

13. Deep Research Copy Section

Not in the original. When the LLM identifies knowledge gaps:

• Web search via Tavily, SerpApi, or SearXNG finds relevant sources with full content extraction (no truncation)
• Provider-specific configuration — Tavily and SerpApi use independent API keys; SerpApi supports selectable engines, while SearXNG uses a configured instance URL and search categories
• Multiple search queries per topic — LLM-generated at ingest time, optimized for search engines
• LLM-optimized research topics — when triggered from Graph Insights, LLM reads overview.md + purpose.md to generate domain-specific topics and queries (not generic keywords)
• User confirmation dialog — editable topic and search queries shown for review before research starts
• LLM synthesizes findings into a wiki research page with cross-references to existing wiki
• Thinking display — `` blocks shown as collapsible sections during synthesis, auto-scroll to latest content
• Auto-ingest — research results automatically processed to extract entities/concepts into the wiki
• Task queue with 3 concurrent tasks
• Research Panel — dedicated sidebar panel with dynamic height, real-time streaming progress

14. Browser Extension (Web Clipper) Copy Section

The original mentions Obsidian Web Clipper. We built a dedicated Chrome Extension (Manifest V3):

• Mozilla Readability.js for accurate article extraction (strips ads, nav, sidebars)
• Turndown.js for HTML → Markdown conversion with table support
• Project picker — choose which wiki to clip into (supports multi-project)
• Local HTTP API (port 19827, tiny_http) — Extension ↔ App communication
• Auto-ingest — clipped content automatically triggers the two-step ingest pipeline
• Clip watcher — polls every 3 seconds for new clips, processes automatically
• Offline preview — shows extracted content even when app is not running

15. Multi-format Document Support Copy Section

The original focuses on text/markdown. We support structured extraction preserving document semantics:

MinerU is optional. Use MinerU Cloud, an official Local API endpoint, or Local Pipeline mode for complex PDFs. Local modes keep processing on your machine, and extracted images are stored in the project-managed wiki/media directory. If MinerU fails, LLM Wiki falls back to the built-in parser.

16. File Deletion with Cascade Cleanup Copy Section

The original has no deletion mechanism. We added intelligent cascade deletion:

• Deleting a source file removes its wiki summary page
• 3-method matching finds related wiki pages: frontmatter sources[] field, source summary page name, frontmatter section references
• Shared entity preservation — entity/concept pages linked to multiple sources only have the deleted source removed from their sources[] array, not deleted entirely
• Index cleanup — removed pages are purged from index.md
• Wikilink cleanup — dead [[wikilinks]] to deleted pages are removed from remaining wiki pages

17. Configurable Context Window Copy Section

Not in the original. Users can configure how much context the LLM receives:

• Slider from 4K to 1M tokens — adapts to different LLM capabilities
• Proportional budget allocation — larger windows get proportionally more wiki content
• 60/20/5/15 split — wiki pages / chat history / index / system prompt

18. Cross-Platform Compatibility Copy Section

The original is platform-agnostic (abstract pattern). We handle concrete cross-platform concerns:

• Path normalization — unified normalizePath() used across 22+ files, backslash → forward slash
• Unicode-safe string handling — char-based slicing instead of byte-based (prevents crashes on CJK filenames)
• macOS close-to-hide — close button hides window (app stays running in background), click dock icon to restore, Cmd+Q to quit
• Windows/Linux close confirmation — confirmation dialog before quitting to prevent accidental data loss
• Tauri v2 — native desktop on macOS, Windows, Linux
• GitHub Actions CI/CD — automated builds for macOS (ARM + Intel), Windows (.msi), Linux (.deb / .AppImage)

19. Other Additions Copy Section

• i18n — English + Chinese interface (react-i18next)
• Settings persistence — LLM provider, API key, model, context size, language saved via Tauri Store
• Obsidian config — auto-generated .obsidian/ directory with recommended settings
• Markdown rendering — GFM tables with borders, proper code blocks, wikilink processing in chat and preview
• Multi-provider LLM support — OpenAI, Anthropic, Google, Ollama, Custom — each with provider-specific streaming and headers
• Configurable LLM timeout — adjust request timeouts for slow local models and long-running operations
• Configurable Firecrawl — optional API key and custom Base URL for hosted or self-hosted services
• Collapsible file sidebar — collapse Knowledge/Files navigation while preserving its state
• Project maintenance — ZIP export/import for migration and deterministic wiki/index.md rebuilding
• dataVersion signaling — graph and UI automatically refresh when wiki content changes

Tech Stack Copy Chapter

Installation Copy Chapter

Pre-built Binaries Copy Section

Download from Releases:

• macOS: .dmg (Apple Silicon + Intel)
• Windows: .msi
• Linux: .deb / .AppImage

Build from Source Copy Section

# Prerequisites: Node.js 20+, Rust 1.70+
git clone https://github.com/nashsu/llm_wiki.git
cd llm_wiki
npm install
npm run tauri dev      # Development
npm run tauri build    # Production build

Chrome Extension Copy Section

1. Open chrome://extensions
2. Enable "Developer mode"
3. Click "Load unpacked"
4. Select the extension/ directory
5. Clip the current page with Alt+Shift+L (Command+Shift+L on macOS). Customize it at chrome://extensions/shortcuts.

Quick Start Copy Chapter

1. Launch the app → Create a new project (choose a template)
2. Go to Settings → Configure your LLM provider (API key + model)
3. Optional: configure Web Search providers and source folder auto-watch in Settings
4. Go to Sources → Import documents (PDF, DOCX, MD, etc.)
5. Watch the Activity Panel — LLM automatically builds wiki pages
6. Use Chat to query your knowledge base
7. Browse the Knowledge Graph to see connections
8. Check Review for items needing your attention
9. Run Lint periodically to maintain wiki health

Local HTTP API + MCP Server + AI Agent Skill Copy Chapter

LLM Wiki ships a built-in local HTTP API at http://127.0.0.1:19828 (token-protected, 127.0.0.1-only) so external tools — including AI agents like Claude Code, Codex, or any HTTP-capable script — can query your wiki:

• GET /api/v1/health — server status (no auth)
• GET /api/v1/projects — list projects
• GET /api/v1/projects/{id}/files / files/content — read files and content
• GET /api/v1/projects/{id}/reviews?status=unresolved — export Review tab items for wiki maintenance (status: unresolved, resolved, or all; optional type and limit)
• PATCH /api/v1/projects/{id}/reviews/{reviewId} — update one Review item (JSON body { "resolved": true, "action": "label" }; resolved defaults to true, pass false to reopen)
• POST /api/v1/projects/{id}/reviews/resolve — bulk-resolve Review items (JSON body { "ids": [...], "action": "label" }), returns { resolved, notFound, count }; the Review tab's Refresh button re-reads the result from disk
• POST /api/v1/projects/{id}/search — hybrid retrieval (keyword + vector) returning mode, tokenHits, vectorHits, per-result vectorScore
• POST /api/v1/projects/{id}/chat — backend Agent chat endpoint for wiki/source/web/AnyTXT retrieval. JSON requests remain non-streaming by default; send "stream": true or Accept: text/event-stream for SSE events (meta, incremental agent, then done, cancelled, or error). The terminal done frame contains the complete aggregate response, so clients should not render both message deltas and the final message as separate answers. mode: "deep" broadens evidence collection, while the full Deep Research workspace remains available in the desktop UI
• GET /api/v1/projects/{id}/graph — wikilinks graph
• POST /api/v1/projects/{id}/sources/rescan — trigger a backend rescan
• POST /api/v1/projects/{id}/pages/embed — index one externally created or updated wiki/*.md page without rebuilding the whole vector database

Enable the API, generate a token, and choose whether local unauthenticated access is allowed in Settings → API + MCP.

For MCP-compatible clients, LLM Wiki also includes a local MCP server in mcp-server/. After building it with npm run mcp:build, Settings → API + MCP shows a copyable MCP client configuration with the correct local path for your machine. The MCP tools call the same API surface, so agent clients can list projects, read files, export unresolved Review items, run hybrid search, inspect the graph, trigger source rescans, and call the same Rust backend Agent chat endpoint without custom HTTP glue code.

Plug your AI agent in with one command Copy Section

A ready-made agent skill for LLM Wiki lives in its own repo. Install it into Claude Code / Codex / any skills-compatible runtime:

npx skills add https://github.com/nashsu/llm_wiki_skill.git --skill llm-wiki

After install, the agent can answer prompts like "what does my LLM Wiki say about X", "search my 知识库 for Y", "show the neighborhood of node Z in my wiki graph", and "rescan my wiki sources" by talking to your locally-running app — read-only by default, citing wiki page paths so you can verify in-app.

• Skill repo: https://github.com/nashsu/llm_wiki_skill
• Trigger discipline: it intentionally does not trigger on generic "search my notes" / "check my Obsidian / Notion / Logseq" — only when you explicitly name LLM Wiki / my wiki / 知识库.

Project Structure Copy Chapter

my-wiki/
├── purpose.md              # Goals, key questions, research scope
├── schema.md               # Wiki structure rules, page types
├── raw/
│   ├── sources/            # Uploaded documents (immutable)
│   └── assets/             # Local images
├── wiki/
│   ├── index.md            # Content catalog
│   ├── log.md              # Operation history
│   ├── overview.md         # Global summary (auto-updated)
│   ├── entities/           # People, organizations, products
│   ├── concepts/           # Theories, methods, techniques
│   ├── sources/            # Source summaries
│   ├── queries/            # Saved chat answers + research
│   ├── synthesis/          # Cross-source analysis
│   └── comparisons/        # Side-by-side comparisons
├── .obsidian/              # Obsidian vault config (auto-generated)
└── .llm-wiki/              # App config, chat history, review items

Star History Copy Chapter

License Copy Chapter

This project is licensed under the GNU General Public License v3.0 — see LICENSE for details.

File: README_CN.md Copy Chapter

LLM Wiki

功能亮点 Copy Chapter

• 两步思维链摄入 — LLM 先分析再生成 Wiki 页面，来源可追溯，支持增量缓存
• 多模态图片摄入 — 自动提取 PDF 内嵌图片，调用视觉模型生成事实性描述，搜索结果按图文分区，支持 lightbox 预览与跳转到原始文档对应位置
• 多格式文档解析 — 支持 PDF、Office 文档、EPUB/MOBI、Org mode、图片、音视频、网页剪藏和批量 URL 导入，并提供内置、云端或本地 MinerU PDF 处理
• 灵活的模型配置 — 支持项目级模型配置、Chat/Ingest 独立模型路由，以及自定义 Provider、请求头和流式输出
• 原始资料检索 — 可使用“只读原文”模式，仅依据导入的原始资料回答
• 项目管理与迁移 — 支持完整项目归档的跨设备导入导出，并可根据现有 Wiki 页面重建索引
• 四信号知识图谱 — 直接链接、来源重叠、Adamic-Adar、类型亲和四维关联度模型
• Louvain 社区检测 — 自动发现知识聚类，内聚度评分
• 图谱洞察 — 惊奇连接与知识空白检测，一键触发 Deep Research
• 向量语义搜索 — 可选的 embedding 检索，基于 LanceDB，支持任意 OpenAI 兼容端点
• 持久化摄入队列 — 串行处理，崩溃恢复，取消/重试，进度可视化
• 文件夹导入 — 递归导入保留目录结构，文件夹路径作为 LLM 分类上下文
• Source 文件夹自动监听 — 检测 raw/sources/ 的外部变更，并同步触发摄入或删除清理
• 深度研究 — LLM 智能生成搜索主题，通过 Tavily、SerpApi 或 SearXNG 进行多查询网络搜索，研究结果自动摄入 Wiki
• Rust 后端 Chat Agent — 支持工具调用的聊天运行时，可进行 Wiki/Source/Graph/Web 检索、workspace 文件生成、shell 审批、取消和流式工具事件展示
• Agent Skills — 扫描并启用本地 SKILL.md 目录，在聊天中用 /skill 选择，让 Agent 按需读取 Skill 指令
• 生成物预览 — Agent 生成的 Markdown、HTML、图片等 workspace 文件会作为生成物展示，支持预览和快速打开目录
• Mermaid 流程图渲染 — 聊天和预览中可直接渲染 Mermaid 代码块，语法错误会显示为紧凑错误卡片
• 异步审核系统 — LLM 在摄入时标记需人工判断的项，预定义操作，预生成搜索查询
• Chrome 网页剪藏 — 一键捕获网页内容，自动摄入知识库
• 本地 HTTP API + MCP Server + AI Agent Skill — 内置 127.0.0.1:19828 JSON API 和随包提供的 MCP Server，支持 Hybrid 检索、文件读取、知识图谱遍历、源资料重新扫描；配套 agent skill 一行命令接入 Claude Code / Codex（npx skills add …）

这是什么？ Copy Chapter

LLM Wiki 是一个跨平台桌面应用，能将你的文档自动转化为有组织、相互关联的知识库。与传统 RAG（每次查询都从头检索和回答）不同，LLM 会从你的资料中增量构建并维护一个持久化的 Wiki。知识只编译一次并持续更新，而非每次查询都重新推导。

本项目基于 Karpathy 的 LLM Wiki 方法论 —— 一套使用 LLM 构建个人知识库的方法论。llm_wiki 由 nash_su 创建和维护，并在保留核心理念的基础上，将其实现为完整桌面应用，加入了大量增强。

致谢 Copy Chapter

基础方法论来自 Andrej Karpathy 的 llm-wiki.md，描述了使用 LLM 增量构建和维护个人 Wiki 的设计模式。原始文档是一个抽象的设计范式；本项目是一个具体的实现，并有大量扩展。

保留的原始设计 Copy Chapter

核心架构忠实遵循 Karpathy 的方法论：

• 三层架构：原始资料（不可变）→ Wiki（LLM 生成）→ Schema（规则和配置）
• 三个核心操作：Ingest（摄入）、Query（查询）、Lint（检查）
• index.md 作为内容目录和 LLM 导航入口
• log.md 作为可解析格式的时序操作记录
• [[wikilink]] 语法用于交叉引用
• YAML frontmatter 存在于每个 Wiki 页面
• Obsidian 兼容 —— Wiki 目录可直接作为 Obsidian 仓库使用
• 人类策展，LLM 维护 —— 基本角色分工

我们的修改与新增 Copy Chapter

1. 从命令行到桌面应用 Copy Section

原始设计是一个抽象的模式文档，设计上是复制粘贴给 LLM agent 使用的。我们将其构建为完整的跨平台桌面应用：

• 三栏布局：知识树 / 文件树（左）+ 聊天（中）+ 预览（右）
• 图标侧边栏 —— 在 Wiki、资料源、搜索、图谱、Lint、审核、深度研究、设置之间快速切换
• 自定义可调面板 —— 左右面板支持拖拽调整大小，带最小/最大约束
• 活动面板 —— 实时处理状态，逐文件显示摄入进度
• 全状态持久化 —— 对话、设置、审核项、项目配置在重启后保持
• 场景模板 —— 研究、阅读、个人成长、商业、通用 —— 每个模板预配置 purpose.md 和 schema.md

2. Purpose.md —— Wiki 的灵魂 Copy Section

原始设计有 Schema（Wiki 如何运作），但没有正式定义 为什么 这个 Wiki 存在。我们新增了 purpose.md：

• 定义目标、关键问题、研究范围、演进中的论点
• LLM 在每次摄入和查询时都会读取它以获取上下文
• LLM 可以根据使用模式建议更新
• 与 schema 不同 —— schema 是结构规则，purpose 是方向意图

3. 两步思维链摄入 Copy Section

原始设计描述的是 LLM 同时阅读和写入的单步摄入。我们将其拆分为两次顺序 LLM 调用，显著提升质量：

第一步（分析）：LLM 阅读资料 → 结构化分析
  - 关键实体、概念、论点
  - 与现有 Wiki 内容的关联
  - 与现有知识的矛盾和张力
  - Wiki 结构建议

第二步（生成）：LLM 基于分析 → 生成 Wiki 文件
  - 带 frontmatter 的资料摘要（type, title, sources[]）
  - 实体页面、概念页面及交叉引用
  - 更新 index.md、log.md、overview.md
  - 需要人工判断的审核项
  - 深度研究的搜索查询

超越原始设计的摄入增强：

• SHA256 增量缓存 —— 摄入前检查源文件内容哈希，未变更则自动跳过，节省 LLM token 和时间
• 持久化摄入队列 —— 串行处理防止并发 LLM 调用；队列持久化到磁盘，应用重启后自动恢复；失败任务自动重试最多 3 次
• 文件夹导入 —— 递归导入保留目录结构；文件夹路径作为分类上下文传给 LLM（如 "papers > energy" 帮助分类）
• Source 文件夹自动监听 —— 在应用外新增、修改或删除 raw/sources/ 文件时会被自动检测，并复用应用内相同的摄入/删除生命周期
• 队列可视化 —— 活动面板显示进度条、排队/处理中/失败任务，支持取消和重试
• 自动 Embedding —— 开启向量搜索时，新页面摄入后自动生成 embedding
• 来源可追溯 —— 每个生成的 Wiki 页面在 YAML frontmatter 中包含 sources: [] 字段，链接回贡献的原始资料文件
• overview.md 自动更新 —— 全局概要页面在每次摄入后重新生成，反映 Wiki 最新状态
• 保证资料摘要生成 —— 兜底机制确保资料摘要页面始终被创建，即使 LLM 遗漏
• 语言感知生成 —— LLM 按用户配置的语言（中文或英文）响应
• 资料源渐进渲染 —— 大型资料目录会随滚动分批渲染，保持 Sources 页面流畅

4. 知识图谱与关联度模型 Copy Section

原始设计提到了 [[wikilinks]] 用于交叉引用，但没有图分析。我们构建了完整的知识图谱可视化和关联度引擎：

四信号关联度模型：

图谱可视化（sigma.js + graphology + ForceAtlas2）：

• 按页面类型或社区着色节点，按链接数缩放节点大小（√ 缩放）
• 边的粗细和颜色按关联权重变化（绿色=强，灰色=弱）
• 悬停交互：邻居节点保持可见，非邻居变暗，边高亮并显示关联度分数
• 缩放控件（放大、缩小、适应屏幕）
• 位置缓存防止数据更新时布局跳动
• 图例根据着色模式自动切换类型计数或社区信息

5. Louvain 社区检测 Copy Section

原始设计中没有。基于 Louvain 算法（graphology-communities-louvain）自动发现知识聚类：

• 自动聚类 —— 根据链接拓扑发现哪些页面自然归为一组，独立于预定义的页面类型
• 类型 / 社区 一键切换 —— 按页面类型（实体、概念、资料...）或按发现的知识集群着色
• 内聚度评分 —— 每个社区按内部边密度（实际边数 / 可能边数）评分；低内聚社区（< 0.15）标警告
• 12 色调色板 —— 集群之间视觉区分清晰
• 社区图例 —— 显示核心节点标签、成员数和内聚度

6. 图谱洞察 —— 惊奇连接与知识空白 Copy Section

原始设计中没有。系统自动分析图谱结构，呈现可操作的洞察：

惊奇连接：

• 检测意外关联：跨社区边、跨类型链接、边缘↔核心耦合
• 复合惊奇度评分排序最值得关注的连接
• 可消除 —— 标记为已查看后不再重复出现

知识空白：

• 孤立页面（度 ≤ 1）—— 与 Wiki 其余部分缺少连接的页面
• 稀疏社区（cohesion < 0.15，≥ 3 页）—— 内部交叉引用薄弱的知识领域
• 桥接节点（连接 3+ 个集群）—— 维系多个知识领域的关键枢纽页面

交互：

• 点击洞察卡片高亮图谱中对应节点和边；再次点击取消
• 知识空白和桥接节点附带 Deep Research 按钮 —— 触发 LLM 智能主题生成（读取 overview.md + purpose.md 获取领域上下文）
• 研究主题在可编辑确认对话框中展示 —— 用户可修改主题和搜索查询后再启动

7. 优化的查询检索管线 Copy Section

原始设计描述了 LLM 读取相关页面的简单查询。我们构建了支持可选向量搜索的多阶段检索管线：

阶段 1：分词搜索
  - 英文：分词 + 停用词过滤
  - 中文：CJK 二元组分词（每个 → [每个, 个…]）
  - 标题匹配加分（+10 分）
  - 同时搜索 wiki/ 和 raw/sources/

阶段 1.5：向量语义搜索（可选）
  - 通过任意 OpenAI 兼容的 /v1/embeddings 端点生成 embedding
  - 存储在 LanceDB（Rust 后端）中进行快速 ANN 检索
  - 余弦相似度发现即使没有关键词重叠也语义相关的页面
  - 结果合并：增强已有匹配 + 添加新发现

阶段 2：图谱扩展
  - 搜索结果作为种子节点
  - 四信号关联度模型发现相关页面
  - 2 跳遍历带衰减，发现更深层关联

阶段 3：预算控制
  - 可配置上下文窗口：4K → 1M tokens
  - 比例分配：60% Wiki 页面，20% 聊天历史，5% 索引，15% 系统提示
  - 页面按搜索 + 图谱关联度综合分数排序

阶段 4：上下文组装
  - 编号页面附完整内容（非仅摘要）
  - 系统提示包含：purpose.md、语言规则、引用格式、index.md
  - LLM 被指示按编号引用页面：[1]、[2] 等

向量搜索完全可选 —— 默认关闭，在设置中开启，有独立的端点、API Key 和模型配置。关闭时管线 fallback 到分词搜索 + 图谱扩展。基准测试：开启向量搜索后整体召回率从 58.2% 提升至 71.4%。

8. 多对话聊天与持久化 Copy Section

原始设计只有单一查询接口。我们构建了完整的多对话支持：

• 独立聊天会话 —— 创建、重命名、删除对话
• 对话侧边栏 —— 快速切换不同主题
• 逐对话持久化 —— 每个对话保存到 .llm-wiki/chats/{id}.json
• 可配置历史深度 —— 限制作为上下文发送的消息数量（默认：10）
• 引用参考面板 —— 每条回复上可折叠的区域，显示使用了哪些 Wiki 页面，按类型分组并附图标
• 引用持久化 —— 引用的页面直接存储在消息数据中，重启后稳定不变
• 重新生成 —— 一键重新生成最后一条回复（移除最后的助手+用户消息对，重新发送）
• 保存到 Wiki —— 将有价值的回答归档到 wiki/queries/，然后自动摄入提取实体/概念到知识网络

9. Rust 后端 Chat Agent 与 Skills Copy Section

原始设计中没有。聊天现在由 Rust 后端 Agent runtime 驱动，而不是只在浏览器端运行 TypeScript 循环：

• 工具型 Agent —— 可自主选择 Wiki 检索、Source 检索、图谱检索、网页搜索、AnyTXT、workspace 文件工具、已批准的 shell 命令和 Skill 文件读取
• Skill 管理 —— 扫描项目级和用户级 Skill 目录，启用或禁用 Skill，并在每个会话中通过 /skill 补全选择 Skill
• 生成物管理 —— Agent 工具生成的文件统一放在 agent-workspace/ 下，并作为生成物在聊天中展示、预览或打开目录
• 用户交互表单 —— Skill 可以请求单选、多选或自由文本等结构化用户输入，不需要为每个 Skill 硬编码专用界面
• 更安全的执行模型 —— 项目 workspace 内的命令可以顺畅继续执行，外部 shell 命令仍需要明确批准

10. 思维链 / 推理过程展示 Copy Section

原始设计中没有。针对会输出 `` 块的 LLM（DeepSeek、QwQ 等）：

• 流式思维展示 —— 生成中滚动显示 5 行，带透明度渐变
• 默认折叠 —— 生成完成后思维块隐藏，点击展开
• 视觉分离 —— 思维内容以独特样式显示，与主回复分开

11. Markdown 渲染：KaTeX 数学公式与 Mermaid 图表 Copy Section

原始设计中没有。聊天和预览支持更丰富的 Markdown 渲染：

• KaTeX 渲染 —— 行内 $...$ 和块级 $$...$$ 公式通过 remark-math + rehype-katex 渲染
• Milkdown 数学插件 —— 预览编辑器通过 @milkdown/plugin-math 原生渲染数学公式
• 自动检测 —— 裸 \begin{aligned} 等 LaTeX 环境自动补上 $$ 定界符
• Unicode 降级 —— 100+ 符号映射（α, ∑, →, ≤ 等）用于数学块外的简单行内符号
• Mermaid 代码块 —— fenced mermaid 代码块可直接渲染为流程图、时序图等 Mermaid 支持的图表
• 紧凑 Mermaid 错误 —— 语法错误会被收敛到小型错误卡片中，不会把原始解析器输出铺满聊天界面

12. 审核系统（异步人机协作） Copy Section

原始设计建议在摄入时全程参与。我们新增了异步审核队列：

• LLM 在摄入过程中标记需要人工判断的项目
• 预定义操作类型：创建页面、深度研究、跳过 —— 约束操作防止 LLM 凭空生成任意操作
• 摄入时生成搜索查询 —— LLM 预先为每个审核项生成优化的网络搜索查询
• 用户可在方便时处理审核 —— 不阻塞摄入流程

13. 深度研究 Copy Section

原始设计中没有。当 LLM 识别出知识空白时：

• 网络搜索 支持 Tavily、SerpApi 或 SearXNG，查找相关资料并返回完整内容（非截断摘要）
• Provider 独立配置 —— Tavily 和 SerpApi 使用各自 API Key；SerpApi 支持选择搜索引擎，SearXNG 使用实例 URL 和搜索分类
• 多条搜索查询 —— 摄入时由 LLM 生成，针对搜索引擎优化
• LLM 智能主题生成 —— 从图谱洞察触发时，LLM 读取 overview.md + purpose.md 生成领域精准的研究主题和查询（非泛泛关键词）
• 用户确认对话框 —— 研究主题和搜索查询可编辑，确认后才开始研究
• LLM 综合 搜索结果生成 Wiki 研究页面，并交叉引用现有 Wiki
• 思维链展示 —— 综合过程中 `` 块显示为可折叠区域，自动滚动到最新内容
• 自动摄入 —— 研究结果自动进入两步摄入流程，提取实体/概念到 Wiki
• 任务队列 —— 最多 3 个并发任务
• 研究面板 —— 专用侧边面板，动态高度，实时流式进度

14. 浏览器扩展（网页剪藏） Copy Section

原始设计提到了 Obsidian Web Clipper。我们构建了专用 Chrome 扩展（Manifest V3）：

• Mozilla Readability.js 精确提取文章内容（去除广告、导航、侧边栏）
• Turndown.js 将 HTML 转换为 Markdown，支持表格
• 项目选择器 —— 选择剪藏到哪个 Wiki（支持多项目）
• 本地 HTTP API（端口 19827，tiny_http）—— 扩展 ↔ 应用通信
• 自动摄入 —— 剪藏内容自动触发两步摄入流程
• 剪藏监听 —— 每 3 秒轮询新剪藏，自动处理
• 离线预览 —— 即使应用未运行也能显示提取的内容

15. 多格式文档支持 Copy Section

原始设计聚焦于纯文本/Markdown。我们支持保留文档语义的结构化提取：

MinerU 是可选功能。复杂 PDF 可使用 MinerU 云端、官方 Local API 或本地 Pipeline 模式；本地模式无需上传文件，提取的图片会保存到项目管理的 wiki/media 目录。若 MinerU 失败，LLM Wiki 会回退到内置解析器。

16. 文件删除级联清理 Copy Section

原始设计没有删除机制。我们新增了智能级联删除：

• 删除资料文件时同时移除其 Wiki 摘要页面
• 三重匹配 查找相关 Wiki 页面：frontmatter sources[] 字段、资料摘要页面名称、frontmatter 章节引用
• 共享实体保护 —— 链接到多个资料的实体/概念页面仅从其 sources[] 数组中移除被删除的资料，而非删除整个页面
• 索引清理 —— 被移除的页面从 index.md 中清除
• Wiki 链接清理 —— 指向已删除页面的失效 [[wikilinks]] 从其余 Wiki 页面中移除

17. 可配置上下文窗口 Copy Section

原始设计中没有。用户可配置 LLM 接收多少上下文：

• 4K 到 1M tokens 滑块 —— 适配不同 LLM 的能力
• 比例预算分配 —— 更大的窗口按比例获得更多 Wiki 内容
• 60/20/5/15 分配 —— Wiki 页面 / 聊天历史 / 索引 / 系统提示

18. 跨平台兼容 Copy Section

原始设计与平台无关（抽象模式）。我们处理了具体的跨平台问题：

• 路径规范化 —— 统一的 normalizePath() 在 22+ 个文件中使用，反斜杠 → 正斜杠
• Unicode 安全字符串处理 —— 基于字符而非字节的切片（防止中文文件名导致崩溃）
• macOS 关闭隐藏 —— 关闭按钮隐藏窗口（程序后台运行），点击 Dock 图标恢复，Cmd+Q 退出
• Windows/Linux 关闭确认 —— 关闭时弹出确认对话框，防止误操作导致数据丢失
• Tauri v2 —— macOS、Windows、Linux 原生桌面
• GitHub Actions CI/CD —— 自动构建 macOS（ARM + Intel）、Windows（.msi）、Linux（.deb / .AppImage）

19. 其他新增 Copy Section

• 国际化 —— 中英文界面（react-i18next）
• 设置持久化 —— LLM 提供商、API 密钥、模型、上下文大小、语言通过 Tauri Store 保存
• Obsidian 配置 —— 自动生成 .obsidian/ 目录及推荐设置
• Markdown 渲染 —— 带边框的 GFM 表格、代码块、聊天和预览中的 wikilink 处理
• 多 LLM 提供商 —— OpenAI、Anthropic、Google、Ollama、自定义 —— 各有特定的流式传输和请求头
• 可配置 LLM 超时 —— 可针对较慢的本地模型和长任务调整请求超时
• 可配置 Firecrawl —— 支持可选 API Key 和自定义 Base URL，可连接托管或自部署服务
• 可折叠文件侧栏 —— 可收起 Knowledge/Files 导航并保存折叠状态
• 项目维护 —— 支持 ZIP 导入导出迁移和确定性重建 wiki/index.md
• dataVersion 信号 —— 图谱和 UI 在 Wiki 内容变更时自动刷新

技术栈 Copy Chapter

安装 Copy Chapter

预编译二进制文件 Copy Section

从 Releases 下载：

• macOS：.dmg（Apple Silicon + Intel）
• Windows：.msi
• Linux：.deb / .AppImage

从源码构建 Copy Section

# 前置条件：Node.js 20+, Rust 1.70+
git clone https://github.com/nashsu/llm_wiki.git
cd llm_wiki
npm install
npm run tauri dev      # 开发模式
npm run tauri build    # 生产构建

Chrome 扩展 Copy Section

1. 打开 chrome://extensions
2. 启用「开发者模式」
3. 点击「加载已解压的扩展程序」
4. 选择 extension/ 目录
5. 使用 Alt+Shift+L（macOS 为 Command+Shift+L）直接剪藏当前页面；可在 chrome://extensions/shortcuts 中自定义快捷键

快速开始 Copy Chapter

1. 启动应用 → 创建新项目（选择模板）
2. 进入 设置 → 配置 LLM 提供商（API 密钥 + 模型）
3. 可选：在 设置 中配置网络搜索 Provider 和 source 文件夹自动监听
4. 进入 资料源 → 导入文档（PDF、DOCX、MD 等）
5. 观察 活动面板 —— LLM 自动构建 Wiki 页面
6. 使用 聊天 查询你的知识库
7. 浏览 知识图谱 查看关联
8. 查看 审核 处理需要你关注的项目
9. 定期运行 Lint 维护 Wiki 健康度

本地 HTTP API + MCP Server + AI Agent Skill Copy Chapter

LLM Wiki 内置一个本地 HTTP API（监听 http://127.0.0.1:19828，Token 鉴权，仅本机可达），任何外部工具——包括 Claude Code、Codex 这类 AI Agent，或者任意能发 HTTP 请求的脚本——都可以直接查询你的知识库：

• GET /api/v1/health —— 服务状态（无需鉴权）
• GET /api/v1/projects —— 项目列表
• GET /api/v1/projects/{id}/files / files/content —— 读取文件树与内容
• POST /api/v1/projects/{id}/search —— Hybrid 混合检索（关键词 + 向量），返回 mode、tokenHits、vectorHits，每条结果带 vectorScore
• POST /api/v1/projects/{id}/chat —— 非流式 Rust 后端 Agent 聊天接口，返回助手消息、引用、用量和工具事件；支持 Wiki/Source/Web/AnyTXT 检索，mode: "deep" 会扩展证据收集范围
• GET /api/v1/projects/{id}/graph —— Wikilinks 知识图谱
• POST /api/v1/projects/{id}/sources/rescan —— 触发后端重新扫描
• POST /api/v1/projects/{id}/pages/embed —— 为外部创建或更新的单个 wiki/*.md 页面建立向量索引，无需重建整个向量数据库

在 设置 → API + MCP 中开启 API、生成 Token，并按需选择是否允许本机无鉴权访问。

对于兼容 MCP 的客户端，LLM Wiki 还内置了 mcp-server/。执行 npm run mcp:build 构建后，设置 → API + MCP 会展示一份可复制的 MCP 客户端配置，并自动填入当前机器上的真实入口路径。MCP 工具复用同一套 API 能力，因此 Agent 可以直接列出项目、读取文件、执行 Hybrid 检索、查看图谱、触发资料源重新扫描，并调用同一套 Rust 后端 Agent 聊天接口，不需要再手写 HTTP 调用。

一条命令把 AI Agent 接进你的知识库 Copy Section

LLM Wiki 配套的 agent skill 单独维护在另一个仓库。把它装进 Claude Code / Codex / 任意兼容 skills 的 runtime：

npx skills add https://github.com/nashsu/llm_wiki_skill.git --skill llm-wiki

安装完成后，Agent 就能响应 "我的 LLM Wiki 里关于 X 是怎么说的"、"在我的知识库里搜 Y"、"展示我 wiki 图谱里 Z 的邻居"、"重新索引我的资料源" 等请求——直接调用本机运行的 App，默认只读，引用 wiki 页面路径方便你在 App 内核对。

• Skill 仓库：https://github.com/nashsu/llm_wiki_skill
• 触发约束：刻意不会响应"搜我的笔记"/"看我的 Obsidian / Notion / Logseq"这类泛指的请求——只有你明确说 LLM Wiki / 我的 wiki / 我的知识库 时才会被调用。

项目结构 Copy Chapter

my-wiki/
├── purpose.md              # 目标、关键问题、研究范围
├── schema.md               # Wiki 结构规则、页面类型
├── raw/
│   ├── sources/            # 上传的文档（不可变）
│   └── assets/             # 本地图片
├── wiki/
│   ├── index.md            # 内容目录
│   ├── log.md              # 操作历史
│   ├── overview.md         # 全局概要（自动更新）
│   ├── entities/           # 人物、组织、产品
│   ├── concepts/           # 理论、方法、技术
│   ├── sources/            # 资料摘要
│   ├── queries/            # 保存的聊天回答 + 研究
│   ├── synthesis/          # 跨资料分析
│   └── comparisons/        # 并列对比
├── .obsidian/              # Obsidian 仓库配置（自动生成）
└── .llm-wiki/              # 应用配置、聊天历史、审核项

Star History Copy Chapter

许可证 Copy Chapter

本项目基于 GNU 通用公共许可证 v3.0 授权 —— 详见 LICENSE。