## 1. Project Overview & Quickstart (PromtEngineer/research_roadmap) # research_roadmap Open-source repository PromtEngineer/research_roadmap ### Repository Details - **Repository:** [PromtEngineer/research_roadmap](https://github.com/PromtEngineer/research_roadmap) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (PromtEngineer/website) ## File: docs/examples/youtube-embed.md --- --- # YouTube Video Embedding Examples ## Method 1: Using HTML iframe You can embed a YouTube video directly using an HTML iframe. This is the most common method: ```html ``` Replace `VIDEO_ID` with your actual YouTube video ID. For example, if your YouTube URL is `https://www.youtube.com/watch?v=dQw4w9WgXcQ`, the video ID is `dQw4w9WgXcQ`. Here's a live example: ## Method 2: Using Material for MkDocs Admonitions You can combine admonitions with iframes for a nicer presentation: ```markdown !!! video "Video Title" ``` Here's how it looks: !!! video "Sample YouTube Video" ## Method 3: Responsive Embedding For responsive videos that adjust to screen size, you can use this CSS approach: ```html
``` With this CSS in your `docs/stylesheets/extra.css` file: ```css .video-wrapper { position: relative; padding-bottom: 56.25%; /* 16:9 aspect ratio */ height: 0; overflow: hidden; max-width: 100%; } .video-wrapper iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } ``` ## Method 4: Using a Plugin If you prefer using a plugin, you can install the `mkdocs-video` plugin. Add it to your `requirements-doc.txt` file: ``` mkdocs-video ``` Then add it to your `mkdocs.yml` file: ```yaml plugins: - search - mkdocs-video ``` And use it in your Markdown: ```markdown ![type:video](https://www.youtube.com/embed/VIDEO_ID) ``` ## Getting the YouTube Video ID The video ID is the part of the YouTube URL after `v=`. For example: - Full URL: `https://www.youtube.com/watch?v=dQw4w9WgXcQ` - Video ID: `dQw4w9WgXcQ` You can also get the embed URL directly from YouTube by: 1. Going to the video on YouTube 2. Clicking "Share" 3. Clicking "Embed" 4. Copying the iframe code provided --- ## File: docs/writing/posts/agents-architecture.md --- authors: - muhammad categories: - LLMs - Agents comments: true date: 2025-03-28 tags: - agents - llm - architecture - rag --- # A Visual Guide to LLM Agents ## Table of Contents 1. [Introduction to Large Language Models](#introduction-to-large-language-models) 2. [From LLMs to Agents](#from-llms-to-agents) 3. [Core Components of LLM Agents](#core-components-of-llm-agents) 4. [Tools and Augmentation](#tools-and-augmentation) 5. [Agent Planning and Reasoning](#agent-planning-and-reasoning) 6. [Agent Memory Systems](#agent-memory-systems) 7. [Advanced Agent Architectures](#advanced-agent-architectures) 8. [Multi-Agent Systems](#multi-agent-systems) 9. [Building and Deploying Agents](#building-and-deploying-agents) 10. [Future Directions](#future-directions) ## Introduction to Large Language Models Before diving into agents, we need to understand what Large Language Models (LLMs) are and how they function. LLMs are sophisticated neural networks trained on vast amounts of text data to understand and generate human-like text. These models have evolved from simple statistical approaches to complex architectures based primarily on the Transformer architecture introduced in 2017. ### How LLMs Work At their core, LLMs predict the next token (word or subword) in a sequence based on the context provided. The basic architecture consists of: ```mermaid flowchart TD A[Input Text] --> B[Tokenization] B --> C[Embedding Layer] C --> D[Transformer Layers] D --> E[Output Layer] E --> F[Generated Text] style A fill:#f9f9f9,stroke:#333,stroke-width:1px style B fill:#e6f3ff,stroke:#333,stroke-width:1px style C fill:#e6f3ff,stroke:#333,stroke-width:1px style D fill:#cce5ff,stroke:#333,stroke-width:1px style E fill:#e6f3ff,stroke:#333,stroke-width:1px style F fill:#f9f9f9,stroke:#333,stroke-width:1px ``` - **Tokenization**: Converting input text into tokens - **Embedding Layer**: Transforming tokens into vector representations - **Transformer Layers**: Processing these vectors through attention mechanisms - **Output Layer**: Generating probability distributions for the next token Most modern LLMs use the transformer architecture, which employs self-attention mechanisms to weigh the importance of different words in context. ### Capabilities and Limitations of Traditional LLMs **Capabilities:** - Text generation across various domains - Understanding context and nuance - Adapting to different writing styles - Performing various language tasks without task-specific training **Limitations:** - No ability to access or verify external information beyond training data - No capability to take actions in the world - Limited understanding of temporal context (when events occurred) - No persistent memory between sessions - No ability to use tools or APIs - Risk of hallucinations (generating false information) These limitations highlight why moving from passive LLMs to active agents is necessary for more complex applications. ## From LLMs to Agents The transition from passive language models to active agents is fundamental to understanding LLM agents. ### What Defines an Agent? As defined by Russell & Norvig, **"an agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators."** This definition introduces two critical components missing in standard LLMs: 1. **Perception**: The ability to sense the environment 2. **Action**: The ability to affect the environment ### The Agent Framework An agent-based framework adapts this definition to work with LLMs: ```mermaid flowchart LR subgraph LLM["Traditional LLM"] A[Input Prompt] --> B[Text Generation] B --> C[Output Text] end subgraph Agent["LLM Agent"] D[Environment] --> E[Perception] E --> F[Reasoning/Planning] F --> G[Tool Use] G --> H[Actions] H --> D I[Memory] --> F F --> I end LLM --> Agent style LLM fill:#f5f5f5,stroke:#333,stroke-width:1px style Agent fill:#e6f3ff,stroke:#333,stroke-width:2px ``` - **Environment**: The context in which the agent operates (could be a chat interface, document, or digital environment) - **Perception**: Input methods like prompts, document content, or API responses - **Reasoning**: Internal processing using the LLM to decide what to do - **Action**: Outputs that affect the environment (generating text, calling functions, using tools) - **Memory**: Retaining information across interactions This framework transforms a passive text prediction system into an entity that can intelligently interact with its surroundings. ## Core Components of LLM Agents Let's explore the essential components that make up an LLM agent: ```mermaid flowchart LR %% Central node with dashed border LLM["Large Language\nModel"] %% Main components with their subgraphs subgraph Tools[" "] T[Tool Use] --> T1[Function Calling] T --> T2[API Integration] T --> T3[Database Access] end subgraph Reasoning[" "] R[Reasoning] --> R1[Task Decomposition] R --> R2[Chain-of-Thought] R --> R3[Decision-Making] end subgraph Perception[" "] P[Perception] --> P1[Text Inputs] P --> P2[Document Understanding] P --> P3[Multimodal Inputs] end subgraph Memory[" "] M[Memory] --> M1[Short-Term] M --> M2[Long-Term] M --> M3[Episodic & Semantic] end %% Connect LLM to main components LLM --> T LLM --> R LLM --> P LLM --> M %% Style nodes style LLM fill:#f5f5f5,stroke:#333,stroke-width:1px,stroke-dasharray:5 5 %% Tool styles style T fill:#fd7e14,stroke:#333,stroke-width:2px style T1 fill:#ffe8cc,stroke:#333,stroke-width:1px style T2 fill:#ffe8cc,stroke:#333,stroke-width:1px style T3 fill:#ffe8cc,stroke:#333,stroke-width:1px %% Reasoning styles style R fill:#40c057,stroke:#333,stroke-width:2px style R1 fill:#d3f9d8,stroke:#333,stroke-width:1px style R2 fill:#d3f9d8,stroke:#333,stroke-width:1px style R3 fill:#d3f9d8,stroke:#333,stroke-width:1px %% Perception styles style P fill:#4dabf7,stroke:#333,stroke-width:2px style P1 fill:#d0ebff,stroke:#333,stroke-width:1px style P2 fill:#d0ebff,stroke:#333,stroke-width:1px style P3 fill:#d0ebff,stroke:#333,stroke-width:1px %% Memory styles style M fill:#ae3ec9,stroke:#333,stroke-width:2px style M1 fill:#f3d9fa,stroke:#333,stroke-width:1px style M2 fill:#f3d9fa,stroke:#333,stroke-width:1px style M3 fill:#f3d9fa,stroke:#333,stroke-width:1px ``` ### Environment Perception Agents need to understand their environment through various inputs: - **Text Input**: The most basic form of perception through prompts - **Document Understanding**: Processing and understanding documents - **Structured Data**: Working with databases, APIs, and structured information - **Multimodal Input**: Processing images, audio, or other data types (in advanced agents) ### Planning and Reasoning An agent must plan its actions and reason about the best course of action: - **Task Decomposition**: Breaking complex tasks into manageable steps - **Chain-of-Thought**: Working through problems step-by-step - **Decision-Making**: Evaluating options and selecting the best course of action - **Self-Reflection**: Evaluating its own reasoning and outputs ### Tool Usage and Integration A defining characteristic of LLM agents is their ability to use tools: - **Function Calling**: Identifying when to call an external function - **API Integration**: Connecting to external services through APIs - **Code Execution**: Running code to perform calculations or manipulate data - **Database Access**: Retrieving or storing information in databases ### Memory Systems Agents require memory to maintain context and learn from past interactions: ```mermaid flowchart TB %% Short-term Memory Section subgraph ShortTerm["Short-Term Memory"] direction LR CH["Conversation History"] --- CD["Current Session Data"] --- AT["Active Task State"] end %% Long-term Memory Section subgraph LongTerm["Long-Term Memory"] direction LR VD["Vector Database"] --- KG["Knowledge Graph"] --- DD["Document Store"] --- UP["User Profiles"] end %% Memory Types Section subgraph MemTypes["Memory Types"] direction LR EM["Episodic Memory
(Specific Interactions)"] --- SM["Semantic Memory
(General Knowledge)"] --- PM["Procedural Memory
(How to Perform Tasks)"] end %% Memory Retrieval Section subgraph Retrieval["Memory Retrieval"] direction LR SS["Semantic Search"] --- TF["Temporal Filtering"] --- SR["Relevance Ranking"] --- CR["Contextual Retrieval"] end %% Decision Making DM["Agent Decision Making"] %% Connections ShortTerm --> MemTypes LongTerm --> MemTypes MemTypes --> Retrieval Retrieval --> DM %% Styling style ShortTerm fill:#fff7e6,stroke:#333,stroke-width:1px style LongTerm fill:#e6ffe6,stroke:#333,stroke-width:1px style MemTypes fill:#e6f7ff,stroke:#333,stroke-width:1px style Retrieval fill:#ffe6e6,stroke:#333,stroke-width:1px style DM fill:#f0f0ff,stroke:#333,stroke-width:1px %% Node Styles classDef default fill:#fff,stroke:#333,stroke-width:1px ``` - **Short-Term Memory**: Recent conversation history - **Long-Term Memory**: Persistent information stored across sessions - **Episodic Memory**: Specific sequences of interactions - **Semantic Memory**: General knowledge and facts These core components transform an LLM into an agent capable of complex, goal-oriented behavior. ## Tools and Augmentation Tools and augmentation techniques enhance the capabilities of LLM agents beyond their built-in knowledge. ### Types of Tools Modern LLM agents can leverage various tools: - **Search Tools**: Accessing up-to-date information from the internet - **Calculators**: Performing precise mathematical operations - **Knowledge Bases**: Accessing specific domain knowledge - **Code Interpreters**: Executing and debugging code - **Database Interfaces**: Querying and manipulating structured data - **API Connectors**: Interacting with external services and platforms ### Retrieval Augmented Generation (RAG) RAG is a powerful technique that combines retrieval of information with text generation: ```mermaid flowchart TD Q[Query/Question] --> E[Embedding Model] E --> VS[Vector Search] subgraph Indexing["Document Indexing (Pre-processing)"] D[Documents] --> DC[Document Chunking] DC --> DE[Document Embedding] DE --> VDB[Vector Database] end VS --> VDB VDB --> RD[Retrieved Documents] Q --> P[Prompt Construction] RD --> P P --> LLM[Large Language Model] LLM --> A[Augmented Response] style Indexing fill:#e6f3ff,stroke:#333,stroke-width:1px style Q fill:#f9f9f9,stroke:#333,stroke-width:1px style P fill:#f9f9f9,stroke:#333,stroke-width:1px style LLM fill:#cce5ff,stroke:#333,stroke-width:2px style A fill:#f9f9f9,stroke:#333,stroke-width:1px ``` 1. **Indexing**: Documents are processed, chunked, and stored in a vector database 2. **Retrieval**: When a query is received, relevant documents are retrieved 3. **Augmentation**: Retrieved content is added to the prompt 4. **Generation**: The LLM generates a response based on both the query and the retrieved information RAG enhances accuracy by grounding responses in specific knowledge sources, reducing hallucinations and improving factual accuracy. ### Function and API Integration Function calling allows agents to interact with the world: ```mermaid sequenceDiagram participant User participant LLM participant Function User->>LLM: Query (e.g., "What's the weather in New York?") LLM->>LLM: Recognize need for external data LLM->>Function: Call function
(get_weather, {location: "New York"}) Function->>LLM: Return data (Temperature: 72°F, Condition: Sunny) LLM->>User: Generate response with function data
"The current weather in New York is sunny with a temperature of 72°F." note over LLM,Function: Modern LLMs can determine when to
call functions and structure the
appropriate parameters ``` 1. **Function Definition**: Functions are defined with names, descriptions, and parameter specifications 2. **Function Detection**: The LLM detects when a function should be called based on the user's query 3. **Parameter Generation**: The LLM generates the appropriate parameters 4. **Function Execution**: The function is executed, and results are returned 5. **Response Integration**: The LLM incorporates the function results into its response This capability enables agents to perform actions like checking the weather, booking appointments, or processing payments. ## Agent Planning and Reasoning Effective planning and reasoning are crucial for complex tasks. ### Prompt Engineering for Agents Agent prompts typically include: - **System Instructions**: Defining the agent's role and capabilities - **Available Tools**: Descriptions of tools the agent can use - **Constraints**: Limitations on the agent's actions - **Output Format**: How the agent should structure its responses - **Examples**: Demonstrations of expected behavior ### Chain-of-Thought (CoT) Reasoning CoT enables an agent to work through problems step-by-step: 1. **Problem Analysis**: Understanding the task and breaking it down 2. **Intermediate Steps**: Working through each step logically 3. **Reflection**: Checking the reasoning at each step 4. **Solution**: Arriving at the final answer based on the steps This approach significantly improves performance on complex reasoning tasks. ### ReAct Framework ReAct (Reasoning + Acting) interleaves thinking and action: ```mermaid sequenceDiagram participant User participant Agent participant Tools as External Tools User->>Agent: Task or Query loop Until task completion Agent->>Agent: Thought: Reasoning about next step Agent->>Agent: Action: Decide which tool to use Agent->>Tools: Call appropriate tool Tools->>Agent: Observation: Return result Agent->>Agent: Thought: Process observation end Agent->>User: Final response note over Agent: ReAct interleaves reasoning (thoughts)
with actions and observations in a cycle ``` 1. **Reasoning**: The agent thinks about what it needs to do 2. **Action**: The agent takes action using available tools 3. **Observation**: The agent observes the results of its action 4. **Continued Reasoning**: The agent incorporates observations into its reasoning This cycle continues until the task is complete, enabling dynamic, adaptive problem-solving. ## Agent Memory Systems Memory systems enable agents to maintain context and learn from past interactions. ```mermaid flowchart TD subgraph SM["Short-Term Memory"] SM1["Conversation History"] SM2["Current Session Data"] SM3["Active Task State"] end subgraph LM["Long-Term Memory"] LM1["Vector Database"] LM2["Knowledge Graph"] LM3["Document Store"] LM4["User Profiles"] end subgraph MT["Memory Types"] MT1["Episodic Memory
(Specific Interactions)"] MT2["Semantic Memory
(General Knowledge)"] MT3["Procedural Memory
(How to Perform Tasks)"] end subgraph MR["Memory Retrieval"] MR1["Semantic Search"] MR2["Temporal Filtering"] MR3["Relevance Ranking"] MR4["Contextual Retrieval"] end SM --> MT LM --> MT MT --> MR MR --> A[Agent Decision Making] style SM fill:#ffffcc,stroke:#333,stroke-width:1px style LM fill:#ccffcc,stroke:#333,stroke-width:1px style MT fill:#ccffff,stroke:#333,stroke-width:1px style MR fill:#ffcccc,stroke:#333,stroke-width:1px ``` ### Short-Term Context Short-term or working memory includes: - **Conversation History**: The recent exchanges between user and agent - **Current Session Data**: Information gathered during the current interaction - **Active Task State**: The current progress on the task being performed These elements are typically handled through context window management. ### Long-Term Memory Storage Long-term memory enables persistent information storage: - **Vector Databases**: Storing semantic representations of past conversations - **Knowledge Graphs**: Structured representations of entities and relationships - **Document Stores**: Persistent storage of important information - **User Profiles**: Preferences and patterns specific to individual users ### Episodic vs. Semantic Memory Agents can implement different types of memory: - **Episodic Memory**: Specific sequences of interactions (e.g., "Last time we discussed home renovation options") - **Semantic Memory**: General knowledge and facts (e.g., "The user prefers minimalist design") ### Memory Retrieval Strategies Effective retrieval is critical for using stored information: - **Semantic Search**: Finding relevant information based on meaning - **Temporal Filtering**: Retrieving information based on when it was stored - **Relevance Ranking**: Prioritizing the most important information - **Contextual Retrieval**: Finding information relevant to the current context A well-designed memory system allows agents to build on past interactions and provide more personalized experiences. ## Advanced Agent Architectures As agents become more sophisticated, their architectures evolve to handle more complex tasks. ### Task Decomposition Complex task handling requires sophisticated decomposition: 1. **Goal Analysis**: Understanding the overall objective 2. **Subtask Identification**: Breaking down the goal into manageable parts 3. **Dependency Mapping**: Determining the order of subtasks 4. **Resource Allocation**: Assigning appropriate tools to each subtask This approach enables agents to tackle problems too complex to solve all at once. ### Self-Reflection and Self-Correction Advanced agents can evaluate and improve their own outputs: 1. **Output Generation**: Producing an initial response 2. **Self-Critique**: Identifying potential issues or improvements 3. **Refinement**: Revising the response based on self-critique 4. **Verification**: Checking the improved response against requirements This recursive improvement process enhances accuracy and quality. ### Verification of Outputs Ensuring reliability through verification: - **Fact-Checking**: Verifying factual claims against reliable sources - **Consistency Checks**: Ensuring internal consistency in responses - **Hallucination Detection**: Identifying when the agent is generating unfounded information - **Confidence Scoring**: Assessing the reliability of different parts of a response ### Meta-Prompting and Prompt Chaining Sophisticated prompting techniques: - **Meta-Prompting**: Using the LLM to generate or refine its own prompts - **Prompt Chaining**: Connecting multiple prompts in sequence to handle complex workflows - **Adaptive Prompting**: Modifying prompts based on user responses or task progress These techniques allow for more flexible and powerful agent behaviors. ## Multi-Agent Systems Multiple specialized agents can collaborate to solve complex problems. ```mermaid flowchart TD U[User] --> C[Coordinator Agent] C --> P[Planner Agent] C --> R[Researcher Agent] C --> E[Expert Agent] C --> CR[Critic Agent] P --> C R --> C E --> C CR --> C C --> U subgraph Communication P <-.-> R R <-.-> E E <-.-> CR P <-.-> CR end style U fill:#f9f9f9,stroke:#333,stroke-width:1px style C fill:#ffcc99,stroke:#333,stroke-width:2px style P fill:#ccffcc,stroke:#333,stroke-width:1px style R fill:#ccffcc,stroke:#333,stroke-width:1px style E fill:#ccffcc,stroke:#333,stroke-width:1px style CR fill:#ccffcc,stroke:#333,stroke-width:1px style Communication opacity:0.2 ``` ### Agent Collaboration Models Different models for agent collaboration: - **Hierarchical**: Supervisor agents coordinate specialized worker agents - **Peer-to-Peer**: Agents communicate directly with each other - **Market-Based**: Agents bid for tasks based on their capabilities - **Consensus-Based**: Agents work together to reach agreement on solutions ### Specialized Agent Roles Multi-agent systems often feature specialized roles: - **Planner**: Designs overall strategy and breaks down tasks - **Researcher**: Gathers information from various sources - **Expert**: Provides domain-specific knowledge and analysis - **Critic**: Evaluates and improves outputs - **Coordinator**: Manages communication between agents ### Communication Protocols Effective inter-agent communication requires: - **Message Formats**: Structured formats for exchanging information - **Dialogue Management**: Tracking conversation state between agents - **Knowledge Sharing**: Methods for sharing relevant information - **Conflict Resolution**: Mechanisms for resolving disagreements ### Consensus Mechanisms When agents must agree on a course of action: - **Voting**: Simple majority or weighted voting schemes - **Debate**: Agents present arguments and counter-arguments - **Evidence Evaluation**: Assessing the quality of evidence presented - **Meta-Evaluation**: Using another agent to evaluate competing proposals Multi-agent systems enable more complex problem-solving than any single agent could achieve alone. ## Building and Deploying Agents Practical considerations for implementing LLM agents. ### Frameworks and Libraries Popular tools for building agents: - **LangChain**: Framework for building language model applications - **LlamaIndex**: Tools for connecting LLMs to external data - **AutoGPT**: Autonomous AI agent framework - **Microsoft Semantic Kernel**: Framework for integrating AI with traditional programming ### Evaluation Metrics Assessing agent performance: - **Task Completion Rate**: How often the agent successfully completes tasks - **Efficiency**: Number of steps or time required to complete tasks - **Accuracy**: Correctness of information and actions - **User Satisfaction**: User ratings and feedback - **Hallucination Rate**: Frequency of unfounded claims ### Safety Considerations Important safety measures: - **Action Limitations**: Restricting potentially harmful actions - **User Confirmation**: Requiring approval for significant actions - **Monitoring**: Tracking agent behavior for unexpected patterns - **Transparency**: Making reasoning and sources clear to users ### Deployment Patterns Common approaches to deployment: - **Serverless Functions**: Deploying components as cloud functions - **Containerization**: Packaging agents and dependencies in containers - **API Services**: Exposing agent capabilities through APIs - **Edge Deployment**: Running lightweight agents on edge devices Careful attention to these aspects ensures agents that are effective, reliable, and safe. ## Future Directions The field of LLM agents is rapidly evolving. Here are some emerging trends and challenges: ### Current Limitations Areas needing improvement: - **Reasoning Abilities**: Enhancing logical and causal reasoning - **Tool Creation**: Enabling agents to create new tools as needed - **True Autonomy**: Reducing the need for human oversight - **Cross-Domain Knowledge**: Applying knowledge across different domains - **Efficiency**: Reducing computational requirements ### Research Frontiers Exciting areas of research: - **Embodied Agents**: Connecting language models to robotic systems - **Multi-Modal Agents**: Integrating text, vision, audio, and other modalities - **Continual Learning**: Agents that learn and improve through interaction - **Collective Intelligence**: Emergent capabilities from agent collaboration - **Neural-Symbolic Approaches**: Combining neural networks with symbolic reasoning ### Potential Applications Promising applications for advanced agents: - **Personalized Education**: Tutors adapted to individual learning styles - **Scientific Discovery**: Agents that generate and test hypotheses - **Healthcare Assistance**: Diagnostic and treatment planning support - **Creative Collaboration**: Partners for writing, design, and other creative tasks - **Autonomous Systems**: Self-directed systems that adapt to changing conditions ### Ethical Considerations Important ethical questions: - **Transparency**: Ensuring users understand agent capabilities and limitations - **Accountability**: Determining responsibility for agent actions - **Privacy**: Protecting sensitive information used by agents - **Bias**: Addressing biases in training data and reasoning - **Human Augmentation**: Enhancing rather than replacing human capabilities The future of LLM agents will depend on thoughtful approaches to these challenges and opportunities. ## Conclusion LLM agents represent a significant evolution in artificial intelligence, transforming passive language models into active, capable assistants. By combining the language understanding of LLMs with the ability to perceive, reason, and act, these agents can solve increasingly complex problems and provide more valuable assistance. As the technology continues to develop, we can expect agents to become more autonomous, capable, and integrated into our daily lives and work. The journey from simple language models to sophisticated agents is just beginning, with many exciting possibilities on the horizon. The most successful approaches will likely be those that thoughtfully combine the strengths of artificial and human intelligence, creating systems that augment human capabilities rather than simply attempting to replace them. --- ## File: docs/writing/index.md --- categories: - Entrepreneurship comments: true personal stories and expert tips. tags: - consulting - AI - open source - writing - RAG --- # Deeper Dives: Thoughts on AI, Code, and Building Better Systems This space gathers my thoughts and experiences at the intersection of AI consulting, open-source development, and the practical application of large language models (LLMs). Expect insights, reflections, and technical deep dives based on my ongoing work. Find this valuable? Subscribe to my newsletter (max twice a month) for curated updates on AI trends, tech insights, and practical tips for building impactful solutions. [:material-email: Subscribe to my Newsletter](https://tally.so/r/3y9bb0){.md-button .md-button--secondary} [:material-twitter: Follow me on X](https://twitter.com/engineerprompt){.md-button .md-button--secondary} For posts about RAG (Retrieval-Augmented Generation) or LLMs (Large Language Models), check out the category labels in the sidebar. Here are some of my best posts on these topics: ## RAG and LLM Expertise - [Agents and Architecture](./posts/agents-architecture.md): Understanding the components of LLM Agents ## Talks and Interviews - [Building AI Agents That work](https://youtu.be/UPqMdszWHy0): My Google NEXT Talk --- ## File: docs/404.md --- title: Page not found --- # 404 - Page Not Found Sorry, the page you were looking for doesn't exist. ## What might have happened? - The page might have been moved or deleted - You might have followed a broken link - There might be a typo in the URL ## What can you do now? - [Go back to the homepage](./index.md) - Use the search function at the top of the page - Check out some of my popular content: - [RAG Beyond Basics Course](./rag-beyond-basics.md) - [Strategic AI Consulting Services](./services.md) - [Expert AI Video Library](./youtube.md) If you believe this is an error, please [contact me via email](mailto:engineerprompt@gmail.com) or [reach out on X](https://x.com/engineerrprompt). --- ## File: docs/book-a-call.md --- --- # Let's Turn Your AI Ambitions Into Real-World Results Struggling with AI implementation, scaling machine learning systems, or making sense of where to begin? With over a decade of experience helping startups and enterprises alike, I'll help you cut through the noise and create a clear, strategic path forward. ## What You'll Get From Our 45-Minute Consultation This isn't a sales call. It's a focused session designed to unlock clarity, fast. 1. **Current State Assessment** — Understand where you are and what's getting in your way 2. **Opportunity Discovery** — Identify the most impactful areas where AI can move the needle 3. **Custom Roadmap** — Walk away with a tailored game plan and clear next steps 4. **Expert Insights** — Get candid answers to your questions about strategy, tools, or implementation ## Book Instantly. No Back-and-Forth. [:material-calendar: Schedule Your Call →](https://calendly.com/engineerprompt/consulting-call){.md-button .md-button--secondary} --- ## File: docs/index.md --- --- # Turn AI Complexity Into Business Advantage ## Independent AI Consultant & Google Developer Expert helping businesses build high-impact AI solutions With over a decade of experience in machine learning systems and a proven track record of transforming ideas into measurable results. [:material-calendar: Book a Consultation](./book-a-call.md){.md-button .md-button--secondary} [:octicons-info-16: Explore My Services](./services.md){.md-button .md-button--secondary} # 👋 I'm Muhammad — Your AI Expert I'm an independent consultant, advisor, and Google Developer Expert with over a decade of hands-on experience in building and scaling machine learning systems. I partner with businesses like yours to turn complex AI challenges into clear, actionable strategies that deliver tangible results—boosting efficiency, unlocking new opportunities, and driving growth. Whether you're navigating the ever-evolving landscape of Generative AI or trying to unlock the full potential of LLMs, I can guide you from ideation to execution—with a focus on measurable results. Before diving into consulting, I earned a Ph.D. in Applied Machine Learning, specializing in wearable sensors and biomedical signal processing. You can browse my academic research and publications on [Google Scholar](https://scholar.google.com/citations?user=vQcpPPAAAAAJ&hl=en). I'm also deeply committed to open-source and community-driven learning. I created [localGPT](https://github.com/PromtEngineer/localGPT), one of the first open-source Retrieval-Augmented Generation (RAG) projects for secure, private document interaction—now with over **20k GitHub ⭐**. On my [YouTube channel](https://www.youtube.com/@engineerprompt) with ***199k+ subscribers***, I regularly share practical insights on LLMs, Generative AI, Agents, and more. Join the conversation on [Discord](https://discord.gg/HuvDMXfCAf), where 4,500+ members exchange ideas, ask questions, and build cool stuff together. --- Ready to bring AI into your business in a meaningful way? --- ## Consulting I help startups and enterprise teams build high-impact machine learning systems, adapt quickly to emerging AI tools, and implement strategies that drive tangible business outcomes. My focus areas include: - **AI Strategy & Roadmapping:** Aligning AI initiatives with your core business goals. - **System Implementation & Optimization:** Building and refining ML systems for performance and efficiency. - **Team Enablement & Training:** Upskilling your team to leverage AI effectively. Learn more about how we can tailor a solution for your specific needs. [:octicons-info-16: Explore My Services](./services.md){.md-button .md-button--secondary} --- ### AI Team Training Programs Want to build lasting AI expertise within your organization? My custom training programs equip your team with practical skills to: - **Master prompt engineering:** Get consistent, high-quality results from LLMs. - **Implement advanced RAG:** Build powerful, context-aware AI applications. - **Integrate AI for developers:** Seamlessly embed AI into your existing products and pipelines. - **Champion AI ethics & governance:** Build responsible, trustworthy AI systems. Each session is customized based on your team's technical background and the specific challenges your business is facing, ensuring immediate applicability and impact. --- ## Connect With Me Stay connected and get the latest updates by following me on [:fontawesome-brands-youtube: YouTube](https://www.youtube.com/@engineerprompt) and [:fontawesome-brands-twitter: Twitter](https://twitter.com/engineerrprompt). --- ## File: docs/rag-beyond-basics.md --- --- # RAG Beyond Basics: Master AI-Powered Document Intelligence ## Transform How You Extract Value From Documents: Build Professional-Grade RAG Systems That Deliver Precise, Contextual Answers At Scale ⭐ Master Both The "How" AND The "Why" Behind Advanced RAG Systems Through Hands-On Implementation ⭐ ## 👀 Why This Course Is A Game-Changer Retrieval-Augmented Generation (RAG) is revolutionizing how businesses interact with their document repositories. Instead of spending hours searching through files, imagine instantly extracting precise insights through natural conversation—unlocking document intelligence that drives real business decisions. 📈📚 This comprehensive course bridges theory with hands-on implementation, taking you beyond basic RAG tutorials into professional-grade system architecture. You'll master both commercial APIs and fully private, on-premise solutions, giving you the flexibility to build systems that meet any security or customization requirement. [:material-school: Enroll Now](https://prompt-s-site.thinkific.com/courses/rag){.md-button .md-button--secondary} ## 👩‍💻 Who Will Get Transformative Value - ✅ **SaaS Founders & Product Leaders**: Transform document-heavy workflows into competitive advantages and create AI features users will pay premium prices for - ✅ **ML & AI Engineers**: Skip months of trial-and-error by implementing battle-tested RAG architectures that overcome common challenges - ✅ **Technical Leaders & Architects**: Make informed decisions about AI infrastructure integration and optimize for performance, cost, and security - ✅ **Enterprise Innovation Teams**: Build proof-of-concepts that demonstrate immediate business value from your untapped document repositories The only prerequisite is basic Python knowledge—everything else is covered step-by-step. Familiarity with LangChain or Streamlit is helpful but not essential. ## Key Topics & Advanced Techniques Covered Go beyond basic tutorials and master the techniques needed for production-ready RAG: - **Advanced Query & Retrieval Strategies:** Implement Query Expansion, multi-model Re-ranking (GPT-4, ColBERT, Cohere), Hypothetical Document Embeddings (HyDE), and Ensemble Retrieval methods. - **Optimized Document Processing:** Learn Hierarchical Chunking and Parent Document Retrieval for richer context and improved answer accuracy. - **Practical Application Building:** Construct and deploy a fully interactive GUI RAG application using Streamlit. ## 📚 Learning Architecture: Theory + Implementation + Application Each module follows a proven learning pattern that maximizes both understanding and practical skill development: 1. **Conceptual Foundation**: Clear explanation of why specific techniques matter and how they solve real problems 2. **Implementation Deep-Dive**: Hands-on coding with step-by-step guidance and complete access to source code 3. **Real-World Application**: Apply what you've learned to actual document sets and see immediate results By the final module, you'll have built a production-ready RAG system that you fully understand and can confidently customize for any domain-specific challenge. [:material-school: Get Access Now](https://prompt-s-site.thinkific.com/courses/rag){.md-button .md-button--secondary} --- ## File: docs/services.md --- --- # Strategic AI Consulting Services ## Unlock the Power of AI for Your Business Whether you're just starting to explore AI or looking to scale advanced capabilities, I offer customized solutions that align with your goals and deliver real business impact. My core service areas focus on: - **AI Strategy & Integration:** Developing clear roadmaps and integrating AI to achieve strategic business objectives, from market leadership to operational excellence. - **Custom AI Solutions & Optimization:** Building, implementing, and refining bespoke AI systems to enhance performance, automate workflows, and boost productivity. - **AI Team Enablement:** Equipping your technical teams and leadership with the practical skills and strategic insights needed to leverage AI effectively through tailored training and coaching. [Schedule a Consultation](./book-a-call.md){.md-button .md-button--secondary} ## Our Collaborative Process We follow a clear, outcome-focused approach: 1. **Discovery & Assessment:** Deep dive into your goals, tech stack, and challenges. 2. **Strategic Solution Design:** Co-create a practical, scalable plan using the best-fit tools. 3. **Implementation & Delivery:** Move efficiently from prototype to production with hands-on support. 4. **Iteration & Scaling:** Refine and expand the solution to ensure long-term value. ## Flexible Engagement Options Every organization is different—so I offer flexible ways to work together depending on your goals, timeline, and internal capabilities: - **Project-Based Engagements:** For well-defined AI initiatives with clear objectives and deliverables, including strategic assessments, implementation support, and custom solution development. - **Advisory Retainers:** Ongoing partnership providing monthly strategic input, technical guidance, and on-call support to keep your AI initiatives on track. - **Custom Workshops & Training:** Tailored sessions, from quick workshops to multi-day deep dives, designed to upskill your team based on their specific needs. - **Strategy Intensives:** Focused, short-term engagements (1-2 days) designed for rapid clarity, prioritization, and direction on specific AI challenges or opportunities. Pricing is tailored to project scope and objectives. Let's start with a discovery call to build a plan that fits your needs and budget. [:material-email: Contact Me](mailto:engineerprompt@gmail.com){.md-button .md-button--secondary} | [:material-calendar: Schedule a Consultation](./book-a-call.md){.md-button .md-button--secondary} --- ## File: docs/youtube.md --- --- # Expert AI Video Library Access my curated collection of in-depth technical tutorials, strategic insights, and practical implementation guides that have helped over 193,000 professionals master AI, LLMs, and Generative AI. Each video distills complex concepts into actionable knowledge you can apply immediately. [:fontawesome-brands-youtube: Join 199k+ Subscribers](https://www.youtube.com/@engineerprompt){.md-button .md-button--secondary} ## Retrieval Augmented Generation Tutorials RAG

Local LightRAG: A GraphRAG Alternative but Fully Local with Ollama

In this video, we explore how to set up and run LightRAG—a retrieval augmented generation (RAG) system that combines knowledge graphs with embedding-based retrieval—locally using OLLAMA. . Agents

RAG with reasoning models

This video explores how reasoning models can help your RAG pipelines. Reasoning model can help with reranking of the returned context without the need for external reranker models. Agents

Graph RAG: Improving RAG with Knowledge Graphs

Discover Microsoft’s groundbreaking GraphRAG, an open-source system combining knowledge graphs with Retrieval Augmented Generation to improve query-focused summarization. I’ll guide you through setting it up on your local machine, demonstrate its functions, and evaluate its cost implications. LLM Training

Multimodal RAG with Vision Models

Learn how to create an end to end multimodal RAG pipeline without the need of parsing your docs, chunking, embedding etc. This technique enables you to feed image of your docs (PDF pages) and create a multi-vector represetnation with techniques like ColPali and direclty feed your returned pages in to a Vision Language Model. --- ## File: farooq-site/src/content/writing/agent-context-management.md --- title: 'How agent harnesses manage context: cap, slice, search, store' image: '/og/agent-context-management.png' date: 2026-06-11 tags: ['agents', 'harness engineering'] --- Context management is how an agent harness decides what the model sees on each turn. It matters because the context window is too small for everything the model might want to see, and a full window makes the agent worse. You can watch this happen: an agent works well for twenty minutes, reads one big file, and gets noticeably dumber. It repeats itself, forgets a decision it made earlier, and re-reads things it already saw. Nothing about the model changed. The window filled up, and the model stopped reasoning about the task and started swimming through its own transcript. Modern harnesses treat context as memory that has to be managed, not as a passive chat log. In [the harness anatomy essay](/writing/what-is-an-agent-harness/) I listed context management as component two of nine. In this post I want to go deeper: what occupies the window, the four moves harnesses make when content doesn't fit, and how four real harnesses handle the same stress test. ## What fills the window The window holds more than the conversation. On any given turn it contains: - **Fixed context.** The system prompt, tool definitions, policies, and skills. Loaded once at session start. - **Conversation history.** Grows every turn, without bound. - **File contents.** A single read can be ten thousand, fifty thousand, sometimes hundreds of thousands of tokens. Often the largest single occupant. - **Tool results.** Grep, bash, search, database output. Every call writes another object into the transcript, so these accumulate fast. - **Sub-agent responses, summaries, and memory state.** Forked task output and compaction artifacts. Almost everything on that list grows, and the window doesn't. All of it competes with the one budget you can't touch: room for the model to answer.
The window fills from all sides. The model sees only what the harness lets through.
The harness ends up acting like an operating system. It decides what stays close to the model, what gets compressed, what gets paged out, and what can be retrieved later. ## How harnesses handle large files The clearest test case is a file that's bigger than the available context. When the model asks to read one, somebody has to decide what happens, because "just read it" would evict everything else the model knows. For the research behind this post I traced that decision through four harnesses: Pi, OpenClaw, Claude Code, and Letta. None of them just reads the file. The answer is always some combination of four moves: - **Cap it.** Enforce a hard limit on lines or bytes and show the head of the file. - **Slice it.** Let the model page through the rest with offset and limit parameters. - **Search it.** Point the model at grep or semantic search instead of a full read. - **Store it elsewhere.** Keep the full content on disk or in a vector store and show only a managed view. Each harness mixes the moves differently: - **Pi** is harness-first and simple. File reads are capped at 2,000 lines or 50 KB. The model sees the beginning of the file plus an explicit continuation nudge: use offset and limit to continue. - **OpenClaw** layers defense on top of the same idea. It keeps Pi-style truncation, adds character budgets for bootstrap files, caps tool results, and uses head-plus-tail truncation when the end of the output looks important. - **Claude Code** is the most aggressive. It checks file size before opening. If the file is too large, the read is rejected outright and the model is pointed toward offset, limit, or grep. After a successful read it token-counts the output, truncates long lines, deduplicates repeated reads of the same file, and can tune all of these limits remotely. - **Letta** takes a different path. It parses, chunks, and embeds files into a vector store. The model gets direct viewing, exact search, and semantic search, and the window only ever shows a managed view of the file.
Four harnesses, four mixes of cap, slice, search, and store. None of them dumps the file into the model.
Different architectures, same lesson: file context isn't dumped into the model, it's mediated. ## Tool results are the quiet offender Tool results eat context in a way that's easy to miss. A file announces itself: you know a 200 KB read is a problem the moment you see it. Tool results feel harmless because each one looks useful in isolation. One grep output is fine. One JSON payload is fine. But every call writes another object into the transcript, so a session's worth of grep outputs, logs, and dataframe previews can eat the working set faster than the conversation itself, and no single result ever looked like the culprit. The fix follows the same logic as the file case: stop treating tool outputs as messages and start treating them as artifacts. Oversized results get persisted to disk, and the model sees a small preview plus a pointer to the rest. Repeated previews get deduplicated. Long values get truncated head-and-tail. Search results get summarized, paginated, or capped per tool and per message. Anthropic's [context engineering guidance](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) points the same way: treat context as a finite resource and clear raw tool outputs from history once they've served their purpose. The insight underneath is that the model doesn't need the entire payload in the prompt. It needs enough visible state to decide what to fetch next. ## Old history gets compacted Long sessions add a deeper problem: deciding what old history still matters. When context pressure crosses a threshold, the harness keeps the recent tail of the conversation and summarizes the older transcript into a synthetic message. Each of the four harnesses does this differently. OpenClaw flushes important state to memory before compacting. Claude Code offloads oversized tool results before each call and can restore recently read files after compaction. Letta warns before the window is full, evicts with a sliding window, and falls back to stronger truncation if the summary itself overflows. Compaction done badly can quietly ruin a session, so I wrote [a separate deep dive](/writing/compaction/) on how each harness triggers, summarizes, and recovers. ## Sub-agents are isolated processes You might expect a child agent to inherit the parent's conversation, but that would double the context problem instead of solving it, and most harnesses refuse to do it. Pi starts a fresh process with just the task string. OpenClaw starts a fresh session and passes only filtered workspace context. Claude Code's typed agents start blank: the delegated prompt becomes the first user message, with restricted tools and permissions. Letta mostly avoids forking and keeps execution inside the main loop, with history reachable through recall and archival tools. Sub-agents are closer to isolated processes than shared threads. They get the task, the permissions, and the workspace slice they need, not the parent's whole mental state. I cover the pattern properly in [the sub-agents essay](/writing/sub-agents/). ## The same moves keep showing up None of this is a coding-agent quirk. A data exploration agent hits the same wall with tables, traces, JSON, notebooks, and charts. Arize's Alyx converged on near-identical answers: cap tool results, binary-search for the largest slice that fits, deduplicate repeated previews, keep full payloads server-side, expose drill-down tools, and force checkpoints when token pressure climbs. Cursor, Aider, Continue, LangGraph, and OpenAI's Agents SDK all point in the same direction.
The full feature matrix. Four very different architectures converge on the same context primitives.
Bigger windows don't dissolve the problem either. [Lost in the Middle](https://arxiv.org/abs/2307.03172) showed that models use information at the start and end of a long context far better than information buried in the middle. Chroma's [Context Rot](https://www.trychroma.com/research/context-rot) report measured 18 models getting less reliable as input grows, even on simple tasks. Putting more tokens in the window doesn't guarantee the model uses them well, so the management problem stays. ## The mental model is virtual memory The best frame for all of this isn't prompt engineering. It's virtual memory: - The prompt is registers and cache: what the model needs right now. - Recent conversation is RAM. - Summaries are compressed pages. - Files, vector stores, databases, logs, and tool artifacts are disk. - Grep, semantic search, offsets, limits, and recall tools are page lookup. - Compaction is garbage collection. - Sub-agent isolation is process management. This analogy isn't new. The [MemGPT paper](https://arxiv.org/abs/2310.08560), the research behind Letta, proposed exactly this in 2023: manage memory tiers inside and outside the window the way an operating system moves pages between RAM and disk. What's changed since then is that every serious harness now implements some version of it. The agent that got dumber after one big file read didn't have a model problem. It had a paging problem. The agent looks like one continuous intelligence, and underneath, the harness is constantly moving memory around to keep that illusion alive. ## Where this fits The future of agents isn't just better models with bigger windows. It's harnesses that make a fixed-size working set feel infinite. They cap, slice, search, store, summarize, restore, and isolate, and the model sees the right working set at the right time. [The research evidence](/writing/harness-engineering/) keeps finding that the harness drives more of an agent's performance than the model. Context management is a large part of where that gap comes from. ## Sources - [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172), Liu et al., 2023 - [Context Rot: How Increasing Input Tokens Impacts LLM Performance](https://www.trychroma.com/research/context-rot), Chroma research, 2025 - [MemGPT: Towards LLMs as Operating Systems](https://arxiv.org/abs/2310.08560), Packer et al., 2023, the paper behind Letta - [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents), Anthropic - [What is an agent harness? The nine components of a great one](/writing/what-is-an-agent-harness/), the companion essay this post deepens - [Harness engineering: why agent performance now lives outside the model](/writing/harness-engineering/), the research evidence --- ## File: farooq-site/src/content/writing/compaction.md --- title: 'Compaction is the hardest problem in agent engineering' image: '/og/compaction.png' date: 2026-06-11 tags: ['agents', 'harness engineering'] --- Compaction is the step where an agent harness summarizes older conversation history to make room for new work. Every long session needs it, because the transcript grows each turn and the context window does not, so at some point something has to give. Done badly, it quietly ruins the session: an agent that worked well for an hour starts re-reading files it already read and proposing approaches it already rejected. In [my post on harness anatomy](/writing/what-is-an-agent-harness/) I flagged compaction with a one-line warning. This post is the full version of that warning. I covered the whole context system in [a companion essay on agent context management](/writing/agent-context-management/): file read caps, tool result handling, paging, the virtual memory framing. In this post I want to stay on one step of that system, the summarize-and-prune move itself: why it exists, when it fires, what a careless pass destroys, and the patterns that make it safe. ## Why the window fills During a real task the window holds more than the conversation: - fixed context: the system prompt, tool definitions, instruction files, skills - conversation history, growing every turn - file contents, where a single read can run tens of thousands of tokens - tool results: grep output, bash output, search results, each call appending another object - sub-agent reports, memory state, and the artifacts of previous compactions All of it competes with the one budget you cannot touch: room for the model to answer. Claude Code's budget was around 200,000 tokens, now up to a million with Opus. A bigger window delays the problem, but it doesn't remove it. Any session that runs long enough arrives at the same wall, and the harness has to decide what to do when it gets there. ## The trigger The harness doesn't wait for a hard failure. It watches usage, and when it crosses a threshold near the budget, compaction fires. The numbers vary, the shape doesn't. Anthropic's API ships compaction as a built-in feature with a default trigger of 150,000 input tokens. Letta warns before the window is full, evicts older messages with a sliding window, and falls back to stronger truncation if summarization itself overflows. Claude Code triggers when usage gets close to its limit and lets you fire early by hand. What happens when it fires is LLM-powered summarization. The harness takes the older portion of the transcript, sends it to a model with a structured prompt, and gets back one synthetic message. Reduced to its floor, compaction is trading a pile of exact records for a paragraph of prose. Anthropic's default compaction prompt asks the model to write down "anything that would be helpful, including the state, next steps, learnings." The conversation is then rebuilt: summary first, recent turns after it, and the session continues. The summary is lossy, though, and the losses fall in a specific place. ## What a naive summary destroys The obvious implementation is a single pass: summarize everything, swap it in, move on. This is the version that ruins sessions, and it's worth being precise about why. A summary preserves narrative. It destroys specifics. And an agent mid-task runs on specifics: - precise file paths, the difference between `config.py` and `config/settings.py` - exact error strings, which the agent must recognize when the bug resurfaces - tool arguments that worked: flags, offsets, query strings - decisions and their reasons: what was tried, what failed, and why an approach was rejected The last item is the most damaging. The decision usually survives summarization. The reason rarely does. "We chose approach B" compresses fine, but the three failed attempts that justified it disappear, and with them the agent's defense against trying approach A again. You don't have to take this on intuition. In the Meta-Harness work I covered in [the harness engineering essay](/writing/harness-engineering/), an optimizer reading raw execution traces hit 50% accuracy. Replacing the traces with summaries dropped it to 34.9%, barely better than no traces at all. The signal lives in the exact text, and a summary is exact about nothing. ## How a session fails afterward Nothing crashes after a bad compaction. The session keeps going and degrades in recognizable ways: - **The agent repeats work.** It re-reads files and re-runs searches, because the results now exist only as "explored the codebase." - **It contradicts earlier decisions.** The conclusion survived, the reasoning did not, so the agent re-litigates a settled question and picks the option it already rejected. - **It loses the thread.** The goal got paraphrased one too many times, and the agent starts solving an adjacent problem. - **It declares done too early.** The summary says tests were written. It does not say two of them are failing. A later turn reads partial progress as completion and stops. The fourth failure is the premature completion problem Anthropic hit in its own agent work. Bad compaction manufactures the conditions for it. ## The patterns that work Look at how production harnesses handle this (Claude Code, OpenClaw, Letta, Pi) and four patterns repeat.
The compaction pipeline: flush first, summarize the old, keep the recent tail exact.
**Keep recent turns verbatim.** The near past is where the model's current state lives, so never summarize the active working set. Pi keeps the recent tail in full and summarizes only the older transcript. Letta's default sliding window summarizes about 30% of messages and keeps 70% untouched. Anthropic's API supports the same shape: pause after the summary, re-append the last few messages exactly, then continue. Touch the tail last, or never. **Stage the summarization.** Not all tokens deserve equal treatment, so don't run one lossy pass over everything. Anthropic calls tool result clearing the safest, lightest-touch form of compaction: a raw grep result from forty turns ago can go long before any conversation text gets rewritten. OpenClaw stages its summarization and prunes tool results in memory without destroying the persistent conversation log. The transcript on disk stays complete. Only the model's view shrinks. **Flush before you summarize.** If a summary is going to lose specifics, get the specifics out of its reach first. OpenClaw's pre-compaction flush gives the agent a chance to write important state to files before history disappears: the plan, key decisions, open problems, exact paths and commands. Anything on disk no longer depends on surviving the summary. Claude Code attacks the same gap from the other side: after compaction it restores the five most recently accessed files, so exact file contents reenter the window from disk rather than from a paraphrase. **Keep durable state outside the transcript.** The strongest version of the flush is never keeping critical state in the conversation in the first place. A plan file, a notes file, a decision log: state on disk is safe by construction, because compaction only touches the transcript. Sub-agents are the same idea applied to whole workstreams. Delegate exploration to [an isolated sub-agent](/writing/sub-agents/) that returns a short report, and the thousand-line search never enters the parent's window at all. The best compaction is the one that fires later because the transcript stayed small. ## This code gets deleted, not refined Every harness component encodes an assumption about what the model cannot do alone, and [those assumptions expire](/writing/harness-engineering/). Compaction encodes two: windows are small, and models lose the plot over long contexts. Both are weakening. Claude Code's budget went from roughly 200,000 tokens to a million with Opus. When Opus 4.6 stopped needing context resets, Anthropic deleted them from its harness. That's the trajectory for compaction logic too. Not refinement into something baroque, but deletion on schedule, as bigger windows and better long-context behavior make each stage unnecessary. The elaborate summarization machinery will be the first thing to go. The flush and the external state will outlive it. Writing decisions, plans, and exact paths to durable files isn't a workaround for a small window. It makes sessions resumable, auditable, and safe to interrupt, whatever the window size. So build compaction as if you will delete it next year, and put everything you cannot afford to lose somewhere compaction cannot reach. ## Sources - [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents), Anthropic, on compaction, tool result clearing, and Claude Code's summarize-then-restore behavior - [Context compaction](https://platform.claude.com/docs/en/build-with-claude/compaction), Claude API docs: trigger thresholds, the default summarization prompt, and preserving recent messages - [Compaction](https://docs.letta.com/guides/core-concepts/messages/compaction/), Letta docs, on sliding window strategies and their defaults - [Agent context management](/writing/agent-context-management/), the companion essay on the full context system - [What is an agent harness?](/writing/what-is-an-agent-harness/) and [Harness engineering](/writing/harness-engineering/), the related essays on this site --- ## File: farooq-site/src/content/writing/deepseek-visual-primitives.md --- title: 'DeepSeek visual primitives: teaching models to reason with a cursor' image: '/og/deepseek-visual-primitives.png' date: 2026-06-11 tags: ['research notes', 'vision'] --- Thinking with Visual Primitives is a DeepSeek paper about how a vision model keeps hold of a specific object while it reasons. The publication history is strange: DeepSeek published the paper in late April 2026, then removed it within about a day. Some people saw the GitHub repository. Others now see a 404 page. The model weights are not broadly released, so this is not a review of a product. It is a note on the research idea, which is worth understanding regardless of what happens to the release. In this post I want to explain the problem the paper targets, the fix it proposes, how the model is trained, and the limits the paper itself admits. ## Why words keep losing their grip The problem is how a model refers to objects while it reasons. Ask a vision model to count the mugs on a cluttered desk, and its reasoning will say things like "the mug on the left, next to the notebook." If there are several candidates, the phrase never locks onto one thing. The problem compounds: over a long chain of reasoning, a reference can drift, so the phrase that meant one mug at step two quietly means a different mug at step nine. The paper calls this the reference gap. In plain terms, the model has no way to say "this one" and have it stay this one. This is a different problem from perception. Most multimodal work in recent years has aimed at helping models see better: higher resolution, stronger encoders, more visual tokens. None of that helps here, because the reference gap is not about seeing. It is about keeping track of what has already been seen. The fix has to give the model a pointing device, not better eyes. ## A cursor made of three shapes The proposed fix is to give the model three visual primitives to use while it reasons. A box refers to an object. A point refers to a location. A sequence of points describes a path. That is the whole vocabulary: a way to say "this thing," "this spot," and "this route." Bounding boxes themselves are nothing new. Models have emitted boxes for years, but as outputs, attached after the answer. The important claim here is the placement. The boxes and points are interleaved into the reasoning trace itself, so coordinates sit between the words, as units of thought rather than as a final report.
The three primitives, and where they live: inside the reasoning trace, between the words.
Here is what that buys in practice. For counting, the model can mark every object first and then count the marked boxes, so the count becomes a property of the trace rather than a guess. On the cluttered desk, instead of "the mug on the left," the trace now holds a box around that mug, and step nine points at the same pixels step two did. For spatial questions, the model can pin down the objects being compared before comparing them. For mazes and line tracing, it can lay down a sequence of points and follow the route through the image step by step. ## Teaching the model to point The pipeline has two broad stages. First the model learns to produce the primitives at all: boxes and points in the right format, grounded in the right pixels. Then it practices the tasks where primitives pay off: counting, spatial reasoning, maze navigation, path tracing. The useful trick in this pipeline is automatic verification. Many of these tasks can be checked by a program with no human in the loop. A maze path either crosses a wall or it does not. A traced line can be compared against the true curve. So the training loop gets a cheap and reliable reward signal, the same property that made verifiable rewards work for math and code, and the model can practice at scale without anyone labeling its homework. The reported gains follow the training menu. They are strongest exactly where you would expect: dense counting, spatial reasoning, maze navigation, and path tracing, the tasks where holding a stable reference is most of the difficulty. ## Where this fits in DeepSeek's vision work DeepSeek has been building toward this for a while. DeepSeek-VL focused on general image understanding. DeepSeek-VL2 extended it with a more efficient mixture-of-experts design. Then DeepSeek-OCR asked a stranger question: can vision serve as a compact way to store and recover text from documents? The new paper continues that line. It describes strong compression of visual tokens before and inside the language model, and it treats coordinates as something the model works with, not only something it outputs. The common thread is vision as a compact working format, not just an input image. DeepSeek is not only trying to make models see more pixels. It is exploring how a model keeps track of what it is seeing. ## The honest limits The paper's limits matter as much as its results, so I will be specific about them. The weights are not broadly available. The original repository is gone, and community mirrors of the paper are what remain. Until weights or an official release appear, none of the reported numbers can be independently reproduced. Some benchmarks are in-house. The tasks that show the largest gains are partly evaluated on benchmarks the team built. That is not disqualifying, but in-house benchmarks tend to flatter the method they were designed around. The primitives need explicit triggers. The model does not yet decide on its own that a question deserves boxes. The behavior has to be invoked, so the cursor exists but the model does not yet reach for it unprompted. And point-based reasoning is still hard outside the training settings. The primitives work best inside the task families the model practiced on. Whether reasoning with coordinates generalizes to messy, open-ended visual questions is exactly the open question. The takeaway is modest but real. The model is being trained to reason with a cursor, a way to say "this one" and have it hold. Whether that becomes a standard part of multimodal models depends on the limits above, and right now nobody outside DeepSeek can check. ## Sources - Thinking with Visual Primitives, DeepSeek-AI, April 2026. The original repository was taken down; a [community mirror of the paper](https://github.com/mitkox/Thinking-with-Visual-Primitives) and an [alphaXiv overview](https://www.alphaxiv.org/overview/visual-primitives) survive - [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://arxiv.org/abs/2403.05525), March 2024 - [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://arxiv.org/abs/2412.10302), December 2024 - [DeepSeek-OCR: Contexts Optical Compression](https://arxiv.org/abs/2510.18234), October 2025