# Technical Documentation: jamwithai/production-agentic-rag-course > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/jamwithai/production-agentic-rag-course) ### Overview (`README.md`) # The Mother of AI Project ## Phase 1 RAG Systems: arXiv Paper Curator

A Learner-Focused Journey into Production RAG Systems

Learn to build modern AI systems from the ground up through hands-on implementation

Master the most in-demand AI engineering skills: RAG (Retrieval-Augmented Generation)

## 📖 About This Course This is a **learner-focused project** where you'll build a complete research assistant system that automatically fetches academic papers, understands their content, and answers your research questions using advanced RAG techniques. **The arXiv Paper Curator** will teach you to build a **production-grade RAG system using industry best practices**. Unlike tutorials that jump straight to vector search, we follow the **professional path**: master keyword search foundations first, then enhance with vectors for hybrid retrieval. > **🎯 The Professional Difference:** We build RAG systems the way successful companies do - solid search foundations enhanced with AI, not AI-first approaches that ignore search fundamentals. By the end of this course, you'll have your own AI research assistant and the deep technical skills to build production RAG systems for any domain. ### **🎓 What You'll Build** - **Week 1:** Complete infrastructure with Docker, FastAPI, PostgreSQL, OpenSearch, and Airflow - **Week 2:** Automated data pipeline fetching and parsing academic papers from arXiv - **Week 3:** Production BM25 keyword search with filtering and relevance scoring - **Week 4:** Intelligent chunking + hybrid search combining keywords with semantic understanding - **Week 5:** Complete RAG pipeline with local LLM, streaming responses, and Gradio interface - **Week 6:** Production monitoring with Langfuse tracing and Redis caching for optimized performance - **Week 7:** **Agentic RAG with LangGraph and Telegram Bot for mobile access** --- ## 🏗️ System Architecture Evolution ### Week 7: Agentic RAG & Telegram Bot Integration

Week 7 Telegram and Agentic AI Architecture

Complete Week 7 architecture showing Telegram bot integration with the agentic RAG system

### LangGraph Agentic RAG Workflow

Detailed LangGraph workflow showing decision nodes, document grading, and adaptive retrieval

**Week 7 Code walkthrough + blog:** [Agentic RAG with LangGraph and Telegram](https://jamwithai.substack.com/p/agentic-rag-with-langgraph-and-telegram) **Key Innovations in Week 7:** - **Intelligent Decision-Making**: Agents evaluate and adapt retrieval strategies - **Document Grading**: Automatic relevance assessment with semantic evaluation - **Query Rewriting**: Adaptive query refinement when results are insufficient - **Guardrails**: Out-of-domain detection prevents hallucination - **Mobile Access**: Telegram bot for conversational AI on any device - **Transparency**: Full reasoning step tracking for debugging and trust --- ## 🚀 Quick Start ### **📋 Prerequisites** - **Docker Desktop** (with Docker Compose) - **Python 3.12+** - **UV Package Manager** ([Install Guide](https://docs.astral.sh/uv/getting-started/installation/)) - **8GB+ RAM** and **20GB+ free disk space** ### **⚡ Get Started** ```bash # 1. Clone and setup git clone cd arxiv-paper-curator # 2. Configure environment (IMPORTANT!) cp .env.example .env # The .env file contains all necessary configuration for OpenSearch, # arXiv API, and service connections. Defaults work out of the box. # You need to add Jina embeddings free api key and langfuse keys (check the blogs) # 3. Install dependencies uv sync # 4. Start all services docker compose up --build -d # 5. Verify everything works curl http://localhost:8000/api/v1/health ``` ### **📚 Weekly Learning Path** | Week | Topic | Blog Post | Code Release | |------|-------|-----------|--------------| | **Week 0** | The Mother of AI project - 6 phases | [The Mother of AI project](https://jamwithai.substack.com/p/the-mother-of-ai-project) | - | | **Week 1** | Infrastructure Foundation | [The Infrastructure That Powers RAG Systems](https://jamwithai.substack.com/p/the-infrastructure-that-powers-rag) | [week1.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week1.0) | | **Week 2** | Data Ingestion Pipeline | [Building Data Ingestion Pipelines for RAG](https://jamwithai.substack.com/p/bringing-your-rag-system-to-life) | [week2.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week2.0) | | **Week 3** | OpenSearch ingestion & BM25 retrieval | [The Search Foundation Every RAG System Needs](https://jamwithai.substack.com/p/the-search-foundation-every-rag-system) | [week3.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week3.0) | | **Week 4** | **Chunking & Hybrid Search** | [The Chunking Strategy That Makes Hybrid Search Work](https://jamwithai.substack.com/p/chunking-strategies-and-hybrid-rag) | [week4.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week4.0) | | **Week 5** | **Complete RAG system** | [The Complete RAG System](https://jamwithai.substack.com/p/the-complete-rag-system) | [week5.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week5.0) | | **Week 6** | **Production monitoring & caching** | [Production-ready RAG: Monitoring & Caching](https://jamwithai.substack.com/p/production-ready-rag-monitoring-and) | [week6.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week6.0) | | **Week 7** | **Agentic RAG & Telegram Bot** | [Agentic RAG with LangGraph and Telegram](https://jamwithai.substack.com/p/agentic-rag-with-langgraph-and-telegram) | [week7.0](https://github.com/jamwithai/arxiv-paper-curator/releases/tag/week7.0) | **📥 Clone a specific week's release:** ```bash # Clone a specific week's code git clone --branch https://github.com/jamwithai/arxiv-paper-curator cd arxiv-paper-curator uv sync docker compose down -v docker compose up --build -d # Replace with: week1.0, week2.0, etc. ``` ### **📊 Access Your Services** | Service | URL | Purpose | |---------|-----|---------| | **API Documentation** | http://localhost:8000/docs | Interactive API testing | | **Gradio RAG Interface** | http://localhost:7861 | User-friendly chat interface | | **Langfuse Dashboard** | http://localhost:3000 | RAG pipeline monitoring & tracing | | **Airflow Dashboard** | http://localhost:8080 | Workflow management | | **OpenSearch Dashboards** | http://localhost:5601 | Hybrid search engine UI | #### **NOTE**: Check airflow/simple_auth_manager_passwords.json.generated for Airflow username and password --- ## 📚 Week 1: Infrastructure Foundation ✅ **Start here!** Master the infrastructure that powers modern RAG systems. ### **🎯 Learning Objectives** - Complete infrastructure setup with Docker Compose - FastAPI development with automatic documentation and health checks - PostgreSQL database configuration and management - OpenSearch hybrid search engine setup - Ollama local LLM service configuration - Service orchestration and health monitoring - Professional development environment with code quality tools ### **🏗️ Architecture Overview**

Week 1 Infrastructure Setup

**Infrastructure Components:** - **FastAPI**: REST endpoints with async support (Port 8000) - **PostgreSQL 16**: Paper metadata storage (Port 5432) - **OpenSearch 2.19**: Search engine with dashboards (Ports 9200, 5601) - **Apache Airflow 3.0**: Workflow orchestration (Port 8080) - **Ollama**: Local LLM server (Port 11434) ### **📓 Setup Guide** ```bash # Launch the Week 1 notebook uv run jupyter notebook notebooks/week1/week1_setup.ipynb ``` **Completion Guide:** Follow the [Week 1 notebook](notebooks/week1/week1_setup.ipynb) for hands-on setup and verification steps. ### **📖 Deep Dive** **Blog Post:** [The Infrastructure That Powers RAG Systems](https://jamwithai.substack.com/p/the-infrastructure-that-powers-rag) - Detailed walkthrough and production insights --- ## 📚 Week 2: Data Ingestion Pipeline ✅ **Building on Week 1 infrastructure:** Learn to fetch, process, and store academic papers automatically. ### **🎯 Learning Objectives** - arXiv API integration with rate limiting and retry logic - Scientific PDF parsing using Docling - Automated data ingestion pipelines with Apache Airflow - Metadata extraction and storage workflows - Complete paper processing from API to database ### **🏗️ Architecture Overview**

Week 2 Data Ingestion Architecture

**Data Pipeline Components:** - **MetadataFetcher**: 🎯 Main orchestrator coordinating the entire pipeline - **ArxivClient**: Rate-limited paper fetching with retry logic - **PDFParserService**: Docling-powered scientific document processing - **Airflow DAGs**: Automated daily paper ingestion workflows - **PostgreSQL Storage**: Structured paper metadata and content ### **📓 Implementation Guide** ```bash # Launch the Week 2 notebook uv run jupyter notebook notebooks/week2/week2_arxiv_integration.ipynb ``` **Completion Guide:** Follow the [Week 2 notebook](notebooks/week2/week2_arxiv_integration.ipynb) for hands-on implementation and verification steps. ### **📖 Deep Dive** **Blog Post:** [Building Data Ingestion Pipelines for RAG](https://jamwithai.substack.com/p/bringing-your-rag-system-to-life) - arXiv API integration and PDF processing --- ## 📚 Week 3: Keyword Search First - The Critical Foundation **Building on Weeks 1-2 foundation:** Implement the keyword search foundation that professional RAG systems rely on. ### **🎯 Learning Objectives** - Why keyword search is essential for RAG systems (foundation first approach) - OpenSearch index management, mappings, and search optimization - BM25 algorithm and the math behind effective keyword search - Query DSL for building complex search queries with filters and boosting - Search analytics for measuring relevance and performance - Production patterns used by real companies ### **🏗️ Architecture Overview**

Week 3 OpenSearch Flow Architecture

**Search Infrastructure Components:** - **OpenSearch Service**: `src/services/opensearch/` - Professional search service implementation - **Search API**: `src/routers/search.py` - Search API endpoints with BM25 scoring - **Learning Materials**: `notebooks/week3/` - Complete OpenSearch integration guide - **Quality Metrics**: Precision, recall, and relevance scoring ### **📓 Setup Guide** ```bash # Launch the Week 3 notebook uv run jupyter notebook notebooks/week3/week3_opensearch.ipynb ``` **Completion Guide:** Follow the [Week 3 notebook](notebooks/week3/week3_opensearch.ipynb) for hands-on OpenSearch setup and BM25 search implementation. ### **📖 Deep Dive** **Blog Post:** [The Search Foundation Every RAG System Needs](https://jamwithai.substack.com/p/the-search-foundation-every-rag-system) - Complete BM25 implementation with OpenSearch --- ## 📚 Week 4: Chunking & Hybrid Search - The Semantic Layer **Building on Week 3 foundation:** Add the semantic layer that makes search truly intelligent. ### **🎯 Learning Objectives** - Section-based chunking with intelligent document segmentation - Production embeddings with Jina AI integration and fallback strategies - Hybrid search mastery using RRF fusion for keyword + semantic retrieval - Unified API design with single endpoint supporting multiple search modes - Performance analysis and trade-offs between search approaches ### **🏗️ Architecture Overview**

Week 4 Hybrid Search Architecture

**Hybrid Search Infrastructure Components:** - **Text Chunker**: `src/services/indexing/text_chunker.py` - Section-aware chunking with overlap strategies - **Embeddings Service**: `src/services/embeddings/` - Production embedding pipeline with Jina AI - **Hybrid Search API**: `src/routers/hybrid_search.py` - Unified search API supporting all modes - **Learning Materials**: `notebooks/week4/` - Complete hybrid search implementation guide ### **📓 Setup Guide** ```bash # Launch the Week 4 notebook uv run jupyter notebook notebooks/week4/week4_hybrid_search.ipynb ``` **Completion Guide:** Follow the [Week 4 notebook](notebooks/week4/week4_hybrid_search.ipynb) for hands-on implementation and verification steps. ### **📖 Deep Dive** **Blog Post:** [The Chunking Strategy That Makes Hybrid Search Work](https://jamwithai.substack.com/p/chunking-strategies-and-hybrid-rag) - Production chunking and RRF fusion implementation --- ## 📚 Week 5: Complete RAG Pipeline with LLM Integration **Building on Week 4 hybrid search:** Add the LLM layer that turns search into intelligent conversation. ### **🎯 Learning Objectives** - Local LLM integration with Ollama for complete data privacy - Performance optimization with 80% prompt reduction (6x speed improvement) - Streaming implementation using Server-Sent Events for real-time responses - Dual API design with standard and streaming endpoints - Interactive Gradio interface with advanced parameter controls ### **🏗️ Architecture Overview**

Week 5 Complete RAG System Architecture

**Complete RAG Infrastructure Components:** - **RAG Endpoints**: `src/routers/ask.py` - Dual endpoints (`/api/v1/ask` + `/api/v1/stream`) - **Ollama Service**: `src/services/ollama/` - LLM client with optimized prompts - **System Prompt**: `src/services/ollama/prompts/rag_system.txt` - Optimized for academic papers - **Gradio Interface**: `src/gradio_app.py` - Interactive web UI with streaming support - **Launcher Script**: `gradio_launcher.py` - Easy-launch script (runs on port 7861) ### **📓 Setup Guide** ```bash # Launch the Week 5 notebook uv run jupyter notebook notebooks/week5/week5_complete_rag_system.ipynb # Launch Gradio interface uv run python gradio_launcher.py # Open http://localhost:7861 ``` **Completion Guide:** Follow the [Week 5 notebook](notebooks/week5/week5_complete_rag_system.ipynb) for hands-on LLM integration and RAG pipeline implementation. ### **📖 Deep Dive** **Blog Post:** [The Complete RAG System](https://jamwithai.substack.com/p/the-complete-rag-system) - Complete RAG system with local LLM integration and optimization techniques --- ## 📚 Week 6: Production Monitoring and Caching **Building on Week 5 complete RAG system:** Add observability, performance optimization, and production-grade monitoring. ### **🎯 Learning Objectives** - Langfuse integration for end-to-end RAG pipeline tracing - Redis caching strategy with intelligent cache keys and TTL management - Performance monitoring with real-time dashboards for latency and costs - Production patterns for observability and optimization - Cost analysis and LLM usage optimization (150-400x speedup with caching) ### **🏗️ Architecture Overview**

Week 6 Monitoring & Caching Architecture

**Production Infrastructure Components:** - **Langfuse Service**: `src/services/langfuse/` - Complete tracing integration with RAG-specific metrics - **Cache Service**: `src/services/cache/` - Redis client with exact-match caching and graceful fallback - **Updated Endpoints**: `src/routers/ask.py` - Integrated tracing and caching middleware - **Docker Config**: `docker-compose.yml` - Added Redis service and Langfuse local instance - **Learning Materials**: `notebooks/week6/` - Complete monitoring and caching implementation guide ### **📓 Setup Guide** ```bash # Launch the Week 6 notebook uv run jupyter notebook notebooks/week6/week6_cache_testing.ipynb ``` **Completion Guide:** Follow the [Week 6 notebook](notebooks/week6/week6_cache_testing.ipynb) for hands-on Langfuse tracing and Redis caching implementation. ### **📖 Deep Dive** **Blog Post:** [Production-ready RAG: Monitoring & Caching](https://jamwithai.substack.com/p/production-ready-rag-monitoring-and) - Production-ready RAG with monitoring and caching --- ## 📚 Week 7: Agentic RAG with LangGraph and Telegram Bot **Building on Week 6 production system:** Add intelligent reasoning, multi-step decision-making, and Telegram bot integration for mobile-first AI interactions. ### **🎯 Learning Objectives** - LangGraph workflows for state-based agent orchestration with decision nodes - Guardrail implementation for query validation and domain boundary detection - Document grading with semantic relevance evaluation - Query rewriting for automatic query refinement and better retrieval - Adaptive retrieval with multi-attempt retrieval and intelligent fallback - Telegram bot integration with async operations and error handling - Reasoning transparency by exposing agent decision-making process ### **🏗️ Architecture Overview**

Week 7 Agentic RAG & Telegram Architecture

**Agentic RAG Infrastructure Components:** - **Agent Nodes**: `src/services/agents/nodes/` - Guardrail, retrieve, grade, rewrite, and generate nodes - **Workflow Orchestration**: `src/services/agents/agentic_rag.py` - LangGraph workflow coordination - **Telegram Bot**: `src/services/telegram/` - Command handlers and message processing - **Agentic Endpoint**: `src/routers/agentic_ask.py` - Agentic RAG API endpoint - **Learning Materials**: `notebooks/week7/` - Week 7 learning materials and examples ### **📓 Setup Guide** ```bash # Launch the Week 7 notebook uv run jupyter notebook notebooks/week7/week7_agentic_rag.ipynb ``` **Completion Guide:** Follow the [Week 7 notebook](notebooks/week7/week7_agentic_rag.ipynb) for hands-on LangGraph agentic RAG and Telegram bot implementation. ### **📖 Deep Dive** **Blog Post:** [Agentic RAG with LangGraph and Telegram](https://jamwithai.substack.com/p/agentic-rag-with-langgraph-and-telegram) - Building intelligent agents with decision-making, adaptive retrieval, and mobile access --- ## ⚙️ Configuration **Setup:** ```bash cp .env.example .env # Edit .env for your environment ``` **Key Variables:** - `JINA_API_KEY` - Required for Week 4+ (hybrid search with embeddings) - `TELEGRAM__BOT_TOKEN` - Required for Week 7 (Telegram bot integration) - `LANGFUSE__PUBLIC_KEY` & `LANGFUSE__SECRET_KEY` - Optional for Week 6 (monitoring) **Complete Configuration:** See [.env.example](.env.example) for all available options and detailed documentation. --- ## 🔧 Reference & Development Guide ### **🛠️ Technology Stack** | Service | Purpose | Status | |---------|---------|--------| | **FastAPI** | REST API with automatic docs | ✅ Ready | | **PostgreSQL 16** | Paper metadata and content storage | ✅ Ready | | **OpenSearch 2.19** | Hybrid search engine (BM25 + Vector) | ✅ Ready | | **Apache Airflow 3.0** | Workflow automation | ✅ Ready | | **Jina AI** | Embedding generation (Week 4) | ✅ Ready | | **Ollama** | Local LLM serving (Week 5) | ✅ Ready | | **Redis** | High-performance caching (Week 6) | ✅ Ready | | **Langfuse** | RAG pipeline observability (Week 6) | ✅ Ready | **Development Tools:** UV, Ruff, MyPy, Pytest, Docker Compose ### **🏗️ Project Structure** ``` arxiv-paper-curator/ ├── src/ # Main application code │ ├── routers/ # API endpoints (search, ask, papers) │ ├── services/ # Business logic (opensearch, ollama, agents, cache) │ ├── models/ # Database models (SQLAlchemy) │ ├── schemas/ # Pydantic validation schemas │ └── config.py # Environment configuration ├── notebooks/ # Weekly learning materials (week1-7) ├── airflow/ # Workflow orchestration (DAGs) ├── tests/ # Test suite └── compose.yml # Docker service orchestration ``` ### **📡 API Endpoints Reference** | Endpoint | Method | Description | Week | |----------|--------|-------------|------| | `/health` | GET | Service health check | Week 1 | | `/api/v1/papers` | GET | List stored papers | Week 2 | | `/api/v1/papers/{id}` | GET | Get specific paper | Week 2 | | `/api/v1/search` | POST | BM25 keyword search | Week 3 | | `/api/v1/hybrid-search/` | POST | Hybrid search (BM25 + Vector) | **Week 4** | **API Documentation:** Visit http://localhost:8000/docs for interactive API explorer ### **🔧 Essential Commands** #### **Using the Makefile** (Recommended) ```bash # View all available commands make help # Quick workflow make start # Start all services make health # Check all services health make test # Run tests make stop # Stop services ``` #### **All Available Commands** | Command | Description | |---------|-------------| | `make start` | Start all services | | `make stop` | Stop all services | | `make restart` | Restart all services | | `make status` | Show service status | | `make logs` | Show service logs | | `make health` | Check all services health | | `make setup` | Install Python dependencies | | `make format` | Format code | | `make lint` | Lint and type check | | `make test` | Run tests | | `make test-cov` | Run tests with coverage | | `make clean` | Clean up everything | #### **Direct Commands** (Alternative) ```bash # If you prefer using commands directly docker compose up --build -d # Start services docker compose ps # Check status docker compose logs # View logs uv run pytest # Run tests ``` ### **🎓 Target Audience** | Who | Why | |-----|-----| | **AI/ML Engineers** | Learn production RAG architecture beyond tutorials | | **Software Engineers** | Build end-to-end AI applications with best practices | | **Data Scientists** | Implement production AI systems using modern tools | --- ## 🛠️ Troubleshooting **Common Issues:** - **Services not starting?** Wait 2-3 minutes, check `docker compose logs` - **Port conflicts?** Stop other services using ports 8000, 8080, 5432, 9200 - **Memory issues?** Increase Docker Desktop memory allocation **Get Help:** - Check the comprehensive Week 1 notebook troubleshooting section - Review service logs: `docker compose logs [service-name]` - Complete reset: `docker compose down --volumes && docker compose up --build -d` --- ## 💰 Cost Structure **This course is completely free!** You'll only need minimal costs for optional services: - **Local Development:** $0 (everything runs locally) - **Optional Cloud APIs:** ~$2-5 for external LLM services (if chosen) ---

🎉 Ready to Start Your AI Engineering Journey?

Begin with the Week 1 setup notebook and build your first production RAG system!

For learners who want to master modern AI engineering

Built with love by Shirin Khosravi Jam & Shantanu Ladhwe

--- ## Star History [](https://star-history.com/#jamwithai/production-agentic-rag-course&Date) --- ## 📄 License MIT License - see [LICENSE](LICENSE) file for details. --- ### Overview (`airflow/README.md`) # Airflow Configuration This directory contains Apache Airflow configuration and DAGs for the arXiv Paper Curator project.

Week 2 Data Ingestion Architecture

## Current Setup (Week 2) ### Production-Ready DAGs - **hello_world_dag.py**: Basic health check DAG for Week 1 - **arxiv_paper_ingestion.py**: Main production DAG for automated arXiv paper fetching and processing ### Production Pipeline Features - **Daily arXiv ingestion**: Automated fetching of CS.AI papers - **PDF processing**: Download and parse papers using Docling - **Database storage**: Store complete paper metadata and content in PostgreSQL - **Error handling**: Comprehensive retry logic and error reporting - **Cross-platform compatibility**: Works on macOS, Linux, WSL, and Ubuntu ## Directory Structure ``` airflow/ ├── README.md # This file ├── Dockerfile # Custom Airflow container with dependencies ├── requirements-airflow.txt # Python dependencies for DAGs └── dags/ ├── hello_world_dag.py # Week 1 health check DAG ├── arxiv_paper_ingestion.py # Week 2 production ingestion DAG └── arxiv_ingestion/ └── tasks.py # Production pipeline tasks with async processing ``` ## Docker Configuration ### Cross-Platform Compatibility The Airflow container is configured for cross-platform deployment: - **User Configuration**: Runs as `airflow` user (50000:0) to avoid permission issues - **Volume Management**: Uses named volumes for logs to prevent bind mount conflicts - **Database Integration**: Connects to shared PostgreSQL instance - **Service Dependencies**: Automatic initialization and health checks ### Container Features - **Python 3.12** with Apache Airflow 2.10.3 - **PostgreSQL support** via psycopg2 - **PDF processing** with Docling, Tesseract OCR, and Poppler utilities - **Rate limiting** and retry logic for arXiv API compliance - **Async processing** for optimal performance with concurrent downloads and parsing ## Usage ### Web Interface - **URL**: http://localhost:8080 - **Credentials**: Auto-generated during container initialization - **Features**: DAG monitoring, task logs, pipeline statistics ### Production DAG (`arxiv_paper_ingestion`) 1. **Environment Setup**: Verify services and initialize caching 2. **Daily Paper Fetch**: Retrieve papers from previous day (10 papers default) 3. **PDF Processing**: Download and parse PDFs with Docling 4. **Failed PDF Retry**: Handle any processing failures 5. **Database Storage**: Store complete paper data with parsed content 6. **OpenSearch Placeholders**: Prepare for Week 3+ search indexing 7. **Daily Report**: Generate comprehensive processing statistics ### Pipeline Performance - **Concurrent Processing**: 5 parallel downloads, 1 parsing operation (laptop-optimized) - **Rate Limiting**: Respects arXiv API guidelines (3-second delays) - **Caching**: PDF files cached locally to avoid re-downloading - **Error Resilience**: Continues processing even with individual paper failures ## Configuration ### Environment Variables ```bash AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db AIRFLOW__CORE__EXECUTOR=LocalExecutor POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db PYTHONPATH=/opt/airflow/src ``` ### Service Dependencies - **PostgreSQL**: Paper metadata and content storage - **Source Code**: Mounted from `../src` for service access - **Shared Network**: Communication with API and database services ## Week 2 Implementation Status ### ✅ Completed Features - Custom Docker container with all dependencies - Production arXiv ingestion DAG with comprehensive error handling - Async PDF processing pipeline with concurrency control - PostgreSQL integration with complete content storage - Cross-platform compatibility (macOS, Linux, WSL, Ubuntu) - Rate limiting and retry logic for arXiv API compliance - Detailed logging and monitoring throughout pipeline ### 🔄 Week 3+ Roadmap - **OpenSearch Integration**: Real search indexing (currently placeholders) - **Advanced Scheduling**: Multiple collection strategies - **Monitoring & Alerting**: Production observability - **Scale Optimization**: Higher concurrency for production workloads --- ### Overview (`notebooks/week1/README.md`) # Week 1: Infrastructure Setup and Verification This folder contains the materials for Week 1 of the arXiv Paper Curator project, which focuses on setting up and verifying the complete infrastructure stack. ## Contents ### `week1_setup.ipynb` A comprehensive Jupyter notebook that guides students through: 1. **System Requirements and Setup** - Understanding each technology component and its purpose - Cross-platform installation instructions (Windows, macOS, Linux) - Prerequisites verification with automated checking 2. **Infrastructure Architecture** - Complete overview of the multi-service architecture - Understanding how Docker containers communicate - Data persistence and volume management concepts

Week 1 Infrastructure Setup

**Architecture Overview:** - **FastAPI** (Port 8000): REST API with async support and automatic documentation - **PostgreSQL 16** (Port 5432): Primary database for paper metadata and content storage - **OpenSearch 2.19** (Ports 9200, 5601): Hybrid search engine with management dashboards - **Apache Airflow 3.0** (Port 8080): Workflow orchestration with DAGs and PostgreSQL backend - **Ollama** (Port 11434): Local LLM server for future RAG implementation - **Docker Network**: All services communicate via `rag-network` with persistent volumes 3. **Service-by-Service Setup** - PostgreSQL database for paper metadata storage - OpenSearch for full-text search capabilities - Apache Airflow for workflow automation - Ollama for local LLM inference - FastAPI for REST API endpoints 4. **Verification and Testing** - Automated health checks for all services - Step-by-step verification procedures - Modular Ollama testing (4 focused test cells) - Common troubleshooting scenarios and solutions ## Learning Objectives By completing this week's materials, students will: - Understand containerization and Docker Compose orchestration - Learn how to set up a production-grade infrastructure stack - Gain experience with database design and API development - Master troubleshooting techniques for multi-service applications - Learn direct HTTP API testing vs service abstraction layers - Build confidence working with professional development tools ## Ollama Testing (Simplified for Week 1) The notebook includes modular Ollama testing broken into focused cells: - **Test 3A**: Check available models - **Test 3B**: Simple model testing (if models installed) - **Test 3C**: Performance analysis - **Test 3D**: Learning notes and setup commands ### Easy Model Installation (Optional for Week 1) ```bash # Using Makefile (recommended) make ollama-pull MODEL=llama3.2:1b make ollama-test MODEL=llama3.2:1b # Direct HTTP calls for learning curl -X POST http://localhost:11434/api/pull -d '{"name":"llama3.2:1b"}' curl -X POST http://localhost:11434/api/generate -d '{"model":"llama3.2:1b","prompt":"Hello","stream":false}' ``` ### Recommended Models for Course - **llama3.2:1b** (1.2GB) - Fast, good for testing - **llama3.2:3b** (2.0GB) - Balance of speed/quality - **llama3.1:8b** (4.7GB) - Better quality, slower **Note**: No models are required for Week 1 - service health check works without them. ## Target Audience This material is designed for: - **Beginners** who want to learn modern software infrastructure - **Students** looking to understand how real-world applications are built - **Professionals** transitioning into software development or DevOps - **Anyone** interested in building their own AI-powered research tools ## Time Commitment - **Setup**: 2-3 hours (including software installation and downloads) - **Notebook completion**: 1 hours - **Total**: 2-4 hours ## 📖 Additional Resources **Week 1 Blog Post:** [The Infrastructure That Powers RAG Systems](https://jamwithai.substack.com/p/the-infrastructure-that-powers-rag) - Deep dive into each infrastructure component - Production deployment considerations - Architecture decision explanations ## Support Resources If you encounter issues: 1. Check the troubleshooting sections in the notebook 2. Review the common problems and solutions 3. Ensure all prerequisites are properly installed 4. Follow the step-by-step verification procedures 5. Ask in Jam With AI substack chat channel ## Next Steps After completing Week 1, you will be ready to: - Understand how each service contributes to the overall system - Modify and extend the infrastructure as needed - Proceed to Week 2: arXiv Integration and PDF Processing - Build confidence in working with professional development environments --- ### Overview (`notebooks/week2/README.md`) # Week 2: arXiv API Integration & PDF Processing This folder contains the materials for Week 2 of the arXiv Paper Curator project, which focuses on building the core data ingestion pipeline that feeds fresh academic content into our RAG system. ## Contents ### `week2_arxiv_integration.ipynb` A comprehensive Jupyter notebook that guides students through: 1. **Infrastructure Validation** - Verify all Week 1 services are running correctly - Container health checks and fresh build verification - Environment setup for Week 2 components 2. **arXiv API Integration** - Building a robust client with rate limiting and retry logic - Implementing date-based filtering for targeted paper retrieval - Testing CS.AI category searches with proper API etiquette

Week 2 Data Ingestion Architecture

**Data Pipeline Overview:** - **MetadataFetcher**: 🎯 Main orchestrator coordinating the entire pipeline - **ArxivClient**: Rate-limited fetching with retry logic (3-second delays) - **PDFParserService**: Scientific PDF parsing with structured content extraction - **PaperRepository**: PostgreSQL integration with upsert operations - **Airflow DAGs**: Automated daily ingestion workflows 3. **PDF Processing Pipeline** - Download and cache PDF files with proper error handling - Parse scientific PDFs using Docling for structured content extraction - Handle parsing failures gracefully with fallback mechanisms 4. **Database Integration** - Store paper metadata and content in PostgreSQL - Implement upsert logic to avoid duplicates - Test retrieval and query operations 5. **Complete Pipeline Testing** - End-to-end processing from arXiv API to database storage - Error handling and graceful degradation testing - Performance metrics and success rate analysis 6. **Production Readiness** - Airflow DAG status verification - Error logging and monitoring capabilities - Ready for automated daily ingestion ## Learning Objectives By completing this week's materials, students will: - Master API integration with proper rate limiting and error handling - Learn PDF processing techniques for scientific documents - Understand database design patterns for research data storage - Build robust data pipelines with comprehensive error handling - Gain experience with async Python programming patterns - Learn workflow automation concepts with Apache Airflow - Develop skills in production-grade data processing systems ## Key Technologies & Services ### Core Services (Built This Week) - **arXiv API Client** - Fetches CS.AI papers with intelligent rate limiting - **PDF Parser (Docling)** - Extracts structured content from scientific PDFs - **Metadata Fetcher** - Orchestrates the complete processing pipeline - **Database Repository** - Handles PostgreSQL operations with SQLAlchemy ### Infrastructure Dependencies (From Week 1) - **PostgreSQL 16** - Paper metadata and content storage - **FastAPI** - REST API endpoints for paper retrieval - **Apache Airflow** - Workflow orchestration and scheduling - **Docker Compose** - Service orchestration and networking ## Pipeline Architecture ``` arXiv Search Query → Rate Limited API Calls → PDF Downloads → Docling Parsing → Database Storage ↓ ↓ ↓ ↓ ↓ Date Filtering → Retry Logic → Caching → Structure → Upsert Logic Category Filter → Error Handling → Validation → Extraction → Transactions Result Limiting → 3s Rate Limit → Size Checks → Metadata → Relationships ``` ## Performance Characteristics **Week 2 System Capabilities:** - **arXiv API**: ~20 papers/minute (respecting 3-second rate limits) - **PDF Processing**: 2-5 seconds per paper (depends on PDF complexity) - **Database Storage**: ~100 papers/second (batch operations) - **Error Handling**: Graceful continuation despite individual failures - **Success Rates**: 95%+ for paper fetching, 80-90% for PDF parsing ## Target Audience This material is designed for: - **Data Engineers** learning research data pipeline construction - **Students** interested in academic research automation - **Developers** building content aggregation systems - **Researchers** wanting to automate literature discovery - **Anyone** building production data ingestion pipelines ## Time Commitment - **Fresh container build**: 10-15 minutes (required for Week 2 dependencies) - **Notebook completion**: 45-60 minutes - **Pipeline testing**: 30-45 minutes - **Total**: 1.5-2 hours ## 📖 Additional Resources **Week 2 Blog Post:** [Building Robust Data Pipelines for Academic Research](https://jamwithai.substack.com/p/building-data-pipelines-academic-research) - Deep dive into arXiv API best practices - PDF processing strategies for scientific documents - Error handling patterns for production systems - Database design for research metadata ## Important Notes ### Fresh Container Build Required Week 2 requires rebuilding containers with new dependencies: ```bash # Shutdown and rebuild (REQUIRED for Week 2) docker compose down docker compose up --build # This ensures: # - New Python dependencies (docling, arxiv client) # - Updated Airflow DAGs # - Fresh service configurations ``` ### PDF Processing Expectations - Not all PDFs parse successfully (expected behavior) - Docling works best with standard academic paper formats - System handles failures gracefully and continues processing - Success rates of 80-90% are normal for academic PDFs ## Support Resources If you encounter issues: 1. Ensure fresh containers are built (`docker compose up --build`) 2. Check the troubleshooting sections in the notebook 3. Review service health checks and logs 4. Verify all Week 1 infrastructure is working 5. Ask in Jam With AI substack chat channel ## Next Steps After completing Week 2, you will be ready to: - Understand production data pipeline architecture - Handle real-world API integration challenges - Implement robust error handling and monitoring - Proceed to Week 3: OpenSearch Integration and Full-Text Search - Build confidence in handling complex data processing workflows ## Success Criteria ✅ **Week 2 Complete When:** - arXiv API client fetches papers with proper rate limiting - PDF download and caching works reliably - Docling parser extracts structured content from scientific papers - Database stores complete paper metadata with relationships - Complete pipeline processes papers end-to-end with error handling - Airflow DAGs are configured and ready for automation - All components demonstrate production-grade error handling and monitoring --- ### Overview (`notebooks/week3/README.md`) # Week 3: Keyword Search First - The Critical Foundation > **🚨 The 90% Problem:** Most RAG systems jump straight to vector search and miss the foundation that powers the best retrieval systems. We're doing it right! This folder contains the materials for Week 3 of the arXiv Paper Curator project, where we implement the **keyword search foundation** that professional RAG systems rely on using OpenSearch and BM25 scoring. ## 🎯 Why Keyword Search First? **The Professional Path:** Unlike tutorials that jump straight to vector embeddings, we build the foundation that successful companies use: 1. **🔍 Exact Match Power:** Keywords excel at finding specific technical terms, paper IDs, and precise phrases 2. **📊 Interpretable Results:** You can understand exactly why a paper was retrieved 3. **⚡ Speed & Efficiency:** BM25 is computationally fast and doesn't require expensive embedding models 4. **📈 Production Reality:** Companies like Elasticsearch, Algolia, and enterprise search use keyword search as their foundation **The Learning Path:** ``` Week 3: Master BM25 keyword search ← YOU ARE HERE Week 4: Add intelligent chunking Week 5: Introduce vector embeddings for hybrid retrieval Week 6: Optimize the complete system ``` ## 🚀 Before You Start **Essential Environment Setup:** ```bash # 1. Ensure you have the correct environment configuration cp .env.example .env # 2. Verify OpenSearch settings are properly configured # Your .env should contain these critical Week 3 settings: # OPENSEARCH__HOST=http://opensearch:9200 # OPENSEARCH__INDEX_NAME=arxiv-papers ``` **Important:** Week 3 requires the `.env` file to be properly configured for OpenSearch connectivity and indexing. The defaults in `.env.example` work out of the box. ## Contents ### `week3_opensearch.ipynb` A comprehensive Jupyter notebook that guides you through building the keyword search foundation: 1. **Infrastructure Validation** - Verify all Week 1-2 services are running correctly - OpenSearch cluster health and connectivity testing - Environment setup for Week 3 search components 2. **OpenSearch Service Integration** - Building a production-grade search client with factory patterns - Implementing dependency injection with FastAPI lifespan management - Creating JSON-based index configurations with proper analyzers ## 🔍 OpenSearch Dashboard in Action *You'll add your OpenSearch dashboard screenshot here showing real paper searches with BM25 scoring and result rankings.*

OpenSearch Dashboard Retrieving Papers
OpenSearch Dashboards showing keyword search results with BM25 relevance scoring

**Search Architecture Overview:** - **OpenSearch Client**: Factory-pattern service with health monitoring and CRUD operations - **Query Builder**: Advanced search query construction with field boosting and filtering - **Index Management**: JSON-based schema with English analyzers and strict mapping - **Search API**: RESTful endpoints (`/search`) with GET/POST methods for different query types - **BM25 Algorithm**: Industry-standard text relevance scoring with multi-field search 3. **Index Management & Schema Design** - Creating indexes with proper field mappings for academic papers - Implementing English language analyzers for improved search relevance - Testing index health, statistics, and document management operations 4. **Document Indexing Pipeline** - Migrating paper data from PostgreSQL to OpenSearch - Bulk indexing operations with error handling and validation - Real-time indexing verification and performance monitoring 5. **BM25 Search Implementation** - Multi-field search across title, abstract, and content with field boosting - Advanced query features: highlighting, pagination, category filtering - Testing two-letter queries (AI, ML, NN, CV) as specifically requested - Fuzzy matching and typo tolerance configuration 6. **Search API Development** - FastAPI integration with OpenSearch dependency injection - RESTful endpoints: `GET /search` for simple queries, `POST /search` for advanced search - Response formatting with pagination, metadata, and highlighted snippets - Error handling and service health monitoring endpoints 7. **Airflow Pipeline Updates** - Sequential task execution: setup → fetch → opensearch → report → cleanup - Real OpenSearch indexing replacing placeholder operations - Updated daily reporting with search statistics and health metrics - Graceful handling of PDF processing limits (>20MB or >30 pages) 8. **End-to-End Pipeline Verification** - Complete flow testing: arXiv API → PostgreSQL → OpenSearch → Search API - Performance benchmarking and response time measurement - Production readiness assessment with monitoring and alerting **Week 3 Architecture:**

Week 3 OpenSearch Flow Architecture
Complete Week 3 architecture showing the OpenSearch integration flow

## Key Features Implemented ### 🔍 **Production-Grade Search System** - **BM25 Scoring**: Industry-standard relevance ranking algorithm - **Multi-field Search**: Query across title (3x boost), abstract (2x boost), and content (1x boost) - **Advanced Filtering**: Category filtering, date ranges, and custom field queries - **Real-time Highlighting**: Search term highlighting in results with HTML markup - **Pagination Support**: Efficient large result set handling with size/from parameters ### 🏗️ **Clean Architecture Implementation** - **Factory Pattern**: Consistent service creation across all components - **Dependency Injection**: Proper service lifecycle management with FastAPI lifespan - **Query Builder Pattern**: Flexible search query construction for complex operations - **Error Handling**: Comprehensive exception management with graceful degradation ### 📊 **Performance & Monitoring** - **Sub-100ms Search**: Optimized query performance for real-time applications - **Health Monitoring**: Cluster status, index statistics, and service availability - **Resource Limits**: PDF processing constraints to prevent memory issues - **Scalability**: Design patterns supporting horizontal scaling ### 🔧 **Development Best Practices** - **Type Safety**: Full type hints throughout the codebase - **Testing**: End-to-end verification with realistic data scenarios - **Documentation**: Comprehensive docstrings and architectural explanations - **Code Quality**: Following established patterns from Week 1-2 implementations ## Expected Outcomes By completing Week 3, students will have: 1. **Working Search System**: Fully functional OpenSearch BM25 search with 28+ indexed papers 2. **RESTful API**: Complete search API with both simple and advanced query capabilities 3. **Production Architecture**: Clean, scalable code following industry best practices 4. **Performance Insights**: Understanding of search optimization and monitoring strategies 5. **End-to-End Pipeline**: Complete RAG system foundation ready for Week 4 development ## Success Criteria - ✅ **OpenSearch Integration**: Factory pattern service with health checks operational - ✅ **BM25 Search**: Relevance scoring with multi-field queries returning relevant results - ✅ **Advanced Features**: Filtering, highlighting, pagination, and fuzzy search working - ✅ **API Endpoints**: RESTful search with both GET and POST methods functional - ✅ **Two-Letter Queries**: AI, ML, NN, CV queries working as specifically requested - ✅ **Pipeline Integration**: Airflow DAGs updated for OpenSearch indexing - ✅ **Performance**: Sub-100ms search response times with proper error handling ## What's Next? **Week 4 Preview**: Chunking Strategies and Hybrid Retrieval - Intelligent document chunking strategies for better context preservation - Combining keyword search (BM25) with vector embeddings for hybrid retrieval - Building the complete foundation for modern RAG systems - Evaluation metrics to measure retrieval quality --- **Ready to build production-grade search? Let's get started with Week 3!** 🔍✨ --- ### Overview (`notebooks/week4/README.md`) # Week 4: Document Chunking and Hybrid Search ## Overview Week 4 implements a **production-grade hybrid search system** that combines the precision of BM25 keyword search with the semantic understanding of vector embeddings. This system provides the foundation for retrieval-augmented generation (RAG) by intelligently breaking documents into searchable chunks and enabling multiple search modes. ## What We Built ### 🧩 **Section-Based Document Chunking** - **Intelligent Segmentation**: Leverages document structure (parsed sections) for natural chunk boundaries - **Context Preservation**: 100-word overlaps between chunks maintain semantic continuity - **Adaptive Processing**: Handles both structured (with sections) and unstructured (paragraph-based) documents - **Optimal Sizing**: Targets 600-word chunks with minimum 100-word threshold ### 🔍 **Unified Hybrid Search System** - **Single Index Architecture**: One OpenSearch index (`arxiv-papers-chunks`) supports all search modes - **Multiple Search Types**: - **BM25 Keyword Search**: Fast (~50ms) traditional text matching - **Vector Similarity Search**: Semantic search using 1024-dimensional embeddings - **Hybrid Search**: RRF (Reciprocal Rank Fusion) combining both approaches - **Production API**: RESTful endpoint `/api/v1/hybrid-search/` with comprehensive validation ### 🤖 **Real Embedding Integration** - **Jina AI Embeddings**: Production-grade 1024-dimensional vectors optimized for retrieval - **Automatic Generation**: FastAPI endpoints automatically generate query embeddings - **Fallback Strategy**: Graceful degradation to BM25 when embeddings unavailable - **Performance Optimized**: Efficient embedding generation and storage ## Architecture ### System Overview

Week 4 Hybrid Search Architecture
Complete Week 4 architecture showing hybrid search with chunking, embeddings, and RRF fusion

### Data Flow ``` Raw Papers → PDF Parsing → Section Extraction → Chunking → Embedding → Indexing → Search ``` The diagram above illustrates the complete Week 4 implementation: - **Data Processing Pipeline**: arXiv papers flow through chunking and embedding generation - **Unified OpenSearch Index**: Single index supporting BM25, vector, and hybrid search modes - **Hybrid Retrieval Pipeline**: RRF fusion combining keyword precision with semantic understanding - **Production API Layer**: FastAPI endpoints with automatic embedding generation ### System Components #### **1. Document Processing Pipeline** ```python # Located in: src/services/indexing/text_chunker.py TextChunker.chunk_paper( title="Paper Title", abstract="Abstract text", full_text="Complete paper content", sections=parsed_sections_dict, target_words=600, overlap_words=100 ) ``` #### **2. Embedding Service** ```python # Located in: src/services/embeddings/factory.py embeddings_service = make_embeddings_service() vectors = await embeddings_service.embed_query(["query text"]) ``` #### **3. Unified Search Client** ```python # Located in: src/services/opensearch/client.py results = opensearch_client.search_unified( query="machine learning", query_embedding=vector, use_hybrid=True, size=10 ) ``` #### **4. Production API** ```python # Located in: src/routers/hybrid_search.py POST /api/v1/hybrid-search/ { "query": "neural networks", "use_hybrid": true, "size": 5 } ``` ## Key Features ### **Hybrid Search Modes** | Mode | Speed | Recall | Precision | Use Case | |------|--------|--------|-----------|----------| | **BM25 Only** | ~50ms | High | Medium | Exact keyword matching | | **Vector Only** | ~100ms | Medium | High | Semantic similarity | | **Hybrid (RRF)** | ~2-4s | High | High | Best overall relevance | ### **RRF (Reciprocal Rank Fusion)** - **Algorithm**: Combines BM25 and vector search rankings using reciprocal rank fusion - **Implementation**: Manual fusion algorithm (OpenSearch 2.19 compatibility) - **Weighting**: Configurable balance between keyword and semantic relevance - **Fallback**: Automatic fallback to BM25 if vector search fails ### **Section-Based Chunking Strategy** ```python # Chunking Parameters (optimized through testing) CHUNK_SIZE = 600 # Target words per chunk OVERLAP_SIZE = 100 # Words overlapping between chunks MIN_CHUNK_SIZE = 100 # Minimum viable chunk size SECTION_BASED = True # Use document structure when available ``` **Benefits**: - **Semantic Coherence**: Chunks respect natural document boundaries - **Context Preservation**: Overlaps prevent information loss at boundaries - **Retrieval Accuracy**: Better matching of user queries to relevant content - **Scalability**: Handles documents from 1,000 to 100,000+ words ## Implementation Details ### **OpenSearch Index Configuration** **Index Name**: `arxiv-papers-chunks` **Key Fields**: ```json { "arxiv_id": "2508.18563v1", "title": "Paper title", "chunk_text": "Chunk content...", "chunk_id": "unique_chunk_identifier", "section_name": "Introduction", "embedding": [0.123, 0.456, ...], // 1024 dimensions "paper_categories": ["cs.AI", "cs.LG"], "published_date": "2025-08-25T23:43:33" } ``` ### **Search Query Structure** **BM25 Query** (keyword matching): ```json { "query": { "bool": { "should": [ {"match": {"chunk_text": {"query": "machine learning", "fuzziness": "AUTO"}}}, {"match": {"title": {"query": "machine learning", "boost": 2.0}}}, {"match": {"abstract": {"query": "machine learning", "boost": 1.5}}} ] } } } ``` **Hybrid Query** (RRF manual fusion): 1. Execute BM25 query → get ranked results 2. Execute vector query → get ranked results 3. Apply RRF fusion algorithm → combine rankings 4. Return merged results with hybrid scores ### **Environment Configuration** **Required Variables**: ```bash # Core Services POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db OPENSEARCH__HOST=http://opensearch:9200 # Embeddings (Required for Hybrid Search) JINA_API_KEY=jina_your_api_key_here # Chunking Configuration CHUNKING__CHUNK_SIZE=600 CHUNKING__OVERLAP_SIZE=100 CHUNKING__MIN_CHUNK_SIZE=100 CHUNKING__SECTION_BASED=true # OpenSearch Configuration OPENSEARCH__INDEX_NAME=arxiv-papers OPENSEARCH__CHUNK_INDEX_SUFFIX=chunks OPENSEARCH__VECTOR_DIMENSION=1024 ``` ## API Reference ### **Hybrid Search Endpoint** **Endpoint**: `POST /api/v1/hybrid-search/` **Request Body**: ```json { "query": "transformer neural networks", "use_hybrid": true, "size": 10, "from": 0, "categories": ["cs.AI", "cs.LG"], "latest_papers": false, "min_score": 0.0 } ``` **Response**: ```json { "query": "transformer neural networks", "total": 15, "hits": [ { "arxiv_id": "2508.18563v1", "title": "Paper Title", "authors": "Author Names", "abstract": "Paper abstract...", "score": 0.8542, "chunk_text": "Relevant chunk content...", "chunk_id": "chunk_uuid", "section_name": "Related Work" } ], "size": 10, "from": 0, "search_mode": "hybrid" } ``` ## Performance Benchmarks **Test Environment**: 3 papers, 81 chunks, single-node OpenSearch | Search Type | Avg Response Time | Throughput | Recall@10 | Precision@10 | |-------------|-------------------|------------|-----------|--------------| | BM25 Only | 52ms | ~200 req/s | 0.78 | 0.65 | | Vector Only | 105ms | ~95 req/s | 0.82 | 0.71 | | Hybrid (RRF) | 2.4s | ~25 req/s | 0.89 | 0.84 | **Key Insights**: - **Hybrid search** provides best relevance at cost of response time - **BM25** excellent for high-throughput keyword matching - **Vector search** good semantic understanding with moderate speed - **Embedding generation** accounts for ~2s of hybrid search time ## Production Deployment ### **Scaling Considerations** **OpenSearch Cluster**: ```yaml # Recommended minimum for production opensearch: image: opensearchproject/opensearch:2.19.0 environment: - cluster.name=rag-cluster - node.name=rag-node-1 - discovery.type=single-node - OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g deploy: resources: limits: memory: 2G reservations: memory: 1G ``` **Embedding Service Optimization**: - **Batch Processing**: Process multiple embeddings per API call - **Caching**: Cache frequent query embeddings - **Rate Limiting**: Respect Jina AI API limits (1000 requests/minute) - **Fallback Strategy**: BM25-only mode when embeddings unavailable ### **Monitoring and Observability** **Key Metrics**: - Search request latency (p50, p95, p99) - Embedding generation success rate - Index document count and size - Search mode usage distribution (BM25 vs Hybrid) **Health Checks**: - OpenSearch cluster health - Embedding service availability - Index document count validation - Sample query execution ## Troubleshooting ### **Common Issues** **1. Hybrid Search Returns BM25 Mode** ```bash # Check embedding service curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \ -H "Content-Type: application/json" \ -d '{"query": "test", "use_hybrid": true}' # Check logs for embedding errors docker compose logs api | grep -i embedding ``` **2. Empty Search Results** ```bash # Verify index exists and has documents curl "http://localhost:9200/arxiv-papers-chunks/_count" # Check index mapping curl "http://localhost:9200/arxiv-papers-chunks/_mapping" ``` **3. Slow Embedding Generation** ```bash # Check Jina API key configuration docker compose exec api env | grep JINA # Test direct embedding service curl -X POST "https://api.jina.ai/v1/embeddings" \ -H "Authorization: Bearer $JINA_API_KEY" \ -d '{"model": "jina-embeddings-v3", "input": ["test"]}' ``` ## Testing ### **Running the Week 4 Notebook** 1. **Start Services**: ```bash docker compose up --build -d ``` 2. **Open Notebook**: ```bash cd notebooks/week4 jupyter notebook week4_hybrid_search.ipynb ``` 3. **Execute All Cells**: The notebook includes: - Environment setup and health checks - Section-based chunking demonstration - Real embedding generation with Jina AI - All search modes (BM25, Vector, Hybrid) - Production API endpoint testing - Performance comparison ### **Manual Testing** **Test BM25 Search**: ```bash curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \ -H "Content-Type: application/json" \ -d '{"query": "machine learning", "use_hybrid": false, "size": 3}' ``` **Test Hybrid Search**: ```bash curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \ -H "Content-Type: application/json" \ -d '{"query": "neural networks", "use_hybrid": true, "size": 3}' ``` ## Next Steps (Week 5) Week 4 provides the search foundation for Week 5's LLM integration: 1. **LLM Integration**: Connect Ollama for answer generation 2. **RAG Pipeline**: Query → Search → Context → Generate → Response 3. **Context Management**: Optimize retrieved chunks for LLM input 4. **Answer Quality**: Implement citation and source attribution 5. **Conversation Memory**: Support multi-turn conversations The hybrid search system is **production-ready** and provides the retrieval accuracy needed for high-quality RAG applications. ## File Structure ``` src/ ├── routers/ │ └── hybrid_search.py # FastAPI endpoints ├── services/ │ ├── opensearch/ │ │ ├── client.py # Unified search client │ │ ├── factory.py # Client factory │ │ └── index_config_hybrid.py # Index configuration │ ├── indexing/ │ │ ├── text_chunker.py # Section-based chunking │ │ ├── hybrid_indexer.py # Document indexing │ │ └── factory.py # Indexing service factory │ └── embeddings/ │ ├── jina_client.py # Jina AI client │ └── factory.py # Embedding service factory ├── schemas/ │ └── api/ │ └── search.py # Request/response models └── config.py # Configuration management notebooks/week4/ ├── README.md # This document ├── week4_hybrid_search.ipynb # Interactive tutorial └── data/ # Sample data directory ``` ## Resources - **OpenSearch Documentation**: https://opensearch.org/docs/ - **Jina AI Embeddings**: https://jina.ai/embeddings/ - **FastAPI Documentation**: https://fastapi.tiangolo.com/ - **Reciprocal Rank Fusion Paper**: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf - **Week 4 Notebook**: [week4_hybrid_search.ipynb](./week4_hybrid_search.ipynb) --- ### Overview (`notebooks/week5/README.md`) # Week 5: Complete RAG System with LLM Integration ## Overview Week 5 completes our **production-grade RAG system** by integrating Ollama LLM with hybrid search. The system delivers **6x faster performance** (120s → 15-20s), real-time streaming, and includes a Gradio web interface. ## What We Built - **Local LLM Integration**: Ollama service with llama3.2 models - **Performance Optimization**: 80% prompt reduction, 6x speed improvement - **Streaming API**: Real-time responses via Server-Sent Events - **Gradio Interface**: Interactive web UI with streaming support - **Production Ready**: Clean API design with two focused endpoints ## Architecture

Week 5 Complete RAG System Architecture
Complete RAG system with LLM generation layer (Ollama), hybrid retrieval pipeline, and Gradio interface

## Quick Start ### 1. Start Services ```bash docker compose up --build -d ``` ### 2. Test RAG Endpoint ```bash curl -X POST "http://localhost:8000/api/v1/ask" \ -H "Content-Type: application/json" \ -d '{"query": "What are transformers?", "top_k": 3, "use_hybrid": true}' ``` ### 3. Launch Gradio Interface ```bash uv run python gradio_launcher.py # Open http://localhost:7861 ``` ## API Endpoints ### Standard RAG - `/api/v1/ask` - **Purpose**: Complete response with metadata - **Response Time**: 15-20 seconds - **Use Case**: Batch processing, API integrations ### Streaming RAG - `/api/v1/stream` - **Purpose**: Real-time token generation - **Time to First Token**: 2-3 seconds - **Use Case**: Interactive UIs, better UX ### Request Format ```json { "query": "Your question", "top_k": 3, // Chunks to retrieve (1-10) "use_hybrid": true, // BM25 + vector search "model": "llama3.2:1b", // LLM model "categories": ["cs.AI"] // Optional filter } ``` ## Performance | Configuration | Response Time | Use Case | |--------------|---------------|----------| | `top_k=1, BM25` | ~2.4s | Quick answers | | `top_k=3, Hybrid` | ~15-20s | Balanced quality | | `top_k=5, Hybrid` | ~25-30s | Comprehensive | **Key Optimizations**: - Removed redundant metadata (80% prompt reduction) - Shared code architecture (DRY principles) - 300-word response limit for focused answers - Automatic source deduplication ## Configuration ```bash # .env file OLLAMA_HOST=http://ollama:11434 OLLAMA__DEFAULT_MODEL=llama3.2:1b JINA_API_KEY=your_key_here # For embeddings ``` ## Testing ### Run the Notebook ```bash jupyter notebook notebooks/week5/week5_complete_rag_system.ipynb ``` ### Test Streaming ```bash curl -X POST "http://localhost:8000/api/v1/stream" \ -H "Content-Type: application/json" \ -d '{"query": "Explain attention mechanism", "top_k": 2}' \ --no-buffer ``` ## Troubleshooting | Issue | Solution | |-------|----------| | 404 on `/stream` | Rebuild API: `docker compose build api && docker compose restart api` | | Slow responses | Use smaller model: `llama3.2:1b` or reduce `top_k` | | No Gradio | Port changed to 7861: `http://localhost:7861` | | Ollama errors | Check service: `docker exec rag-ollama ollama list` | ## Project Structure ``` src/ ├── routers/ │ └── ask.py # RAG endpoints ├── services/ │ └── ollama/ │ ├── client.py # LLM client │ └── prompts/ # System prompts ├── gradio_app.py # Web interface └── gradio_launcher.py # Launcher script notebooks/week5/ ├── README.md # This file └── week5_complete_rag_system.ipynb ``` ## Next Steps - **Enhance**: Add conversation memory, feedback loops - **Optimize**: Implement caching, fine-tune models - **Deploy**: Add authentication, monitoring, load balancing ## Resources - [Notebook Tutorial](./week5_complete_rag_system.ipynb) - [API Documentation](http://localhost:8000/docs) - [Gradio Interface](http://localhost:7861) - [Ollama Models](https://ollama.ai/library) --- ### Overview (`notebooks/week6/README.md`) # Week 6: Production Monitoring and Caching with Langfuse and Redis ## Overview Week 6 adds production-grade monitoring and intelligent caching to our RAG system. We integrate Langfuse for complete pipeline observability and Redis for high-performance response caching. ## What We Built - **Langfuse Integration**: End-to-end RAG pipeline tracing and analytics - **Redis Caching**: 150-400x faster responses for repeated queries - **Performance Monitoring**: Real-time metrics and system health - **Production Ready**: Enterprise-grade observability and optimization ## Architecture

Week 6 Monitoring & Caching Architecture
Week 6 architecture with Langfuse tracing and Redis caching integration

### Data Flow ``` Query → Cache Check → [Hit: ~100ms] | [Miss: Full Pipeline ~15s] → Cache Store → Langfuse Trace ``` ## Key Features ### **Langfuse Observability** - Complete RAG pipeline tracing with performance breakdowns - User analytics, query patterns, and success rate tracking - Real-time monitoring dashboard with cost and usage metrics - Quality insights with answer relevance and source attribution ### **Redis Intelligent Caching** - **Exact-Match Strategy**: Parameter-aware cache keys for precise matching - **Performance**: 150-400x faster responses for repeated queries (~100ms vs 15-20s) - **TTL Management**: 24-hour default expiration with configurable settings - **Future Enhancement**: Can be upgraded to semantic similarity caching for fuzzy matching ## Quick Start ### Environment Setup ```bash # Required environment variables LANGFUSE__SECRET_KEY=sk_lf_your_secret_key LANGFUSE__PUBLIC_KEY=pk_lf_your_public_key REDIS__HOST=redis REDIS__TTL_HOURS=24 ``` ### Start Services ```bash docker compose up --build -d ``` ### Test Caching Performance ```bash # First request (cache miss ~15-20s) curl -X POST "http://localhost:8000/api/v1/ask" \ -H "Content-Type: application/json" \ -d '{"query": "What are transformers?", "top_k": 3}' # Second identical request (cache hit ~100ms) curl -X POST "http://localhost:8000/api/v1/ask" \ -H "Content-Type: application/json" \ -d '{"query": "What are transformers?", "top_k": 3}' ``` ## Performance Benchmarks | Scenario | Response Time | Improvement | |----------|---------------|-------------| | **Cache Miss** | 15-20 seconds | Baseline | | **Cache Hit** | 50-100ms | **150-400x faster** | | **Monitoring Overhead** | <2% | Negligible impact | ## Testing ### Run the Notebook ```bash jupyter notebook notebooks/week6/week6_cache_testing.ipynb ``` ### Monitor System Health ```bash # Check Redis connectivity redis-cli ping # View cache statistics curl "http://localhost:8000/api/v1/health" # Access Langfuse dashboard # Visit: https://cloud.langfuse.com (or your self-hosted instance) ``` ## Troubleshooting | Issue | Solution | |-------|----------| | **Cache not working** | Check Redis: `redis-cli ping` | | **No Langfuse traces** | Verify environment variables: `LANGFUSE__*` | | **Slow responses** | Monitor cache hit rate and system resources | ## Next Steps - **Enhanced Caching**: Upgrade to semantic similarity caching for fuzzy matching - **Advanced Analytics**: Custom dashboards and A/B testing frameworks - **Production Scaling**: Distributed caching and automated monitoring - **Quality Optimization**: User feedback integration and answer scoring ## Resources - **Notebook**: [week6_cache_testing.ipynb](./week6_cache_testing.ipynb) - **Langfuse Dashboard**: https://cloud.langfuse.com - **Redis Documentation**: https://redis.io/docs --- Week 6 transforms your RAG system into a production-grade service with 150-400x performance improvements and comprehensive observability. --- ### Overview (`notebooks/week7/README.md`) # Week 7: Agentic RAG with LangGraph + Telegram Bot ## Overview Week 7 adds TWO major enhancements to the arXiv Paper Curator: 1. **🤖 Agentic RAG with LangGraph** - Intelligent, adaptive retrieval with decision-making 2. **💬 Telegram Bot Integration** - Conversational interface for mobile/desktop access --- ## 🧠 Part 1: Agentic RAG with LangGraph ### What is Agentic RAG? **Traditional RAG** (Weeks 5-6): ``` Query → Always Retrieve → Generate Answer ``` **Agentic RAG** (Week 7): ``` Query → Agent Decides: ├─ Simple question? → Respond directly (faster!) └─ Research needed? → Retrieve ├─ Relevant docs? → Generate answer └─ Not relevant? → Rewrite query → Try again ``` ### Key Features - **🎯 Intelligent Decision Making** - LLM decides when retrieval is actually needed - **📊 Document Grading** - Validates that retrieved papers are relevant - **🔄 Query Refinement** - Rewrites vague queries for better results - **🔍 Reasoning Transparency** - Shows the agent's decision-making steps - **♻️ Iterative Improvement** - Can retry with better queries if needed ### What We Built ``` src/services/agents/ ├── tools.py # Retriever tool wrapping OpenSearch ├── nodes.py # 4 graph nodes (query, grade, rewrite, generate) ├── agentic_rag.py # LangGraph workflow + service ├── prompts.py # LLM prompt templates └── factory.py # Dependency injection src/routers/ └── agentic_ask.py # FastAPI endpoint Total: ~750 LOC following SOLID, KISS, DRY principles ``` ### Architecture ``` LangGraph Workflow: START ↓ generate_query_or_respond ├─ No retrieval needed → END (direct response) └─ Needs retrieval → retrieve (ToolNode) ↓ grade_documents ├─ Relevant → generate_answer → END └─ Not relevant → rewrite_query → (loop back) ``` ### New API Endpoint **`POST /api/v1/ask-agentic`** ```json // Request { "query": "What are transformers in ML?", "top_k": 3, "use_hybrid": true } // Response { "query": "What are transformers in ML?", "answer": "Transformers are neural network architectures...", "sources": ["https://arxiv.org/pdf/1706.03762.pdf"], "chunks_used": 3, "search_mode": "hybrid", "reasoning_steps": [ "Decided to retrieve relevant papers", "Retrieved documents from database", "Generated answer from relevant documents" ], "retrieval_attempts": 1 } ``` ### Quick Start: Agentic RAG **1. Ensure services are running:** ```bash docker compose up --build -d ``` **2. Test with cURL:** ```bash # Simple question (should respond directly) curl -X POST http://localhost:8000/api/v1/ask-agentic \ -H "Content-Type: application/json" \ -d '{ "query": "What is 2+2?", "top_k": 3, "use_hybrid": true }' # Research question (should retrieve papers) curl -X POST http://localhost:8000/api/v1/ask-agentic \ -H "Content-Type: application/json" \ -d '{ "query": "What are attention mechanisms?", "top_k": 3, "use_hybrid": true }' ``` **3. Interactive Testing:** ```bash # Open Jupyter notebook jupyter notebook notebooks/week7/week7_agentic_rag.ipynb ``` ### Comparison: Traditional vs. Agentic | Feature | Traditional RAG | Agentic RAG | |---------|----------------|-------------| | **Retrieval** | Always retrieves | Decides when needed | | **Relevance Check** | None | Grades documents | | **Query Refinement** | None | Rewrites if needed | | **Iterations** | Single pass | Multiple attempts | | **Transparency** | Black box | Shows reasoning | | **Simple Questions** | ~15-20s | ~2-5s (no retrieval) | | **Complex Questions** | Single attempt | Iterative refinement | ### Testing Scenarios **Scenario 1: Direct Response (No Retrieval)** - Query: "What is 5 + 7?" - Expected: Agent responds "12" without retrieving papers - Reasoning: "Responded directly without retrieval" **Scenario 2: Successful Retrieval** - Query: "What are transformers in machine learning?" - Expected: Agent retrieves papers, grades as relevant, generates answer - Reasoning: "Decided to retrieve" → "Retrieved documents" → "Generated answer" **Scenario 3: Query Rewriting** - Query: "Tell me about ML stuff" (vague) - Expected: Agent retrieves, grades as not relevant, rewrites query, tries again - Reasoning: "Retrieved" → "Not relevant" → "Rewritten query" → "Retrieved again" → "Generated answer" ### Design Principles Followed - ✅ **SOLID** - Single responsibility, dependency inversion, composition - ✅ **KISS** - Simple nodes (<30 lines), clear logic - ✅ **DRY** - Reused existing services (OpenSearch, Ollama, Jina) - ✅ **YAGNI** - Only implemented what's needed - ✅ **Explicit** - Type hints, docstrings, clear names - ✅ **2025 Best Practices** - MessagesState, ToolNode, tools_condition ### Documentation - **Implementation Plan**: `docs/AGENTIC_RAG_IMPLEMENTATION_PLAN.md` - **Testing Plan**: `docs/AGENTIC_RAG_TESTING_PLAN.md` - **LangGraph 2025 Patterns**: `docs/LANGGRAPH_2025_BEST_PRACTICES.md` - **Design Principles**: `docs/DESIGN_PRINCIPLES.md` - **Interactive Notebook**: `notebooks/week7/week7_agentic_rag.ipynb` --- ## 💬 Part 2: Telegram Bot Integration ### What We Built - **🤖 Full Telegram Bot Integration**: Conversational interface with command support - **💬 Natural Language Queries**: Ask questions in plain language, get answers with sources - **⚡ All Week 6 Features**: Redis caching (150-400x speedup) and Langfuse tracing - **🎯 Interactive Commands**: `/start`, `/help`, `/ask`, `/search`, `/settings`, `/status` - **👤 User Session Management**: Per-user preferences and conversation history - **📱 Mobile-First**: Rich message formatting with clickable arXiv links - **🔐 Optional Access Control**: Whitelist specific users if needed ## Architecture ### Data Flow ``` Telegram User ↓ Telegram Bot (Polling/Webhook) ↓ TelegramService + Handlers ↓ [Langfuse Tracing] Cache Check (Redis) ├─ Hit → Instant Response (~100ms) └─ Miss → Full RAG Pipeline ↓ Hybrid Search (OpenSearch BM25 + Vector) ↓ LLM Generation (Ollama) ↓ Cache Store (Redis) ↓ Format Response → Send to Telegram ``` ### New Components ``` src/services/telegram/ ├── client.py # Main Telegram bot service ├── handlers.py # Command and message handlers ├── formatters.py # Message formatting utilities ├── keyboards.py # Interactive inline keyboards ├── user_manager.py # User settings and sessions └── factory.py # Factory function src/schemas/telegram/ ├── messages.py # Message validation schemas ├── commands.py # Command schemas └── user_settings.py # User preferences schema ``` ## Key Features ### **Conversational Interface** - Natural language queries: Just send a message - Contextual conversation with history tracking - Automatic query routing (commands vs regular messages) ### **Rich Commands** ``` /start - Welcome message with bot capabilities /help - Detailed usage instructions /ask - Explicitly ask a research question /search - Quick paper search by keywords /settings - Customize preferences (search mode, results count) /status - Check system health and statistics /clear - Clear conversation history ``` ### **Interactive Settings** - Search mode: Hybrid (BM25 + semantic) or BM25 only - Results per query: 3, 5, or 10 papers - Category filters: All, cs.AI, cs.LG, cs.CL, etc. - Model selection: Choose LLM model - Toggles: Streaming, source display, compact mode ### **User Experience** - **Typing indicators** during processing - **Rich formatting** with Markdown support - **Clickable links** to arXiv papers - **Automatic message splitting** for long responses - **Source citations** with paper metadata - **Cache indicators** (⚡) for instant responses ## Quick Start ### Prerequisites 1. **Telegram Account** - Install Telegram on your phone or computer 2. **All Week 1-6 Services Running** - Full RAG stack must be operational ### Step 1: Create Your Telegram Bot 1. **Open Telegram** and search for `@BotFather` 2. **Send** `/newbot` to BotFather 3. **Follow prompts**: - Choose a name (e.g., "My arXiv Curator") - Choose a username (e.g., "my_arxiv_curator_bot" - must end in "bot") 4. **Copy the bot token** - You'll receive something like: ``` 1234567890:ABCdefGHIjklMNOpqrsTUVwxyz-1234567 ``` ### Step 2: Configure Environment Add these variables to your `.env` file: ```bash # Enable Telegram bot TELEGRAM__ENABLED=true TELEGRAM__BOT_TOKEN=your_token_from_botfather_here # Optional: Restrict to specific users (comma-separated Telegram user IDs) # Leave empty to allow all users TELEGRAM__ALLOWED_USER_IDS= # Use polling mode for development (webhook requires HTTPS) TELEGRAM__USE_WEBHOOK=false ``` ### Step 3: Install Dependencies ```bash # Install python-telegram-bot library uv sync ``` ### Step 4: Start Services ```bash # Start all services (includes Telegram bot) docker compose up --build -d # Check logs to verify Telegram bot started docker compose logs -f api ``` You should see: ``` INFO - Telegram bot started successfully INFO - Starting Telegram bot in polling mode INFO - Bot commands set successfully ``` ### Step 5: Test Your Bot 1. **Open Telegram** and search for your bot username 2. **Send** `/start` to your bot 3. **Try asking**: "What are transformers in machine learning?" 4. **Verify** you receive an answer with sources! ## Testing Instructions ### Manual Testing Scenarios #### Scenario 1: Basic Commands **Test `/start` command:** ``` You: /start Bot: 👋 Welcome to arXiv Paper Curator! [Shows capabilities and quick commands] ``` **Test `/help` command:** ``` You: /help Bot: 📚 arXiv Paper Curator Help [Shows detailed command documentation] ``` **Test `/status` command:** ``` You: /status Bot: 🔧 System Status ✅ OPENSEARCH ✅ OLLAMA ✅ CACHE [Shows system health] ``` #### Scenario 2: RAG Question Answering **Test simple question:** ``` You: What are attention mechanisms? Bot: [15-20s first time] *Answer:* Attention mechanisms allow models to dynamically focus on... 📚 *Sources:* [1] *Attention Is All You Need* 🔗 Read on arXiv - 1706.03762 📊 Score: 12.456 [2] *Neural Machine Translation...* 🔗 Read on arXiv - 1409.0473 📊 Score: 11.234 ⚙️ Mode: hybrid ``` **Test cached query:** ``` You: What are attention mechanisms? Bot: [~100ms second time ⚡] [Same answer as above] ⚙️ Mode: hybrid ⚡ Cached ``` #### Scenario 3: Search Command **Test paper search:** ``` You: /search transformer neural networks Bot: 📖 Found 145 papers (showing top 10) 1. *Attention Is All You Need* 🔗 Read on arXiv - 1706.03762 📊 Score: 12.456 2. *BERT: Pre-training of Deep Bidirectional...* 🔗 Read on arXiv - 1810.04805 📊 Score: 11.234 [...] ``` #### Scenario 4: Settings Customization **Test settings command:** ``` You: /settings Bot: ⚙️ Your Settings *Search Mode:* HYBRID *Results per query:* 3 papers *Model:* llama3.2:1b *Categories:* All [Interactive buttons appear]: [🔍 Hybrid Search] [⚡ BM25 Only] [3 Results] [5 Results] [10 Results] [All Categories] [cs.AI] [cs.LG] ``` **Test changing settings:** ``` You: [Click "5 Results" button] Bot: ✅ Results per query: 5 You: [Click "cs.AI" button] Bot: ✅ Category filter: cs.AI ``` #### Scenario 5: Natural Conversation **Test multi-turn conversation:** ``` You: Tell me about BERT Bot: [Provides answer about BERT with sources] You: How does it differ from GPT? Bot: [Answers about BERT vs GPT differences] You: /clear Bot: 🗑️ Conversation history cleared! Your settings have been preserved. ``` #### Scenario 6: Error Handling **Test invalid query:** ``` You: asdfghjkl Bot: ❌ No relevant papers found. Try different keywords or check your category filters. ``` **Test when services are down:** ``` You: test query Bot: ❌ Error processing your question: [User-friendly error message] ``` ### Verification Checklist - [ ] **Bot responds to `/start`** - Shows welcome message - [ ] **Bot responds to all commands** - `/help`, `/ask`, `/search`, `/settings`, `/status`, `/clear` - [ ] **Natural language queries work** - Can ask questions without commands - [ ] **RAG answers include sources** - Papers cited with arXiv links - [ ] **Cache works** - Repeated queries return instantly (⚡ indicator) - [ ] **Settings persist** - Changes remain across conversation - [ ] **Interactive buttons work** - Can click inline keyboard buttons - [ ] **Long responses split correctly** - Messages don't exceed Telegram limit - [ ] **Markdown formatting works** - Bold, italics, links render correctly - [ ] **Langfuse shows traces** - Check http://localhost:3000 for Telegram events - [ ] **Typing indicator shows** - "Bot is typing..." appears during processing - [ ] **Error messages are friendly** - No stack traces exposed to user ### Performance Testing **Cache Performance:** ```bash # First query (cache miss) You: What is machine learning? Bot: [Response in ~15-20 seconds] # Identical query (cache hit) You: What is machine learning? Bot: [Response in ~100ms ⚡] # Verify 150-400x speedup! ``` **Concurrent Users:** ```bash # Test with multiple Telegram accounts # Each user should have independent settings and sessions ``` **Session Management:** ```bash # Stop using bot for 30 minutes (default timeout) # Verify session cleanup happens automatically ``` ## Configuration ### Environment Variables ```bash # Enable/Disable Bot TELEGRAM__ENABLED=true # Set to false to disable # Bot Token (Required) TELEGRAM__BOT_TOKEN=your_token_here # Deployment Mode TELEGRAM__USE_WEBHOOK=false # true for production TELEGRAM__WEBHOOK_URL=https://your-domain.com TELEGRAM__WEBHOOK_PATH=/telegram/webhook # Access Control (Optional) TELEGRAM__ALLOWED_USER_IDS=123456789,987654321 # Empty = allow all # Behavior Settings TELEGRAM__MAX_MESSAGE_LENGTH=4000 # Telegram limit is 4096 TELEGRAM__ENABLE_STREAMING=true TELEGRAM__SESSION_TIMEOUT_MINUTES=30 TELEGRAM__RATE_LIMIT_MESSAGES_PER_MINUTE=20 # Default User Preferences TELEGRAM__DEFAULT_TOP_K=3 TELEGRAM__DEFAULT_USE_HYBRID=true TELEGRAM__DEFAULT_MODEL=llama3.2:1b ``` ### User Settings (Customizable per User) Each user can customize: - **Search mode**: Hybrid (BM25 + semantic) or BM25 only - **Results count**: 3, 5, or 10 papers per query - **Categories**: All, cs.AI, cs.LG, cs.CL, cs.CV, cs.NE - **Model**: LLM model for answer generation - **Display**: Show sources, compact mode, streaming Settings persist across sessions and are stored in-memory (for production, consider Redis/PostgreSQL storage). ## Architecture Details ### Service Integration The Telegram bot integrates with all existing Week 1-6 services: 1. **OpenSearch** - Hybrid search for relevant papers 2. **Jina Embeddings** - Semantic search capabilities 3. **Ollama LLM** - Answer generation 4. **Redis Cache** - 150-400x speedup for repeated queries 5. **Langfuse** - Complete tracing of Telegram interactions 6. **PostgreSQL** - Paper metadata (via OpenSearch) ### Message Flow ```python # Simplified message flow async def handle_message(update, context): 1. Extract user_id, chat_id, text 2. Check rate limits 3. Get/create user settings 4. Show "typing..." indicator 5. Check cache (Redis) 6. If cache hit → format and send 7. If cache miss: a. Generate embedding (Jina) b. Search papers (OpenSearch) c. Build prompt with context d. Generate answer (Ollama) e. Cache result (Redis) f. Trace interaction (Langfuse) 8. Format response (Markdown) 9. Split if too long 10. Send to Telegram 11. Update conversation history ``` ### Error Handling The bot implements graceful degradation: - **Markdown formatting fails** → Falls back to plain text - **Embedding generation fails** → Falls back to BM25 search - **Cache unavailable** → Continues without caching - **Langfuse tracing fails** → Logs warning, continues - **Service errors** → User-friendly error messages ### Rate Limiting Built-in rate limiting protects against abuse: - **Per-user limits**: 20 messages/minute (configurable) - **Automatic throttling**: Enforced by Telegram library - **Session timeouts**: 30 minutes of inactivity ## Troubleshooting | Issue | Solution | |-------|----------| | **Bot doesn't respond** | Check `TELEGRAM__ENABLED=true` and valid `BOT_TOKEN` | | **"Unauthorized" error** | Bot token is invalid, regenerate with @BotFather | | **Bot responds but no answers** | Check OpenSearch, Ollama, and embeddings services | | **Slow responses** | First query always slow, subsequent queries use cache | | **Markdown formatting broken** | Bot automatically falls back to plain text | | **Can't find bot** | Ensure bot username is correct, ends with "bot" | | **"Forbidden" errors** | Check user_id restrictions in `ALLOWED_USER_IDS` | | **Memory issues** | User sessions stored in-memory, monitor RAM usage | ### Debug Mode Enable detailed logging: ```bash # In .env DEBUG=true # Check bot logs docker compose logs -f api | grep telegram ``` ### Health Check ```bash # Check if bot is running curl http://localhost:8000/api/v1/health # Should show telegram_service status ``` ### Common Fixes **Bot not starting:** ```bash # 1. Check token is valid # 2. Restart API service docker compose restart api # 3. Check logs for errors docker compose logs api ``` **Responses too slow:** ```bash # 1. Check cache is working docker exec rag-redis redis-cli ping # Should return PONG # 2. Check cache hit rate # Look for ⚡ indicator in bot responses # 3. Verify Langfuse tracing not blocking LANGFUSE__ENABLED=false # Temporarily disable ``` ## Performance Benchmarks | Metric | Value | Notes | |--------|-------|-------| | **First Query** | 15-20s | Full RAG pipeline execution | | **Cached Query** | 50-100ms | **150-400x faster** | | **Typing Indicator** | <500ms | Shows immediately | | **Search Only** | 2-3s | `/search` command | | **Status Check** | <1s | `/status` command | | **Settings Update** | <100ms | Instant button response | | **Concurrent Users** | 10+ | Tested simultaneously | ### Cache Hit Rates (Expected) - **Repeated exact queries**: 100% hit rate - **Popular questions**: 60-80% hit rate - **Unique queries**: 0% hit rate (first time) ### Resource Usage - **Memory**: ~50MB per active user session - **CPU**: Minimal (<5% idle, spikes during generation) - **Network**: ~10KB per message (excluding LLM generation) ## Production Deployment ### Webhook Mode (Recommended for Production) ```bash # .env for production TELEGRAM__USE_WEBHOOK=true TELEGRAM__WEBHOOK_URL=https://your-domain.com TELEGRAM__WEBHOOK_PATH=/telegram/webhook # Requires: # - HTTPS domain with valid certificate # - Nginx/Caddy for TLS termination # - Public IP or reverse proxy ``` ### Security Best Practices 1. **Restrict Users**: Use `ALLOWED_USER_IDS` for private bots 2. **Rate Limiting**: Keep default limits (20 msgs/min) 3. **Environment Variables**: Never commit `.env` with real tokens 4. **HTTPS Only**: Use webhook mode in production 5. **Monitoring**: Track usage via Langfuse dashboard ### Scaling Considerations For high-traffic deployments: 1. **User Storage**: Migrate from in-memory to Redis/PostgreSQL 2. **Cache**: Increase Redis memory allocation 3. **Load Balancing**: Multiple API instances (webhook mode) 4. **Rate Limits**: Adjust per your requirements 5. **Monitoring**: Set up alerts for errors and latency ## Next Steps ### Optional Enhancements (Week 7.1+) - **📸 Image Support**: Upload paper PDFs, get summaries - **🗣️ Voice Messages**: Ask questions via voice (speech-to-text) - **📊 User Analytics**: Dashboard for usage patterns - **🤝 Group Chat**: Multi-user discussions - **🌍 Multi-Language**: Internationalization support - **🔔 Notifications**: Push alerts for new papers in categories - **📈 Personalization**: ML-based paper recommendations - **🔗 Semantic Cache**: Fuzzy matching for similar queries ### Integration Ideas - **Slack Bot**: Port to Slack with same architecture - **Discord Bot**: Extend to Discord communities - **WhatsApp Bot**: Use WhatsApp Business API - **Web Widget**: Embed in websites - **API Access**: Expose RESTful API for integrations ## Resources - **Telegram Bot API**: https://core.telegram.org/bots/api - **python-telegram-bot**: https://python-telegram-bot.org - **@BotFather**: https://t.me/botfather - **Langfuse Docs**: https://langfuse.com/docs - **Redis Docs**: https://redis.io/docs ## Code Structure ```python # Entry point: src/main.py telegram_service = make_telegram_service(...) await telegram_service.start() # Service: src/services/telegram/client.py class TelegramService: async def start() -> Start bot in polling/webhook mode async def stop() -> Stop bot gracefully async def health_check() -> Check bot status # Handlers: src/services/telegram/handlers.py class TelegramHandlers: async def start_command() -> /start async def help_command() -> /help async def ask_command() -> /ask async def search_command() -> /search async def settings_command() -> /settings async def handle_message() -> Regular text messages # Formatters: src/services/telegram/formatters.py format_rag_response() -> Rich Markdown formatting format_search_results() -> Search result display format_welcome_message() -> /start message escape_markdown_v2() -> Telegram MarkdownV2 escaping split_long_message() -> Auto-split >4000 chars ``` ## Success Criteria Week 7 is complete when: - ✅ Bot responds to all commands - ✅ Natural language queries return RAG answers - ✅ Cache provides 150-400x speedup - ✅ Settings persist across sessions - ✅ Interactive keyboards work - ✅ Langfuse shows Telegram traces - ✅ Error handling is graceful - ✅ Markdown formatting renders correctly - ✅ Long messages split automatically - ✅ Multiple users can use concurrently --- **Week 7 transforms your RAG system into a mobile-first, conversational research assistant accessible anywhere via Telegram!** 🚀 --- ## FAQ **Q: Do I need a public IP for the Telegram bot?** A: No! Polling mode works perfectly for development and low-traffic deployments. Webhook requires HTTPS. **Q: Can multiple users use the bot simultaneously?** A: Yes! Each user has independent settings and sessions. **Q: How much does it cost to run the bot?** A: Free! Telegram bot API is completely free with no limits. **Q: Can I restrict the bot to specific users?** A: Yes! Set `TELEGRAM__ALLOWED_USER_IDS=123456,789012` with comma-separated Telegram user IDs. **Q: How do I get my Telegram user ID?** A: Message `@userinfobot` on Telegram or check bot logs after sending a message. **Q: Does the bot store conversation history?** A: Yes, in-memory for the last 10 messages per user. For production, migrate to Redis/PostgreSQL. **Q: Can I deploy this on a server without Docker?** A: Yes! Just run `python src/main.py` after installing dependencies with `uv sync`. **Q: How do I update the bot token?** A: Update `TELEGRAM__BOT_TOKEN` in `.env` and restart: `docker compose restart api` **Q: Can I customize the bot's personality?** A: Yes! Edit prompts in `src/services/ollama/prompts/` and message templates in `src/services/telegram/formatters.py`. **Q: Does this work with other LLM models?** A: Yes! Change `OLLAMA_MODEL` or use `/settings` command to select different Ollama models. --- Enjoy your new conversational RAG interface! 🎉 --- --- METRICS --- - Files Extracted: 9 - Estimated Token Budget: ~23222 tokens - Canonical Reference: https://codewiki.google/github.com/jamwithai/production-agentic-rag-course