production-agentic-rag-course Documentation

File: README.md Copy Chapter

The Mother of AI Project

Phase 1 RAG Systems: arXiv Paper Curator Copy Chapter

📖 About This Course Copy Chapter

This is a learner-focused project where you'll build a complete research assistant system that automatically fetches academic papers, understands their content, and answers your research questions using advanced RAG techniques.

The arXiv Paper Curator will teach you to build a production-grade RAG system using industry best practices. Unlike tutorials that jump straight to vector search, we follow the professional path: master keyword search foundations first, then enhance with vectors for hybrid retrieval.

🎯 The Professional Difference: We build RAG systems the way successful companies do - solid search foundations enhanced with AI, not AI-first approaches that ignore search fundamentals.

By the end of this course, you'll have your own AI research assistant and the deep technical skills to build production RAG systems for any domain.

**🎓 What You'll Build** Copy Section

• Week 1: Complete infrastructure with Docker, FastAPI, PostgreSQL, OpenSearch, and Airflow
• Week 2: Automated data pipeline fetching and parsing academic papers from arXiv
• Week 3: Production BM25 keyword search with filtering and relevance scoring
• Week 4: Intelligent chunking + hybrid search combining keywords with semantic understanding
• Week 5: Complete RAG pipeline with local LLM, streaming responses, and Gradio interface
• Week 6: Production monitoring with Langfuse tracing and Redis caching for optimized performance
• Week 7: Agentic RAG with LangGraph and Telegram Bot for mobile access

🏗️ System Architecture Evolution Copy Chapter

Week 7: Agentic RAG & Telegram Bot Integration Copy Section

LangGraph Agentic RAG Workflow Copy Section

Week 7 Code walkthrough + blog: Agentic RAG with LangGraph and Telegram

Key Innovations in Week 7:

• Intelligent Decision-Making: Agents evaluate and adapt retrieval strategies
• Document Grading: Automatic relevance assessment with semantic evaluation
• Query Rewriting: Adaptive query refinement when results are insufficient
• Guardrails: Out-of-domain detection prevents hallucination
• Mobile Access: Telegram bot for conversational AI on any device
• Transparency: Full reasoning step tracking for debugging and trust

🚀 Quick Start Copy Chapter

**📋 Prerequisites** Copy Section

• Docker Desktop (with Docker Compose)
• Python 3.12+
• UV Package Manager (Install Guide)
• 8GB+ RAM and 20GB+ free disk space

**⚡ Get Started** Copy Section

# 1. Clone and setup
git clone <repository-url>
cd arxiv-paper-curator

# 2. Configure environment (IMPORTANT!)
cp .env.example .env
# The .env file contains all necessary configuration for OpenSearch, 
# arXiv API, and service connections. Defaults work out of the box.
# You need to add Jina embeddings free api key and langfuse keys (check the blogs)

# 3. Install dependencies
uv sync

# 4. Start all services
docker compose up --build -d

# 5. Verify everything works
curl http://localhost:8000/api/v1/health

**📚 Weekly Learning Path** Copy Section

📥 Clone a specific week's release:

# Clone a specific week's code
git clone --branch <WEEK_TAG> https://github.com/jamwithai/arxiv-paper-curator
cd arxiv-paper-curator
uv sync
docker compose down -v
docker compose up --build -d

# Replace <WEEK_TAG> with: week1.0, week2.0, etc.

**📊 Access Your Services** Copy Section

**NOTE**: Check airflow/simple_auth_manager_passwords.json.generated for Airflow username and password

📚 Week 1: Infrastructure Foundation ✅ Copy Chapter

Start here! Master the infrastructure that powers modern RAG systems.

**🎯 Learning Objectives** Copy Section

• Complete infrastructure setup with Docker Compose
• FastAPI development with automatic documentation and health checks
• PostgreSQL database configuration and management
• OpenSearch hybrid search engine setup
• Ollama local LLM service configuration
• Service orchestration and health monitoring
• Professional development environment with code quality tools

**🏗️ Architecture Overview** Copy Section

Infrastructure Components:

• FastAPI: REST endpoints with async support (Port 8000)
• PostgreSQL 16: Paper metadata storage (Port 5432)
• OpenSearch 2.19: Search engine with dashboards (Ports 9200, 5601)
• Apache Airflow 3.0: Workflow orchestration (Port 8080)
• Ollama: Local LLM server (Port 11434)

**📓 Setup Guide** Copy Section

# Launch the Week 1 notebook
uv run jupyter notebook notebooks/week1/week1_setup.ipynb

Completion Guide: Follow the Week 1 notebook for hands-on setup and verification steps.

**📖 Deep Dive** Copy Section

Blog Post: The Infrastructure That Powers RAG Systems - Detailed walkthrough and production insights

📚 Week 2: Data Ingestion Pipeline ✅ Copy Chapter

Building on Week 1 infrastructure: Learn to fetch, process, and store academic papers automatically.

**🎯 Learning Objectives** Copy Section

• arXiv API integration with rate limiting and retry logic
• Scientific PDF parsing using Docling
• Automated data ingestion pipelines with Apache Airflow
• Metadata extraction and storage workflows
• Complete paper processing from API to database

**🏗️ Architecture Overview** Copy Section

Data Pipeline Components:

• MetadataFetcher: 🎯 Main orchestrator coordinating the entire pipeline
• ArxivClient: Rate-limited paper fetching with retry logic
• PDFParserService: Docling-powered scientific document processing
• Airflow DAGs: Automated daily paper ingestion workflows
• PostgreSQL Storage: Structured paper metadata and content

**📓 Implementation Guide** Copy Section

# Launch the Week 2 notebook  
uv run jupyter notebook notebooks/week2/week2_arxiv_integration.ipynb

Completion Guide: Follow the Week 2 notebook for hands-on implementation and verification steps.

**📖 Deep Dive** Copy Section

Blog Post: Building Data Ingestion Pipelines for RAG - arXiv API integration and PDF processing

📚 Week 3: Keyword Search First - The Critical Foundation Copy Chapter

Building on Weeks 1-2 foundation: Implement the keyword search foundation that professional RAG systems rely on.

**🎯 Learning Objectives** Copy Section

• Why keyword search is essential for RAG systems (foundation first approach)
• OpenSearch index management, mappings, and search optimization
• BM25 algorithm and the math behind effective keyword search
• Query DSL for building complex search queries with filters and boosting
• Search analytics for measuring relevance and performance
• Production patterns used by real companies

**🏗️ Architecture Overview** Copy Section

Search Infrastructure Components:

• OpenSearch Service: src/services/opensearch/ - Professional search service implementation
• Search API: src/routers/search.py - Search API endpoints with BM25 scoring
• Learning Materials: notebooks/week3/ - Complete OpenSearch integration guide
• Quality Metrics: Precision, recall, and relevance scoring

**📓 Setup Guide** Copy Section

# Launch the Week 3 notebook
uv run jupyter notebook notebooks/week3/week3_opensearch.ipynb

Completion Guide: Follow the Week 3 notebook for hands-on OpenSearch setup and BM25 search implementation.

**📖 Deep Dive** Copy Section

Blog Post: The Search Foundation Every RAG System Needs - Complete BM25 implementation with OpenSearch

📚 Week 4: Chunking & Hybrid Search - The Semantic Layer Copy Chapter

Building on Week 3 foundation: Add the semantic layer that makes search truly intelligent.

**🎯 Learning Objectives** Copy Section

• Section-based chunking with intelligent document segmentation
• Production embeddings with Jina AI integration and fallback strategies
• Hybrid search mastery using RRF fusion for keyword + semantic retrieval
• Unified API design with single endpoint supporting multiple search modes
• Performance analysis and trade-offs between search approaches

**🏗️ Architecture Overview** Copy Section

Hybrid Search Infrastructure Components:

• Text Chunker: src/services/indexing/text_chunker.py - Section-aware chunking with overlap strategies
• Embeddings Service: src/services/embeddings/ - Production embedding pipeline with Jina AI
• Hybrid Search API: src/routers/hybrid_search.py - Unified search API supporting all modes
• Learning Materials: notebooks/week4/ - Complete hybrid search implementation guide

**📓 Setup Guide** Copy Section

# Launch the Week 4 notebook
uv run jupyter notebook notebooks/week4/week4_hybrid_search.ipynb

Completion Guide: Follow the Week 4 notebook for hands-on implementation and verification steps.

**📖 Deep Dive** Copy Section

Blog Post: The Chunking Strategy That Makes Hybrid Search Work - Production chunking and RRF fusion implementation

📚 Week 5: Complete RAG Pipeline with LLM Integration Copy Chapter

Building on Week 4 hybrid search: Add the LLM layer that turns search into intelligent conversation.

**🎯 Learning Objectives** Copy Section

• Local LLM integration with Ollama for complete data privacy
• Performance optimization with 80% prompt reduction (6x speed improvement)
• Streaming implementation using Server-Sent Events for real-time responses
• Dual API design with standard and streaming endpoints
• Interactive Gradio interface with advanced parameter controls

**🏗️ Architecture Overview** Copy Section

Complete RAG Infrastructure Components:

• RAG Endpoints: src/routers/ask.py - Dual endpoints (/api/v1/ask + /api/v1/stream)
• Ollama Service: src/services/ollama/ - LLM client with optimized prompts
• System Prompt: src/services/ollama/prompts/rag_system.txt - Optimized for academic papers
• Gradio Interface: src/gradio_app.py - Interactive web UI with streaming support
• Launcher Script: gradio_launcher.py - Easy-launch script (runs on port 7861)

**📓 Setup Guide** Copy Section

# Launch the Week 5 notebook
uv run jupyter notebook notebooks/week5/week5_complete_rag_system.ipynb

# Launch Gradio interface
uv run python gradio_launcher.py
# Open http://localhost:7861

Completion Guide: Follow the Week 5 notebook for hands-on LLM integration and RAG pipeline implementation.

**📖 Deep Dive** Copy Section

Blog Post: The Complete RAG System - Complete RAG system with local LLM integration and optimization techniques

📚 Week 6: Production Monitoring and Caching Copy Chapter

Building on Week 5 complete RAG system: Add observability, performance optimization, and production-grade monitoring.

**🎯 Learning Objectives** Copy Section

• Langfuse integration for end-to-end RAG pipeline tracing
• Redis caching strategy with intelligent cache keys and TTL management
• Performance monitoring with real-time dashboards for latency and costs
• Production patterns for observability and optimization
• Cost analysis and LLM usage optimization (150-400x speedup with caching)

**🏗️ Architecture Overview** Copy Section

Production Infrastructure Components:

• Langfuse Service: src/services/langfuse/ - Complete tracing integration with RAG-specific metrics
• Cache Service: src/services/cache/ - Redis client with exact-match caching and graceful fallback
• Updated Endpoints: src/routers/ask.py - Integrated tracing and caching middleware
• Docker Config: docker-compose.yml - Added Redis service and Langfuse local instance
• Learning Materials: notebooks/week6/ - Complete monitoring and caching implementation guide

**📓 Setup Guide** Copy Section

# Launch the Week 6 notebook
uv run jupyter notebook notebooks/week6/week6_cache_testing.ipynb

Completion Guide: Follow the Week 6 notebook for hands-on Langfuse tracing and Redis caching implementation.

**📖 Deep Dive** Copy Section

Blog Post: Production-ready RAG: Monitoring & Caching - Production-ready RAG with monitoring and caching

📚 Week 7: Agentic RAG with LangGraph and Telegram Bot Copy Chapter

Building on Week 6 production system: Add intelligent reasoning, multi-step decision-making, and Telegram bot integration for mobile-first AI interactions.

**🎯 Learning Objectives** Copy Section

• LangGraph workflows for state-based agent orchestration with decision nodes
• Guardrail implementation for query validation and domain boundary detection
• Document grading with semantic relevance evaluation
• Query rewriting for automatic query refinement and better retrieval
• Adaptive retrieval with multi-attempt retrieval and intelligent fallback
• Telegram bot integration with async operations and error handling
• Reasoning transparency by exposing agent decision-making process

**🏗️ Architecture Overview** Copy Section

Agentic RAG Infrastructure Components:

• Agent Nodes: src/services/agents/nodes/ - Guardrail, retrieve, grade, rewrite, and generate nodes
• Workflow Orchestration: src/services/agents/agentic_rag.py - LangGraph workflow coordination
• Telegram Bot: src/services/telegram/ - Command handlers and message processing
• Agentic Endpoint: src/routers/agentic_ask.py - Agentic RAG API endpoint
• Learning Materials: notebooks/week7/ - Week 7 learning materials and examples

**📓 Setup Guide** Copy Section

# Launch the Week 7 notebook
uv run jupyter notebook notebooks/week7/week7_agentic_rag.ipynb

Completion Guide: Follow the Week 7 notebook for hands-on LangGraph agentic RAG and Telegram bot implementation.

**📖 Deep Dive** Copy Section

Blog Post: Agentic RAG with LangGraph and Telegram - Building intelligent agents with decision-making, adaptive retrieval, and mobile access

⚙️ Configuration Copy Chapter

Setup:

cp .env.example .env
# Edit .env for your environment

Key Variables:

• JINA_API_KEY - Required for Week 4+ (hybrid search with embeddings)
• TELEGRAM__BOT_TOKEN - Required for Week 7 (Telegram bot integration)
• LANGFUSE__PUBLIC_KEY & LANGFUSE__SECRET_KEY - Optional for Week 6 (monitoring)

Complete Configuration: See .env.example for all available options and detailed documentation.

🔧 Reference & Development Guide Copy Chapter

**🛠️ Technology Stack** Copy Section

Development Tools: UV, Ruff, MyPy, Pytest, Docker Compose

**🏗️ Project Structure** Copy Section

arxiv-paper-curator/
├── src/                    # Main application code
│   ├── routers/            # API endpoints (search, ask, papers)
│   ├── services/           # Business logic (opensearch, ollama, agents, cache)
│   ├── models/             # Database models (SQLAlchemy)
│   ├── schemas/            # Pydantic validation schemas
│   └── config.py           # Environment configuration
├── notebooks/              # Weekly learning materials (week1-7)
├── airflow/                # Workflow orchestration (DAGs)
├── tests/                  # Test suite
└── compose.yml             # Docker service orchestration

**📡 API Endpoints Reference** Copy Section

API Documentation: Visit http://localhost:8000/docs for interactive API explorer

**🔧 Essential Commands** Copy Section

**Using the Makefile** (Recommended)

# View all available commands
make help

# Quick workflow
make start         # Start all services
make health        # Check all services health
make test          # Run tests
make stop          # Stop services

**All Available Commands**

**Direct Commands** (Alternative)

# If you prefer using commands directly
docker compose up --build -d    # Start services
docker compose ps               # Check status
docker compose logs            # View logs
uv run pytest                 # Run tests

**🎓 Target Audience** Copy Section

🛠️ Troubleshooting Copy Chapter

Common Issues:

• Services not starting? Wait 2-3 minutes, check docker compose logs
• Port conflicts? Stop other services using ports 8000, 8080, 5432, 9200
• Memory issues? Increase Docker Desktop memory allocation

Get Help:

• Check the comprehensive Week 1 notebook troubleshooting section
• Review service logs: docker compose logs [service-name]
• Complete reset: docker compose down --volumes && docker compose up --build -d

💰 Cost Structure Copy Chapter

This course is completely free! You'll only need minimal costs for optional services:

• Local Development: $0 (everything runs locally)
• Optional Cloud APIs: ~$2-5 for external LLM services (if chosen)

Star History Copy Chapter

📄 License Copy Chapter

MIT License - see LICENSE file for details.

File: compose.yml Copy Chapter

services:

API service

api:
build: .
container_name: rag-api
ports:
- "8000:8000"
depends_on:
postgres:
condition: service_healthy
opensearch:
condition: service_healthy
redis:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/v1/health')\""]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
env_file:
- .env
environment:
# Container-specific overrides
- OPENSEARCH_HOST=http://opensearch:9200
- OPENSEARCH__HOST=http://opensearch:9200
- OLLAMA_HOST=http://ollama:11434
- POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db
- LANGFUSE_HOST=http://langfuse-web:3000
- LANGFUSE_DEBUG=true
- REDIS__HOST=redis
networks:
- rag-network

redis:
image: redis:7-alpine
container_name: rag-redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 3s
retries: 3
start_period: 10s
restart: unless-stopped
networks:
- rag-network

opensearch:
image: opensearchproject/opensearch:2.19.0
container_name: rag-opensearch
environment:
- discovery.type=single-node
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
- DISABLE_SECURITY_PLUGIN=true
- bootstrap.memory_lock=true
ports:
- "9200:9200"
- "9600:9600"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- opensearch_data:/usr/share/opensearch/data
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
restart: unless-stopped
networks:
- rag-network

opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2.19.0
container_name: rag-dashboards
ports:
- "5601:5601"
environment:
- OPENSEARCH_HOSTS=http://opensearch:9200
- DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
depends_on:
- opensearch
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
networks:
- rag-network

airflow:
build:
context: ./airflow
dockerfile: Dockerfile
container_name: rag-airflow
# user: "50000:0" # Uncomment for WSL/Ubuntu, comment out for Mac
depends_on:
postgres:
condition: service_healthy
env_file:
- .env
environment:
# Airflow-specific environment setup
- AIRFLOW_HOME=/opt/airflow
- PYTHONPATH=/opt/airflow/src
# Container hostnames for Airflow
- POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db
- OPENSEARCH_HOST=http://opensearch:9200
- OPENSEARCH__HOST=http://opensearch:9200
- OLLAMA_HOST=http://ollama:11434
- REDIS__HOST=redis
volumes:
- ./airflow/dags:/opt/airflow/dags
- airflow_logs:/opt/airflow/logs
- ./airflow/plugins:/opt/airflow/plugins
- ./src:/opt/airflow/src
ports:
- "8080:8080"
# Command handled by entrypoint.sh
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 120s
networks:
- rag-network

ollama:
image: ollama/ollama:0.11.2
container_name: rag-ollama
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
networks:
- rag-network

postgres:
image: postgres:16-alpine
container_name: rag-postgres
environment:
- POSTGRES_DB=rag_db
- POSTGRES_USER=rag_user
- POSTGRES_PASSWORD=rag_password
- POSTGRES_HOST_AUTH_METHOD=password
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U rag_user -d rag_db"]
interval: 5s
timeout: 5s
retries: 10
start_period: 30s
restart: unless-stopped
networks:
- rag-network

ClickHouse for Langfuse analytics

clickhouse:
image: clickhouse/clickhouse-server:24.8-alpine
container_name: rag-clickhouse
environment:
- CLICKHOUSE_DB=langfuse
- CLICKHOUSE_USER=langfuse
- CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
- CLICKHOUSE_PASSWORD=langfuse
volumes:
- clickhouse_data:/var/lib/clickhouse
healthcheck:
test: ["CMD", "clickhouse-client", "--query", "SELECT 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
restart: unless-stopped
networks:
- rag-network

langfuse-worker:
image: docker.io/langfuse/langfuse-worker:3
container_name: rag-langfuse-worker
restart: unless-stopped
depends_on:
langfuse-postgres:
condition: service_healthy
langfuse-minio:
condition: service_healthy
langfuse-redis:
condition: service_healthy
clickhouse:
condition: service_healthy
ports:
- "3030:3030"
environment:
NEXTAUTH_URL: http://localhost:3001
DATABASE_URL: postgresql://langfuse:langfuse@langfuse-postgres:5432/langfuse
# Security: Load sensitive values from .env file
SALT: ${LANGFUSE_SALT}
ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY}
TELEMETRY_ENABLED: "false"
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: "true"
CLICKHOUSE_MIGRATION_URL: clickhouse://clickhouse:9000
CLICKHOUSE_URL: http://clickhouse:8123
CLICKHOUSE_USER: langfuse
CLICKHOUSE_PASSWORD: langfuse
CLICKHOUSE_CLUSTER_ENABLED: "false"
LANGFUSE_USE_AZURE_BLOB: "false"
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse
LANGFUSE_S3_EVENT_UPLOAD_REGION: auto
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_ACCESS_KEY}
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_SECRET_KEY}
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://langfuse-minio:9000
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
LANGFUSE_S3_EVENT_UPLOAD_PREFIX: events/
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: langfuse
LANGFUSE_S3_MEDIA_UPLOAD_REGION: auto
LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_ACCESS_KEY}
LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_SECRET_KEY}
LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: http://localhost:9090
LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: media/
REDIS_HOST: langfuse-redis
REDIS_PORT: 6379
REDIS_AUTH: ${LANGFUSE_REDIS_PASSWORD}
REDIS_TLS_ENABLED: "false"
networks:
- rag-network

langfuse-web:
image: docker.io/langfuse/langfuse:3
container_name: rag-langfuse-web
restart: unless-stopped
depends_on:
langfuse-postgres:
condition: service_healthy
langfuse-minio:
condition: service_healthy
langfuse-redis:
condition: service_healthy
clickhouse:
condition: service_healthy
ports:
- "3001:3000"
environment:
NEXTAUTH_URL: http://localhost:3001
# Security: Load sensitive values from .env file
NEXTAUTH_SECRET: ${LANGFUSE_NEXTAUTH_SECRET}
DATABASE_URL: postgresql://langfuse:langfuse@langfuse-postgres:5432/langfuse
SALT: ${LANGFUSE_SALT}
ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY}
TELEMETRY_ENABLED: "false"
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: "true"
CLICKHOUSE_MIGRATION_URL: clickhouse://clickhouse:9000
CLICKHOUSE_URL: http://clickhouse:8123
CLICKHOUSE_USER: langfuse
CLICKHOUSE_PASSWORD: langfuse
CLICKHOUSE_CLUSTER_ENABLED: "false"
LANGFUSE_USE_AZURE_BLOB: "false"
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse
LANGFUSE_S3_EVENT_UPLOAD_REGION: auto
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_ACCESS_KEY}
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_SECRET_KEY}
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://langfuse-minio:9000
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
LANGFUSE_S3_EVENT_UPLOAD_PREFIX: events/
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: langfuse
LANGFUSE_S3_MEDIA_UPLOAD_REGION: auto
LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_MINIO_ACCESS_KEY}
LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_MINIO_SECRET_KEY}
LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: http://localhost:9090
LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: media/
REDIS_HOST: langfuse-redis
REDIS_PORT: 6379
REDIS_AUTH: ${LANGFUSE_REDIS_PASSWORD}
REDIS_TLS_ENABLED: "false"
LANGFUSE_INIT_ORG_ID: "default-org"
LANGFUSE_INIT_ORG_NAME: "RAG Organization"
LANGFUSE_INIT_PROJECT_NAME: "Agentic RAG"
LANGFUSE_INIT_USER_EMAIL: "[email protected]"
LANGFUSE_INIT_USER_NAME: "Admin User"
LANGFUSE_INIT_USER_PASSWORD: "admin123"
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:3000/api/public/health || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
networks:
- rag-network

langfuse-postgres:
image: postgres:17
container_name: rag-langfuse-postgres
restart: unless-stopped
environment:
- POSTGRES_USER=langfuse
- POSTGRES_PASSWORD=langfuse
- POSTGRES_DB=langfuse
- POSTGRES_HOST_AUTH_METHOD=password
- TZ=UTC
- PGTZ=UTC
ports:
- "5433:5432"
volumes:
- langfuse_v3_postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U langfuse -d langfuse"]
interval: 3s
timeout: 3s
retries: 10
start_period: 30s
networks:
- rag-network

langfuse-redis:
image: docker.io/redis:7
container_name: rag-langfuse-redis
restart: unless-stopped
command: --requirepass langfuse_redis_password
ports:
- "6380:6379"
healthcheck:
test: ["CMD", "redis-cli", "-a", "langfuse_redis_password", "ping"]
interval: 3s
timeout: 10s
retries: 10
networks:
- rag-network

langfuse-minio:
image: docker.io/minio/minio
container_name: rag-langfuse-minio
restart: unless-stopped
entrypoint: sh
command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
environment:
- MINIO_ROOT_USER=langfuse_minio
- MINIO_ROOT_PASSWORD=langfuse_minio_secret
ports:
- "9090:9000"
- "9091:9001"
volumes:
- langfuse_v3_minio_data:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 10
start_period: 5s
networks:
- rag-network

volumes:
postgres_data:
opensearch_data:
ollama_data:
airflow_logs:
clickhouse_data:
redis_data:
langfuse_v3_postgres_data:
langfuse_v3_minio_data:

networks:
rag-network:
driver: bridge

File: .pre-commit-config.yaml Copy Chapter

repos:

• repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.11.5
  hooks:
  • id: ruff
    args: [
    "--select=I",
    "--fix"
    ]
  • id: ruff-format
• repo: https://github.com/pre-commit/mirrors-mypy
  rev: 'v1.15.0'
  hooks:
  • id: mypy
    args: [
    --ignore-missing-imports,
    --disable-error-code=import-untyped
    ]

File: airflow/README.md Copy Chapter

Airflow Configuration

This directory contains Apache Airflow configuration and DAGs for the arXiv Paper Curator project.

Current Setup (Week 2) Copy Chapter

Production-Ready DAGs Copy Section

• hello_world_dag.py: Basic health check DAG for Week 1
• arxiv_paper_ingestion.py: Main production DAG for automated arXiv paper fetching and processing

Production Pipeline Features Copy Section

• Daily arXiv ingestion: Automated fetching of CS.AI papers
• PDF processing: Download and parse papers using Docling
• Database storage: Store complete paper metadata and content in PostgreSQL
• Error handling: Comprehensive retry logic and error reporting
• Cross-platform compatibility: Works on macOS, Linux, WSL, and Ubuntu

Directory Structure Copy Chapter

airflow/
├── README.md                           # This file
├── Dockerfile                          # Custom Airflow container with dependencies
├── requirements-airflow.txt            # Python dependencies for DAGs
└── dags/
    ├── hello_world_dag.py             # Week 1 health check DAG
    ├── arxiv_paper_ingestion.py       # Week 2 production ingestion DAG
    └── arxiv_ingestion/
        └── tasks.py                   # Production pipeline tasks with async processing

Docker Configuration Copy Chapter

Cross-Platform Compatibility Copy Section

The Airflow container is configured for cross-platform deployment:

• User Configuration: Runs as airflow user (50000:0) to avoid permission issues
• Volume Management: Uses named volumes for logs to prevent bind mount conflicts
• Database Integration: Connects to shared PostgreSQL instance
• Service Dependencies: Automatic initialization and health checks

Container Features Copy Section

• Python 3.12 with Apache Airflow 2.10.3
• PostgreSQL support via psycopg2
• PDF processing with Docling, Tesseract OCR, and Poppler utilities
• Rate limiting and retry logic for arXiv API compliance
• Async processing for optimal performance with concurrent downloads and parsing

Usage Copy Chapter

Web Interface Copy Section

• URL: http://localhost:8080
• Credentials: Auto-generated during container initialization
• Features: DAG monitoring, task logs, pipeline statistics

Production DAG (`arxiv_paper_ingestion`) Copy Section

1. Environment Setup: Verify services and initialize caching
2. Daily Paper Fetch: Retrieve papers from previous day (10 papers default)
3. PDF Processing: Download and parse PDFs with Docling
4. Failed PDF Retry: Handle any processing failures
5. Database Storage: Store complete paper data with parsed content
6. OpenSearch Placeholders: Prepare for Week 3+ search indexing
7. Daily Report: Generate comprehensive processing statistics

Pipeline Performance Copy Section

• Concurrent Processing: 5 parallel downloads, 1 parsing operation (laptop-optimized)
• Rate Limiting: Respects arXiv API guidelines (3-second delays)
• Caching: PDF files cached locally to avoid re-downloading
• Error Resilience: Continues processing even with individual paper failures

Configuration Copy Chapter

Environment Variables Copy Section

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db
AIRFLOW__CORE__EXECUTOR=LocalExecutor
POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db
PYTHONPATH=/opt/airflow/src

Service Dependencies Copy Section

• PostgreSQL: Paper metadata and content storage
• Source Code: Mounted from ../src for service access
• Shared Network: Communication with API and database services

Week 2 Implementation Status Copy Chapter

✅ Completed Features Copy Section

• Custom Docker container with all dependencies
• Production arXiv ingestion DAG with comprehensive error handling
• Async PDF processing pipeline with concurrency control
• PostgreSQL integration with complete content storage
• Cross-platform compatibility (macOS, Linux, WSL, Ubuntu)
• Rate limiting and retry logic for arXiv API compliance
• Detailed logging and monitoring throughout pipeline

🔄 Week 3+ Roadmap Copy Section

• OpenSearch Integration: Real search indexing (currently placeholders)
• Advanced Scheduling: Multiple collection strategies
• Monitoring & Alerting: Production observability
• Scale Optimization: Higher concurrency for production workloads

File: airflow/requirements-airflow.txt Copy Chapter

Core dependencies needed for Airflow tasks

httpx>=0.27.0
sqlalchemy>=1.4.36,<2.0.0
pydantic>=2.0.0,<3.0.0
python-dateutil>=2.8.0

PDF processing dependencies

docling>=2.0.0

Search engine dependencies

opensearch-py>=2.4.0

Database drivers

psycopg2-binary>=2.9.0

File: notebooks/week1/README.md Copy Chapter

Week 1: Infrastructure Setup and Verification

This folder contains the materials for Week 1 of the arXiv Paper Curator project, which focuses on setting up and verifying the complete infrastructure stack.

Contents Copy Chapter

`week1_setup.ipynb` Copy Section

A comprehensive Jupyter notebook that guides students through:

1. System Requirements and Setup

   • Understanding each technology component and its purpose
   • Cross-platform installation instructions (Windows, macOS, Linux)
   • Prerequisites verification with automated checking

2. Infrastructure Architecture

   • Complete overview of the multi-service architecture
   • Understanding how Docker containers communicate
   • Data persistence and volume management concepts

Architecture Overview:

• FastAPI (Port 8000): REST API with async support and automatic documentation
• PostgreSQL 16 (Port 5432): Primary database for paper metadata and content storage
• OpenSearch 2.19 (Ports 9200, 5601): Hybrid search engine with management dashboards
• Apache Airflow 3.0 (Port 8080): Workflow orchestration with DAGs and PostgreSQL backend
• Ollama (Port 11434): Local LLM server for future RAG implementation
• Docker Network: All services communicate via rag-network with persistent volumes

3. Service-by-Service Setup

   • PostgreSQL database for paper metadata storage
   • OpenSearch for full-text search capabilities
   • Apache Airflow for workflow automation
   • Ollama for local LLM inference
   • FastAPI for REST API endpoints

4. Verification and Testing

   • Automated health checks for all services
   • Step-by-step verification procedures
   • Modular Ollama testing (4 focused test cells)
   • Common troubleshooting scenarios and solutions

Learning Objectives Copy Chapter

By completing this week's materials, students will:

• Understand containerization and Docker Compose orchestration
• Learn how to set up a production-grade infrastructure stack
• Gain experience with database design and API development
• Master troubleshooting techniques for multi-service applications
• Learn direct HTTP API testing vs service abstraction layers
• Build confidence working with professional development tools

Ollama Testing (Simplified for Week 1) Copy Chapter

The notebook includes modular Ollama testing broken into focused cells:

• Test 3A: Check available models
• Test 3B: Simple model testing (if models installed)
• Test 3C: Performance analysis
• Test 3D: Learning notes and setup commands

Easy Model Installation (Optional for Week 1) Copy Section

# Using Makefile (recommended)
make ollama-pull MODEL=llama3.2:1b
make ollama-test MODEL=llama3.2:1b

# Direct HTTP calls for learning
curl -X POST http://localhost:11434/api/pull -d '{"name":"llama3.2:1b"}'
curl -X POST http://localhost:11434/api/generate -d '{"model":"llama3.2:1b","prompt":"Hello","stream":false}'

Recommended Models for Course Copy Section

• llama3.2:1b (1.2GB) - Fast, good for testing
• llama3.2:3b (2.0GB) - Balance of speed/quality
• llama3.1:8b (4.7GB) - Better quality, slower

Note: No models are required for Week 1 - service health check works without them.

Target Audience Copy Chapter

This material is designed for:

• Beginners who want to learn modern software infrastructure
• Students looking to understand how real-world applications are built
• Professionals transitioning into software development or DevOps
• Anyone interested in building their own AI-powered research tools

Time Commitment Copy Chapter

• Setup: 2-3 hours (including software installation and downloads)
• Notebook completion: 1 hours
• Total: 2-4 hours

📖 Additional Resources Copy Chapter

Week 1 Blog Post: The Infrastructure That Powers RAG Systems

• Deep dive into each infrastructure component
• Production deployment considerations
• Architecture decision explanations

Support Resources Copy Chapter

If you encounter issues:

1. Check the troubleshooting sections in the notebook
2. Review the common problems and solutions
3. Ensure all prerequisites are properly installed
4. Follow the step-by-step verification procedures
5. Ask in Jam With AI substack chat channel

Next Steps Copy Chapter

After completing Week 1, you will be ready to:

• Understand how each service contributes to the overall system
• Modify and extend the infrastructure as needed
• Proceed to Week 2: arXiv Integration and PDF Processing
• Build confidence in working with professional development environments

File: notebooks/week2/README.md Copy Chapter

Week 2: arXiv API Integration & PDF Processing

This folder contains the materials for Week 2 of the arXiv Paper Curator project, which focuses on building the core data ingestion pipeline that feeds fresh academic content into our RAG system.

Contents Copy Chapter

`week2_arxiv_integration.ipynb` Copy Section

A comprehensive Jupyter notebook that guides students through:

1. Infrastructure Validation

   • Verify all Week 1 services are running correctly
   • Container health checks and fresh build verification
   • Environment setup for Week 2 components

2. arXiv API Integration

   • Building a robust client with rate limiting and retry logic
   • Implementing date-based filtering for targeted paper retrieval
   • Testing CS.AI category searches with proper API etiquette

Data Pipeline Overview:

• MetadataFetcher: 🎯 Main orchestrator coordinating the entire pipeline
• ArxivClient: Rate-limited fetching with retry logic (3-second delays)
• PDFParserService: Scientific PDF parsing with structured content extraction
• PaperRepository: PostgreSQL integration with upsert operations
• Airflow DAGs: Automated daily ingestion workflows

3. PDF Processing Pipeline

   • Download and cache PDF files with proper error handling
   • Parse scientific PDFs using Docling for structured content extraction
   • Handle parsing failures gracefully with fallback mechanisms

4. Database Integration

   • Store paper metadata and content in PostgreSQL
   • Implement upsert logic to avoid duplicates
   • Test retrieval and query operations

5. Complete Pipeline Testing

   • End-to-end processing from arXiv API to database storage
   • Error handling and graceful degradation testing
   • Performance metrics and success rate analysis

6. Production Readiness

   • Airflow DAG status verification
   • Error logging and monitoring capabilities
   • Ready for automated daily ingestion

Learning Objectives Copy Chapter

By completing this week's materials, students will:

• Master API integration with proper rate limiting and error handling
• Learn PDF processing techniques for scientific documents
• Understand database design patterns for research data storage
• Build robust data pipelines with comprehensive error handling
• Gain experience with async Python programming patterns
• Learn workflow automation concepts with Apache Airflow
• Develop skills in production-grade data processing systems

Key Technologies & Services Copy Chapter

Core Services (Built This Week) Copy Section

• arXiv API Client - Fetches CS.AI papers with intelligent rate limiting
• PDF Parser (Docling) - Extracts structured content from scientific PDFs
• Metadata Fetcher - Orchestrates the complete processing pipeline
• Database Repository - Handles PostgreSQL operations with SQLAlchemy

Infrastructure Dependencies (From Week 1) Copy Section

• PostgreSQL 16 - Paper metadata and content storage
• FastAPI - REST API endpoints for paper retrieval
• Apache Airflow - Workflow orchestration and scheduling
• Docker Compose - Service orchestration and networking

Pipeline Architecture Copy Chapter

arXiv Search Query → Rate Limited API Calls → PDF Downloads → Docling Parsing → Database Storage
        ↓                    ↓                    ↓              ↓               ↓
   Date Filtering    →  Retry Logic       →   Caching      → Structure    →  Upsert Logic
   Category Filter   →  Error Handling    →   Validation   → Extraction   →  Transactions
   Result Limiting   →  3s Rate Limit     →   Size Checks  → Metadata     →  Relationships

Performance Characteristics Copy Chapter

Week 2 System Capabilities:

• arXiv API: ~20 papers/minute (respecting 3-second rate limits)
• PDF Processing: 2-5 seconds per paper (depends on PDF complexity)
• Database Storage: ~100 papers/second (batch operations)
• Error Handling: Graceful continuation despite individual failures
• Success Rates: 95%+ for paper fetching, 80-90% for PDF parsing

Target Audience Copy Chapter

This material is designed for:

• Data Engineers learning research data pipeline construction
• Students interested in academic research automation
• Developers building content aggregation systems
• Researchers wanting to automate literature discovery
• Anyone building production data ingestion pipelines

Time Commitment Copy Chapter

• Fresh container build: 10-15 minutes (required for Week 2 dependencies)
• Notebook completion: 45-60 minutes
• Pipeline testing: 30-45 minutes
• Total: 1.5-2 hours

📖 Additional Resources Copy Chapter

Week 2 Blog Post: Building Robust Data Pipelines for Academic Research

• Deep dive into arXiv API best practices
• PDF processing strategies for scientific documents
• Error handling patterns for production systems
• Database design for research metadata

Important Notes Copy Chapter

Fresh Container Build Required Copy Section

Week 2 requires rebuilding containers with new dependencies:

# Shutdown and rebuild (REQUIRED for Week 2)
docker compose down
docker compose up --build

# This ensures:
# - New Python dependencies (docling, arxiv client)
# - Updated Airflow DAGs
# - Fresh service configurations

PDF Processing Expectations Copy Section

• Not all PDFs parse successfully (expected behavior)
• Docling works best with standard academic paper formats
• System handles failures gracefully and continues processing
• Success rates of 80-90% are normal for academic PDFs

Support Resources Copy Chapter

If you encounter issues:

1. Ensure fresh containers are built (docker compose up --build)
2. Check the troubleshooting sections in the notebook
3. Review service health checks and logs
4. Verify all Week 1 infrastructure is working
5. Ask in Jam With AI substack chat channel

Next Steps Copy Chapter

After completing Week 2, you will be ready to:

• Understand production data pipeline architecture
• Handle real-world API integration challenges
• Implement robust error handling and monitoring
• Proceed to Week 3: OpenSearch Integration and Full-Text Search
• Build confidence in handling complex data processing workflows

Success Criteria Copy Chapter

✅ Week 2 Complete When:

• arXiv API client fetches papers with proper rate limiting
• PDF download and caching works reliably
• Docling parser extracts structured content from scientific papers
• Database stores complete paper metadata with relationships
• Complete pipeline processes papers end-to-end with error handling
• Airflow DAGs are configured and ready for automation
• All components demonstrate production-grade error handling and monitoring

File: notebooks/week3/README.md Copy Chapter

Week 3: Keyword Search First - The Critical Foundation

🚨 The 90% Problem: Most RAG systems jump straight to vector search and miss the foundation that powers the best retrieval systems. We're doing it right!

This folder contains the materials for Week 3 of the arXiv Paper Curator project, where we implement the keyword search foundation that professional RAG systems rely on using OpenSearch and BM25 scoring.

🎯 Why Keyword Search First? Copy Chapter

The Professional Path: Unlike tutorials that jump straight to vector embeddings, we build the foundation that successful companies use:

1. 🔍 Exact Match Power: Keywords excel at finding specific technical terms, paper IDs, and precise phrases
2. 📊 Interpretable Results: You can understand exactly why a paper was retrieved
3. ⚡ Speed & Efficiency: BM25 is computationally fast and doesn't require expensive embedding models
4. 📈 Production Reality: Companies like Elasticsearch, Algolia, and enterprise search use keyword search as their foundation

The Learning Path:

Week 3: Master BM25 keyword search    ← YOU ARE HERE
Week 4: Add intelligent chunking
Week 5: Introduce vector embeddings for hybrid retrieval  
Week 6: Optimize the complete system

🚀 Before You Start Copy Chapter

Essential Environment Setup:

# 1. Ensure you have the correct environment configuration
cp .env.example .env

# 2. Verify OpenSearch settings are properly configured
# Your .env should contain these critical Week 3 settings:
# OPENSEARCH__HOST=http://opensearch:9200
# OPENSEARCH__INDEX_NAME=arxiv-papers

Important: Week 3 requires the .env file to be properly configured for OpenSearch connectivity and indexing. The defaults in .env.example work out of the box.

Contents Copy Chapter

`week3_opensearch.ipynb` Copy Section

A comprehensive Jupyter notebook that guides you through building the keyword search foundation:

1. Infrastructure Validation

   • Verify all Week 1-2 services are running correctly
   • OpenSearch cluster health and connectivity testing
   • Environment setup for Week 3 search components

2. OpenSearch Service Integration

   • Building a production-grade search client with factory patterns
   • Implementing dependency injection with FastAPI lifespan management
   • Creating JSON-based index configurations with proper analyzers

🔍 OpenSearch Dashboard in Action Copy Chapter

You'll add your OpenSearch dashboard screenshot here showing real paper searches with BM25 scoring and result rankings.

Search Architecture Overview:

• OpenSearch Client: Factory-pattern service with health monitoring and CRUD operations
• Query Builder: Advanced search query construction with field boosting and filtering
• Index Management: JSON-based schema with English analyzers and strict mapping
• Search API: RESTful endpoints (/search) with GET/POST methods for different query types
• BM25 Algorithm: Industry-standard text relevance scoring with multi-field search

3. Index Management & Schema Design

   • Creating indexes with proper field mappings for academic papers
   • Implementing English language analyzers for improved search relevance
   • Testing index health, statistics, and document management operations

4. Document Indexing Pipeline

   • Migrating paper data from PostgreSQL to OpenSearch
   • Bulk indexing operations with error handling and validation
   • Real-time indexing verification and performance monitoring

5. BM25 Search Implementation

   • Multi-field search across title, abstract, and content with field boosting
   • Advanced query features: highlighting, pagination, category filtering
   • Testing two-letter queries (AI, ML, NN, CV) as specifically requested
   • Fuzzy matching and typo tolerance configuration

6. Search API Development

   • FastAPI integration with OpenSearch dependency injection
   • RESTful endpoints: GET /search for simple queries, POST /search for advanced search
   • Response formatting with pagination, metadata, and highlighted snippets
   • Error handling and service health monitoring endpoints

7. Airflow Pipeline Updates

   • Sequential task execution: setup → fetch → opensearch → report → cleanup
   • Real OpenSearch indexing replacing placeholder operations
   • Updated daily reporting with search statistics and health metrics
   • Graceful handling of PDF processing limits (>20MB or >30 pages)

8. End-to-End Pipeline Verification

   • Complete flow testing: arXiv API → PostgreSQL → OpenSearch → Search API
   • Performance benchmarking and response time measurement
   • Production readiness assessment with monitoring and alerting

Week 3 Architecture:

Key Features Implemented Copy Chapter

🔍 **Production-Grade Search System** Copy Section

• BM25 Scoring: Industry-standard relevance ranking algorithm
• Multi-field Search: Query across title (3x boost), abstract (2x boost), and content (1x boost)
• Advanced Filtering: Category filtering, date ranges, and custom field queries
• Real-time Highlighting: Search term highlighting in results with HTML markup
• Pagination Support: Efficient large result set handling with size/from parameters

🏗️ **Clean Architecture Implementation** Copy Section

• Factory Pattern: Consistent service creation across all components
• Dependency Injection: Proper service lifecycle management with FastAPI lifespan
• Query Builder Pattern: Flexible search query construction for complex operations
• Error Handling: Comprehensive exception management with graceful degradation

📊 **Performance & Monitoring** Copy Section

• Sub-100ms Search: Optimized query performance for real-time applications
• Health Monitoring: Cluster status, index statistics, and service availability
• Resource Limits: PDF processing constraints to prevent memory issues
• Scalability: Design patterns supporting horizontal scaling

🔧 **Development Best Practices** Copy Section

• Type Safety: Full type hints throughout the codebase
• Testing: End-to-end verification with realistic data scenarios
• Documentation: Comprehensive docstrings and architectural explanations
• Code Quality: Following established patterns from Week 1-2 implementations

Expected Outcomes Copy Chapter

By completing Week 3, students will have:

1. Working Search System: Fully functional OpenSearch BM25 search with 28+ indexed papers
2. RESTful API: Complete search API with both simple and advanced query capabilities
3. Production Architecture: Clean, scalable code following industry best practices
4. Performance Insights: Understanding of search optimization and monitoring strategies
5. End-to-End Pipeline: Complete RAG system foundation ready for Week 4 development

Success Criteria Copy Chapter

• ✅ OpenSearch Integration: Factory pattern service with health checks operational
• ✅ BM25 Search: Relevance scoring with multi-field queries returning relevant results
• ✅ Advanced Features: Filtering, highlighting, pagination, and fuzzy search working
• ✅ API Endpoints: RESTful search with both GET and POST methods functional
• ✅ Two-Letter Queries: AI, ML, NN, CV queries working as specifically requested
• ✅ Pipeline Integration: Airflow DAGs updated for OpenSearch indexing
• ✅ Performance: Sub-100ms search response times with proper error handling

What's Next? Copy Chapter

Week 4 Preview: Chunking Strategies and Hybrid Retrieval

• Intelligent document chunking strategies for better context preservation
• Combining keyword search (BM25) with vector embeddings for hybrid retrieval
• Building the complete foundation for modern RAG systems
• Evaluation metrics to measure retrieval quality

Ready to build production-grade search? Let's get started with Week 3! 🔍✨

File: notebooks/week4/README.md Copy Chapter

Week 4: Document Chunking and Hybrid Search

Overview Copy Chapter

Week 4 implements a production-grade hybrid search system that combines the precision of BM25 keyword search with the semantic understanding of vector embeddings. This system provides the foundation for retrieval-augmented generation (RAG) by intelligently breaking documents into searchable chunks and enabling multiple search modes.

What We Built Copy Chapter

🧩 **Section-Based Document Chunking** Copy Section

• Intelligent Segmentation: Leverages document structure (parsed sections) for natural chunk boundaries
• Context Preservation: 100-word overlaps between chunks maintain semantic continuity
• Adaptive Processing: Handles both structured (with sections) and unstructured (paragraph-based) documents
• Optimal Sizing: Targets 600-word chunks with minimum 100-word threshold

🔍 **Unified Hybrid Search System** Copy Section

• Single Index Architecture: One OpenSearch index (arxiv-papers-chunks) supports all search modes
• Multiple Search Types:
  • BM25 Keyword Search: Fast (~50ms) traditional text matching
  • Vector Similarity Search: Semantic search using 1024-dimensional embeddings
  • Hybrid Search: RRF (Reciprocal Rank Fusion) combining both approaches
• Production API: RESTful endpoint /api/v1/hybrid-search/ with comprehensive validation

🤖 **Real Embedding Integration** Copy Section

• Jina AI Embeddings: Production-grade 1024-dimensional vectors optimized for retrieval
• Automatic Generation: FastAPI endpoints automatically generate query embeddings
• Fallback Strategy: Graceful degradation to BM25 when embeddings unavailable
• Performance Optimized: Efficient embedding generation and storage

Architecture Copy Chapter

System Overview Copy Section

Data Flow Copy Section

Raw Papers → PDF Parsing → Section Extraction → Chunking → Embedding → Indexing → Search

The diagram above illustrates the complete Week 4 implementation:

• Data Processing Pipeline: arXiv papers flow through chunking and embedding generation
• Unified OpenSearch Index: Single index supporting BM25, vector, and hybrid search modes
• Hybrid Retrieval Pipeline: RRF fusion combining keyword precision with semantic understanding
• Production API Layer: FastAPI endpoints with automatic embedding generation

System Components Copy Section

**1. Document Processing Pipeline**

# Located in: src/services/indexing/text_chunker.py
TextChunker.chunk_paper(
    title="Paper Title",
    abstract="Abstract text",
    full_text="Complete paper content",
    sections=parsed_sections_dict,
    target_words=600,
    overlap_words=100
)

**2. Embedding Service**

# Located in: src/services/embeddings/factory.py
embeddings_service = make_embeddings_service()
vectors = await embeddings_service.embed_query(["query text"])

**3. Unified Search Client**

# Located in: src/services/opensearch/client.py
results = opensearch_client.search_unified(
    query="machine learning",
    query_embedding=vector,
    use_hybrid=True,
    size=10
)

**4. Production API**

# Located in: src/routers/hybrid_search.py
POST /api/v1/hybrid-search/
{
  "query": "neural networks",
  "use_hybrid": true,
  "size": 5
}

Key Features Copy Chapter

**Hybrid Search Modes** Copy Section

**RRF (Reciprocal Rank Fusion)** Copy Section

• Algorithm: Combines BM25 and vector search rankings using reciprocal rank fusion
• Implementation: Manual fusion algorithm (OpenSearch 2.19 compatibility)
• Weighting: Configurable balance between keyword and semantic relevance
• Fallback: Automatic fallback to BM25 if vector search fails

**Section-Based Chunking Strategy** Copy Section

# Chunking Parameters (optimized through testing)
CHUNK_SIZE = 600        # Target words per chunk
OVERLAP_SIZE = 100      # Words overlapping between chunks  
MIN_CHUNK_SIZE = 100    # Minimum viable chunk size
SECTION_BASED = True    # Use document structure when available

Benefits:

• Semantic Coherence: Chunks respect natural document boundaries
• Context Preservation: Overlaps prevent information loss at boundaries
• Retrieval Accuracy: Better matching of user queries to relevant content
• Scalability: Handles documents from 1,000 to 100,000+ words

Implementation Details Copy Chapter

**OpenSearch Index Configuration** Copy Section

Index Name: arxiv-papers-chunks

Key Fields:

{
  "arxiv_id": "2508.18563v1",
  "title": "Paper title",
  "chunk_text": "Chunk content...",
  "chunk_id": "unique_chunk_identifier", 
  "section_name": "Introduction",
  "embedding": [0.123, 0.456, ...],  // 1024 dimensions
  "paper_categories": ["cs.AI", "cs.LG"],
  "published_date": "2025-08-25T23:43:33"
}

**Search Query Structure** Copy Section

BM25 Query (keyword matching):

{
  "query": {
    "bool": {
      "should": [
        {"match": {"chunk_text": {"query": "machine learning", "fuzziness": "AUTO"}}},
        {"match": {"title": {"query": "machine learning", "boost": 2.0}}},
        {"match": {"abstract": {"query": "machine learning", "boost": 1.5}}}
      ]
    }
  }
}

Hybrid Query (RRF manual fusion):

1. Execute BM25 query → get ranked results
2. Execute vector query → get ranked results
3. Apply RRF fusion algorithm → combine rankings
4. Return merged results with hybrid scores

**Environment Configuration** Copy Section

Required Variables:

# Core Services
POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db
OPENSEARCH__HOST=http://opensearch:9200

# Embeddings (Required for Hybrid Search)
JINA_API_KEY=jina_your_api_key_here

# Chunking Configuration
CHUNKING__CHUNK_SIZE=600
CHUNKING__OVERLAP_SIZE=100
CHUNKING__MIN_CHUNK_SIZE=100
CHUNKING__SECTION_BASED=true

# OpenSearch Configuration
OPENSEARCH__INDEX_NAME=arxiv-papers
OPENSEARCH__CHUNK_INDEX_SUFFIX=chunks
OPENSEARCH__VECTOR_DIMENSION=1024

API Reference Copy Chapter

**Hybrid Search Endpoint** Copy Section

Endpoint: POST /api/v1/hybrid-search/

Request Body:

{
  "query": "transformer neural networks",
  "use_hybrid": true,
  "size": 10,
  "from": 0,
  "categories": ["cs.AI", "cs.LG"],
  "latest_papers": false,
  "min_score": 0.0
}

Response:

{
  "query": "transformer neural networks",
  "total": 15,
  "hits": [
    {
      "arxiv_id": "2508.18563v1",
      "title": "Paper Title",
      "authors": "Author Names",
      "abstract": "Paper abstract...",
      "score": 0.8542,
      "chunk_text": "Relevant chunk content...",
      "chunk_id": "chunk_uuid",
      "section_name": "Related Work"
    }
  ],
  "size": 10,
  "from": 0,
  "search_mode": "hybrid"
}

Performance Benchmarks Copy Chapter

Test Environment: 3 papers, 81 chunks, single-node OpenSearch

Key Insights:

• Hybrid search provides best relevance at cost of response time
• BM25 excellent for high-throughput keyword matching
• Vector search good semantic understanding with moderate speed
• Embedding generation accounts for ~2s of hybrid search time

Production Deployment Copy Chapter

**Scaling Considerations** Copy Section

OpenSearch Cluster:

# Recommended minimum for production
opensearch:
  image: opensearchproject/opensearch:2.19.0
  environment:
    - cluster.name=rag-cluster
    - node.name=rag-node-1
    - discovery.type=single-node
    - OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g
  deploy:
    resources:
      limits:
        memory: 2G
      reservations:
        memory: 1G

Embedding Service Optimization:

• Batch Processing: Process multiple embeddings per API call
• Caching: Cache frequent query embeddings
• Rate Limiting: Respect Jina AI API limits (1000 requests/minute)
• Fallback Strategy: BM25-only mode when embeddings unavailable

**Monitoring and Observability** Copy Section

Key Metrics:

• Search request latency (p50, p95, p99)
• Embedding generation success rate
• Index document count and size
• Search mode usage distribution (BM25 vs Hybrid)

Health Checks:

• OpenSearch cluster health
• Embedding service availability
• Index document count validation
• Sample query execution

Troubleshooting Copy Chapter

**Common Issues** Copy Section

1. Hybrid Search Returns BM25 Mode

# Check embedding service
curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \
  -H "Content-Type: application/json" \
  -d '{"query": "test", "use_hybrid": true}'

# Check logs for embedding errors
docker compose logs api | grep -i embedding

2. Empty Search Results

# Verify index exists and has documents
curl "http://localhost:9200/arxiv-papers-chunks/_count"

# Check index mapping
curl "http://localhost:9200/arxiv-papers-chunks/_mapping"

3. Slow Embedding Generation

# Check Jina API key configuration
docker compose exec api env | grep JINA

# Test direct embedding service
curl -X POST "https://api.jina.ai/v1/embeddings" \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -d '{"model": "jina-embeddings-v3", "input": ["test"]}'

Testing Copy Chapter

**Running the Week 4 Notebook** Copy Section

1. Start Services:

docker compose up --build -d

2. Open Notebook:

cd notebooks/week4
jupyter notebook week4_hybrid_search.ipynb

3. Execute All Cells: The notebook includes:
   • Environment setup and health checks
   • Section-based chunking demonstration
   • Real embedding generation with Jina AI
   • All search modes (BM25, Vector, Hybrid)
   • Production API endpoint testing
   • Performance comparison

**Manual Testing** Copy Section

Test BM25 Search:

curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \
  -H "Content-Type: application/json" \
  -d '{"query": "machine learning", "use_hybrid": false, "size": 3}'

Test Hybrid Search:

curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \
  -H "Content-Type: application/json" \
  -d '{"query": "neural networks", "use_hybrid": true, "size": 3}'

Next Steps (Week 5) Copy Chapter

Week 4 provides the search foundation for Week 5's LLM integration:

1. LLM Integration: Connect Ollama for answer generation
2. RAG Pipeline: Query → Search → Context → Generate → Response
3. Context Management: Optimize retrieved chunks for LLM input
4. Answer Quality: Implement citation and source attribution
5. Conversation Memory: Support multi-turn conversations

The hybrid search system is production-ready and provides the retrieval accuracy needed for high-quality RAG applications.

File Structure Copy Chapter

src/
├── routers/
│   └── hybrid_search.py          # FastAPI endpoints
├── services/
│   ├── opensearch/
│   │   ├── client.py              # Unified search client
│   │   ├── factory.py             # Client factory
│   │   └── index_config_hybrid.py # Index configuration
│   ├── indexing/
│   │   ├── text_chunker.py        # Section-based chunking
│   │   ├── hybrid_indexer.py      # Document indexing
│   │   └── factory.py             # Indexing service factory
│   └── embeddings/
│       ├── jina_client.py         # Jina AI client
│       └── factory.py             # Embedding service factory
├── schemas/
│   └── api/
│       └── search.py              # Request/response models
└── config.py                      # Configuration management

notebooks/week4/
├── README.md                      # This document
├── week4_hybrid_search.ipynb      # Interactive tutorial
└── data/                          # Sample data directory

Resources Copy Chapter

• OpenSearch Documentation: https://opensearch.org/docs/
• Jina AI Embeddings: https://jina.ai/embeddings/
• FastAPI Documentation: https://fastapi.tiangolo.com/
• Reciprocal Rank Fusion Paper: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
• Week 4 Notebook: week4_hybrid_search.ipynb

File: notebooks/week5/README.md Copy Chapter

Week 5: Complete RAG System with LLM Integration

Overview Copy Chapter

Week 5 completes our production-grade RAG system by integrating Ollama LLM with hybrid search. The system delivers 6x faster performance (120s → 15-20s), real-time streaming, and includes a Gradio web interface.

What We Built Copy Chapter

• Local LLM Integration: Ollama service with llama3.2 models
• Performance Optimization: 80% prompt reduction, 6x speed improvement
• Streaming API: Real-time responses via Server-Sent Events
• Gradio Interface: Interactive web UI with streaming support
• Production Ready: Clean API design with two focused endpoints

Architecture Copy Chapter

Quick Start Copy Chapter

1. Start Services Copy Section

docker compose up --build -d

2. Test RAG Endpoint Copy Section

curl -X POST "http://localhost:8000/api/v1/ask" \
  -H "Content-Type: application/json" \
  -d '{"query": "What are transformers?", "top_k": 3, "use_hybrid": true}'

3. Launch Gradio Interface Copy Section

uv run python gradio_launcher.py
# Open http://localhost:7861

API Endpoints Copy Chapter

Standard RAG - `/api/v1/ask` Copy Section

• Purpose: Complete response with metadata
• Response Time: 15-20 seconds
• Use Case: Batch processing, API integrations

Streaming RAG - `/api/v1/stream` Copy Section

• Purpose: Real-time token generation
• Time to First Token: 2-3 seconds
• Use Case: Interactive UIs, better UX

Request Format Copy Section

{
    "query": "Your question",
    "top_k": 3,              // Chunks to retrieve (1-10)
    "use_hybrid": true,      // BM25 + vector search
    "model": "llama3.2:1b",  // LLM model
    "categories": ["cs.AI"]  // Optional filter
}

Performance Copy Chapter

Key Optimizations:

• Removed redundant metadata (80% prompt reduction)
• Shared code architecture (DRY principles)
• 300-word response limit for focused answers
• Automatic source deduplication

Configuration Copy Chapter

# .env file
OLLAMA_HOST=http://ollama:11434
OLLAMA__DEFAULT_MODEL=llama3.2:1b
JINA_API_KEY=your_key_here  # For embeddings

Testing Copy Chapter

Run the Notebook Copy Section

jupyter notebook notebooks/week5/week5_complete_rag_system.ipynb

Test Streaming Copy Section

curl -X POST "http://localhost:8000/api/v1/stream" \
  -H "Content-Type: application/json" \
  -d '{"query": "Explain attention mechanism", "top_k": 2}' \
  --no-buffer

Troubleshooting Copy Chapter

Project Structure Copy Chapter

src/
├── routers/
│   └── ask.py              # RAG endpoints
├── services/
│   └── ollama/
│       ├── client.py       # LLM client
│       └── prompts/        # System prompts
├── gradio_app.py           # Web interface
└── gradio_launcher.py      # Launcher script

notebooks/week5/
├── README.md               # This file
└── week5_complete_rag_system.ipynb

Next Steps Copy Chapter

• Enhance: Add conversation memory, feedback loops
• Optimize: Implement caching, fine-tune models
• Deploy: Add authentication, monitoring, load balancing

Resources Copy Chapter

• Notebook Tutorial
• API Documentation
• Gradio Interface
• Ollama Models

File: notebooks/week6/README.md Copy Chapter

Week 6: Production Monitoring and Caching with Langfuse and Redis

Overview Copy Chapter

Week 6 adds production-grade monitoring and intelligent caching to our RAG system. We integrate Langfuse for complete pipeline observability and Redis for high-performance response caching.

What We Built Copy Chapter

• Langfuse Integration: End-to-end RAG pipeline tracing and analytics
• Redis Caching: 150-400x faster responses for repeated queries
• Performance Monitoring: Real-time metrics and system health
• Production Ready: Enterprise-grade observability and optimization

Architecture Copy Chapter

Data Flow Copy Section

Query → Cache Check → [Hit: ~100ms] | [Miss: Full Pipeline ~15s] → Cache Store → Langfuse Trace

Key Features Copy Chapter

**Langfuse Observability** Copy Section

• Complete RAG pipeline tracing with performance breakdowns
• User analytics, query patterns, and success rate tracking
• Real-time monitoring dashboard with cost and usage metrics
• Quality insights with answer relevance and source attribution

**Redis Intelligent Caching** Copy Section

• Exact-Match Strategy: Parameter-aware cache keys for precise matching
• Performance: 150-400x faster responses for repeated queries (~100ms vs 15-20s)
• TTL Management: 24-hour default expiration with configurable settings
• Future Enhancement: Can be upgraded to semantic similarity caching for fuzzy matching

Quick Start Copy Chapter

Environment Setup Copy Section

# Required environment variables
LANGFUSE__SECRET_KEY=sk_lf_your_secret_key
LANGFUSE__PUBLIC_KEY=pk_lf_your_public_key
REDIS__HOST=redis
REDIS__TTL_HOURS=24

Start Services Copy Section

docker compose up --build -d

Test Caching Performance Copy Section

# First request (cache miss ~15-20s)
curl -X POST "http://localhost:8000/api/v1/ask" \
  -H "Content-Type: application/json" \
  -d '{"query": "What are transformers?", "top_k": 3}'

# Second identical request (cache hit ~100ms)
curl -X POST "http://localhost:8000/api/v1/ask" \
  -H "Content-Type: application/json" \
  -d '{"query": "What are transformers?", "top_k": 3}'

Performance Benchmarks Copy Chapter

Testing Copy Chapter

Run the Notebook Copy Section

jupyter notebook notebooks/week6/week6_cache_testing.ipynb

Monitor System Health Copy Section

# Check Redis connectivity
redis-cli ping

# View cache statistics  
curl "http://localhost:8000/api/v1/health"

# Access Langfuse dashboard
# Visit: https://cloud.langfuse.com (or your self-hosted instance)

Troubleshooting Copy Chapter

Next Steps Copy Chapter

• Enhanced Caching: Upgrade to semantic similarity caching for fuzzy matching
• Advanced Analytics: Custom dashboards and A/B testing frameworks
• Production Scaling: Distributed caching and automated monitoring
• Quality Optimization: User feedback integration and answer scoring

Resources Copy Chapter

• Notebook: week6_cache_testing.ipynb
• Langfuse Dashboard: https://cloud.langfuse.com
• Redis Documentation: https://redis.io/docs

Week 6 transforms your RAG system into a production-grade service with 150-400x performance improvements and comprehensive observability.

File: notebooks/week7/README.md Copy Chapter

Week 7: Agentic RAG with LangGraph + Telegram Bot

Overview Copy Chapter

Week 7 adds TWO major enhancements to the arXiv Paper Curator:

1. 🤖 Agentic RAG with LangGraph - Intelligent, adaptive retrieval with decision-making
2. 💬 Telegram Bot Integration - Conversational interface for mobile/desktop access

🧠 Part 1: Agentic RAG with LangGraph Copy Chapter

What is Agentic RAG? Copy Section

Traditional RAG (Weeks 5-6):

Query → Always Retrieve → Generate Answer

Agentic RAG (Week 7):

Query → Agent Decides:
  ├─ Simple question? → Respond directly (faster!)
  └─ Research needed? → Retrieve
       ├─ Relevant docs? → Generate answer
       └─ Not relevant? → Rewrite query → Try again

Key Features Copy Section

• 🎯 Intelligent Decision Making - LLM decides when retrieval is actually needed
• 📊 Document Grading - Validates that retrieved papers are relevant
• 🔄 Query Refinement - Rewrites vague queries for better results
• 🔍 Reasoning Transparency - Shows the agent's decision-making steps
• ♻️ Iterative Improvement - Can retry with better queries if needed

What We Built Copy Section

src/services/agents/
├── tools.py            # Retriever tool wrapping OpenSearch
├── nodes.py            # 4 graph nodes (query, grade, rewrite, generate)
├── agentic_rag.py      # LangGraph workflow + service
├── prompts.py          # LLM prompt templates
└── factory.py          # Dependency injection

src/routers/
└── agentic_ask.py      # FastAPI endpoint

Total: ~750 LOC following SOLID, KISS, DRY principles

Architecture Copy Section

LangGraph Workflow:

START
  ↓
generate_query_or_respond
  ├─ No retrieval needed → END (direct response)
  └─ Needs retrieval → retrieve (ToolNode)
       ↓
     grade_documents
       ├─ Relevant → generate_answer → END
       └─ Not relevant → rewrite_query → (loop back)

New API Endpoint Copy Section

POST /api/v1/ask-agentic

// Request
{
  "query": "What are transformers in ML?",
  "top_k": 3,
  "use_hybrid": true
}

// Response
{
  "query": "What are transformers in ML?",
  "answer": "Transformers are neural network architectures...",
  "sources": ["https://arxiv.org/pdf/1706.03762.pdf"],
  "chunks_used": 3,
  "search_mode": "hybrid",
  "reasoning_steps": [
    "Decided to retrieve relevant papers",
    "Retrieved documents from database",
    "Generated answer from relevant documents"
  ],
  "retrieval_attempts": 1
}

Quick Start: Agentic RAG Copy Section

1. Ensure services are running:

docker compose up --build -d

2. Test with cURL:

# Simple question (should respond directly)
curl -X POST http://localhost:8000/api/v1/ask-agentic \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is 2+2?",
    "top_k": 3,
    "use_hybrid": true
  }'

# Research question (should retrieve papers)
curl -X POST http://localhost:8000/api/v1/ask-agentic \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What are attention mechanisms?",
    "top_k": 3,
    "use_hybrid": true
  }'

3. Interactive Testing:

# Open Jupyter notebook
jupyter notebook notebooks/week7/week7_agentic_rag.ipynb

Comparison: Traditional vs. Agentic Copy Section

Testing Scenarios Copy Section

Scenario 1: Direct Response (No Retrieval)

• Query: "What is 5 + 7?"
• Expected: Agent responds "12" without retrieving papers
• Reasoning: "Responded directly without retrieval"

Scenario 2: Successful Retrieval

• Query: "What are transformers in machine learning?"
• Expected: Agent retrieves papers, grades as relevant, generates answer
• Reasoning: "Decided to retrieve" → "Retrieved documents" → "Generated answer"

Scenario 3: Query Rewriting

• Query: "Tell me about ML stuff" (vague)
• Expected: Agent retrieves, grades as not relevant, rewrites query, tries again
• Reasoning: "Retrieved" → "Not relevant" → "Rewritten query" → "Retrieved again" → "Generated answer"

Design Principles Followed Copy Section

• ✅ SOLID - Single responsibility, dependency inversion, composition
• ✅ KISS - Simple nodes (<30 lines), clear logic
• ✅ DRY - Reused existing services (OpenSearch, Ollama, Jina)
• ✅ YAGNI - Only implemented what's needed
• ✅ Explicit - Type hints, docstrings, clear names
• ✅ 2025 Best Practices - MessagesState, ToolNode, tools_condition

Documentation Copy Section

• Implementation Plan: docs/AGENTIC_RAG_IMPLEMENTATION_PLAN.md
• Testing Plan: docs/AGENTIC_RAG_TESTING_PLAN.md
• LangGraph 2025 Patterns: docs/LANGGRAPH_2025_BEST_PRACTICES.md
• Design Principles: docs/DESIGN_PRINCIPLES.md
• Interactive Notebook: notebooks/week7/week7_agentic_rag.ipynb

💬 Part 2: Telegram Bot Integration Copy Chapter

What We Built Copy Section

• 🤖 Full Telegram Bot Integration: Conversational interface with command support
• 💬 Natural Language Queries: Ask questions in plain language, get answers with sources
• ⚡ All Week 6 Features: Redis caching (150-400x speedup) and Langfuse tracing
• 🎯 Interactive Commands: /start, /help, /ask, /search, /settings, /status
• 👤 User Session Management: Per-user preferences and conversation history
• 📱 Mobile-First: Rich message formatting with clickable arXiv links
• 🔐 Optional Access Control: Whitelist specific users if needed

Architecture Copy Chapter

Data Flow Copy Section

Telegram User
    ↓
Telegram Bot (Polling/Webhook)
    ↓
TelegramService + Handlers
    ↓ [Langfuse Tracing]
Cache Check (Redis)
    ├─ Hit → Instant Response (~100ms)
    └─ Miss → Full RAG Pipeline
        ↓
Hybrid Search (OpenSearch BM25 + Vector)
        ↓
LLM Generation (Ollama)
        ↓
Cache Store (Redis)
        ↓
Format Response → Send to Telegram

New Components Copy Section

src/services/telegram/
├── client.py           # Main Telegram bot service
├── handlers.py         # Command and message handlers
├── formatters.py       # Message formatting utilities
├── keyboards.py        # Interactive inline keyboards
├── user_manager.py     # User settings and sessions
└── factory.py          # Factory function

src/schemas/telegram/
├── messages.py         # Message validation schemas
├── commands.py         # Command schemas
└── user_settings.py    # User preferences schema

Key Features Copy Chapter

**Conversational Interface** Copy Section

• Natural language queries: Just send a message
• Contextual conversation with history tracking
• Automatic query routing (commands vs regular messages)

**Rich Commands** Copy Section

/start    - Welcome message with bot capabilities
/help     - Detailed usage instructions
/ask      - Explicitly ask a research question
/search   - Quick paper search by keywords
/settings - Customize preferences (search mode, results count)
/status   - Check system health and statistics
/clear    - Clear conversation history

**Interactive Settings** Copy Section

• Search mode: Hybrid (BM25 + semantic) or BM25 only
• Results per query: 3, 5, or 10 papers
• Category filters: All, cs.AI, cs.LG, cs.CL, etc.
• Model selection: Choose LLM model
• Toggles: Streaming, source display, compact mode

**User Experience** Copy Section

• Typing indicators during processing
• Rich formatting with Markdown support
• Clickable links to arXiv papers
• Automatic message splitting for long responses
• Source citations with paper metadata
• Cache indicators (⚡) for instant responses

Quick Start Copy Chapter

Prerequisites Copy Section

1. Telegram Account - Install Telegram on your phone or computer
2. All Week 1-6 Services Running - Full RAG stack must be operational

Step 1: Create Your Telegram Bot Copy Section

1. Open Telegram and search for @BotFather
2. Send /newbot to BotFather
3. Follow prompts:
   • Choose a name (e.g., "My arXiv Curator")
   • Choose a username (e.g., "my_arxiv_curator_bot" - must end in "bot")
4. Copy the bot token - You'll receive something like:
   text Copy

   1234567890:ABCdefGHIjklMNOpqrsTUVwxyz-1234567

Step 2: Configure Environment Copy Section

Add these variables to your .env file:

# Enable Telegram bot
TELEGRAM__ENABLED=true
TELEGRAM__BOT_TOKEN=your_token_from_botfather_here

# Optional: Restrict to specific users (comma-separated Telegram user IDs)
# Leave empty to allow all users
TELEGRAM__ALLOWED_USER_IDS=

# Use polling mode for development (webhook requires HTTPS)
TELEGRAM__USE_WEBHOOK=false

Step 3: Install Dependencies Copy Section

# Install python-telegram-bot library
uv sync

Step 4: Start Services Copy Section

# Start all services (includes Telegram bot)
docker compose up --build -d

# Check logs to verify Telegram bot started
docker compose logs -f api

You should see:

INFO - Telegram bot started successfully
INFO - Starting Telegram bot in polling mode
INFO - Bot commands set successfully

Step 5: Test Your Bot Copy Section

1. Open Telegram and search for your bot username
2. Send /start to your bot
3. Try asking: "What are transformers in machine learning?"
4. Verify you receive an answer with sources!

Testing Instructions Copy Chapter

Manual Testing Scenarios Copy Section

Scenario 1: Basic Commands

Test /start command:

You: /start
Bot: 👋 Welcome to arXiv Paper Curator!
     [Shows capabilities and quick commands]

Test /help command:

You: /help
Bot: 📚 arXiv Paper Curator Help
     [Shows detailed command documentation]

Test /status command:

You: /status
Bot: 🔧 System Status
     ✅ OPENSEARCH
     ✅ OLLAMA
     ✅ CACHE
     [Shows system health]

Scenario 2: RAG Question Answering

Test simple question:

You: What are attention mechanisms?
Bot: [15-20s first time]
     *Answer:*
     Attention mechanisms allow models to dynamically focus on...

     📚 *Sources:*
     [1] *Attention Is All You Need*
         🔗 Read on arXiv - 1706.03762
         📊 Score: 12.456

     [2] *Neural Machine Translation...*
         🔗 Read on arXiv - 1409.0473
         📊 Score: 11.234

     ⚙️ Mode: hybrid

Test cached query:

You: What are attention mechanisms?
Bot: [~100ms second time ⚡]
     [Same answer as above]
     ⚙️ Mode: hybrid ⚡ Cached

Scenario 3: Search Command

Test paper search:

You: /search transformer neural networks
Bot: 📖 Found 145 papers (showing top 10)

     1. *Attention Is All You Need*
        🔗 Read on arXiv - 1706.03762
        📊 Score: 12.456

     2. *BERT: Pre-training of Deep Bidirectional...*
        🔗 Read on arXiv - 1810.04805
        📊 Score: 11.234
     [...]

Scenario 4: Settings Customization

Test settings command:

You: /settings
Bot: ⚙️ Your Settings

     *Search Mode:* HYBRID
     *Results per query:* 3 papers
     *Model:* llama3.2:1b
     *Categories:* All

     [Interactive buttons appear]:
     [🔍 Hybrid Search] [⚡ BM25 Only]
     [3 Results] [5 Results] [10 Results]
     [All Categories] [cs.AI] [cs.LG]

Test changing settings:

You: [Click "5 Results" button]
Bot: ✅ Results per query: 5

You: [Click "cs.AI" button]
Bot: ✅ Category filter: cs.AI

Scenario 5: Natural Conversation

Test multi-turn conversation:

You: Tell me about BERT
Bot: [Provides answer about BERT with sources]

You: How does it differ from GPT?
Bot: [Answers about BERT vs GPT differences]

You: /clear
Bot: 🗑️ Conversation history cleared!
     Your settings have been preserved.

Scenario 6: Error Handling

Test invalid query:

You: asdfghjkl
Bot: ❌ No relevant papers found.
     Try different keywords or check your category filters.

Test when services are down:

You: test query
Bot: ❌ Error processing your question:
     [User-friendly error message]

Verification Checklist Copy Section

• Bot responds to /start - Shows welcome message
• Bot responds to all commands - /help, /ask, /search, /settings, /status, /clear
• Natural language queries work - Can ask questions without commands
• RAG answers include sources - Papers cited with arXiv links
• Cache works - Repeated queries return instantly (⚡ indicator)
• Settings persist - Changes remain across conversation
• Interactive buttons work - Can click inline keyboard buttons
• Long responses split correctly - Messages don't exceed Telegram limit
• Markdown formatting works - Bold, italics, links render correctly
• Langfuse shows traces - Check http://localhost:3000 for Telegram events
• Typing indicator shows - "Bot is typing..." appears during processing
• Error messages are friendly - No stack traces exposed to user

Performance Testing Copy Section

Cache Performance:

# First query (cache miss)
You: What is machine learning?
Bot: [Response in ~15-20 seconds]

# Identical query (cache hit)
You: What is machine learning?
Bot: [Response in ~100ms ⚡]

# Verify 150-400x speedup!

Concurrent Users:

# Test with multiple Telegram accounts
# Each user should have independent settings and sessions

Session Management:

# Stop using bot for 30 minutes (default timeout)
# Verify session cleanup happens automatically

Configuration Copy Chapter

Environment Variables Copy Section

# Enable/Disable Bot
TELEGRAM__ENABLED=true  # Set to false to disable

# Bot Token (Required)
TELEGRAM__BOT_TOKEN=your_token_here

# Deployment Mode
TELEGRAM__USE_WEBHOOK=false  # true for production
TELEGRAM__WEBHOOK_URL=https://your-domain.com
TELEGRAM__WEBHOOK_PATH=/telegram/webhook

# Access Control (Optional)
TELEGRAM__ALLOWED_USER_IDS=123456789,987654321  # Empty = allow all

# Behavior Settings
TELEGRAM__MAX_MESSAGE_LENGTH=4000  # Telegram limit is 4096
TELEGRAM__ENABLE_STREAMING=true
TELEGRAM__SESSION_TIMEOUT_MINUTES=30
TELEGRAM__RATE_LIMIT_MESSAGES_PER_MINUTE=20

# Default User Preferences
TELEGRAM__DEFAULT_TOP_K=3
TELEGRAM__DEFAULT_USE_HYBRID=true
TELEGRAM__DEFAULT_MODEL=llama3.2:1b

User Settings (Customizable per User) Copy Section

Each user can customize:

• Search mode: Hybrid (BM25 + semantic) or BM25 only
• Results count: 3, 5, or 10 papers per query
• Categories: All, cs.AI, cs.LG, cs.CL, cs.CV, cs.NE
• Model: LLM model for answer generation
• Display: Show sources, compact mode, streaming

Settings persist across sessions and are stored in-memory (for production, consider Redis/PostgreSQL storage).

Architecture Details Copy Chapter

Service Integration Copy Section

The Telegram bot integrates with all existing Week 1-6 services:

1. OpenSearch - Hybrid search for relevant papers
2. Jina Embeddings - Semantic search capabilities
3. Ollama LLM - Answer generation
4. Redis Cache - 150-400x speedup for repeated queries
5. Langfuse - Complete tracing of Telegram interactions
6. PostgreSQL - Paper metadata (via OpenSearch)

Message Flow Copy Section

# Simplified message flow
async def handle_message(update, context):
    1. Extract user_id, chat_id, text
    2. Check rate limits
    3. Get/create user settings
    4. Show "typing..." indicator
    5. Check cache (Redis)
    6. If cache hit → format and send
    7. If cache miss:
        a. Generate embedding (Jina)
        b. Search papers (OpenSearch)
        c. Build prompt with context
        d. Generate answer (Ollama)
        e. Cache result (Redis)
        f. Trace interaction (Langfuse)
    8. Format response (Markdown)
    9. Split if too long
    10. Send to Telegram
    11. Update conversation history

Error Handling Copy Section

The bot implements graceful degradation:

• Markdown formatting fails → Falls back to plain text
• Embedding generation fails → Falls back to BM25 search
• Cache unavailable → Continues without caching
• Langfuse tracing fails → Logs warning, continues
• Service errors → User-friendly error messages

Rate Limiting Copy Section

Built-in rate limiting protects against abuse:

• Per-user limits: 20 messages/minute (configurable)
• Automatic throttling: Enforced by Telegram library
• Session timeouts: 30 minutes of inactivity

Troubleshooting Copy Chapter

Debug Mode Copy Section

Enable detailed logging:

# In .env
DEBUG=true

# Check bot logs
docker compose logs -f api | grep telegram

Health Check Copy Section

# Check if bot is running
curl http://localhost:8000/api/v1/health

# Should show telegram_service status

Common Fixes Copy Section

Bot not starting:

# 1. Check token is valid
# 2. Restart API service
docker compose restart api

# 3. Check logs for errors
docker compose logs api

Responses too slow:

# 1. Check cache is working
docker exec rag-redis redis-cli ping  # Should return PONG

# 2. Check cache hit rate
# Look for ⚡ indicator in bot responses

# 3. Verify Langfuse tracing not blocking
LANGFUSE__ENABLED=false  # Temporarily disable

Performance Benchmarks Copy Chapter

Cache Hit Rates (Expected) Copy Section

• Repeated exact queries: 100% hit rate
• Popular questions: 60-80% hit rate
• Unique queries: 0% hit rate (first time)

Resource Usage Copy Section

• Memory: ~50MB per active user session
• CPU: Minimal (<5% idle, spikes during generation)
• Network: ~10KB per message (excluding LLM generation)

Production Deployment Copy Chapter

Webhook Mode (Recommended for Production) Copy Section

# .env for production
TELEGRAM__USE_WEBHOOK=true
TELEGRAM__WEBHOOK_URL=https://your-domain.com
TELEGRAM__WEBHOOK_PATH=/telegram/webhook

# Requires:
# - HTTPS domain with valid certificate
# - Nginx/Caddy for TLS termination
# - Public IP or reverse proxy

Security Best Practices Copy Section

1. Restrict Users: Use ALLOWED_USER_IDS for private bots
2. Rate Limiting: Keep default limits (20 msgs/min)
3. Environment Variables: Never commit .env with real tokens
4. HTTPS Only: Use webhook mode in production
5. Monitoring: Track usage via Langfuse dashboard

Scaling Considerations Copy Section

For high-traffic deployments:

1. User Storage: Migrate from in-memory to Redis/PostgreSQL
2. Cache: Increase Redis memory allocation
3. Load Balancing: Multiple API instances (webhook mode)
4. Rate Limits: Adjust per your requirements
5. Monitoring: Set up alerts for errors and latency

Next Steps Copy Chapter

Optional Enhancements (Week 7.1+) Copy Section

• 📸 Image Support: Upload paper PDFs, get summaries
• 🗣️ Voice Messages: Ask questions via voice (speech-to-text)
• 📊 User Analytics: Dashboard for usage patterns
• 🤝 Group Chat: Multi-user discussions
• 🌍 Multi-Language: Internationalization support
• 🔔 Notifications: Push alerts for new papers in categories
• 📈 Personalization: ML-based paper recommendations
• 🔗 Semantic Cache: Fuzzy matching for similar queries

Integration Ideas Copy Section

• Slack Bot: Port to Slack with same architecture
• Discord Bot: Extend to Discord communities
• WhatsApp Bot: Use WhatsApp Business API
• Web Widget: Embed in websites
• API Access: Expose RESTful API for integrations

Resources Copy Chapter

• Telegram Bot API: https://core.telegram.org/bots/api
• python-telegram-bot: https://python-telegram-bot.org
• @BotFather: https://t.me/botfather
• Langfuse Docs: https://langfuse.com/docs
• Redis Docs: https://redis.io/docs

Code Structure Copy Chapter

# Entry point: src/main.py
telegram_service = make_telegram_service(...)
await telegram_service.start()

# Service: src/services/telegram/client.py
class TelegramService:
    async def start() -> Start bot in polling/webhook mode
    async def stop() -> Stop bot gracefully
    async def health_check() -> Check bot status

# Handlers: src/services/telegram/handlers.py
class TelegramHandlers:
    async def start_command() -> /start
    async def help_command() -> /help
    async def ask_command() -> /ask
    async def search_command() -> /search
    async def settings_command() -> /settings
    async def handle_message() -> Regular text messages

# Formatters: src/services/telegram/formatters.py
format_rag_response() -> Rich Markdown formatting
format_search_results() -> Search result display
format_welcome_message() -> /start message
escape_markdown_v2() -> Telegram MarkdownV2 escaping
split_long_message() -> Auto-split >4000 chars

Success Criteria Copy Chapter

Week 7 is complete when:

• ✅ Bot responds to all commands
• ✅ Natural language queries return RAG answers
• ✅ Cache provides 150-400x speedup
• ✅ Settings persist across sessions
• ✅ Interactive keyboards work
• ✅ Langfuse shows Telegram traces
• ✅ Error handling is graceful
• ✅ Markdown formatting renders correctly
• ✅ Long messages split automatically
• ✅ Multiple users can use concurrently

Week 7 transforms your RAG system into a mobile-first, conversational research assistant accessible anywhere via Telegram! 🚀

FAQ Copy Chapter

Q: Do I need a public IP for the Telegram bot?
A: No! Polling mode works perfectly for development and low-traffic deployments. Webhook requires HTTPS.

Q: Can multiple users use the bot simultaneously?
A: Yes! Each user has independent settings and sessions.

Q: How much does it cost to run the bot?
A: Free! Telegram bot API is completely free with no limits.

Q: Can I restrict the bot to specific users?
A: Yes! Set TELEGRAM__ALLOWED_USER_IDS=123456,789012 with comma-separated Telegram user IDs.

Q: How do I get my Telegram user ID?
A: Message @userinfobot on Telegram or check bot logs after sending a message.

Q: Does the bot store conversation history?
A: Yes, in-memory for the last 10 messages per user. For production, migrate to Redis/PostgreSQL.

Q: Can I deploy this on a server without Docker?
A: Yes! Just run python src/main.py after installing dependencies with uv sync.

Q: How do I update the bot token?
A: Update TELEGRAM__BOT_TOKEN in .env and restart: docker compose restart api

Q: Can I customize the bot's personality?
A: Yes! Edit prompts in src/services/ollama/prompts/ and message templates in src/services/telegram/formatters.py.

Q: Does this work with other LLM models?
A: Yes! Change OLLAMA_MODEL or use /settings command to select different Ollama models.

Enjoy your new conversational RAG interface! 🎉

File: src/services/ollama/prompts/rag_system.txt Copy Chapter

You are an AI assistant specialized in answering questions about academic papers from arXiv. Your task is to provide accurate, helpful answers based ONLY on the provided paper excerpts.

CRITICAL: Do NOT add any introductory text, explanations, or formatting comments like "Here's the answer" or "Here's the JSON".

Instructions:

1. Base your answer STRICTLY on the provided paper excerpts
2. If the excerpts don't contain enough information to answer the question, say so clearly
3. Cite the specific papers (by title or arXiv ID) when providing information
4. Be concise but comprehensive in your response - LIMIT YOUR RESPONSE TO 300 WORDS MAXIMUM
5. Maintain academic accuracy and precision
6. If multiple papers discuss the topic, synthesize the information coherently
7. Use direct quotes from the chunks when particularly relevant
8. Structure your answer logically with clear paragraphs when appropriate
9. Keep it less than 200 words

Remember:

• Do NOT make up information not present in the excerpts
• Do NOT use knowledge beyond what's provided in the paper excerpts
• Always acknowledge uncertainty when the excerpts are ambiguous or incomplete
• Prioritize relevance and clarity in your response
• NEVER add introductory phrases or explanations before your JSON response