crawl4ai Documentation

File: README.md Copy Chapter

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

✨ Check out latest update v0.9.2

✨ New in v0.9.2: Maintenance patch release. Fixes a MemoryAdaptiveDispatcher task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (ENABLE_GPU=true) Docker builds. Release notes →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →

✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →

✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →

✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →

🤓 My Personal Story

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

Why developers pick Crawl4AI

• LLM ready output, smart Markdown with headings, tables, code, citation hints
• Fast in practice, async browser pool, caching, minimal hops
• Full control, sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence, learns site patterns, explores only what matters
• Deploy anywhere, zero keys, CLI and Docker, cloud friendly

🚀 Quick Start Copy Chapter

1. Install Crawl4AI:

# Install the package
pip install -U crawl4ai

# For pre release versions
pip install crawl4ai --pre

# Run post-installation setup
crawl4ai-setup

# Verify your installation
crawl4ai-doctor

If you encounter any browser-related issues, you can install them manually:

python -m playwright install --with-deps chromium

2. Run a simple web crawl with Python:

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

3. Or use the new command-line interface:

# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown

# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"

💖 Support Crawl4AI Copy Chapter

🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.

Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.

🤝 Sponsorship Tiers Copy Section

• 🌱 Believer ($5/mo) — Join the movement for data democratization
• 🚀 Builder ($50/mo) — Priority support & early access to features
• 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
• 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support
  Custom arrangements available - see SPONSORS.md for details & contact

Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.

See All Tiers & Benefits →

✨ Features Copy Chapter

📝 Markdown Generation

• 🧹 Clean Markdown: Generates clean, structured Markdown with accurate formatting.
• 🎯 Fit Markdown: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
• 🔗 Citations and References: Converts page links into a numbered reference list with clean citations.
• 🛠️ Custom Strategies: Users can create their own Markdown generation strategies tailored to specific needs.
• 📚 BM25 Algorithm: Employs BM25-based filtering for extracting core information and removing irrelevant content.

📊 Structured Data Extraction

• 🤖 LLM-Driven Extraction: Supports all LLMs (open-source and proprietary) for structured data extraction.
• 🧱 Chunking Strategies: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
• 🌌 Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
• 🔎 CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
• 🔧 Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.

🌐 Browser Integration

• 🖥️ Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
• 🔄 Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
• 👤 Browser Profiler: Create and manage persistent profiles with saved authentication states, cookies, and settings.
• 🔒 Session Management: Preserve browser states and reuse them for multi-step crawling.
• 🧩 Proxy Support: Seamlessly connect to proxies with authentication for secure access.
• ⚙️ Full Browser Control: Modify headers, cookies, user agents, and more for tailored crawling setups.
• 🌍 Multi-Browser Support: Compatible with Chromium, Firefox, and WebKit.
• 📐 Dynamic Viewport Adjustment: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.

🔎 Crawling & Scraping

• 🖼️ Media Support: Extract images, audio, videos, and responsive image formats like srcset and picture.
• 🚀 Dynamic Crawling: Execute JS and wait for async or sync for dynamic content extraction.
• 📸 Screenshots: Capture page screenshots during crawling for debugging or analysis.
• 📂 Raw Data Crawling: Directly process raw HTML (raw:) or local files (file://).
• 🔗 Comprehensive Link Extraction: Extracts internal, external links, and embedded iframe content.
• 🛠️ Customizable Hooks: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).
• 💾 Caching: Cache data for improved speed and to avoid redundant fetches.
• 📄 Metadata Extraction: Retrieve structured metadata from web pages.
• 📡 IFrame Content Extraction: Seamless extraction from embedded iframe content.
• 🕵️ Lazy Load Handling: Waits for images to fully load, ensuring no content is missed due to lazy loading.
• 🔄 Full-Page Scanning: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.

🚀 Deployment

• 🐳 Dockerized Setup: Optimized Docker image with FastAPI server for easy deployment.
• 🔑 Secure Authentication: Built-in JWT token authentication for API security.
• 🔄 API Gateway: One-click deployment with secure token authentication for API-based workflows.
• 🌐 Scalable Architecture: Designed for mass-scale production and optimized server performance.
• ☁️ Cloud Deployment: Ready-to-deploy configurations for major cloud platforms.

🎯 Additional Features

• 🕶️ Stealth Mode: Avoid bot detection by mimicking real users.
• 🏷️ Tag-Based Content Extraction: Refine crawling based on custom tags, headers, or metadata.
• 🔗 Link Analysis: Extract and analyze all links for detailed data exploration.
• 🛡️ Error Handling: Robust error management for seamless execution.
• 🔐 CORS & Static Serving: Supports filesystem-based caching and cross-origin requests.
• 📖 Clear Documentation: Simplified and updated guides for onboarding and advanced usage.
• 🙌 Community Recognition: Acknowledges contributors and pull requests for transparency.

Try it Now! Copy Chapter

✨ Play around with this

✨ Visit our Documentation Website

Installation 🛠️ Copy Chapter

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

🐍 Using pip

Choose the installation option that best fits your needs:

Basic Installation Copy Section

For basic web crawling and scraping tasks:

pip install crawl4ai
crawl4ai-setup # Setup the browser

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

👉 Note: When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

1. Through the command line:

   bash Copy

   playwright install

2. If the above doesn't work, try this more specific command:

   bash Copy

   python -m playwright install chromium

This second method has proven to be more reliable in some cases.

Installation with Synchronous Version Copy Section

The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:

pip install crawl4ai[sync]

Development Installation Copy Section

For contributors who plan to modify the source code:

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .                    # Basic installation in editable mode

Install optional features:

pip install -e ".[torch]"           # With PyTorch features
pip install -e ".[transformer]"     # With Transformer features
pip install -e ".[cosine]"          # With cosine similarity features
pip install -e ".[sync]"            # With synchronous crawling (Selenium)
pip install -e ".[all]"             # Install all optional features

🐳 Docker Deployment

🚀 Now Available! Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.

New Docker Features Copy Section

The new Docker implementation includes:

• Real-time Monitoring Dashboard with live system metrics and browser pool visibility
• Browser pooling with page pre-warming for faster response times
• Interactive playground to test and generate request code
• MCP integration for direct connection to AI tools like Claude Code
• Comprehensive API endpoints including HTML extraction, screenshots, PDF generation, and JavaScript execution
• Multi-architecture support with automatic detection (AMD64/ARM64)
• Optimized resources with improved memory management

Getting Started Copy Section

# Pull and run the latest release
docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

# Visit the monitoring dashboard at http://localhost:11235/dashboard
# Or the playground at http://localhost:11235/playground

Quick Test Copy Section

Run a quick test (works for both Docker options):

import requests

# Submit a crawl job
response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
    print("Crawl job submitted successfully.")
    
if "results" in response.json():
    results = response.json()["results"]
    print("Crawl job completed. Results:")
    for result in results:
        print(result)
else:
    task_id = response.json()["task_id"]
    print(f"Crawl job submitted. Task ID:: {task_id}")
    result = requests.get(f"http://localhost:11235/task/{task_id}")

For more examples, see our Docker Examples. For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide.

🔬 Advanced Usage Examples 🔬 Copy Chapter

You can check the project structure in the directory docs/examples. Over there, you can find a variety of examples; here, some popular examples are shared.

📝 Heuristic Markdown Generation with Clean and Fit Markdown

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    browser_config = BrowserConfig(
        headless=True,  
        verbose=True,
    )
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.ENABLED,
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
        ),
        # markdown_generator=DefaultMarkdownGenerator(
        #     content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)
        # ),
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://docs.micronaut.io/4.9.9/guide/",
            config=run_config
        )
        print(len(result.markdown.raw_markdown))
        print(len(result.markdown.fit_markdown))

if __name__ == "__main__":
    asyncio.run(main())

🖥️ Executing JavaScript & Extract Structured Data without LLMs

/* Detailed source-code truncated for AI context efficiency. */

📚 Extracting Structured Data with LLMs

import os
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy
from pydantic import BaseModel, Field

class OpenAIModelFee(BaseModel):
    model_name: str = Field(..., description="Name of the OpenAI model.")
    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
    output_fee: str = Field(..., description="Fee for output token for the OpenAI model.")

async def main():
    browser_config = BrowserConfig(verbose=True)
    run_config = CrawlerRunConfig(
        word_count_threshold=1,
        extraction_strategy=LLMExtractionStrategy(
            # Here you can use any provider that Litellm library supports, for instance: ollama/qwen2
            # provider="ollama/qwen2", api_token="no-token", 
            llm_config = LLMConfig(provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY')), 
            schema=OpenAIModelFee.schema(),
            extraction_type="schema",
            instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
            Do not miss any models in the entire content. One extracted model JSON format should look like this: 
            {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}."""
        ),            
        cache_mode=CacheMode.BYPASS,
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url='https://openai.com/api/pricing/',
            config=run_config
        )
        print(result.extracted_content)

if __name__ == "__main__":
    asyncio.run(main())

🤖 Using Your own Browser with Custom User Profile

import os, sys
from pathlib import Path
import asyncio, time
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def test_news_crawl():
    # Create a persistent user data directory
    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
    os.makedirs(user_data_dir, exist_ok=True)

    browser_config = BrowserConfig(
        verbose=True,
        headless=True,
        user_data_dir=user_data_dir,
        use_persistent_context=True,
    )
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
        
        result = await crawler.arun(
            url,
            config=run_config,
            magic=True,
        )
        
        print(f"Successfully crawled {url}")
        print(f"Content length: {len(result.markdown)}")

✨ Recent Updates Copy Chapter

Version 0.9.2 Release Highlights - Maintenance Bug Fixes

A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. MemoryAdaptiveDispatcher no longer leaks crawl tasks and browser pages when a streaming crawl is closed. Docker fixes cover the Playground "Advanced Config" 400, the Monitor WebSocket 500 under JWT auth, and Playwright headless-shell packaging. ENABLE_GPU=true Docker builds no longer fail on the CUDA toolkit.

No new features, no breaking changes.

pip install -U crawl4ai

Full v0.9.2 Release Notes →

Version 0.9.1 Release Highlights - Bug Fixes & PruningContentFilter Whitelist

A patch release with 12 bug fixes and one new feature. The new preserve_classes / preserve_tags parameters for PruningContentFilter let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps.

Bug fixes span Docker (auth gate UI, supervisord/redis dirs, FastAPI compatibility, redis auth), browser (Windows channel crash, context snapshot leak), core (HTTP timeout unit mismatch, best-first ordering), and extraction (html2text table attributes).

pip install -U crawl4ai

Full v0.9.1 Release Notes →

Version 0.9.0 Release Highlights - Secure-by-Default Docker Server

A major, secure-by-default release of the Docker API server. The out-of-the-box deployment is hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary.

pip install -U crawl4ai

Migration Guide → · Full v0.9.0 Release Notes →

Version 0.8.7 Release Highlights - Security Hardening, DomainMapper & Community Fixes

A security-hardening release. Fixes critical Docker API vulnerabilities (AST sandbox escape RCE, hook sandbox RCE, hardcoded JWT secret, SSRF on webhook and crawl endpoints, arbitrary file write, monitor auth bypass, stored XSS, and unauthenticated JS execution), adds the DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes. If you self-host the Docker API, upgrade immediately.

pip install -U crawl4ai

Full v0.8.7 Release Notes →

Version 0.8.6 - Security Hotfix: litellm Supply Chain Fix

Replaced litellm dependency with unclecode-litellm due to a PyPI supply chain compromise affecting the original package. If you're on v0.8.5 or earlier, upgrade immediately.

pip install -U crawl4ai

Version 0.8.5 Release Highlights - Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes

Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.

• 🛡️ Anti-Bot Detection & Proxy Escalation:

  • 3-tier detection: known vendors, generic block indicators, structural integrity checks
  • Automatic retry with proxy chain and fallback fetch function
  python Copy

  from crawl4ai import CrawlerRunConfig
  from crawl4ai.async_configs import ProxyConfig
  
  config = CrawlerRunConfig(
      proxy_config=[ProxyConfig.DIRECT, ProxyConfig(server="http://my-proxy:8080")],
      max_retries=2,
      fallback_fetch_function=my_web_unlocker,
  )

• 🌑 Shadow DOM Flattening:

  • Extract content hidden inside shadow DOM components
  python Copy

  config = CrawlerRunConfig(flatten_shadow_dom=True)

• 🛑 Deep Crawl Cancellation:

  • Stop long crawls gracefully with cancel() or should_cancel callback
  • Works with BFS, DFS, and BestFirst strategies

• ⚙️ Config Defaults API:

  • set_defaults() / get_defaults() / reset_defaults() on BrowserConfig and CrawlerRunConfig

• 🔒 Critical Security Fixes:

  • RCE via deserialization in Docker /crawl endpoint — removed eval(), added allowlist
  • Redis CVE-2025-49844 (CVSS 10.0) — upgraded to 7.2.7

• 60+ Bug Fixes across browser management, proxy, deep crawling, extraction, CLI, and Docker

Full v0.8.5 Release Notes →

Version 0.8.0 Release Highlights - Crash Recovery & Prefetch Mode

This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.

• 🔄 Deep Crawl Crash Recovery:

  • on_state_change callback fires after each URL for real-time state persistence
  • resume_state parameter to continue from a saved checkpoint
  • JSON-serializable state for Redis/database storage
  • Works with BFS, DFS, and Best-First strategies
  python Copy

  from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
  
  strategy = BFSDeepCrawlStrategy(
      max_depth=3,
      resume_state=saved_state,  # Continue from checkpoint
      on_state_change=save_to_redis,  # Called after each URL
  )

• ⚡ Prefetch Mode for Fast URL Discovery:

  • prefetch=True skips markdown, extraction, and media processing
  • 5-10x faster than full processing
  • Perfect for two-phase crawling: discover first, process selectively
  python Copy

  config = CrawlerRunConfig(prefetch=True)
  result = await crawler.arun("https://example.com", config=config)
  # Returns HTML and links only - no markdown generation

• 🔒 Security Fixes (Docker API):

  • Hooks disabled by default (CRAWL4AI_HOOKS_ENABLED=false)
  • file:// URLs blocked on API endpoints to prevent LFI
  • __import__ removed from hook execution sandbox

Full v0.8.0 Release Notes →

Version 0.7.8 Release Highlights - Stability & Bug Fix Release

This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.

• 🐳 Docker API Fixes:

  • Fixed ContentRelevanceFilter deserialization in deep crawl requests (#1642)
  • Fixed ProxyConfig JSON serialization in BrowserConfig.to_dict() (#1629)
  • Fixed .cache folder permissions in Docker image (#1638)

• 🤖 LLM Extraction Improvements:

  • Configurable rate limiter backoff with new LLMConfig parameters (#1269):
    python Copy

    from crawl4ai import LLMConfig
    
    config = LLMConfig(
        provider="openai/gpt-4o-mini",
        backoff_base_delay=5,           # Wait 5s on first retry
        backoff_max_attempts=5,          # Try up to 5 times
        backoff_exponential_factor=3     # Multiply delay by 3 each attempt
    )

  • HTML input format support for LLMExtractionStrategy (#1178):
    python Copy

    from crawl4ai import LLMExtractionStrategy
    
    strategy = LLMExtractionStrategy(
        llm_config=config,
        instruction="Extract table data",
        input_format="html"  # Now supports: "html", "markdown", "fit_markdown"
    )

  • Fixed raw HTML URL variable - extraction strategies now receive "Raw HTML" instead of HTML blob (#1116)

• 🔗 URL Handling:

  • Fixed relative URL resolution after JavaScript redirects (#1268)
  • Fixed import statement formatting in extracted code (#1181)

• 📦 Dependency Updates:

  • Replaced deprecated PyPDF2 with pypdf (#1412)
  • Pydantic v2 ConfigDict compatibility - no more deprecation warnings (#678)

• 🧠 AdaptiveCrawler:

  • Fixed query expansion to actually use LLM instead of hardcoded mock data (#1621)

Full v0.7.8 Release Notes →

Version 0.7.7 Release Highlights - The Self-Hosting & Monitoring Update

• 📊 Real-time Monitoring Dashboard: Interactive web UI with live system metrics and browser pool visibility

  python Copy

  # Access the monitoring dashboard
  # Visit: http://localhost:11235/dashboard
  
  # Real-time metrics include:
  # - System health (CPU, memory, network, uptime)
  # - Active and completed request tracking
  # - Browser pool management (permanent/hot/cold)
  # - Janitor cleanup events
  # - Error monitoring with full context

• 🔌 Comprehensive Monitor API: Complete REST API for programmatic access to all monitoring data

  python Copy

  import httpx
  
  async with httpx.AsyncClient() as client:
      # System health
      health = await client.get("http://localhost:11235/monitor/health")
  
      # Request tracking
      requests = await client.get("http://localhost:11235/monitor/requests")
  
      # Browser pool status
      browsers = await client.get("http://localhost:11235/monitor/browsers")
  
      # Endpoint statistics
      stats = await client.get("http://localhost:11235/monitor/endpoints/stats")

• ⚡ WebSocket Streaming: Real-time updates every 2 seconds for custom dashboards

• 🔥 Smart Browser Pool: 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup

• 🧹 Janitor System: Automatic resource management with event logging

• 🎮 Control Actions: Manual browser management (kill, restart, cleanup) via API

• 📈 Production Metrics: 6 critical metrics for operational excellence with Prometheus integration

• 🐛 Critical Bug Fixes:

  • Fixed async LLM extraction blocking issue (#1055)
  • Enhanced DFS deep crawl strategy (#1607)
  • Fixed sitemap parsing in AsyncUrlSeeder (#1598)
  • Resolved browser viewport configuration (#1495)
  • Fixed CDP timing with exponential backoff (#1528)
  • Security update for pyOpenSSL (>=25.3.0)

Full v0.7.7 Release Notes →

Version 0.7.5 Release Highlights - The Docker Hooks & Security Update

• 🔧 Docker Hooks System: Complete pipeline customization with user-provided Python functions at 8 key points

• ✨ Function-Based Hooks API (NEW): Write hooks as regular Python functions with full IDE support:

  python Copy

  from crawl4ai import hooks_to_string
  from crawl4ai.docker_client import Crawl4aiDockerClient
  
  # Define hooks as regular Python functions
  async def on_page_context_created(page, context, **kwargs):
      """Block images to speed up crawling"""
      await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
      await page.set_viewport_size({"width": 1920, "height": 1080})
      return page
  
  async def before_goto(page, context, url, **kwargs):
      """Add custom headers"""
      await page.set_extra_http_headers({'X-Crawl4AI': 'v0.7.5'})
      return page
  
  # Option 1: Use hooks_to_string() utility for REST API
  hooks_code = hooks_to_string({
      "on_page_context_created": on_page_context_created,
      "before_goto": before_goto
  })
  
  # Option 2: Docker client with automatic conversion (Recommended)
  client = Crawl4aiDockerClient(base_url="http://localhost:11235")
  results = await client.crawl(
      urls=["https://httpbin.org/html"],
      hooks={
          "on_page_context_created": on_page_context_created,
          "before_goto": before_goto
      }
  )
  # ✓ Full IDE support, type checking, and reusability!

• 🤖 Enhanced LLM Integration: Custom providers with temperature control and base_url configuration

• 🔒 HTTPS Preservation: Secure internal link handling with preserve_https_for_internal_links=True

• 🐍 Python 3.10+ Support: Modern language features and enhanced performance

• 🛠️ Bug Fixes: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration

Full v0.7.5 Release Notes →

Version 0.7.4 Release Highlights - The Intelligent Table Extraction & Performance Update

• 🚀 LLMTableExtraction: Revolutionary table extraction with intelligent chunking for massive tables:

  python Copy

  from crawl4ai import LLMTableExtraction, LLMConfig
  
  # Configure intelligent table extraction
  table_strategy = LLMTableExtraction(
      llm_config=LLMConfig(provider="openai/gpt-4.1-mini"),
      enable_chunking=True,           # Handle massive tables
      chunk_token_threshold=5000,     # Smart chunking threshold
      overlap_threshold=100,          # Maintain context between chunks
      extraction_type="structured"    # Get structured data output
  )
  
  config = CrawlerRunConfig(table_extraction_strategy=table_strategy)
  result = await crawler.arun("https://complex-tables-site.com", config=config)
  
  # Tables are automatically chunked, processed, and merged
  for table in result.tables:
      print(f"Extracted table: {len(table['data'])} rows")

• ⚡ Dispatcher Bug Fix: Fixed sequential processing bottleneck in arun_many for fast-completing tasks

• 🧹 Memory Management Refactor: Consolidated memory utilities into main utils module for cleaner architecture

• 🔧 Browser Manager Fixes: Resolved race conditions in concurrent page creation with thread-safe locking

• 🔗 Advanced URL Processing: Better handling of raw:// URLs and base tag link resolution

• 🛡️ Enhanced Proxy Support: Flexible proxy configuration supporting both dict and string formats

Full v0.7.4 Release Notes →

Version 0.7.3 Release Highlights - The Multi-Config Intelligence Update

• 🕵️ Undetected Browser Support: Bypass sophisticated bot detection systems:

  python Copy

  from crawl4ai import AsyncWebCrawler, BrowserConfig
  
  browser_config = BrowserConfig(
      browser_type="undetected",  # Use undetected Chrome
      headless=True,              # Can run headless with stealth
      extra_args=[
          "--disable-blink-features=AutomationControlled",
          "--disable-web-security"
      ]
  )
  
  async with AsyncWebCrawler(config=browser_config) as crawler:
      result = await crawler.arun("https://protected-site.com")
  # Successfully bypass Cloudflare, Akamai, and custom bot detection

• 🎨 Multi-URL Configuration: Different strategies for different URL patterns in one batch:

from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode
  
  configs = [
      # Documentation sites - aggressive caching
      CrawlerRunConfig(
          url_matcher=["*docs*", "*documentation*"],
          cache_mode=CacheMode.WRITE_ONLY,
          markdown_generator_options={"include_links": True}
      ),
      
      # News/blog sites - fresh content
      CrawlerRunConfig(
          url_matcher=lambda url: 'blog' in url or 'news' in url,
          cache_mode=CacheMode.BYPASS
      ),
      
      # Fallback for everything else
      CrawlerRunConfig()
  ]
  
  results = await crawler.arun_many(urls, config=configs)
  # Each URL gets the perfect configuration automatically

• 🧠 Memory Monitoring: Track and optimize memory usage during crawling:

  python Copy

  from crawl4ai.memory_utils import MemoryMonitor
  
  monitor = MemoryMonitor()
  monitor.start_monitoring()
  
  results = await crawler.arun_many(large_url_list)
  
  report = monitor.get_report()
  print(f"Peak memory: {report['peak_mb']:.1f} MB")
  print(f"Efficiency: {report['efficiency']:.1f}%")
  # Get optimization recommendations

• 📊 Enhanced Table Extraction: Direct DataFrame conversion from web tables:

  python Copy

  result = await crawler.arun("https://site-with-tables.com")
  
  # New way - direct table access
  if result.tables:
      import pandas as pd
      for table in result.tables:
          df = pd.DataFrame(table['data'])
          print(f"Table: {df.shape[0]} rows × {df.shape[1]} columns")

• 💰 GitHub Sponsors: 4-tier sponsorship system for project sustainability

• 🐳 Docker LLM Flexibility: Configure providers via environment variables

Full v0.7.3 Release Notes →

Version 0.7.0 Release Highlights - The Adaptive Intelligence Update

• 🧠 Adaptive Crawling: Your crawler now learns and adapts to website patterns automatically:

  python Copy

  config = AdaptiveConfig(
      confidence_threshold=0.7, # Min confidence to stop crawling
      max_depth=5, # Maximum crawl depth
      max_pages=20, # Maximum number of pages to crawl
      strategy="statistical"
  )
  
  async with AsyncWebCrawler() as crawler:
      adaptive_crawler = AdaptiveCrawler(crawler, config)
      state = await adaptive_crawler.digest(
          start_url="https://news.example.com",
          query="latest news content"
      )
  # Crawler learns patterns and improves extraction over time

• 🌊 Virtual Scroll Support: Complete content extraction from infinite scroll pages:

  python Copy

  scroll_config = VirtualScrollConfig(
      container_selector="[data-testid='feed']",
      scroll_count=20,
      scroll_by="container_height",
      wait_after_scroll=1.0
  )
  
  result = await crawler.arun(url, config=CrawlerRunConfig(
      virtual_scroll_config=scroll_config
  ))

• 🔗 Intelligent Link Analysis: 3-layer scoring system for smart link prioritization:

  python Copy

  link_config = LinkPreviewConfig(
      query="machine learning tutorials",
      score_threshold=0.3,
      concurrent_requests=10
  )
  
  result = await crawler.arun(url, config=CrawlerRunConfig(
      link_preview_config=link_config,
      score_links=True
  ))
  # Links ranked by relevance and quality

• 🎣 Async URL Seeder: Discover thousands of URLs in seconds:

  python Copy

  seeder = AsyncUrlSeeder(SeedingConfig(
      source="sitemap+cc",
      pattern="*/blog/*",
      query="python tutorials",
      score_threshold=0.4
  ))
  
  urls = await seeder.discover("https://example.com")

• ⚡ Performance Boost: Up to 3x faster with optimized resource handling and memory efficiency

Read the full details in our 0.7.0 Release Notes or check the CHANGELOG.

Version Numbering in Crawl4AI Copy Chapter

Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.

📈 Version Numbers Explained

Our version numbers follow this pattern: MAJOR.MINOR.PATCH (e.g., 0.4.3)

Pre-release Versions

We use different suffixes to indicate development stages:

• dev (0.4.3dev1): Development versions, unstable
• a (0.4.3a1): Alpha releases, experimental features
• b (0.4.3b1): Beta releases, feature complete but needs testing
• rc (0.4.3): Release candidates, potential final version

Installation

• Regular installation (stable version):

  bash Copy

  pip install -U crawl4ai

• Install pre-release versions:

  bash Copy

  pip install crawl4ai --pre

• Install specific version:

  bash Copy

  pip install crawl4ai==0.4.3b1

Why Pre-releases?

We use pre-releases to:

• Test new features in real-world scenarios
• Gather feedback before final releases
• Ensure stability for production users
• Allow early adopters to try new features

For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the --pre flag.

📖 Documentation & Roadmap Copy Chapter

🚨 Documentation Update Alert: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!

For current documentation, including installation instructions, advanced features, and API reference, visit our Documentation Website.

To check our development plans and upcoming features, visit our Roadmap.

📈 Development TODOs

• 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
• 1. Question-Based Crawler: Natural language driven web discovery and content extraction
• 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
• 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
• 4. Automated Schema Generator: Convert natural language to extraction schemas
• 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
• 6. Web Embedding Index: Semantic search infrastructure for crawled content
• 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
• 8. Performance Monitor: Real-time insights into crawler operations
• 9. Cloud Integration: One-click deployment solutions across cloud providers
• 10. Sponsorship Program: Structured support system with tiered benefits
• 11. Educational Content: "How to Crawl" video series and interactive tutorials

🤝 Contributing Copy Chapter

We welcome contributions from the open-source community. Check out our contribution guidelines for more information.

I'll help modify the license section with badges. For the halftone effect, here's a version with it:

Here's the updated license section:

📄 License & Attribution Copy Chapter

This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the Apache 2.0 License file for details.

Attribution Requirements Copy Section

When using Crawl4AI, you must include one of the following attribution methods:

📈 1. Badge Attribution (Recommended)
Add one of these badges to your README, documentation, or website:

HTML code for adding the badges:

<a href="https://github.com/unclecode/crawl4ai">
  
</a>


<a href="https://github.com/unclecode/crawl4ai">
  
</a>


<a href="https://github.com/unclecode/crawl4ai">
  
</a>


<a href="https://github.com/unclecode/crawl4ai">
  
</a>


<a href="https://github.com/unclecode/crawl4ai">
  
</a>

📖 2. Text Attribution
Add this line to your documentation:

This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.

📚 Citation Copy Chapter

If you use Crawl4AI in your research or project, please cite:

@software{crawl4ai2024,
  author = {UncleCode},
  title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub Repository},
  howpublished = {\url{https://github.com/unclecode/crawl4ai}},
  commit = {Please use the commit hash you're working with}
}

Text citation format:

UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software]. 
GitHub. https://github.com/unclecode/crawl4ai

📧 Contact Copy Chapter

For questions, suggestions, or feedback, feel free to reach out:

• GitHub: unclecode
• Twitter: @unclecode
• Website: crawl4ai.com

Happy Crawling! 🕸️🚀

🗾 Mission Copy Chapter

Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.

We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.

🔑 Key Opportunities

• Data Capitalization: Transform digital footprints into measurable, valuable assets.
• Authentic AI Data: Provide AI systems with real human insights.
• Shared Economy: Create a fair data marketplace that benefits data creators.

🚀 Development Pathway

1. Open-Source Tools: Community-driven platforms for transparent data extraction.
2. Digital Asset Structuring: Tools to organize and value digital knowledge.
3. Ethical Data Marketplace: A secure, fair platform for exchanging structured data.

For more details, see our full mission statement.

🌟 Current Sponsors Copy Chapter

🤝 Strategic Partners Copy Section

These companies provide core infrastructure and technology that power Crawl4AI’s capabilities — from web access and proxy networks to AI tooling and data pipelines.

🏢 Enterprise Sponsors Copy Section

Our enterprise sponsors support Crawl4AI and help scale it to power production-grade data pipelines.

💼 Become a Strategic Partner or Sponsor Copy Section

Interested in partnering with Crawl4AI?

Whether you’re a proxy provider, AI infrastructure company, cloud platform, or an organization looking to support the Crawl4AI ecosystem, we’d love to hear from you.

📩 Contact: [email protected]

🧑‍🤝 Individual Sponsors Copy Section

A heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!

Want to join them? Sponsor Crawl4AI →

Star History Copy Chapter

File: deploy/docker/README.md Copy Chapter

Crawl4AI Docker Guide 🐳

Table of Contents Copy Chapter

• Prerequisites
• Installation
  • Option 1: Using Pre-built Docker Hub Images (Recommended)
  • Option 2: Using Docker Compose
  • Option 3: Manual Local Build & Run
• Dockerfile Parameters
• Using the API
  • Playground Interface
  • Python SDK
  • Understanding Request Schema
  • REST API Examples
  • Asynchronous Jobs with Webhooks
• Additional API Endpoints
  • HTML Extraction Endpoint
  • Screenshot Endpoint
  • PDF Export Endpoint
  • JavaScript Execution Endpoint
  • Library Context Endpoint
• MCP (Model Context Protocol) Support
  • What is MCP?
  • Connecting via MCP
  • Using with Claude Code
  • Available MCP Tools
  • Testing MCP Connections
  • MCP Schemas
• Metrics & Monitoring
• Deployment Scenarios
• Complete Examples
• Server Configuration
  • Understanding config.yml
  • JWT Authentication
  • Configuration Tips and Best Practices
  • Customizing Your Configuration
  • Configuration Recommendations
• Getting Help
• Summary

Prerequisites Copy Chapter

Before we dive in, make sure you have:

• Docker installed and running (version 20.10.0 or higher), including docker compose (usually bundled with Docker Desktop).
• git for cloning the repository.
• At least 4GB of RAM available for the container (more recommended for heavy use).
• Python 3.10+ (if using the Python SDK).
• Node.js 16+ (if using the Node.js examples).

💡 Pro tip: Run docker info to check your Docker installation and available resources.

Installation Copy Chapter

We offer several ways to get the Crawl4AI server running. The quickest way is to use our pre-built Docker Hub images.

Option 1: Using Pre-built Docker Hub Images (Recommended) Copy Section

Pull and run images directly from Docker Hub without building locally.

1. Pull the Image

Our latest stable release is 0.9.2. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.

# Pull the latest stable version (0.9.2)
docker pull unclecode/crawl4ai:0.8.6

# Or use the latest tag
docker pull unclecode/crawl4ai:latest

2. Setup Environment (API Keys)

If you plan to use LLMs, create a .llm.env file in your working directory:

# Create a .llm.env file with your API keys
cat > .llm.env << EOL
# OpenAI
OPENAI_API_KEY=sk-your-key

# Anthropic
ANTHROPIC_API_KEY=your-anthropic-key

# Other providers as needed
# DEEPSEEK_API_KEY=your-deepseek-key
# GROQ_API_KEY=your-groq-key
# TOGETHER_API_KEY=your-together-key
# MISTRAL_API_KEY=your-mistral-key
# GEMINI_API_TOKEN=your-gemini-token
EOL

🔑 Note: Keep your API keys secure! Never commit .llm.env to version control.

3. Run the Container

• Basic run:

  bash Copy

  docker run -d \
    -p 11235:11235 \
    --name crawl4ai \
    --shm-size=1g \
    unclecode/crawl4ai:0.8.6

• With LLM support:

  bash Copy

  # Make sure .llm.env is in the current directory
  docker run -d \
    -p 11235:11235 \
    --name crawl4ai \
    --env-file .llm.env \
    --shm-size=1g \
    unclecode/crawl4ai:0.8.6

The server will be available at http://localhost:11235. Visit /playground to access the interactive testing interface.

4. Stopping the Container

docker stop crawl4ai && docker rm crawl4ai

Docker Hub Versioning Explained

• Image Name: unclecode/crawl4ai
• Tag Format: LIBRARY_VERSION[-SUFFIX] (e.g., 0.7.0-r1)
  • LIBRARY_VERSION: The semantic version of the core crawl4ai Python library
  • SUFFIX: Optional tag for release candidates (``) and revisions (r1)
• latest Tag: Points to the most recent stable version
• Multi-Architecture Support: All images support both linux/amd64 and linux/arm64 architectures through a single tag

Option 2: Using Docker Compose Copy Section

Docker Compose simplifies building and running the service, especially for local development and testing.

1. Clone Repository

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai

2. Environment Setup (API Keys)

If you plan to use LLMs, copy the example environment file and add your API keys. This file should be in the project root directory.

# Make sure you are in the 'crawl4ai' root directory
cp deploy/docker/.llm.env.example .llm.env

# Now edit .llm.env and add your API keys

Flexible LLM Provider Configuration:

The Docker setup now supports flexible LLM provider configuration through three methods:

1. Environment Variable (Highest Priority): Set LLM_PROVIDER to override the default

   bash Copy

   export LLM_PROVIDER="anthropic/claude-3-opus"
   # Or in your .llm.env file:
   # LLM_PROVIDER=anthropic/claude-3-opus

2. API Request Parameter: Specify provider per request

   json Copy

   {
     "url": "https://example.com",
     "provider": "groq/mixtral-8x7b"
   }

3. Config File Default: Falls back to config.yml (default: openai/gpt-4o-mini)

The system automatically selects the appropriate API key based on the provider.

3. Build and Run with Compose

The docker-compose.yml file in the project root provides a simplified approach that automatically handles architecture detection using buildx.

• Run Pre-built Image from Docker Hub:

  bash Copy

  # Pulls and runs the release candidate from Docker Hub
  # Automatically selects the correct architecture
  IMAGE=unclecode/crawl4ai:0.8.6 docker compose up -d

• Build and Run Locally:

  bash Copy

  # Builds the image locally using Dockerfile and runs it
  # Automatically uses the correct architecture for your machine
  docker compose up --build -d

• Customize the Build:

  bash Copy

  # Build with all features (includes torch and transformers)
  INSTALL_TYPE=all docker compose up --build -d
  
  # Build with GPU support (for AMD64 platforms)
  ENABLE_GPU=true docker compose up --build -d

The server will be available at http://localhost:11235.

4. Stopping the Service

# Stop the service
docker compose down

Option 3: Manual Local Build & Run Copy Section

If you prefer not to use Docker Compose for direct control over the build and run process.

1. Clone Repository & Setup Environment

Follow steps 1 and 2 from the Docker Compose section above (clone repo, cd crawl4ai, create .llm.env in the root).

2. Build the Image (Multi-Arch)

Use docker buildx to build the image. Crawl4AI now uses buildx to handle multi-architecture builds automatically.

# Make sure you are in the 'crawl4ai' root directory
# Build for the current architecture and load it into Docker
docker buildx build -t crawl4ai-local:latest --load .

# Or build for multiple architectures (useful for publishing)
docker buildx build --platform linux/amd64,linux/arm64 -t crawl4ai-local:latest --load .

# Build with additional options
docker buildx build \
  --build-arg INSTALL_TYPE=all \
  --build-arg ENABLE_GPU=false \
  -t crawl4ai-local:latest --load .

3. Run the Container

• Basic run (no LLM support):

  bash Copy

  docker run -d \
    -p 11235:11235 \
    --name crawl4ai-standalone \
    --shm-size=1g \
    crawl4ai-local:latest

• With LLM support:

  bash Copy

  # Make sure .llm.env is in the current directory (project root)
  docker run -d \
    -p 11235:11235 \
    --name crawl4ai-standalone \
    --env-file .llm.env \
    --shm-size=1g \
    crawl4ai-local:latest

The server will be available at http://localhost:11235.

4. Stopping the Manual Container

docker stop crawl4ai-standalone && docker rm crawl4ai-standalone

MCP (Model Context Protocol) Support Copy Chapter

Crawl4AI server includes support for the Model Context Protocol (MCP), allowing you to connect the server's capabilities directly to MCP-compatible clients like Claude Code.

What is MCP? Copy Section

MCP is an open protocol that standardizes how applications provide context to LLMs. It allows AI models to access external tools, data sources, and services through a standardized interface.

Connecting via MCP Copy Section

The Crawl4AI server exposes two MCP endpoints:

• Server-Sent Events (SSE): http://localhost:11235/mcp/sse
• WebSocket: ws://localhost:11235/mcp/ws

Using with Claude Code Copy Section

You can add Crawl4AI as an MCP tool provider in Claude Code with a simple command:

# Add the Crawl4AI server as an MCP provider
claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse

# List all MCP providers to verify it was added
claude mcp list

Once connected, Claude Code can directly use Crawl4AI's capabilities like screenshot capture, PDF generation, and HTML processing without having to make separate API calls.

Available MCP Tools Copy Section

When connected via MCP, the following tools are available:

• md - Generate markdown from web content
• html - Extract preprocessed HTML
• screenshot - Capture webpage screenshots
• pdf - Generate PDF documents
• execute_js - Run JavaScript on web pages
• crawl - Perform multi-URL crawling
• ask - Query the Crawl4AI library context

Testing MCP Connections Copy Section

You can test the MCP WebSocket connection using the test file included in the repository:

# From the repository root
python tests/mcp/test_mcp_socket.py

MCP Schemas Copy Section

Access the MCP tool schemas at http://localhost:11235/mcp/schema for detailed information on each tool's parameters and capabilities.

Additional API Endpoints Copy Chapter

In addition to the core /crawl and /crawl/stream endpoints, the server provides several specialized endpoints:

HTML Extraction Endpoint Copy Section

POST /html

Crawls the URL and returns preprocessed HTML optimized for schema extraction.

{
  "url": "https://example.com"
}

Screenshot Endpoint Copy Section

POST /screenshot

Captures a full-page PNG screenshot of the specified URL.

{
  "url": "https://example.com",
  "screenshot_wait_for": 2,
  "output_path": "/path/to/save/screenshot.png"
}

• screenshot_wait_for: Optional delay in seconds before capture (default: 2)
• output_path: Optional path to save the screenshot (recommended)

PDF Export Endpoint Copy Section

POST /pdf

Generates a PDF document of the specified URL.

{
  "url": "https://example.com",
  "output_path": "/path/to/save/document.pdf"
}

• output_path: Optional path to save the PDF (recommended)

JavaScript Execution Endpoint Copy Section

POST /execute_js

Executes JavaScript snippets on the specified URL and returns the full crawl result.

{
  "url": "https://example.com",
  "scripts": [
    "return document.title",
    "return Array.from(document.querySelectorAll('a')).map(a => a.href)"
  ]
}

• scripts: List of JavaScript snippets to execute sequentially

Dockerfile Parameters Copy Chapter

You can customize the image build process using build arguments (--build-arg). These are typically used via docker buildx build or within the docker-compose.yml file.

# Example: Build with 'all' features using buildx
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg INSTALL_TYPE=all \
  -t yourname/crawl4ai-all:latest \
  --load \
  . # Build from root context

Build Arguments Explained Copy Section

(Note: PYTHON_VERSION is fixed by the FROM instruction in the Dockerfile)

Build Best Practices Copy Section

1. Choose the Right Install Type
   • default: Basic installation, smallest image size. Suitable for most standard web scraping and markdown generation.
   • all: Full features including torch and transformers for advanced extraction strategies (e.g., CosineStrategy, certain LLM filters). Significantly larger image. Ensure you need these extras.
2. Platform Considerations
   • Use buildx for building multi-architecture images, especially for pushing to registries.
   • Use docker compose profiles (local-amd64, local-arm64) for easy platform-specific local builds.
3. Performance Optimization
   • The image automatically includes platform-specific optimizations (OpenMP for AMD64, OpenBLAS for ARM64).

Using the API Copy Chapter

Communicate with the running Docker server via its REST API (defaulting to http://localhost:11235). You can use the Python SDK or make direct HTTP requests.

Playground Interface Copy Section

A built-in web playground is available at http://localhost:11235/playground for testing and generating API requests. The playground allows you to:

1. Configure CrawlerRunConfig and BrowserConfig using the main library's Python syntax
2. Test crawling operations directly from the interface
3. Generate corresponding JSON for REST API requests based on your configuration

This is the easiest way to translate Python configuration to JSON requests when building integrations.

Python SDK Copy Section

Install the SDK: pip install crawl4ai

/* Detailed source-code truncated for AI context efficiency. */

(SDK parameters like timeout, verify_ssl etc. remain the same)

Second Approach: Direct API Calls Copy Section

Crucially, when sending configurations directly via JSON, they must follow the {"type": "ClassName", "params": {...}} structure for any non-primitive value (like config objects or strategies). Dictionaries must be wrapped as {"type": "dict", "value": {...}}.

(Keep the detailed explanation of Configuration Structure, Basic Pattern, Simple vs Complex, Strategy Pattern, Complex Nested Example, Quick Grammar Overview, Important Rules, Pro Tip)

More Examples *(Ensure Schema example uses type/value wrapper)*

Advanced Crawler Configuration
(Keep example, ensure cache_mode uses valid enum value like "bypass")

Extraction Strategy

{
    "crawler_config": {
        "type": "CrawlerRunConfig",
        "params": {
            "extraction_strategy": {
                "type": "JsonCssExtractionStrategy",
                "params": {
                    "schema": {
                        "type": "dict",
                        "value": {
                           "baseSelector": "article.post",
                           "fields": [
                               {"name": "title", "selector": "h1", "type": "text"},
                               {"name": "content", "selector": ".content", "type": "html"}
                           ]
                         }
                    }
                }
            }
        }
    }
}

LLM Extraction Strategy (Keep example, ensure schema uses type/value wrapper)
(Keep Deep Crawler Example)

REST API Examples Copy Section

Update URLs to use port 11235.

Simple Crawl

import requests

# Configuration objects converted to the required JSON structure
browser_config_payload = {
    "type": "BrowserConfig",
    "params": {"headless": True}
}
crawler_config_payload = {
    "type": "CrawlerRunConfig",
    "params": {"stream": False, "cache_mode": "bypass"} # Use string value of enum
}

crawl_payload = {
    "urls": ["https://httpbin.org/html"],
    "browser_config": browser_config_payload,
    "crawler_config": crawler_config_payload
}
response = requests.post(
    "http://localhost:11235/crawl", # Updated port
    # headers={"Authorization": f"Bearer {token}"},  # If JWT is enabled
    json=crawl_payload
)
print(f"Status Code: {response.status_code}")
if response.ok:
    print(response.json())
else:
    print(f"Error: {response.text}")

Streaming Results

/* Detailed source-code truncated for AI context efficiency. */

Asynchronous Jobs with Webhooks Copy Section

For long-running crawls or when you want to avoid keeping connections open, use the job queue endpoints. Instead of polling for results, configure a webhook to receive notifications when jobs complete.

Why Use Jobs & Webhooks?

• No Polling Required - Get notified when crawls complete instead of constantly checking status
• Better Resource Usage - Free up client connections while jobs run in the background
• Scalable Architecture - Ideal for high-volume crawling with TypeScript/Node.js clients or microservices
• Reliable Delivery - Automatic retry with exponential backoff (5 attempts: 1s → 2s → 4s → 8s → 16s)

How It Works

1. Submit Job → POST to /crawl/job with optional webhook_config
2. Get Task ID → Receive a task_id immediately
3. Job Runs → Crawl executes in the background
4. Webhook Fired → Server POSTs completion notification to your webhook URL
5. Fetch Results → If data wasn't included in webhook, GET /crawl/job/{task_id}

Quick Example

# Submit a crawl job with webhook notification
curl -X POST http://localhost:11235/crawl/job \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "webhook_config": {
      "webhook_url": "https://myapp.com/webhooks/crawl-complete",
      "webhook_data_in_payload": false
    }
  }'

# Response: {"task_id": "crawl_a1b2c3d4"}

Your webhook receives:

{
  "task_id": "crawl_a1b2c3d4",
  "task_type": "crawl",
  "status": "completed",
  "timestamp": "2025-10-21T10:30:00.000000+00:00",
  "urls": ["https://example.com"]
}

Then fetch the results:

curl http://localhost:11235/crawl/job/crawl_a1b2c3d4

Include Data in Webhook

Set webhook_data_in_payload: true to receive the full crawl results directly in the webhook:

curl -X POST http://localhost:11235/crawl/job \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "webhook_config": {
      "webhook_url": "https://myapp.com/webhooks/crawl-complete",
      "webhook_data_in_payload": true
    }
  }'

Your webhook receives the complete data:

{
  "task_id": "crawl_a1b2c3d4",
  "task_type": "crawl",
  "status": "completed",
  "timestamp": "2025-10-21T10:30:00.000000+00:00",
  "urls": ["https://example.com"],
  "data": {
    "markdown": "...",
    "html": "...",
    "links": {...},
    "metadata": {...}
  }
}

Webhook Authentication

Add custom headers for authentication:

{
  "urls": ["https://example.com"],
  "webhook_config": {
    "webhook_url": "https://myapp.com/webhooks/crawl",
    "webhook_data_in_payload": false,
    "webhook_headers": {
      "X-Webhook-Secret": "your-secret-token",
      "X-Service-ID": "crawl4ai-prod"
    }
  }
}

Global Default Webhook

Configure a default webhook URL in config.yml for all jobs:

webhooks:
  enabled: true
  default_url: "https://myapp.com/webhooks/default"
  data_in_payload: false
  retry:
    max_attempts: 5
    initial_delay_ms: 1000
    max_delay_ms: 32000
    timeout_ms: 30000

Now jobs without webhook_config automatically use the default webhook.

Job Status Polling (Without Webhooks)

If you prefer polling instead of webhooks, just omit webhook_config:

# Submit job
curl -X POST http://localhost:11235/crawl/job \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"]}'
# Response: {"task_id": "crawl_xyz"}

# Poll for status
curl http://localhost:11235/crawl/job/crawl_xyz

The response includes status field: "processing", "completed", or "failed".

LLM Extraction Jobs with Webhooks

The same webhook system works for LLM extraction jobs via /llm/job:

# Submit LLM extraction job with webhook
curl -X POST http://localhost:11235/llm/job \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "q": "Extract the article title, author, and main points",
    "provider": "openai/gpt-4o-mini",
    "webhook_config": {
      "webhook_url": "https://myapp.com/webhooks/llm-complete",
      "webhook_data_in_payload": true,
      "webhook_headers": {
        "X-Webhook-Secret": "your-secret-token"
      }
    }
  }'

# Response: {"task_id": "llm_1234567890"}

Your webhook receives:

{
  "task_id": "llm_1234567890",
  "task_type": "llm_extraction",
  "status": "completed",
  "timestamp": "2025-10-22T12:30:00.000000+00:00",
  "urls": ["https://example.com/article"],
  "data": {
    "extracted_content": {
      "title": "Understanding Web Scraping",
      "author": "John Doe",
      "main_points": ["Point 1", "Point 2", "Point 3"]
    }
  }
}

Key Differences for LLM Jobs:

• Task type is "llm_extraction" instead of "crawl"
• Extracted data is in data.extracted_content
• Single URL only (not an array)
• Supports schema-based extraction with schema parameter

💡 Pro tip: See WEBHOOK_EXAMPLES.md for detailed examples including TypeScript client code, Flask webhook handlers, and failure handling.

Metrics & Monitoring Copy Chapter

Keep an eye on your crawler with these endpoints:

• /health - Quick health check
• /metrics - Detailed Prometheus metrics
• /schema - Full API schema

Example health check:

curl http://localhost:11235/health

(Deployment Scenarios and Complete Examples sections remain the same, maybe update links if examples moved)

Server Configuration Copy Chapter

The server's behavior can be customized through the config.yml file.

Understanding config.yml Copy Section

The configuration file is loaded from /app/config.yml inside the container. By default, the file from deploy/docker/config.yml in the repository is copied there during the build.

Here's a detailed breakdown of the configuration options (using defaults from deploy/docker/config.yml):

/* Detailed source-code truncated for AI context efficiency. */

(JWT Authentication section remains the same, just note the default port is now 11235 for requests)

(Configuration Tips and Best Practices remain the same)

Customizing Your Configuration Copy Section

You can override the default config.yml.

Method 1: Modify Before Build

1. Edit the deploy/docker/config.yml file in your local repository clone.
2. Build the image using docker buildx or docker compose --profile local-... up --build. The modified file will be copied into the image.

Method 2: Runtime Mount (Recommended for Custom Deploys)

1. Create your custom configuration file, e.g., my-custom-config.yml locally. Ensure it contains all necessary sections.

2. Mount it when running the container:

   • Using docker run:

     bash Copy

     # Assumes my-custom-config.yml is in the current directory
     docker run -d -p 11235:11235 \
       --name crawl4ai-custom-config \
       --env-file .llm.env \
       --shm-size=1g \
       -v $(pwd)/my-custom-config.yml:/app/config.yml \
       unclecode/crawl4ai:latest # Or your specific tag

   • Using docker-compose.yml: Add a volumes section to the service definition:

     yaml Copy

     services:
       crawl4ai-hub-amd64: # Or your chosen service
         image: unclecode/crawl4ai:latest
         profiles: ["hub-amd64"]
         <<: *base-config
         volumes:
           # Mount local custom config over the default one in the container
           - ./my-custom-config.yml:/app/config.yml
           # Keep the shared memory volume from base-config
           - /dev/shm:/dev/shm

     (Note: Ensure my-custom-config.yml is in the same directory as docker-compose.yml)

💡 When mounting, your custom file completely replaces the default one. Ensure it's a valid and complete configuration.

Configuration Recommendations Copy Section

1. Security First 🔒

   • Always enable security in production
   • Use specific trusted_hosts instead of wildcards
   • Set up proper rate limiting to protect your server
   • Consider your environment before enabling HTTPS redirect

2. Resource Management 💻

   • Adjust memory_threshold_percent based on available RAM
   • Set timeouts according to your content size and network conditions
   • Use Redis for rate limiting in multi-container setups

3. Monitoring 📊

   • Enable Prometheus if you need metrics
   • Set DEBUG logging in development, INFO in production
   • Regular health check monitoring is crucial

4. Performance Tuning ⚡

   • Start with conservative rate limiter delays
   • Increase batch_process timeout for large content
   • Adjust stream_init timeout based on initial response times

Getting Help Copy Chapter

We're here to help you succeed with Crawl4AI! Here's how to get support:

• 📖 Check our full documentation
• 🐛 Found a bug? Open an issue
• 💬 Join our Discord community
• ⭐ Star us on GitHub to show support!

Summary Copy Chapter

In this guide, we've covered everything you need to get started with Crawl4AI's Docker deployment:

• Building and running the Docker container
• Configuring the environment
• Using the interactive playground for testing
• Making API requests with proper typing
• Using the Python SDK
• Asynchronous job queues with webhook notifications
• Leveraging specialized endpoints for screenshots, PDFs, and JavaScript execution
• Connecting via the Model Context Protocol (MCP)
• Monitoring your deployment

The new playground interface at http://localhost:11235/playground makes it much easier to test configurations and generate the corresponding JSON for API requests.

For AI application developers, the MCP integration allows tools like Claude Code to directly access Crawl4AI's capabilities without complex API handling.

Remember, the examples in the examples folder are your friends - they show real-world usage patterns that you can adapt for your needs.

Keep exploring, and don't hesitate to reach out if you need help! We're building something amazing together. 🚀

Happy crawling! 🕷️

File: docs/md_v2/extraction/chunking.md Copy Chapter

Chunking Strategies

Chunking strategies are critical for dividing large texts into manageable parts, enabling effective content processing and extraction. These strategies are foundational in cosine similarity-based extraction techniques, which allow users to retrieve only the most relevant chunks of content for a given query. Additionally, they facilitate direct integration into RAG (Retrieval-Augmented Generation) systems for structured and scalable workflows.

Why Use Chunking? Copy Section

1. Cosine Similarity and Query Relevance: Prepares chunks for semantic similarity analysis.
2. RAG System Integration: Seamlessly processes and stores chunks for retrieval.
3. Structured Processing: Allows for diverse segmentation methods, such as sentence-based, topic-based, or windowed approaches.

Methods of Chunking Copy Section

1. Regex-Based Chunking

Splits text based on regular expression patterns, useful for coarse segmentation.

Code Example:

class RegexChunking:
    def __init__(self, patterns=None):
        self.patterns = patterns or [r'\n\n']  # Default pattern for paragraphs

    def chunk(self, text):
        paragraphs = [text]
        for pattern in self.patterns:
            paragraphs = [seg for p in paragraphs for seg in re.split(pattern, p)]
        return paragraphs

# Example Usage
text = """This is the first paragraph.

This is the second paragraph."""
chunker = RegexChunking()
print(chunker.chunk(text))

2. Sentence-Based Chunking

Divides text into sentences using NLP tools, ideal for extracting meaningful statements.

Code Example:

from nltk.tokenize import sent_tokenize

class NlpSentenceChunking:
    def chunk(self, text):
        sentences = sent_tokenize(text)
        return [sentence.strip() for sentence in sentences]

# Example Usage
text = "This is sentence one. This is sentence two."
chunker = NlpSentenceChunking()
print(chunker.chunk(text))

3. Topic-Based Segmentation

Uses algorithms like TextTiling to create topic-coherent chunks.

Code Example:

from nltk.tokenize import TextTilingTokenizer

class TopicSegmentationChunking:
    def __init__(self):
        self.tokenizer = TextTilingTokenizer()

    def chunk(self, text):
        return self.tokenizer.tokenize(text)

# Example Usage
text = """This is an introduction.
This is a detailed discussion on the topic."""
chunker = TopicSegmentationChunking()
print(chunker.chunk(text))

4. Fixed-Length Word Chunking

Segments text into chunks of a fixed word count.

Code Example:

class FixedLengthWordChunking:
    def __init__(self, chunk_size=100):
        self.chunk_size = chunk_size

    def chunk(self, text):
        words = text.split()
        return [' '.join(words[i:i + self.chunk_size]) for i in range(0, len(words), self.chunk_size)]

# Example Usage
text = "This is a long text with many words to be chunked into fixed sizes."
chunker = FixedLengthWordChunking(chunk_size=5)
print(chunker.chunk(text))

5. Sliding Window Chunking

Generates overlapping chunks for better contextual coherence.

Code Example:

class SlidingWindowChunking:
    def __init__(self, window_size=100, step=50):
        self.window_size = window_size
        self.step = step

    def chunk(self, text):
        words = text.split()
        chunks = []
        for i in range(0, len(words) - self.window_size + 1, self.step):
            chunks.append(' '.join(words[i:i + self.window_size]))
        return chunks

# Example Usage
text = "This is a long text to demonstrate sliding window chunking."
chunker = SlidingWindowChunking(window_size=5, step=2)
print(chunker.chunk(text))

Combining Chunking with Cosine Similarity Copy Section

To enhance the relevance of extracted content, chunking strategies can be paired with cosine similarity techniques. Here’s an example workflow:

Code Example:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

class CosineSimilarityExtractor:
    def __init__(self, query):
        self.query = query
        self.vectorizer = TfidfVectorizer()

    def find_relevant_chunks(self, chunks):
        vectors = self.vectorizer.fit_transform([self.query] + chunks)
        similarities = cosine_similarity(vectors[0:1], vectors[1:]).flatten()
        return [(chunks[i], similarities[i]) for i in range(len(chunks))]

# Example Workflow
text = """This is a sample document. It has multiple sentences. 
We are testing chunking and similarity."""

chunker = SlidingWindowChunking(window_size=5, step=3)
chunks = chunker.chunk(text)
query = "testing chunking"
extractor = CosineSimilarityExtractor(query)
relevant_chunks = extractor.find_relevant_chunks(chunks)

print(relevant_chunks)

File: docs/md_v2/extraction/clustring-strategies.md Copy Chapter

Cosine Strategy

The Cosine Strategy in Crawl4AI uses similarity-based clustering to identify and extract relevant content sections from web pages. This strategy is particularly useful when you need to find and extract content based on semantic similarity rather than structural patterns.

How It Works Copy Chapter

The Cosine Strategy:

1. Breaks down page content into meaningful chunks
2. Converts text into vector representations
3. Calculates similarity between chunks
4. Clusters similar content together
5. Ranks and filters content based on relevance

Basic Usage Copy Chapter

from crawl4ai import CosineStrategy

strategy = CosineStrategy(
    semantic_filter="product reviews",    # Target content type
    word_count_threshold=10,             # Minimum words per cluster
    sim_threshold=0.3                    # Similarity threshold
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://example.com/reviews",
        extraction_strategy=strategy
    )
    
    content = result.extracted_content

Configuration Options Copy Chapter

Core Parameters Copy Section

CosineStrategy(
    # Content Filtering
    semantic_filter: str = None,       # Keywords/topic for content filtering
    word_count_threshold: int = 10,    # Minimum words per cluster
    sim_threshold: float = 0.3,        # Similarity threshold (0.0 to 1.0)
    
    # Clustering Parameters
    max_dist: float = 0.2,            # Maximum distance for clustering
    linkage_method: str = 'ward',      # Clustering linkage method
    top_k: int = 3,                   # Number of top categories to extract
    
    # Model Configuration
    model_name: str = 'sentence-transformers/all-MiniLM-L6-v2',  # Embedding model
    
    verbose: bool = False             # Enable logging
)

Parameter Details Copy Section

1. semantic_filter

• Sets the target topic or content type
• Use keywords relevant to your desired content
• Example: "technical specifications", "user reviews", "pricing information"

2. sim_threshold

• Controls how similar content must be to be grouped together
• Higher values (e.g., 0.8) mean stricter matching
• Lower values (e.g., 0.3) allow more variation

# Strict matching
strategy = CosineStrategy(sim_threshold=0.8)

# Loose matching
strategy = CosineStrategy(sim_threshold=0.3)

3. word_count_threshold

• Filters out short content blocks
• Helps eliminate noise and irrelevant content

# Only consider substantial paragraphs
strategy = CosineStrategy(word_count_threshold=50)

4. top_k

• Number of top content clusters to return
• Higher values return more diverse content

# Get top 5 most relevant content clusters
strategy = CosineStrategy(top_k=5)

Use Cases Copy Chapter

1. Article Content Extraction Copy Section

strategy = CosineStrategy(
    semantic_filter="main article content",
    word_count_threshold=100,  # Longer blocks for articles
    top_k=1                   # Usually want single main content
)

result = await crawler.arun(
    url="https://example.com/blog/post",
    extraction_strategy=strategy
)

2. Product Review Analysis Copy Section

strategy = CosineStrategy(
    semantic_filter="customer reviews and ratings",
    word_count_threshold=20,   # Reviews can be shorter
    top_k=10,                 # Get multiple reviews
    sim_threshold=0.4         # Allow variety in review content
)

3. Technical Documentation Copy Section

strategy = CosineStrategy(
    semantic_filter="technical specifications documentation",
    word_count_threshold=30,
    sim_threshold=0.6,        # Stricter matching for technical content
    max_dist=0.3             # Allow related technical sections
)

Advanced Features Copy Chapter

Custom Clustering Copy Section

strategy = CosineStrategy(
    linkage_method='complete',  # Alternative clustering method
    max_dist=0.4,              # Larger clusters
    model_name='sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2'  # Multilingual support
)

Content Filtering Pipeline Copy Section

strategy = CosineStrategy(
    semantic_filter="pricing plans features",
    word_count_threshold=15,
    sim_threshold=0.5,
    top_k=3
)

async def extract_pricing_features(url: str):
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url=url,
            extraction_strategy=strategy
        )
        
        if result.success:
            content = json.loads(result.extracted_content)
            return {
                'pricing_features': content,
                'clusters': len(content),
                'similarity_scores': [item['score'] for item in content]
            }

Best Practices Copy Chapter

1. Adjust Thresholds Iteratively

• Start with default values
• Adjust based on results
• Monitor clustering quality

2. Choose Appropriate Word Count Thresholds

• Higher for articles (100+)
• Lower for reviews/comments (20+)
• Medium for product descriptions (50+)

3. Optimize Performance

strategy = CosineStrategy(
    word_count_threshold=10,  # Filter early
    top_k=5,                 # Limit results
    verbose=True             # Monitor performance
)

4. Handle Different Content Types

# For mixed content pages
strategy = CosineStrategy(
    semantic_filter="product features",
    sim_threshold=0.4,      # More flexible matching
    max_dist=0.3,          # Larger clusters
    top_k=3                # Multiple relevant sections
)

Error Handling Copy Chapter

try:
    result = await crawler.arun(
        url="https://example.com",
        extraction_strategy=strategy
    )
    
    if result.success:
        content = json.loads(result.extracted_content)
        if not content:
            print("No relevant content found")
    else:
        print(f"Extraction failed: {result.error_message}")
        
except Exception as e:
    print(f"Error during extraction: {str(e)}")

The Cosine Strategy is particularly effective when:

• Content structure is inconsistent
• You need semantic understanding
• You want to find similar content blocks
• Structure-based extraction (CSS/XPath) isn't reliable

It works well with other strategies and can be used as a pre-processing step for LLM-based extraction.

File: docs/md_v2/extraction/llm-strategies.md Copy Chapter

Extracting JSON (LLM)

In some cases, you need to extract complex or unstructured information from a webpage that a simple CSS/XPath schema cannot easily parse. Or you want AI-driven insights, classification, or summarization. For these scenarios, Crawl4AI provides an LLM-based extraction strategy that:

1. Works with any large language model supported by LiteLLM (Ollama, OpenAI, Claude, and more).
2. Automatically splits content into chunks (if desired) to handle token limits, then combines results.
3. Lets you define a schema (like a Pydantic model) or a simpler “block” extraction approach.

Important: LLM-based extraction can be slower and costlier than schema-based approaches. If your page data is highly structured, consider using JsonCssExtractionStrategy or JsonXPathExtractionStrategy first. But if you need AI to interpret or reorganize content, read on!

1. Why Use an LLM? Copy Chapter

• Complex Reasoning: If the site’s data is unstructured, scattered, or full of natural language context.
• Semantic Extraction: Summaries, knowledge graphs, or relational data that require comprehension.
• Flexible: You can pass instructions to the model to do more advanced transformations or classification.

2. Provider-Agnostic via LiteLLM Copy Chapter

You can use LLMConfig, to quickly configure multiple variations of LLMs and experiment with them to find the optimal one for your use case. You can read more about LLMConfig here.

llm_config = LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY"))

Crawl4AI uses a “provider string” (e.g., "openai/gpt-4o", "ollama/llama2.0", "aws/titan") to identify your LLM. Any model that LiteLLM supports is fair game. You just provide:

• provider: The /<model_name> identifier (e.g., "openai/gpt-4", "ollama/llama2", "huggingface/google-flan", etc.).
• api_token: If needed (for OpenAI, HuggingFace, etc.); local models or Ollama might not require it.
• base_url (optional): If your provider has a custom endpoint.

This means you aren’t locked into a single LLM vendor. Switch or experiment easily.

3. How LLM Extraction Works Copy Chapter

3.1 Flow Copy Section

1. Chunking (optional): The HTML or markdown is split into smaller segments if it’s very long (based on chunk_token_threshold, overlap, etc.).
2. Prompt Construction: For each chunk, the library forms a prompt that includes your instruction (and possibly schema or examples).
3. LLM Inference: Each chunk is sent to the model in parallel or sequentially (depending on your concurrency).
4. Combining: The results from each chunk are merged and parsed into JSON.

3.2 `extraction_type` Copy Section

• "schema": The model tries to return JSON conforming to your Pydantic-based schema.
• "block": The model returns freeform text, or smaller JSON structures, which the library collects.

For structured data, "schema" is recommended. You provide schema=YourPydanticModel.model_json_schema().

4. Key Parameters Copy Chapter

Below is an overview of important LLM extraction parameters. All are typically set inside LLMExtractionStrategy(...). You then put that strategy in your CrawlerRunConfig(..., extraction_strategy=...).

1. llm_config (LLMConfig): e.g., "openai/gpt-4", "ollama/llama2".
   2. schema (dict): A JSON schema describing the fields you want. Usually generated by YourModel.model_json_schema().
   3. extraction_type (str): "schema" or "block".
   4. instruction (str): Prompt text telling the LLM what you want extracted. E.g., “Extract these fields as a JSON array.”
   5. chunk_token_threshold (int): Maximum tokens per chunk. If your content is huge, you can break it up for the LLM.
   6. overlap_rate (float): Overlap ratio between adjacent chunks. E.g., 0.1 means 10% of each chunk is repeated to preserve context continuity.
   7. apply_chunking (bool): Set True to chunk automatically. If you want a single pass, set False.
   8. input_format (str): Determines which crawler result is passed to the LLM. Options include:
   • "markdown": The raw markdown (default).
   • "fit_markdown": The filtered “fit” markdown if you used a content filter.
   • "html": The cleaned or raw HTML.
     9. extra_args (dict): Additional LLM parameters like temperature, max_tokens, top_p, etc.
     10. show_usage(): A method you can call to print out usage info (token usage per chunk, total cost if known).

Example:

extraction_strategy = LLMExtractionStrategy(
    llm_config = LLMConfig(provider="openai/gpt-4", api_token="YOUR_OPENAI_KEY"),
    schema=MyModel.model_json_schema(),
    extraction_type="schema",
    instruction="Extract a list of items from the text with 'name' and 'price' fields.",
    chunk_token_threshold=1200,
    overlap_rate=0.1,
    apply_chunking=True,
    input_format="html",
    extra_args={"temperature": 0.1, "max_tokens": 1000},
    verbose=True
)

5. Putting It in `CrawlerRunConfig` Copy Chapter

Important: In Crawl4AI, all strategy definitions should go inside the CrawlerRunConfig, not directly as a param in arun(). Here’s a full example:

/* Detailed source-code truncated for AI context efficiency. */

6. Chunking Details Copy Chapter

6.1 `chunk_token_threshold` Copy Section

If your page is large, you might exceed your LLM’s context window. chunk_token_threshold sets the approximate max tokens per chunk. The library calculates word→token ratio using word_token_rate (often ~0.75 by default). If chunking is enabled (apply_chunking=True), the text is split into segments.

6.2 `overlap_rate` Copy Section

To keep context continuous across chunks, we can overlap them. E.g., overlap_rate=0.1 means each subsequent chunk includes 10% of the previous chunk’s text. This is helpful if your needed info might straddle chunk boundaries.

6.3 Performance & Parallelism Copy Section

By chunking, you can potentially process multiple chunks in parallel (depending on your concurrency settings and the LLM provider). This reduces total time if the site is huge or has many sections.

7. Input Format Copy Chapter

By default, LLMExtractionStrategy uses input_format="markdown", meaning the crawler’s final markdown is fed to the LLM. You can change to:

• html: The cleaned HTML or raw HTML (depending on your crawler config) goes into the LLM.
• fit_markdown: If you used, for instance, PruningContentFilter, the “fit” version of the markdown is used. This can drastically reduce tokens if you trust the filter.
• markdown: Standard markdown output from the crawler’s markdown_generator.

This setting is crucial: if the LLM instructions rely on HTML tags, pick "html". If you prefer a text-based approach, pick "markdown".

LLMExtractionStrategy(
    # ...
    input_format="html",  # Instead of "markdown" or "fit_markdown"
)

8. Token Usage & Show Usage Copy Chapter

To keep track of tokens and cost, each chunk is processed with an LLM call. We record usage in:

• usages (list): token usage per chunk or call.
• total_usage: sum of all chunk calls.
• show_usage(): prints a usage report (if the provider returns usage data).

llm_strategy = LLMExtractionStrategy(...)
# ...
llm_strategy.show_usage()
# e.g. “Total usage: 1241 tokens across 2 chunk calls”

If your model provider doesn't return usage info, these fields might be partial or empty.

Tip: JsonCssExtractionStrategy.generate_schema() also supports token usage tracking via an optional usage parameter. See Token Usage Tracking in Schema Generation for details.

9. Example: Building a Knowledge Graph Copy Chapter

Below is a snippet combining LLMExtractionStrategy with a Pydantic schema for a knowledge graph. Notice how we pass an instruction telling the model what to parse.

import os
import json
import asyncio
from typing import List
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy

class Entity(BaseModel):
    name: str
class Relationship(BaseModel):
    entity1: Entity
    entity2: Entity
    relation_type: str

class KnowledgeGraph(BaseModel):
    entities: List[Entity]
    relationships: List[Relationship]

async def main():
    # LLM extraction strategy
    llm_strat = LLMExtractionStrategy(
        llm_config = LLMConfig(provider="openai/gpt-4", api_token=os.getenv('OPENAI_API_KEY')),
        schema=KnowledgeGraph.model_json_schema(),
        extraction_type="schema",
        instruction="Extract entities and relationships from the content. Return valid JSON.",
        chunk_token_threshold=1400,
        apply_chunking=True,
        input_format="html",
        extra_args={"temperature": 0.1, "max_tokens": 1500}
    )

    crawl_config = CrawlerRunConfig(
        extraction_strategy=llm_strat,
        cache_mode=CacheMode.BYPASS
    )

    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        # Example page
        url = "https://www.nbcnews.com/business"
        result = await crawler.arun(url=url, config=crawl_config)

        print("--- LLM RAW RESPONSE ---")
        print(result.extracted_content)
        print("--- END LLM RAW RESPONSE ---")

        if result.success:
            with open("kb_result.json", "w", encoding="utf-8") as f:
                f.write(result.extracted_content)
            llm_strat.show_usage()
        else:
            print("Crawl failed:", result.error_message)

if __name__ == "__main__":
    asyncio.run(main())

Key Observations:

• extraction_type="schema" ensures we get JSON fitting our KnowledgeGraph.
• input_format="html" means we feed HTML to the model.
• instruction guides the model to output a structured knowledge graph.

10. Best Practices & Caveats Copy Chapter

1. Cost & Latency: LLM calls can be slow or expensive. Consider chunking or smaller coverage if you only need partial data.
2. Model Token Limits: If your page + instruction exceed the context window, chunking is essential.
3. Instruction Engineering: Well-crafted instructions can drastically improve output reliability.
4. Schema Strictness: "schema" extraction tries to parse the model output as JSON. If the model returns invalid JSON, partial extraction might happen, or you might get an error.
5. Parallel vs. Serial: The library can process multiple chunks in parallel, but you must watch out for rate limits on certain providers.
6. Check Output: Sometimes, an LLM might omit fields or produce extraneous text. You may want to post-validate with Pydantic or do additional cleanup.

11. Conclusion Copy Chapter

LLM-based extraction in Crawl4AI is provider-agnostic, letting you choose from hundreds of models via LiteLLM. It’s perfect for semantically complex tasks or generating advanced structures like knowledge graphs. However, it’s slower and potentially costlier than schema-based approaches. Keep these tips in mind:

• Put your LLM strategy in CrawlerRunConfig.
• Use input_format to pick which form (markdown, HTML, fit_markdown) the LLM sees.
• Tweak chunk_token_threshold, overlap_rate, and apply_chunking to handle large content efficiently.
• Monitor token usage with show_usage().

If your site’s data is consistent or repetitive, consider JsonCssExtractionStrategy first for speed and simplicity. But if you need an AI-driven approach, LLMExtractionStrategy offers a flexible, multi-provider solution for extracting structured JSON from any website.

Next Steps:

1. Experiment with Different Providers

• Try switching the provider (e.g., "ollama/llama2", "openai/gpt-4o", etc.) to see differences in speed, accuracy, or cost.
• Pass different extra_args like temperature, top_p, and max_tokens to fine-tune your results.

2. Performance Tuning

• If pages are large, tweak chunk_token_threshold, overlap_rate, or apply_chunking to optimize throughput.
• Check the usage logs with show_usage() to keep an eye on token consumption and identify potential bottlenecks.

3. Validate Outputs

• If using extraction_type="schema", parse the LLM’s JSON with a Pydantic model for a final validation step.
• Log or handle any parse errors gracefully, especially if the model occasionally returns malformed JSON.

4. Explore Hooks & Automation

• Integrate LLM extraction with hooks for complex pre/post-processing.
• Use a multi-step pipeline: crawl, filter, LLM-extract, then store or index results for further analysis.

Last Updated: 2025-01-01

That’s it for Extracting JSON (LLM)—now you can harness AI to parse, classify, or reorganize data on the web. Happy crawling!

File: docs/md_v2/core/ask-ai.md Copy Chapter

File: docs/md_v2/core/browser-crawler-config.md Copy Chapter

Browser, Crawler & LLM Configuration (Quick Overview)

Crawl4AI's flexibility stems from two key classes:

1. BrowserConfig – Dictates how the browser is launched and behaves (e.g., headless or visible, proxy, user agent).
2. CrawlerRunConfig – Dictates how each crawl operates (e.g., caching, extraction, timeouts, JavaScript code to run, etc.).
3. LLMConfig - Dictates how LLM providers are configured. (model, api token, base url, temperature etc.)

In most examples, you create one BrowserConfig for the entire crawler session, then pass a fresh or re-used CrawlerRunConfig whenever you call arun(). This tutorial shows the most commonly used parameters. If you need advanced or rarely used fields, see the Configuration Parameters.

1. BrowserConfig Essentials Copy Chapter

class BrowserConfig:
    def __init__(
        browser_type="chromium",
        headless=True,
        browser_mode="dedicated",
        use_managed_browser=False,
        cdp_url=None,
        debugging_port=9222,
        host="localhost",
        proxy_config=None,
        viewport_width=1080,
        viewport_height=600,
        verbose=True,
        use_persistent_context=False,
        user_data_dir=None,
        cookies=None,
        headers=None,
        user_agent=(
            # "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) AppleWebKit/537.36 "
            # "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
            # "(KHTML, like Gecko) Chrome/116.0.5845.187 Safari/604.1 Edg/117.0.2045.47"
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36"
        ),
        user_agent_mode="",
        text_mode=False,
        light_mode=False,
        extra_args=None,
        enable_stealth=False,
        # ... other advanced parameters omitted here
    ):
        ...

Key Fields to Note Copy Section

1.⠀browser_type

• Options: "chromium", "firefox", or "webkit".
• Defaults to "chromium".
• If you need a different engine, specify it here.

2.⠀headless

• True: Runs the browser in headless mode (invisible browser).
• False: Runs the browser in visible mode, which helps with debugging.

3.⠀browser_mode

• Determines how the browser should be initialized:
  • "dedicated" (default): Creates a new browser instance each time
  • "builtin": Uses the builtin CDP browser running in background
  • "custom": Uses explicit CDP settings provided in cdp_url
  • "docker": Runs browser in Docker container with isolation

4.⠀use_managed_browser & cdp_url

• use_managed_browser=True: Launch browser using Chrome DevTools Protocol (CDP) for advanced control
• cdp_url: URL for CDP endpoint (e.g., "ws://localhost:9222/devtools/browser/")
• Automatically set based on browser_mode

5.⠀debugging_port & host

• debugging_port: Port for browser debugging protocol (default: 9222)
• host: Host for browser connection (default: "localhost")

6.⠀proxy_config

• A ProxyConfig object or dictionary with fields like:

{
    "server": "http://proxy.example.com:8080", 
    "username": "...", 
    "password": "..."
}

• Leave as None if a proxy is not required.

7.⠀viewport_width & viewport_height

• The initial window size.
• Some sites behave differently with smaller or bigger viewports.

8.⠀device_scale_factor

• Controls the device pixel ratio (DPR) for rendering. Default is 1.0.
• Set to 2.0 for Retina-quality screenshots (e.g., a 1920×1080 viewport produces 3840×2160 images).
• Higher values increase screenshot size and rendering time proportionally.

9.⠀verbose

• If True, prints extra logs.
• Handy for debugging.

9.⠀use_persistent_context

• If True, uses a persistent browser profile, storing cookies/local storage across runs.
• Typically also set user_data_dir to point to a folder.

10.⠀cookies & headers
- If you want to start with specific cookies or add universal HTTP headers to the browser context, set them here.
- E.g. cookies=[{"name": "session", "value": "abc123", "domain": "example.com"}].

11.⠀user_agent & user_agent_mode
- user_agent: Custom User-Agent string. If None, a default is used.
- user_agent_mode: Set to "random" for randomization (helps fight bot detection).

12.⠀text_mode & light_mode
- text_mode=True disables images, possibly speeding up text-only crawls.
- light_mode=True turns off certain background features for performance.

13.⠀avoid_ads & avoid_css
- avoid_ads=True blocks requests to common ad and tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level. Reduces network overhead and memory usage.
- avoid_css=True blocks loading of CSS files (.css, .less, .scss, .sass), useful when you only need text content and want faster, leaner crawls.
- Both default to False (opt-in). Can be combined with each other and with text_mode.

14.⠀extra_args
- Additional flags for the underlying browser.
- E.g. ["--disable-extensions"].

15.⠀enable_stealth
- If True, enables stealth mode using playwright-stealth.
- Modifies browser fingerprints to avoid basic bot detection.
- Default is False. Recommended for sites with bot protection.

Helper Methods Copy Section

Both configuration classes provide a clone() method to create modified copies:

# Create a base browser config
base_browser = BrowserConfig(
    browser_type="chromium",
    headless=True,
    text_mode=True
)

# Create a visible browser config for debugging
debug_browser = base_browser.clone(
    headless=False,
    verbose=True
)

Class-Level Defaults Copy Section

Both BrowserConfig and CrawlerRunConfig support class-level default overrides via set_defaults(). This is useful in server/cloud deployments where every config instance needs the same base settings — set them once at startup instead of repeating at every call site.

from crawl4ai import BrowserConfig, CrawlerRunConfig

# At application startup — one time
BrowserConfig.set_defaults(
    cache_cdp_connection=True,
    cdp_close_delay=0,
    create_isolated_context=True,
)
CrawlerRunConfig.set_defaults(verbose=False)

# Every new instance automatically inherits those defaults
cfg = BrowserConfig(cdp_url="ws://localhost:9222")
# → cache_cdp_connection=True, cdp_close_delay=0, create_isolated_context=True

# Explicit values still win
cfg = BrowserConfig(cdp_url="ws://localhost:9222", cache_cdp_connection=False)
# → cache_cdp_connection=False (explicit overrides the class default)

Available methods (on both BrowserConfig and CrawlerRunConfig):

Note: Class defaults are independent per class — BrowserConfig.set_defaults() does not affect CrawlerRunConfig, and vice versa. Defaults are stored in memory and apply for the lifetime of the process.

Minimal Example:

from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_conf = BrowserConfig(
    browser_type="firefox",
    headless=False,
    text_mode=True
)

async with AsyncWebCrawler(config=browser_conf) as crawler:
    result = await crawler.arun("https://example.com")
    print(result.markdown[:300])

2. CrawlerRunConfig Essentials Copy Chapter

class CrawlerRunConfig:
    def __init__(
        word_count_threshold=200,
        extraction_strategy=None,
        chunking_strategy=RegexChunking(),
        markdown_generator=None,
        cache_mode=CacheMode.BYPASS,
        js_code=None,
        c4a_script=None,
        wait_for=None,
        screenshot=False,
        pdf=False,
        capture_mhtml=False,
        # Location and Identity Parameters
        locale=None,            # e.g. "en-US", "fr-FR"
        timezone_id=None,       # e.g. "America/New_York"
        geolocation=None,       # GeolocationConfig object
        # Proxy Configuration
        proxy_config=None,
        proxy_rotation_strategy=None,
        # Page Interaction Parameters
        scan_full_page=False,
        scroll_delay=0.2,
        wait_until="domcontentloaded",
        page_timeout=60000,
        delay_before_return_html=0.1,
        # URL Matching Parameters
        url_matcher=None,       # For URL-specific configurations
        match_mode=MatchMode.OR,
        verbose=True,
        stream=False,  # Enable streaming for arun_many()
        # ... other advanced parameters omitted
    ):
        ...

Key Fields to Note Copy Section

1.⠀word_count_threshold:

• The minimum word count before a block is considered.
• If your site has lots of short paragraphs or items, you can lower it.

2.⠀extraction_strategy:

• Where you plug in JSON-based extraction (CSS, LLM, etc.).
• If None, no structured extraction is done (only raw/cleaned HTML + markdown).

3.⠀chunking_strategy:

• Strategy to chunk content before extraction.
• Defaults to RegexChunking(). Can be customized for different chunking approaches.

4.⠀markdown_generator:

• E.g., DefaultMarkdownGenerator(...), controlling how HTML→Markdown conversion is done.
• If None, a default approach is used.

5.⠀cache_mode:

• Controls caching behavior (ENABLED, BYPASS, DISABLED, etc.).
• Defaults to CacheMode.BYPASS.

6.⠀js_code, js_code_before_wait, & c4a_script:

• js_code: JavaScript to run after wait_for completes — on the fully-loaded page.
• js_code_before_wait: JavaScript to run before wait_for — for triggering loading that wait_for then checks.
• c4a_script: C4A script that compiles to JavaScript.
• Great for "Load More" buttons or user interactions.

7.⠀wait_for:

• A CSS or JS expression to wait for before extracting content.
• Common usage: wait_for="css:.main-loaded" or wait_for="js:() => window.loaded === true".

8.⠀flatten_shadow_dom:

• If True, flattens Shadow DOM content into the light DOM before HTML capture.
• Essential for sites built with Web Components (Stencil, Lit, Shoelace, etc.).
• Also force-opens closed shadow roots. See Flattening Shadow DOM.

9.⠀screenshot, pdf, & capture_mhtml:

• If True, captures a screenshot, PDF, or MHTML snapshot after the page is fully loaded.
• The results go to result.screenshot (base64), result.pdf (bytes), or result.mhtml (string).
• Use force_viewport_screenshot=True to capture only the visible viewport instead of the full page. This is faster and produces smaller images when you don't need a full-page screenshot.

9.⠀Location Parameters:

• locale: Browser's locale (e.g., "en-US", "fr-FR") for language preferences
• timezone_id: Browser's timezone (e.g., "America/New_York", "Europe/Paris")
• geolocation: GPS coordinates via GeolocationConfig(latitude=48.8566, longitude=2.3522)
• See Identity Based Crawling

10.⠀Proxy Configuration:
- proxy_config: Single ProxyConfig or list[ProxyConfig] — proxies tried in order. Pass a list for automatic escalation.
- proxy_rotation_strategy: Strategy for rotating proxies during crawls

11.⠀Anti-Bot Retry & Fallback (see Anti-Bot & Fallback):
- max_retries: Number of retry rounds when blocking is detected (default: 0). Each round tries all proxies in proxy_config.
- fallback_fetch_function: Async function called as last resort — takes URL, returns raw HTML

12.⠀Page Interaction Parameters:
- scan_full_page: If True, scroll through the entire page to load all content
- wait_until: Condition to wait for when navigating (e.g., "domcontentloaded", "networkidle")
- page_timeout: Timeout in milliseconds for page operations (default: 60000)
- delay_before_return_html: Delay in seconds before retrieving final HTML.

13.⠀url_matcher & match_mode:
- Enable URL-specific configurations when used with arun_many().
- Set url_matcher to patterns (glob, function, or list) to match specific URLs.
- Use match_mode (OR/AND) to control how multiple patterns combine.
- See URL-Specific Configurations for examples.

13.⠀verbose:
- Logs additional runtime details.
- Overlaps with the browser's verbosity if also set to True in BrowserConfig.

14.⠀stream:
- If True, enables streaming mode for arun_many() to process URLs as they complete.
- Allows handling results incrementally instead of waiting for all URLs to finish.

Helper Methods Copy Section

The clone() method is particularly useful for creating variations of your crawler configuration:

# Create a base configuration
base_config = CrawlerRunConfig(
    cache_mode=CacheMode.ENABLED,
    word_count_threshold=200,
    wait_until="networkidle"
)

# Create variations for different use cases
stream_config = base_config.clone(
    stream=True,  # Enable streaming mode
    cache_mode=CacheMode.BYPASS
)

debug_config = base_config.clone(
    page_timeout=120000,  # Longer timeout for debugging
    verbose=True
)

The clone() method:

• Creates a new instance with all the same settings
• Updates only the specified parameters
• Leaves the original configuration unchanged
• Perfect for creating variations without repeating all parameters

3. LLMConfig Essentials Copy Chapter

Key fields to note Copy Section

1.⠀provider:

• Which LLM provider to use.
• Possible values are "ollama/llama3","groq/llama3-70b-8192","groq/llama3-8b-8192", "openai/gpt-4o-mini" ,"openai/gpt-4o","openai/o1-mini","openai/o1-preview","openai/o3-mini","openai/o3-mini-high","anthropic/claude-3-haiku-20240307","anthropic/claude-3-opus-20240229","anthropic/claude-3-sonnet-20240229","anthropic/claude-3-5-sonnet-20240620","gemini/gemini-pro","gemini/gemini-1.5-pro","gemini/gemini-2.0-flash","gemini/gemini-2.0-flash-exp","gemini/gemini-2.0-flash-lite-preview-02-05","deepseek/deepseek-chat"(default: "openai/gpt-4o-mini")

2.⠀api_token:
- Optional. When not provided explicitly, api_token will be read from environment variables based on provider. For example: If a gemini model is passed as provider then,"GEMINI_API_KEY" will be read from environment variables
- API token of LLM provider eg: api_token = "gsk_1ClHGGJ7Lpn4WGybR7vNWGdyb3FY7zXEw3SCiy0BAVM9lL8CQv"
- Environment variable - use with prefix "env:" eg:api_token = "env: GROQ_API_KEY"

3.⠀base_url:

• If your provider has a custom endpoint

4.⠀Retry/backoff controls (optional):

• backoff_base_delay (default 2 seconds) – base delay inserted before the first retry when the provider returns a rate-limit response.
• backoff_max_attempts (default 3) – total number of attempts (initial call plus retries) before the request is surfaced as an error.
• backoff_exponential_factor (default 2) – growth rate for the retry delay (delay = base_delay * factor^attempt).
• These values are forwarded to the shared perform_completion_with_backoff helper, ensuring every strategy that consumes your LLMConfig honors the same throttling policy.

llm_config = LLMConfig(
    provider="openai/gpt-4o-mini",
    api_token=os.getenv("OPENAI_API_KEY"),
    backoff_base_delay=1, # optional
    backoff_max_attempts=5, # optional
    backoff_exponential_factor=3, #optional
)

4. Putting It All Together Copy Chapter

In a typical scenario, you define one BrowserConfig for your crawler session, then create one or more CrawlerRunConfig & LLMConfig depending on each call's needs:

/* Detailed source-code truncated for AI context efficiency. */

5. Next Steps Copy Chapter

For a detailed list of available parameters (including advanced ones), see:

• BrowserConfig, CrawlerRunConfig & LLMConfig Reference

You can explore topics like:

• Custom Hooks & Auth (Inject JavaScript or handle login forms).
• Session Management (Re-use pages, preserve state across multiple calls).
• Magic Mode or Identity-based Crawling (Fight bot detection by simulating user behavior).
• Advanced Caching (Fine-tune read/write cache modes).

6. Conclusion Copy Chapter

BrowserConfig, CrawlerRunConfig and LLMConfig give you straightforward ways to define:

• Which browser to launch, how it should run, and any proxy or user agent needs.
• How each crawl should behave—caching, timeouts, JavaScript code, extraction strategies, etc.
• Which LLM provider to use, api token, temperature and base url for custom endpoints

Use them together for clear, maintainable code, and when you need more specialized behavior, check out the advanced parameters in the reference docs. Happy crawling!

File: docs/md_v2/core/cache-modes.md Copy Chapter

Crawl4AI Cache System and Migration Guide

Overview Copy Chapter

Starting from version 0.5.0, Crawl4AI introduces a new caching system that replaces the old boolean flags with a more intuitive CacheMode enum. This change simplifies cache control and makes the behavior more predictable.

Old vs New Approach Copy Chapter

Old Way (Deprecated) Copy Section

The old system used multiple boolean flags:

• bypass_cache: Skip cache entirely
• disable_cache: Disable all caching
• no_cache_read: Don't read from cache
• no_cache_write: Don't write to cache

New Way (Recommended) Copy Section

The new system uses a single CacheMode enum:

• CacheMode.ENABLED: Normal caching (read/write)
• CacheMode.DISABLED: No caching at all
• CacheMode.READ_ONLY: Only read from cache
• CacheMode.WRITE_ONLY: Only write to cache
• CacheMode.BYPASS: Skip cache for this operation

Migration Example Copy Chapter

Old Code (Deprecated) Copy Section

from crawl4ai import AsyncWebCrawler

async def old_code(crawler: AsyncWebCrawler):
    # Legacy `bypass_cache` / `disable_cache` / `no_cache_read` / `no_cache_write`
    # were removed in v0.5+. This example no longer applies:
    result = await crawler.arun(
        url="https://www.nbcnews.com/business",
        # cache_mode is the only cache option now.
    )
    print(len(result.markdown))

New Code (Recommended) Copy Section

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode
from crawl4ai.async_configs import CrawlerRunConfig

async def use_proxy():
    # Use CacheMode in CrawlerRunConfig
    config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)  
    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
            config=config  # Pass the configuration object
        )
        print(len(result.markdown))

async def main():
    await use_proxy()

if __name__ == "__main__":
    asyncio.run(main())

Common Migration Patterns Copy Chapter