openevolve Documentation

File: README.md Copy Chapter

OpenEvolve

Why OpenEvolve? Copy Chapter

| ### Autonomous Discovery
LLMs don't just optimize—they discover entirely new algorithms. No human guidance needed. | ### Proven Results
2-3x speedups on real hardware. State-of-the-art circle packing. Breakthrough optimizations. | ### Research Grade

Full reproducibility, extensive evaluation pipelines, and scientific rigor built-in. |
| --- | --- | --- |

OpenEvolve vs Manual Optimization:

Proven Achievements Copy Chapter

🚀 Quick Start Copy Chapter

Get from zero to evolving code in 30 seconds:

# Install OpenEvolve
pip install openevolve

# The example uses Google Gemini by default (free tier available)
# Get your API key from: https://aistudio.google.com/apikey
export OPENAI_API_KEY="your-gemini-api-key"  # Yes, use OPENAI_API_KEY env var

# Run your first evolution!
python openevolve-run.py examples/function_minimization/initial_program.py \
  examples/function_minimization/evaluator.py \
  --config examples/function_minimization/config.yaml \
  --iterations 50

Note: The example config uses Gemini by default, but you can use any OpenAI-compatible provider by modifying the config.yaml. See the configs for full configuration options.

**Library Usage** Copy Section

OpenEvolve can be used as a library without any external files:

from openevolve import run_evolution, evolve_function

# Evolution with inline code (no files needed!)
result = run_evolution(
    initial_program='''
    def fibonacci(n):
        if n <= 1: return n
        return fibonacci(n-1) + fibonacci(n-2)
    ''',
    evaluator=lambda path: {"score": benchmark_fib(path)},
    iterations=100
)

# Evolve Python functions directly
def bubble_sort(arr):
    for i in range(len(arr)):
        for j in range(len(arr)-1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[j] 
    return arr

result = evolve_function(
    bubble_sort,
    test_cases=[([3,1,2], [1,2,3]), ([5,2,8], [2,5,8])],
    iterations=50
)
print(f"Evolved sorting algorithm: {result.best_code}")

Prefer Docker? See the Installation & Setup section for Docker options.

See It In Action Copy Chapter

Circle Packing: From Random to State-of-the-Art

Watch OpenEvolve discover optimal circle packing in real-time:

Result: Matches published benchmarks for n=26 circle packing problem.

GPU Kernel Evolution

Before (Baseline):

// Standard attention implementation
kernel void attention_baseline(/* ... */) {
    // Generic matrix multiplication
    float sum = 0.0;
    for (int i = 0; i < seq_len; i++) {
        sum += query[tid] * key[i];
    }
}

After Evolution (2.8x faster):

// OpenEvolve discovered optimization
kernel void attention_evolved(/* ... */) {
    // Hardware-aware tiling + unified memory optimization
    threadgroup float shared_mem[256];
    // ... evolved algorithm exploiting Apple Silicon architecture
}

Performance Impact: 2.8x speedup on Apple M1 Pro, maintaining numerical accuracy.

How OpenEvolve Works Copy Chapter

OpenEvolve implements a sophisticated evolutionary coding pipeline that goes far beyond simple optimization:

**Core Innovation**: MAP-Elites + LLMs Copy Section

• Quality-Diversity Evolution: Maintains diverse populations across feature dimensions
• Island-Based Architecture: Multiple populations prevent premature convergence
• LLM Ensemble: Multiple models with intelligent fallback strategies
• Artifact Side-Channel: Error feedback improves subsequent generations

**Advanced Features** Copy Section

Scientific Reproducibility

• Comprehensive Seeding: Every component (LLM, database, evaluation) is seeded
• Default Seed=42: Immediate reproducible results out of the box
• Deterministic Evolution: Exact reproduction of runs across machines
• Component Isolation: Hash-based isolation prevents cross-contamination

Advanced LLM Integration

• Universal API: Works with OpenAI, Google, Claude Code CLI, local models, and proxies
• Intelligent Ensembles: Weighted combinations with sophisticated fallback
• Test-Time Compute: Enhanced reasoning through proxy systems (see OptiLLM setup)
• Plugin Ecosystem: Support for advanced reasoning plugins

Evolution Algorithm Innovations

• Double Selection: Different programs for performance vs inspiration
• Adaptive Feature Dimensions: Custom quality-diversity metrics
• Migration Patterns: Ring topology with controlled gene flow
• Multi-Strategy Sampling: Elite, diverse, and exploratory selection

Perfect For Copy Chapter

🛠 Installation & Setup Copy Chapter

Requirements Copy Section

• Python: 3.10+
• LLM Access: Any OpenAI-compatible API, or Claude Code CLI
• Optional: Docker for containerized runs

Installation Options Copy Section

📦 PyPI (Recommended)

pip install openevolve

🔧 Development Install

git clone https://github.com/algorithmicsuperintelligence/openevolve.git
cd openevolve
pip install -e ".[dev]"

🐳 Docker

# Pull the image
docker pull ghcr.io/algorithmicsuperintelligence/openevolve:latest

# Run an example
docker run --rm -v $(pwd):/app ghcr.io/algorithmicsuperintelligence/openevolve:latest \
  examples/function_minimization/initial_program.py \
  examples/function_minimization/evaluator.py --iterations 100

Cost Estimation Copy Section

Cost depends on your LLM provider and iterations:

• o3: ~$0.15-0.60 per iteration (depending on code size)
• o3-mini: ~$0.03-0.12 per iteration (more cost-effective)
• Gemini-2.5-Pro: ~$0.08-0.30 per iteration
• Gemini-2.5-Flash: ~$0.01-0.05 per iteration (fastest and cheapest)
• Local models: Nearly free after setup
• OptiLLM: Use cheaper models with test-time compute for better results

Cost-saving tips:

• Start with fewer iterations (100-200)
• Use o3-mini, Gemini-2.5-Flash or local models for exploration
• Use cascade evaluation to filter bad programs early
• Configure smaller population sizes initially

LLM Provider Setup Copy Section

OpenEvolve works with any OpenAI-compatible API:

🔥 OpenAI (Direct)

export OPENAI_API_KEY="sk-..."
# Uses OpenAI endpoints by default

🤖 Google Gemini

# config.yaml
llm:
  api_base: "https://generativelanguage.googleapis.com/v1beta/openai/"
  model: "gemini-2.5-pro"

export OPENAI_API_KEY="your-gemini-api-key"

🏠 Local Models (Ollama/vLLM)

# config.yaml
llm:
  api_base: "http://localhost:11434/v1"  # Ollama
  model: "codellama:7b"

⚡ OptiLLM (Advanced)

For maximum flexibility with rate limiting, model routing, and test-time compute:

# Install OptiLLM
pip install optillm

# Start OptiLLM proxy
optillm --port 8000

# Point OpenEvolve to OptiLLM
export OPENAI_API_KEY="your-actual-key"

llm:
  api_base: "http://localhost:8000/v1"
  model: "moa&readurls-o3"  # Test-time compute + web access

🔮 Claude Code CLI (No API Key)

Use the Claude Code CLI as the LLM backend — no API keys needed, authentication uses the CLI's OAuth session.

# Install and authenticate
npm install @anthropic-ai/claude-code
claude login

# config.yaml
llm:
  provider: "claude_code"
  models:
    - name: "sonnet"
      weight: 0.8
      max_tokens: 16000
      max_budget_usd: 1.0
    - name: "haiku"
      weight: 0.2
      max_tokens: 8000

See the Claude Code quickstart example for a complete walkthrough.

Examples Gallery Copy Chapter

**Quick Example**: Function Minimization Copy Section

Watch OpenEvolve evolve from random search to sophisticated optimization:

# Initial Program (Random Search)
def minimize_function(func, bounds, max_evals=1000):
    best_x, best_val = None, float('inf')
    for _ in range(max_evals):
        x = random_point_in_bounds(bounds)
        val = func(x)
        if val < best_val:
            best_x, best_val = x, val
    return best_x, best_val

Evolution Process

# Evolved Program (Simulated Annealing + Adaptive Cooling)
def minimize_function(func, bounds, max_evals=1000):
    x = random_point_in_bounds(bounds)
    temp = adaptive_initial_temperature(func, bounds)
    
    for i in range(max_evals):
        neighbor = generate_neighbor(x, temp, bounds)
        delta = func(neighbor) - func(x)
        
        if delta < 0 or random.random() < exp(-delta/temp):
            x = neighbor
            
        temp *= adaptive_cooling_rate(i, max_evals)  # Dynamic cooling
    
    return x, func(x)

Performance: 100x improvement in convergence speed!

**Advanced Examples** Copy Section

Prompt Evolution

Evolve prompts instead of code for better LLM performance. See the LLM Prompt Optimization example for a complete case study with HotpotQA achieving +23% accuracy improvement.

Full Example

🏁 Competitive Programming

Automatic solution generation for programming contests:

# Problem: Find maximum subarray sum
# OpenEvolve discovers multiple approaches:

# Evolution Path 1: Brute Force → Kadane's Algorithm
# Evolution Path 2: Divide & Conquer → Optimized Kadane's
# Evolution Path 3: Dynamic Programming → Space-Optimized DP

Online Judge Integration

Configuration Copy Chapter

OpenEvolve offers extensive configuration for advanced users:

# Advanced Configuration Example
max_iterations: 1000
random_seed: 42  # Full reproducibility

llm:
  # Ensemble configuration
  models:
    - name: "gemini-2.5-pro"
      weight: 0.6
    - name: "gemini-2.5-flash"
      weight: 0.4
  temperature: 0.7

database:
  # MAP-Elites quality-diversity
  population_size: 500
  num_islands: 5  # Parallel evolution
  migration_interval: 20
  feature_dimensions: ["complexity", "diversity", "performance"]

  # Optional novelty filtering with Gemini embeddings
  embedding_model: "gemini-embedding-001"
  similarity_threshold: 0.99

evaluator:
  enable_artifacts: true      # Error feedback to LLM
  cascade_evaluation: true    # Multi-stage testing
  use_llm_feedback: true      # AI code quality assessment

prompt:
  # Sophisticated inspiration system
  num_top_programs: 3         # Best performers
  num_diverse_programs: 2     # Creative exploration
  include_artifacts: true     # Execution feedback
  
  # Custom templates
  template_dir: "custom_prompts/"
  use_template_stochasticity: true  # Randomized prompts

For Gemini embeddings, set GEMINI_API_KEY. GOOGLE_API_KEY is also supported
as a fallback. OpenEvolve uses Google's OpenAI-compatible endpoint automatically.

🎯 Feature Engineering

Control how programs are organized in the quality-diversity grid:

database:
  feature_dimensions: 
    - "complexity"      # Built-in: code length
    - "diversity"       # Built-in: structural diversity
    - "performance"     # Custom: from your evaluator
    - "memory_usage"    # Custom: from your evaluator
    
  feature_bins:
    complexity: 10      # 10 complexity levels
    performance: 20     # 20 performance buckets
    memory_usage: 15    # 15 memory usage categories

Important: Return raw values from evaluator, OpenEvolve handles binning automatically.

🎨 Custom Prompt Templates

Advanced prompt engineering with custom templates:

prompt:
  template_dir: "custom_templates/"
  use_template_stochasticity: true
  template_variations:
    greeting:
      - "Let's enhance this code:"
      - "Time to optimize:"
      - "Improving the algorithm:"
    improvement_suggestion:
      - "Here's how we could improve this code:"
      - "I suggest the following improvements:"
      - "We can enhance this code by:"

How it works: Place {greeting} or {improvement_suggestion} placeholders in your templates, and OpenEvolve will randomly choose from the variations for each generation, adding diversity to prompts.

See prompt examples for complete template customization.

Crafting Effective System Messages Copy Chapter

System messages are the secret to successful evolution. They guide the LLM's understanding of your domain, constraints, and optimization goals. A well-crafted system message can be the difference between random mutations and targeted improvements.

Why System Messages Matter Copy Section

The system message in your config.yaml is arguably the most important component for evolution success:

• Domain Expertise: Provides LLM with specific knowledge about your problem space
• Constraint Awareness: Defines what can and cannot be changed during evolution
• Optimization Focus: Guides the LLM toward meaningful improvements
• Error Prevention: Helps avoid common pitfalls and compilation errors

The Iterative Creation Process Copy Section

Based on successful OpenEvolve implementations, system messages are best created through iteration:

🔄 Step-by-Step Process

Phase 1: Initial Draft

1. Start with a basic system message describing your goal
2. Run 20-50 evolution iterations to observe behavior
3. Note where the system gets "stuck" or makes poor choices

Phase 2: Refinement

4. Add specific guidance based on observed issues
5. Include domain-specific terminology and concepts
6. Define clear constraints and optimization targets
7. Run another batch of iterations

Phase 3: Specialization

8. Add detailed examples of good vs bad approaches
9. Include specific library/framework guidance
10. Add error avoidance patterns you've observed
11. Fine-tune based on artifact feedback

Phase 4: Optimization

12. Consider using OpenEvolve itself to optimize your prompt
13. Measure improvements using combined score metrics

Examples by Complexity Copy Section

**Simple: General Optimization**

prompt:
  system_message: |
    You are an expert programmer specializing in optimization algorithms.
    Your task is to improve a function minimization algorithm to find the
    global minimum reliably, escaping local minima that might trap simple algorithms.

**Intermediate: Domain-Specific Guidance**

prompt:
  system_message: |
    You are an expert prompt engineer. Your task is to revise prompts for LLMs.

    Your improvements should:
    * Clarify vague instructions and eliminate ambiguity
    * Strengthen alignment between prompt and desired task outcome
    * Improve robustness against edge cases
    * Include formatting instructions and examples where helpful
    * Avoid unnecessary verbosity

    Return only the improved prompt text without explanations.

⚡ **Advanced: Hardware-Specific Optimization**

prompt:
  system_message: |
    You are an expert Metal GPU programmer specializing in custom attention
    kernels for Apple Silicon.

    # TARGET: Optimize Metal Kernel for Grouped Query Attention (GQA)
    # HARDWARE: Apple M-series GPUs with unified memory architecture
    # GOAL: 5-15% performance improvement

    # OPTIMIZATION OPPORTUNITIES:
    **1. Memory Access Pattern Optimization:**
    - Coalesced access patterns for Apple Silicon
    - Vectorized loading using SIMD
    - Pre-compute frequently used indices

    **2. Algorithm Fusion:**
    - Combine max finding with score computation
    - Reduce number of passes through data

    # CONSTRAINTS - CRITICAL SAFETY RULES:
    **MUST NOT CHANGE:**
    ❌ Kernel function signature
    ❌ Template parameter names or types
    ❌ Overall algorithm correctness

    **ALLOWED TO OPTIMIZE:**
    ✅ Memory access patterns and indexing
    ✅ Computation order and efficiency
    ✅ Vectorization and SIMD utilization
    ✅ Apple Silicon specific optimizations

Best Practices Copy Section

🎨 Prompt Engineering Patterns

Structure Your Message: Start with role definition → Define task/context → List optimization opportunities → Set constraints → Success criteria

Use Specific Examples:

# Good: "Focus on reducing memory allocations. Example: Replace `new Vector()` with pre-allocated arrays."
# Avoid: "Make the code faster"

Include Domain Knowledge:

# Good: "For GPU kernels: 1) Memory coalescing 2) Occupancy 3) Shared memory usage"
# Avoid: "Optimize the algorithm"

Set Clear Boundaries:

system_message: |
  MUST NOT CHANGE: ❌ Function signatures ❌ Algorithm correctness ❌ External API
  ALLOWED: ✅ Internal implementation ✅ Data structures ✅ Performance optimizations

🔬 Advanced Techniques

Artifact-Driven Iteration: Enable artifacts in config → Include common error patterns in system message → Add guidance based on stderr/warning patterns

Multi-Phase Evolution: Start broad ("Explore different algorithmic approaches"), then focus ("Given successful simulated annealing, focus on parameter tuning")

Template Stochasticity: See the Configuration section for complete template variation examples.

Meta-Evolution: Using OpenEvolve to Optimize Prompts Copy Section

You can use OpenEvolve to evolve your system messages themselves! This powerful technique lets you optimize prompts for better LLM performance automatically.

See the LLM Prompt Optimization example for a complete implementation, including the HotpotQA case study with +23% accuracy improvement.

Common Pitfalls to Avoid Copy Section

• Too Vague: "Make the code better" → Specify exactly what "better" means
• Too Restrictive: Over-constraining can prevent useful optimizations
• Missing Context: Include relevant domain knowledge and terminology
• No Examples: Concrete examples guide LLM better than abstract descriptions
• Ignoring Artifacts: Don't refine prompts based on error feedback

Artifacts & Debugging Copy Chapter

Artifacts side-channel provides rich feedback to accelerate evolution:

# Evaluator can return execution context
from openevolve.evaluation_result import EvaluationResult

return EvaluationResult(
    metrics={"performance": 0.85, "correctness": 1.0},
    artifacts={
        "stderr": "Warning: suboptimal memory access pattern",
        "profiling_data": {...},
        "llm_feedback": "Code is correct but could use better variable names",
        "build_warnings": ["unused variable x"]
    }
)

Next generation prompt automatically includes:

## Previous Execution Feedback
⚠️ Warning: suboptimal memory access pattern
💡 LLM Feedback: Code is correct but could use better variable names
🔧 Build Warnings: unused variable x

This creates a feedback loop where each generation learns from previous mistakes!

Visualization Copy Chapter

Real-time evolution tracking with interactive web interface:

# Install visualization dependencies
pip install -r scripts/requirements.txt

# Launch interactive visualizer
python scripts/visualizer.py

# Or visualize specific checkpoint
python scripts/visualizer.py --path examples/function_minimization/openevolve_output/checkpoints/checkpoint_100/

Features:

• 🌳 Evolution tree with parent-child relationships
• 📈 Performance tracking across generations
• 🔍 Code diff viewer showing mutations
• 📊 MAP-Elites grid visualization
• 🎯 Multi-metric analysis with custom dimensions

Roadmap Copy Chapter

**🔥 Upcoming Features** Copy Section

• Multi-Modal Evolution: Images, audio, and text simultaneously
• Federated Learning: Distributed evolution across multiple machines
• AutoML Integration: Hyperparameter and architecture evolution
• Benchmark Suite: Standardized evaluation across domains

**🌟 Research Directions** Copy Section

• Self-Modifying Prompts: Evolution modifies its own prompting strategy
• Cross-Language Evolution: Python → Rust → C++ optimization chains
• Neurosymbolic Reasoning: Combine neural and symbolic approaches
• Human-AI Collaboration: Interactive evolution with human feedback

Want to contribute? Check out our roadmap discussions!

FAQ Copy Chapter

💰 How much does it cost to run?

See the Cost Estimation section in Installation & Setup for detailed pricing information and cost-saving tips.

🆚 How does this compare to manual optimization?

OpenEvolve shines when you need to explore large solution spaces or optimize for multiple objectives simultaneously.

🔧 Can I use my own LLM?

Yes! OpenEvolve supports any OpenAI-compatible API:

• Commercial: OpenAI, Google, Cohere
• Local: Ollama, vLLM, LM Studio, text-generation-webui
• Advanced: OptiLLM for routing and test-time compute

Just set the api_base in your config to point to your endpoint.

🚨 What if evolution gets stuck?

Built-in mechanisms prevent stagnation:

• Island migration: Fresh genes from other populations
• Temperature control: Exploration vs exploitation balance
• Diversity maintenance: MAP-Elites prevents convergence
• Artifact feedback: Error messages guide improvements
• Template stochasticity: Randomized prompts break patterns

Manual interventions:

• Increase num_diverse_programs for more exploration
• Add custom feature dimensions to diversify search
• Use template variations to randomize prompts
• Adjust migration intervals for more cross-pollination

📈 How do I measure success?

Multiple success metrics:

1. Primary Metric: Your evaluator's combined_score or metric average
2. Convergence: Best score improvement over time
3. Diversity: MAP-Elites grid coverage
4. Efficiency: Iterations to reach target performance
5. Robustness: Performance across different test cases

Use the visualizer to track all metrics in real-time and identify when evolution has converged.

**Contributors** Copy Section

Thanks to all our amazing contributors who make OpenEvolve possible!

**Contributing** Copy Section

We welcome contributions! Here's how to get started:

1. 🍴 Fork the repository
2. 🌿 Create your feature branch: git checkout -b feat-amazing-feature
3. ✨ Add your changes and tests
4. ✅ Test everything: python -m unittest discover tests
5. 📝 Commit with a clear message
6. 🚀 Push and create a Pull Request

New to open source? Check out our Contributing Guide and look for good-first-issue labels!

**Academic & Research** Copy Section

Articles & Blog Posts About OpenEvolve:

• Towards Open Evolutionary Agents - Evolution of coding agents and the open-source movement
• OpenEvolve: GPU Kernel Discovery - Automated discovery of optimized GPU kernels
• OpenEvolve: Evolutionary Coding with LLMs - Introduction to evolutionary algorithm discovery using large language models

Citation Copy Chapter

If you use OpenEvolve in your research, please cite:

@software{openevolve,
  title = {OpenEvolve: an open-source evolutionary coding agent},
  author = {Asankhaya Sharma},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/algorithmicsuperintelligence/openevolve}
}

File: configs/README.md Copy Chapter

OpenEvolve Configuration Files

This directory contains configuration files for OpenEvolve with examples for different use cases.

Configuration Files Copy Chapter

`default_config.yaml` Copy Section

The main configuration file containing all available options with sensible defaults. This file includes:

• Complete documentation for all configuration parameters
• Default values for all settings
• Island-based evolution parameters for proper evolutionary diversity

Use this file as a template for your own configurations.

`island_config_example.yaml` Copy Section

A practical example configuration demonstrating proper island-based evolution setup. Shows:

• Recommended island settings for most use cases
• Balanced migration parameters
• Complete working configuration

`island_examples.yaml` Copy Section

Multiple example configurations for different scenarios:

• Maximum Diversity: Many islands, frequent migration
• Focused Exploration: Few islands, rare migration
• Balanced Approach: Default recommended settings
• Quick Exploration: Small-scale rapid testing
• Large-Scale Evolution: Complex optimization runs

Includes guidelines for choosing parameters based on your problem characteristics.

Island-Based Evolution Parameters Copy Chapter

The key new parameters for proper evolutionary diversity are:

database:
  num_islands: 5                      # Number of separate populations
  migration_interval: 50              # Migrate every N generations  
  migration_rate: 0.1                 # Fraction of top programs to migrate

Parameter Guidelines Copy Section

• num_islands: 3-10 for most problems (more = more diversity)
• migration_interval: 25-100 generations (higher = more independence)
• migration_rate: 0.05-0.2 (5%-20%, higher = faster knowledge sharing)

When to Use What Copy Section

• Complex problems → More islands, less frequent migration
• Simple problems → Fewer islands, more frequent migration
• Long runs → More islands to maintain diversity
• Short runs → Fewer islands for faster convergence

Usage Copy Chapter

Copy any of these files as a starting point for your configuration:

cp configs/default_config.yaml my_config.yaml
# Edit my_config.yaml for your specific needs

Then use with OpenEvolve:

from openevolve import OpenEvolve
evolve = OpenEvolve(
    initial_program_path="program.py",
    evaluation_file="evaluator.py", 
    config_path="my_config.yaml"
)

File: examples/r_robust_regression/README.md Copy Chapter

R Robust Regression Evolution

This example demonstrates how to use OpenEvolve with the R programming language. The example focuses on evolving robust regression algorithms that can handle outliers in data, showcasing OpenEvolve's ability to work with statistical computing languages beyond Python.

Files Copy Chapter

• initial_program.r: Starting R implementation with basic least squares regression
• evaluator.py: Python evaluator that runs R code and measures performance
• config.yaml: Configuration optimized for statistical algorithm evolution
• requirements.txt: Dependencies for both R and Python components

Prerequisites Copy Chapter

R Dependencies Copy Section

Install R (version 3.6 or higher) and the required packages:

install.packages(c("jsonlite"))

Python Dependencies Copy Section

pip install -r requirements.txt

Usage Copy Chapter

Run the evolution process:

cd examples/r_robust_regression
python ../../openevolve-run.py initial_program.r evaluator.py --config config.yaml --iterations 100

This example shows how OpenEvolve can evolve algorithms in R by starting with a basic least squares implementation and improving it to handle outliers through various robust regression techniques.

File: examples/online_judge_programming/README.md Copy Chapter

Online Judge Programming Example

This example demonstrates how OpenEvolve can solve programming problems and pass all test cases on Kattis online judge starting from scratch.

Problem Description Copy Chapter

We take the Alphabet problem from Kattis as following:

A string of lowercase letters is called **alphabetical** if some of the letters can be deleted so that the only letters that remain are the letters from 'a' to 'z' in order. Given a string s, determine the minimum number of letters to add anywhere in the string to make it alphabetical.

Input:
Each input will consist of a single test case. Note that your program may be run multiple times on different inputs. The only line of input contains a string s (1 ≤ |s| ≤ 50) which contains only lowercase letters.

Output:
Output a single integer, which is the smallest number of letters needed to add to `s` to make it alphabetical.

Sample Input 1:
xyzabcdefghijklmnopqrstuvw

Sample Output 1:
3

Sample Input 2:
aiemckgobjfndlhp

Sample Output 2:
20

Getting Started Copy Chapter

First, fill your username and token in example.kattisrc according to your personal configuration file (must be logged in) from Kattis and rename the file as .kittisrc.

Then, to run this example:

cd examples/online_judge_programming
python ../../openevolve-run.py initial_program.py evaluator.py --config config.yaml

Algorithm Evolution Copy Chapter

Initial Algorithm (dummy output) Copy Section

The initial implementation was a simple dummy output that returned 0 directly.

import sys
for line in sys.stdin:
    s = line.strip()

ans = 0
print(ans)

Evolved Algorithm (Dynamic Programming) Copy Section

After running OpenEvolve for just 4 iterations, it discovered a dynamic programming algorithm that passes all test cases on Kattis:

import sys

for line in sys.stdin:
    s = line.strip()

n = len(s)
dp = [1] * n

for i in range(1, n):
    for j in range(i):
        if s[i] > s[j]:
            dp[i] = max(dp[i], dp[j] + 1)

longest_alphabetical_subsequence_length = max(dp)
ans = 26 - longest_alphabetical_subsequence_length
print(ans)

Next Steps Copy Chapter

Try modifying the config.yaml file to:

• Change the programming problem in system prompt
• Change the LLM model configuration

File: examples/mlx_metal_kernel_opt/README.md Copy Chapter

MLX Metal Kernel Optimization Example

This example uses OpenEvolve to automatically discover optimized Metal GPU kernels for Grouped Query Attention (GQA) in Qwen3-0.6B on Apple Silicon.

Target Configuration Copy Chapter

• Model: Qwen3-0.6B-bf16
• Architecture: 16 query heads : 8 KV heads (2:1 ratio), 2048 hidden size, 128 head dimension
• Hardware: Apple M-series GPUs with unified memory
• Baseline: mx.fast.scaled_dot_product_attention via mlx_lm.generate
• Goal: Evolve custom Metal kernel source code to outperform baseline

Quick Start Copy Chapter

Prerequisites Copy Section

pip install mlx mlx-lm openevolve

# Set API key (Gemini via OpenAI-compatible endpoint)
export OPENAI_API_KEY="your-gemini-key"

Run Evolution Copy Section

cd openevolve/examples/mlx_metal_kernel_opt

# Using the experiment runner script
./run_evolve_experiment.sh --run-name test_run --iterations 25

# Or directly
python -m openevolve.cli \
    --initial-program initial_program.py \
    --evaluator evaluator.py \
    --config config.yaml \
    --iterations 25 \
    --output ./openevolve_output

Verify Evaluation Validity Copy Section

# Run a single benchmark with verbose output
python -c "
from evaluator import Qwen3GQAEvaluator
e = Qwen3GQAEvaluator()
result = e.evaluate('initial_program.py')
print(result['summary'])
"

Files Copy Chapter

Validity Fixes (This PR) Copy Chapter

This PR corrects critical issues that invalidated prior evaluation results:

1. Subprocess Kernel Hook: Evolved kernels are now properly applied in benchmark subprocesses via mlx_lm_generate_with_hook.py

2. bfloat16 Correctness Gate: Correctness tests now use mx.bfloat16 inputs to match actual inference dtype

3. Architecture Alignment: Fixed head ratio from 40:8 to correct 16:8 (2:1 GQA pattern)

4. Evaluation Flow Optimizations: Early exit on compilation errors, correctness-before-baseline ordering, GPU state cleanup between runs

Current Status Copy Chapter

After fixing validity issues, we ran 25 evolution iterations.

Result: The best evolved kernel is 3.2% SLOWER than MLX's baseline implementation.

The evolution improved from an initial -11.5% regression to -3.2%, but never exceeded baseline. This indicates fundamental limitations in the current evolution mechanism that require further investigation.

For detailed experiment results and analysis, see EVOLUTION_ANALYSIS.md.

Demo Results (Committed) Copy Section

For review and reproducibility, this example repo includes a committed snapshot of one post-fix evolution run:

• best_program.py: best evolved program (iteration 23)
• best_program_info.json: metrics + baseline comparisons (includes the -3.2% result)

The full run output directory is intentionally git-ignored (see .gitignore) to avoid committing large run artifacts.

Known Limitations Copy Section

1. MAP-Elites selection uses abstract combined_score instead of direct speedup ratios
2. LLM context underutilized (only 1 parent + 5 samples per iteration)
3. No GPU profiling data to guide optimization
4. 32% bf16 compilation failure rate

References Copy Chapter

• OpenEvolve
• MLX
• MLX-LM
• KernelBench

File: examples/lm_eval/README.md Copy Chapter

lm-eval.py

lm-eval.py provides basic benchmark capability for LLM feedback-based evolutionary task solving. The benchmark framework is EleutherAI's lm-evaluation-harness.

Limitation: Only generation-only tasks such as gsm8k are supported. This is because tasks that require loglikelihood probabilities are not well applicable to agents.

Usage Copy Chapter

$ python3 examples/lm_eval/lm-eval.py -h
usage: lm-eval.py [-h] [--config CONFIG] [--init_file INIT_FILE] [--evaluator_file EVALUATOR_FILE] [--iterations ITERATIONS] [--limit LIMIT] [--tasks TASKS]
                  [--output_path OUTPUT_PATH]

OpenEvolve <-> lm-evaluation-harness adapter.

options:
  -h, --help            show this help message and exit
  --config CONFIG       config file
  --init_file INIT_FILE
                        initial content file
  --evaluator_file EVALUATOR_FILE
                        evaluator file
  --iterations ITERATIONS
                        number of iterations
  --limit LIMIT         limit the number of examples per task that are executed
  --tasks TASKS         list of tasks to evaluate
  --output_path OUTPUT_PATH
                        output path for results

Early examples that were meant to indicate that more evolution iterations improve task performance -- I suspect the prompting may not be ideal yet:

$ python3 examples/lm_eval/lm-eval.py --tasks gsm8k --limit 10 --iterations 1
[..]
Headline metrics:
  gsm8k           exact_match,strict-match 80.000%
[..]


$ python3 examples/lm_eval/lm-eval.py --tasks gsm8k --limit 10 --iterations 3
[..]
Headline metrics:
  gsm8k           exact_match,strict-match 90.000%
[..]

$ python3 examples/lm_eval/lm-eval.py --tasks gsm8k --limit 10 --iterations 10
[..]
Headline metrics:
  gsm8k           exact_match,strict-match 80.000%
[..]

$ python3 examples/lm_eval/lm-eval.py --tasks gsm8k --limit 10 --iterations 15
[..]
Headline metrics:
  gsm8k           exact_match,strict-match 70.000%
[..]

Warning Copy Chapter

• Be aware that this is an early implementation. No extensive benchmarks have been executed so far. With a limit to 10 tasks and 10 iterations, the benchmark is meaningless as is.
• Use the --limit parameter only for tests, not for metric generation.
• Do not cite the metrics that result from the script execution blindly without reviewing the solution first.

References Copy Chapter

@misc{eval-harness,
    author       = {Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy},
    title        = {The Language Model Evaluation Harness},
    month        = 07,
    year         = 2024,
    publisher    = {Zenodo},
    version      = {v0.4.3},
    doi          = {10.5281/zenodo.12608602},
    url          = {https://zenodo.org/records/12608602}
}

File: examples/llm_prompt_optimization/README.md Copy Chapter

LLM Prompt Optimization with OpenEvolve 🚀

This example demonstrates how to use OpenEvolve to automatically optimize prompts for Large Language Models across various benchmark datasets. The system uses evolutionary search to discover high-performing prompts, achieving significant improvements across multiple tasks.

📊 Latest Performance Results (GEPA Benchmarks) Copy Chapter

OpenEvolve successfully improved prompt performance across three challenging GEPA benchmarks:

Key Achievements: Copy Section

• 767 more correct answers across all datasets
• 38% fewer empty responses with evolved prompts
• Near-perfect performance on instruction following (IFEval: 97.41%)
• Major improvement in multi-hop reasoning (HotpotQA: 88.62%)

🎯 Overview Copy Chapter

OpenEvolve automatically:

• Evolves prompts through multiple generations using LLMs
• Uses cascading evaluation for efficient testing
• Employs MAP-Elites algorithm to maintain diversity
• Incorporates LLM feedback for qualitative assessment
• Supports various datasets from HuggingFace

📊 All Supported Datasets Copy Chapter

GEPA Benchmarks (Latest Focus) Copy Section

IFEval (Instruction Following Eval)

• Task: Follow complex, multi-constraint instructions
• Size: 541 samples (train split)
• Metric: Binary success on instruction adherence
• Results: 95.01% → 97.41% (+2.40%)
• Config: ifeval_prompt_dataset.yaml

HoVer (Claim Verification)

• Task: Verify claims as SUPPORTED or NOT_SUPPORTED
• Size: 4,000 samples (validation split)
• Metric: Binary classification accuracy
• Results: 43.83% → 42.90% (-0.93%)
• Config: hover_prompt_dataset.yaml
• Note: Uses integer labels (0=SUPPORTED, 1=NOT_SUPPORTED)

HotpotQA (Multi-hop Question Answering)

• Task: Answer questions requiring reasoning over multiple paragraphs
• Size: 7,405 samples (validation split)
• Metric: Exact match with answer
• Results: 77.93% → 88.62% (+10.69%)
• Config: hotpotqa_prompt_dataset.yaml

Additional Datasets (Earlier Experiments) Copy Section

Emotion Classification

• Task: Classify emotions in text (6 classes)
• Dataset: dair-ai/emotion
• Config: emotion_prompt_dataset.yaml
• Benchmark: Compared against DSPy results

GSM8K (Grade School Math)

• Task: Solve grade school math word problems
• Dataset: gsm8k
• Config: gsm8k_prompt_dataset.yaml
• Benchmark: DSPy achieves 97.1%

IMDB Sentiment Analysis

• Task: Binary sentiment classification
• Dataset: stanfordnlp/imdb
• Config: initial_prompt_dataset.yaml
• Example Evolution: 72% → 94% accuracy

🚀 Quick Start Copy Chapter

1. Install Dependencies Copy Section

cd examples/llm_prompt_optimization
pip install -r requirements.txt

2. Set Your API Key Copy Section

export OPENAI_API_KEY="your_openrouter_api_key"

Note: Despite the variable name, this uses OpenRouter API. Get your key at https://openrouter.ai/

3. Evaluate Prompts Copy Section

Use the unified evaluation script to test baseline or evolved prompts:

# Evaluate baseline prompts on a single dataset
python evaluate_prompts.py --dataset ifeval --prompt-type baseline --samples 100

# Evaluate evolved prompts on a single dataset
python evaluate_prompts.py --dataset hover --prompt-type evolved --samples 100

# Evaluate all GEPA datasets with evolved prompts (full dataset)
python evaluate_prompts.py --dataset all --prompt-type evolved

# Specify output file
python evaluate_prompts.py --dataset all --prompt-type evolved --output results.json

4. Run Evolution Copy Section

To evolve prompts from scratch:

# For GEPA benchmarks
python ../../openevolve-run.py ifeval_prompt.txt evaluator.py \
  --config config_qwen3_evolution.yaml \
  --iterations 50

# For other datasets (using wrapper script)
./run_evolution.sh emotion_prompt.txt --iterations 50
./run_evolution.sh gsm8k_prompt.txt --iterations 100

⚙️ Configuration Files Copy Chapter

Evolution Configurations Copy Section

GEPA Benchmarks (`config_qwen3_evolution.yaml`)

llm:
  models:
    - name: "qwen/qwen3-8b"
      weight: 1.0
  temperature: 0.7
  max_tokens: 4096

evaluator:
  cascade_evaluation: true
  cascade_thresholds: [0.9]  # 2-stage evaluation
  timeout: 1800  # 30 minutes
  use_llm_feedback: true
  llm_feedback_weight: 0.3

database:
  n_islands: 4  # Island-based evolution
  migration_interval: 10

General Configuration (`config.yaml`)

llm:
  api_base: "https://openrouter.ai/api/v1"
  models:
    - name: "google/gemini-2.5-flash"
      weight: 1.0

Dataset Configurations Copy Section

Each dataset has its own configuration file following the pattern *_prompt_dataset.yaml:

# Example: ifeval_prompt_dataset.yaml
dataset_name: "google/IFEval"
input_field: "prompt"
target_field: "instruction_id_list"
split: "train"
is_ifeval: true  # Special handling flag

🧬 Evolution Process Copy Chapter

How It Works Copy Section

1. Initial Population: Start with baseline prompt
2. Variation: LLM generates prompt mutations
3. Evaluation: Test on dataset samples (10 for Stage 1, 40 for Stage 2)
4. Selection: Keep best performers based on combined score
5. Island Evolution: 4 isolated populations with periodic migration
6. Iteration: Repeat for specified generations (typically 50-100)

Cascade Evaluation Copy Section

• Stage 1: Quick test on 10 samples (must achieve 90% to proceed)
• Stage 2: Comprehensive test on 40 samples
• Combined Score: 70% task accuracy + 30% LLM feedback

LLM Feedback Metrics Copy Section

Evolved prompts are evaluated on:

• Clarity: Unambiguous instructions
• Specificity: Appropriate detail level
• Robustness: Edge case handling
• Format Specification: Clear output requirements

📁 Complete File Structure Copy Chapter

llm_prompt_optimization/
├── evaluate_prompts.py          # Unified evaluation script
├── evaluator.py                 # OpenEvolve evaluator
├── run_evolution.sh             # Wrapper script for evolution
│
├── Configuration Files
│   ├── config.yaml              # General LLM config
│   ├── config_qwen3_evolution.yaml  # GEPA evolution config
│   └── config_qwen3_baseline.yaml   # GEPA baseline config
│
├── Dataset Configurations & Prompts
│   ├── ifeval_prompt.txt & ifeval_prompt_dataset.yaml
│   ├── hover_prompt.txt & hover_prompt_dataset.yaml
│   ├── hotpotqa_prompt.txt & hotpotqa_prompt_dataset.yaml
│   ├── emotion_prompt.txt & emotion_prompt_dataset.yaml
│   ├── gsm8k_prompt.txt & gsm8k_prompt_dataset.yaml
│   └── initial_prompt.txt & initial_prompt_dataset.yaml
│
├── Evolution Templates
│   └── templates/
│       ├── full_rewrite_user.txt
│       ├── evaluation.txt
│       └── evaluator_system_message.txt
│
├── Results
│   ├── evaluation_results_baseline_20250809_070942.json
│   ├── evaluation_results_evolved_20250809_103002.json
│   └── openevolve_output_qwen3_*/
│       └── best/
│           └── best_program.txt     # Evolved prompt
│
└── requirements.txt

🔍 Example Evolved Prompts Copy Chapter

IFEval (97.41% accuracy) Copy Section

Follow the instruction below precisely. Structure your response into two 
distinct parts: 1) a step-by-step reasoning process that explicitly 
identifies the task, constraints, and required output format, and 2) the 
final answer in the exact format specified...

HotpotQA (88.62% accuracy) Copy Section

Answer the following question using the provided context. The answer must 
integrate information from multiple paragraphs and follow these steps:
1. Paragraph Analysis: Extract key details from each relevant paragraph...
2. Synthesis: Combine these details into a single, coherent response...
3. Citation: Attribute all assertions to their source paragraphs...

IMDB Sentiment (Example Evolution) Copy Section

Starting prompt:

Analyze the sentiment: "{input_text}"

Evolved prompt after 100 iterations:

Analyze the sentiment of the following text. Determine if the overall 
emotional tone is positive or negative.

Text: "{input_text}"

Response: Provide only a single digit - either 1 for positive sentiment 
or 0 for negative sentiment. Do not include any explanation or additional text.

Accuracy improvement: 72% → 94%

🐛 Troubleshooting Copy Chapter

HoVer Dataset Issues Copy Section

• Problem: Test split has no labels (all -1)
• Solution: Use validation split (configured automatically)
• Labels: Integer format (0=SUPPORTED, 1=NOT_SUPPORTED)

Empty Responses Copy Section

• Cause: Complex evolved prompts exceeding token limits
• Solution: Increase max_tokens in evaluation or simplify prompts

Slow Evaluation Copy Section

• IFEval: ~1 minute per 100 samples
• HoVer: ~30 minutes for full dataset
• HotpotQA: ~45 minutes for full dataset
• Tip: Use --samples flag for faster testing

Dataset Not Found Copy Section

• Check the exact dataset name and source
• Some datasets require acceptance of terms
• Use trust_remote_code=True for certain datasets

🚀 Advanced Usage Copy Chapter

Custom Datasets Copy Section

To add a new dataset:

1. Create initial prompt: mydataset_prompt.txt
2. Create configuration: mydataset_prompt_dataset.yaml
3. Run evolution:
   bash Copy

   ./run_evolution.sh mydataset_prompt.txt --iterations 50
   # or directly:
   python ../../openevolve-run.py mydataset_prompt.txt evaluator.py --config config.yaml

Batch Evaluation Copy Section

Evaluate multiple configurations:

# Create a script to run multiple evaluations
for dataset in ifeval hover hotpotqa; do
    python evaluate_prompts.py --dataset $dataset --prompt-type evolved
done

Resume Evolution Copy Section

Continue from a checkpoint:

python ../../openevolve-run.py prompt.txt evaluator.py \
  --config config_qwen3_evolution.yaml \
  --checkpoint openevolve_output_qwen3_ifeval/checkpoints/checkpoint_30 \
  --iterations 20

Custom Templates Copy Section

The templates/ directory contains customizable templates for prompt evolution:

• full_rewrite_user.txt: Instructions for prompt rewriting
• evaluation.txt: LLM feedback template
• evaluator_system_message.txt: System message for evaluation

📈 Tips for Best Results Copy Chapter

1. Start Simple: Begin with clear, working baseline prompts
2. Sufficient Samples: Use at least 40 samples for Stage 2 evaluation
3. Monitor Progress: Check openevolve_output_*/logs/ for progress
4. Multiple Runs: Evolution has randomness; try multiple runs
5. Token Limits: Ensure max_tokens accommodates prompt + response
6. Dataset Variety: Test on multiple datasets to ensure generalization

📚 References Copy Chapter

• OpenEvolve Documentation
• IFEval Paper
• HoVer Dataset
• HotpotQA Paper
• GSM8K Dataset
• DSPy Framework
• OpenRouter API

Happy prompt evolving! 🧬✨

File: examples/k_module_problem/README.md Copy Chapter

K-Module Problem: Evolution vs Iterative Refinement

This example demonstrates a fundamental limitation of iterative refinement approaches and shows how evolutionary search with population-based exploration can solve problems that defeat single-trajectory optimization.

The Problem Copy Chapter

The K-Module Problem is a pipeline configuration task where you must find the correct combination of 4 independent modules:

Search space: 5⁴ = 625 possible combinations

The Challenge: The evaluator only tells you how many modules are correct (0-4), not which ones. This creates a deceptive fitness landscape with no gradient information.

Why Iterative Refinement Fails Copy Chapter

Consider this scenario:

Initial:  [json_reader, standardize, mergesort, xml]     → Score: 0/4
Refine 1: [csv_reader, standardize, mergesort, xml]      → Score: 1/4 ✓
Refine 2: [csv_reader, normalize, mergesort, xml]        → Score: 2/4 ✓✓
Refine 3: [csv_reader, normalize, heapsort, xml]         → Score: 1/4 ✗ (went backwards!)
Refine 4: [csv_reader, normalize, mergesort, json]       → Score: 2/4 (no progress)

The Problem: When the model changes mergesort to heapsort, it has no way to know this was wrong because:

• The score decreased, but was that because of the algorithm change?
• Or because normalize wasn't actually correct?
• The model can't tell which modules are contributing to the score

This leads to random walk behavior requiring O(625) evaluations on average.

Why Evolution Succeeds Copy Chapter

Evolution maintains a population that explores different regions simultaneously:

Generation 1:
  Individual A: [csv_reader, scale, quicksort, xml]      → 2/4 (loader, algorithm correct)
  Individual B: [json_reader, normalize, bubble, json]   → 2/4 (preprocess, formatter correct)
  Individual C: [xml_reader, minmax, mergesort, csv]     → 0/4

Generation 2 (crossover):
  Child(A,B): [csv_reader, normalize, quicksort, json]   → 4/4 SUCCESS!

Key insight: Evolution discovers correct modules in different individuals and crossover combines them. This is the "Building Block Hypothesis" - complex solutions are assembled from simpler discovered components.

Running the Experiment Copy Chapter

Prerequisites Copy Section

1. OpenEvolve (this repo):

   bash Copy

   pip install -e .

2. API Key (both methods use the same model via OpenRouter for fair comparison):

   bash Copy

   export OPENROUTER_API_KEY=your_key

Both OpenEvolve and the iterative agent use google/gemini-2.5-flash-lite via OpenRouter API for a fair comparison.

Run OpenEvolve Copy Section

cd examples/k_module_problem
chmod +x run_openevolve.sh
./run_openevolve.sh 50

Or directly:

openevolve-run initial_program.py evaluator.py --config config.yaml --iterations 50

Run Iterative Agent Copy Section

python iterative_agent.py --iterations 100

Or run multiple trials to get statistics:

python run_iterative_trials.py --trials 3 --iterations 100

Run Random Baseline Copy Section

Establish the "no learning" baseline (no LLM needed):

python run_random_baseline.py --samples 100 --trials 100

This runs 100 independent trials of random search, each with up to 100 samples, and calculates empirical pass@k metrics.

Compare Results Copy Section

python compare_results.py

This generates:

• comparison_plot.png: Visual comparison of convergence
• Summary statistics printed to console

Experimental Results Copy Chapter

Random Baseline (100 trials, 100 samples each) Copy Section

This establishes the "no learning" baseline. Any method that beats this is demonstrating actual optimization, not just random sampling.

Iterative Refinement Results (3 trials, 100 iterations max) Copy Section

Summary:

• Success rate: 33% (1/3 trials found solution)
• When successful: 13 iterations
• Failure mode: Gets stuck at 75% - keeps trying standardize instead of normalize

Key observation: The iterative agent repeatedly finds configurations with 3/4 correct modules (csv_reader, quicksort, json) but cannot identify that preprocess is the wrong module. It keeps cycling through variations without escaping this local optimum.

OpenEvolve (Evolutionary) Results (3 trials, 100 iterations max) Copy Section

Summary:

• Success rate: 100% (3/3 trials found solution)
• Avg iterations to solution: 52.3
• Min iterations: 18
• Max iterations: 89

Key advantage: OpenEvolve's population-based approach maintains diverse configurations that explore different module combinations in parallel. Even when some individuals get stuck at local optima (75% with wrong preprocessing), others explore alternatives and eventually discover the correct solution.

Comparison Summary Copy Section

Key insight: While OpenEvolve takes more iterations on average (52.3 vs 13), it has a 100% success rate compared to iterative refinement's 33%. The evolutionary approach's population diversity ensures it eventually escapes local optima that trap single-trajectory methods.

Rich Feedback Mode: Proving Attribution Matters Copy Section

To verify that feedback attribution is the key factor, we added a RICH_FEEDBACK=1 mode that tells the agent exactly which modules are correct/incorrect:

RICH_FEEDBACK=1 python run_iterative_trials.py --trials 3 --iterations 100

With rich feedback, iterative refinement achieves 100% success rate in only 3 iterations - dramatically faster than OpenEvolve's 52 iterations! This proves that:

1. Feedback attribution is the key factor, not the optimization method
2. When feedback is attributable, iterative refinement is highly effective
3. Evolution is necessary when feedback is NOT attributable (you can't tell which component is wrong)

Why This Matters Copy Chapter

This example illustrates when you should prefer evolutionary approaches:

1. Combinatorial Configuration: When solutions are combinations of independent choices
2. Deceptive Fitness: When partial solutions don't clearly indicate which components are correct
3. No Gradient: When small changes don't reliably improve or degrade solutions
4. Building Block Problems: When good solutions are assembled from discovered components

Real-world examples:

• Hyperparameter tuning (learning rate + batch size + architecture)
• Feature selection (which features to include)
• API composition (which services to combine)
• Configuration optimization (compiler flags, system settings)

Files Copy Chapter

Configuration Details Copy Chapter

The OpenEvolve config uses settings optimized for this combinatorial problem:

# High temperature for diverse exploration
temperature: 0.9

# Very high exploration ratio
exploration_ratio: 0.6
exploitation_ratio: 0.25

# Multiple islands for parallel search with frequent migration
num_islands: 5
migration_interval: 3
migration_rate: 0.3

# Larger population for more diversity
population_size: 25

Key config improvements over default:

• Higher temperature (0.9 vs 0.7) - more exploration of different options
• Prompt emphasizes systematic exploration, not following hints
• More islands (5) with faster migration (interval=3) - combines building blocks faster
• Larger population (25) - maintains more diverse configurations

References Copy Chapter

• Building Block Hypothesis: Holland, J.H. (1975). Adaptation in Natural and Artificial Systems
• Schema Theorem: Explains how evolution propagates good partial solutions
• No Free Lunch: Wolpert & Macready (1997) - Evolution excels on problems with structure

Conclusion Copy Chapter

This example demonstrates that iterative refinement is not sufficient for problems with independent, combinatorial components. Evolutionary search with population-based exploration and crossover can solve these problems orders of magnitude faster by:

1. Exploring multiple regions of the search space simultaneously
2. Discovering correct "building blocks" in different individuals
3. Combining discoveries through crossover to assemble complete solutions

When your optimization problem has this structure, consider evolutionary approaches like OpenEvolve over single-trajectory iterative refinement.

File: examples/function_minimization/README.md Copy Chapter

Function Minimization Example

This example demonstrates how OpenEvolve can discover sophisticated optimization algorithms starting from a simple implementation.

Problem Description Copy Chapter

The task is to minimize a complex non-convex function with multiple local minima:

f(x, y) = sin(x) * cos(y) + sin(x*y) + (x^2 + y^2)/20

The global minimum is approximately at (-1.704, 0.678) with a value of -1.519.

Getting Started Copy Chapter

To run this example:

cd examples/function_minimization
python ../../openevolve-run.py initial_program.py evaluator.py --config config.yaml

Algorithm Evolution Copy Chapter

Initial Algorithm (Random Search) Copy Section

The initial implementation was a simple random search that had no memory between iterations:

def search_algorithm(iterations=1000, bounds=(-5, 5)):
    """
    A simple random search algorithm that often gets stuck in local minima.
    
    Args:
        iterations: Number of iterations to run
        bounds: Bounds for the search space (min, max)
        
    Returns:
        Tuple of (best_x, best_y, best_value)
    """
    # Initialize with a random point
    best_x = np.random.uniform(bounds[0], bounds[1])
    best_y = np.random.uniform(bounds[0], bounds[1])
    best_value = evaluate_function(best_x, best_y)
    
    for _ in range(iterations):
        # Simple random search
        x = np.random.uniform(bounds[0], bounds[1])
        y = np.random.uniform(bounds[0], bounds[1])
        value = evaluate_function(x, y)
        
        if value < best_value:
            best_value = value
            best_x, best_y = x, y
    
    return best_x, best_y, best_value

Evolved Algorithm (Simulated Annealing) Copy Section

After running OpenEvolve, it discovered a simulated annealing algorithm with a completely different approach:

/* Detailed source-code truncated for AI context efficiency. */

Key Improvements Copy Chapter

Through evolutionary iterations, OpenEvolve discovered several key algorithmic concepts:

1. Exploration via Temperature: Simulated annealing uses a temperature parameter to allow uphill moves early in the search, helping escape local minima that would trap simpler methods.

   python Copy

   probability = np.exp((current_value - new_value) / temperature)

2. Adaptive Step Size: The step size is adjusted dynamically—shrinking as the search converges and expanding if progress stalls—leading to better coverage and faster convergence.

   python Copy

   if i > iterations * 0.75:  # Reduce step size towards the end
       step_size *= 0.5
   if no_improvement_count > step_size_increase_threshold: # Increase step size if stuck
       step_size *= 1.1
       no_improvement_count = 0 # Reset the counter

3. Bounded Moves: The algorithm ensures all candidate solutions remain within the feasible domain, avoiding wasted evaluations.

   python Copy

   # Keep the new points within the bounds
   new_x = max(bounds[0], min(new_x, bounds[1]))
   new_y = max(bounds[0], min(new_y, bounds[1]))

4. Stagnation Handling: By counting iterations without improvement, the algorithm responds by boosting exploration when progress stalls.

   python Copy

   if no_improvement_count > step_size_increase_threshold: # Increase step size if stuck
       step_size *= 1.1
       no_improvement_count = 0 # Reset the counter

Results Copy Chapter

The evolved algorithm shows substantial improvement in finding better solutions:

The simulated annealing algorithm:

• Achieves higher quality solutions (closer to the global minimum)
• Has perfect reliability (100% success rate in completing runs)
• Maintains a good balance between performance and reliability

How It Works Copy Chapter

This example demonstrates key features of OpenEvolve:

• Code Evolution: Only the code inside the evolve blocks is modified
• Complete Algorithm Redesign: The system transformed a random search into a completely different algorithm
• Automatic Discovery: The system discovered simulated annealing without being explicitly programmed with knowledge of optimization algorithms
• Function Renaming: The system even recognized that the algorithm should have a more descriptive name

Next Steps Copy Chapter

Try modifying the config.yaml file to:

• Increase the number of iterations
• Change the LLM model configuration
• Adjust the evaluator settings to prioritize different metrics
• Try a different objective function by modifying evaluate_function()

File: examples/circle_packing_with_artifacts/README.md Copy Chapter

Circle Packing Example with Artifacts

This example demonstrates how OpenEvolve can be used to tackle the challenging mathematical problem of circle packing, a classic problem in computational geometry. Specifically, we focus on packing 26 circles of varying sizes into a unit square to maximize the sum of their radii, replicating one of the tasks from the AlphaEvolve paper.

Enhanced with Artifacts: This version showcases OpenEvolve's artifacts feature, which provides detailed execution feedback to help the LLM understand what went wrong and how to fix it.

Artifacts Enhancement Copy Chapter

This example has been enhanced to demonstrate OpenEvolve's artifacts side-channel, which captures detailed execution information beyond just numeric metrics. When a program fails or succeeds, the evaluator now provides rich context that gets included in the next generation's prompt.

What Artifacts Capture Copy Section

The enhanced evaluator captures different types of information:

🚨 **Failure Artifacts**

When compilation or runtime errors occur:

## Last Execution Output
### Stderr

Invalid shapes: centers=(25, 2), radii=(26,), expected (26, 2) and (26,)

### Failure_stage

shape_validation

### Suggestion

Check for syntax errors, import issues, or runtime exceptions

⚠️ **Validation Artifacts**

When geometric constraints are violated:

## Last Execution Output
### Boundary_violations

Circle 23 at (0.950000, 0.950000) with radius 0.055000 is outside unit square
Circle 24 at (0.980000, 0.980000) with radius 0.030000 is outside unit square

### Overlap_violations

Circles 5 and 12 overlap: dist=0.180000, r1+r2=0.190000

### Validation_report

Valid: False, Violations: 2 boundary, 1 overlaps

✅ **Success Artifacts**

For excellent solutions (>95% of target):

## Last Execution Output
### Stdout

Excellent packing! Achieved 99.7% of target value

### Radius_stats

Min: 0.045123, Max: 0.167500, Avg: 0.101319

### Packing_summary

Sum of radii: 2.634292/2.635 = 0.9997

⏱️ **Performance Artifacts**

Always included:

### Execution_time

12.45s

How Artifacts Help Evolution Copy Section

The artifacts provide crucial context that helps the LLM make better decisions:

1. Specific Error Fixing: Instead of just seeing validity: 0.0, the LLM sees exactly which circles are problematic and why
2. Performance Guidance: Execution time helps the LLM understand if algorithms are too slow
3. Success Recognition: When a solution works well, artifacts explain why it's good
4. Debugging Context: Full stack traces help fix syntax and runtime errors

Example Evolution with Artifacts Copy Section

Here's how artifacts help in a typical evolution scenario:

Generation N: Program fails with overlapping circles

# Faulty code
centers[5] = [0.3, 0.3]  
centers[6] = [0.3, 0.3]  # Same position!
radii[5] = radii[6] = 0.1

Artifacts captured:

Circles 5 and 6 overlap: dist=0.000000, r1+r2=0.200000

Generation N+1: LLM sees the overlap details and fixes it

# Fixed code  
centers[5] = [0.3, 0.3]
centers[6] = [0.5, 0.3]  # Different position!
radii[5] = radii[6] = 0.1

This leads to faster convergence because the LLM gets specific, actionable feedback instead of just numeric scores.

Backward Compatibility Copy Section

The artifacts enhancement is fully backward compatible:

• Existing evaluators continue to work unchanged
• Enhanced evaluators return EvaluationResult with both metrics and artifacts
• Disable artifacts by setting export ENABLE_ARTIFACTS=false if needed

To run with artifacts disabled:

export ENABLE_ARTIFACTS=false
python openevolve-run.py examples/circle_packing_with_artifacts/initial_program.py \
  examples/circle_packing_with_artifacts/evaluator.py \
  --config examples/circle_packing_with_artifacts/config_phase_1.yaml

Problem Overview Copy Chapter

The circle packing problem involves placing n non-overlapping circles inside a container (in this case, a unit square) to optimize a specific metric. For this example:

• We pack exactly 26 circles
• Each circle must lie entirely within the unit square
• No circles may overlap
• We aim to maximize the sum of all circle radii

According to the AlphaEvolve paper, a solution with a sum of radii of approximately 2.635 is achievable for n=26. Our goal was to match or exceed this result.

Our Approach Copy Chapter

We structured our evolution in two phases, each with a different configuration to encourage exploration and exploitation at different stages:

Phase 1: Initial Exploration Copy Section

In the first phase, we focused on exploring different fundamental approaches to the packing problem:

• Used a constructor-based approach that places circles in strategic positions
• Explored various geometric patterns (concentric rings, grid-based arrangements, etc.)
• Developed simple optimization routines to maximize circle sizes without overlaps

Configuration highlights:

max_iterations: 100
population_size: 60  
num_islands: 4
exploitation_ratio: 0.7

Phase 2: Breaking Through the Plateau Copy Section

After the initial exploration phase, we observed our solutions plateauing around 2.377. For the second phase, we reconfigured OpenEvolve to encourage more radical innovations:

• Increased the population size to promote diversity
• Lowered the exploitation ratio to favor exploration
• Updated the system prompt to suggest different optimization techniques
• Allowed for longer and more complex code solutions

Configuration highlights:

max_iterations: 100
population_size: 70  
num_islands: 5
exploitation_ratio: 0.6

Evolution Progress Copy Chapter

We tracked the evolution over 470 generations, capturing visualizations at each checkpoint. The progression shows dramatic improvements in the packing strategy:

Initial Solution (Generation 0) Copy Section

The initial program used a simple constructive approach with a central circle and two concentric rings:

# Initial attempt
# Place a large circle in the center
centers[0] = [0.5, 0.5]

# Place 8 circles around it in a ring
for i in range(8):
    angle = 2 * np.pi * i / 8
    centers[i + 1] = [0.5 + 0.3 * np.cos(angle), 0.5 + 0.3 * np.sin(angle)]

# Place 16 more circles in an outer ring
for i in range(16):
    angle = 2 * np.pi * i / 16
    centers[i + 9] = [0.5 + 0.7 * np.cos(angle), 0.5 + 0.7 * np.sin(angle)]

This approach yielded a sum of radii of approximately 0.959.

Generation 10 Breakthrough Copy Section

By generation 10, OpenEvolve had already discovered a more sophisticated approach:

# Generation 10
# Parameters for the arrangement (fine-tuned)
r_center = 0.1675  # Central circle radius

# 1. Place central circle
centers[0] = [0.5, 0.5]
radii[0] = r_center

# 2. First ring: 6 circles in hexagonal arrangement
r_ring1 = 0.1035
ring1_distance = r_center + r_ring1 + 0.0005  # Small gap for stability
for i in range(6):
    angle = 2 * np.pi * i / 6
    centers[i+1] = [
        0.5 + ring1_distance * np.cos(angle),
        0.5 + ring1_distance * np.sin(angle)
    ]
    radii[i+1] = r_ring1

The key innovations at this stage included:

• A carefully tuned hexagonal arrangement for the first ring
• Strategic placement of corner circles
• An additional optimization step to maximize each circle's radius

This approach achieved a sum of radii of approximately 1.795.

Generation 100: Grid-Based Approach Copy Section

By generation 100, OpenEvolve had pivoted to a grid-based approach with variable sized circles:

# Generation 100
# Row 1: 5 circles
centers[0] = [0.166, 0.166]
centers[1] = [0.333, 0.166]
centers[2] = [0.500, 0.166]
centers[3] = [0.667, 0.166]
centers[4] = [0.834, 0.166]

# Row 2: 6 circles (staggered)
centers[5] = [0.100, 0.333]
centers[6] = [0.266, 0.333]
# ... additional circles

Key innovations:

• Grid-like pattern with staggered rows
• Variable circle sizes based on position (larger in the center)
• More aggressive optimization routine with 50 iterations

This approach achieved a sum of radii of approximately 2.201.

Final Solution: Mathematical Optimization Copy Section

The breakthrough came when OpenEvolve discovered the power of mathematical optimization techniques. The final solution uses:

# Final solution with scipy.optimize
def construct_packing():
    # ... initialization code ...
    
    # Objective function: Negative sum of radii (to maximize)
    def objective(x):
        centers = x[:2*n].reshape(n, 2)
        radii = x[2*n:]
        return -np.sum(radii)

    # Constraint: No overlaps and circles stay within the unit square
    def constraint(x):
        centers = x[:2*n].reshape(n, 2)
        radii = x[2*n:]
        
        # Overlap constraint
        overlap_constraints = []
        for i in range(n):
            for j in range(i + 1, n):
                dist = np.sqrt(np.sum((centers[i] - centers[j])**2))
                overlap_constraints.append(dist - (radii[i] + radii[j]))
        # ... boundary constraints ...
        
    # Optimization using SLSQP
    result = minimize(objective, x0, method='SLSQP', bounds=bounds, constraints=constraints)

The key innovation in the final solution:

• Using scipy.optimize.minimize with SLSQP method to find the optimal configuration
• Formulating circle packing as a constrained optimization problem
• Representing both circle positions and radii as optimization variables
• Carefully crafted constraints to enforce non-overlap and boundary conditions

This approach achieved a sum of radii of 2.634, matching the AlphaEvolve paper's result of 2.635 to within 0.04%!

Results Copy Chapter

Our final solution achieves:

Sum of radii: 2.634292402141039
Target ratio: 0.9997314619131079 (99.97% of AlphaEvolve's result)

This demonstrates that OpenEvolve can successfully reproduce the results from the AlphaEvolve paper on this mathematical optimization problem.

Fast Convergence with Dual-Model Configuration Copy Chapter

Using a dual-model configuration with weighted sampling, OpenEvolve achieves near-optimal results in remarkably few iterations:

Configuration Copy Section

The config.yaml uses two Gemini models with different weights:

• google/gemini-2.5-flash-lite (weight: 0.8) - Fast, cost-effective for exploration
• google/gemini-2.5-flash (weight: 0.2) - Higher capability for breakthroughs

llm:
  models:
    - name: "google/gemini-2.5-flash-lite"
      weight: 0.8
    - name: "google/gemini-2.5-flash"
      weight: 0.2

Rapid Convergence Copy Section

The plot shows the evolution of sum_radii across program versions:

• Version 0: Starts at ~0.96 (basic initial program)
• Version 6: First major improvement to ~2.09
• Version 21: Reaches 2.63 (99.8% of target)
• Final: Achieves 2.6304 sum of radii

Key insight: OpenEvolve discovers the mathematical optimization approach (using scipy.optimize.minimize with SLSQP) by version 21, achieving 99.8% of the AlphaEvolve target in just ~40 program evaluations. The dual-model approach allows rapid exploration with the lighter model while leveraging the more capable model for breakthrough discoveries.

Why It Works Copy Section

1. Artifacts provide rich feedback: Failed programs return detailed error information (boundary violations, overlaps), helping the LLM quickly correct mistakes
2. MAP-Elites diversity: The feature dimensions (radius_variance, spatial_spread) maintain diverse solutions in the population
3. Island-based evolution: 4 islands evolve independently, preventing premature convergence
4. Efficient model weighting: 80% lightweight model for broad exploration, 20% capable model for sophisticated solutions

Key Observations Copy Chapter

The evolution process demonstrated several interesting patterns:

1. Algorithm Transition: OpenEvolve discovered increasingly sophisticated algorithms, from basic geometric constructions to advanced mathematical optimization techniques.

2. Exploration-Exploitation Balance: The two-phase approach was crucial - initial exploration of different patterns followed by exploitation and refinement of the most promising approaches.

3. Breakthrough Discoveries: The most significant improvements came from fundamental changes in approach (e.g., switching from manual construction to mathematical optimization), not just parameter tuning.

4. Code Complexity Evolution: As the solutions improved, the code grew in complexity, adopting more sophisticated mathematical techniques.

Running the Example Copy Chapter

To reproduce our results:

# Phase 1: Initial exploration
python openevolve-run.py examples/circle_packing/initial_program.py \
  examples/circle_packing/evaluator.py \
  --config examples/circle_packing/config_phase_1.yaml \
  --iterations 100

# Phase 2: Breaking through the plateau
python openevolve-run.py examples/circle_packing/openevolve_output/checkpoints/checkpoint_100/best_program.py \
  examples/circle_packing/evaluator.py \
  --config examples/circle_packing/config_phase_2.yaml \
  --iterations 100

To visualize the best solution:

from examples.circle_packing.openevolve_output.best.best_program import run_packing, visualize

centers, radii, sum_radii = run_packing()
print(f"Sum of radii: {sum_radii}")
visualize(centers, radii)

Conclusion Copy Chapter

This example demonstrates OpenEvolve's ability to discover sophisticated algorithms for mathematical optimization problems. By evolving from simple constructive approaches to advanced numerical optimization techniques, OpenEvolve was able to match the results reported in the AlphaEvolve paper.

The circle packing problem shows how OpenEvolve can discover not just improvements to existing algorithms, but entirely new algorithmic approaches, transitioning from manual geometric construction to principled mathematical optimization.