## File: README.md # Build A Reasoning Model (From Scratch) This repository contains the code for developing an LLM reasoning model and is the official code repository for the book [*Build a Reasoning Model (From Scratch)*](https://mng.bz/lZ5B). [](https://mng.bz/lZ5B) (Printed in color.) In [*Build a Reasoning Model (From Scratch)*](https://mng.bz/lZ5B), you will learn and understand how a reasoning large language model (LLM) works. Reasoning is one of the most exciting and important recent advances in improving LLMs, but it’s also one of the easiest to misunderstand if you only hear the term reasoning and read about it in theory. This is why this book takes a hands-on approach. We will start with a pre-trained base LLM and then add reasoning capabilities ourselves, step by step in code, so you can see exactly how it works. The methods described in this book walk you through the process of developing your own small-but-functional reasoning model for educational purposes. It mirrors the approaches used in creating large-scale reasoning models such as DeepSeek R1, GPT-5 Thinking, and others. In addition, this book includes code for loading the weights of existing, pretrained models. - Link to the official [source code repository](https://github.com/rasbt/reasoning-from-scratch) - Link to the [book at Manning](https://mng.bz/lZ5B) (the publisher's website) - Link to the book page on Amazon.com (TBD) - ISBN 9781633434677 To download a copy of this repository, click on the [Download ZIP](https://github.com/rasbt/reasoning-from-scratch/archive/refs/heads/main.zip) button or execute the following command in your terminal: ```bash git clone --depth 1 https://github.com/rasbt/reasoning-from-scratch.git ``` > **Tip:** > Chapter 2 provides additional tips on installing Python, managing Python packages, and setting up your coding environment. ## Table of Contents (In Progress) [](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-linux.yml) [](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-macos.yml) [](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-windows.yml) - [Troubleshooting Guide](./troubleshooting.md) | Chapter Title | Main Code | | ----------------------------------------------------------- | ------------------------------------------------------------ | | Ch 1: Understanding reasoning Models | No code | | Ch 2: Generating Text with a Pre-trained LLM | - [ch02_main.ipynb](ch02/01_main-chapter-code/ch02_main.ipynb)- [ch02_exercise-solutions.ipynb](ch02/01_main-chapter-code/ch02_exercise-solutions.ipynb) | | Ch 3: Evaluating Reasoning Models | - [ch03_main.ipynb](ch03/01_main-chapter-code/ch03_main.ipynb)- [ch03_exercise-solutions.ipynb](ch03/01_main-chapter-code/ch03_exercise-solutions.ipynb) | | Ch 4: Improving Reasoning with Inference-Time Scaling | - [ch04_main.ipynb](ch04/01_main-chapter-code/ch04_main.ipynb)- [ch04_exercise-solutions.ipynb](ch04/01_main-chapter-code/ch04_exercise-solutions.ipynb) | | Ch 5: Inference-Time Scaling via Self-Refinement | - [ch05_main.ipynb](ch05/01_main-chapter-code/ch05_main.ipynb)- [ch05_exercise-solutions.ipynb](ch05/01_main-chapter-code/ch05_exercise-solutions.ipynb) | | Ch 6: Training Reasoning Models with Reinforcement Learning | - [ch06_main.ipynb](ch06/01_main-chapter-code/ch06_main.ipynb)- [ch06_exercise-solutions.ipynb](ch06/01_main-chapter-code/ch06_exercise-solutions.ipynb) | | Ch 7: Improving GRPO for Reinforcement Learning | - [ch07_main.ipynb](ch07/01_main-chapter-code/ch07_main.ipynb)- [ch07_exercise-solutions.ipynb](ch07/01_main-chapter-code/ch07_exercise-solutions.ipynb) | | Ch 8: Distilling Reasoning Models for Efficient Reasoning | - [ch08_main.ipynb](ch08/01_main-chapter-code/ch08_main.ipynb)- [ch08_exercise-solutions.ipynb](ch08/01_main-chapter-code/ch08_exercise-solutions.ipynb) | | Appendix A: References and Further Reading | No code | | Appendix B: Exercise Solutions | Code and solutions are in each chapter's subfolder | | Appendix C: Qwen3 LLM Source Code | - [chC_main.ipynb](chC/01_main-chapter-code/chC_main.ipynb) | | Appendix D: Using larger LLMs | - [chD_main.ipynb](chD/chD_main.ipynb) | | Appendix E: Batching and throughput-oriented execution | - [chE_main.ipynb](chE/chE_main.ipynb) | | Appendix F: Common Approaches to LLM Evaluation | - [chF_main.ipynb](chF/01_main-chapter-code/chF_main.ipynb) | | Appendix G: Building a Chat Interface | - [chG](chG) |   The mental model below summarizes the main techniques covered in this book.   ## Companion Book Please note that *Build A Reasoning Model (From Scratch)* is a standalone book focused on methods to improve LLM reasoning. In this book, we work with a pre-trained open-source base LLM (Qwen3) on top of which we code apply reasoning methods from scratch. This includes inference-time scaling, reinforcement learning, and distillation. However, if you are interested in understanding how a conventional base LLM is implemented, you may like my previous book, [*Build a Large Language Model (From Scratch)*](https://amzn.to/4fqvn0D). [](https://amzn.to/4fqvn0D) - [Amazon link](https://amzn.to/4fqvn0D) - [Manning link](http://mng.bz/orYv) - [GitHub repository](https://github.com/rasbt/LLMs-from-scratch)   ## Hardware Requirements The code in the main chapters of this book is designed to mostly run on consumer hardware within a reasonable timeframe and does not require specialized server hardware. This approach ensures that a wide audience can engage with the material. Additionally, the code automatically utilizes GPUs if they are available. That being said, chapters 2-4 will work well on CPUs and GPUs. For chapters 5 and 6, it is recommended to use a GPU if you want to replicate the results in the chapter. (Please see the [setup_tips](ch02/02_setup-tips/python-instructions.md) doc for additional recommendations.)   ## Exercises Each chapter of the book includes several exercises. The solutions are summarized in Appendix B, and the corresponding code notebooks are available in the main chapter folders of this repository (for example, [`ch02/01_main-chapter-code/ch02_exercise-solutions.ipynb`](ch02/01_main-chapter-code/ch02_exercise-solutions.ipynb)).   ## Bonus Material Several folders contain optional materials as a bonus for interested readers: - **Chapter 2: Generating Text with a Pre-trained LLM** - [Optional Python Setup and Cloud GPU Recommendations](ch02/02_setup-tips) - [Using a GPU-optimized version of the LLM](ch02/03_optimized-LLM) - [Using `torch.compile()` on Windows](ch02/04_torch-compile-windows) - [Run inference and chat with the model](ch02/05_use_model) - **Chapter 3: Evaluating LLMs** - [MATH-500 Verifier Scripts](ch03/02_math500-verifier-scripts) - [Advanced Parser](ch03/03_advanced-parser) (hybrid LaTeX parser) - **Chapter 4: Improving Reasoning with Inference-Time Scaling** - [Inference Scaling on MATH-500](ch04/02_math500-inference-scaling-scripts) (CoT prompting, self-consistency) - **Chapter 5: Inference-Time Scaling Via Self-Refinement** - [More Inference Scaling on MATH-500](ch05/02_math500-more-inference-scaling-scripts) (Best-of-N, self-refinement) - **Chapter 6: Training Reasoning Models with Reinforcement Learning** - [GRPO scripts](ch06/02_rlvr_grpo_scripts_intro) with a batched mode - **Chapter 7: Improving GRPO for Reinforcement Learning** - [Advanced GRPO scripts](ch07/03_rlvr_grpo_scripts_advanced) (including DeepSeek-V3.2-, Olmo3-, and GDPO-style training) - [Download training checkpoints](ch07/04_download_trainining_checkpoints) (how to download and use the chapter 6 and 7 GRPO checkpoints) - **Chapter 8: Distilling Reasoning Models for Efficient Reasoning** - [Generate distillation data](ch08/02_generate_distillation_data) (teacher-output generation via Ollama or OpenRouter) - [Train with distillation](ch08/04_train_with_distillation) (including single-example and batched distillation scripts) - [Download training checkpoints](ch08/05_download_training_checkpoints) (how to download and use the chapter 8 distillation checkpoints) - [Use Qwen3 with Hugging Face](ch08/06_use_via_huggingface) (how to use the base model and chapter 6-8 checkpoints with `transformers`) - **Appendix F: Common Approaches to LLM Evaluation** - [MMLU Evaluation Methods](chF/02_mmlu) - [LLM leaderboards](chF/03_leaderboards) - [LLM-as-a-judge](chF/04_llm-judge) - **Appendix G: Building a Chat Interface** - [Chat interface code](chG/01_main-chapter-code)   ## Questions, Feedback, and Contributing to This Repository For common problems, please see the [Troubleshooting Guide](./troubleshooting.md). I welcome all sorts of feedback, best shared via the [Manning Discussion Forum](https://livebook.manning.com/forum?product=raschka2&page=1) or [GitHub Discussions](https://github.com/rasbt/reasoning-from-scratch/discussions). Likewise, if you have any questions or just want to bounce ideas off others, please don't hesitate to post these in the forum as well. Please note that since this repository contains the code corresponding to a print book, I currently cannot accept contributions that would extend the contents of the main chapter code, as it would introduce deviations from the physical book. Keeping it consistent helps ensure a smooth experience for everyone.   ## Citation If you find this book or code useful for your research, please consider citing it. Chicago-style citation: > Raschka, Sebastian. *Build A Reasoning Model (From Scratch)*. Manning, 2025. ISBN: 9781633434677. BibTeX entry: ``` @book{build-llms-from-scratch-book, author = {Sebastian Raschka}, title = {Build A Reasoning Model (From Scratch)}, publisher = {Manning}, year = {2025}, isbn = {9781633434677}, url = {https://mng.bz/lZ5B}, github = {https://github.com/rasbt/reasoning-from-scratch} } ``` --- ## File: ch01/README.md # Chapter 1: Understanding Reasoning Models   ## Main chapter code There is no code in this chapter. --- ## File: ch02/05_use_model/README.md # Run Inference and Chat With the Model     This folder contains standalone example scripts to generate text with the model we loaded in chapter 2 (and exercises): - `generate_simple.py`: Generates text similar to the main chapter. - `chat.py`: Similar to the code above, as an interactive wrapper so that we can prompt the model multiple times without having to reload the model into memory each time. - `chat_multiturn.py`: Same as above, but with a memory feature to remember the message history. More usage details are provided in the sections below.   ## generate_simple.py This simple function loads the model as described in chapter 2 and uses the `generate_text_simple_cache_stream` function from the chapter 2 exercises. You can use the function as follows (replace `uv run` with `python` if you are not using `uv`): ```bash uv run ch02/05_use_model/generate_simple.py Using Apple Silicon GPU (MPS) ✓ qwen3/qwen3-0.6B-base.pth already up-to-date ============================================================ torch : 2.7.1 device : mps cache : True compile : False reasoning : False ============================================================ Large language models are artificial intelligence systems that can understand, generate, and process human language, enabling them to perform a wide range of tasks, from answering questions to writing essays. Time: 1.52 sec 22 tokens/sec ``` The function is useful if you want to quickly try out different prompts with the base or reasoning variant. The additional options are listed below: ```bash usage: generate_simple.py [-h] [--device DEVICE] [--max_new_tokens MAX_NEW_TOKENS] [--compile] [--reasoning] [--prompt PROMPT] Run Qwen3 text generation options: -h, --help show this help message and exit --device DEVICE Device to run on (e.g. 'cpu', 'cuda', 'mps'). If not provided, will auto-detect with get_device(). --max_new_tokens MAX_NEW_TOKENS Maximum number of new tokens to generate (default: 2048). --compile Compile PyTorch model (default: False). --reasoning Use reasoning model variant (default: False). --prompt PROMPT Use a custom prompt. If not explicitly provided, uses the following defaults: 'Explain large language models in a single sentence.' for the base model, and 'Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.' for the reasoning model. ```   ## chat.py Similar to the function above, this function is useful to try different prompts on the base and reasoning models. However, in contrast to the previous function, this function keeps the user in an interactive mode so that the model doesn't have to be reloaded each time: ```bash uv run ch02/05_use_model/chat.py Using Apple Silicon GPU (MPS) ✓ qwen3/qwen3-0.6B-base.pth already up-to-date ============================================================ torch : 2.7.1 device : mps cache : True compile : False reasoning : False memory : False ============================================================ Interactive REPL (no memory). Type '\exit' or '\quit' to quit. >> Explain language models in 1 sentence ------------------------------------------------------------ [User] Explain language models in 1 sentence [Model] Language models are algorithms that analyze and predict the likelihood of future words in a text based on the words already seen, enabling them to generate coherent and contextually relevant text. [Stats] Time: 1.53 sec 22 tokens/sec ------------------------------------------------------------ >> Explain machine learning in 1 sentence. ------------------------------------------------------------ [User] Explain machine learning in 1 sentence. [Model] Machine learning is a subset of artificial intelligence that enables computers to learn from data and improve their performance over time without being explicitly programmed. [Stats] Time: 1.04 sec 24 tokens/sec ------------------------------------------------------------ ``` Additional options are listed below: ```bash usage: chat.py [-h] [--device DEVICE] [--max_new_tokens MAX_NEW_TOKENS] [--compile] [--reasoning] Run Qwen3 text generation (interactive REPL) options: -h, --help show this help message and exit --device DEVICE Device to run on (e.g. 'cpu', 'cuda', 'mps'). If not provided, will auto-detect with get_device(). --max_new_tokens MAX_NEW_TOKENS Maximum number of new tokens to generate (default: 2048). --compile Compile PyTorch model (default: False). --reasoning Use reasoning model variant (default: False). ```   ## chat_multiturn.py This function is similar to the one above, except it adds a multi-turn memory so that the LLM remembers the conversation from the past turns. It is highly recommended to use the reasoning variant here as the base model struggles with conversations: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Additional options are listed below: ```bash usage: chat_multiturn.py [-h] [--device DEVICE] [--max_new_tokens MAX_NEW_TOKENS] [--compile] [--reasoning] Run Qwen3 text generation (interactive REPL) options: -h, --help show this help message and exit --device DEVICE Device to run on (e.g. 'cpu', 'cuda', 'mps'). If not provided, will auto-detect with get_device(). --max_new_tokens MAX_NEW_TOKENS Maximum number of new tokens to generate in each turn (default: 2048). --compile Compile PyTorch model (default: False). --reasoning Use reasoning model variant (default: False). ``` --- ## File: ch02/04_torch-compile-windows/README.md # Using `torch.compile()` on Windows `torch.compile()` relies on *TorchInductor*, which JIT-compiles kernels and requires a working C/C++ compiler toolchain. So, on Windows, the setup required to make `torch.compile` work can be a bit more involved than on Linux or macOS, which usually don't require any extra steps besides installing PyTorch. If you are a Windows user and using `torch.compile` sounds too tricky or complicated, don't worry, all code examples in this repository will work fine without compilation. Below are some tips that I compiled based on recommendations by [Daniel Kleine](https://github.com/d-kleine) and the following [PyTorch guide](https://docs.pytorch.org/tutorials/unstable/inductor_windows.html).   ## 1 Basic Setup (CPU or CUDA)   ### 1.1 Install Visual Studio 2022 - Select the **“Desktop development with C++”** workload. - Make sure to include the **English language pack** (without it, you may run into UTF-8 encoding errors.)   ### 1.2 Open the correct command prompt Launch Python from the **"x64 Native Tools Command Prompt for VS 2022"** or from the **"Visual Studio 2022 Developer Command Prompt"**. Alternatively, you can initialize the environment manually by running: ```bash "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" ```   ### 1.3 Verify that the compiler works Run ```bash cl.exe ``` If you see version information printed, the compiler is ready.   ## 2 Troubleshooting Common Errors   ### 2.1 Error: `cl not found` Install **Visual Studio Build Tools** with the "C++ build tools" workload and run Python from a developer command prompt. (See this Microsoft [guide](https://learn.microsoft.com/en-us/cpp/build/vscpp-step-0-installation?view=msvc-170) for details)   ### 2.2 Error: `triton not found` (when using CUDA) Install the Windows build of Triton manually: ```bash pip install "triton-windows<3.4" ``` or, if you are using `uv`: ```bash uv pip install "triton-windows<3.4" ``` (As mentioned earlier, triton is required by TorchInductor for CUDA kernel compilation.)   ## 3 Additional Notes On Windows, the `cl.exe` compiler is only accessible from within the Visual Studio Developer environment. This means that using `torch.compile()` in notebooks such as Jupyter may not work unless the notebook was launched from a Developer Command Prompt. As mentioned at the beginning of this article, there is also a [PyTorch guide](https://docs.pytorch.org/tutorials/unstable/inductor_windows.html) that some users found helpful when getting `torch.compile()` running on Windows CPU builds. However, note that it refers to PyTorch's unstable branch, so use it as a reference only. **If compilation continues to cause issues, please feel free to skip it. It's a nice bonus, but it's not important to follow the book.** --- ## File: ch02/03_optimized-LLM/README.md # Optimized Qwen3 The Qwen3 from-scratch implementation used in this book strikes a balance between being efficient (both on CPU and GPU) and lean while remaining easy to read by a human. As an alternative, you can use the optional `Qwen3Model` drop-in replacement, which is slightly more GPU-efficient. The optimized version in [`qwen3_optimized.py`](../../reasoning_from_scratch/qwen3_optimized.py) (discussed further in Appendix C) differs from the baseline implementation in [`qwen3.py`](../../reasoning_from_scratch/qwen3.py) in two key ways: - It implements attention using PyTorch’s built-in `torch.nn.functional.scaled_dot_product` instead of a custom implementation. - It introduces a modified `KVCache` that pre-allocates key/value tensors. This increases memory usage but avoids repeatedly allocating new storage during execution. To explore the differences, I recommend opening [`qwen3.py`](../../reasoning_from_scratch/qwen3.py) and [`qwen3_optimized.py`](../../reasoning_from_scratch/qwen3_optimized.py) side by side and/or looking at a file-diff:   ## How to use The optimized code can be used as drop-in replacement for the code used in the main chapters as shown below. **Before:** ```python from reasoning_from_scratch.qwen3 import Qwen3Model from reasoning_from_scratch.ch02 import generate_text_basic_stream_cache ``` **After:** ```python from reasoning_from_scratch.qwen3_optimized import Qwen3Model from reasoning_from_scratch.ch02 import generate_text_basic_stream_cache ```   ## How to run comparisons To evaluate the performance on your system, you can use the [`compare_inference.py`](compare_inference.py) function contained in this folder: ```python python compare_inference.py ``` or ```python uv run compare_inference.py ``` Then, add the following flags: - `--device`: Select the device, e.g., `cpu`, `mps`, or `cuda` - `--cache`: Enables the KV cache - `--compile`: Uses `torch.compile` - `--reasoning`: Uses the Qwen3 reasoning variant instead of the base model. The base model generates approximately 50 tokens in response to the given prompt. The reasoning variant generates about 2000 tokens. - `--optimize`: Uses the optimized model from `qwen3_optimized.py` instead of the standard model from `qwen3.py`.   ### Standard model | Model | Mode | Command | Hardware | Tokens/sec | GPU Memory (VRAM) | | -------- | ----------------- | ------------------------------- | --------------- | ------------- | ----------------- | | qwen3.py | Regular | --device cpu | Mac Mini M4 CPU | 6 | - | | qwen3.py | Regular compiled | --device cpu --compile | Mac Mini M4 CPU | 6 | - | | qwen3.py | KV cache | --device cpu --cache | Mac Mini M4 CPU | 28 | - | | qwen3.py | KV cache compiled | --device cpu --compile --cache | Mac Mini M4 CPU | 68 | - | | | | | | | | | qwen3.py | Regular | --device mps | Mac Mini M4 GPU | 17 | - | | qwen3.py | Regular compiled | --device mps --compile | Mac Mini M4 GPU | InductorError | - | | qwen3.py | KV cache | --device mps --cache | Mac Mini M4 GPU | 18 | - | | qwen3.py | KV cache compiled | --device mps --compile --cache | Mac Mini M4 GPU | InductorError | - | | | | | | | | | qwen3.py | Regular | --device cuda | NVIDIA H100 GPU | 51 | 1.55 GB | | qwen3.py | Regular compiled | --device cuda --compile | NVIDIA H100 GPU | 164 | 1.81 GB | | qwen3.py | KV cache | --device cuda --cache | NVIDIA H100 GPU | 48 | 1.52 GB | | qwen3.py | KV cache compiled | --device cuda --compile --cache | NVIDIA H100 GPU | 141 | 1.81 GB |   ### Optimized model | Model | Mode | Command | Hardware | Tokens/sec | GPU Memory (VRAM) | | ------------------ | ----------------- | ------------------------------------------- | --------------- | ---------- | ----------------- | | qwen3_optimized.py | Regular | --optimized --device cpu | Mac Mini M4 CPU | 5 | - | | qwen3_optimized.py | Regular compiled | --optimized --device cpu --compile | Mac Mini M4 CPU | 7 | - | | qwen3_optimized.py | KV cache | --optimized --device cpu --cache | Mac Mini M4 CPU | 49 | - | | qwen3_optimized.py | KV cache compiled | --optimized --device cpu --compile --cache | Mac Mini M4 CPU | 51 | - | | | | | | | | | qwen3_optimized.py | Regular | --optimized --device mps | Mac Mini M4 GPU | 21 | - | | qwen3_optimized.py | Regular compiled | --optimized --device mps --compile | Mac Mini M4 GPU | NameError | - | | qwen3_optimized.py | KV cache | --optimized --device mps --cache | Mac Mini M4 GPU | 29 | - | | qwen3_optimized.py | KV cache compiled | --optimized --device mps --compile --cache | Mac Mini M4 GPU | 38 | - | | | | | | | | | qwen3_optimized.py | Regular | --optimized --device cuda | NVIDIA H100 GPU | 55 | 1.50 GB | | qwen3_optimized.py | Regular compiled | --optimized --device cuda --compile | NVIDIA H100 GPU | 173 | 1.81 GB | | qwen3_optimized.py | KV cache | --optimized --device cuda --cache | NVIDIA H100 GPU | 56 | 5.85 GB | | qwen3_optimized.py | KV cache compiled | --optimized --device cuda --compile --cache | NVIDIA H100 GPU | 177 | 5.85 GB | Comparing the 2 tables above, we can see that the optimized variant is clearly faster in terms of tokens/second in most cases. However, note that the unoptimized version is faster (68 tok/sec) than the optimized version (51 tok/sec) when using the compiled version with KV cache. The optimized version also uses more base RAM (5.85 GB with KV Cache) than the unoptimized version (1.5 GB). This is because it pre-allocates the tensors holding the KV values for the maximum supported context length. (So, when running the unoptimized version on a prompt with 41k context length, the RAM usage would be approximately similar.) **Perhaps the best recommendation is to use the unoptimized version (with `--cache` and `--compile`) when using a CPU. When using a GPU, use the optimized version (with `--cache` and `--compile`).** --- ## File: ch02/02_setup-tips/README.md # Chapter 2: Generating Text with a Pre-Trained LLM   ## Bonus material - [python-instructions.md](python-instructions.md): optional Python setup recommendations and instructions - [gpu-instructions.md](gpu-instructions.md): recommendations for cloud compute resources --- ## File: ch02/01_main-chapter-code/README.md # Chapter 2: Generating Text with a Pre-Trained LLM   ## Main chapter code - [ch02_main.ipynb](ch02_main.ipynb): main chapter code - [ch02_exercise-solutions.ipynb](ch02_exercise-solutions.ipynb): exercise solutions --- ## File: ch02/README.md # Chapter 2: Generating Text with a Pre-Trained LLM   ## Main chapter code - [01_main-chapter-code](01_main-chapter-code): main chapter code and exercise solutions   ## Bonus material - [02_setup-tips](02_setup-tips/): optional Python setup recommendations and cloud GPU recommendations - [03_optimized-LLM](03_optimized-LLM): info on how to use a GPU-optimized version of the LLM --- ## File: ch03/03_advanced-parser/README.md # Chapter 3: Advanced Parser (Bonus Material) This folder contains the parser experiment from [issue #133](https://github.com/rasbt/reasoning-from-scratch/issues/133), where a hybrid LaTeX parser was proposed to handle edge cases that the current chapter parser may miss.   ## Files - [compare_with_current_parser.ipynb](compare_with_current_parser.ipynb): notebook with usage examples - [math500_gpt_answers.json](math500_gpt_answers.json): MATH-500 examples with LLM answers, used for a section in the notebook above - [gen_llm_answers.py](gen_llm_answers.py): Convenience script to get boxed answers from the Qwen3 model in json format - [evaluate_math500_advanced.py](evaluate_math500_advanced.py): Same as the chapter 3 LLM evaluation script [evaluate_math500.py](../02_math500-verifier-scripts/evaluate_math500.py) but supports `--hybrid_parser` as an additional argument to use the alternative hybrid parser, for example, ```python uv run evaluate_math500_advanced.py --dataset_size 500 --hybrid_parser ```   ## How This Differs From The Chapter 3 Parser evaluate_math500_advanced.py The chapter parser in [reasoning_from_scratch/ch03.py](../../reasoning_from_scratch/ch03.py) is designed to stay compact and teachable: - It focuses on lightweight normalization plus symbolic equivalence checks - It mainly treats answers as arithmetic/symbolic expressions The hybrid parser in this folder (`latex_normalizer_hybrid.py`) is pattern-first and broader: - It recognizes answer formats before fallback parsing. - It adds support for intervals, unions, equations, matrices, set notation, membership (`\\in`), and `\\pm` - It preserves important edge cases better, such as base-subscript answers (`52_8`) and text casing (`\\text{Evelyn}`) Examples where behavior differs: - `52_8` -> chapter path often resolves to `528`; hybrid keeps `52_8` - `11,\\! 111,\\! 111,\\! 100` -> chapter path can become a tuple; hybrid normalizes to `11111111100` - `(0,9) \\cup (9,36)` -> chapter path usually remains text; hybrid returns a symbolic union Tradeoffs: - Chapter parser: simpler, faster, and easier to interpret - Hybrid parser: better coverage on LaTeX edge cases, but more rules and complexity; also adds SymPy LaTeX backend dependencies   ## Usage You can import the hybrid parser directly from the package: ```python from reasoning_from_scratch.bonus.parser import normalize_text_hybrid, sympy_parser_hybrid ``` See [compare_with_current_parser.ipynb](compare_with_current_parser.ipynb) for more detailed usage examples. --- ## File: ch03/02_math500-verifier-scripts/README.md # Chapter 3: Evaluating Reasoning Models     ## Bonus materials - [evaluate_math500.py](evaluate_math500.py): standalone script to evaluate models on the MATH-500 dataset - [evaluate_math500_batched.py](evaluate_math500_batched.py): same as above, but processes multiple examples in parallel during generation (for higher throughput) - [evaluate_json.py](evaluate_json.py): evaluate saved records JSON/JSONL files and report accuracy Both evaluation scripts import functionality from the [`reasoning_from_scratch`](../../reasoning_from_scratch) package to avoid code duplication. (See [chapter 2 setup instructions](../../ch02/02_setup-tips/python-instructions.md) for installation details.) --- **Note**: If you are not a `uv` user, replace `uv run ...py` with `python ...py` in the examples below. ---   ## `evaluate_math500.py` usage Run with: ```bash python evaluate_math500.py ``` Or, with `uv:` ```bash uv run evaluate_math500.py ``` Options: ```bash uv run evaluate_math500.py --help options: -h, --help show this help message and exit --device DEVICE Device to use: "auto" (default) or any torch device string (e.g., "cpu", "cuda", "cuda:0", "mps"). --which_model {base,reasoning} Model variant to load (default: "base"). --dataset_size DATASET_SIZE Number of MATH-500 examples to evaluate (default: 10). --max_new_tokens MAX_NEW_TOKENS Max new tokens to generate (default: 2048). --compile Enable torch.compile. --verbose Print per-sample correctness while evaluating. ```   ## `evaluate_math500_batch.py` usage This version extends batching to generation itself, enabling parallel decoding: ```bash uv run evaluate_math500_batched.py --help ``` Extra options: ```bash --batch_size BATCH_SIZE Number of examples to generate in parallel (default: 4). --disable_efficient_mode Use a simpler batched inference method. Slower and more memory-intensive, but easier to debug. ```   **Implementation note:** By default, batched generation halts for sequences that emit a stop token. With `--disable_efficient_mode`, all sequences continue until the longest finishes. This affects compute efficiency only, not qualitative results, since tokens after the stop token are discarded.   **Tip (MPS devices):** Run with: ```bash PYTORCH_ENABLE_MPS_FALLBACK=1 uv run evaluate_math500_batched.py ``` Some PyTorch ops used in efficient batched inference are not yet supported on MPS. As a fallback, you can also use `--disable_efficient_mode`.   - `evaluate_math500.py --dataset_size 500` | Device / Dataset size | Base model | Reasoning model | | ------------------------------------------- | ---------- | --------------- | | **Mac Mini M4 CPU** (500 examples, sequential | 43.6 min | Didn't run (too hot) | | **Mac Mini M4 GPU** (500 examples, sequential) | 37.5 min | Didn't run (too hot) | | **DGX Spark** (500 examples, sequential) | 10.0 min | 182.2 min | | **H100 GPU** (500 examples, sequential) | 13.3 min | 185.4 min | - `evaluate_math500_batched.py --dataset_size 500 --batch_size 128` | Device / Dataset size | Base model | Reasoning model | | ------------------------------------------------------------ | ---------- | --------------- | | **Mac Mini M4 CPU** (500 examples, batched, `--batch_size 128`) | 167.2 min | Didn't run (too hot) | | **Mac Mini M4 GPU** (500 examples, batched, `--batch_size 128`) | Error* | Error | | **DGX Spark** (500 examples, batched, `--batch_size 128`) | 16.3 min | 119.3 min | | **H100 GPU** (500 examples, batched, `--batch_size 128`) | 3.3 min | 14.6 min | - The accuracy of the base model is 15.6% (78/500); the accuracy of the reasoning model is 50.8% (254/500).   ## `evaluate_json.py` usage Use this if you already have saved records and only want to (re)compute accuracy: ```bash uv run evaluate_json.py --json_path math500_base-mps-evaluate-script.jsonl # Accuracy 15.6% (78/500) Optional keys: ```bash uv run evaluate_json.py \ --json_path my_records.json \ --gtruth_answer "gtruth_answer" \ --generated_text "generated_text" ```