tiny-llm

tiny-llm is a hands-on course for systems engineers who want to understand LLM
inference end to end. You can think of it as an LLM-serving counterpart to
CMU's Needle
project: build the path that loads a Qwen3 model, turns tokens into logits, and
generates text.

The course begins with array and matrix operations, then introduces kernels and
serving machinery as the running model needs them. Keeping the implementation
small enough to read end to end makes it possible to connect the equations to
memory traffic, kernel occupancy, KV-cache growth, batching, and request
scheduling.

The course is built on MLX arrays and the MLX extension runtime, without using
high-level neural-network layers. When a chapter teaches an operator, your
solution implements that operator in Python, C++, or Metal rather than calling
the corresponding optimized MLX operation. MLX remains the correctness oracle
and performance baseline.

The Learning Path Copy Chapter

The course follows a four-week learning path:

• Week 1: From Matmul to Text. Build a Qwen3 model directly from mlx.core
  array operations: attention, RoPE, GQA, RMSNorm, the MLP, sampling, and
  the autoregressive loop.
• Week 2: A Step Closer to vLLM. Add a KV cache, establish a
  synchronized MLX baseline, and let matched benchmarks choose each optimization.
  The path moves from quantized decode matvec to fused model kernels, tiled
  prefill, and split-K where the measured Qwen shapes need it.
• Week 3: Build a Mini vLLM. Introduce continuous
  batching and chunked admission, then make paged KV the canonical serving
  layout. Decode attention and FlashAttention learn to read pages directly so
  the scheduler does not rebuild dense history on every step.
• Week 4: Build a Coding Agent. Start with a bounded, validated agent loop,
  then connect it to a small workspace. The course is publishing one reviewed
  checkpoint at a time; Days 1 through 9 now cover inspection, approved edits,
  one validation command, simple effect receipts, and one visible
  checkpoint-and-resume boundary, receipt-backed context compaction, and one
  visible inspect-and-steer pause, and deterministic evaluation of observable
  outcomes, then tokenizer/KV-prefix reuse for two isolated steered branches
  and one explicit evidence-backed selection, followed by bounded,
  range-retrievable evidence for oversized tool results.

Why MLX and Qwen3? Copy Chapter

Apple silicon provides a practical local environment with one shared memory
space and direct access to Metal kernels. Students can inspect the complete
path on one machine instead of depending on an expensive CUDA GPU setup.

Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache
costs, but small enough to iterate on locally. Its grouped-query attention,
QK normalization, BF16 activations, and 4-bit weights also keep the exercises
close to current model-serving work.

Start Here Copy Chapter

The book is published at
skyzh.github.io/tiny-llm. Begin with the
environment setup, or verify an
existing checkout with:

pdm install -v
pdm run check-installation
pdm run test-refsol -- -- -k week_1

The tiny_llm package is where students implement the exercises.
tiny_llm_ref contains the reference solution used by the tests and benchmark
appendix. The book summary lists the chapter order;
implementation, test, and publication readiness is tracked below.

Roadmap Copy Chapter

The table tracks implementation (Code), tests (Test), rendered chapters (Doc), and Chi's review of learner-facing material (Audit). Week 4 is publishing one reviewed day at a time; Days 1 through 9 are currently available to learners. The Audit column reflects Chi's personal editorial pass on the published course content and is independent of code/test/doc readiness.

Day 3 can send file contents to the model, modify files after approval, and run
one exact configured command. Use a disposable workspace without secrets and
read the Week 4 overview before running the loop.
Day 4 checkpoints a complete tool-observation boundary with the scripted
model's fake cache metadata, then restores a fresh model without replaying the
completed edit or command.
Day 5 compacts older completed effects in the model-visible transcript while
their exact receipts retain the full action, result, and changed artifacts.
Day 6 inspects one complete-observation checkpoint, appends one visible operator
instruction, and resumes a fresh model without replaying the completed effect.
Day 7 evaluates one completed run from declared final, file, result, and receipt
facts without grading hidden reasoning or exact transcript shape.
Day 8 reuses one real tokenizer/KV checkpoint for two differently steered,
effect-isolated continuations, evaluates both with Day 7's harness, and makes
one explicit passing selection without pretending completed effects were
rewound.
Day 9 stores exact oversized tool-result bytes outside the model prompt, shows
a bounded identity/digest/head-tail observation, and lets the model retrieve
one explicit byte range through the existing loop.

Other topics not covered include quantized or compressed KV caches,
cross-request prefix caching, fine-tuning, and long-context techniques.

Community Copy Chapter

Join skyzh's Discord server to study with the tiny-llm community.