{"owner":"stas00","repo":"ml-engineering","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["SKILL.md"],"skills":{"SKILL.md":"---\nname: ml-engineering\ndescription: >-\n  Field-tested methodology and concrete recipes for training and operating\n  large-scale LLM/VLM/multi-modal models end to end - choosing and benchmarking\n  accelerators, storage and network; SLURM/Kubernetes orchestration; maximizing\n  training throughput and fitting models in memory; diagnosing and surviving\n  training instabilities, NaN/Inf, and hardware/job failures; checkpointing and\n  fault tolerance; inference performance and memory; debugging multi-node/\n  multi-GPU hangs; and writing/running tests. Use when the user is training or\n  fine-tuning large models, hits low TFLOPS/MFU, OOM, slow dataloading, a\n  loss spike/divergence, a NCCL/InfiniBand or multi-node hang, node/GPU\n  failures, checkpoint or preemption problems, storage/network bottlenecks, or\n  needs to pick GPUs/cloud/file-systems or size inference latency/throughput.\n  Distilled from \"Machine Learning Engineering\", the latest version of which can\n  be found at https://github.com/stas00/ml-engineering\n\n  The latest SKILL.md version can be found at https://github.com/stas00/ml-engineering/blob/master/SKILL.md\n---\n\n# Machine Learning Engineering\n\nDistilled from **Machine Learning Engineering Open Book** by Stas Bekman - source: https://github.com/stas00/ml-engineering (CC BY-SA 4.0). Know-how from training BLOOM-176B, IDEFICS-80B and production RAG and RL training and inference systems. This skill is a condensed index; each section links back to the full chapter for depth, scripts, and benchmarks.\n\nA field-tested, end-to-end guide to **training and serving large models** (LLMs, VLMs, multi-modal, RAG) on real hardware at scale - distilled from actually training BLOOM-176B and IDEFICS-80B and building production inference/RAG systems. It is a practitioner's brain dump: opinionated guidance backed by copy-paste scripts, benchmark tools, and comparison tables, written for the engineers and operators who have to make expensive clusters actually deliver a finished model.\n\nIt spans the entire stack that decides whether a run succeeds and how much it costs: selecting and *benchmarking* accelerators, storage, and network so the fast compute is never starved; orchestrating jobs with SLURM/Kubernetes; maximizing throughput (MFU) and fitting models in memory via parallelism (DP/TP/PP/ZeRO), activation recomputation, and offload; keeping training numerically stable through loss spikes and NaN/Inf; and surviving the *inevitable* hardware and job failures with frequent checkpointing, spare capacity, and automatic restarts. On the serving side it covers inference latency/throughput/cost trade-offs, KV-cache and memory sizing, and framework selection - plus diagnosing multi-node/multi-GPU hangs and testing the whole thing.\n\nUse it as an operator's runbook: figure out which resource is *actually* the bottleneck (compute? memory? network? storage? dataloader?), then jump to the targeted recipe. For pure debugging technique (gdb/strace/py-spy/CUDA), pair this with [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n\n## Core principles\n\n- **Measure, don't assume.** Vendor/theoretical TFLOPS are marketing; benchmark *your* hardware and software stack before optimizing or buying. Track MFU/throughput, not vibes.\n- **Find the actual bottleneck.** A training step is gated by the slowest of: accelerator compute, memory bandwidth/capacity, inter/intra-node network, storage IO, or CPU dataloading. Optimizing anything else is wasted effort.\n- **At scale, failure is the steady state.** With hundreds/thousands of accelerators, hardware *will* fail mid-run. Design for frequent checkpoints, automatic restarts, spare nodes, and kill/save switches from day one.\n- **Reproduce small and fast.** Debug on a tiny model / few layers / one node before burning cluster time - see [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n- **Watch the logbooks.** Others have already hit your instability; training chronicles document the loss spikes and the fixes. See [LLM/VLM chronicles](https://github.com/stas00/ml-engineering/tree/master/resources#publicly-available-training-llmvlm-logbooks).\n\n## Compute / accelerators\n\nFull chapter: [Compute](https://github.com/stas00/ml-engineering/tree/master/compute) · [Accelerators](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md).\n\n- **The number that matters is achievable, not theoretical.** Real matmul FLOPS are well below spec. Measure Maximum Achievable Matmul FLOPS with [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py); compute **MFU** (model FLOPS utilization) to compare setups. See [the most important thing to understand](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#the-most-important-thing-to-understand) and [what accelerator characteristics we care for](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#what-accelerator-characteristics-do-we-care-for).\n- **Memory capacity + bandwidth often bind before FLOPS.** Check the [accelerator memory size and speed table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#accelerator-memory-size-and-speed) and the [TFLOPS comparison table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table) when choosing hardware.\n- **Don't forget power and cooling** - throttling silently caps sustained throughput. See [power and cooling](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#power-and-cooling).\n- **CPU and CPU memory matter too** (dataloading, offload, pinned memory). See [CPU](https://github.com/stas00/ml-engineering/tree/master/compute/cpu) and [CPU memory](https://github.com/stas00/ml-engineering/tree/master/compute/cpu-memory).\n\n## Storage (IO)\n\nFull chapter: [Storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md).\n\n- **Pick the file system for the job:** distributed/parallel FS for shared checkpoints and datasets; fast **local** NVMe beats network storage for scratch/hot data. See [which file system to choose](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#which-file-system-to-choose) and [local storage beats cloud storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#local-storage-beats-cloud-storage).\n- **Benchmark IO before you trust it** (checkpoint save/load and dataset streaming are common stalls). See [storage benchmarks](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#benchmarks).\n- **Gotchas that bite at scale:** you often get less usable capacity than sold; some clouds put backups on the same partition; always keep [checksums](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#dont-forget-the-checksums). Clean up instead of overpaying - see [why pay for more storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#why-pay-for-more-storage-when-you-can-easily-clean-it-up-instead).\n\n## Network\n\nFull chapter: [Network](https://github.com/stas00/ml-engineering/blob/master/network/README.md).\n\n- **Inter-node speed can dominate the whole training's speed.** For sharded/parallel training, slow inter-node links stall everyone. Understand why before scaling out: [why inter-node speed is hugely important](https://github.com/stas00/ml-engineering/blob/master/network/README.md#understanding-why-inter-node-network-speed-is-of-a-huge-importance).\n- **Know the two speeds:** [intra-node](https://github.com/stas00/ml-engineering/blob/master/network/README.md#intra-node-networking) (NVLink/PCIe) vs [inter-node](https://github.com/stas00/ml-engineering/blob/master/network/README.md#inter-node-networking) (InfiniBand/RoCE/Ethernet), and [RDMA](https://github.com/stas00/ml-engineering/blob/master/network/README.md#rdma-networking).\n- **Benchmark real collective throughput,** not marketing numbers, with [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) (far simpler than nccl-tests). Mind the [important nuances](https://github.com/stas00/ml-engineering/blob/master/network/README.md#important-nuances) (e.g. unidirectional vs bidirectional, payload size, busbw vs algbw).\n- **When collectives hang or crawl:** see [network debug](https://github.com/stas00/ml-engineering/tree/master/network/debug) and [NCCL performance debug](https://github.com/stas00/ml-engineering/blob/master/debug/nccl-performance-debug.md).\n\n## Orchestration & SLURM\n\nFull chapter: [Orchestration](https://github.com/stas00/ml-engineering/tree/master/orchestration) · [SLURM](https://github.com/stas00/ml-engineering/tree/master/orchestration/slurm) · [Kubernetes](https://github.com/stas00/ml-engineering/tree/master/orchestration/kubernetes).\n\n- **Verify the cluster before the big run:** every GPU on every node must talk to every other. Run [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) across all nodes first.\n- **SLURM day-to-day:** the [users cheatsheet](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/users.md) covers `sbatch`/`srun`/`salloc`, job arrays, dependencies, and inspecting the queue; keep the allocation and re-`srun` for fast debug iterations.\n- **Launchers** (torchrun/accelerate/deepspeed under SLURM): see [launchers](https://github.com/stas00/ml-engineering/tree/master/orchestration/slurm/launchers/README.md).\n\n## Training: performance & memory\n\nFull chapter: [Performance](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md).\n\n- **Start from the checklist:** [how to improve speed and save memory](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#how-to-improve-speed-and-save-memory) enumerates the high-impact levers (parallelism choice, activation checkpointing/recomputation, offload, fused kernels, mixed precision).\n- **Fit the model:** know the [anatomy of memory usage](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#anatomy-of-models-memory-usage) (weights + grads + optimizer states + activations) and profile with [memory profiler tools](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#memory-profiler-tools).\n- **Free throughput wins:** keep tensor dims [divisible/aligned](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#vector-and-matrix-size-divisibility) for Tensor Cores, set [NUMA affinity](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#numa-affinity), stop the [DataLoader](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#dataloader) from starving the GPUs, try [`torch.compile`](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#torchcompile), and tame [automatic garbage collection](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#automatic-garbage-collection) jitter in synchronized runs.\n- **Model parallelism** (DP/TP/PP/sequence/ZeRO) concepts and trade-offs: [model parallelism](https://github.com/stas00/ml-engineering/tree/master/training/model-parallelism).\n\n## Training: stability (instabilities & NaN/Inf)\n\nFull chapter: [Instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md).\n\n- **Loss spikes/divergence are expected at scale.** Compare against public [training logbooks](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#learning-from-training-logbooks) - your symptom is probably documented with a known mitigation.\n- **Numerical hygiene:** sane [weight init / STD](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#std-init), watch for [numerical instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#numerical-instabilities), and bad [data-batch × parameter-state combinations](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#bad-combination-of-data-batch-and-model-parameter-state).\n- **Catch NaN/Inf early:** [underflow/overflow detection](https://github.com/stas00/ml-engineering/blob/master/debug/underflow_overflow.md) and PyTorch [tensor debugging](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#debugging-tensors) (fp16/bf16 range issues, per-tensor min/max/NaN scans).\n- **Reproducibility** for isolating a divergence: [reproducibility](https://github.com/stas00/ml-engineering/tree/master/training/reproducibility).\n\n## Training: fault tolerance & checkpoints\n\nFull chapter: [Fault tolerance](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md) · [Checkpoints](https://github.com/stas00/ml-engineering/tree/master/training/checkpoints).\n\n- **Provision slack:** [always plan for more nodes than needed](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#always-plan-to-have-more-nodes-than-needed) so a dead node doesn't stop the run; [queue up multiple jobs](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#queue-up-multiple-training-jobs) for auto-continuation.\n- **Checkpoint often enough** that a crash costs minutes, not hours - see [frequent checkpoint saving](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#frequent-checkpoint-saving) and [prevention](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#prevention).\n- **Operator controls:** a [kill switch](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#kill-switch) to stop a doomed run cleanly and a [save switch](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#save-switch) to force a checkpoint on demand.\n- **Survive the cluster:** handle [forced job preemption](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#dealing-with-forced-job-preemption) and prefer [fixed over dynamic allocations](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#preferring-fixed-accelerator-allocations-to-dynamic-ones).\n\n## Inference\n\nFull chapter: [Inference](https://github.com/stas00/ml-engineering/blob/master/inference/README.md).\n\n- **Speak the metrics:** TTFT, TPOT/ITL, throughput vs latency, and how batching trades them off. See [key inference performance metrics](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#key-inference-performance-metrics) and [concepts](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#concepts).\n- **Size the memory:** weights + KV cache + activations - see [anatomy of model's memory usage](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#anatomy-of-models-memory-usage).\n- **Cut cold starts:** [speeding up model loading time](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#speeding-up-model-loading-time); pick a serving stack from [inference frameworks](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#inference-frameworks) and validate with [benchmarks](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#benchmarks).\n\n## Debugging distributed / PyTorch at scale\n\nFull chapter: [Debugging](https://github.com/stas00/ml-engineering/tree/master/debug) · [PyTorch](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md).\n\n- **Iterate cheaply:** shrink to tiny models/tokenizers/datasets and [fast debug of PyTorch models](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#faster-debug-and-development-with-tiny-models-tokenizers-and-datasets).\n- **Multi-node/multi-GPU hang or deadlock** (the classic scale bug):\n  1. Rule out comms with [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py).\n  2. Dump every rank's Python stack at once with `py-spy`; ranks stuck at *different* collectives reveal the desync. See [diagnosing crashes, hangs and tracing execution](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#diagnosing-crashes-hangs-and-tracing-execution) and [hanging solutions](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-hanging-solutions.md).\n  3. Set `NCCL_DEBUG=INFO`; escalate to [NCCL performance debug](https://github.com/stas00/ml-engineering/blob/master/debug/nccl-performance-debug.md).\n- **Cryptic CUDA errors:** `CUDA_LAUNCH_BLOCKING=1` for a real traceback (CUDA is async). See [The Art of Debugging - PyTorch](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#dealing-with-async-cuda-bugs).\n- **OOM / memory:** [memory usage](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#memory-usage); fragmentation via `PYTORCH_ALLOC_CONF`.\n- **NaN/Inf:** [underflow and overflow detection](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#underflow-and-overflow-detection) and [debugging tensors](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#debugging-tensors).\n- **GPU-specific faults:** [troubleshooting NVIDIA GPUs](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/nvidia/debug.md).\n\n## Testing\n\nFull chapter: [Testing](https://github.com/stas00/ml-engineering/blob/master/testing/README.md).\n\n- **Run tests surgically** (select, parametrize, repeat, control output/parallelism): [running tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#running-tests).\n- **Write robust tests** (fixtures, temp dirs, RNG control for reproducibility, distributed tests): [writing tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#writing-tests).\n- **When a test misbehaves:** [debugging tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#debugging-tests).\n\n## Key tools\n\n| Need | Tool |\n|---|---|\n| Verify all GPUs/nodes can talk & allocate | [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) |\n| Real network throughput (all-reduce busbw) | [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) |\n| Actual achievable matmul FLOPS of an accelerator | [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py) |\n| Tiny models/tokenizers/datasets for fast iteration | [make-tiny guide](https://github.com/stas00/ml-engineering/blob/master/debug/make-tiny-models-tokenizers-datasets.md) |\n| Better `trace` for distributed hangs | [NicerTrace](https://github.com/stas00/ml-engineering/blob/master/debug/tools.md) |\n\n## Pick the fix by symptom\n\n| Symptom | Reach for |\n|---|---|\n| Low TFLOPS / MFU, \"GPUs feel idle\" | Find the bottleneck: `mamf-finder`, [performance checklist](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#how-to-improve-speed-and-save-memory), DataLoader, NUMA, dim divisibility |\n| Training OOM | Memory anatomy → activation checkpointing/offload/parallelism; profile; `PYTORCH_ALLOC_CONF` |\n| Slow steps but GPUs busy on comms | Benchmark network (`all_reduce_bench`), check intra/inter-node, NCCL settings |\n| Slow dataloading / GPU starvation | [DataLoader](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#dataloader), local NVMe, prefetch/workers |\n| Loss spike / divergence / NaN | Logbooks, init/STD, underflow-overflow detection, tensor scans |\n| Multi-node/GPU hang or deadlock | `torch-distributed-gpu-test.py` → `py-spy` all ranks → `NCCL_DEBUG=INFO` |\n| Node/GPU dies mid-run | Spare nodes, frequent checkpoints, auto-restart, kill/save switch |\n| Job keeps getting preempted | [forced preemption](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#dealing-with-forced-job-preemption), queue chained jobs |\n| Checkpoint save/load is slow | Benchmark storage, choose FS, local vs shared |\n| Choosing GPUs / cloud / storage | Comparison tables, MAMF, [choose a cloud provider](https://github.com/stas00/ml-engineering/blob/master/insights/how-to-choose-cloud-provider.md) |\n| Inference too slow / won't fit | Metrics (TTFT/TPOT), KV-cache memory, framework choice, model-load speedups |\n\n## Notes for AI agents\n\n- **Diagnose before optimizing.** Identify which resource (compute/memory/network/storage/dataloader) is the actual bottleneck with a measurement; don't tune blindly.\n- **Prefer measured numbers over spec sheets.** Use the provided benchmark scripts on the target hardware/software stack before recommending changes or purchases.\n- **Assume failures at scale.** For any long/large run, verify checkpointing, restart, spare capacity, and a kill switch exist before worrying about peak speed.\n- **Verify the cluster first.** Run the distributed connectivity test before blaming model code for a multi-node problem.\n- **Reuse the community's hard-won lessons.** Check the training logbooks for known instabilities and fixes before re-deriving them.\n- **Read the linked chapter section** before applying a recipe - each has worked examples, exact commands, caveats, and scripts.\n- For deep single-process/tool debugging (gdb, strace, py-spy, cProfile, core files), use the companion skill: [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n"},"files":{"SKILL.md":"---\nname: ml-engineering\ndescription: >-\n  Field-tested methodology and concrete recipes for training and operating\n  large-scale LLM/VLM/multi-modal models end to end - choosing and benchmarking\n  accelerators, storage and network; SLURM/Kubernetes orchestration; maximizing\n  training throughput and fitting models in memory; diagnosing and surviving\n  training instabilities, NaN/Inf, and hardware/job failures; checkpointing and\n  fault tolerance; inference performance and memory; debugging multi-node/\n  multi-GPU hangs; and writing/running tests. Use when the user is training or\n  fine-tuning large models, hits low TFLOPS/MFU, OOM, slow dataloading, a\n  loss spike/divergence, a NCCL/InfiniBand or multi-node hang, node/GPU\n  failures, checkpoint or preemption problems, storage/network bottlenecks, or\n  needs to pick GPUs/cloud/file-systems or size inference latency/throughput.\n  Distilled from \"Machine Learning Engineering\", the latest version of which can\n  be found at https://github.com/stas00/ml-engineering\n\n  The latest SKILL.md version can be found at https://github.com/stas00/ml-engineering/blob/master/SKILL.md\n---\n\n# Machine Learning Engineering\n\nDistilled from **Machine Learning Engineering Open Book** by Stas Bekman - source: https://github.com/stas00/ml-engineering (CC BY-SA 4.0). Know-how from training BLOOM-176B, IDEFICS-80B and production RAG and RL training and inference systems. This skill is a condensed index; each section links back to the full chapter for depth, scripts, and benchmarks.\n\nA field-tested, end-to-end guide to **training and serving large models** (LLMs, VLMs, multi-modal, RAG) on real hardware at scale - distilled from actually training BLOOM-176B and IDEFICS-80B and building production inference/RAG systems. It is a practitioner's brain dump: opinionated guidance backed by copy-paste scripts, benchmark tools, and comparison tables, written for the engineers and operators who have to make expensive clusters actually deliver a finished model.\n\nIt spans the entire stack that decides whether a run succeeds and how much it costs: selecting and *benchmarking* accelerators, storage, and network so the fast compute is never starved; orchestrating jobs with SLURM/Kubernetes; maximizing throughput (MFU) and fitting models in memory via parallelism (DP/TP/PP/ZeRO), activation recomputation, and offload; keeping training numerically stable through loss spikes and NaN/Inf; and surviving the *inevitable* hardware and job failures with frequent checkpointing, spare capacity, and automatic restarts. On the serving side it covers inference latency/throughput/cost trade-offs, KV-cache and memory sizing, and framework selection - plus diagnosing multi-node/multi-GPU hangs and testing the whole thing.\n\nUse it as an operator's runbook: figure out which resource is *actually* the bottleneck (compute? memory? network? storage? dataloader?), then jump to the targeted recipe. For pure debugging technique (gdb/strace/py-spy/CUDA), pair this with [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n\n## Core principles\n\n- **Measure, don't assume.** Vendor/theoretical TFLOPS are marketing; benchmark *your* hardware and software stack before optimizing or buying. Track MFU/throughput, not vibes.\n- **Find the actual bottleneck.** A training step is gated by the slowest of: accelerator compute, memory bandwidth/capacity, inter/intra-node network, storage IO, or CPU dataloading. Optimizing anything else is wasted effort.\n- **At scale, failure is the steady state.** With hundreds/thousands of accelerators, hardware *will* fail mid-run. Design for frequent checkpoints, automatic restarts, spare nodes, and kill/save switches from day one.\n- **Reproduce small and fast.** Debug on a tiny model / few layers / one node before burning cluster time - see [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n- **Watch the logbooks.** Others have already hit your instability; training chronicles document the loss spikes and the fixes. See [LLM/VLM chronicles](https://github.com/stas00/ml-engineering/tree/master/resources#publicly-available-training-llmvlm-logbooks).\n\n## Compute / accelerators\n\nFull chapter: [Compute](https://github.com/stas00/ml-engineering/tree/master/compute) · [Accelerators](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md).\n\n- **The number that matters is achievable, not theoretical.** Real matmul FLOPS are well below spec. Measure Maximum Achievable Matmul FLOPS with [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py); compute **MFU** (model FLOPS utilization) to compare setups. See [the most important thing to understand](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#the-most-important-thing-to-understand) and [what accelerator characteristics we care for](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#what-accelerator-characteristics-do-we-care-for).\n- **Memory capacity + bandwidth often bind before FLOPS.** Check the [accelerator memory size and speed table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#accelerator-memory-size-and-speed) and the [TFLOPS comparison table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table) when choosing hardware.\n- **Don't forget power and cooling** - throttling silently caps sustained throughput. See [power and cooling](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#power-and-cooling).\n- **CPU and CPU memory matter too** (dataloading, offload, pinned memory). See [CPU](https://github.com/stas00/ml-engineering/tree/master/compute/cpu) and [CPU memory](https://github.com/stas00/ml-engineering/tree/master/compute/cpu-memory).\n\n## Storage (IO)\n\nFull chapter: [Storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md).\n\n- **Pick the file system for the job:** distributed/parallel FS for shared checkpoints and datasets; fast **local** NVMe beats network storage for scratch/hot data. See [which file system to choose](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#which-file-system-to-choose) and [local storage beats cloud storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#local-storage-beats-cloud-storage).\n- **Benchmark IO before you trust it** (checkpoint save/load and dataset streaming are common stalls). See [storage benchmarks](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#benchmarks).\n- **Gotchas that bite at scale:** you often get less usable capacity than sold; some clouds put backups on the same partition; always keep [checksums](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#dont-forget-the-checksums). Clean up instead of overpaying - see [why pay for more storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#why-pay-for-more-storage-when-you-can-easily-clean-it-up-instead).\n\n## Network\n\nFull chapter: [Network](https://github.com/stas00/ml-engineering/blob/master/network/README.md).\n\n- **Inter-node speed can dominate the whole training's speed.** For sharded/parallel training, slow inter-node links stall everyone. Understand why before scaling out: [why inter-node speed is hugely important](https://github.com/stas00/ml-engineering/blob/master/network/README.md#understanding-why-inter-node-network-speed-is-of-a-huge-importance).\n- **Know the two speeds:** [intra-node](https://github.com/stas00/ml-engineering/blob/master/network/README.md#intra-node-networking) (NVLink/PCIe) vs [inter-node](https://github.com/stas00/ml-engineering/blob/master/network/README.md#inter-node-networking) (InfiniBand/RoCE/Ethernet), and [RDMA](https://github.com/stas00/ml-engineering/blob/master/network/README.md#rdma-networking).\n- **Benchmark real collective throughput,** not marketing numbers, with [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) (far simpler than nccl-tests). Mind the [important nuances](https://github.com/stas00/ml-engineering/blob/master/network/README.md#important-nuances) (e.g. unidirectional vs bidirectional, payload size, busbw vs algbw).\n- **When collectives hang or crawl:** see [network debug](https://github.com/stas00/ml-engineering/tree/master/network/debug) and [NCCL performance debug](https://github.com/stas00/ml-engineering/blob/master/debug/nccl-performance-debug.md).\n\n## Orchestration & SLURM\n\nFull chapter: [Orchestration](https://github.com/stas00/ml-engineering/tree/master/orchestration) · [SLURM](https://github.com/stas00/ml-engineering/tree/master/orchestration/slurm) · [Kubernetes](https://github.com/stas00/ml-engineering/tree/master/orchestration/kubernetes).\n\n- **Verify the cluster before the big run:** every GPU on every node must talk to every other. Run [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) across all nodes first.\n- **SLURM day-to-day:** the [users cheatsheet](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/users.md) covers `sbatch`/`srun`/`salloc`, job arrays, dependencies, and inspecting the queue; keep the allocation and re-`srun` for fast debug iterations.\n- **Launchers** (torchrun/accelerate/deepspeed under SLURM): see [launchers](https://github.com/stas00/ml-engineering/tree/master/orchestration/slurm/launchers/README.md).\n\n## Training: performance & memory\n\nFull chapter: [Performance](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md).\n\n- **Start from the checklist:** [how to improve speed and save memory](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#how-to-improve-speed-and-save-memory) enumerates the high-impact levers (parallelism choice, activation checkpointing/recomputation, offload, fused kernels, mixed precision).\n- **Fit the model:** know the [anatomy of memory usage](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#anatomy-of-models-memory-usage) (weights + grads + optimizer states + activations) and profile with [memory profiler tools](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#memory-profiler-tools).\n- **Free throughput wins:** keep tensor dims [divisible/aligned](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#vector-and-matrix-size-divisibility) for Tensor Cores, set [NUMA affinity](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#numa-affinity), stop the [DataLoader](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#dataloader) from starving the GPUs, try [`torch.compile`](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#torchcompile), and tame [automatic garbage collection](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#automatic-garbage-collection) jitter in synchronized runs.\n- **Model parallelism** (DP/TP/PP/sequence/ZeRO) concepts and trade-offs: [model parallelism](https://github.com/stas00/ml-engineering/tree/master/training/model-parallelism).\n\n## Training: stability (instabilities & NaN/Inf)\n\nFull chapter: [Instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md).\n\n- **Loss spikes/divergence are expected at scale.** Compare against public [training logbooks](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#learning-from-training-logbooks) - your symptom is probably documented with a known mitigation.\n- **Numerical hygiene:** sane [weight init / STD](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#std-init), watch for [numerical instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#numerical-instabilities), and bad [data-batch × parameter-state combinations](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#bad-combination-of-data-batch-and-model-parameter-state).\n- **Catch NaN/Inf early:** [underflow/overflow detection](https://github.com/stas00/ml-engineering/blob/master/debug/underflow_overflow.md) and PyTorch [tensor debugging](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#debugging-tensors) (fp16/bf16 range issues, per-tensor min/max/NaN scans).\n- **Reproducibility** for isolating a divergence: [reproducibility](https://github.com/stas00/ml-engineering/tree/master/training/reproducibility).\n\n## Training: fault tolerance & checkpoints\n\nFull chapter: [Fault tolerance](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md) · [Checkpoints](https://github.com/stas00/ml-engineering/tree/master/training/checkpoints).\n\n- **Provision slack:** [always plan for more nodes than needed](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#always-plan-to-have-more-nodes-than-needed) so a dead node doesn't stop the run; [queue up multiple jobs](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#queue-up-multiple-training-jobs) for auto-continuation.\n- **Checkpoint often enough** that a crash costs minutes, not hours - see [frequent checkpoint saving](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#frequent-checkpoint-saving) and [prevention](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#prevention).\n- **Operator controls:** a [kill switch](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#kill-switch) to stop a doomed run cleanly and a [save switch](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#save-switch) to force a checkpoint on demand.\n- **Survive the cluster:** handle [forced job preemption](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#dealing-with-forced-job-preemption) and prefer [fixed over dynamic allocations](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#preferring-fixed-accelerator-allocations-to-dynamic-ones).\n\n## Inference\n\nFull chapter: [Inference](https://github.com/stas00/ml-engineering/blob/master/inference/README.md).\n\n- **Speak the metrics:** TTFT, TPOT/ITL, throughput vs latency, and how batching trades them off. See [key inference performance metrics](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#key-inference-performance-metrics) and [concepts](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#concepts).\n- **Size the memory:** weights + KV cache + activations - see [anatomy of model's memory usage](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#anatomy-of-models-memory-usage).\n- **Cut cold starts:** [speeding up model loading time](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#speeding-up-model-loading-time); pick a serving stack from [inference frameworks](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#inference-frameworks) and validate with [benchmarks](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#benchmarks).\n\n## Debugging distributed / PyTorch at scale\n\nFull chapter: [Debugging](https://github.com/stas00/ml-engineering/tree/master/debug) · [PyTorch](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md).\n\n- **Iterate cheaply:** shrink to tiny models/tokenizers/datasets and [fast debug of PyTorch models](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#faster-debug-and-development-with-tiny-models-tokenizers-and-datasets).\n- **Multi-node/multi-GPU hang or deadlock** (the classic scale bug):\n  1. Rule out comms with [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py).\n  2. Dump every rank's Python stack at once with `py-spy`; ranks stuck at *different* collectives reveal the desync. See [diagnosing crashes, hangs and tracing execution](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#diagnosing-crashes-hangs-and-tracing-execution) and [hanging solutions](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-hanging-solutions.md).\n  3. Set `NCCL_DEBUG=INFO`; escalate to [NCCL performance debug](https://github.com/stas00/ml-engineering/blob/master/debug/nccl-performance-debug.md).\n- **Cryptic CUDA errors:** `CUDA_LAUNCH_BLOCKING=1` for a real traceback (CUDA is async). See [The Art of Debugging - PyTorch](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#dealing-with-async-cuda-bugs).\n- **OOM / memory:** [memory usage](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#memory-usage); fragmentation via `PYTORCH_ALLOC_CONF`.\n- **NaN/Inf:** [underflow and overflow detection](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#underflow-and-overflow-detection) and [debugging tensors](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#debugging-tensors).\n- **GPU-specific faults:** [troubleshooting NVIDIA GPUs](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/nvidia/debug.md).\n\n## Testing\n\nFull chapter: [Testing](https://github.com/stas00/ml-engineering/blob/master/testing/README.md).\n\n- **Run tests surgically** (select, parametrize, repeat, control output/parallelism): [running tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#running-tests).\n- **Write robust tests** (fixtures, temp dirs, RNG control for reproducibility, distributed tests): [writing tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#writing-tests).\n- **When a test misbehaves:** [debugging tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#debugging-tests).\n\n## Key tools\n\n| Need | Tool |\n|---|---|\n| Verify all GPUs/nodes can talk & allocate | [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) |\n| Real network throughput (all-reduce busbw) | [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) |\n| Actual achievable matmul FLOPS of an accelerator | [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py) |\n| Tiny models/tokenizers/datasets for fast iteration | [make-tiny guide](https://github.com/stas00/ml-engineering/blob/master/debug/make-tiny-models-tokenizers-datasets.md) |\n| Better `trace` for distributed hangs | [NicerTrace](https://github.com/stas00/ml-engineering/blob/master/debug/tools.md) |\n\n## Pick the fix by symptom\n\n| Symptom | Reach for |\n|---|---|\n| Low TFLOPS / MFU, \"GPUs feel idle\" | Find the bottleneck: `mamf-finder`, [performance checklist](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#how-to-improve-speed-and-save-memory), DataLoader, NUMA, dim divisibility |\n| Training OOM | Memory anatomy → activation checkpointing/offload/parallelism; profile; `PYTORCH_ALLOC_CONF` |\n| Slow steps but GPUs busy on comms | Benchmark network (`all_reduce_bench`), check intra/inter-node, NCCL settings |\n| Slow dataloading / GPU starvation | [DataLoader](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#dataloader), local NVMe, prefetch/workers |\n| Loss spike / divergence / NaN | Logbooks, init/STD, underflow-overflow detection, tensor scans |\n| Multi-node/GPU hang or deadlock | `torch-distributed-gpu-test.py` → `py-spy` all ranks → `NCCL_DEBUG=INFO` |\n| Node/GPU dies mid-run | Spare nodes, frequent checkpoints, auto-restart, kill/save switch |\n| Job keeps getting preempted | [forced preemption](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#dealing-with-forced-job-preemption), queue chained jobs |\n| Checkpoint save/load is slow | Benchmark storage, choose FS, local vs shared |\n| Choosing GPUs / cloud / storage | Comparison tables, MAMF, [choose a cloud provider](https://github.com/stas00/ml-engineering/blob/master/insights/how-to-choose-cloud-provider.md) |\n| Inference too slow / won't fit | Metrics (TTFT/TPOT), KV-cache memory, framework choice, model-load speedups |\n\n## Notes for AI agents\n\n- **Diagnose before optimizing.** Identify which resource (compute/memory/network/storage/dataloader) is the actual bottleneck with a measurement; don't tune blindly.\n- **Prefer measured numbers over spec sheets.** Use the provided benchmark scripts on the target hardware/software stack before recommending changes or purchases.\n- **Assume failures at scale.** For any long/large run, verify checkpointing, restart, spare capacity, and a kill switch exist before worrying about peak speed.\n- **Verify the cluster first.** Run the distributed connectivity test before blaming model code for a multi-node problem.\n- **Reuse the community's hard-won lessons.** Check the training logbooks for known instabilities and fixes before re-deriving them.\n- **Read the linked chapter section** before applying a recipe - each has worked examples, exact commands, caveats, and scripts.\n- For deep single-process/tool debugging (gdb, strace, py-spy, cProfile, core files), use the companion skill: [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n"},"items":[{"name":"SKILL.md","path":"SKILL.md","title":"SKILL.md","content":"---\nname: ml-engineering\ndescription: >-\n  Field-tested methodology and concrete recipes for training and operating\n  large-scale LLM/VLM/multi-modal models end to end - choosing and benchmarking\n  accelerators, storage and network; SLURM/Kubernetes orchestration; maximizing\n  training throughput and fitting models in memory; diagnosing and surviving\n  training instabilities, NaN/Inf, and hardware/job failures; checkpointing and\n  fault tolerance; inference performance and memory; debugging multi-node/\n  multi-GPU hangs; and writing/running tests. Use when the user is training or\n  fine-tuning large models, hits low TFLOPS/MFU, OOM, slow dataloading, a\n  loss spike/divergence, a NCCL/InfiniBand or multi-node hang, node/GPU\n  failures, checkpoint or preemption problems, storage/network bottlenecks, or\n  needs to pick GPUs/cloud/file-systems or size inference latency/throughput.\n  Distilled from \"Machine Learning Engineering\", the latest version of which can\n  be found at https://github.com/stas00/ml-engineering\n\n  The latest SKILL.md version can be found at https://github.com/stas00/ml-engineering/blob/master/SKILL.md\n---\n\n# Machine Learning Engineering\n\nDistilled from **Machine Learning Engineering Open Book** by Stas Bekman - source: https://github.com/stas00/ml-engineering (CC BY-SA 4.0). Know-how from training BLOOM-176B, IDEFICS-80B and production RAG and RL training and inference systems. This skill is a condensed index; each section links back to the full chapter for depth, scripts, and benchmarks.\n\nA field-tested, end-to-end guide to **training and serving large models** (LLMs, VLMs, multi-modal, RAG) on real hardware at scale - distilled from actually training BLOOM-176B and IDEFICS-80B and building production inference/RAG systems. It is a practitioner's brain dump: opinionated guidance backed by copy-paste scripts, benchmark tools, and comparison tables, written for the engineers and operators who have to make expensive clusters actually deliver a finished model.\n\nIt spans the entire stack that decides whether a run succeeds and how much it costs: selecting and *benchmarking* accelerators, storage, and network so the fast compute is never starved; orchestrating jobs with SLURM/Kubernetes; maximizing throughput (MFU) and fitting models in memory via parallelism (DP/TP/PP/ZeRO), activation recomputation, and offload; keeping training numerically stable through loss spikes and NaN/Inf; and surviving the *inevitable* hardware and job failures with frequent checkpointing, spare capacity, and automatic restarts. On the serving side it covers inference latency/throughput/cost trade-offs, KV-cache and memory sizing, and framework selection - plus diagnosing multi-node/multi-GPU hangs and testing the whole thing.\n\nUse it as an operator's runbook: figure out which resource is *actually* the bottleneck (compute? memory? network? storage? dataloader?), then jump to the targeted recipe. For pure debugging technique (gdb/strace/py-spy/CUDA), pair this with [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n\n## Core principles\n\n- **Measure, don't assume.** Vendor/theoretical TFLOPS are marketing; benchmark *your* hardware and software stack before optimizing or buying. Track MFU/throughput, not vibes.\n- **Find the actual bottleneck.** A training step is gated by the slowest of: accelerator compute, memory bandwidth/capacity, inter/intra-node network, storage IO, or CPU dataloading. Optimizing anything else is wasted effort.\n- **At scale, failure is the steady state.** With hundreds/thousands of accelerators, hardware *will* fail mid-run. Design for frequent checkpoints, automatic restarts, spare nodes, and kill/save switches from day one.\n- **Reproduce small and fast.** Debug on a tiny model / few layers / one node before burning cluster time - see [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n- **Watch the logbooks.** Others have already hit your instability; training chronicles document the loss spikes and the fixes. See [LLM/VLM chronicles](https://github.com/stas00/ml-engineering/tree/master/resources#publicly-available-training-llmvlm-logbooks).\n\n## Compute / accelerators\n\nFull chapter: [Compute](https://github.com/stas00/ml-engineering/tree/master/compute) · [Accelerators](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md).\n\n- **The number that matters is achievable, not theoretical.** Real matmul FLOPS are well below spec. Measure Maximum Achievable Matmul FLOPS with [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py); compute **MFU** (model FLOPS utilization) to compare setups. See [the most important thing to understand](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#the-most-important-thing-to-understand) and [what accelerator characteristics we care for](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#what-accelerator-characteristics-do-we-care-for).\n- **Memory capacity + bandwidth often bind before FLOPS.** Check the [accelerator memory size and speed table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#accelerator-memory-size-and-speed) and the [TFLOPS comparison table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table) when choosing hardware.\n- **Don't forget power and cooling** - throttling silently caps sustained throughput. See [power and cooling](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#power-and-cooling).\n- **CPU and CPU memory matter too** (dataloading, offload, pinned memory). See [CPU](https://github.com/stas00/ml-engineering/tree/master/compute/cpu) and [CPU memory](https://github.com/stas00/ml-engineering/tree/master/compute/cpu-memory).\n\n## Storage (IO)\n\nFull chapter: [Storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md).\n\n- **Pick the file system for the job:** distributed/parallel FS for shared checkpoints and datasets; fast **local** NVMe beats network storage for scratch/hot data. See [which file system to choose](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#which-file-system-to-choose) and [local storage beats cloud storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#local-storage-beats-cloud-storage).\n- **Benchmark IO before you trust it** (checkpoint save/load and dataset streaming are common stalls). See [storage benchmarks](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#benchmarks).\n- **Gotchas that bite at scale:** you often get less usable capacity than sold; some clouds put backups on the same partition; always keep [checksums](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#dont-forget-the-checksums). Clean up instead of overpaying - see [why pay for more storage](https://github.com/stas00/ml-engineering/blob/master/storage/README.md#why-pay-for-more-storage-when-you-can-easily-clean-it-up-instead).\n\n## Network\n\nFull chapter: [Network](https://github.com/stas00/ml-engineering/blob/master/network/README.md).\n\n- **Inter-node speed can dominate the whole training's speed.** For sharded/parallel training, slow inter-node links stall everyone. Understand why before scaling out: [why inter-node speed is hugely important](https://github.com/stas00/ml-engineering/blob/master/network/README.md#understanding-why-inter-node-network-speed-is-of-a-huge-importance).\n- **Know the two speeds:** [intra-node](https://github.com/stas00/ml-engineering/blob/master/network/README.md#intra-node-networking) (NVLink/PCIe) vs [inter-node](https://github.com/stas00/ml-engineering/blob/master/network/README.md#inter-node-networking) (InfiniBand/RoCE/Ethernet), and [RDMA](https://github.com/stas00/ml-engineering/blob/master/network/README.md#rdma-networking).\n- **Benchmark real collective throughput,** not marketing numbers, with [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) (far simpler than nccl-tests). Mind the [important nuances](https://github.com/stas00/ml-engineering/blob/master/network/README.md#important-nuances) (e.g. unidirectional vs bidirectional, payload size, busbw vs algbw).\n- **When collectives hang or crawl:** see [network debug](https://github.com/stas00/ml-engineering/tree/master/network/debug) and [NCCL performance debug](https://github.com/stas00/ml-engineering/blob/master/debug/nccl-performance-debug.md).\n\n## Orchestration & SLURM\n\nFull chapter: [Orchestration](https://github.com/stas00/ml-engineering/tree/master/orchestration) · [SLURM](https://github.com/stas00/ml-engineering/tree/master/orchestration/slurm) · [Kubernetes](https://github.com/stas00/ml-engineering/tree/master/orchestration/kubernetes).\n\n- **Verify the cluster before the big run:** every GPU on every node must talk to every other. Run [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) across all nodes first.\n- **SLURM day-to-day:** the [users cheatsheet](https://github.com/stas00/ml-engineering/blob/master/orchestration/slurm/users.md) covers `sbatch`/`srun`/`salloc`, job arrays, dependencies, and inspecting the queue; keep the allocation and re-`srun` for fast debug iterations.\n- **Launchers** (torchrun/accelerate/deepspeed under SLURM): see [launchers](https://github.com/stas00/ml-engineering/tree/master/orchestration/slurm/launchers/README.md).\n\n## Training: performance & memory\n\nFull chapter: [Performance](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md).\n\n- **Start from the checklist:** [how to improve speed and save memory](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#how-to-improve-speed-and-save-memory) enumerates the high-impact levers (parallelism choice, activation checkpointing/recomputation, offload, fused kernels, mixed precision).\n- **Fit the model:** know the [anatomy of memory usage](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#anatomy-of-models-memory-usage) (weights + grads + optimizer states + activations) and profile with [memory profiler tools](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#memory-profiler-tools).\n- **Free throughput wins:** keep tensor dims [divisible/aligned](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#vector-and-matrix-size-divisibility) for Tensor Cores, set [NUMA affinity](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#numa-affinity), stop the [DataLoader](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#dataloader) from starving the GPUs, try [`torch.compile`](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#torchcompile), and tame [automatic garbage collection](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#automatic-garbage-collection) jitter in synchronized runs.\n- **Model parallelism** (DP/TP/PP/sequence/ZeRO) concepts and trade-offs: [model parallelism](https://github.com/stas00/ml-engineering/tree/master/training/model-parallelism).\n\n## Training: stability (instabilities & NaN/Inf)\n\nFull chapter: [Instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md).\n\n- **Loss spikes/divergence are expected at scale.** Compare against public [training logbooks](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#learning-from-training-logbooks) - your symptom is probably documented with a known mitigation.\n- **Numerical hygiene:** sane [weight init / STD](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#std-init), watch for [numerical instabilities](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#numerical-instabilities), and bad [data-batch × parameter-state combinations](https://github.com/stas00/ml-engineering/blob/master/training/instabilities/README.md#bad-combination-of-data-batch-and-model-parameter-state).\n- **Catch NaN/Inf early:** [underflow/overflow detection](https://github.com/stas00/ml-engineering/blob/master/debug/underflow_overflow.md) and PyTorch [tensor debugging](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#debugging-tensors) (fp16/bf16 range issues, per-tensor min/max/NaN scans).\n- **Reproducibility** for isolating a divergence: [reproducibility](https://github.com/stas00/ml-engineering/tree/master/training/reproducibility).\n\n## Training: fault tolerance & checkpoints\n\nFull chapter: [Fault tolerance](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md) · [Checkpoints](https://github.com/stas00/ml-engineering/tree/master/training/checkpoints).\n\n- **Provision slack:** [always plan for more nodes than needed](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#always-plan-to-have-more-nodes-than-needed) so a dead node doesn't stop the run; [queue up multiple jobs](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#queue-up-multiple-training-jobs) for auto-continuation.\n- **Checkpoint often enough** that a crash costs minutes, not hours - see [frequent checkpoint saving](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#frequent-checkpoint-saving) and [prevention](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#prevention).\n- **Operator controls:** a [kill switch](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#kill-switch) to stop a doomed run cleanly and a [save switch](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#save-switch) to force a checkpoint on demand.\n- **Survive the cluster:** handle [forced job preemption](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#dealing-with-forced-job-preemption) and prefer [fixed over dynamic allocations](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#preferring-fixed-accelerator-allocations-to-dynamic-ones).\n\n## Inference\n\nFull chapter: [Inference](https://github.com/stas00/ml-engineering/blob/master/inference/README.md).\n\n- **Speak the metrics:** TTFT, TPOT/ITL, throughput vs latency, and how batching trades them off. See [key inference performance metrics](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#key-inference-performance-metrics) and [concepts](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#concepts).\n- **Size the memory:** weights + KV cache + activations - see [anatomy of model's memory usage](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#anatomy-of-models-memory-usage).\n- **Cut cold starts:** [speeding up model loading time](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#speeding-up-model-loading-time); pick a serving stack from [inference frameworks](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#inference-frameworks) and validate with [benchmarks](https://github.com/stas00/ml-engineering/blob/master/inference/README.md#benchmarks).\n\n## Debugging distributed / PyTorch at scale\n\nFull chapter: [Debugging](https://github.com/stas00/ml-engineering/tree/master/debug) · [PyTorch](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md).\n\n- **Iterate cheaply:** shrink to tiny models/tokenizers/datasets and [fast debug of PyTorch models](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#faster-debug-and-development-with-tiny-models-tokenizers-and-datasets).\n- **Multi-node/multi-GPU hang or deadlock** (the classic scale bug):\n  1. Rule out comms with [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py).\n  2. Dump every rank's Python stack at once with `py-spy`; ranks stuck at *different* collectives reveal the desync. See [diagnosing crashes, hangs and tracing execution](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#diagnosing-crashes-hangs-and-tracing-execution) and [hanging solutions](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-hanging-solutions.md).\n  3. Set `NCCL_DEBUG=INFO`; escalate to [NCCL performance debug](https://github.com/stas00/ml-engineering/blob/master/debug/nccl-performance-debug.md).\n- **Cryptic CUDA errors:** `CUDA_LAUNCH_BLOCKING=1` for a real traceback (CUDA is async). See [The Art of Debugging - PyTorch](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#dealing-with-async-cuda-bugs).\n- **OOM / memory:** [memory usage](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#memory-usage); fragmentation via `PYTORCH_ALLOC_CONF`.\n- **NaN/Inf:** [underflow and overflow detection](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#underflow-and-overflow-detection) and [debugging tensors](https://github.com/stas00/ml-engineering/blob/master/debug/pytorch.md#debugging-tensors).\n- **GPU-specific faults:** [troubleshooting NVIDIA GPUs](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/nvidia/debug.md).\n\n## Testing\n\nFull chapter: [Testing](https://github.com/stas00/ml-engineering/blob/master/testing/README.md).\n\n- **Run tests surgically** (select, parametrize, repeat, control output/parallelism): [running tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#running-tests).\n- **Write robust tests** (fixtures, temp dirs, RNG control for reproducibility, distributed tests): [writing tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#writing-tests).\n- **When a test misbehaves:** [debugging tests](https://github.com/stas00/ml-engineering/blob/master/testing/README.md#debugging-tests).\n\n## Key tools\n\n| Need | Tool |\n|---|---|\n| Verify all GPUs/nodes can talk & allocate | [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) |\n| Real network throughput (all-reduce busbw) | [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) |\n| Actual achievable matmul FLOPS of an accelerator | [`mamf-finder.py`](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/benchmarks/mamf-finder.py) |\n| Tiny models/tokenizers/datasets for fast iteration | [make-tiny guide](https://github.com/stas00/ml-engineering/blob/master/debug/make-tiny-models-tokenizers-datasets.md) |\n| Better `trace` for distributed hangs | [NicerTrace](https://github.com/stas00/ml-engineering/blob/master/debug/tools.md) |\n\n## Pick the fix by symptom\n\n| Symptom | Reach for |\n|---|---|\n| Low TFLOPS / MFU, \"GPUs feel idle\" | Find the bottleneck: `mamf-finder`, [performance checklist](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#how-to-improve-speed-and-save-memory), DataLoader, NUMA, dim divisibility |\n| Training OOM | Memory anatomy → activation checkpointing/offload/parallelism; profile; `PYTORCH_ALLOC_CONF` |\n| Slow steps but GPUs busy on comms | Benchmark network (`all_reduce_bench`), check intra/inter-node, NCCL settings |\n| Slow dataloading / GPU starvation | [DataLoader](https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md#dataloader), local NVMe, prefetch/workers |\n| Loss spike / divergence / NaN | Logbooks, init/STD, underflow-overflow detection, tensor scans |\n| Multi-node/GPU hang or deadlock | `torch-distributed-gpu-test.py` → `py-spy` all ranks → `NCCL_DEBUG=INFO` |\n| Node/GPU dies mid-run | Spare nodes, frequent checkpoints, auto-restart, kill/save switch |\n| Job keeps getting preempted | [forced preemption](https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md#dealing-with-forced-job-preemption), queue chained jobs |\n| Checkpoint save/load is slow | Benchmark storage, choose FS, local vs shared |\n| Choosing GPUs / cloud / storage | Comparison tables, MAMF, [choose a cloud provider](https://github.com/stas00/ml-engineering/blob/master/insights/how-to-choose-cloud-provider.md) |\n| Inference too slow / won't fit | Metrics (TTFT/TPOT), KV-cache memory, framework choice, model-load speedups |\n\n## Notes for AI agents\n\n- **Diagnose before optimizing.** Identify which resource (compute/memory/network/storage/dataloader) is the actual bottleneck with a measurement; don't tune blindly.\n- **Prefer measured numbers over spec sheets.** Use the provided benchmark scripts on the target hardware/software stack before recommending changes or purchases.\n- **Assume failures at scale.** For any long/large run, verify checkpointing, restart, spare capacity, and a kill switch exist before worrying about peak speed.\n- **Verify the cluster first.** Run the distributed connectivity test before blaming model code for a multi-node problem.\n- **Reuse the community's hard-won lessons.** Check the training logbooks for known instabilities and fixes before re-deriving them.\n- **Read the linked chapter section** before applying a recipe - each has worked examples, exact commands, caveats, and scripts.\n- For deep single-process/tool debugging (gdb, strace, py-spy, cProfile, core files), use the companion skill: [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).\n","category":"root","tokens":5477}]}