Repository: sharkdp/hyperfine

Stars: 27885

README.md Copy Chapter

hyperfine

A command-line benchmarking tool.

Demo: Benchmarking fd and
find:

!hyperfine

Sponsors Copy Section

A special thank you goes to our biggest <a href="doc/sponsors.md">sponsor</a>:<br>

<a href="https://www.warp.dev/hyperfine">
<img src="doc/sponsors/warp-logo.png" width="200" alt="Warp">
<br>
<strong>Warp, the intelligent terminal</strong>
<br>
_{Available on MacOS, Linux, Windows}
</a>

Features Copy Chapter

* Statistical analysis across multiple runs.
* Support for arbitrary shell commands.
* Constant feedback about the benchmark progress and current estimates.
* Warmup runs can be executed before the actual benchmark.
* Cache-clearing commands can be set up before each timing run.
* Statistical outlier detection to detect interference from other programs and caching effects.
* Export results to various formats: CSV, JSON, Markdown, AsciiDoc.
* Parameterized benchmarks (e.g. vary the number of threads).
* Cross-platform

Usage Copy Chapter

Basic benchmarks Copy Section

To run a benchmark, you can simply call hyperfine <command>.... The argument(s) can be any
shell command. For example:

hyperfine 'sleep 0.3'

Hyperfine will automatically determine the number of runs to perform for each command. By default,
it will perform at least 10 benchmarking runs and measure for at least 3 seconds. To change this,
you can use the -r/--runs option:

hyperfine --runs 5 'sleep 0.3'

If you want to compare the runtimes of different programs, you can pass multiple commands:

hyperfine 'hexdump file' 'xxd file'

Warmup runs and preparation commands Copy Section

For programs that perform a lot of disk I/O, the benchmarking results can be heavily influenced
by disk caches and whether they are cold or warm.

If you want to run the benchmark on a warm cache, you can use the -w/--warmup option to
perform a certain number of program executions before the actual benchmark:

hyperfine --warmup 3 'grep -R TODO *'

Conversely, if you want to run the benchmark for a cold cache, you can use the -p/--prepare
option to run a special command before each timing run. For example, to clear harddisk caches
on Linux, you can run

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'grep -R TODO *'

Parameterized benchmarks Copy Section

If you want to run a series of benchmarks where a single parameter is varied (say, the number of
threads), you can use the -P/--parameter-scan option and call:

hyperfine --prepare 'make clean' --parameter-scan num_threads 1 12 'make -j {num_threads}'

hyperfine --parameter-scan delay 0.3 0.7 -D 0.2 'sleep {delay}'

For non-numeric parameters, you can also supply a list of values with the -L/--parameter-list
option:

hyperfine -L compiler gcc,clang '{compiler} -O2 main.cpp'

Intermediate shell Copy Section

By default, commands are executed using a predefined shell (/bin/sh on Unix, cmd.exe on Windows).
If you want to use a different shell, you can use the -S, --shell <SHELL> option:

hyperfine --shell zsh 'for i in {1..10000}; do echo test; done'

Note that hyperfine always corrects for the shell spawning time. To do this, it performs a calibration
procedure where it runs the shell with an empty command (multiple times), to measure the startup time
of the shell. It will then subtract this time from the total to show the actual time used by the command
in question.

If you want to run a benchmark without an intermediate shell, you can use the -N or --shell=none
option. This is helpful for very fast commands (< 5 ms) where the shell startup overhead correction would
produce a significant amount of noise. Note that you cannot use shell syntax like * or ~ in this case.

hyperfine -N 'grep TODO /home/user'

Shell functions and aliases Copy Section

If you are using bash, you can export shell functions to directly benchmark them with hyperfine:

my_function() { sleep 1; }
export -f my_function
hyperfine --shell=bash my_function

Otherwise, inline them into or source them from the benchmarked program:

hyperfine 'my_function() { sleep 1; }; my_function'
echo 'alias my_alias="sleep 1"' > /tmp/my_alias.sh
hyperfine '. /tmp/my_alias.sh; my_alias'

Exporting results Copy Section

Hyperfine has multiple options for exporting benchmark results to CSV, JSON, Markdown and other
formats (see --help text for details).

#### Markdown

You can use the --export-markdown <file> option to create tables like the following:

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| find . -iregex '.*[0-9]\.jpg$' | 2.275 ± 0.046 | 2.243 | 2.397 | 9.79 ± 0.22 |
| find . -iname '*[0-9].jpg' | 1.427 ± 0.026 | 1.405 | 1.468 | 6.14 ± 0.13 |
| fd -HI '.*[0-9]\.jpg$' | 0.232 ± 0.002 | 0.230 | 0.236 | 1.00 |

#### JSON

The JSON output is useful if you want to analyze the benchmark results in more detail. The
scripts/ folder includes a lot
of helpful Python programs to further analyze benchmark results and create helpful
visualizations, like a histogram of runtimes or a whisker plot to compare
multiple benchmarks:

|  |  |
|---:|---:|

Detailed benchmark flowchart Copy Section

The following chart explains the execution order of various timing runs when using options
like --warmup, --prepare <cmd>, --setup <cmd> or --cleanup <cmd>:


Installation Copy Chapter


On Ubuntu Copy Section

On Ubuntu, hyperfine can be installed from the official repositories:

apt install hyperfine

Alternatively, for the latest version, you can download the appropriate .deb package from the Release page and install it via dpkg:

wget https://github.com/sharkdp/hyperfine/releases/download/v1.20.0/hyperfine_1.20.0_amd64.deb
sudo dpkg -i hyperfine_1.20.0_amd64.deb

On Fedora Copy Section

On Fedora, hyperfine can be installed from the official repositories:

dnf install hyperfine

On Alpine Linux Copy Section

On Alpine Linux, hyperfine can be installed from the official repositories:

apk add hyperfine

On Arch Linux Copy Section

On Arch Linux, hyperfine can be installed from the official repositories:

pacman -S hyperfine

On Debian Linux Copy Section

On Debian Linux, hyperfine can be installed from the official repositories:

apt install hyperfine

On Exherbo Linux Copy Section

On Exherbo Linux, hyperfine can be installed from the rust repositories:

cave resolve -x repository/rust
cave resolve -x hyperfine

On Funtoo Linux Copy Section

On Funtoo Linux, hyperfine can be installed from core-kit:

emerge app-benchmarks/hyperfine

On NixOS Copy Section

On NixOS, hyperfine can be installed from the official repositories:

nix-env -i hyperfine

On Flox Copy Section

On Flox, hyperfine can be installed as follows.

flox install hyperfine

On openSUSE Copy Section

On openSUSE, hyperfine can be installed from the official repositories:

zypper install hyperfine

On Void Linux Copy Section

Hyperfine can be installed via xbps

xbps-install -S hyperfine

On macOS Copy Section

Hyperfine can be installed via Homebrew:

brew install hyperfine

Or you can install using MacPorts:

sudo port selfupdate
sudo port install hyperfine

On FreeBSD Copy Section

Hyperfine can be installed via pkg:

pkg install hyperfine

On OpenBSD Copy Section

doas pkg_add hyperfine

On Windows Copy Section

Hyperfine can be installed via Chocolatey, Scoop, or Winget:

choco install hyperfine

scoop install hyperfine

winget install hyperfine

With conda Copy Section

Hyperfine can be installed via conda from the conda-forge channel:

conda install -c conda-forge hyperfine

With cargo (Linux, macOS, Windows) Copy Section

Hyperfine can be installed from source via cargo:

cargo install --locked hyperfine

Make sure that you use Rust 1.76 or newer.

From binaries (Linux, macOS, Windows) Copy Section

Download the corresponding archive from the Release page.

Alternative tools Copy Chapter

Hyperfine is inspired by bench.

Integration with other tools Copy Chapter

Chronologer is a tool that uses hyperfine to
visualize changes in benchmark timings across your Git history.

Bencher is a continuous benchmarking tool that supports hyperfine to
track benchmarks and catch performance regressions in CI.

Drop hyperfine JSON outputs onto the Venz chart to visualize the results,
and manage hyperfine configurations.

Make sure to check out the scripts folder
in this repository for a set of tools to work with hyperfine benchmark results.

Origin of the name Copy Chapter

The name hyperfine was chosen in reference to the hyperfine levels of caesium 133 which play a crucial role in the
definition of our base unit of time
— the second.

Citing hyperfine Copy Chapter

Thank you for considering to cite hyperfine in your research work. Please see the information
in the sidebar on how to properly cite hyperfine.

License Copy Chapter

hyperfine is dual-licensed under the terms of the MIT License and the Apache License 2.0.

See the LICENSE-APACHE and LICENSE-MIT files for details.