benchmark Documentation

Index Copy Section

Benchmark

• Assembly Tests
• Bazel
• Dependencies
• Perf Counters
• Platform Specific Build Instructions
• Python Bindings
• Random Interleaving
• Reducing Variance
• Releasing
• Tools
• User Guide

AssemblyTests Copy Section

Assembly Tests

The Benchmark library provides a number of functions whose primary
purpose in to affect assembly generation, including DoNotOptimize
and ClobberMemory. In addition there are other functions,
such as KeepRunning, for which generating good assembly is paramount.

For these functions it's important to have tests that verify the
correctness and quality of the implementation. This requires testing
the code generated by the compiler.

This document describes how the Benchmark library tests compiler output,
as well as how to properly write new tests.

Anatomy of a Test Copy Chapter

Writing a test has two steps:

• Write the code you want to generate assembly for.
• Add // CHECK lines to match against the verified assembly.

Example:

// CHECK-LABEL: test_add:
extern "C" int test_add() {
    extern int ExternInt;
    return ExternInt + 1;

    // CHECK: movl ExternInt(%rip), %eax
    // CHECK: addl %eax
    // CHECK: ret
}

LLVM Filecheck

LLVM's Filecheck
is used to test the generated assembly against the // CHECK lines
specified in the tests source file. Please see the documentation
linked above for information on how to write CHECK directives.

Tips and Tricks:

• Tests should match the minimal amount of output required to establish
  correctness. CHECK directives don't have to match on the exact next line
  after the previous match, so tests should omit checks for unimportant
  bits of assembly. (CHECK-NEXT
  can be used to ensure a match occurs exactly after the previous match).

• The tests are compiled with -O3 -g0. So we're only testing the
  optimized output.

• The assembly output is further cleaned up using tools/strip_asm.py.
  This removes comments, assembler directives, and unused labels before
  the test is run.

• The generated and stripped assembly file for a test is output under
  /test/<test-name>.s

• Filecheck supports using CHECK prefixes
  to specify lines that should only match in certain situations.
  The Benchmark tests use CHECK-CLANG and CHECK-GNU for lines that
  are only expected to match Clang or GCC's output respectively. Normal
  CHECK lines match against all compilers. (Note: CHECK-NOT and
  CHECK-LABEL are NOT prefixes. They are versions of non-prefixed
  CHECK lines)

• Use extern "C" to disable name mangling for specific functions. This
  makes them easier to name in the CHECK lines.

Problems Writing Portable Tests Copy Chapter

Writing tests which check the code generated by a compiler are
inherently non-portable. Different compilers and even different compiler
versions may generate entirely different code. The Benchmark tests
must tolerate this.

LLVM Filecheck provides a number of mechanisms to help write
"more portable" tests; including matching using regular expressions,
allowing the creation of named variables
for later matching, and checking non-sequential matches.

Capturing Variables

For example, say GCC stores a variable in a register but Clang stores
it in memory. To write a test that tolerates both cases we "capture"
the destination of the store, and then use the captured expression
to write the remainder of the test.

// CHECK-LABEL: test_div_no_op_into_shr:
extern "C" void test_div_no_op_into_shr(int value) {
    int divisor = 2;
    benchmark::DoNotOptimize(divisor); // hide the value from the optimizer
    return value / divisor;

    // CHECK: movl $2, [[DEST:.*]]
    // CHECK: idivl [[DEST]]
    // CHECK: ret
}

Using Regular Expressions to Match Differing Output

Often tests require testing assembly lines which may subtly differ
between compilers or compiler versions. A common example of this
is matching stack frame addresses. In this case regular expressions
can be used to match the differing bits of output. For example:

int ExternInt;
struct Point { int x, y, z; };

// CHECK-LABEL: test_store_point:
extern "C" void test_store_point() {
    Point p{ExternInt, ExternInt, ExternInt};
    benchmark::DoNotOptimize(p);

    // CHECK: movl ExternInt(%rip), %eax
    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
    // CHECK: ret
}

Current Requirements and Limitations Copy Chapter

The tests require Filecheck to be installed along the PATH of the
build machine. Otherwise the tests will be disabled.

Additionally, as mentioned in the previous section, codegen tests are
inherently non-portable. Currently the tests are limited to:

• x86_64 targets.
• Compiled with GCC or Clang

Further work could be done, at least on a limited basis, to extend the
tests to other architectures and compilers (using CHECK prefixes).

Furthermore, the tests fail for builds which specify additional flags
that modify code generation, including --coverage or -fsanitize=.

Bazel Copy Section

Bazel

Google Benchmark provides Bazel targets for both the benchmark library and the
optional default main function:

• @google_benchmark//:benchmark provides the benchmark library.
• @google_benchmark//:benchmark_main provides the default main function and
  depends on @google_benchmark//:benchmark.

Use @google_benchmark//:benchmark when the benchmark target defines its own
main function, including through BENCHMARK_MAIN(). Use
@google_benchmark//:benchmark_main when the benchmark target should use the
default Google Benchmark entry point.

Bzlmod Copy Chapter

With Bzlmod enabled, add Google Benchmark to your MODULE.bazel file:

bazel_dep(name = "google_benchmark", version = "<VERSION>")

Replace <VERSION> with the Google Benchmark release version you want to use.

Then depend on the Bazel target from a cc_binary or cc_test:

load("@rules_cc//cc:defs.bzl", "cc_binary")

cc_binary(
    name = "string_benchmark",
    srcs = ["string_benchmark.cc"],
    deps = ["@google_benchmark//:benchmark_main"],
)

The source file should register benchmarks, but it should not call
BENCHMARK_MAIN() when linking against @google_benchmark//:benchmark_main:

#include <benchmark/benchmark.h>
#include <string>

static void BM_StringCreation(benchmark::State& state) {
  for (auto _ : state) {
    std::string empty_string;
  }
}
BENCHMARK(BM_StringCreation);

Run the benchmark with Bazel:

bazel run //:string_benchmark

Pass Google Benchmark flags after Bazel's -- separator:

bazel run //:string_benchmark -- --benchmark_filter=StringCreation

WORKSPACE Copy Chapter

Projects that still use WORKSPACE can declare Google Benchmark as an external
repository and load its dependencies from bazel/benchmark_deps.bzl:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "google_benchmark",
    strip_prefix = "benchmark-<VERSION>",
    urls = ["https://github.com/google/benchmark/archive/refs/tags/v<VERSION>.tar.gz"],
    # Add sha256 for reproducible builds.
)

load("@google_benchmark//:bazel/benchmark_deps.bzl", "benchmark_deps")

benchmark_deps()

Use the same <VERSION> value without the leading v; the archive URL adds the tag prefix explicitly.

After declaring the repository, use the same target labels shown above:
@google_benchmark//:benchmark or @google_benchmark//:benchmark_main.

Perf Counters Copy Chapter

When using Bazel, enable libpfm support by adding --define pfm=1 to the build
or run command. See Perf Counters for more details.

Dependencies Copy Section

Build tool dependency policy

We follow the Foundational C++ support policy for our build tools. In
particular the "Build Systems" section.

CMake Copy Chapter

The current supported version is CMake 3.13 as of 2024-10-24. Most modern
distributions include newer versions, for example:

• Ubuntu 20.04 provides CMake 3.16.3
• Debian 11.4 provides CMake 3.18.4
• Ubuntu 22.04 provides CMake 3.22.1

Python Copy Chapter

The Python bindings require Python 3.10+ as of v1.9.0 (2024-08-16) for installation from PyPI.
Building from source for older versions probably still works, though. See the user guide for details on how to build from source.
The minimum theoretically supported version is Python 3.8, since the used bindings generator (nanobind) only supports Python 3.8+.

Perf Counters Copy Section

• The benchmark is run on an architecture featuring a Performance Monitoring
  Unit (PMU),
• The benchmark is compiled with support for collecting counters. Currently,
  this requires libpfm, which is built as a
  dependency via Bazel.

The feature does not require modifying benchmark code. Counter collection is
handled at the boundaries where timer collection is also handled.

To opt-in:

• If using a Bazel build, add --define pfm=1 to your build flags
• If using CMake:
  • Install libpfm4-dev, e.g. apt-get install libpfm4-dev.
  • Enable the CMake flag BENCHMARK_ENABLE_LIBPFM in CMakeLists.txt.

To use, pass a comma-separated list of counter names through the
--benchmark_perf_counters flag. The names are decoded through libpfm - meaning,
they are platform specific, but some (e.g. CYCLES or INSTRUCTIONS) are
mapped by libpfm to platform-specifics - see libpfm
documentation for more details.

The counter values are reported back through the User Counters
mechanism, meaning, they are available in all the formats (e.g. JSON) supported
by User Counters.

Platform Specific Build Instructions Copy Section

Platform Specific Build Instructions

Building with GCC Copy Chapter

When the library is built using GCC it is necessary to link with the pthread
library due to how GCC implements std::thread. Failing to link to pthread will
lead to runtime exceptions (unless you're using libc++), not linker errors. See
issue #67 for more details. You
can link to pthread by adding -pthread to your linker command. Note, you can
also use -lpthread, but there are potential issues with ordering of command
line parameters if you use that.

On QNX, the pthread library is part of libc and usually included automatically
(see
pthread_create()).
There's no separate pthread library to link.

Building with Visual Studio 2015, 2017 or 2022 Copy Chapter

The shlwapi library (-lshlwapi) is required to support a call to CPUInfo which reads the registry. Either add shlwapi.lib under [ Configuration Properties > Linker > Input ], or use the following:

// Alternatively, can add libraries using linker options.

// First, Add the path to the generated library files (directory containing the `benchmark.lib`) in `[Configuration Properties > Linker > General > Additional Library Directories]`. Then do the following:
#ifdef _WIN32
#pragma comment ( lib, "Shlwapi.lib" )
#ifdef _DEBUG
#pragma comment ( lib, "benchmark.lib" )
#else
#pragma comment ( lib, "benchmark.lib" )
#endif
#endif

When using the static library, make sure to add BENCHMARK_STATIC_DEFINE under [Configuration Properties > C/C++ > Preprocessor > Preprocessor Definitions]

Can also use the graphical version of CMake:

• Open CMake GUI.
• Under Where to build the binaries, same path as source plus build.
• Under CMAKE_INSTALL_PREFIX, same path as source plus install.
• Click Configure, Generate, Open Project.
• If build fails, try deleting entire directory and starting again, or unticking options to build less.

Building with Intel 2015 Update 1 or Intel System Studio Update 4 Copy Chapter

See instructions for building with Visual Studio. Once built, right click on the solution and change the build to Intel.

Building on Solaris Copy Chapter

If you're running benchmarks on solaris, you'll want the kstat library linked in
too (-lkstat).

Python Bindings Copy Section

Building and installing Python bindings

Python bindings are available as wheels on PyPI for importing and
using Google Benchmark directly in Python.
Currently, pre-built wheels exist for macOS (both ARM64 and Intel x86), Linux x86-64 and 64-bit Windows.
Supported Python versions are Python 3.8 - 3.12.

To install Google Benchmark's Python bindings, run:

python -m pip install --upgrade pip  # for manylinux2014 support
python -m pip install google-benchmark

In order to keep your system Python interpreter clean, it is advisable to run these commands in a virtual
environment. See the official Python documentation
on how to create virtual environments.

To build a wheel directly from source, you can follow these steps:

git clone https://github.com/google/benchmark.git
cd benchmark
# create a virtual environment and activate it
python3 -m venv venv --system-site-packages
source venv/bin/activate  # .\venv\Scripts\Activate.ps1 on Windows

# upgrade Python's system-wide packages
python -m pip install --upgrade pip build
# builds the wheel and stores it in the directory "dist".
python -m build

NB: Building wheels from source requires Bazel. For platform-specific instructions on how to install Bazel,
refer to the Bazel installation docs.

Random Interleaving Copy Section

Random Interleaving is a
technique to lower run-to-run variance. It randomly interleaves repetitions of a
microbenchmark with repetitions from other microbenchmarks in the same benchmark
test. Data shows it is able to lower run-to-run variance by
40% on average.

To use, you mainly need to set --benchmark_enable_random_interleaving=true,
and optionally specify non-zero repetition count --benchmark_repetitions=9
and optionally decrease the per-repetition time --benchmark_min_time=0.1.

Reducing Variance Copy Section

Reducing Variance

***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.

sudo cpupower frequency-set --governor performance

cpupower frequency-info -o proc

***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.

benchmark::MaybeReenterWithoutASLR(argc, argv);

echo 0 > /proc/sys/kernel/randomize_va_space

setarch `uname -m` -R ./a_benchmark

The Linux CPU frequency governor discussed
above is not the only source
of noise in benchmarks. Some, but not all, of the sources of variance
include:

1. On multi-core machines not all CPUs/CPU cores/CPU threads run the same
   speed, so running a benchmark one time and then again may give a
   different result depending on which CPU it ran on.
2. CPU scaling features that run on the CPU, like Intel's Turbo Boost and
   AMD Turbo Core and Precision Boost, can temporarily change the CPU
   frequency even when the using the "performance" governor on Linux.
3. Context switching between CPUs, or scheduling competition on the CPU the
   benchmark is running on.
4. Intel Hyperthreading or AMD SMT causing the same issue as above.
5. Cache effects caused by code running on other CPUs.
6. Non-uniform memory architectures (NUMA).

These can cause variance in benchmarks results within a single run
(--benchmark_repetitions=N) or across multiple runs of the benchmark
program.

Reducing sources of variance is OS and architecture dependent, which is one
reason some companies maintain machines dedicated to performance testing.

Some of the easier and effective ways of reducing variance on a typical
Linux workstation are:

1. Use the performance governor as discussed
   above.
2. Disable processor boosting by:
   sh Copy

   echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost

   See the Linux kernel's
   boost.txt
   for more information.
3. Set the benchmark program's task affinity to a fixed cpu. For example:
   sh Copy

   taskset -c 0 ./mybenchmark

4. Increase the program's scheduling priority to minimize context switches using nice or chrt:
   sh Copy

   sudo nice -n -20 ./mybenchmark
   sudo chrt -f 80 ./mybenchmark

5. Disabling Hyperthreading/SMT. This can be done in the Bios or using the
   /sys file system (see the LLVM project's Benchmarking
   tips).
6. Close other programs that do non-trivial things based on timers, such as
   your web browser, desktop environment, etc.
7. Reduce the working set of your benchmark to fit within the L1 cache, but
   do be aware that this may lead you to optimize for an unrealistic
   situation.

Further resources on this topic:

1. The LLVM project's Benchmarking
   tips.
2. The Arch Wiki Cpu frequency
   scaling page.

Releasing Copy Section

How to release

• Make sure you're on main and synced to HEAD
• Ensure the project builds and tests run
  • parallel -j0 exec ::: test/*_test can help ensure everything at least
    passes
• Prepare release notes
  • git log $(git describe --abbrev=0 --tags)..HEAD gives you the list of
    commits between the last annotated tag and HEAD
  • Pick the most interesting.
• Create one last commit that updates the version saved in CMakeLists.txt, MODULE.bazel,
  bindings/python/google_benchmark/__init__.py, and bindings/rust/Cargo.toml to the release version you're creating.
  (This version will be used if benchmark is installed from the archive you'll be creating
  in the next step.)

# CMakeLists.txt
project (benchmark VERSION 1.9.0 LANGUAGES CXX)

# MODULE.bazel
module(name = "com_github_google_benchmark", version="1.9.0")

# google_benchmark/__init__.py
__version__ = "1.9.0"

# bindings/rust/Cargo.toml
version = "1.9.0"

• Create a release through github's interface
  • Note this will create a lightweight tag.
  • Update this to an annotated tag:
    • git pull --tags
    • git tag -a -f <tag> <tag>
    • git push --force --tags origin
• Confirm that the "Build and upload Python wheels" action runs to completion
  • Run it manually if it hasn't run.

Tools Copy Section

Benchmark Tools

compare.py Copy Chapter

The compare.py can be used to compare the result of benchmarks.

Dependencies Copy Section

The utility relies on the scipy package which can be installed using pip:

pip3 install -r requirements.txt

Displaying aggregates only Copy Section

The switch -a / --display_aggregates_only can be used to control the
displayment of the normal iterations vs the aggregates. When passed, it will
be passthrough to the benchmark binaries to be run, and will be accounted for
in the tool itself; only the aggregates will be displayed, but not normal runs.
It only affects the display, the separate runs will still be used to calculate
the U test.

Modes of operation Copy Section

There are three modes of operation:

1. Just compare two benchmarks
   The program is invoked like:

$ compare.py benchmarks <benchmark_baseline> <benchmark_contender> [benchmark options]...

Where and either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file.

[benchmark options] will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal --benchmark_* parameters, or some custom parameters your binary takes.

Example output:

/* Detailed source-code truncated for AI context efficiency. */

What it does is for the every benchmark from the first run it looks for the benchmark with exactly the same name in the second run, and then compares the results. If the names differ, the benchmark is omitted from the diff.
As you can note, the values in Time and CPU columns are calculated as (new - old) / |old|.

2. Compare two different filters of one benchmark
   The program is invoked like:

$ compare.py filters <benchmark> <filter_baseline> <filter_contender> [benchmark options]...

Where `` either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file.

Where <filter_baseline> and <filter_contender> are the same regex filters that you would pass to the [--benchmark_filter=<regex>] parameter of the benchmark binary.

[benchmark options] will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal --benchmark_* parameters, or some custom parameters your binary takes.

Example output:

/* Detailed source-code truncated for AI context efficiency. */

As you can see, it applies filter to the benchmarks, both when running the benchmark, and before doing the diff. And to make the diff work, the matches are replaced with some common string. Thus, you can compare two different benchmark families within one benchmark binary.
As you can note, the values in Time and CPU columns are calculated as (new - old) / |old|.

3. Compare filter one from benchmark one to filter two from benchmark two:
   The program is invoked like:

$ compare.py filters <benchmark_baseline> <filter_baseline> <benchmark_contender> <filter_contender> [benchmark options]...

Where and either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file.

Where <filter_baseline> and <filter_contender> are the same regex filters that you would pass to the [--benchmark_filter=<regex>] parameter of the benchmark binary.

[benchmark options] will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal --benchmark_* parameters, or some custom parameters your binary takes.

Example output:

/* Detailed source-code truncated for AI context efficiency. */

This is a mix of the previous two modes, two (potentially different) benchmark binaries are run, and a different filter is applied to each one.
As you can note, the values in Time and CPU columns are calculated as (new - old) / |old|.

Note: Interpreting the output Copy Section

Performance measurements are an art, and performance comparisons are doubly so.
Results are often noisy and don't necessarily have large absolute differences to
them, so just by visual inspection, it is not at all apparent if two
measurements are actually showing a performance change or not. It is even more
confusing with multiple benchmark repetitions.

Thankfully, what we can do, is use statistical tests on the results to determine
whether the performance has statistically-significantly changed. compare.py
uses Mann–Whitney U
test, with a null
hypothesis being that there's no difference in performance.

The below output is a summary of a benchmark comparison with statistics
provided for a multi-threaded process.

Benchmark                                               Time        CPU    Time Old      Time New       CPU Old       CPU New
-----------------------------------------------------------------------------------------------------------------------------
benchmark/threads:1/process_time/real_time_pvalue     0.0000     0.0000    U Test, Repetitions: 27 vs 27
benchmark/threads:1/process_time/real_time_mean      -0.1442    -0.1442          90            77            90            77
benchmark/threads:1/process_time/real_time_median    -0.1444    -0.1444          90            77            90            77
benchmark/threads:1/process_time/real_time_stddev    +0.3974    +0.3933           0             0             0             0
benchmark/threads:1/process_time/real_time_cv        +0.6329    +0.6280           0             0             0             0
OVERALL_GEOMEAN                                      -0.1442    -0.1442           0             0             0             0

Here's a breakdown of each row:

benchmark/threads:1/process_time/real_time_pvalue: This shows the p-value for
the statistical test comparing the performance of the process running with one
thread. A value of 0.0000 suggests a statistically significant difference in
performance. The comparison was conducted using the U Test (Mann-Whitney
U Test) with 27 repetitions for each case.

benchmark/threads:1/process_time/real_time_mean: This shows the relative
difference in mean execution time between two different cases. The negative
value (-0.1442) implies that the new process is faster by about 14.42%. The old
time was 90 units, while the new time is 77 units.

benchmark/threads:1/process_time/real_time_median: Similarly, this shows the
relative difference in the median execution time. Again, the new process is
faster by 14.44%.

benchmark/threads:1/process_time/real_time_stddev: This is the relative
difference in the standard deviation of the execution time, which is a measure
of how much variation or dispersion there is from the mean. A positive value
(+0.3974) implies there is more variance in the execution time in the new
process.

benchmark/threads:1/process_time/real_time_cv: CV stands for Coefficient of
Variation. It is the ratio of the standard deviation to the mean. It provides a
standardized measure of dispersion. An increase (+0.6329) indicates more
relative variability in the new process.

OVERALL_GEOMEAN: Geomean stands for geometric mean, a type of average that is
less influenced by outliers. The negative value indicates a general improvement
in the new process. However, given the values are all zero for the old and new
times, this seems to be a mistake or placeholder in the output.

Let's first try to see what the different columns represent in the above
compare.py benchmarking output:

1. Benchmark: The name of the function being benchmarked, along with the
   size of the input (after the slash).

2. Time: The average time per operation, across all iterations.

3. CPU: The average CPU time per operation, across all iterations.

4. Iterations: The number of iterations the benchmark was run to get a
   stable estimate.

5. Time Old and Time New: These represent the average time it takes for a
   function to run in two different scenarios or versions. For example, you
   might be comparing how fast a function runs before and after you make some
   changes to it.

6. CPU Old and CPU New: These show the average amount of CPU time that the
   function uses in two different scenarios or versions. This is similar to
   Time Old and Time New, but focuses on CPU usage instead of overall time.

In the comparison section, the relative differences in both time and CPU time
are displayed for each input size.

A statistically-significant difference is determined by a p-value, which is
a measure of the probability that the observed difference could have occurred
just by random chance. A smaller p-value indicates stronger evidence against the
null hypothesis.

Therefore:

1. If the p-value is less than the chosen significance level (alpha), we
   reject the null hypothesis and conclude the benchmarks are significantly
   different.
2. If the p-value is greater than or equal to alpha, we fail to reject the
   null hypothesis and treat the two benchmarks as similar.

The result of said the statistical test is additionally communicated through color coding:

+ Green:

The benchmarks are statistically different. This could mean the
performance has either significantly improved or significantly
deteriorated. You should look at the actual performance numbers to see which
is the case.

- Red:

The benchmarks are statistically similar. This means the performance
hasn't significantly changed.

In statistical terms, 'green' means we reject the null hypothesis that
there's no difference in performance, and 'red' means we fail to reject the
null hypothesis. This might seem counter-intuitive if you're expecting 'green'
to mean 'improved performance' and 'red' to mean 'worsened performance'.

But remember, in this context:

    'Success' means 'successfully finding a difference'.
    'Failure' means 'failing to find a difference'.

Also, please note that even if we determine that there is a
statistically-significant difference between the two measurements, it does not
necessarily mean that the actual benchmarks that were measured are
different, or vice versa, even if we determine that there is no
statistically-significant difference between the two measurements, it does not
necessarily mean that the actual benchmarks that were measured are not
different.

U test Copy Section

If there is a sufficient repetition count of the benchmarks, the tool can do
a U Test, of the
null hypothesis that it is equally likely that a randomly selected value from
one sample will be less than or greater than a randomly selected value from a
second sample.

If the calculated p-value is below this value is lower than the significance
level alpha, then the result is said to be statistically significant and the
null hypothesis is rejected. Which in other words means that the two benchmarks
aren't identical.

WARNING: requires LARGE (no less than 9) number of repetitions to be
meaningful!

User Guide Copy Section

User Guide

Command Line Copy Chapter

Command Line Options

Output Formats

Output Files

Running Benchmarks

Running a Subset of Benchmarks

Result Comparison

Extra Context

Library Copy Chapter

Runtime and Reporting Considerations

Setup/Teardown

Passing Arguments

Custom Benchmark Name

Calculating Asymptotic Complexity

Templated Benchmarks

Templated Benchmarks that take arguments

Fixtures

Custom Counters

Multithreaded Benchmarks

CPU Timers

Manual Timing

Setting the Time Unit

Random Interleaving

User-Requested Performance Counters

Preventing Optimization

Reporting Statistics

Custom Statistics

Memory Usage

Using RegisterBenchmark

Exiting with an Error

A Faster KeepRunning Loop

Benchmarking Tips Copy Chapter

Disabling CPU Frequency Scaling

Reducing Variance in Benchmarks

$ ./benchmark --benchmark_list_tests
BM_SomeFunction
BM_AnotherFunction

$ ./benchmark --benchmark_filter=BM_memcpy/32

$ ./benchmark --benchmark_dry_run

If set, enable random interleaving of repetitions of all benchmarks. This can help reduce the impact of system state changes on benchmark results. See GitHub issue #1051 for details.

Example:

$ ./benchmark --benchmark_enable_random_interleaving

Timing and Repetition Control Copy Section

`--benchmark_min_time=` (BENCHMARK_MIN_TIME)

Specifies the minimum amount of time (in seconds) that each benchmark should run. For CPU-time based tests, this is the lower bound on the total CPU time used by all threads that make up the test. For real-time based tests, this is the lower bound on the elapsed time of the benchmark execution, regardless of number of threads.

Default: 0.5 seconds

Example:

$ ./benchmark --benchmark_min_time=1.0

`--benchmark_min_warmup_time=` (BENCHMARK_MIN_WARMUP_TIME)

Minimum number of seconds a benchmark should be run before results should be taken into account. This can be necessary for benchmarks of code which needs to fill some form of cache before performance is of interest. Results gathered within this period are discarded and not used for the reported result.

Default: 0.0 seconds

Example:

$ ./benchmark --benchmark_min_warmup_time=0.5

`--benchmark_repetitions=` (BENCHMARK_REPETITIONS)

The number of runs of each benchmark. If greater than 1, the mean and standard deviation of the runs will be reported.

Default: 1

Example:

$ ./benchmark --benchmark_repetitions=5

Output Formatting Copy Section

`--benchmark_format=` (BENCHMARK_FORMAT)

The format to use for console output. Valid values are 'console', 'json', or 'csv'. See Output Formats for more details.

Default: console

Example:

$ ./benchmark --benchmark_format=json

`--benchmark_out=` (BENCHMARK_OUT)

The file to write additional output to. The output format is controlled by --benchmark_out_format. Specifying this option does not suppress console output.

Example:

$ ./benchmark --benchmark_out=results.json

`--benchmark_out_format=` (BENCHMARK_OUT_FORMAT)

The format to use for file output specified by --benchmark_out. Valid values are 'console', 'json', or 'csv'.

Default: json

Example:

$ ./benchmark --benchmark_out=results.csv --benchmark_out_format=csv

`--benchmark_color=` (BENCHMARK_COLOR)

Whether to use colors in the output. Valid values are 'true'/'yes'/1, 'false'/'no'/0, and 'auto'. 'auto' means to use colors if the output is being sent to a terminal and the TERM environment variable is set to a terminal type that supports colors.

Default: auto

Example:

$ ./benchmark --benchmark_color=false

`--benchmark_time_unit=` (BENCHMARK_TIME_UNIT)

Set the default time unit to use for reports. Valid values are 'ns' (nanoseconds), 'us' (microseconds), 'ms' (milliseconds), or 's' (seconds).

Default: (empty, uses automatic selection)

Example:

$ ./benchmark --benchmark_time_unit=us

Reporting Options Copy Section

`--benchmark_report_aggregates_only` (BENCHMARK_REPORT_AGGREGATES_ONLY)

When enabled, only the mean, standard deviation, and other statistics are reported for repeated benchmarks. This affects all reporters (both console and file output).

Default: false

Example:

$ ./benchmark --benchmark_repetitions=5 --benchmark_report_aggregates_only

`--benchmark_display_aggregates_only` (BENCHMARK_DISPLAY_AGGREGATES_ONLY)

When enabled, only the mean, standard deviation, and other statistics are displayed for repeated benchmarks. Unlike --benchmark_report_aggregates_only, this only affects the display (console) reporter, not the file reporter, which will still contain all output.

Default: false

Example:

$ ./benchmark --benchmark_repetitions=5 --benchmark_display_aggregates_only

`--benchmark_counters_tabular` (BENCHMARK_COUNTERS_TABULAR)

Whether to use tabular format when printing user counters to the console. Valid values: 'true'/'yes'/1, 'false'/'no'/0.

Default: false

Example:

$ ./benchmark --benchmark_counters_tabular=true

Performance Counters and Context Copy Section

`--benchmark_perf_counters=` (BENCHMARK_PERF_COUNTERS)

List of additional performance counters to collect, in libpfm format. For more information about libpfm, see the libpfm documentation.

Example:

$ ./benchmark --benchmark_perf_counters=cycles,instructions,cache-misses

`--benchmark_context=` (BENCHMARK_CONTEXT)

Extra context to include in the output, formatted as comma-separated key-value pairs. This context is included in the JSON output's context object.

Example:

$ ./benchmark --benchmark_context=compiler=clang,version=13

Miscellaneous Copy Section

`-v` (V)

The level of verbose logging to output. Higher values produce more verbose output.

Default: 0

Example:

$ ./benchmark -v

Benchmark                               Time(ns)    CPU(ns) Iterations
----------------------------------------------------------------------
BM_SetInsert/1024/1                        28928      29349      23853  133.097kiB/s   33.2742k items/s
BM_SetInsert/1024/8                        32065      32913      21375  949.487kiB/s   237.372k items/s
BM_SetInsert/1024/10                       33157      33648      21431  1.13369MiB/s   290.225k items/s

{
  "context": {
    "date": "2015/03/17-18:40:25",
    "host_name": "my-host",
    "num_cpus": 40,
    "mhz_per_cpu": 2801,
    "cpu_scaling_enabled": false,
    "caches": [
      {
        "type": "Data",
        "level": 1,
        "size": 32768,
        "num_sharing": 2
      }
    ],
    "load_avg": [],
    "library_version": "vX.Y.Z",
    "library_build_type": "debug",
    "json_schema_version": 1
  },
  "benchmarks": [
    {
      "name": "BM_SetInsert/1024/1",
      "run_name": "BM_SetInsert/1024/1",
      "run_type": "iteration",
      "iterations": 94877,
      "real_time": 29275,
      "cpu_time": 29836,
      "time_unit": "ns",
      "bytes_per_second": 134066,
      "items_per_second": 33516
    },
    {
      "name": "BM_SetInsert/1024/8",
      "run_name": "BM_SetInsert/1024/8",
      "run_type": "iteration",
      "iterations": 21609,
      "real_time": 32317,
      "cpu_time": 32429,
      "time_unit": "ns",
      "bytes_per_second": 986770,
      "items_per_second": 246693
    },
    {
      "name": "BM_SetInsert/1024/10",
      "run_name": "BM_SetInsert/1024/10",
      "run_type": "iteration",
      "iterations": 21393,
      "real_time": 32724,
      "cpu_time": 33355,
      "time_unit": "ns",
      "bytes_per_second": 1199226,
      "items_per_second": 299807
    }
  ]
}

name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label
"BM_SetInsert/1024/1",65465,17890.7,8407.45,475768,118942,
"BM_SetInsert/1024/8",116606,18810.1,9766.64,3.27646e+06,819115,
"BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,

Write benchmark results to a file with the --benchmark_out=<filename> option
(or set BENCHMARK_OUT). Specify the output format with
--benchmark_out_format={json|console|csv} (or set
BENCHMARK_OUT_FORMAT={json|console|csv}). Note that the 'csv' reporter is
deprecated and the saved .csv file
is not parsable by csv
parsers.

Specifying --benchmark_out does not suppress the console output.

$ ./run_benchmarks.x --benchmark_filter=BM_memcpy/32
Run on (1 X 2300 MHz CPU )
2016-06-25 19:34:24
Benchmark              Time           CPU Iterations
----------------------------------------------------
BM_memcpy/32          11 ns         11 ns   79545455
BM_memcpy/32k       2181 ns       2185 ns     324074
BM_memcpy/32          12 ns         12 ns   54687500
BM_memcpy/32k       1834 ns       1837 ns     357143

It is possible to compare the benchmarking results.
See Additional Tooling Documentation

$ ./run_benchmarks --benchmark_context=pwd=`pwd`
Run on (1 x 2300 MHz CPU)
pwd: /home/user/benchmark/
Benchmark              Time           CPU Iterations
----------------------------------------------------
BM_memcpy/32          11 ns         11 ns   79545455
BM_memcpy/32k       2181 ns       2185 ns     324074

benchmark::AddCustomContext("foo", "bar");

$ ./run_benchmarks.x --benchmark_min_time=0.5s
$ ./run_benchmarks.x --benchmark_min_time=100x

static void DoSetup(const benchmark::State& state) {
}

static void DoTeardown(const benchmark::State& state) {
}

static void BM_func(benchmark::State& state) {...}

BENCHMARK(BM_func)->Arg(1)->Arg(3)->Threads(16)->Threads(32)->Setup(DoSetup)->Teardown(DoTeardown);

static void BM_memcpy(benchmark::State& state) {
  char* src = new char[state.range(0)];
  char* dst = new char[state.range(0)];
  memset(src, 'x', state.range(0));
  for (auto _ : state)
    memcpy(dst, src, state.range(0));
  state.SetBytesProcessed(int64_t(state.iterations()) *
                          int64_t(state.range(0)));
  delete[] src;
  delete[] dst;
}
BENCHMARK(BM_memcpy)->Arg(8)->Arg(64)->Arg(512)->Arg(4<<10)->Arg(8<<10);

BENCHMARK(BM_memcpy)->Range(8, 8<<10);

BENCHMARK(BM_memcpy)->RangeMultiplier(2)->Range(8, 8<<10);

static void BM_DenseRange(benchmark::State& state) {
  for(auto _ : state) {
    std::vector<int> v(state.range(0), state.range(0));
    auto data = v.data();
    benchmark::DoNotOptimize(data);
    benchmark::ClobberMemory();
  }
}
BENCHMARK(BM_DenseRange)->DenseRange(0, 1024, 128);

static void BM_SetInsert(benchmark::State& state) {
  std::set<int> data;
  for (auto _ : state) {
    state.PauseTiming();
    data = ConstructRandomSet(state.range(0));
    state.ResumeTiming();
    for (int j = 0; j < state.range(1); ++j)
      data.insert(RandomNumber());
  }
}
BENCHMARK(BM_SetInsert)
    ->Args({1<<10, 128})
    ->Args({2<<10, 128})
    ->Args({4<<10, 128})
    ->Args({8<<10, 128})
    ->Args({1<<10, 512})
    ->Args({2<<10, 512})
    ->Args({4<<10, 512})
    ->Args({8<<10, 512});

BENCHMARK(BM_SetInsert)->Ranges({{1<<10, 8<<10}, {128, 512}});

BENCHMARK(BM_SetInsert)
    ->ArgsProduct({{1<<10, 3<<10, 8<<10}, {20, 40, 60, 80}})
// would generate the same benchmark arguments as
BENCHMARK(BM_SetInsert)
    ->Args({1<<10, 20})
    ->Args({3<<10, 20})
    ->Args({8<<10, 20})
    ->Args({3<<10, 40})
    ->Args({8<<10, 40})
    ->Args({1<<10, 40})
    ->Args({1<<10, 60})
    ->Args({3<<10, 60})
    ->Args({8<<10, 60})
    ->Args({1<<10, 80})
    ->Args({3<<10, 80})
    ->Args({8<<10, 80});

BENCHMARK(BM_SetInsert)
    ->ArgsProduct({
      benchmark::CreateRange(8, 128, /*multi=*/2),
      benchmark::CreateDenseRange(1, 4, /*step=*/1)
    })
// would generate the same benchmark arguments as
BENCHMARK(BM_SetInsert)
    ->ArgsProduct({
      {8, 16, 32, 64, 128},
      {1, 2, 3, 4}
    });

static void CustomArguments(benchmark::Benchmark* b) {
  for (int i = 0; i <= 10; ++i)
    for (int j = 32; j <= 1024*1024; j *= 8)
      b->Args({i, j});
}
BENCHMARK(BM_SetInsert)->Apply(CustomArguments);

BENCHMARK(BM_memcpy)->Range(8, 512)->ArgName("bytes");

BENCHMARK(BM_SetInsert)
    ->Args({100, 128})
    ->Args({200, 512})
    ->ArgNames({"size", "inserts"});

template <class ...Args>
void BM_takes_args(benchmark::State& state, Args&&... args) {
  auto args_tuple = std::make_tuple(std::move(args)...);
  for (auto _ : state) {
    std::cout << std::get<0>(args_tuple) << ": " << std::get<1>(args_tuple)
              << '\n';
    [...]
  }
}
// Registers a benchmark named "BM_takes_args/int_string_test" that passes
// the specified values to `args`.
BENCHMARK_CAPTURE(BM_takes_args, int_string_test, 42, std::string("abc"));

// Registers the same benchmark "BM_takes_args/int_test" that passes
// the specified values to `args`.
BENCHMARK_CAPTURE(BM_takes_args, int_test, 42, 43);

void BM_Foo(benchmark::State& state) {
  for (auto _ : state) {}
}
// Registers a benchmark named "BM_Foo/my_variant"
BENCHMARK_NAMED(BM_Foo, my_variant);

static void BM_StringCompare(benchmark::State& state) {
  std::string s1(state.range(0), '-');
  std::string s2(state.range(0), '-');
  for (auto _ : state) {
    auto comparison_result = s1.compare(s2);
    benchmark::DoNotOptimize(comparison_result);
  }
  state.SetComplexityN(state.range(0));
}
BENCHMARK(BM_StringCompare)
    ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity(benchmark::oN);

BENCHMARK(BM_StringCompare)
    ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity();

BENCHMARK(BM_StringCompare)->RangeMultiplier(2)
    ->Range(1<<10, 1<<18)->Complexity([](benchmark::IterationCount n)->double{return n; });

BENCHMARK(BM_memcpy)->Name("memcpy")->RangeMultiplier(2)->Range(8, 8<<10);

template <class Q> void BM_Sequential(benchmark::State& state) {
  Q q;
  typename Q::value_type v;
  for (auto _ : state) {
    for (int i = state.range(0); i--; )
      q.push(v);
    for (int e = state.range(0); e--; )
      q.Wait(&v);
  }
  // actually messages, not bytes:
  state.SetBytesProcessed(
      static_cast<int64_t>(state.iterations())*state.range(0));
}

// You can use the BENCHMARK macro with template parameters:
BENCHMARK(BM_Sequential<WaitQueue<int>>)->Range(1<<0, 1<<10);

// Old, legacy verbose C++03 syntax:
BENCHMARK_TEMPLATE(BM_Sequential, WaitQueue<int>)->Range(1<<0, 1<<10);

#define BENCHMARK(func<...>) // Takes any number of parameters.
#define BENCHMARK_TEMPLATE1(func, arg1)
#define BENCHMARK_TEMPLATE2(func, arg1, arg2)

template <class Q> void BM_Sequential_With_Step(benchmark::State& state, int step) {
  Q q;
  typename Q::value_type v;
  for (auto _ : state) {
    for (int i = state.range(0); i-=step; )
      q.push(v);
    for (int e = state.range(0); e-=step; )
      q.Wait(&v);
  }
  // actually messages, not bytes:
  state.SetBytesProcessed(
      static_cast<int64_t>(state.iterations())*state.range(0));
}

BENCHMARK_TEMPLATE1_CAPTURE(BM_Sequential, WaitQueue<int>, Step1, 1)->Range(1<<0, 1<<10);

class MyFixture : public benchmark::Fixture {
public:
  void SetUp(::benchmark::State& state) {
  }

  void TearDown(::benchmark::State& state) {
  }
};

// Defines and registers `FooTest` using the class `MyFixture`.
BENCHMARK_F(MyFixture, FooTest)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}

// Only defines `BarTest` using the class `MyFixture`.
BENCHMARK_DEFINE_F(MyFixture, BarTest)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}
// `BarTest` is NOT registered.
BENCHMARK_REGISTER_F(MyFixture, BarTest)->Threads(2);
// `BarTest` is now registered.

template<typename T>
class MyFixture : public benchmark::Fixture {};

// Defines and registers `IntTest` using the class template `MyFixture<int>`.
BENCHMARK_TEMPLATE_F(MyFixture, IntTest, int)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}

// Only defines `DoubleTest` using the class template `MyFixture<double>`.
BENCHMARK_TEMPLATE_DEFINE_F(MyFixture, DoubleTest, double)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}
// `DoubleTest` is NOT registered.
BENCHMARK_REGISTER_F(MyFixture, DoubleTest)->Threads(2);
// `DoubleTest` is now registered.

// Defines `Test` using the class template `MyFixture`.
BENCHMARK_TEMPLATE_METHOD_F(MyFixture, Test)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}

// Instantiates and registers the benchmark `MyFixture<int>::Test`.
BENCHMARK_TEMPLATE_INSTANTIATE_F(MyFixture, Test, int)->Threads(2);
// Instantiates and registers the benchmark `MyFixture<double>::Test`.
BENCHMARK_TEMPLATE_INSTANTIATE_F(MyFixture, Test, double)->Threads(4);

template<typename Sz>
class SizedFixture : public benchmark::Fixture {
  static constexpr auto Size = Sz::value;
  int myValue;
};

BENCHMARK_TEMPLATE_METHOD_F(SizedFixture, Test)(benchmark::State& st) {
   for (auto _ : st) {
     this->myValue = Base::Size;
  }
}

BENCHMARK_TEMPLATE_INSTANTIATE_F(SizedFixture, Test, std::integral_constant<5>)->Threads(2);

static void UserCountersExample1(benchmark::State& state) {
  double numFoos = 0, numBars = 0, numBazs = 0;
  for (auto _ : state) {
    // ... count Foo,Bar,Baz events
  }
  state.counters["Foo"] = numFoos;
  state.counters["Bar"] = numBars;
  state.counters["Baz"] = numBazs;
}

// sets a simple counter
  state.counters["Foo"] = numFoos;

  // Set the counter as a rate. It will be presented divided
  // by the duration of the benchmark.
  // Meaning: per one second, how many 'foo's are processed?
  state.counters["FooRate"] = Counter(numFoos, benchmark::Counter::kIsRate);

  // Set the counter as a rate. It will be presented divided
  // by the duration of the benchmark, and the result inverted.
  // Meaning: how many seconds it takes to process one 'foo'?
  state.counters["FooInvRate"] = Counter(numFoos, benchmark::Counter::kIsRate | benchmark::Counter::kInvert);

  // Set the counter as a thread-average quantity. It will
  // be presented divided by the number of threads.
  state.counters["FooAvg"] = Counter(numFoos, benchmark::Counter::kAvgThreads);

  // There's also a combined flag:
  state.counters["FooAvgRate"] = Counter(numFoos,benchmark::Counter::kAvgThreadsRate);

  // This says that we process with the rate of state.range(0) bytes every iteration:
  state.counters["BytesProcessed"] = Counter(state.range(0), benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::OneK::kIs1024);

state.counters.insert({{"Foo", numFoos}, {"Bar", numBars}, {"Baz", numBazs}});
  // ... instead of:
  state.counters["Foo"] = numFoos;
  state.counters["Bar"] = numBars;
  state.counters["Baz"] = numBazs;

------------------------------------------------------------------------------
Benchmark                        Time           CPU Iterations UserCounters...
------------------------------------------------------------------------------
BM_UserCounter/threads:8      2248 ns      10277 ns      68808 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:1      9797 ns       9788 ns      71523 Bar=2 Bat=5 Baz=3 Foo=1024m
BM_UserCounter/threads:2      4924 ns       9842 ns      71036 Bar=4 Bat=10 Baz=6 Foo=2
BM_UserCounter/threads:4      2589 ns      10284 ns      68012 Bar=8 Bat=20 Baz=12 Foo=4
BM_UserCounter/threads:8      2212 ns      10287 ns      68040 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:16     1782 ns      10278 ns      68144 Bar=32 Bat=80 Baz=48 Foo=16
BM_UserCounter/threads:32     1291 ns      10296 ns      68256 Bar=64 Bat=160 Baz=96 Foo=32
BM_UserCounter/threads:4      2615 ns      10307 ns      68040 Bar=8 Bat=20 Baz=12 Foo=4
BM_Factorial                    26 ns         26 ns   26608979 40320
BM_Factorial/real_time          26 ns         26 ns   26587936 40320
BM_CalculatePiRange/1           16 ns         16 ns   45704255 0
BM_CalculatePiRange/8           73 ns         73 ns    9520927 3.28374
BM_CalculatePiRange/64         609 ns        609 ns    1140647 3.15746
BM_CalculatePiRange/512       4900 ns       4901 ns     142696 3.14355

/* Detailed source-code truncated for AI context efficiency. */

static void BM_MultiThreaded(benchmark::State& state) {
  if (state.thread_index() == 0) {
    // Setup code here.
  }
  for (auto _ : state) {
    // Run the test as normal.
  }
  if (state.thread_index() == 0) {
    // Teardown code here.
  }
}
BENCHMARK(BM_MultiThreaded)->Threads(2);

BENCHMARK(BM_MultiThreaded)->ThreadRange(1, 8);

BENCHMARK(BM_test)->Range(8, 8<<10)->UseRealTime();

class OpenMPThreadRunner : public benchmark::ThreadRunnerBase
{
  OpenMPThreadRunner(int num_threads)
  : num_threads_(num_threads)
  {}

  void RunThreads(const std::function<void(int)>& fn) final
  {
#pragma omp parallel num_threads(num_threads_)
    fn(omp_get_thread_num());
  }

private:
  int num_threads_;
};

BENCHMARK(BM_MultiThreaded)
  ->ThreadRunner([](int num_threads) {
    return std::make_unique<OpenMPThreadRunner>(num_threads);
  })
  ->Threads(1)->Threads(2)->Threads(4);

void callee(int i);

static void MyMain(int size) {
#pragma omp parallel for
  for(int i = 0; i < size; i++)
    callee(i);
}

static void BM_OpenMP(benchmark::State& state) {
  for (auto _ : state)
    MyMain(state.range(0));
}

// Measure the time spent by the main thread, use it to decide for how long to
// run the benchmark loop. Depending on the internal implementation detail may
// measure to anywhere from near-zero (the overhead spent before/after work
// handoff to worker thread[s]) to the whole single-thread time.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10);

// Measure the user-visible time, the wall clock (literally, the time that
// has passed on the clock on the wall), use it to decide for how long to
// run the benchmark loop. This will always be meaningful, and will match the
// time spent by the main thread in single-threaded case, in general decreasing
// with the number of internal threads doing the work.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->UseRealTime();

// Measure the total CPU consumption, use it to decide for how long to
// run the benchmark loop. This will always measure to no less than the
// time spent by the main thread in single-threaded case.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime();

// A mixture of the last two. Measure the total CPU consumption, but use the
// wall clock to decide for how long to run the benchmark loop.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime()->UseRealTime();

static void BM_SetInsert_With_Timer_Control(benchmark::State& state) {
  std::set<int> data;
  for (auto _ : state) {
    state.PauseTiming(); // Stop timers. They will not count until they are resumed.
    data = ConstructRandomSet(state.range(0)); // Do something that should not be measured
    state.ResumeTiming(); // And resume timers. They are now counting again.
    // The rest will be measured.
    for (int j = 0; j < state.range(1); ++j)
      data.insert(RandomNumber());
  }
}
BENCHMARK(BM_SetInsert_With_Timer_Control)->Ranges({{1<<10, 8<<10}, {128, 512}});

static void BM_SetInsert_With_Scoped_Timer_Control(benchmark::State& state) {
  std::set<int> data;
  for (auto _ : state) {
    {
      benchmark::ScopedPauseTiming pause(state); // Pauses timing
      data = ConstructRandomSet(state.range(0));
    } // Timing resumes automatically when 'pause' goes out of scope

    // The rest will be measured.
    for (int j = 0; j < state.range(1); ++j)
      data.insert(RandomNumber());
  }
}
BENCHMARK(BM_SetInsert_With_Scoped_Timer_Control)->Ranges({{1<<10, 8<<10}, {128, 512}});

static void BM_ManualTiming(benchmark::State& state) {
  int microseconds = state.range(0);
  std::chrono::duration<double, std::micro> sleep_duration {
    static_cast<double>(microseconds)
  };

  for (auto _ : state) {
    auto start = std::chrono::high_resolution_clock::now();
    // Simulate some useful workload with a sleep
    std::this_thread::sleep_for(sleep_duration);
    auto end = std::chrono::high_resolution_clock::now();

    auto elapsed_seconds =
      std::chrono::duration_cast<std::chrono::duration<double>>(
        end - start);

    state.SetIterationTime(elapsed_seconds.count());
  }
}
BENCHMARK(BM_ManualTiming)->Range(1, 1<<17)->UseManualTime();

BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);

static void BM_test(benchmark::State& state) {
  for (auto _ : state) {
      int x = 0;
      for (int i=0; i < 64; ++i) {
        benchmark::DoNotOptimize(x += i);
      }
  }
}

// Example 1: `<expr>` is removed entirely.
  int foo(int x) { return x + 42; }
  while (...) DoNotOptimize(foo(0)); // Optimized to DoNotOptimize(42);

  // Example 2: Result of '<expr>' is only reused.
  int bar(int) __attribute__((const));
  while (...) DoNotOptimize(bar(0)); // Optimized to:
  // int __result__ = bar(0);
  // while (...) DoNotOptimize(__result__);

// Avoid: may call the deprecated const-reference overload.
  while (...) DoNotOptimize(foo(0));

  // Prefer: materialize the result, then pass the local lvalue.
  while (...) {
    auto result = foo(0);
    DoNotOptimize(result);
  }

static void BM_vector_push_back(benchmark::State& state) {
  for (auto _ : state) {
    std::vector<int> v;
    v.reserve(1);
    auto data = v.data();           // Allow v.data() to be clobbered. Pass as non-const
    benchmark::DoNotOptimize(data); // lvalue to avoid undesired compiler optimizations
    v.push_back(42);
    benchmark::ClobberMemory(); // Force 42 to be written to memory.
  }
}

void BM_spin_empty(benchmark::State& state) {
  for (auto _ : state) {
    for (int x = 0; x < state.range(0); ++x) {
      benchmark::DoNotOptimize(x);
    }
  }
}

BENCHMARK(BM_spin_empty)
  ->Repetitions(3) // or add option --benchmark_repetitions=3
  ->ComputeStatistics("max", [](const std::vector<double>& v) -> double {
    return *(std::max_element(std::begin(v), std::end(v)));
  })
  ->Arg(512);

void BM_spin_empty(benchmark::State& state) {
  for (auto _ : state) {
    for (int x = 0; x < state.range(0); ++x) {
      benchmark::DoNotOptimize(x);
    }
  }
}

BENCHMARK(BM_spin_empty)
  ->Repetitions(3) // or add option --benchmark_repetitions=3
  ->ComputeStatistics("ratio", [](const std::vector<double>& v) -> double {
    return v.front() / v.back();
  }, benchmark::StatisticUnit::kPercentage)
  ->Arg(512);

auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ };

int main(int argc, char** argv) {
  benchmark::MaybeReenterWithoutASLR(argc, argv);
  for (auto& test_input : { /* ... */ })
      benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input);
  benchmark::Initialize(&argc, argv);
  benchmark::RunSpecifiedBenchmarks();
  benchmark::Shutdown();
}

static void BM_test(benchmark::State& state) {
  auto resource = GetResource();
  if (!resource.good()) {
    state.SkipWithError("Resource is not good!");
    // KeepRunning() loop will not be entered.
  }
  while (state.KeepRunning()) {
    auto data = resource.read_data();
    if (!resource.good()) {
      state.SkipWithError("Failed to read data!");
      break; // Needed to skip the rest of the iteration.
    }
    do_stuff(data);
  }
}

static void BM_test_ranged_fo(benchmark::State & state) {
  auto resource = GetResource();
  if (!resource.good()) {
    state.SkipWithError("Resource is not good!");
    return; // Early return is allowed when SkipWithError() has been used.
  }
  for (auto _ : state) {
    auto data = resource.read_data();
    if (!resource.good()) {
      state.SkipWithError("Failed to read data!");
      break; // REQUIRED to prevent all further iterations.
    }
    do_stuff(data);
  }
}

static void BM_Fast(benchmark::State &state) {
  for (auto _ : state) {
    FastOperation();
  }
}
BENCHMARK(BM_Fast);

# Loop Init
  mov rbx, qword ptr [r14 + 104]
  call benchmark::State::StartKeepRunning()
  test rbx, rbx
  je .LoopEnd
.LoopHeader: # =>This Inner Loop Header: Depth=1
  add rbx, -1
  jne .LoopHeader
.LoopEnd:

.LoopHeader: # in Loop: Header=BB0_3 Depth=1
  cmp byte ptr [rbx], 1
  jne .LoopInit
.LoopBody: # =>This Inner Loop Header: Depth=1
  mov rax, qword ptr [rbx + 8]
  lea rcx, [rax + 1]
  mov qword ptr [rbx + 8], rcx
  cmp rax, qword ptr [rbx + 104]
  jb .LoopHeader
  jmp .LoopEnd
.LoopInit:
  mov rdi, rbx
  call benchmark::State::StartKeepRunning()
  jmp .LoopBody
.LoopEnd:

***WARNING*** CPU scaling is enabled, the benchmark real time measurements may
be noisy and will incur extra overhead.

See Reducing Variance for more information.

CONTRIBUTING Copy Section

How to contribute

We'd love to accept your patches and contributions to this project. There are
a just a few small guidelines you need to follow.

Contributor License Agreement Copy Chapter

Contributions to any Google project must be accompanied by a Contributor
License Agreement. This is not a copyright assignment, it simply gives
Google permission to use and redistribute your contributions as part of the
project.

• If you are an individual writing original source code and you're sure you
  own the intellectual property, then you'll need to sign an individual
  CLA.

• If you work for a company that wants to allow you to contribute your work,
  then you'll need to sign a corporate CLA.

You generally only need to submit a CLA once, so if you've already submitted
one (even if it was for a different project), you probably don't need to do it
again.

Once your CLA is submitted (or if you already submitted one for
another Google project), make a commit adding yourself to the
AUTHORS and CONTRIBUTORS files. This commit can be part
of your first pull request.

Submitting a patch Copy Chapter

1. It's generally best to start by opening a new issue describing the bug or
   feature you're intending to fix. Even if you think it's relatively minor,
   it's helpful to know what people are working on. Mention in the initial
   issue that you are planning to work on that bug or feature so that it can
   be assigned to you.

2. Follow the normal process of forking the project, and setup a new
   branch to work in. It's important that each group of changes be done in
   separate branches in order to ensure that a pull request only includes the
   commits related to that bug or feature.

3. Do your best to have well-formed commit messages for each change.
   This provides consistency throughout the project, and ensures that commit
   messages are able to be formatted properly by various git tools.

4. Finally, push the commits to your fork and submit a pull request.

AI usage Copy Chapter

Please consult AGENTS file.

README Copy Section

Benchmark

A library to benchmark code snippets, similar to unit tests. Example:

#include <benchmark/registration.h>
#include <benchmark/state.h>

static void BM_SomeFunction(benchmark::State& state) {
  // Perform setup here
  for (auto _ : state) {
    // This code gets timed
    SomeFunction();
  }
}
// Register the function as a benchmark
BENCHMARK(BM_SomeFunction);
// Run the benchmark
BENCHMARK_MAIN();

Getting Started Copy Chapter

To get started, see Requirements and
Installation. See Usage for a full example and the
User Guide for a more comprehensive feature overview.

It may also help to read the Google Test documentation
as some of the structural aspects of the APIs are similar.

Resources Copy Chapter

Discussion group

IRC channels:

• libera #benchmark

Additional Tooling Documentation

Assembly Testing Documentation

Building and installing Python bindings

Requirements Copy Chapter

The library can be used with C++11. However, it requires C++17 to build,
including compiler and standard library support.

See dependencies.md for more details regarding supported
compilers and standards.

If you have need for a particular compiler to be supported, patches are very welcome.

See Platform-Specific Build Instructions.

Installation Copy Chapter

This describes the installation process using cmake. As pre-requisites, you'll
need git and cmake installed.

See dependencies.md for more details regarding supported
versions of build tools.

# Check out the library.
$ git clone https://github.com/google/benchmark.git
# Go to the library root directory
$ cd benchmark
# Make a build directory to place the build output.
$ cmake -E make_directory "build"
# Generate build system files with cmake, and download any dependencies.
$ cmake -DBENCHMARK_DOWNLOAD_DEPENDENCIES=on -DCMAKE_BUILD_TYPE=Release -S . -B "build"
# Build the library.
$ cmake --build "build" --config Release

This builds the benchmark and benchmark_main libraries and tests.
On a unix system, the build directory should now look something like this:

/benchmark
  /build
    /src
      /libbenchmark.a
      /libbenchmark_main.a
    /test
      ...

Next, you can run the tests to check the build.

$ cmake -E chdir "build" ctest --build-config Release

If you want to install the library globally, also run:

sudo cmake --build "build" --config Release --target install

Note that Google Benchmark requires Google Test to build and run the tests. This
dependency can be provided two ways:

• Checkout the Google Test sources into benchmark/googletest.
• Otherwise, if -DBENCHMARK_DOWNLOAD_DEPENDENCIES=ON is specified during
  configuration as above, the library will automatically download and build
  any required dependencies.

If you do not wish to build and run the tests, add -DBENCHMARK_ENABLE_GTEST_TESTS=OFF
to CMAKE_ARGS.

Debug vs Release Copy Section

By default, benchmark builds as a debug library. You will see a warning in the
output when this is the case. To build it as a release library instead, add
-DCMAKE_BUILD_TYPE=Release when generating the build system files, as shown
above. The use of --config Release in build commands is needed to properly
support multi-configuration tools (like Visual Studio for example) and can be
skipped for other build systems (like Makefile).

To enable link-time optimisation, also add -DBENCHMARK_ENABLE_LTO=true when
generating the build system files.

If you are using gcc, you might need to set GCC_AR and GCC_RANLIB cmake
cache variables, if autodetection fails.

If you are using clang, you may need to set LLVMAR_EXECUTABLE,
LLVMNM_EXECUTABLE and LLVMRANLIB_EXECUTABLE cmake cache variables.

To enable sanitizer checks (eg., asan and tsan), add:

-DCMAKE_C_FLAGS="-g -O2 -fno-omit-frame-pointer -fsanitize=address -fsanitize=thread -fno-sanitize-recover=all"
 -DCMAKE_CXX_FLAGS="-g -O2 -fno-omit-frame-pointer -fsanitize=address -fsanitize=thread -fno-sanitize-recover=all "

Stable and Experimental Library Versions Copy Section

The main branch contains the latest stable version of the benchmarking library;
the API of which can be considered largely stable, with source breaking changes
being made only upon the release of a new major version.

Newer, experimental, features are implemented and tested on the
v2 branch. Users who wish
to use, test, and provide feedback on the new features are encouraged to try
this branch. However, this branch provides no stability guarantees and reserves
the right to change and break the API at any time.

Usage Copy Chapter

Basic usage Copy Section

Define a function that executes the code to measure, register it as a benchmark
function using the BENCHMARK macro, and ensure an appropriate main function
is available:

#include <benchmark/benchmark.h>

static void BM_StringCreation(benchmark::State& state) {
  for (auto _ : state)
    std::string empty_string;
}
// Register the function as a benchmark
BENCHMARK(BM_StringCreation);

// Define another benchmark
static void BM_StringCopy(benchmark::State& state) {
  std::string x = "hello";
  for (auto _ : state)
    std::string copy(x);
}
BENCHMARK(BM_StringCopy);

BENCHMARK_MAIN();

To run the benchmark, compile and link against the benchmark library
(libbenchmark.a/.so). If you followed the build steps above, this library will
be under the build directory you created.

# Example on linux after running the build steps above. Assumes the
# `benchmark` and `build` directories are under the current directory.
$ g++ mybenchmark.cc -std=c++11 -isystem benchmark/include \
  -Lbenchmark/build/src -lbenchmark -lpthread -o mybenchmark

Alternatively, link against the benchmark_main library and remove
BENCHMARK_MAIN(); above to get the same behavior.

The compiled executable will run all benchmarks by default. Pass the --help
flag for option information or see the User Guide.

Usage with CMake Copy Section

If using CMake, it is recommended to link against the project-provided
benchmark::benchmark or benchmark::benchmark_main targets using
target_link_libraries. Link to benchmark::benchmark when your target
defines its own main function, or link to benchmark::benchmark_main to use
the default benchmark entry point. The benchmark::benchmark_main target links
benchmark::benchmark transitively.
It is possible to use find_package to import an installed version of the
library.

find_package(benchmark REQUIRED)

Alternatively, add_subdirectory will incorporate the library directly in
to one's CMake project.

add_subdirectory(benchmark)

Either way, link to the library as follows.

target_link_libraries(MyTarget benchmark::benchmark)
# Or, when you do not define your own main:
target_link_libraries(MyTarget benchmark::benchmark_main)

When benchmark sources are shared through an intermediate CMake target, choose
an object library instead of a static library:

add_library(shared_benchmarks OBJECT bench.cc)
target_link_libraries(shared_benchmarks benchmark::benchmark_main)
add_executable(runnable_benchmarks)
target_link_libraries(runnable_benchmarks shared_benchmarks)

This links the object file that contains BENCHMARK registrations into the
final executable. If those registrations are placed only in an intermediate
STATIC library, the linker may not copy static registration symbols, and thus
benchmarks will not be part of the final executable.

Embedding Google Benchmark in another CMake project

There are two common ways to consume Google Benchmark from a CMake project:

• Use an installed or package-managed copy, for example from a system package
  manager or vcpkg, and import it with find_package(benchmark REQUIRED).
• Add this repository to the source tree, for example as a submodule or
  FetchContent dependency, and call add_subdirectory.

The installed form keeps Google Benchmark's build separate from the parent
project and is usually the simplest choice for system packages and vcpkg. The
source-tree form is useful when the parent project wants to pin a specific
commit or build Google Benchmark as part of its normal CMake configure step.

When embedding from source, most projects should turn off Google Benchmark's
tests and install rules:

set(BENCHMARK_ENABLE_GTEST_TESTS OFF CACHE BOOL "" FORCE)
set(BENCHMARK_ENABLE_TESTING OFF CACHE BOOL "" FORCE)
set(BENCHMARK_ENABLE_INSTALL OFF CACHE BOOL "" FORCE)
add_subdirectory(third_party/benchmark)
target_link_libraries(MyTarget benchmark::benchmark)

If Google Test is not already provided by the parent build, either check out the
Google Test sources under benchmark/googletest or configure with
BENCHMARK_DOWNLOAD_DEPENDENCIES=ON. For projects that only link the benchmark
library and do not build Google Benchmark's tests, disabling
BENCHMARK_ENABLE_GTEST_TESTS avoids the Google Test dependency.

Google Benchmark follows CMake's BUILD_SHARED_LIBS setting when selecting
static or shared library output. On Windows, keep this setting consistent with
the rest of the project and make sure the same runtime library configuration is
used across the benchmark library and the targets that link it.

Usage with Bazel Copy Section

If using Bazel with Bzlmod, add Google Benchmark to your MODULE.bazel file:

bazel_dep(name = "google_benchmark", version = "<VERSION>")

Replace <VERSION> with the Google Benchmark release version you want to use.

Then link a cc_binary or cc_test against one of the provided targets:

load("@rules_cc//cc:defs.bzl", "cc_binary")

cc_binary(
    name = "my_benchmark",
    srcs = ["my_benchmark.cc"],
    deps = ["@google_benchmark//:benchmark_main"],
)

Use @google_benchmark//:benchmark when your target defines its own main
function, including through BENCHMARK_MAIN(). Use
@google_benchmark//:benchmark_main to use the default Google Benchmark entry
point.

For WORKSPACE setup and more examples, see Bazel.