kicad Documentation

***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.

you might want to disable the CPU frequency scaling while running the
benchmark, as well as consider other ways to stabilize the performance of
your system while benchmarking.

Exactly how to do this depends on the Linux distribution,
desktop environment, and installed programs. Specific details are a moving
target, so we will not attempt to exhaustively document them here.

One simple option is to use the cpupower program to change the
performance governor to "performance". This tool is maintained along with
the Linux kernel and provided by your distribution.

It must be run as root, like this:

sudo cpupower frequency-set --governor performance

After this you can verify that all CPUs are using the performance governor
by running this command:

Running a Subset of Benchmarks

cpupower frequency-info -o proc

The benchmarks you subsequently run will have less variance.

Reducing Variance in Benchmarks

The Linux CPU frequency governor discussed
above is not the only source
of noise in benchmarks. Some, but not all, of the sources of variance
include:

On multi-core machines not all CPUs/CPU cores/CPU threads run the same
speed, so running a benchmark one time and then again may give a
different result depending on which CPU it ran on.
CPU scaling features that run on the CPU, like Intel's Turbo Boost and
AMD Turbo Core and Precision Boost, can temporarily change the CPU
frequency even when the using the "performance" governor on Linux.
Context switching between CPUs, or scheduling competition on the CPU the
benchmark is running on.
Intel Hyperthreading or AMD SMT causing the same issue as above.
Cache effects caused by code running on other CPUs.
Non-uniform memory architectures (NUMA).

These can cause variance in benchmarks results within a single run
(--benchmark_repetitions=N) or across multiple runs of the benchmark
program.

Reducing sources of variance is OS and architecture dependent, which is one
reason some companies maintain machines dedicated to performance testing.

Some of the easier and effective ways of reducing variance on a typical
Linux workstation are:

Use the performance governor as discussed
above.
Disable processor boosting by:
sh
```
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost
```
See the Linux kernel's
boost.txt
for more information.
Set the benchmark program's task affinity to a fixed cpu. For example:
sh
```
taskset -c 0 ./mybenchmark
```
Disabling Hyperthreading/SMT. This can be done in the Bios or using the
/sys file system (see the LLVM project's Benchmarking
tips).
Close other programs that do non-trivial things based on timers, such as
your web browser, desktop environment, etc.
Reduce the working set of your benchmark to fit within the L1 cache, but
do be aware that this may lead you to optimize for an unrealistic
situation.

Further resources on this topic:

The LLVM project's Benchmarking
tips.
The Arch Wiki Cpu frequency
scaling page.

Thirdparty Sentry Native External Benchmark Docs Releasing (`thirdparty/sentry-native/external/benchmark/docs/releasing.md`)

How to release

Make sure you're on main and synced to HEAD
Ensure the project builds and tests run
- parallel -j0 exec ::: test/*_test can help ensure everything at least
  passes
Prepare release notes
- git log $(git describe --abbrev=0 --tags)..HEAD gives you the list of
  commits between the last annotated tag and HEAD
- Pick the most interesting.
Create one last commit that updates the version saved in CMakeLists.txt, MODULE.bazel,
and bindings/python/google_benchmark/__init__.py to the release version you're creating.
(This version will be used if benchmark is installed from the archive you'll be creating
in the next step.)

text

# CMakeLists.txt
project (benchmark VERSION 1.9.0 LANGUAGES CXX)

text

# MODULE.bazel
module(name = "com_github_google_benchmark", version="1.9.0")

text

# google_benchmark/__init__.py
__version__ = "1.9.0"

Create a release through github's interface
- Note this will create a lightweight tag.
- Update this to an annotated tag:
  - git pull --tags
  - git tag -a -f <tag> <tag>
  - git push --force --tags origin
Confirm that the "Build and upload Python wheels" action runs to completion
- Run it manually if it hasn't run.

Thirdparty Sentry Native External Benchmark Docs Tools (`thirdparty/sentry-native/external/benchmark/docs/tools.md`)

Benchmark Tools

compare.py

The compare.py can be used to compare the result of benchmarks.

Dependencies

The utility relies on the scipy package which can be installed using pip:

bash

pip3 install -r requirements.txt

Displaying aggregates only

The switch -a / --display_aggregates_only can be used to control the
displayment of the normal iterations vs the aggregates. When passed, it will
be passthrough to the benchmark binaries to be run, and will be accounted for
in the tool itself; only the aggregates will be displayed, but not normal runs.
It only affects the display, the separate runs will still be used to calculate
the U test.

Modes of operation

There are three modes of operation:

Just compare two benchmarks
The program is invoked like:

bash

$ compare.py benchmarks <benchmark_baseline> <benchmark_contender> [benchmark options]...

Where and either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file.

[benchmark options] will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal --benchmark_* parameters, or some custom parameters your binary takes.

Example output:

text

/* Detailed source-code truncated for AI context efficiency. */

What it does is for the every benchmark from the first run it looks for the benchmark with exactly the same name in the second run, and then compares the results. If the names differ, the benchmark is omitted from the diff.
As you can note, the values in Time and CPU columns are calculated as (new - old) / |old|.

Compare two different filters of one benchmark
The program is invoked like:

bash

$ compare.py filters <benchmark> <filter_baseline> <filter_contender> [benchmark options]...

Where `` either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file.

Where <filter_baseline> and <filter_contender> are the same regex filters that you would pass to the [--benchmark_filter=<regex>] parameter of the benchmark binary.

Example output:

text

/* Detailed source-code truncated for AI context efficiency. */

As you can see, it applies filter to the benchmarks, both when running the benchmark, and before doing the diff. And to make the diff work, the matches are replaced with some common string. Thus, you can compare two different benchmark families within one benchmark binary.
As you can note, the values in Time and CPU columns are calculated as (new - old) / |old|.

Compare filter one from benchmark one to filter two from benchmark two:
The program is invoked like:

bash

$ compare.py filters <benchmark_baseline> <filter_baseline> <benchmark_contender> <filter_contender> [benchmark options]...

Where <filter_baseline> and <filter_contender> are the same regex filters that you would pass to the [--benchmark_filter=<regex>] parameter of the benchmark binary.

Example output:

text

/* Detailed source-code truncated for AI context efficiency. */

This is a mix of the previous two modes, two (potentially different) benchmark binaries are run, and a different filter is applied to each one.
As you can note, the values in Time and CPU columns are calculated as (new - old) / |old|.

Note: Interpreting the output

Performance measurements are an art, and performance comparisons are doubly so.
Results are often noisy and don't necessarily have large absolute differences to
them, so just by visual inspection, it is not at all apparent if two
measurements are actually showing a performance change or not. It is even more
confusing with multiple benchmark repetitions.

Thankfully, what we can do, is use statistical tests on the results to determine
whether the performance has statistically-significantly changed. compare.py
uses Mann–Whitney U
test, with a null
hypothesis being that there's no difference in performance.

The below output is a summary of a benchmark comparison with statistics
provided for a multi-threaded process.

text

Benchmark                                               Time        CPU    Time Old      Time New       CPU Old       CPU New
-----------------------------------------------------------------------------------------------------------------------------
benchmark/threads:1/process_time/real_time_pvalue     0.0000     0.0000    U Test, Repetitions: 27 vs 27
benchmark/threads:1/process_time/real_time_mean      -0.1442    -0.1442          90            77            90            77
benchmark/threads:1/process_time/real_time_median    -0.1444    -0.1444          90            77            90            77
benchmark/threads:1/process_time/real_time_stddev    +0.3974    +0.3933           0             0             0             0
benchmark/threads:1/process_time/real_time_cv        +0.6329    +0.6280           0             0             0             0
OVERALL_GEOMEAN                                      -0.1442    -0.1442           0             0             0             0

Here's a breakdown of each row:

benchmark/threads:1/process_time/real_time_pvalue: This shows the p-value for
the statistical test comparing the performance of the process running with one
thread. A value of 0.0000 suggests a statistically significant difference in
performance. The comparison was conducted using the U Test (Mann-Whitney
U Test) with 27 repetitions for each case.

benchmark/threads:1/process_time/real_time_mean: This shows the relative
difference in mean execution time between two different cases. The negative
value (-0.1442) implies that the new process is faster by about 14.42%. The old
time was 90 units, while the new time is 77 units.

benchmark/threads:1/process_time/real_time_median: Similarly, this shows the
relative difference in the median execution time. Again, the new process is
faster by 14.44%.

benchmark/threads:1/process_time/real_time_stddev: This is the relative
difference in the standard deviation of the execution time, which is a measure
of how much variation or dispersion there is from the mean. A positive value
(+0.3974) implies there is more variance in the execution time in the new
process.

benchmark/threads:1/process_time/real_time_cv: CV stands for Coefficient of
Variation. It is the ratio of the standard deviation to the mean. It provides a
standardized measure of dispersion. An increase (+0.6329) indicates more
relative variability in the new process.

OVERALL_GEOMEAN: Geomean stands for geometric mean, a type of average that is
less influenced by outliers. The negative value indicates a general improvement
in the new process. However, given the values are all zero for the old and new
times, this seems to be a mistake or placeholder in the output.

Let's first try to see what the different columns represent in the above
compare.py benchmarking output:

Benchmark: The name of the function being benchmarked, along with the
size of the input (after the slash).
Time: The average time per operation, across all iterations.
CPU: The average CPU time per operation, across all iterations.
Iterations: The number of iterations the benchmark was run to get a
stable estimate.
Time Old and Time New: These represent the average time it takes for a
function to run in two different scenarios or versions. For example, you
might be comparing how fast a function runs before and after you make some
changes to it.
CPU Old and CPU New: These show the average amount of CPU time that the
function uses in two different scenarios or versions. This is similar to
Time Old and Time New, but focuses on CPU usage instead of overall time.

In the comparison section, the relative differences in both time and CPU time
are displayed for each input size.

A statistically-significant difference is determined by a p-value, which is
a measure of the probability that the observed difference could have occurred
just by random chance. A smaller p-value indicates stronger evidence against the
null hypothesis.

Therefore:

If the p-value is less than the chosen significance level (alpha), we
reject the null hypothesis and conclude the benchmarks are significantly
different.
If the p-value is greater than or equal to alpha, we fail to reject the
null hypothesis and treat the two benchmarks as similar.

The result of said the statistical test is additionally communicated through color coding:

diff

+ Green:

The benchmarks are statistically different. This could mean the
performance has either significantly improved or significantly
deteriorated. You should look at the actual performance numbers to see which
is the case.

diff

- Red:

The benchmarks are statistically similar. This means the performance
hasn't significantly changed.

In statistical terms, 'green' means we reject the null hypothesis that
there's no difference in performance, and 'red' means we fail to reject the
null hypothesis. This might seem counter-intuitive if you're expecting 'green'
to mean 'improved performance' and 'red' to mean 'worsened performance'.

bash

But remember, in this context:

    'Success' means 'successfully finding a difference'.
    'Failure' means 'failing to find a difference'.

Also, please note that even if we determine that there is a
statistically-significant difference between the two measurements, it does not
necessarily mean that the actual benchmarks that were measured are
different, or vice versa, even if we determine that there is no
statistically-significant difference between the two measurements, it does not
necessarily mean that the actual benchmarks that were measured are not
different.

U test

If there is a sufficient repetition count of the benchmarks, the tool can do
a U Test, of the
null hypothesis that it is equally likely that a randomly selected value from
one sample will be less than or greater than a randomly selected value from a
second sample.

If the calculated p-value is below this value is lower than the significance
level alpha, then the result is said to be statistically significant and the
null hypothesis is rejected. Which in other words means that the two benchmarks
aren't identical.

WARNING: requires LARGE (no less than 9) number of repetitions to be
meaningful!

Thirdparty Sentry Native External Benchmark Docs User Guide (`thirdparty/sentry-native/external/benchmark/docs/user_guide.md`)

User Guide

Command Line

Output Formats

Output Files

Running Benchmarks

Result Comparison

Extra Context

Library

Runtime and Reporting Considerations

Setup/Teardown

Passing Arguments

Custom Benchmark Name

Calculating Asymptotic Complexity

Templated Benchmarks

Templated Benchmarks that take arguments

Fixtures

Custom Counters

Multithreaded Benchmarks

CPU Timers

Manual Timing

Setting the Time Unit

Random Interleaving

User-Requested Performance Counters

Preventing Optimization

Reporting Statistics

Custom Statistics

Memory Usage

Using RegisterBenchmark

Exiting with an Error

A Faster KeepRunning Loop

Benchmarking Tips

Disabling CPU Frequency Scaling

Reducing Variance in Benchmarks

Output Formats

The library supports multiple output formats. Use the
--benchmark_format=<console|json|csv> flag (or set the
BENCHMARK_FORMAT=<console|json|csv> environment variable) to set
the format type. console is the default format.

The Console format is intended to be a human readable format. By default
the format generates color output. Context is output on stderr and the
tabular data on stdout. Example tabular output looks like:

Benchmark                               Time(ns)    CPU(ns) Iterations
----------------------------------------------------------------------
BM_SetInsert/1024/1                        28928      29349      23853  133.097kiB/s   33.2742k items/s
BM_SetInsert/1024/8                        32065      32913      21375  949.487kiB/s   237.372k items/s
BM_SetInsert/1024/10                       33157      33648      21431  1.13369MiB/s   290.225k items/s

The JSON format outputs human readable json split into two top level attributes.
The context attribute contains information about the run in general, including
information about the CPU and the date.
The benchmarks attribute contains a list of every benchmark run. Example json
output looks like:

json

{
  "context": {
    "date": "2015/03/17-18:40:25",
    "num_cpus": 40,
    "mhz_per_cpu": 2801,
    "cpu_scaling_enabled": false,
    "build_type": "debug"
  },
  "benchmarks": [
    {
      "name": "BM_SetInsert/1024/1",
      "iterations": 94877,
      "real_time": 29275,
      "cpu_time": 29836,
      "bytes_per_second": 134066,
      "items_per_second": 33516
    },
    {
      "name": "BM_SetInsert/1024/8",
      "iterations": 21609,
      "real_time": 32317,
      "cpu_time": 32429,
      "bytes_per_second": 986770,
      "items_per_second": 246693
    },
    {
      "name": "BM_SetInsert/1024/10",
      "iterations": 21393,
      "real_time": 32724,
      "cpu_time": 33355,
      "bytes_per_second": 1199226,
      "items_per_second": 299807
    }
  ]
}

The CSV format outputs comma-separated values. The context is output on stderr
and the CSV itself on stdout. Example CSV output looks like:

Write benchmark results to a file with the --benchmark_out=<filename> option
(or set BENCHMARK_OUT). Specify the output format with
--benchmark_out_format={json|console|csv} (or set
BENCHMARK_OUT_FORMAT={json|console|csv}). Note that the 'csv' reporter is
deprecated and the saved .csv file
is not parsable by csv
parsers.

name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label
"BM_SetInsert/1024/1",65465,17890.7,8407.45,475768,118942,
"BM_SetInsert/1024/8",116606,18810.1,9766.64,3.27646e+06,819115,
"BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,

Output Files

Specifying --benchmark_out does not suppress the console output.

Running Benchmarks

Benchmarks are executed by running the produced binaries. Benchmarks binaries,
by default, accept options that may be specified either through their command
line interface or by setting environment variables before execution. For every
--option_flag=<value> CLI switch, a corresponding environment variable
OPTION_FLAG=<value> exist and is used as default if set (CLI switches always
prevails). A complete list of CLI options is available running benchmarks
with the --help switch.

Dry runs

To confirm that benchmarks can run successfully without needing to wait for
multiple repetitions and iterations, the --benchmark_dry_run flag can be
used. This will run the benchmarks as normal, but for 1 iteration and 1
repetition only.

Running a Subset of Benchmarks

The --benchmark_filter=<regex> option (or BENCHMARK_FILTER=<regex>
environment variable) can be used to only run the benchmarks that match
the specified <regex>. For example:

It is possible to compare the benchmarking results.
See Additional Tooling Documentation

$ ./run_benchmarks.x --benchmark_filter=BM_memcpy/32
Run on (1 X 2300 MHz CPU )
2016-06-25 19:34:24
Benchmark              Time           CPU Iterations
----------------------------------------------------
BM_memcpy/32          11 ns         11 ns   79545455
BM_memcpy/32k       2181 ns       2185 ns     324074
BM_memcpy/32          12 ns         12 ns   54687500
BM_memcpy/32k       1834 ns       1837 ns     357143

Disabling Benchmarks

It is possible to temporarily disable benchmarks by renaming the benchmark
function to have the prefix "DISABLED_". This will cause the benchmark to
be skipped at runtime.

Result comparison

Extra Context

Sometimes it's useful to add extra context to the content printed before the
results. By default this section includes information about the CPU on which
the benchmarks are running. If you do want to add more context, you can use
the benchmark_context command line flag:

$ ./run_benchmarks --benchmark_context=pwd=`pwd`
Run on (1 x 2300 MHz CPU)
pwd: /home/user/benchmark/
Benchmark              Time           CPU Iterations
----------------------------------------------------
BM_memcpy/32          11 ns         11 ns   79545455
BM_memcpy/32k       2181 ns       2185 ns     324074

You can get the same effect with the API:

benchmark::AddCustomContext("foo", "bar");

Note that attempts to add a second value with the same key will fail with an
error message.

Runtime and Reporting Considerations

When the benchmark binary is executed, each benchmark function is run serially.
The number of iterations to run is determined dynamically by running the
benchmark a few times and measuring the time taken and ensuring that the
ultimate result will be statistically stable. As such, faster benchmark
functions will be run for more iterations than slower benchmark functions, and
the number of iterations is thus reported.

In all cases, the number of iterations for which the benchmark is run is
governed by the amount of time the benchmark takes. Concretely, the number of
iterations is at least one, not more than 1e9, until CPU time is greater than
the minimum time, or the wallclock time is 5x minimum time. The minimum time is
set per benchmark by calling MinTime on the registered benchmark object.

Furthermore warming up a benchmark might be necessary in order to get
stable results because of e.g caching effects of the code under benchmark.
Warming up means running the benchmark a given amount of time, before
results are actually taken into account. The amount of time for which
the warmup should be run can be set per benchmark by calling
MinWarmUpTime on the registered benchmark object or for all benchmarks
using the --benchmark_min_warmup_time command-line option. Note that
MinWarmUpTime will overwrite the value of --benchmark_min_warmup_time
for the single benchmark. How many iterations the warmup run of each
benchmark takes is determined the same way as described in the paragraph
above. Per default the warmup phase is set to 0 seconds and is therefore
disabled.

Average timings are then reported over the iterations run. If multiple
repetitions are requested using the --benchmark_repetitions command-line
option, or at registration time, the benchmark function will be run several
times and statistical results across these repetitions will also be reported.

As well as the per-benchmark entries, a preamble in the report will include
information about the machine on which the benchmarks are run.

Setup/Teardown

Global setup/teardown specific to each benchmark can be done by
passing a callback to Setup/Teardown:

The setup/teardown callbacks will be invoked once for each benchmark. If the
benchmark is multi-threaded (will run in k threads), they will be invoked
exactly once before each run with k threads.

If the benchmark uses different size groups of threads, the above will be true
for each size group.

Eg.,

static void DoSetup(const benchmark::State& state) {
}

static void DoTeardown(const benchmark::State& state) {
}

static void BM_func(benchmark::State& state) {...}

BENCHMARK(BM_func)->Arg(1)->Arg(3)->Threads(16)->Threads(32)->Setup(DoSetup)->Teardown(DoTeardown);

In this example, DoSetup and DoTearDown will be invoked 4 times each,
specifically, once for each of this family:

BM_func_Arg_1_Threads_16, BM_func_Arg_1_Threads_32
BM_func_Arg_3_Threads_16, BM_func_Arg_3_Threads_32

Passing Arguments

Sometimes a family of benchmarks can be implemented with just one routine that
takes an extra argument to specify which one of the family of benchmarks to
run. For example, the following code defines a family of benchmarks for
measuring the speed of memcpy() calls of different lengths:

static void BM_memcpy(benchmark::State& state) {
  char* src = new char[state.range(0)];
  char* dst = new char[state.range(0)];
  memset(src, 'x', state.range(0));
  for (auto _ : state)
    memcpy(dst, src, state.range(0));
  state.SetBytesProcessed(int64_t(state.iterations()) *
                          int64_t(state.range(0)));
  delete[] src;
  delete[] dst;
}
BENCHMARK(BM_memcpy)->Arg(8)->Arg(64)->Arg(512)->Arg(4<<10)->Arg(8<<10);

The preceding code is quite repetitive, and can be replaced with the following
short-hand. The following invocation will pick a few appropriate arguments in
the specified range and will generate a benchmark for each such argument.

BENCHMARK(BM_memcpy)->Range(8, 8<<10);

By default the arguments in the range are generated in multiples of eight and
the command above selects [ 8, 64, 512, 4k, 8k ]. In the following code the
range multiplier is changed to multiples of two.

BENCHMARK(BM_memcpy)->RangeMultiplier(2)->Range(8, 8<<10);

Now arguments generated are [ 8, 16, 32, 64, 128, 256, 512, 1024, 2k, 4k, 8k ].

The preceding code shows a method of defining a sparse range. The following
example shows a method of defining a dense range. It is then used to benchmark
the performance of std::vector initialization for uniformly increasing sizes.

static void BM_DenseRange(benchmark::State& state) {
  for(auto _ : state) {
    std::vector<int> v(state.range(0), state.range(0));
    auto data = v.data();
    benchmark::DoNotOptimize(data);
    benchmark::ClobberMemory();
  }
}
BENCHMARK(BM_DenseRange)->DenseRange(0, 1024, 128);

Now arguments generated are [ 0, 128, 256, 384, 512, 640, 768, 896, 1024 ].

You might have a benchmark that depends on two or more inputs. For example, the
following code defines a family of benchmarks for measuring the speed of set
insertion.

static void BM_SetInsert(benchmark::State& state) {
  std::set<int> data;
  for (auto _ : state) {
    state.PauseTiming();
    data = ConstructRandomSet(state.range(0));
    state.ResumeTiming();
    for (int j = 0; j < state.range(1); ++j)
      data.insert(RandomNumber());
  }
}
BENCHMARK(BM_SetInsert)
    ->Args({1<<10, 128})
    ->Args({2<<10, 128})
    ->Args({4<<10, 128})
    ->Args({8<<10, 128})
    ->Args({1<<10, 512})
    ->Args({2<<10, 512})
    ->Args({4<<10, 512})
    ->Args({8<<10, 512});

The preceding code is quite repetitive, and can be replaced with the following
short-hand. The following macro will pick a few appropriate arguments in the
product of the two specified ranges and will generate a benchmark for each such
pair.

BENCHMARK(BM_SetInsert)->Ranges({{1<<10, 8<<10}, {128, 512}});

Some benchmarks may require specific argument values that cannot be expressed
with Ranges. In this case, ArgsProduct offers the ability to generate a
benchmark input for each combination in the product of the supplied vectors.

BENCHMARK(BM_SetInsert)
    ->ArgsProduct({{1<<10, 3<<10, 8<<10}, {20, 40, 60, 80}})
// would generate the same benchmark arguments as
BENCHMARK(BM_SetInsert)
    ->Args({1<<10, 20})
    ->Args({3<<10, 20})
    ->Args({8<<10, 20})
    ->Args({3<<10, 40})
    ->Args({8<<10, 40})
    ->Args({1<<10, 40})
    ->Args({1<<10, 60})
    ->Args({3<<10, 60})
    ->Args({8<<10, 60})
    ->Args({1<<10, 80})
    ->Args({3<<10, 80})
    ->Args({8<<10, 80});

For the most common scenarios, helper methods for creating a list of
integers for a given sparse or dense range are provided.

BENCHMARK(BM_SetInsert)
    ->ArgsProduct({
      benchmark::CreateRange(8, 128, /*multi=*/2),
      benchmark::CreateDenseRange(1, 4, /*step=*/1)
    })
// would generate the same benchmark arguments as
BENCHMARK(BM_SetInsert)
    ->ArgsProduct({
      {8, 16, 32, 64, 128},
      {1, 2, 3, 4}
    });

For more complex patterns of inputs, passing a custom function to Apply allows
programmatic specification of an arbitrary set of arguments on which to run the
benchmark. The following example enumerates a dense range on one parameter,
and a sparse range on the second.

static void CustomArguments(benchmark::internal::Benchmark* b) {
  for (int i = 0; i <= 10; ++i)
    for (int j = 32; j <= 1024*1024; j *= 8)
      b->Args({i, j});
}
BENCHMARK(BM_SetInsert)->Apply(CustomArguments);

Passing Arbitrary Arguments to a Benchmark

It is possible to define a benchmark that takes an arbitrary number
of extra arguments. The BENCHMARK_CAPTURE(func, test_case_name, ...args)
macro creates a benchmark that invokes func with the benchmark::State as
the first argument followed by the specified args....
The test_case_name is appended to the name of the benchmark and
should describe the values passed.

template <class ...Args>
void BM_takes_args(benchmark::State& state, Args&&... args) {
  auto args_tuple = std::make_tuple(std::move(args)...);
  for (auto _ : state) {
    std::cout << std::get<0>(args_tuple) << ": " << std::get<1>(args_tuple)
              << '\n';
    [...]
  }
}
// Registers a benchmark named "BM_takes_args/int_string_test" that passes
// the specified values to `args`.
BENCHMARK_CAPTURE(BM_takes_args, int_string_test, 42, std::string("abc"));

// Registers the same benchmark "BM_takes_args/int_test" that passes
// the specified values to `args`.
BENCHMARK_CAPTURE(BM_takes_args, int_test, 42, 43);

Note that elements of ...args may refer to global variables. Users should
avoid modifying global state inside of a benchmark.

Calculating Asymptotic Complexity (Big O)

Asymptotic complexity might be calculated for a family of benchmarks. The
following code will calculate the coefficient for the high-order term in the
running time and the normalized root-mean square error of string comparison.

static void BM_StringCompare(benchmark::State& state) {
  std::string s1(state.range(0), '-');
  std::string s2(state.range(0), '-');
  for (auto _ : state) {
    auto comparison_result = s1.compare(s2);
    benchmark::DoNotOptimize(comparison_result);
  }
  state.SetComplexityN(state.range(0));
}
BENCHMARK(BM_StringCompare)
    ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity(benchmark::oN);

As shown in the following invocation, asymptotic complexity might also be
calculated automatically.

BENCHMARK(BM_StringCompare)
    ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity();

The following code will specify asymptotic complexity with a lambda function,
that might be used to customize high-order term calculation.

BENCHMARK(BM_StringCompare)->RangeMultiplier(2)
    ->Range(1<<10, 1<<18)->Complexity([](benchmark::IterationCount n)->double{return n; });

Custom Benchmark Name

You can change the benchmark's name as follows:

BENCHMARK(BM_memcpy)->Name("memcpy")->RangeMultiplier(2)->Range(8, 8<<10);

The invocation will execute the benchmark as before using BM_memcpy but changes
the prefix in the report to memcpy.

Templated Benchmarks

This example produces and consumes messages of size sizeof(v) range_x
times. It also outputs throughput in the absence of multiprogramming.

template <class Q> void BM_Sequential(benchmark::State& state) {
  Q q;
  typename Q::value_type v;
  for (auto _ : state) {
    for (int i = state.range(0); i--; )
      q.push(v);
    for (int e = state.range(0); e--; )
      q.Wait(&v);
  }
  // actually messages, not bytes:
  state.SetBytesProcessed(
      static_cast<int64_t>(state.iterations())*state.range(0));
}

// You can use the BENCHMARK macro with template parameters:
BENCHMARK(BM_Sequential<WaitQueue<int>>)->Range(1<<0, 1<<10);

// Old, legacy verbose C++03 syntax:
BENCHMARK_TEMPLATE(BM_Sequential, WaitQueue<int>)->Range(1<<0, 1<<10);

Three macros are provided for adding benchmark templates.

#define BENCHMARK(func<...>) // Takes any number of parameters.
#define BENCHMARK_TEMPLATE1(func, arg1)
#define BENCHMARK_TEMPLATE2(func, arg1, arg2)

Templated Benchmarks that take arguments

Sometimes there is a need to template benchmarks, and provide arguments to them.

template <class Q> void BM_Sequential_With_Step(benchmark::State& state, int step) {
  Q q;
  typename Q::value_type v;
  for (auto _ : state) {
    for (int i = state.range(0); i-=step; )
      q.push(v);
    for (int e = state.range(0); e-=step; )
      q.Wait(&v);
  }
  // actually messages, not bytes:
  state.SetBytesProcessed(
      static_cast<int64_t>(state.iterations())*state.range(0));
}

BENCHMARK_TEMPLATE1_CAPTURE(BM_Sequential, WaitQueue<int>, Step1, 1)->Range(1<<0, 1<<10);

Fixtures

Fixture tests are created by first defining a type that derives from
::benchmark::Fixture and then creating/registering the tests using the
following macros:

BENCHMARK_F(ClassName, Method)
BENCHMARK_DEFINE_F(ClassName, Method)
BENCHMARK_REGISTER_F(ClassName, Method)

For Example:

class MyFixture : public benchmark::Fixture {
public:
  void SetUp(::benchmark::State& state) {
  }

  void TearDown(::benchmark::State& state) {
  }
};

// Defines and registers `FooTest` using the class `MyFixture`.
BENCHMARK_F(MyFixture, FooTest)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}

// Only defines `BarTest` using the class `MyFixture`.
BENCHMARK_DEFINE_F(MyFixture, BarTest)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}
// `BarTest` is NOT registered.
BENCHMARK_REGISTER_F(MyFixture, BarTest)->Threads(2);
// `BarTest` is now registered.

Templated Fixtures

Also you can create templated fixture by using the following macros:

BENCHMARK_TEMPLATE_F(ClassName, Method, ...)
BENCHMARK_TEMPLATE_DEFINE_F(ClassName, Method, ...)

For example:

template<typename T>
class MyFixture : public benchmark::Fixture {};

// Defines and registers `IntTest` using the class template `MyFixture<int>`.
BENCHMARK_TEMPLATE_F(MyFixture, IntTest, int)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}

// Only defines `DoubleTest` using the class template `MyFixture<double>`.
BENCHMARK_TEMPLATE_DEFINE_F(MyFixture, DoubleTest, double)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}
// `DoubleTest` is NOT registered.
BENCHMARK_REGISTER_F(MyFixture, DoubleTest)->Threads(2);
// `DoubleTest` is now registered.

If you want to use a method template for your fixtures,
which you instantiate afterward, use the following macros:

BENCHMARK_TEMPLATE_METHOD_F(ClassName, Method)
BENCHMARK_TEMPLATE_INSTANTIATE_F(ClassName, Method, ...)

With these macros you can define one method for several instantiations.
Example (using MyFixture from above):

// Defines `Test` using the class template `MyFixture`.
BENCHMARK_TEMPLATE_METHOD_F(MyFixture, Test)(benchmark::State& st) {
   for (auto _ : st) {
     ...
  }
}

// Instantiates and registers the benchmark `MyFixture<int>::Test`.
BENCHMARK_TEMPLATE_INSTANTIATE_F(MyFixture, Test, int)->Threads(2);
// Instantiates and registers the benchmark `MyFixture<double>::Test`.
BENCHMARK_TEMPLATE_INSTANTIATE_F(MyFixture, Test, double)->Threads(4);

Inside the method definition of BENCHMARK_TEMPLATE_METHOD_F the type Base refers
to the type of the instantiated fixture.
Accesses to members of the fixture must be prefixed by this->.

BENCHMARK_TEMPLATE_METHOD_Fand BENCHMARK_TEMPLATE_INSTANTIATE_F can only be used,
if the fixture does not use non-type template parameters.
If you want to pass values as template parameters, use e.g. std::integral_constant.
For example:

template<typename Sz>
class SizedFixture : public benchmark::Fixture {
  static constexpr auto Size = Sz::value;
  int myValue;
};

BENCHMARK_TEMPLATE_METHOD_F(SizedFixture, Test)(benchmark::State& st) {
   for (auto _ : st) {
     this->myValue = Base::Size;
  }
}

BENCHMARK_TEMPLATE_INSTANTIATE_F(SizedFixture, Test, std::integral_constant<5>)->Threads(2);

Custom Counters

You can add your own counters with user-defined names. The example below
will add columns "Foo", "Bar" and "Baz" in its output:

static void UserCountersExample1(benchmark::State& state) {
  double numFoos = 0, numBars = 0, numBazs = 0;
  for (auto _ : state) {
    // ... count Foo,Bar,Baz events
  }
  state.counters["Foo"] = numFoos;
  state.counters["Bar"] = numBars;
  state.counters["Baz"] = numBazs;
}

The state.counters object is a std::map with std::string keys
and Counter values. The latter is a double-like class, via an implicit
conversion to double&. Thus you can use all of the standard arithmetic
assignment operators (=,+=,-=,*=,/=) to change the value of each counter.

In multithreaded benchmarks, each counter is set on the calling thread only.
When the benchmark finishes, the counters from each thread will be summed;
the resulting sum is the value which will be shown for the benchmark.

The Counter constructor accepts three parameters: the value as a double
; a bit flag which allows you to show counters as rates, and/or as per-thread
iteration, and/or as per-thread averages, and/or iteration invariants,
and/or finally inverting the result; and a flag specifying the 'unit' - i.e.
is 1k a 1000 (default, benchmark::Counter::OneK::kIs1000), or 1024
(benchmark::Counter::OneK::kIs1024)?

// sets a simple counter
  state.counters["Foo"] = numFoos;

  // Set the counter as a rate. It will be presented divided
  // by the duration of the benchmark.
  // Meaning: per one second, how many 'foo's are processed?
  state.counters["FooRate"] = Counter(numFoos, benchmark::Counter::kIsRate);

  // Set the counter as a rate. It will be presented divided
  // by the duration of the benchmark, and the result inverted.
  // Meaning: how many seconds it takes to process one 'foo'?
  state.counters["FooInvRate"] = Counter(numFoos, benchmark::Counter::kIsRate | benchmark::Counter::kInvert);

  // Set the counter as a thread-average quantity. It will
  // be presented divided by the number of threads.
  state.counters["FooAvg"] = Counter(numFoos, benchmark::Counter::kAvgThreads);

  // There's also a combined flag:
  state.counters["FooAvgRate"] = Counter(numFoos,benchmark::Counter::kAvgThreadsRate);

  // This says that we process with the rate of state.range(0) bytes every iteration:
  state.counters["BytesProcessed"] = Counter(state.range(0), benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::OneK::kIs1024);

You can use insert() with std::initializer_list:

state.counters.insert({{"Foo", numFoos}, {"Bar", numBars}, {"Baz", numBazs}});
  // ... instead of:
  state.counters["Foo"] = numFoos;
  state.counters["Bar"] = numBars;
  state.counters["Baz"] = numBazs;

Counter Reporting

When using the console reporter, by default, user counters are printed at
the end after the table, the same way as bytes_processed and
items_processed. This is best for cases in which there are few counters,
or where there are only a couple of lines per benchmark. Here's an example of
the default output:

------------------------------------------------------------------------------
Benchmark                        Time           CPU Iterations UserCounters...
------------------------------------------------------------------------------
BM_UserCounter/threads:8      2248 ns      10277 ns      68808 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:1      9797 ns       9788 ns      71523 Bar=2 Bat=5 Baz=3 Foo=1024m
BM_UserCounter/threads:2      4924 ns       9842 ns      71036 Bar=4 Bat=10 Baz=6 Foo=2
BM_UserCounter/threads:4      2589 ns      10284 ns      68012 Bar=8 Bat=20 Baz=12 Foo=4
BM_UserCounter/threads:8      2212 ns      10287 ns      68040 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:16     1782 ns      10278 ns      68144 Bar=32 Bat=80 Baz=48 Foo=16
BM_UserCounter/threads:32     1291 ns      10296 ns      68256 Bar=64 Bat=160 Baz=96 Foo=32
BM_UserCounter/threads:4      2615 ns      10307 ns      68040 Bar=8 Bat=20 Baz=12 Foo=4
BM_Factorial                    26 ns         26 ns   26608979 40320
BM_Factorial/real_time          26 ns         26 ns   26587936 40320
BM_CalculatePiRange/1           16 ns         16 ns   45704255 0
BM_CalculatePiRange/8           73 ns         73 ns    9520927 3.28374
BM_CalculatePiRange/64         609 ns        609 ns    1140647 3.15746
BM_CalculatePiRange/512       4900 ns       4901 ns     142696 3.14355

If this doesn't suit you, you can print each counter as a table column by
passing the flag --benchmark_counters_tabular=true to the benchmark
application. This is best for cases in which there are a lot of counters, or
a lot of lines per individual benchmark. Note that this will trigger a
reprinting of the table header any time the counter set changes between
individual benchmarks. Here's an example of corresponding output when
--benchmark_counters_tabular=true is passed:

/* Detailed source-code truncated for AI context efficiency. */

Note above the additional header printed when the benchmark changes from
BM_UserCounter to BM_Factorial. This is because BM_Factorial does
not have the same counter set as BM_UserCounter.

Multithreaded Benchmarks

In a multithreaded test (benchmark invoked by multiple threads simultaneously),
it is guaranteed that none of the threads will start until all have reached
the start of the benchmark loop, and all will have finished before any thread
exits the benchmark loop. (This behavior is also provided by the KeepRunning()
API) As such, any global setup or teardown can be wrapped in a check against the thread
index:

static void BM_MultiThreaded(benchmark::State& state) {
  if (state.thread_index() == 0) {
    // Setup code here.
  }
  for (auto _ : state) {
    // Run the test as normal.
  }
  if (state.thread_index() == 0) {
    // Teardown code here.
  }
}
BENCHMARK(BM_MultiThreaded)->Threads(2);

To run the benchmark across a range of thread counts, instead of Threads, use
ThreadRange. This takes two parameters (min_threads and max_threads) and
runs the benchmark once for values in the inclusive range. For example:

BENCHMARK(BM_MultiThreaded)->ThreadRange(1, 8);

will run BM_MultiThreaded with thread counts 1, 2, 4, and 8.

If the benchmarked code itself uses threads and you want to compare it to
single-threaded code, you may want to use real-time ("wallclock") measurements
for latency comparisons:

BENCHMARK(BM_test)->Range(8, 8<<10)->UseRealTime();

Without UseRealTime, CPU time is used by default.

Manual Multithreaded Benchmarks

Google/benchmark uses std::thread as multithreading environment per default.
If you want to use another multithreading environment (e.g. OpenMP), you can provide
a factory function to your benchmark using the ThreadRunner function.
The factory function takes the number of threads as argument and creates a custom class
derived from benchmark::ThreadRunnerBase.
This custom class must override the function
void RunThreads(const std::function<void(int)>& fn).
RunThreads is called by the main thread and spawns the requested number of threads.
Each spawned thread must call fn(thread_index), where thread_index is its own
thread index. Before RunThreads returns, all spawned threads must be joined.

class OpenMPThreadRunner : public benchmark::ThreadRunnerBase
{
  OpenMPThreadRunner(int num_threads)
  : num_threads_(num_threads)
  {}

  void RunThreads(const std::function<void(int)>& fn) final
  {
#pragma omp parallel num_threads(num_threads_)
    fn(omp_get_thread_num());
  }

private:
  int num_threads_;
};

BENCHMARK(BM_MultiThreaded)
  ->ThreadRunner([](int num_threads) {
    return std::make_unique<OpenMPThreadRunner>(num_threads);
  })
  ->Threads(1)->Threads(2)->Threads(4);

The above example creates a parallel OpenMP region before it enters BM_MultiThreaded.
The actual benchmark code can remain the same and is therefore not tied to a specific
thread runner. The measurement does not include the time for creating and joining the
threads.

CPU Timers

By default, the CPU timer only measures the time spent by the main thread.
If the benchmark itself uses threads internally, this measurement may not
be what you are looking for. Instead, there is a way to measure the total
CPU usage of the process, by all the threads.

void callee(int i);

static void MyMain(int size) {
#pragma omp parallel for
  for(int i = 0; i < size; i++)
    callee(i);
}

static void BM_OpenMP(benchmark::State& state) {
  for (auto _ : state)
    MyMain(state.range(0));
}

// Measure the time spent by the main thread, use it to decide for how long to
// run the benchmark loop. Depending on the internal implementation detail may
// measure to anywhere from near-zero (the overhead spent before/after work
// handoff to worker thread[s]) to the whole single-thread time.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10);

// Measure the user-visible time, the wall clock (literally, the time that
// has passed on the clock on the wall), use it to decide for how long to
// run the benchmark loop. This will always be meaningful, and will match the
// time spent by the main thread in single-threaded case, in general decreasing
// with the number of internal threads doing the work.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->UseRealTime();

// Measure the total CPU consumption, use it to decide for how long to
// run the benchmark loop. This will always measure to no less than the
// time spent by the main thread in single-threaded case.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime();

// A mixture of the last two. Measure the total CPU consumption, but use the
// wall clock to decide for how long to run the benchmark loop.
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime()->UseRealTime();

Controlling Timers

Normally, the entire duration of the work loop (for (auto _ : state) {})
is measured. But sometimes, it is necessary to do some work inside of
that loop, every iteration, but without counting that time to the benchmark time.
That is possible, although it is not recommended, since it has high overhead.

static void BM_SetInsert_With_Timer_Control(benchmark::State& state) {
  std::set<int> data;
  for (auto _ : state) {
    state.PauseTiming(); // Stop timers. They will not count until they are resumed.
    data = ConstructRandomSet(state.range(0)); // Do something that should not be measured
    state.ResumeTiming(); // And resume timers. They are now counting again.
    // The rest will be measured.
    for (int j = 0; j < state.range(1); ++j)
      data.insert(RandomNumber());
  }
}
BENCHMARK(BM_SetInsert_With_Timer_Control)->Ranges({{1<<10, 8<<10}, {128, 512}});

Manual Timing

For benchmarking something for which neither CPU time nor real-time are
correct or accurate enough, completely manual timing is supported using
the UseManualTime function.

When UseManualTime is used, the benchmarked code must call
SetIterationTime once per iteration of the benchmark loop to
report the manually measured time.

An example use case for this is benchmarking GPU execution (e.g. OpenCL
or CUDA kernels, OpenGL or Vulkan or Direct3D draw calls), which cannot
be accurately measured using CPU time or real-time. Instead, they can be
measured accurately using a dedicated API, and these measurement results
can be reported back with SetIterationTime.

static void BM_ManualTiming(benchmark::State& state) {
  int microseconds = state.range(0);
  std::chrono::duration<double, std::micro> sleep_duration {
    static_cast<double>(microseconds)
  };

  for (auto _ : state) {
    auto start = std::chrono::high_resolution_clock::now();
    // Simulate some useful workload with a sleep
    std::this_thread::sleep_for(sleep_duration);
    auto end = std::chrono::high_resolution_clock::now();

    auto elapsed_seconds =
      std::chrono::duration_cast<std::chrono::duration<double>>(
        end - start);

    state.SetIterationTime(elapsed_seconds.count());
  }
}
BENCHMARK(BM_ManualTiming)->Range(1, 1<<17)->UseManualTime();

Setting the Time Unit

If a benchmark runs a few milliseconds it may be hard to visually compare the
measured times, since the output data is given in nanoseconds per default. In
order to manually set the time unit, you can specify it manually:

BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);

Additionally the default time unit can be set globally with the
--benchmark_time_unit={ns|us|ms|s} command line argument. The argument only
affects benchmarks where the time unit is not set explicitly.

Preventing Optimization

To prevent a value or expression from being optimized away by the compiler
the benchmark::DoNotOptimize(...) and benchmark::ClobberMemory()
functions can be used.

static void BM_test(benchmark::State& state) {
  for (auto _ : state) {
      int x = 0;
      for (int i=0; i < 64; ++i) {
        benchmark::DoNotOptimize(x += i);
      }
  }
}

DoNotOptimize(<expr>) forces the result of <expr> to be stored in either
memory or a register. For GNU based compilers it acts as read/write barrier
for global memory. More specifically it forces the compiler to flush pending
writes to memory and reload any other values as necessary.

Note that DoNotOptimize(<expr>) does not prevent optimizations on <expr>
in any way. <expr> may even be removed entirely when the result is already
known. For example:

// Example 1: `<expr>` is removed entirely.
  int foo(int x) { return x + 42; }
  while (...) DoNotOptimize(foo(0)); // Optimized to DoNotOptimize(42);

  // Example 2: Result of '<expr>' is only reused.
  int bar(int) __attribute__((const));
  while (...) DoNotOptimize(bar(0)); // Optimized to:
  // int __result__ = bar(0);
  // while (...) DoNotOptimize(__result__);

The second tool for preventing optimizations is ClobberMemory(). In essence
ClobberMemory() forces the compiler to perform all pending writes to global
memory. Memory managed by block scope objects must be "escaped" using
DoNotOptimize(...) before it can be clobbered. In the below example
ClobberMemory() prevents the call to v.push_back(42) from being optimized
away.

static void BM_vector_push_back(benchmark::State& state) {
  for (auto _ : state) {
    std::vector<int> v;
    v.reserve(1);
    auto data = v.data();           // Allow v.data() to be clobbered. Pass as non-const
    benchmark::DoNotOptimize(data); // lvalue to avoid undesired compiler optimizations
    v.push_back(42);
    benchmark::ClobberMemory(); // Force 42 to be written to memory.
  }
}

Note that ClobberMemory() is only available for GNU or MSVC based compilers.

Statistics: Reporting the Mean, Median and Standard Deviation / Coefficient of variation of Repeated Benchmarks

By default each benchmark is run once and that single result is reported.
However benchmarks are often noisy and a single result may not be representative
of the overall behavior. For this reason it's possible to repeatedly rerun the
benchmark.

The number of runs of each benchmark is specified globally by the
--benchmark_repetitions flag or on a per benchmark basis by calling
Repetitions on the registered benchmark object. When a benchmark is run more
than once the mean, median, standard deviation and coefficient of variation
of the runs will be reported.

Additionally the --benchmark_report_aggregates_only={true|false},
--benchmark_display_aggregates_only={true|false} flags or
ReportAggregatesOnly(bool), DisplayAggregatesOnly(bool) functions can be
used to change how repeated tests are reported. By default the result of each
repeated run is reported. When report aggregates only option is true,
only the aggregates (i.e. mean, median, standard deviation and coefficient
of variation, maybe complexity measurements if they were requested) of the runs
is reported, to both the reporters - standard output (console), and the file.
However when only the display aggregates only option is true,
only the aggregates are displayed in the standard output, while the file
output still contains everything.
Calling ReportAggregatesOnly(bool) / DisplayAggregatesOnly(bool) on a
registered benchmark object overrides the value of the appropriate flag for that
benchmark.

Custom Statistics

While having these aggregates is nice, this may not be enough for everyone.
For example you may want to know what the largest observation is, e.g. because
you have some real-time constraints. This is easy. The following code will
specify a custom statistic to be calculated, defined by a lambda function.

void BM_spin_empty(benchmark::State& state) {
  for (auto _ : state) {
    for (int x = 0; x < state.range(0); ++x) {
      benchmark::DoNotOptimize(x);
    }
  }
}

BENCHMARK(BM_spin_empty)
  ->Repetitions(3) // or add option --benchmark_repetitions=3
  ->ComputeStatistics("max", [](const std::vector<double>& v) -> double {
    return *(std::max_element(std::begin(v), std::end(v)));
  })
  ->Arg(512);

While usually the statistics produce values in time units,
you can also produce percentages:

void BM_spin_empty(benchmark::State& state) {
  for (auto _ : state) {
    for (int x = 0; x < state.range(0); ++x) {
      benchmark::DoNotOptimize(x);
    }
  }
}

BENCHMARK(BM_spin_empty)
  ->Repetitions(3) // or add option --benchmark_repetitions=3
  ->ComputeStatistics("ratio", [](const std::vector<double>& v) -> double {
    return v.front() / v.back();
  }, benchmark::StatisticUnit::kPercentage)
  ->Arg(512);

Memory Usage

It's often useful to also track memory usage for benchmarks, alongside CPU
performance. For this reason, benchmark offers the RegisterMemoryManager
method that allows a custom MemoryManager to be injected.

If set, the MemoryManager::Start and MemoryManager::Stop methods will be
called at the start and end of benchmark runs to allow user code to fill out
a report on the number of allocations, bytes used, etc.

This data will then be reported alongside other performance data, currently
only when using JSON output.

Profiling

It's often useful to also profile benchmarks in particular ways, in addition to
CPU performance. For this reason, benchmark offers the RegisterProfilerManager
method that allows a custom ProfilerManager to be injected.

If set, the ProfilerManager::AfterSetupStart and
ProfilerManager::BeforeTeardownStop methods will be called at the start and
end of a separate benchmark run to allow user code to collect and report
user-provided profile metrics.

Output collected from this profiling run must be reported separately.

Using RegisterBenchmark(name, fn, args...)

The RegisterBenchmark(name, func, args...) function provides an alternative
way to create and register benchmarks.
RegisterBenchmark(name, func, args...) creates, registers, and returns a
pointer to a new benchmark with the specified name that invokes
func(st, args...) where st is a benchmark::State object.

Unlike the BENCHMARK registration macros, which can only be used at the global
scope, the RegisterBenchmark can be called anywhere. This allows for
benchmark tests to be registered programmatically.

Additionally RegisterBenchmark allows any callable object to be registered
as a benchmark. Including capturing lambdas and function objects.

For Example:

auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ };

int main(int argc, char** argv) {
  for (auto& test_input : { /* ... */ })
      benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input);
  benchmark::Initialize(&argc, argv);
  benchmark::RunSpecifiedBenchmarks();
  benchmark::Shutdown();
}

Exiting with an Error

When errors caused by external influences, such as file I/O and network
communication, occur within a benchmark the
State::SkipWithError(const std::string& msg) function can be used to skip that run
of benchmark and report the error. Note that only future iterations of the
KeepRunning() are skipped. For the ranged-for version of the benchmark loop
Users must explicitly exit the loop, otherwise all iterations will be performed.
Users may explicitly return to exit the benchmark immediately.

The SkipWithError(...) function may be used at any point within the benchmark,
including before and after the benchmark loop. Moreover, if SkipWithError(...)
has been used, it is not required to reach the benchmark loop and one may return
from the benchmark function early.

For example:

static void BM_test(benchmark::State& state) {
  auto resource = GetResource();
  if (!resource.good()) {
    state.SkipWithError("Resource is not good!");
    // KeepRunning() loop will not be entered.
  }
  while (state.KeepRunning()) {
    auto data = resource.read_data();
    if (!resource.good()) {
      state.SkipWithError("Failed to read data!");
      break; // Needed to skip the rest of the iteration.
    }
    do_stuff(data);
  }
}

static void BM_test_ranged_fo(benchmark::State & state) {
  auto resource = GetResource();
  if (!resource.good()) {
    state.SkipWithError("Resource is not good!");
    return; // Early return is allowed when SkipWithError() has been used.
  }
  for (auto _ : state) {
    auto data = resource.read_data();
    if (!resource.good()) {
      state.SkipWithError("Failed to read data!");
      break; // REQUIRED to prevent all further iterations.
    }
    do_stuff(data);
  }
}

A Faster KeepRunning Loop

A ranged-based for loop should be used in preference to
the KeepRunning loop for running the benchmarks. For example:

static void BM_Fast(benchmark::State &state) {
  for (auto _ : state) {
    FastOperation();
  }
}
BENCHMARK(BM_Fast);

The reason the ranged-for loop is faster than using KeepRunning, is
because KeepRunning requires a memory load and store of the iteration count
ever iteration, whereas the ranged-for variant is able to keep the iteration count
in a register.

For example, an empty inner loop of using the ranged-based for method looks like:

asm

# Loop Init
  mov rbx, qword ptr [r14 + 104]
  call benchmark::State::StartKeepRunning()
  test rbx, rbx
  je .LoopEnd
.LoopHeader: # =>This Inner Loop Header: Depth=1
  add rbx, -1
  jne .LoopHeader
.LoopEnd:

Compared to an empty KeepRunning loop, which looks like:

asm

.LoopHeader: # in Loop: Header=BB0_3 Depth=1
  cmp byte ptr [rbx], 1
  jne .LoopInit
.LoopBody: # =>This Inner Loop Header: Depth=1
  mov rax, qword ptr [rbx + 8]
  lea rcx, [rax + 1]
  mov qword ptr [rbx + 8], rcx
  cmp rax, qword ptr [rbx + 104]
  jb .LoopHeader
  jmp .LoopEnd
.LoopInit:
  mov rdi, rbx
  call benchmark::State::StartKeepRunning()
  jmp .LoopBody
.LoopEnd:

Unless C++03 compatibility is required, the ranged-for variant of writing
the benchmark loop should be preferred.

Disabling CPU Frequency Scaling

If you see this error: