## File: README.md # Coz: Finding Code that Counts with Causal Profiling by [Charlie Curtsinger](https://curtsinger.cs.grinnell.edu/) and [Emery Berger](https://emeryberger.com) [](https://crates.io/crates/coz) Coz is a profiler for native code (C/C++/Rust) that unlocks optimization opportunities missed by traditional profilers. Coz employs a novel technique called *causal profiling* that measures optimization potential. It predicts what the impact of optimizing code will have on overall throughput or latency. Profiles generated by Coz show the "bang for buck" of optimizing a line of code in an application. In the below profile, almost every effort to optimize the performance of this line of code directly leads to an increase in overall performance, making it an excellent candidate for optimization efforts. Coz's measurement matches developers' assumptions about profilers: that optimizing highly-ranked code will have the greatest impact on performance. Causal profiling measures optimization potential for serial, parallel, and asynchronous programs without instrumentation of special handling for library calls and concurrency primitives. Instead, a causal profiler uses performance experiments to predict the effect of optimizations. This allows the profiler to establish causality: "optimizing function X will have effect Y," exactly the measurement developers had assumed they were getting all along. Full details of Coz are available in our paper, [Coz: Finding Code that Counts with Causal Profiling (pdf)](http://arxiv.org/pdf/1608.03676v1.pdf), SOSP 2015, October 2015 (recipient of a Best Paper Award). [](http://www.youtube.com/watch?v=jE0V-p1odPg&t=0m28s "Coz presentation at SOSP") ## Installation ### Pre-built Packages (Recommended) Download the latest release for your platform from the [GitHub Releases page](https://github.com/plasma-umass/coz/releases). **Debian/Ubuntu (`.deb`):** ```shell # Download the .deb for your architecture (amd64 or arm64) sudo dpkg -i coz_VERSION_amd64.deb ``` **Fedora/RHEL/CentOS (`.rpm`):** ```shell # Download the .rpm for your architecture (x86_64 or aarch64) sudo rpm -i coz-VERSION-1.x86_64.rpm ``` **Generic Linux (tarball):** ```shell tar xzf coz-VERSION-linux-x86_64.tar.gz cd coz-VERSION-linux-x86_64 sudo ./install.sh # Installs to /usr/local by default sudo ldconfig ``` ### Requirements Coz works on Linux systems (running version 2.6.32 or later, with support for the `perf_event_open` system call) and macOS (using Apple's kperf framework). Both platforms require a Python 3.x interpreter. **macOS Note**: The macOS port uses Apple's private kperf framework for sampling. This requires either running with elevated privileges or adjusting System Integrity Protection settings. The kperf API is undocumented and may change in future macOS versions. ## Libraries/Wrappers By default, Coz works for C, C++, and Rust programs. It has been ported or has wrappers for several other languages, listed below: | Language | Link | | ----------- | ----------- | Java | JCoz: https://github.com/Decave/JCoz| | Go | Cozgo: https://github.com/urjitbhatia/cozgo| | Swift | Swift Coz: https://github.com/funcmike/swift-coz | ## Building Coz From Source ### Install build prerequisites On Debian/Ubuntu: ```shell sudo apt-get install -y build-essential cmake pkg-config ``` libelfin is fetched automatically during the build, so no additional dependencies are required. ### Configure and build Use the standard out-of-source workflow (shown with `build/`, but any directory works): ```shell cmake -S . -B build # Configure (defaults to Release with debug info) cmake --build build -j # Build libcoz and the CLI sudo cmake --install build # Install to /usr/local (or CMAKE_INSTALL_PREFIX) sudo ldconfig # Update shared library cache ``` Before running Coz on Linux, relax `perf_event_paranoid` so sampling works: ```shell sudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid' ``` ### Building the Benchmarks The benchmark suite is off by default because it pulls in extra dependencies. Enable it when configuring: ```shell cmake -S . -B build-bench -DBUILD_BENCHMARKS=ON cmake --build build-bench -j ``` When `BUILD_BENCHMARKS` is set, CMake automatically switches the build type to `RelWithDebInfo` (or keeps `Debug`) so DWARF line tables are available. Benchmark binaries live under `build-bench/benchmarks//`. A number of the benchmarks are from the Phoenix benchmark suite, and several require data files. These are available for download via links in the README from [the Phoenix repository](https://github.com/kozyraki/phoenix). ### Viewer After profiling, run `coz plot` to automatically open your results in the browser. To view a specific profile, use `coz plot -i /path/to/profile.coz`. You may need to adjust the "Minimum Points" slider to see results if the profile has limited data. #### AI-Powered Optimization Suggestions The viewer includes an AI assistant that analyzes causal profiling results and suggests concrete optimizations. Click the magic wand icon on any plot to get context-aware suggestions based on the profiling data and source code. Supported LLM providers: - **Anthropic** (Claude) — set `ANTHROPIC_API_KEY` - **OpenAI** (GPT-4o, o3, etc.) — set `OPENAI_API_KEY` - **Amazon Bedrock** — set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` - **Ollama** — local models, no API key required The viewer dynamically fetches available models from each provider and caches them locally. Provider settings, API keys, and model selections persist across sessions via cookies. ## Using Coz Using Coz requires a small amount of setup, but you can jump ahead to the section on the included [sample applications](#sample-applications) in this repository if you want to try Coz right away. To run your program with Coz, you will need to build it with debug information (`-g`). Coz now supports modern DWARF versions (including DWARF 5), so you can use your compiler's default debug format. You do not need to include debug symbols in the main executable: coz uses the same procedure as `gdb` to locate debug information for stripped binaries. Once you have your program built with debug information, you can run it with Coz using the command `coz run {coz options} --- {program name and arguments}`. But, to produce a useful profile you need to decide which part(s) of the application you want to speed up by specifying one or more progress points. ### Profiling Modes Coz departs from conventional profiling by making it possible to view the effect of optimizations on both throughput and latency. To profile throughput, you must specify a progress point. To profile latency, you must specify a pair of progress points. #### Throughput Profiling: Specifying Progress Points To profile throughput you must indicate a line in the code that corresponds to the end of a unit of work. For example, a progress point could be the point at which a transaction concludes, when a web page finishes rendering, or when a query completes. Coz then measures the rate of visits to each progress point to determine any potential optimization's effect on throughput. To place a progress point, include `coz.h` (under the `include` directory in this repository) and add the `COZ_PROGRESS` macro to at least one line you would like to execute more frequently. Don't forget to link your program with libdl: use the `-ldl` option. By default, Coz uses the source file and line number as the name for your progress points. If you use `COZ_PROGRESS_NAMED("name for progress point")` instead, you can provide an informative name for your progress points. This also allows you to mark multiple source locations that correspond to the same progress point. #### Latency Profiling: Specifying Progress Points To profile latency, you must place two progress points that correspond to the start and end of an event of interest, such as when a transaction begins and completes. Simply mark the beginning of a transaction with the `COZ_BEGIN("transaction name")` macro, and the end with the `COZ_END("transaction name")` macro. Unlike regular progress points, you always need to specify a name for your latency progress points. Don't forget to link your program with libdl: use the `-ldl` option. When coz tests a hypothetical optimization it will report the effect of that optimization on the average latency between these two points. Coz can track this information without any knowledge of individual transactions thanks to [Little's Law](https://en.wikipedia.org/wiki/Little%27s_law). ### AI-Suggested Progress Points (`coz suggest-points`) If you're new to Coz or working in an unfamiliar codebase, the hardest part is deciding *where* to place progress points. The `coz suggest-points` subcommand uses an LLM agent to read your source, identify what counts as a unit of work, and propose concrete `COZ_PROGRESS_NAMED` / `COZ_BEGIN` / `COZ_END` placements. Each proposal is shown with a rationale and a unified diff; nothing is written until you confirm. ```shell export ANTHROPIC_API_KEY=... coz suggest-points src/ # explore src/, then prompt to apply coz suggest-points --dry-run src/ # print diffs only, no prompt coz suggest-points --apply src/ # skip the prompt, apply everything coz suggest-points --kind latency src/ # only propose COZ_BEGIN/COZ_END pairs coz suggest-points --hint "HTTP server" src/ # give the agent a domain hint ``` The agent has read-only tools (`list_files`, `read_file`, `grep`) scoped to the paths you pass, and emits one proposal per call. When you accept a proposal, the macro is inserted at the chosen line (matching surrounding indentation), `#include "coz.h"` is added if missing, and the original file is backed up to `*.coz.bak`. Latency points are only applied in matched `BEGIN`/`END` pairs. Files that already contain Coz macros are left alone. #### Choosing an LLM provider `coz suggest-points` supports the same providers as the viewer's optimization assistant. Pick one with `--provider`; the agentic tool-use loop is used on all of them, so results are comparable across providers (quality depends on how well the chosen model follows tool-use instructions). | Provider | Flag | Credentials | Default model | | --- | --- | --- | --- | | Anthropic (Claude) | `--provider anthropic` *(default)* | `ANTHROPIC_API_KEY` (or `--api-key`) | `claude-opus-4-7` | | OpenAI (GPT-4o, o3, …) | `--provider openai` | `OPENAI_API_KEY` (or `--api-key`) | `gpt-4o` | | Amazon Bedrock | `--provider bedrock` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `AWS_SESSION_TOKEN`); region via `AWS_REGION` / `AWS_DEFAULT_REGION` or `--region`. Any credential source that boto3's default chain picks up (IAM role, `~/.aws/credentials`, SSO) also works. Requires `pip install boto3`. | `anthropic.claude-opus-4-20250514-v1:0` | | Ollama (local) | `--provider ollama` | No key. Endpoint via `OLLAMA_HOST` or `--ollama-host` (defaults to `http://localhost:11434`). Use a tool-capable model (e.g. `llama3.1`, `qwen2.5`). | `llama3.1` | Examples: ```shell # Anthropic (default) export ANTHROPIC_API_KEY=... coz suggest-points src/ # OpenAI export OPENAI_API_KEY=... coz suggest-points --provider openai --model gpt-4o src/ # Amazon Bedrock — uses the boto3 credential chain export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-west-2 coz suggest-points --provider bedrock \ --model anthropic.claude-opus-4-20250514-v1:0 src/ # Ollama on a local server (no API key) coz suggest-points --provider ollama --model llama3.1 src/ ``` You can also pass `--api-key ` inline for Anthropic or OpenAI instead of exporting an env var, and `--model ` to select a specific model on any provider. See `coz suggest-points --help` for the full option list (`--include`, `--exclude`, `--max-points`, `--region`, `--ollama-host`, etc.). ### Specifying Progress Points on the Command Line Coz has command line options to specify progress points when profiling the application instead of modifying its source. This feature is currently disabled because it did not work particularly well. Adding support for better command line-specified progress points is planned in the near future. ## Processing Results Run `coz plot` to view your profile in the browser. Use `coz plot --text` for terminal output, or `coz plot --text --verbose` for detailed data points. ## Sample Applications The `benchmarks/` directory includes several small programs with progress points already wired up. Once you configure with `-DBUILD_BENCHMARKS=ON` (see above), you can run them straight from the build tree: ```shell ./build-bench/benchmarks/toy/toy coz run --- ./build-bench/benchmarks/toy/toy ``` These programs may need several runs before Coz accumulates enough samples to emit a useful profile. Run `coz plot` to view the results. ## CMake When you install coz it installs a cmake config file. To add coz to a cmake project simply use the command `find_package(coz-profiler)`. This will import a target for the library and includes called `coz::coz` and a target for the coz binary `coz::profiler`. For guidance on how to use these targets refer to the CMake documentation. ## Limitations Coz currently does not support interpreted or JIT-compiled languages such as Python, Ruby, or JavaScript. Interpreted languages will likely not be supported at any point, but support for JIT-compiled languages that produce debug information could be added in the future. ## License All source code is licensed under the BSD 2-clause license unless otherwise indicated. See LICENSE.md for details. Sample applications (in the `benchmarks` directory) include several [Phoenix](https://github.com/kozyraki/phoenix) programs and [pbzip2](http://compression.ca/pbzip2/), which are licensed separately and included with this release for convenience. --- ## File: docs/coz.rst ===== coz ===== -------------------------------------------------- profiler running experiments on multithreaded code -------------------------------------------------- :Author: Emery Berger - emery@cs.umass.edu :Author: Charlie Curtsinger - curtsinger@grinnell.edu :Date: 2017-08-06 :Copyright: public domain :Version: 0.2 :Manual section: 1 :Manual group: User Commands SYNOPSIS ======== coz run [profiling options] --- [args] coz plot [-h] DESCRIPTION =========== Coz is a new kind of profiler that unlocks optimization opportunities missed by traditional profilers. Coz employs a novel technique we call *causal profiling* that measures optimization potential. This measurement matches developers' assumptions about profilers: that optimizing highly-ranked code will have the greatest impact on performance. Causal profiling measures optimization potential for serial, parallel, and asynchronous programs without instrumentation of special handling for library calls and concurrency primitives. Instead, a causal profiler uses performance experiments to predict the effect of optimizations. This allows the profiler to establish causality: "optimizing function X will have effect Y," exactly the measurement developers had assumed they were getting all along. OPTIONS ======= -h, --help show this help message and exit --binary-scope , -b Profile matching executables. Use '%' as a wildcard, or 'MAIN' to include the main executable (default=MAIN) --source-scope , -s Profile matching source files. Use '%' as a wildcard. (default=%) --progress :, -p : [NOT SUPPORTED] Add a sampling-based progress point --output , -o Profiler output (default=`profile.coz`) --end-to-end Run a single performance experiment per-execution --fixed-line : Evaluate optimizations of a specific source line --fixed-speedup (0-100) Evaluate optimizations of a specific amount SEE ALSO ======== * ``__ * ``man gdb`` --- ## File: docs/README.md The file `coz.1` was automatically generated by using `rst2man`: ``` rst2man coz.rst coz.1 ``` To install `rst2man` on Linux systems, execute the following command: ``` sudo apt-get install python-docutils ``` --- ## File: rust/README.md # coz-rs Rust support for the [`coz` Causal Profiler](https://github.com/plasma-umass/coz) [](https://img.shields.io/crates/d/coz)[](https://docs.rs/coz) ## Usage First, follow the instructions in [`coz`] to install the `coz` command. [`coz`]: https://github.com/plasma-umass/coz/#installation Next, `coz` is a profiler that, for the best results, typically requires source-level modifications of your code. To do this first add this to your `Cargo.toml` ```toml [dependencies] coz = "0.1" ``` Then you'll want to either add throughput or latency tracepoints. More information on this [can be found upstream](https://github.com/plasma-umass/coz/#profiling-modes). If you have something you'd wish would execute more often, you can add: ```rust fn main() { loop { // ... // For example you wish this `loop` executed more iterations coz::progress!(); // equivalent of `COZ_PROGRESS` } } ``` Note that `coz::progress!("name")` is the equivalent of `COZ_PROGRESS_NAMED` as well. If you'd like to profile the latency of an operation you can instead use: ```rust // Boy I wish this function executed more quickly... fn foo() { coz::scope!("foo"); } ``` Instead of `scope!` you may also use `coz::begin!("foo"); ... coz::end!("foo");`. After you've instrumented your code, you need to also ensure that you're compiling with DWARF debug information. To do this you'll want to configure `Cargo.toml` again: ```toml [profile.release] debug = 1 ``` Next up you'll build your application with `cargo build --release`, and then finally you can run it with `coz run --- ./target/release/$your_binary`. ## Caveats Known caveats so far to generate a report that collects information are: * Debug information looks to be critical to get a report from `coz`. Make sure that your program is compiled with at least line-table information (`debug = 1`) to ensure you get the best experience using `coz`. * `coz` works on Linux and macOS. This crate compiles on all platforms, but only does something when the `libcoz` runtime is injected (which is what `coz run` does). ## Examples You can find an example toy program at `rust/examples/toy.rs` in this repository, and we can execute it with `coz`: NOTE: If Rust is not yet installed, first run `sudo apt install curl` and then `curl --proto '=https' --tlsv1.3 https://sh.rustup.rs -sSf | sh`. ``` $ cargo build --release --examples $ coz run --- ./target/release/examples/toy ... [profiler.cpp:75] Starting profiler thread $ ``` That should generate `profile.coz` in the current directory, which if you plot with `coz plot` should look something like this: --- ## File: viewer/README.md # Causal Profile Viewer Prerequisites for building: * Install NodeJS. To build, run `npm install`. To use, simply run an HTTP server from the directory. For example: 1. Install `http-server`: `npm i http-server` 2. Run `http-server` in this directory 3. Open a browser to `http://localhost:8080/`