rr Documentation

Technical Documentation: rr-debugger/rr

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Overview

rr is a lightweight tool for recording, replaying and debugging execution of applications (trees of processes and threads).
Debugging extends gdb with very efficient reverse-execution, which in combination with standard gdb/x86 features like hardware data watchpoints, makes debugging much more fun. More information about the project, including instructions on how to install, run, and build rr, is at https://rr-project.org. The best technical overview is currently the paper Engineering Record And Replay For Deployability: Extended Technical Report.

Or go directly to the installation and building instructions.

Please contribute! Make sure to review the pull request checklist before submitting a pull request.

If you find rr useful, please add a testimonial.

rr development is sponsored by Pernosco and was originated by Mozilla.

System requirements

• Linux kernel >= 4.7 (for support of __WALL in waitid())
  • rr 5.6.0 worked with kernel 3.11 (requiring PTRACE_SETSIGMASK)
• rr currently requires either:
  • An Intel CPU with Nehalem (2010) or later microarchitecture.
  • Certain AMD Zen or later processors (see https://github.com/rr-debugger/rr/wiki/Zen)
  • Certain AArch64 microarchitectures (e.g. ARM Neoverse N1 or the Apple Silicon M-series)
• Running in a VM guest is supported, as long as the VM supports virtualization of hardware performance counters.
  • Some configurations of VMware and KVM are known to work.
  • Some configurations of AWS and GCP VMs are known to work.
  • No configurations of Xen are known to work.

AI usage policy

When submitting issues or PRs, please clearly indicate how LLMs were used. Issues or PRs which appear LLM-generated but carry no such indications may be closed without comment.

When submitting PRs for review, the submitter must be human and take full responsibility for understanding the changes, regardless of how LLMs were used.

Overview (`.android/README.md`) Copy Section

Building for Android

To build for Android (from the root of the rr source tree):

docker build .android -t rr-android
mkdir -p obj/dist
docker run -it --rm \
    -u $UID:$GID \
    -v $(pwd):/src/rr \
    -v $(pwd)/obj/dist:/dist \
    rr-android

-u $UID:GID ensures that the build runs with your current UID/GID, which is
necessary to avoid the output being only writable by root.

-v $(pwd):/src/rr mounts the source tree in the container so it can be built.

-v $(pwd)/obj/dist:/dist sets the output directory for the container to the
current directory. The last step of the build will copy the rr tarball to the
directory on the left of :.

Overview (`release-process/README.md`) Copy Section

Setting up AWS-based rr release testing

• Create an AWS account.
• Switch to the us-east-2 (Ohio) region. The AMI IDs under distro-configs are all for the us-east-2 region so this region must be used.
• Use the EC2 console to create a keypair named rr-testing. This will download a file called rr-testing.pem containing the private key; move it somewhere safe and chmod go-r /rr-testing.pem to make ssh happy.
• Install boto3 locally, e.g. using pip install boto3.
• Install aws-cli locally, e.g. using these instructions.
• Set AWS_DEFAULT_REGION=us-east-2, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your environment.
• Create AWS resources using aws cloudformation create-stack --stack-name rr-testing --template-body file://path/to/rr-testing-cloud-formation.json.
  • In the future, use aws cloudformation update-stack --stack-name rr-testing --template-body file://path/to/rr-testing-cloud-formation.json to update when that configuration file changes.

Preparing a release

If you want to support any new distro versions, add files to release-process/distro-configs. Likewise remove any config files for distro versions no longer supported. Make sure to commit all changes to the local master branch before runnging prepare-release.py. Then:

release-process/prepare-release.py <major>.<minor>.0 <path>/rr-testing.pem

This will (re)create a release branch in Github, then start a number of EC2 VMs for testing and packaging. When a VM successfully completes its work, you will see lines like

Tests succeeded for centos9 arm64

When a VM fails, you will see something like

Tests failed (VM kept): see /tmp/rr-release-logs/debian12.arm64

The output log should help diagnose the problem. At the end of the log is an SSH command you can use to log into the VM for further debugging. Make sure to shut down any failing VMs when you're done, either using sudo shutdown now in an SSH session, or from the AWS EC2 console.

Repeat until there are no more failures or any failures are ignorable (e.g. due to #3607). The built packages will be in /tmp/rr-dist; there should be two .debs, two .rpms, and two .tar.gzs.

Completing the release

• Cherry-pick release to master and update tag: git cherry-pick release; git tag -f <major>.<minor>.0 (Not sure why the tag needs to be updated...)
• Delete the local and remote release branches: git branch -d release; git push origin -d release
• Push changes to Github: git push origin; git push --tags origin. Once this is done there is no turning back!
• Create release and upload packages from /tmp/rr-dist
• Update gh-pages: ./scripts/update-gh-pages.sh && git push origin
• Update News wiki page
• Post to rr-devel mailing list.

Testing a specific distro/CPU configuration

The release-process/test-system.py script can be used to start an AWS EC2 VM to test a specific distro version and CPU combination. E.g.

release-process/test-system.py --keep-vm release-process/distro-configs/centos9.json arm64 ~/rr-testing.pem >& /tmp/output

This will start the VM, build rr, and run tests. Whether or not the tests succed, the VM will keep running and /tmp/output will end with an SSH command to log into the VM --- very handy for testing new distro versions or just running rr tests on a beefy machine. test-system.py has various options for customizing the instance type etc.

This is helpful for debugging intermittent rr test failures. For example, in a VM, you can run

cd ~/rr/src/test
for i in `seq 1 100`; do bash cont_race.run & done

to run 100 parallel instances of the cont_race test. If that doesn't show your bug, try 1000 instances instead. Running instances in parallel stresses thread scheduling and exposes more bugs.

Overview (`src/counters-test/README.md`) Copy Section

counters test

This does a quick check to determine whether hardware performance counter measurement and interrupts, and the kernel/hypervisor support for them, are working as expected. It's useful for testing PMU infrastructure without the full complexity of rr.

It copies some code from rr sources to make a standalone test for easier debugging and/or inclusion in CI tests for other projects. It's under the same MIT-style license as the rest of the rr code.

Basic test Copy Chapter

Run with

g++ counters.cc -o /tmp/counters && /tmp/counters

Intensive interrupt testing Copy Chapter

counters takes optional command-line parameters:

/tmp/counters [number of interrupts to test] [interrupt period]

The number of interrupts to test defaults to 1. The interrupt period defaults to 1000000.

To test interrupts thoroughly, try something like

for i in `seq 1 1000`; do /tmp/counters 1000 10000 > /tmp/output$i & done

Don't run this on your laptop! This tests 1M interrupts with the machine under load. If there are failures you will see lines like

[425]   Aborted                 (core dumped) ~/obj/bin/counters 1000 10000 > /tmp/output$i

Overview (`third-party/brotli/README.md`) Copy Section

Origin and License Copy Section

Taken verbatim from https://github.com/google/brotli/commit/c1bd196833e4ce05ef49b82ec5e124bb79e095ff
below "c", with the exception of this file and removal of the directories "fuzz" and "tools".

Sources are under an MIT-style license. Also, Google has made a RF commitment here:
https://datatracker.ietf.org/ipr/2396
Excerpt:
"In addition, to facilitate the use of the Brotli Compression Algorithm in
other applications, Google hereby provides a W3C RF commitment (that is not
limited to font data) to the Brotli Compression Algorithm specification
linked above and, if that specification leads to a final IETF RFC for
Brotli, then to the specification in that RFC."

Introduction to Brotli Copy Section

Brotli is a generic-purpose lossless compression algorithm that compresses data
using a combination of a modern variant of the LZ77 algorithm, Huffman coding
and 2nd order context modeling, with a compression ratio comparable to the best
currently available general-purpose compression methods. It is similar in speed
with deflate but offers more dense compression.

The specification of the Brotli Compressed Data Format is defined in RFC 7932.

Brotli is open-sourced under the MIT License, see the LICENSE file.

Please note: brotli is a "stream" format; it does not contain
meta-information, like checksums or uncompresssed data length. It is possible
to modify "raw" ranges of the compressed stream and the decoder will not
notice that.

--- METRICS ---

• Files Extracted: 5
• Estimated Token Budget: ~2484 tokens
• Canonical Reference: https://codewiki.google/github.com/rr-debugger/rr