gitoxide Documentation

An idiomatic, lean, fast & safe pure Rust implementation of Git

12,022 stars Rust Rust 1 file · ~13,504 tokens #blazingly-fast#built-with-rust#cli#git#version-control
RAW Doc

Technical Documentation: GitoxideLabs/gitoxide

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)


gitoxide is an implementation of git written in Rust for developing future-proof applications which strive for correctness and
performance while providing a pleasant and unsurprising developer experience.

There are two primary ways to use gitoxide:

  1. As Rust library: Use the gix crate as a Cargo dependency for API access.
  2. As command-line tool: The gix binary as development tool to help testing the API in real repositories,
    and the ein binary with workflow-enhancing tools. Both binaries may forever be unstable,
    do not rely on them in scripts.

Development Status

The command-line tools as well as the status of each crate is described in
the crate status document.

For use in applications, look for the gix crate,
which serves as entrypoint to the functionality provided by various lower-level plumbing crates like
gix-config.

Feature Discovery

Can gix do what I need it to do?

The above can be hard to answer and this paragraph is here to help with feature discovery.

Look at crate-status.md for a rather exhaustive document that contains
both implemented and planned features.

Further, the gix crate documentation with the git2 search term helps to find all currently
known git2 equivalent method calls. Please note that this list is definitely not exhaustive yet, but might help if you are coming from git2.

What follows is a high-level list of features and those which are planned:

  • clone
  • fetch
  • push
  • blame (plumbing)
  • status
  • blob and tree-diff
  • merge
    • blobs
    • trees
    • commits
  • commit
    • hooks
  • commit-graph traversal
  • rebase
  • worktree checkout and worktree stream
  • reset
  • reading and writing of objects
  • reading and writing of refs
  • reading and writing of .git/index
  • reading and writing of git configuration
  • pathspecs
  • revspecs
  • .gitignore and .gitattributes

Crates

Follow linked crate name for detailed status. Please note that all crates follow semver as well as the stability guide.

Production Grade

Stabilization Candidates

Crates that seem feature complete and need to see some more use before they can be released as 1.0.
Documentation is complete and was reviewed at least once.

Initial Development

These crates may be missing some features and thus are somewhat incomplete, but what's there
is usable to some extent.

Stress Testing

  • Verify huge packs
  • Explode a pack to disk
  • Generate and verify large commit graphs
  • Generate huge pack from a lot of loose objects

Stability and MSRV

Our stability guide helps to judge how much churn can be expected when depending on crates in this workspace.

Installation

Download a Binary Release

Using cargo binstall, one is able to fetch binary releases. You can install it via cargo install cargo-binstall, assuming
the rust toolchain is present.

Then install gitoxide with cargo binstall gitoxide.

See the releases section for manual installation and various alternative builds that are slimmer or smaller, depending
on your needs, for Linux, MacOS and Windows.

Install with Homebrew

For macOS and Linux, gitoxide can be installed from Homebrew:

sh
brew install gitoxide

Download from Arch Linux repository

For Arch Linux you can download gitoxide from community repository:

sh
pacman -S gitoxide

Download from Exherbo Linux Rust repository

For Exherbo Linux you can download gitoxide from the Rust repository:

sh
cave resolve -x repository/rust
cave resolve -x gitoxide

From Source via Cargo

cargo is the Rust package manager which can easily be obtained through rustup. With it, you can build your own binary
effortlessly and for your particular CPU for additional performance gains.

The minimum supported Rust version is documented in the Cargo package,
the latest stable one will work as well.

There are various build configurations, all of them are documented here. The documentation should also be useful
for packagers who need to tune external dependencies.

sh
# A way to install `gitoxide` with just Rust and a C compiler installed.
# If there are problems with SSL certificates during clones, try to omit `--locked`.
cargo install gitoxide --locked --no-default-features --features max-pure

# The default installation, 'max', is the fastest, but also needs `cmake` to build successfully.
# Installing it is platform-dependent.
cargo install gitoxide

# For smaller binaries and even faster build times that are traded for a less fancy CLI implementation,
# use the `lean` feature.
cargo install gitoxide --locked --no-default-features --features lean

The following installs the latest unpublished max release directly from git:

sh
cargo install --git https://github.com/GitoxideLabs/gitoxide gitoxide

How to deal with build failures

On some platforms, installation may fail due to lack of tools required by C toolchains. This can generally be avoided by installation with:

sh
cargo install gitoxide --no-default-features --features max-pure

What follows is a list of known failures.

  • On Fedora, perl needs to be installed for OpenSSL to build properly. This can be done with the following command (see issue #592):

    sh
    dnf install perl

Using Docker

Some CI/CD pipelines leverage repository cloning. Below is a copy-paste-able example to build docker images for such workflows.
As no official image exists (at this time), an image must first be built.

[!NOTE]
The dockerfile isn't continuously tested as it costs too much time and thus might already be broken.
PRs are welcome.

Building the most compatible base image

sh
docker build -f etc/docker/Dockerfile.alpine -t gitoxide:latest --compress . --target=pipeline

Basic usage in a Pipeline

For example, if a Dockerfile currently uses something like RUN git clone https://github.com/GitoxideLabs/gitoxide, first build the image:

sh
docker build -f etc/docker/Dockerfile.alpine -t gitoxide:latest --compress .

Then copy the binaries into your image and replace the git directive with a gix equivalent.

dockerfile
COPY --from gitoxide:latest /bin/gix /usr/local/bin/
COPY --from gitoxide:latest /bin/ein /usr/local/bin/

RUN /usr/local/bin/gix clone --depth 1 https://github.com/GitoxideLabs/gitoxide gitoxide

Usage

Once installed, there are two binaries:

  • ein
    • high level commands, porcelain, for every-day use, optimized for a pleasant user experience
  • gix
    • low level commands, plumbing, for use in more specialized cases and to validate newly written code in real-world scenarios

Project Goals

Project goals can change over time as we learn more, and they can be challenged.

  • a pure-rust implementation of git
    • including transport, object database, references, cli and tui
    • a simple command-line interface is provided for the most common git operations, optimized for
      user experience. A simple-git if you so will.
    • be the go-to implementation for anyone who wants to solve problems around git, and become
      the alternative to GitPython and libgit2 in the process.
    • become the foundation for a distributed alternative to GitHub, and maybe even for use within GitHub itself
  • learn from the best to write the best possible idiomatic Rust
    • libgit2 is a fantastic resource to see what abstractions work, we will use them
    • use Rust's type system to make misuse impossible
  • be the best performing implementation
    • use Rust's type system to optimize for work not done without being hard to use
    • make use of parallelism from the get go
    • sparse checkout support from day one
  • assure on-disk consistency
    • assure reads never interfere with concurrent writes
    • assure multiple concurrent writes don't cause trouble
  • take shortcuts, but not in quality
    • binaries use gix::Result and gix::Error to preserve error context and source locations, with helpers available through gix::error instead of a separate gix-error dependency.
    • libraries use light-weight custom errors implemented using quick-error or thiserror.
    • internationalization is nothing we are concerned with right now.
    • IO errors due to insufficient amount of open file handles don't always lead to operation failure
  • Cross platform support, including Windows
    • With the tools and experience available here there is no reason not to support Windows.
    • Windows is tested on CI
      and failures do prevent releases.

Non-Goals

Project non-goals can change over time as we learn more, and they can be challenged.

  • replicate git command functionality perfectly
    • git is git, and there is no reason to not use it. Our path is the one of simplicity to make
      getting started with git easy.
  • be incompatible to git
    • the on-disk format must remain compatible, and we will never contend with it.
  • use async IO everywhere
    • for the most part, git operations are heavily reliant on memory mapped IO as well as CPU to decompress data,
      which doesn't lend itself well to async IO out of the box.
    • Use blocking as well as gix_utils::interrupt to bring operations into the async world and to control
      long running operations.
    • When connecting or streaming over TCP connections, especially when receiving on the server, async seems like a must
      though, but behind a feature flag.

Contributions

If what you have seen so far sparked your interest to contribute, then let us say: We are happy to have you and help you to get started. What's paricularly welcome is the discovery of deviations from Git, or the prodding
and probing to find bugs and other forms of undesirable behaviour.

Please read the contribution guide before starting. It covers when prior discussion is required and how to prepare a submission.

Getting started with Video Tutorials

  • Learning Rust with Gitoxide
    • In 17 episodes you can learn all you need to meaningfully contribute to gitoxide.
  • Getting into Gitoxide
    • Get an introduction to gitoxide itself which should be a good foundation for any contribution, but isn't a requirement for contributions either.
  • Gifting Gitoxide
    • See how PRs are reviewed along with a lot of inner monologue.

Other Media

Roadmap

Features for 1.0

Provide a CLI to for the most basic user journey:

  • initialize a repository
  • fetch
    • and update worktree
  • clone a repository
    • bare
    • with working tree
  • create a commit after adding worktree files
  • add a remote
  • push
    • create (thin) pack

Ideas for Examples

  • gix tool open-remote open the URL of the remote, possibly after applying known transformations to go from ssh to https.
  • tix as example implementation of tig, displaying a version of the commit graph, useful for practicing how highly responsive GUIs can be made.
  • Something like git-sizer, but leveraging extreme decompression speeds of indexed packs.
  • Open up SQL for git using sqlite virtual tables. Check out gitqlite
    as well. What would an MVP look like? Maybe even something that could ship with gitoxide. See this go implementation as example.
  • A truly awesome history rewriter which makes it easy to understand what happened while avoiding all pitfalls. Think BFG, but more awesome, if that's possible.
  • gix-tui should learn a lot from fossil-scm regarding the presentation of data. Maybe this can be used for prompts. Probably magit has a lot to offer, too.

Ideas for Spin-Offs

  • A system to integrate tightly with gix-lfs to allow a multi-tier architecture so that assets can be stored in git and are accessible quickly from an intranet location
    (for example by accessing the storage read-only over the network) while changes are pushed immediately by the server to other edge locations, like the cloud or backups. Sparse checkouts along with explorer/finder integrations
    make it convenient to only work on a small subset of files locally. Clones can contain all configuration somebody would need to work efficiently from their location,
    and authentication for the git history as well as LFS resources make the system secure. One could imagine encryption support for untrusted locations in the cloud
    even though more research would have to be done to make it truly secure.
  • A syncthing like client/server application. This is to demonstrate how lower-level crates can be combined into custom applications that use
    only part of git's technology to achieve their very own thing. Watch out for big file support, multi-device cross-syncing, the possibility for
    untrusted destinations using full-encryption, case-insensitive and sensitive filesystems, and extended file attributes as well as ignore files.
  • An event-based database that uses commit messages to store deltas, while occasionally aggregating the actual state in a tree. Of course it's distributed by nature, allowing
    people to work offline.
    • It's abstracted to completely hide the actual data model behind it, allowing for all kinds of things to be implemented on top.
    • Commits probably need a nanosecond component for the timestamp, which can be added via custom header field.
    • having recording all changes allows for perfect merging, both on the client or on the server, while keeping a natural audit log which makes it useful for mission critical
      databases in business.
    • Applications
      • Can markdown be used as database so issue-trackers along with meta-data could just be markdown files which are mostly human-editable? Could user interfaces
        be meta-data aware and just hide the meta-data chunks which are now editable in the GUI itself? Doing this would make conflicts easier to resolve than an sqlite
        database.
      • A time tracker - simple data, very likely naturally conflict free, and interesting to see it in terms of teams or companies using it with maybe GitHub as Backing for authentication.
        • How about supporting multiple different trackers, as in different remotes?

Shortcomings & Limitations

Please take a look at the SHORTCOMINGS.md file for details.

Credits

  • itertools (MIT Licensed)
    • We use the izip! macro in code
  • flate2 (MIT Licensed)
    • We use the high-level flate2 library to implement decompression and compression, which builds on the high-performance zlib-rs crate.

🙏 Special Thanks 🙏

At least for now this section is exclusive to highlight the incredible support that Josh Triplett has provided to me
in the form of advice, sponsorship and countless other benefits that were incredibly meaningful. Going full time with gitoxide would hardly have been
feasible without his involvement, and I couldn't be more grateful 😌.

License

This project is licensed under either of

at your option.

Fun facts

  • Originally @Byron was really fascinated by this problem
    and believes that with gitoxide it will be possible to provide the fastest solution for it.
  • @Byron has been absolutely blown away by git from the first time he experienced git more than 13 years ago, and
    tried to implement it in various shapes and forms
    multiple times. Now with Rust @Byron finally feels to have found the right tool for the job!

Overview (`gix-config/README.md`)

gix-config

gix-config is a library for interacting with git-config files.

This crate intents to be a performant Rust implementation for reading and
writing git-config files. It exposes tiers of abstractions, from simple
config value wrappers to a high level reader and writer.

The parser retains the syntactic structure of the input in a self-contained
representation, allowing configuration files to be reproduced after parsing.
Higher-level abstractions use acceleration structures for efficient lookup.

Currently, this is not a binary. While we do intent to have a drop-in
replacement for the git config sub-command, we're currently missing
system-level abstractions to do so.

Contributing

Contributions are always welcome!

Code quality

This repository enables pedantic, cargo, and nursery clippy lints. Make sure
to run cargo clean && cargo clippy (the clean stage is very important!) to
ensure your code is linted.

Testing

Since this is a performance oriented crate, in addition to well tested code via
cargo test, we also perform benchmarks to measure notable gains or losses in
performance. We use criterion so benches can be run via cargo bench after
installing it via cargo install cargo-criterion.

Changes to parsing code may include a request to fuzz to ensure that it cannot
panic on inputs. This can be done by executing cargo +nightly fuzz run parse after
installing the fuzz sub-command via cargo install cargo-fuzz.

License

Licensed under either of Apache License, Version 2.0 or [MIT license](LICENSE-MIT) at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in gix-config by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Overview (`gix-imara-diff/README.md`)

imara-diff

Modified for gitoxide from the upstream imara-diff crate.
Upstream source: git cat-file -p 32d1e45d3df061e6ccba6db7fdce92db29e345d8:README.md


imara-diff is a solid (imara in swahili) diff library for rust.
Solid refers to the fact that imara-diff provides very good runtime performance even
in pathological cases so that your application never appears to freeze while waiting on a diff.
The performance improvements are achieved using battle tested heuristics used in gnu-diff and git
that are known to perform well while still providing good results.

imara-diff is also designed to be flexible so that it can be used with arbitrary collections and
not just lists and strings and even allows reusing large parts of the computation when
comparing the same file to multiple different files.

imara-diff provides two diff algorithms:

  • The linear-space variant of the well known Myers algorithm
  • The Histogram algorithm which is a variant of the patience diff algorithm.

Myers algorithm has been enhanced with preprocessing and multiple heuristics to ensure fast runtime in pathological
cases to avoid quadratic time complexity and closely matches the behavior of gnu-diff and git.
The histogram algorithm was originally ported from git but has been heavily optimized.
The Histogram algorithm outperforms Myers algorithm by 10% - 100% across a wide variety of workloads.

Limitations

Even with the optimizations in this crate, performing a large diff without any tokenization (like character diff for a string) does not perform well.
To work around this problem a diff of the entire file with large tokens (like lines for a string) can be performed first.
The Sink implementation can then perform fine-grained diff on changed regions.
Note that this fine-grained diff should not be performed for pure insertions, pure deletions and very large changes.

In an effort to improve performance, imara-diff makes heavy use of pointer compression.
That means that it can only support files with at most 2^31 - 2 tokens.
This should be rarely an issue in practice for textual diffs, because most (large) real-world files
have an average line-length of at least 8.
That means that this limitation only becomes a problem for files above 16GB while performing line-diffs.

Benchmarks

The most used diffing libraries in the rust ecosystem are similar and dissimilar.
The fastest diff implementation both of these offer is a simple implementation of Myers algorithm
without preprocessing or additional heuristics.
As these implementations are very similar only similar was included in the benchmark.

To provide a benchmark to reflects real-world workloads, the git history of different open source projects were used.
For each repo two (fairly different) tags were chosen.
A tree diff is performed with gitoxide and the pairs of files that should be saved are stored in memory.
The diffs collected using this method are often fairly large, because the repositories are compared over a large span of time.
Therefore, the tree diff of the last 30 commit before the tag (equivalent of git diff TAG^ TAG, git diff TAG^^ TAG^^) were also used to also include smaller diffs.

The benchmark measures the runtime of performing a line diff between the collected files.
As a measure of complexity for each change (M + N) D was used where M and N are the lengths of the two compared files
and D is the length of the edit script required to transform these files into each other (determined with Myers algorithm).
This complexity measure is used to divide the changes into 10 badges.
The time to compute the line diffs in each badge was benchmarked.

The plots below show the runtime for each average complexity (runtime is normalized by the number of diffs).
Note that these plots are shown in logarithmic scale due to the large runtime of similar for complex diffs.
Furthermore, to better highlight the performance of the Histogram algorithm, the speedup of the Histogram algorithm
compared to the Myers algorithm is shown separately.

Linux

The sourcecode of the linux kernel.

Rust

The sourcecode of the rust compiler, standard library and various related tooling.

VSCode

The sourcecode of the vscode editor.

Helix

The sourcecode of the helix editor.

Testing

imara-diff includes comprehensive fuzz testing using cargo-fuzz to ensure robustness against arbitrary inputs. Fuzz tests cover all major algorithms (Myers, Histogram, MyersMinimal), postprocessing with different heuristics, and unified diff printing. See the fuzz/README.md for more details on running fuzz tests locally.

Stability Policy

imara-diff uses Semantic Versioning (SemVer).
All non-breaking changes to the public rust API will cause a minor SemVer bump.
All breaking changes to to the public rust API will cause a major SemVer bump.
Changes in the produced diffs are also considered breaking changes if the produced diff was valid.
If the produced diff was invalid the change will be considered a bugfix.

Additionally all changes to the minimum stable rust version (MSRV) are also considered breaking changes.
The current MSRV is 1.85, matching the surrounding gitoxide workspace so hash-related dependencies
can use Rust 2024 crates such as sha2 0.11 and hashbrown 0.17.

Overview (`gix-index/README.md`)

Test fixtures

Most of the test indices are snatched directly from the unit test suite of git itself, usually by running something like the following

shell
./t1700-split-index.sh -r 2 --debug

Then one finds all test state and the index in particular in trash directory/t1700-split-index/.git/index and can possibly copy it over and use as fixture.
The preferred way is to find a test of interest, and use its setup code within one of our own fixture scripts that are executed once to generate the file of interest.

Overview (`gix-lock/README.md`)

Use lock-files in the way git does with auto-cleanup being the most notable feature.

  • writable lock files that can be committed to atomically replace the resource they lock
  • read-only markers that lock a resource without the intend to overwrite it
  • auto-removal of the lockfiles and intermediate directories on drop or on signal

Overview (`gix-pathspec/README.md`)

`gix-pathspec`

Testing

Fuzzing

cargo fuzz is used for fuzzing, installable with cargo install cargo-fuzz.

Targets can be listed with cargo fuzz list and executed via cargo +nightly fuzz run <target>,
where <target> can be parse for example.

Notes

  • There is one additional keyword that git can parse, but that this crate doesn't support yet: the prefix keyword

    Here is a commit in which prefix is somewhat explained.

Overview (`gix-prompt/README.md`)

Acknowledgements

Portions of this crate are loosely based on rpassword.

Overview (`gix-refspec/README.md`)

`gix-refspec`

Testing

Fuzzing

cargo fuzz is used for fuzzing, installable with cargo install cargo-fuzz.

Targets can be listed with cargo fuzz list and executed via cargo +nightly fuzz run <target>,
where <target> can be parse for example.

Overview (`gix-revision/README.md`)

`gix-revision`

Testing

Fuzzing

cargo fuzz is used for fuzzing, installable with cargo install cargo-fuzz.

Targets can be listed with cargo fuzz list and executed via cargo +nightly fuzz run <target>,
where <target> can be parse for example.

Overview (`gix-tempfile/README.md`)

Use tempfiles to minimize the risk of resource leakage when preparing to overwrite or create a file with new content
in a signal-safe way, making the change atomic.

Tempfiles can also be used as locks as only one tempfile can exist at a given path at a time.

  • registered temporary files which are deleted automatically as the process terminates or on drop
    • write to temporary file and persist it under new name
    • close temporary files to convert them into a marker while saving system resources
    • mark paths with a closed temporary file
  • persist temporary files to prevent them from perishing.
  • signal-handler integration with gix to clean lockfiles before the process is aborted.
  • use a temporary file transparently due thanks to implementations of std::io traits

Overview (`gix-tix/README.md`)

This is a tig inspired completely generated program to do what I used tig for, namely:

  • show project histories, but allow to trim them to hide given branches
  • copy selected hashes
  • but be faster and less memory hungry than tig when looking at big repositories.

The commits that created it are clearly identified as authored by GPT, without myself as co-author.
After all, I intentionally didn't look at the code.

And from what I can tell, it does what I want it to, and seems to be worth maintaining.

Keyboard shortcuts

Press ? for keyboard help or p for the searchable command menu. The v, a,
and n groups contain display controls, actions, and enrichment commands.
These shortcuts also work directly, without a prefix:

Key Action
C (Shift+C) Hide visible changes panes or show Tree + Worktree together. ? e still cycles both panes, Tree only, and hidden.
H (Shift+H) Show or hide explicit or inferred integration-branch history from history or a changes pane.
P (Shift+P) Push the active branch from history or Worktree. In Tree, cycle the comparison parent; a P pushes from there.

See the full keyboard reference for
navigation and the remaining direct shortcuts.

Undo and redo

tix op (or tix op log) shows this worktree's operation history, newest first,
with @ marking the current position. tix op undo and tix op redo move one
operation at a time without confirmation. tix op clear discards this
worktree's history without reversing its operations. These commands also work
under gix tix.

Marking reviewed patches

Press n r to mark the selected patch refackiewed (refactored and reviewed)
with ✨, or to clear the mark. The command menu also finds refackiewed.
From the shell, use tix enrich patch refackiewed [REVSPEC] and add --clear
to remove the mark; the target defaults to HEAD.

The mark belongs to that version of a Tix change's patch. It survives rewording
and rebasing when the patch's edits stay the same, even when ancestor changes
move lines or alter unrelated files. Changing the patch hides its old mark;
returning to the approved version restores it. The existing ✔️ checks-pass
mark still belongs to one exact tree.

Tix stores the patch identity in a commit header and the mark in worktree-local
notes. Marking a commit without that header rewrites it and its descendants,
preserving staged changes and worktree files; previously final descendants stay
final. Lazy rebases hide the mark until replay refreshes the identity. Browsing
only reads existing metadata and never calculates patch hashes or backfills old
commits. See patch identity and enrichment
for identity, eligibility, and caching rules.

Worktrees

tix worktrunk (or tix wt) opens a worktree picker with the selected
worktree's interactive history below it. Install its wt shell wrapper by
evaluating tix worktrunk shell-init bash or zsh, piping the fish output to
source, or loading the generated nushell/powershell script from the
corresponding shell profile. The same command below gix tix generates a
wrapper which uses gix tix throughout.

wt switch BRANCH switches to an existing worktree or creates one for an
unchecked-out local branch. wt switch --new-branch BRANCH creates a missing
branch at the logical Tix HEAD, or reuses it if it exists. --path PATH
overrides the default sibling path.

Selecting a worktree in the picker also returns to the shell and changes its
directory. Open tix -X (or tix --auto-hide) there to immediately hide commits
reachable from the integration branches inferred by tix show. Combine it with
-x REVISION to hide additional history. The same options work with gix tix.
Plain tix starts with full history; explicit -x filters apply immediately.
Press Shift+H (View hide unrelated history / show related history) to
toggle the combined filter.

wt switch --detach [COMMIT] creates a detached worktree at the current HEAD
or a supplied commit. --path PATH chooses its directory; otherwise a sibling
directory uses the commit's short hash, adding a number if occupied. The source
worktree gains a symbolic pin following the new worktree's HEAD so its experiment
stays visible in the source's history, even without existing Tix pins.

wt show prints the fully populated worktree table without opening the picker.

It's also an experiment to see how long, or if at all, this is maintainable.

Overview (`etc/corpus/README.md`)

gitoxide uses a corpus of popular git repositories to validate its own algorithms against.
This file contains all information needed to build a corpus of varying sizes and to run gitoxide against it.

Setup corpus

The corpus is a set of git repositories to run algorithms against. The set is a filtered list…

  • …ordered by popularity on GitHub by stars and…
  • …filtered by repos smaller than 5GB…
  • …and limited until a certain size on disk is reached…

which is then cloned to disk into a directory structure that mirrors its URL.

Obtaining the set of repository metadata

The ~1.8GB file can be downloaded here https://www.kaggle.com/datasets/pelmers/github-repository-metadata-with-5-stars (needs account).
When downloaded it has to be converted to JSONL for consumption.

shell
# convert the downloaded JSON into JSONL into the same directory
json-to-jsonl.sh repo_metadata.json

Figuring out how many repos fit the available space

Skipping all repositories larger than 5GB and filling a limit, one can use the Jupyter Notebook at https://www.kaggle.com/code/pelmers/explore-github-repository-metadata (*account required).

Add the following snippet to the notebook and adjust limit to your needs.

python
five_gb_in_kb = 5 * 1024 * 1024
limit = 3500 * 1024 * 1024

# Order by 'stars' column and filter by 'diskUsageKb'
df = df[df['diskUsageKb'] < five_gb_in_kb]
df_sorted = df.sort_values(by='stars', ascending=False)

# Calculate how many entries would fit into 350GB
disk_usage_cumsum = df_sorted['diskUsageKb'].cumsum()
(disk_usage_cumsum <= limit).sum()

In the example above, one would manage to fit 68568 repositories into 3.5TB.

Cloning the repositories

Run head -n 999 repo_metadata.sample.jsonl | ./clone-repos.sh <corpus> to clone into the given <corpus> location, or any other invocation with
your respective repo_metadata.jsonl and the computed amount of repos to include as in head -n <your-count> <your.jsonl>.

Add one large (100GB+) repository and one with a lot of commits repository by hand

Invoke git clone --bare https://github.com/NagatoDEV/PlayStation-Home-Master-Archive <corpus>/github.com/NagatoDEV/PlayStation-Home-Master-Archive.git (after replacing <curpus> with your base path)
to obtain one sample of a huge repository with a lot of assets and other binary data whose tree spans more than 440k files.

That way, we also get to see what happens when we have to handle huge binary files in massive trees.

Another massive tree and a more than 1.3m commits comes in with this invocation:

git clone --bare https://github.com/archlinux/svntogit-community <corpus>/github.com/archlinux/svntogit-community.git.

This repo has 100MB+ files with a lot of append-only changes to it, giving it a very imbalanced delta-tree that triggers worst-case behaviour that needed
special mitigations:

git clone --bare https://github.com/fz139/vigruzki <corpus>/github.com/fz139/vigruzki.git.

All repos should be topped off with…

shell
cd <corpus>
for d in github.com/archlinux/svntogit-community.git github.com/NagatoDEV/PlayStation-Home-Master-Archive.git github.com/fz139/vigruzki.git; do
    git -C $d read-tree @
    git -C $d commit-graph write --no-progress --reachable
done

Run on-off `gix` commands by hand

Sometimes it's interesting to try a new command against all available repositories to see if it fails:

ein t find <corpus> | xargs -P10 -I {} bash -c 'echo {}; gix -r {} <command>

Run `gix corpus`

The corpus sub-command runs specifically implemented commands against the corpus in a parallel fashion and stores these results in a local sqlite database for
later comparison.

TBD

Overview (`etc/security/README.md`)

Security process documents

[!NOTE]
To report a vulnerability, see the security policy, which can also be read in the top-level SECURITY.md.

Table of Contents

The documents in this directory are things we use when managing vulnerabilities:

Incident Response Plan

  • irp.md is our incident response plan.

Threat Model

  • threat-model.md is our provisional threat model outline.

  • threat-model-notes.md are some notes, in a different form, that have informed it. It overlaps significantly with threat-model.md, but it may also occasionally be useful to refer to. (These notes are less detailed in some significant ways, including not mentioning all significant concerns and not indicating STRIDE categories.)

Overview (`gix-bitmap/fuzz/README.md`)

Fuzz Testing

This directory contains fuzz tests for gix-bitmap using cargo-fuzz.

Running Fuzz Tests

Prerequisites

  • Nightly Rust toolchain: rustup install nightly
  • cargo-fuzz: cargo install cargo-fuzz

Targets

  • ewah: exercises EWAH bitmap decoding.

Running the target

bash
cargo +nightly fuzz run ewah -- -max_total_time=60

Artifact Reproducers

The artifacts/ewah directory contains minimized reproducer inputs produced by libFuzzer or Google OSS-Fuzz.

The integration test module fuzzed reads every regular file in that directory and runs it through the parser under test. This lets cargo test -p gix-bitmap fuzzed reproduce known fuzz failures without starting a fuzzing session.

When OSS-Fuzz reports a new failure, place the minimized testcase in artifacts/ewah, then run cargo test -p gix-bitmap fuzzed. To confirm it against the original harness, run cargo fuzz run ewah artifacts/ewah/<reproducer> from this directory.

Overview (`gix-diff/tests/README.md`)

Running the diff-slider tests

The tests compare gix-diff with Git using data produced by
diff-slider-tools. The commands below don't create or discover
a corpus: they require a .sliders file and its corresponding Git repository.

Prepare the corpus once

From a diff-slider-tools checkout:

shell
repo=gitoxide
echo https://github.com/GitoxideLabs/gitoxide >"corpus/$repo.info"
./get-corpus "$repo"
git -C "corpus/$repo.git" log --min-parents=1 --max-parents=1 --format='%P..%H' HEAD |
  ./enumerate-sliders --repo="$repo" >"corpus/$repo.sliders"

export SLIDERS_FILE="$PWD/corpus/$repo.sliders"
export SLIDER_REPOSITORY="$PWD/corpus/$repo.git"

get-corpus reads corpus/gitoxide.info and creates corpus/gitoxide.git. enumerate-sliders then reads that
repository and creates corpus/gitoxide.sliders.

Optionally, see the complete
upstream corpus instructions
for other repositories or commit ranges.

If the corpus already exists, only set its explicit paths:

shell
export SLIDERS_FILE=/path/to/diff-slider-tools/corpus/gitoxide.sliders
export SLIDER_REPOSITORY=/path/to/diff-slider-tools/corpus/gitoxide.git

Then run these two commands from the gitoxide root:

shell
cargo run --package internal-tools -- create-diff-cases \
  --count 2024 \
  --sliders-file "$SLIDERS_FILE" \
  --worktree-dir "$SLIDER_REPOSITORY" \
  --destination-dir gix-diff/tests/fixtures/

cargo test -p gix-diff slider::baseline -- --nocapture

--count limits how many slider records are read; lower it for a quicker run. With the current gitoxide
corpus, --count 2024 produces 3,094 cases after duplicate blob pairs are removed.

The default report prints all mismatch categories without failing. Set GIX_DIFF_SLIDER_STRICT=1 to require a
non-empty external baseline and fail on any mismatch. The small built-in baseline always runs strictly.

Inspect one case

Copy a primary baseline filename from the report and select it explicitly:

shell
GIX_DIFF_SLIDER_CASE='<old>-<new>.myers.baseline' \
  cargo test -p gix-diff --test diff blob::slider::baseline -- --exact --nocapture

The selector must match the complete filename, including the algorithm and .baseline suffix.
Inspection replaces the aggregate report with a StrComparison between the two
unified diffs: Gitoxide with slider heuristics on the left and Git with the indent heuristic on the right.
Matching output is reported explicitly. A mismatch doesn't fail inspection; an empty, missing,
or ambiguous selection does.

GIX_DIFF_SLIDER_CASE cannot be combined with GIX_DIFF_SLIDER_STRICT. Only the selected external case is loaded and
diffed when GIX_DIFF_SLIDER_CASE is given.

Overview (`gix-imara-diff/fuzz/README.md`)

Fuzz Testing

This directory contains fuzz tests for imara-diff using cargo-fuzz.

Running Fuzz Tests

Prerequisites

  • Nightly Rust toolchain: rustup install nightly
  • cargo-fuzz: cargo install cargo-fuzz

Running a specific target

bash
# Run for a specific time (e.g., 60 seconds)
cargo +nightly fuzz run comprehensive_diff -- -max_total_time=60

# Run with a specific number of runs
cargo +nightly fuzz run comprehensive_diff -- -runs=1000000

Running all targets

bash
for target in comprehensive_diff diff_compute_with postprocess_heuristics unified_diff_printer; do
    cargo +nightly fuzz run --release $target -- -max_total_time=60
done

Analyzing coverage

bash
cargo +nightly fuzz coverage comprehensive_diff

CI Integration

Fuzz tests are automatically run in CI for 3 minutes total (45 seconds per target × 4 targets) to ensure no regressions in robustness.

Adding New Fuzz Targets

  1. Create a new file in fuzz_targets/ directory
  2. Add a new [[bin]] entry in Cargo.toml
  3. Update the CI workflow to run the new target
  4. Update this README with a description of the new target

Overview (`gix-index/fuzz/README.md`)

Fuzz Testing

This directory contains fuzz tests for gix-index using cargo-fuzz.

Running Fuzz Tests

Prerequisites

  • Nightly Rust toolchain: rustup install nightly
  • cargo-fuzz: cargo install cargo-fuzz

Targets

  • index_file: exercises index decoding, entry/path access, extension access, verification, and round-trip serialization.

Running the target

bash
cargo +nightly fuzz run index_file -- -max_total_time=60

Artifact Reproducers

The artifacts/index_file directory contains minimized reproducer inputs produced by libFuzzer or Google OSS-Fuzz.

The integration test module fuzzed reads every regular file in that directory and runs it through the parser under test. This lets cargo test -p gix-index fuzzed reproduce known fuzz failures without starting a fuzzing session.

When OSS-Fuzz reports a new failure, place the minimized testcase in artifacts/index_file, then run cargo test -p gix-index fuzzed. To confirm it against the original harness, run cargo fuzz run index_file artifacts/index_file/<reproducer> from this directory.

Overview (`gix-pack/fuzz/README.md`)

Fuzz Testing

This directory contains fuzz tests for gix-pack using cargo-fuzz.

Running Fuzz Tests

Prerequisites

  • Nightly Rust toolchain: rustup install nightly
  • cargo-fuzz: cargo install cargo-fuzz

Targets

  • data_file: exercises .pack parsing, checksum verification, entry lookup, and entry decoding.
  • index_file: exercises .idx parsing, checksum verification, lookups, prefix lookups, and iteration.
  • multi_index_file: exercises multi-pack-index parsing, checksum verification, lookups, prefix lookups, and iteration.

Running a specific target

bash
cargo +nightly fuzz run data_file -- -max_total_time=60
cargo +nightly fuzz run index_file -- -max_total_time=60
cargo +nightly fuzz run multi_index_file -- -max_total_time=60

Running all targets

bash
for target in data_file index_file multi_index_file; do
    cargo +nightly fuzz run "$target" -- -max_total_time=60
done

Artifact Reproducers

The artifacts/<target> directories contain minimized reproducer inputs produced by libFuzzer or Google OSS-Fuzz.

The integration test module fuzzed reads every regular file in the populated artifact directories and runs it through the parser under test. This lets cargo test -p gix-pack fuzzed reproduce known fuzz failures without starting a fuzzing session.

When OSS-Fuzz reports a new failure, place the minimized testcase in the matching artifacts/<target> directory, then run cargo test -p gix-pack fuzzed. To confirm it against the original harness, run cargo fuzz run <target> artifacts/<target>/<reproducer> from this directory.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#blazingly-fast Browse blazingly-fast catalog → #built-with-rust Browse built-with-rust catalog → #cli Browse cli catalog → #git Browse git catalog → #version-control Browse version-control catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →