## File: README.md

git filter-repo is a versatile tool for rewriting history, which includes
[capabilities I have not found anywhere
else](#design-rationale-behind-filter-repo).  It roughly falls into the
same space of tool as [git
filter-branch](https://git-scm.com/docs/git-filter-branch) but without the
capitulation-inducing poor
[performance](https://public-inbox.org/git/CABPp-BGOz8nks0+Tdw5GyGqxeYR-3FF6FT5JcgVqZDYVRQ6qog@mail.gmail.com/),
with far more capabilities, and with a design that scales usability-wise
beyond trivial rewriting cases.  [git filter-repo is now recommended by the
git project](https://git-scm.com/docs/git-filter-branch#_warning) instead
of git filter-branch.

While most users will probably just use filter-repo as a simple command
line tool (and likely only use a few of its flags), at its core filter-repo
contains a library for creating history rewriting tools.  As such, users
with specialized needs can leverage it to quickly create [entirely new
history rewriting tools](contrib/filter-repo-demos).

# Table of Contents

  * [Prerequisites](#prerequisites)
  * [How do I install it?](#how-do-i-install-it)
  * [How do I use it?](#how-do-i-use-it)
  * [Why filter-repo instead of other alternatives?](#why-filter-repo-instead-of-other-alternatives)
    * [filter-branch](#filter-branch)
    * [BFG Repo Cleaner](#bfg-repo-cleaner)
  * [Simple example, with comparisons](#simple-example-with-comparisons)
    * [Solving this with filter-repo](#solving-this-with-filter-repo)
    * [Solving this with BFG Repo Cleaner](#solving-this-with-bfg-repo-cleaner)
    * [Solving this with filter-branch](#solving-this-with-filter-branch)
    * [Solving this with fast-export/fast-import](#solving-this-with-fast-exportfast-import)
  * [Design rationale behind filter-repo](#design-rationale-behind-filter-repo)
  * [How do I contribute?](#how-do-i-contribute)
  * [Is there a Code of Conduct?](#is-there-a-code-of-conduct)
  * [Upstream Improvements](#upstream-improvements)

# Prerequisites

filter-repo requires:

  * git >= 2.36.0
  * python3 >= 3.6

# How do I install it?

While the `git-filter-repo` repository has many files, the main logic
is all contained in a single-file python script named
`git-filter-repo`, which was done to make installation for basic use
on many systems trivial: just place that one file into your $PATH.

See [INSTALL.md](INSTALL.md) for things beyond basic usage or special
cases.  The more involved instructions are only needed if one of the
following apply:

  * you do not find the above comment about trivial installation intuitively
    obvious
  * you are working with a python3 executable named something other than
    "python3"
  * you want to install documentation (beyond the builtin docs shown with -h)
  * you want to run some of the [contrib](contrib/filter-repo-demos/) examples
  * you want to create your own python filtering scripts using filter-repo as
    a module/library

# How do I use it?

For comprehensive documentation:
  * see the [user manual](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html)
  * alternative formating of the user manual is available on various
    external sites
    ([example](https://www.mankier.com/1/git-filter-repo)), for those
    that don't like the htmlpreview.github.io layout, though it may
    only be up-to-date as of the latest release

If you prefer learning from examples:
  * there is a [cheat sheet for converting filter-branch
    commands](Documentation/converting-from-filter-branch.md#cheat-sheet-conversion-of-examples-from-the-filter-branch-manpage),
    which covers every example from the filter-branch manual
  * there is a [cheat sheet for converting BFG Repo Cleaner
    commands](Documentation/converting-from-bfg-repo-cleaner.md#cheat-sheet-conversion-of-examples-from-bfg),
    which covers every example from the BFG website
  * the [simple example](#simple-example-with-comparisons) below may
    be of interest
  * the user manual has an extensive [examples
section](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#EXAMPLES)
  * I have collected a set of [example filterings based on user-filed issues](Documentation/examples-from-user-filed-issues.md)

In either case, you may also find the [Frequently Answered Questions](Documentation/FAQ.md) useful.

# Why filter-repo instead of other alternatives?

This was covered in more detail in a [Git Rev News article on
filter-repo](https://git.github.io/rev_news/2019/08/21/edition-54/#an-introduction-to-git-filter-repo--written-by-elijah-newren),
but some highlights for the main competitors:

## filter-branch

  * filter-branch is [extremely to unusably
    slow](https://public-inbox.org/git/CABPp-BGOz8nks0+Tdw5GyGqxeYR-3FF6FT5JcgVqZDYVRQ6qog@mail.gmail.com/)
    ([multiple orders of magnitude slower than it should
    be](https://git-scm.com/docs/git-filter-branch#PERFORMANCE))
    for non-trivial repositories.

  * [filter-branch is riddled with
    gotchas](https://git-scm.com/docs/git-filter-branch#SAFETY) that can
    silently corrupt your rewrite or at least thwart your "cleanup"
    efforts by giving you something more problematic and messy than what
    you started with.

  * filter-branch is [very onerous](#simple-example-with-comparisons)
    [to
    use](https://github.com/newren/git-filter-repo/blob/a6a6a1b0f62d365bbe2e76f823e1621857ec4dbd/contrib/filter-repo-demos/filter-lamely#L9-L61)
    for any rewrite which is even slightly non-trivial.

  * the git project has stated that the above issues with filter-branch
    cannot be backward compatibly fixed; they recommend that you [stop
    using
    filter-branch](https://git-scm.com/docs/git-filter-branch#_warning)

  * die-hard fans of filter-branch may be interested in
    [filter-lamely](contrib/filter-repo-demos/filter-lamely)
    (a.k.a. [filter-branch-ish](contrib/filter-repo-demos/filter-branch-ish)),
    a reimplementation of filter-branch based on filter-repo which is
    more performant (though not nearly as fast or safe as
    filter-repo).

  * a [cheat
    sheet](Documentation/converting-from-filter-branch.md#cheat-sheet-conversion-of-examples-from-the-filter-branch-manpage)
    is available showing how to convert example commands from the manual of
    filter-branch into filter-repo commands.

## BFG Repo Cleaner

  * great tool for its time, but while it makes some things simple, it
    is limited to a few kinds of rewrites.

  * its architecture is not amenable to handling more types of
    rewrites.

  * its architecture presents some shortcomings and bugs even for its
    intended usecase.

  * fans of bfg may be interested in
    [bfg-ish](contrib/filter-repo-demos/bfg-ish), a reimplementation of bfg
    based on filter-repo which includes several new features and bugfixes
    relative to bfg.

  * a [cheat
    sheet](Documentation/converting-from-bfg-repo-cleaner.md#cheat-sheet-conversion-of-examples-from-bfg)
    is available showing how to convert example commands from the manual of
    BFG Repo Cleaner into filter-repo commands.

# Simple example, with comparisons

Let's say that we want to extract a piece of a repository, with the intent
on merging just that piece into some other bigger repo.  For extraction, we
want to:

  * extract the history of a single directory, src/.  This means that only
    paths under src/ remain in the repo, and any commits that only touched
    paths outside this directory will be removed.
  * rename all files to have a new leading directory, my-module/ (e.g. so that
    src/foo.c becomes my-module/src/foo.c)
  * rename any tags in the extracted repository to have a 'my-module-'
    prefix (to avoid any conflicts when we later merge this repo into
    something else)

## Solving this with filter-repo

Doing this with filter-repo is as simple as the following command:
```shell
  git filter-repo --path src/ --to-subdirectory-filter my-module --tag-rename '':'my-module-'
```
(the single quotes are unnecessary, but make it clearer to a human that we
are replacing the empty string as a prefix with `my-module-`)

## Solving this with BFG Repo Cleaner

BFG Repo Cleaner is not capable of this kind of rewrite; in fact, all
three types of wanted changes are outside of its capabilities.

## Solving this with filter-branch

filter-branch comes with a pile of caveats (more on that below) even
once you figure out the necessary invocation(s):

```shell
  git filter-branch \
      --tree-filter 'mkdir -p my-module && \
                     git ls-files \
                         | grep -v ^src/ \
                         | xargs git rm -f -q && \
                     ls -d * \
                         | grep -v my-module \
                         | xargs -I files mv files my-module/' \
          --tag-name-filter 'echo "my-module-$(cat)"' \
	  --prune-empty -- --all
  git clone file://$(pwd) newcopy
  cd newcopy
  git for-each-ref --format="delete %(refname)" refs/tags/ \
      | grep -v refs/tags/my-module- \
      | git update-ref --stdin
  git gc --prune=now
```

Some might notice that the above filter-branch invocation will be really
slow due to using --tree-filter; you could alternatively use the
--index-filter option of filter-branch, changing the above commands to:

```shell
  git filter-branch \
      --index-filter 'git ls-files \
                          | grep -v ^src/ \
                          | xargs git rm -q --cached;
                      git ls-files -s \
                          | sed "s%$(printf \\t)%&my-module/%" \
                          | git update-index --index-info;
                      git ls-files \
                          | grep -v ^my-module/ \
                          | xargs git rm -q --cached' \
      --tag-name-filter 'echo "my-module-$(cat)"' \
      --prune-empty -- --all
  git clone file://$(pwd) newcopy
  cd newcopy
  git for-each-ref --format="delete %(refname)" refs/tags/ \
      | grep -v refs/tags/my-module- \
      | git update-ref --stdin
  git gc --prune=now
```

However, for either filter-branch command there are a pile of caveats.
First, some may be wondering why I list five commands here for
filter-branch.  Despite the use of --all and --tag-name-filter, and
filter-branch's manpage claiming that a clone is enough to get rid of
old objects, the extra steps to delete the other tags and do another
gc are still required to clean out the old objects and avoid mixing
new and old history before pushing somewhere.  Other caveats:
  * Commit messages are not rewritten; so if some of your commit
    messages refer to prior commits by (abbreviated) sha1, after the
    rewrite those messages will now refer to commits that are no longer
    part of the history.  It would be better to rewrite those
    (abbreviated) sha1 references to refer to the new commit ids.
  * The --prune-empty flag sometimes misses commits that should be
    pruned, and it will also prune commits that *started* empty rather
    than just ended empty due to filtering.  For repositories that
    intentionally use empty commits for versioning and publishing
    related purposes, this can be detrimental.
  * The commands above are OS-specific.  GNU vs. BSD issues for sed,
    xargs, and other commands often trip up users; I think I failed to
    get most folks to use --index-filter since the only example in the
    filter-branch manpage that both uses it and shows how to move
    everything into a subdirectory is linux-specific, and it is not
    obvious to the reader that it has a portability issue since it
    silently misbehaves rather than failing loudly.
  * The --index-filter version of the filter-branch command may be two to
    three times faster than the --tree-filter version, but both
    filter-branch commands are going to be multiple orders of magnitude
    slower than filter-repo.
  * Both commands assume all filenames are composed entirely of ascii
    characters (even special ascii characters such as tabs or double
    quotes will wreak havoc and likely result in missing files or
    misnamed files)

## Solving this with fast-export/fast-import

One can kind of hack this together with something like:

```shell
  git fast-export --no-data --reencode=yes --mark-tags --fake-missing-tagger \
      --signed-tags=strip --tag-of-filtered-object=rewrite --all \
      | grep -vP '^M [0-9]+ [0-9a-f]+ (?!src/)' \
      | grep -vP '^D (?!src/)' \
      | perl -pe 's%^(M [0-9]+ [0-9a-f]+ )(.*)$%\1my-module/\2%' \
      | perl -pe 's%^(D )(.*)$%\1my-module/\2%' \
      | perl -pe s%refs/tags/%refs/tags/my-module-% \
      | git -c core.ignorecase=false fast-import --date-format=raw-permissive \
            --force --quiet
  git for-each-ref --format="delete %(refname)" refs/tags/ \
      | grep -v refs/tags/my-module- \
      | git update-ref --stdin
  git reset --hard
  git reflog expire --expire=now --all
  git gc --prune=now
```

But this comes with some nasty caveats and limitations:
  * The various greps and regex replacements operate on the entire
    fast-export stream and thus might accidentally corrupt unintended
    portions of it, such as commit messages.  If you needed to edit
    file contents and thus dropped the --no-data flag, it could also
    end up corrupting file contents.
  * This command assumes all filenames in the repository are composed
    entirely of ascii characters, and also exclude special characters
    such as tabs or double quotes.  If such a special filename exists
    within the old src/ directory, it will be pruned even though it
    was intended to be kept.  (In slightly different repository
    rewrites, this type of editing also risks corrupting filenames
    with special characters by adding extra double quotes near the end
    of the filename and in some leading directory name.)
  * This command will leave behind huge numbers of useless empty
    commits, and has no realistic way of pruning them.  (And if you
    tried to combine this technique with another tool to prune the
    empty commits, then you now have no way to distinguish between
    commits which were made empty by the filtering that you want to
    remove, and commits which were empty before the filtering process
    and which you thus may want to keep.)
  * Commit messages which reference other commits by hash will now
    reference old commits that no longer exist.  Attempting to edit
    the commit messages to update them is extraordinarily difficult to
    add to this kind of direct rewrite.

# Design rationale behind filter-repo

None of the existing repository filtering tools did what I wanted;
they all came up short for my needs. No tool provided any of the
first eight traits below I wanted, and no tool provided more than
two of the last four traits either:

  1. [Starting report] Provide user an analysis of their repo to help
     them get started on what to prune or rename, instead of expecting
     them to guess or find other tools to figure it out.  (Triggered, e.g.
     by running the first time with a special flag, such as --analyze.)

  1. [Keep vs. remove] Instead of just providing a way for users to
     easily remove selected paths, also provide flags for users to
     only *keep* certain paths.  Sure, users could workaround this by
     specifying to remove all paths other than the ones they want to
     keep, but the need to specify all paths that *ever* existed in
     **any** version of the repository could sometimes be quite
     painful.  For filter-branch, using pipelines like `git ls-files |
     grep -v ... | xargs -r git rm` might be a reasonable workaround
     but can get unwieldy and isn't as straightforward for users; plus
     those commands are often operating-system specific (can you spot
     the GNUism in the snippet I provided?).

  1. [Renaming] It should be easy to rename paths.  For example, in
     addition to allowing one to treat some subdirectory as the root
     of the repository, also provide options for users to make the
     root of the repository just become a subdirectory.  And more
     generally allow files and directories to be easily renamed.
     Provide sanity checks if renaming causes multiple files to exist
     at the same path.  (And add special handling so that if a commit
     merely copied oldname->newname without modification, then
     filtering oldname->newname doesn't trigger the sanity check and
     die on that commit.)

  1. [More intelligent safety] Writing copies of the original refs to
     a special namespace within the repo does not provide a
     user-friendly recovery mechanism.  Many would struggle to recover
     using that.  Almost everyone I've ever seen do a repository
     filtering operation has done so with a fresh clone, because
     wiping out the clone in case of error is a vastly easier recovery
     mechanism.  Strongly encourage that workflow by [detecting and
     bailing if we're not in a fresh
     clone](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#FRESHCLONE),
     unless the user overrides with --force.

  1. [Auto shrink] Automatically remove old cruft and repack the
     repository for the user after filtering (unless overridden); this
     simplifies things for the user, helps avoid mixing old and new
     history together, and avoids problems where the multi-step
     process for shrinking the repo documented in the manpage doesn't
     actually work in some cases.  (I'm looking at you,
     filter-branch.)

  1. [Clean separation] Avoid confusing users (and prevent accidental
     re-pushing of old stuff) due to mixing old repo and rewritten
     repo together.  (This is particularly a problem with filter-branch
     when using the --tag-name-filter option, and sometimes also an
     issue when only filtering a subset of branches.)

  1. [Versatility] Provide the user the ability to extend the tool or
     even write new tools that leverage existing capabilities, and
     provide this extensibility in a way that (a) avoids the need to
     fork separate processes (which would destroy performance), (b)
     avoids making the user specify OS-dependent shell commands (which
     would prevent users from sharing commands with each other), (c)
     takes advantage of rich data structures (because hashes, dicts,
     lists, and arrays are prohibitively difficult in shell) and (d)
     provides reasonable string manipulation capabilities (which are
     sorely lacking in shell).

  1. [Old commit references] Provide a way for users to use old commit
     IDs with the new repository (in particular via mapping from old to
     new hashes with refs/replace/ references).

  1. [Commit message consistency] If commit messages refer to other
     commits by ID (e.g. "this reverts commit 01234567890abcdef", "In
     commit 0013deadbeef9a..."), those commit messages should be
     rewritten to refer to the new commit IDs.

  1. [Become-empty pruning] Commits which become empty due to filtering
     should be pruned.  If the parent of a commit is pruned, the first
     non-pruned ancestor needs to become the new parent.  If no
     non-pruned ancestor exists and the commit was not a merge, then it
     becomes a new root commit.  If no non-pruned ancestor exists and
     the commit was a merge, then the merge will have one less parent
     (and thus make it likely to become a non-merge commit which would
     itself be pruned if it had no file changes of its own).  One
     special thing to note here is that we prune commits which become
     empty, NOT commits which start empty.  Some projects intentionally
     create empty commits for versioning or publishing reasons, and
     these should not be removed.  (As a special case, commits which
     started empty but whose parent was pruned away will also be
     considered to have "become empty".)

  1. [Become-degenerate pruning] Pruning of commits which become empty
     can potentially cause topology changes, and there are lots of
     special cases.  Normally, merge commits are not removed since they
     are needed to preserve the graph topology, but the pruning of
     parents and other ancestors can ultimately result in the loss of
     one or more parents.  A simple case was already noted above: if a
     merge commit loses enough parents to become a non-merge commit and
     it has no file changes, then it too can be pruned.  Merge commits
     can also have a topology that becomes degenerate: it could end up
     with the merge_base serving as both parents (if all intervening
     commits from the original repo were pruned), or it could end up
     with one parent which is an ancestor of its other parent.  In such
     cases, if the merge has no file changes of its own, then the merge
     commit can also be pruned.  However, much as we do with empty
     pruning we do not prune merge commits that started degenerate
     (which indicates it may have been intentional, such as with --no-ff
     merges) but only merge commits that become degenerate and have no
     file changes of their own.

  1. [Speed] Filtering should be reasonably fast

# How do I contribute?

See the [contributing guidelines](Documentation/Contributing.md).

# Is there a Code of Conduct?

Participants in the filter-repo community are expected to adhere to
the same standards as for the git project, so the [git Code of
Conduct](https://git.kernel.org/pub/scm/git/git.git/tree/CODE_OF_CONDUCT.md)
applies.

# Upstream Improvements

Work on filter-repo and [its
predecessor](https://public-inbox.org/git/51419b2c0904072035u1182b507o836a67ac308d32b9@mail.gmail.com/)
has also driven numerous improvements to fast-export and fast-import
(and occasionally other commands) in core git, based on things
filter-repo needs to do its work:

  * git-2.48.0
    * [fast-import: disallow more path components](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=da91a90c2f)
    * [fast-import: disallow "." and ".." path components](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=4a2790a257)
    * [fast-import: avoid making replace refs point to themselves](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=5e904f1a4a)
  * git-2.28.0
    * [fast-import: add new --date-format=raw-permissive format](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=d42a2fb72f)
  * git-2.24.0
    * [fast-export: handle nested tags](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=941790d7de)
    * [t9350: add tests for tags of things other than a commit](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=8d7d33c1ce)
    * [fast-export: allow user to request tags be marked with --mark-tags](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=a1638cfe12)
    * [fast-export: add support for --import-marks-if-exists](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=208d69246e)
    * [fast-import: add support for new 'alias' command](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=b8f50e5b60)
    * [fast-import: allow tags to be identified by mark labels](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=f73b2aba05)
    * [fast-import: fix handling of deleted tags](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=3164e6bd24)
    * [fast-export: fix exporting a tag and nothing else](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=af2abd870b)
    * [git-fast-import.txt: clarify that multiple merge commits are allowed](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=d1387d3895)
  * git-2.23.0
    * [t9350: fix encoding test to actually test reencoding](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=32615ce762)
    * [fast-import: support 'encoding' commit header](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=3edfcc65fd)
    * [fast-export: avoid stripping encoding header if we cannot reencode](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=ccbfc96dc4)
    * [fast-export: differentiate between explicitly UTF-8 and implicitly
      UTF-8](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=57a8be2cb0)
    * [fast-export: do automatic reencoding of commit messages only if
      requested](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=e80001f8fd)
  * git-2.22.0
    * [log,diff-tree: add --combined-all-paths option](
        https://git.kernel.org/pub/scm/git/git.git/commit/?id=d76ce4f734)
    * [t9300: demonstrate bug with get-mark and empty orphan commits](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=62edbec7de)
    * [git-fast-import.txt: fix wording about where ls command can appear](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=a63c54a019)
    * [fast-import: check most prominent commands first](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=5056bb7646)
    * [fast-import: only allow cat-blob requests where it makes sense](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=7ffde293f2)
    * [fast-import: fix erroneous handling of get-mark with empty orphan
      commits](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=cf7b857a77)
    * [Honor core.precomposeUnicode in more places](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=8e712ef6fc)
  * git-2.21.0
    * [fast-export: convert sha1 to oid](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=843b9e6d48)
    * [git-fast-import.txt: fix documentation for --quiet option](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=f55c979b14)
    * [git-fast-export.txt: clarify misleading documentation about rev-list
      args](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=4532be7cba)
    * [fast-export: use value from correct enum](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=b93b81e799)
    * [fast-export: avoid dying when filtering by paths and old tags exist](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=1f30c904b3)
    * [fast-export: move commit rewriting logic into a function for reuse](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=f129c4275c)
    * [fast-export: when using paths, avoid corrupt stream with non-existent
      mark](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=cd13762d8f)
    * [fast-export: ensure we export requested refs](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=fdf31b6369)
    * [fast-export: add --reference-excluded-parents option](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=530ca19c02)
    * [fast-import: remove unmaintained duplicate documentation](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=25dd3e4889)
    * [fast-export: add a --show-original-ids option to show
      original names](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=a965bb3116)
    * [git-show-ref.txt: fix order of flags](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=bd8d6f0def)
  * git-2.20.0
    * [update-ref: fix type of update_flags variable to
      match its usage](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=e4c34855a2)
    * [update-ref: allow --no-deref with --stdin](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=d345e9fbe7)
  * git-1.7.3
    * [fast-export: Fix dropping of files with --import-marks and path
      limiting](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=4087a02e45)
    * [fast-export: Add a --full-tree option](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=7f40ab0916)
    * [fast-export: Fix output order of D/F changes](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=060df62422)
    * [fast-import: Improve robustness when D->F changes provided in wrong
      order](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=253fb5f889)
  * git-1.6.4:
    * [fast-export: Set revs.topo_order before calling setup_revisions](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=668f3aa776)
    * [fast-export: Omit tags that tag trees](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=02c48cd69b)
    * [fast-export: Make sure we show actual ref names instead of "(null)"](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=2374502c6c)
    * [fast-export: Do parent rewriting to avoid dropping relevant commits](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=32164131db)
    * [fast-export: Add a --tag-of-filtered-object option for newly
      dangling tags](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=2d8ad46919)
    * [Add new fast-export testcases](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=25e0ca5dd6)
    * [fast-export: Document the fact that git-rev-list arguments are
      accepted](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=8af15d282e)
  * git-1.6.3:
    * [git-filter-branch: avoid collisions with variables in eval'ed
      commands](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=d5b0c97d13)
    * [Correct missing SP characters in grammar comment at top of
      fast-import.c](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=98e1a4186a)
    * [fast-export: Avoid dropping files from commits](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=ebeec7dbc5)
  * git-1.6.1.4:
    * [fast-export: ensure we traverse commits in topological order](
      https://git.kernel.org/pub/scm/git/git.git/commit/?id=784f8affe4)

---

## File: contrib/filter-repo-demos/README.md

## Background

filter-repo is not merely a history rewriting tool, it also contains a
library that can be used to write new history rewriting tools.  This
directory contains several examples showing the breadth of different things
that could be done.

## Quick overview

Command                         |Description
-------|-----------
barebones-example    |Simple example with no modifications to filter-repo behavior, just showing what to import and run.
insert-beginning     |Add a new file (e.g. LICENSE/COPYING) to the beginning of history.
signed-off-by        |Add a Signed-off-by tag to a range of commits
lint-history         |Run some lint command on all non-binary files in history.
clean-ignore         |Delete files from history which match current gitignore rules.
filter-lamely (or filter‑branch‑ish) |A nearly bug compatible re-implementation of filter-branch (the git testsuite passes using it instead of filter-branch), with some performance tricks to make it several times faster (though it's still glacially slow compared to filter-repo).
bfg-ish              |A re-implementation of most of BFG Repo Cleaner, with new features and bug fixes.
convert-svnexternals |Insert Git submodules according to SVN externals.

## Purpose

Please note that the point of these examples is not to provide new complete
tools, but simply to demonstrate that extremely varied history rewriting
tools can be created which automatically inherit lots of useful base
functionality: rewriting hashes in commit messages, pruning commits that
become empty, handling filenames with funny characters, non-standard
encodings, handling of replace refs, etc.  (Additional examples of using
filter-repo as a library can also be found in [the
testsuite](../../t/t9391/).)  My sincerest hope is that these examples
provide lots of useful functionality, but that each is missing at least one
critical piece for your usecase.  Go forth and extend and improve.

## Usage

All the examples require a symlink to git-filter-repo in your PYTHONPATH
named git_filter_repo.py in order to run; also, all have a --help flag to
get a description of their usage and flags.

---

## File: Documentation/Contributing.md

Welcome to the community!

Contributions need to meet the bar for inclusion in git.git.  Although
filter-repo is not part of the git.git repository, I want to leave the
option open for it to be merged in the future.  As such, any
contributions need to follow the same [guidelines for contribution to
git.git](https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/SubmittingPatches),
with a few exceptions:

  * While I
    [hate](https://public-inbox.org/git/CABPp-BG2SkH0GrRYpHLfp2Wey91ThwQoTgf9UmPa9f5Szn+v3Q@mail.gmail.com/)
    [GitHub](https://public-inbox.org/git/CABPp-BEcpasV4vBTm0uxQ4Vzm88MQAX-ArDG4e9QU8tEoNsZWw@mail.gmail.com/)
    [PRs](https://public-inbox.org/git/CABPp-BEHy8c3raHwf9aFXvXN0smf_WwCcNiYxQBwh7W6An60qQ@mail.gmail.com/)
    (as others point out, [it's mind-boggling in a bad way that
    web-based Git hosting and code review systems do such a poor
    job](http://nhaehnle.blogspot.com/2020/06/they-want-to-be-small-they-want-to-be.html)),
    git-format-patch and git-send-email can be a beast and I have not
    yet found time to modify Dscho's excellent
    [GitGitGadget](https://github.com/gitgitgadget/gitgitgadget) to
    work with git-filter-repo.  As such:
      * For very short single-commit changes, feel free to open GitHub PRs.
      * For more involved changes, if format-patch or send-email give you
        too much trouble, go ahead and open a GitHub PR and just mention
        that email didn't work out.
  * If emailing patches to the git list:
    * Include "filter-repo" at the start of the subject,
      e.g. "[filter-repo PATCH] Add packaging scripts for uploading to PyPI"
      instead of just "[PATCH] Add packaging scripts for uploading to PyPI"
    * CC me instead of the git maintainer
  * Git's [CodingGuidlines for python
    code](https://github.com/git/git/blob/v2.24.0/Documentation/CodingGuidelines#L482-L494)
    are only partially applicable:
    * python3 is a hard requirement; python2 is/was EOL at the end of
      2019 and should not be used.  (Commit 4d0264ab723c
      ("filter-repo: workaround python<2.7.9 exec bug", 2019-04-30)
      was the last version of filter-repo that worked with python2).
    * You can depend on anything in python 3.6 or earlier.  I may bump
      this minimum version over time, but do want to generally work
      with the python3 version found in current enterprise Linux
      distributions.
    * In filter-repo, it's not just OK to use bytestrings, you are
      expected to use them a lot.  Using unicode strings result in
      lots of ugly errors since input comes from filesystem names,
      commit messages, file contents, etc., none of which are
      guaranteed to be unicode.  (Plus unicode strings require lots of
      effort to verify, encode, and decode -- slowing the filtering
      process down).  I tried to work with unicode strings more
      broadly in the code base multiple times; but it's just a bad
      idea to use an abstraction that doesn't fit the data.
    * I generally like [PEP
      8](https://www.python.org/dev/peps/pep-0008/), but used
      two-space indents for years before learning of it and have just
      continued that habit.  For consistency, contributions should also
      use two-space indents and otherwise generally follow PEP 8.

There are a few extra things I would like folks to keep in mind:

  * Please test line coverage if you add or modify code

    * `make test` will run the testsuite under
      [coverage3](https://pypi.org/project/coverage/) (which you will
      need to install), and report on line coverage.  Line coverage of
      git-filter-repo needs to remain at 100%; line coverage of
      contrib and test scripts can be ignored.

  * Please do not be intimidated by detailed feedback:

    * In the git community, I have been contributing for years and
      have had hundreds of patches accepted but I still find that even
      when I try to make patches perfect I am not surprised when I
      have to spend as much or more time fixing up patches after
      submitting them than I did figuring out the patches in the first
      place.  git folks tend to do thorough reviews, which has taught
      me a lot, and I try to do the same for filter-repo.  Plus, as
      noted above, I want contributions from others to be acceptable
      in git.git itself.

---

## File: Documentation/converting-from-bfg-repo-cleaner.md

# Cheat Sheet: Converting from BFG Repo Cleaner

This document is aimed at folks who are familiar with BFG Repo Cleaner
and want to learn how to convert over to using filter-repo.

## Table of Contents

  * [Half-hearted conversions](#half-hearted-conversions)
  * [Intention of "equivalent" commands](#intention-of-equivalent-commands)
  * [Basic Differences](#basic-differences)
  * [Cheat Sheet: Conversion of Examples from BFG](#cheat-sheet-conversion-of-examples-from-bfg)

## Half-hearted conversions

You can switch most any BFG command to use filter-repo under the
covers by just replacing the `java -jar bfg.jar` part of the command
with [`bfg-ish`](../contrib/filter-repo-demos/bfg-ish).

bfg-ish is a reasonable tool, and provides a number of bug fixes and
features on top of bfg, but most of my focus is naturally on
filter-repo which has a number of capabilities lacking in bfg-ish.

## Intention of "equivalent" commands

BFG and filter-repo have a few differences, highlighted in the Basic
Differences section below, that make it hard to get commands that
behave identically.  Rather than focusing on matching BFG output as
exactly as possible, I treat the BFG examples as idiomatic ways to
solve a certain type of problem with BFG, and express how one would
idiomatically solve the same problem in filter-repo.  Sometimes that
means the results are not identical, but they are largely the same in
each case.

## Basic Differences

BFG operates directly on tree objects, which have no notion of their
leading path.  Thus, it has no way of differentiating between
'README.md' at the toplevel versus in some subdirectory.  You simply
operate on the basename of files and directories.  This precludes
doing things like renaming files and directories or other bigger
restructures.  By directly operating on trees, it also runs into
problems with loose vs. packed objects, loose vs. packed refs, not
understanding replace refs or grafts, and not understanding the index
and working tree as another data source.

With `git filter-repo`, you are essentially given an editing tool to
operate on the [fast-export](https://git-scm.com/docs/git-fast-export)
serialization of a repo, which operates on filenames including their
full paths from the toplevel of the repo.  Directories are not
separately specified, so any directory-related filtering is done by
checking the leading path of each file.  Further, you aren't limited
to the pre-defined filtering types, python callbacks which operate on
the data structures from the fast-export stream can be provided to do
just about anything you want.  By leveraging fast-export and
fast-import, filter-repo gains automatic handling of objects and refs
whether they are packed or not, automatic handling of replace refs and
grafts, and future features that may appear.  It also tries hard to
provide a full rewrite solution, so it takes care of additional
important concerns such as updating the index and working tree and
running an automatic gc for the user afterwards.

The "protection" and "privacy" defaults in BFG are something I
fundamentally disagreed with for a variety of reasons; see the
comments at the top of the
[bfg-ish](../contrib/filter-repo-demos/bfg-ish) script if you want
details.  The bfg-ish script implemented these protection and privacy
options since it was designed to act like BFG, but still flipped the
default to the opposite of what BFG chose.  I left the "protection"
and "non-private" features out of filter-repo entirely.  This means a
number of things with filter-repo:
  * any filters you specify will also be applied to HEAD, so that you
    don't have a weird disconnect from your history transformations
    only being applied to most commits
  * `[formerly OLDHASH]` references are not munged into commit
    messages; the replace refs that filter-repo adds are a much
    cleaner way of looking up commits by old commit hashes.
  * `Former-commit-id:` footers are not added to commit messages; the
    replace refs that filter-repo adds are a much cleaner way of
    looking up commits by old commit hashes.
  * History is not littered with `<filename>.REMOVED.git-id` files.

BFG expects you to specify the repository to rewrite as its final
argument, whereas filter-repo expects you to cd into the repo and then
run filter-repo.

## Cheat Sheet: Conversion of Examples from BFG

### Stripping big blobs

```shell
  java -jar bfg.jar --strip-blobs-bigger-than 100M some-big-repo.git
```

becomes

```shell
  git filter-repo --strip-blobs-bigger-than 100M
```

### Deleting files

```shell
  java -jar bfg.jar --delete-files id_{dsa,rsa}  my-repo.git
```

becomes

```shell
  git filter-repo --use-base-name --path id_dsa --path id_rsa --invert-paths
```

### Removing sensitive content

```shell
  java -jar bfg.jar --replace-text passwords.txt my-repo.git
```

becomes

```shell
  git filter-repo --replace-text passwords.txt
```

The `--replace-text` was a really clever idea that the BFG came up
with and I just implemented mostly as-is within filter-repo.  Sadly,
BFG didn't document the format of files passed to --replace text very
well, but I added more detail in the filter-repo documentation.

There is one small but important difference between the two tools: if
you use both "regex:" and "==>" on a single line to specify a regex
search and replace, then filter-repo will use "\1", "\2", "\3",
etc. for replacement strings whereas BFG used "$1", "$2", "$3", etc.
The reason for this difference is simply that python used backslashes
in its regex format while scala used dollar signs, and both tools
wanted to just pass along the strings unmodified to the underlying
language.  (Since bfg-ish attempts to emulate the BFG, it accepts
"$1", "$2" and so forth and translates them to "\1", "\2", etc. so
that filter-repo/python will understand it.)

### Removing files and folders with a certain name

```shell
  java -jar bfg.jar --delete-folders .git --delete-files .git --no-blob-protection  my-repo.git
```

becomes

```shell
  git filter-repo --invert-paths --path-glob '*/.git' --path .git
```

Yes, that glob will handle .git directories one or more directories
deep; it's a git-style glob rather than a shell-style glob.  Also, the
`--path .git` was added because `--path-glob '*/.git'` won't match a
directory named .git in the toplevel directory since it has a '/'
character in the glob expression (though I would hope the repository
doesn't have a tracked .git toplevel directory in its history).

---

## File: Documentation/converting-from-filter-branch.md

# Cheat Sheet: Converting from filter-branch

This document is aimed at folks who are familiar with filter-branch and want
to learn how to convert over to using filter-repo.

## Table of Contents

  * [Half-hearted conversions](#half-hearted-conversions)
  * [Intention of "equivalent" commands](#intention-of-equivalent-commands)
  * [Basic Differences](#basic-differences)
  * [Cheat Sheet: Conversion of Examples from the filter-branch manpage](#cheat-sheet-conversion-of-examples-from-the-filter-branch-manpage)
  * [Cheat Sheet: Additional conversion examples](#cheat-sheet-additional-conversion-examples)

## Half-hearted conversions

You can switch nearly any `git filter-branch` command to use
filter-repo under the covers by just replacing the `git filter-branch`
part of the command with
[`filter-lamely`](../contrib/filter-repo-demos/filter-lamely).  The
git.git regression testsuite passes when I swap out the filter-branch
script with filter-lamely, for example.  (However, the filter-branch
tests are not very comprehensive, so don't rely on that too much.)

Doing a half-hearted conversion has nearly all of the drawbacks of
filter-branch and nearly none of the benefits of filter-repo, but it
will make your command run a few times faster and makes for a very
simple conversion.

You'll get a lot more performance, safety, and features by just
switching to direct filter-repo commands.

## Intention of "equivalent" commands

filter-branch and filter-repo have different defaults, as highlighted
in the Basic Differences section below.  As such, getting a command
which behaves identically is not possible.  Also, sometimes the
filter-branch manpage lies, e.g. it says "suppose you want to...from
all commits" and then uses a command line like "git filter-branch
... HEAD", which only operates on commits in the current branch rather
than on all commits.

Rather than focusing on matching filter-branch output as exactly as
possible, I treat the filter-branch examples as idiomatic ways to
solve a certain type of problem with filter-branch, and express how
one would idiomatically solve the same problem in filter-repo.
Sometimes that means the results are not identical, but they are
largely the same in each case.

## Basic Differences

With `git filter-branch`, you have a git repository where every single
commit (within the branches or revisions you specify) is checked out
and then you run one or more shell commands to transform the working
copy into your desired end state.

With `git filter-repo`, you are essentially given an editing tool to
operate on the [fast-export](https://git-scm.com/docs/git-fast-export)
serialization of a repo.  That means there is an input stream of all
the contents of the repository, and rather than specifying filters in
the form of commands to run, you usually employ a number of common
pre-defined filters that provide various ways to slice, dice, or
modify the repo based on its components (such as pathnames, file
content, user names or emails, etc.)  That makes common operations
easier, even if it's not as versatile as shell callbacks.  For cases
where more complexity or special casing is needed, filter-repo
provides python callbacks that can operate on the data structures
populated from the fast-export stream to do just about anything you
want.

filter-branch defaults to working on a subset of the repository, and
requires you to specify a branch or branches, meaning you need to
specify `-- --all` to modify all commits.  filter-repo by contrast
defaults to rewriting everything, and you need to specify `--refs
<rev-list-args>` if you want to limit to just a certain set of
branches or range of commits.  (Though any `<rev-list-args>` that
begin with a hyphen are not accepted by filter-repo as they look like
the start of different options.)

filter-repo also takes care of additional concerns automatically, like
rewriting commit messages that reference old commit IDs to instead
reference the rewritten commit IDs, pruning commits which do not start
empty but become empty due to the specified filters, and automatically
shrinking and gc'ing the repo at the end of the filtering operation.

## Cheat Sheet: Conversion of Examples from the filter-branch manpage

### Removing a file

The filter-branch manual provided three different examples of removing
a single file, based on different levels of ease vs. carefulness and
performance:

```shell
  git filter-branch --tree-filter 'rm filename' HEAD
```
```shell
  git filter-branch --tree-filter 'rm -f filename' HEAD
```
```shell
  git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD
```

All of these just become

```shell
  git filter-repo --invert-paths --path filename
```

### Extracting a subdirectory

Extracting a subdirectory via

```shell
  git filter-branch --subdirectory-filter foodir -- --all
```

is one of the easiest commands to convert; it just becomes

```shell
  git filter-repo --subdirectory-filter foodir
```

### Moving the whole tree into a subdirectory

Keeping all files but placing them in a new subdirectory via

```shell
  git filter-branch --index-filter \
      'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
              GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                      git update-index --index-info &&
       mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD
```

(which happens to be GNU-specific and will fail with BSD userland in
very subtle ways) becomes

```shell
  git filter-repo --to-subdirectory-filter newsubdir
```

(which works fine regardless of GNU vs BSD userland differences.)

### Re-grafting history

The filter-branch manual provided one example with three different
commands that could be used to achieve it, though the first of them
had limited applicability (only when the repo had a single initial
commit).  These three examples were:
```shell
  git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD
```
```shell
  git filter-branch --parent-filter \
      'test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>" || cat' HEAD
```
```shell
  git replace --graft $commit-id $graft-id
  git filter-branch $graft-id..HEAD
```

git-replace did not exist when the original two examples were written,
but it is clear that the last example is far easier to understand.  As
such, filter-repo just uses the same mechanism:

```shell
  git replace --graft $commit-id $graft-id
  git filter-repo --proceed
```

NOTE: --proceed is needed here because filter-repo errors out if no
arguments are specified (doing so is usually an error).

### Removing commits by a certain author

WARNING: This is a BAD example for BOTH filter-branch and filter-repo.
It does not remove the changes the user made from the repo, it just
removes the commit in question while smashing the changes from it into
any subsequent commits as though the subsequent authors had been
responsible for those changes as well.  `git rebase` is likely to be a
better fit for what you really want if you are looking at this
example.  (See also [this explanation of the differences between
rebase and
filter-repo](https://github.com/newren/git-filter-repo/issues/62#issuecomment-597725502))

This filter-branch example

```shell
  git filter-branch --commit-filter '
      if [ "$GIT_AUTHOR_NAME" = "Darl McBribe" ];
      then
          skip_commit "$@";
      else
          git commit-tree "$@";
      fi' HEAD
```

becomes

```shell
  git filter-repo --commit-callback '
      if commit.author_name == b"Darl McBribe":
          commit.skip()
      '
```

### Rewriting commit messages -- removing text

Removing git-svn-id: lines from commit messages via

```shell
  git filter-branch --msg-filter '
      sed -e "/^git-svn-id:/d"
      '
```

becomes

```shell
  git filter-repo --message-callback '
      return re.sub(b"^git-svn-id:.*\n", b"", message, flags=re.MULTILINE)
      '
```

### Rewriting commit messages -- adding text

Adding Acked-by lines to the last ten commits via

```shell
  git filter-branch --msg-filter '
          cat &&
          echo "Acked-by: Bugs Bunny "
      ' master~10..master
```

becomes

```shell
  git filter-repo --message-callback '
          return message + b"Acked-by: Bugs Bunny \n"
      ' --refs master~10..master
```

### Changing author/committer(/tagger?) information

```shell
  git filter-branch --env-filter '
      if test "$GIT_AUTHOR_EMAIL" = "root@localhost"
      then
              GIT_AUTHOR_EMAIL=john@example.com
      fi
      if test "$GIT_COMMITTER_EMAIL" = "root@localhost"
      then
              GIT_COMMITTER_EMAIL=john@example.com
      fi
      ' -- --all
```

becomes either

```shell
  # Ensure '<john@example.com> <root@localhost>' is a line in .mailmap, then:
  git filter-repo --use-mailmap
```

or

```shell
  git filter-repo --email-callback '
    return email if email != b"root@localhost" else b"john@example.com"
    '
```

(and as a bonus both filter-repo alternatives will fix tagger emails
too, unlike the filter-branch example)


### Restricting to a range

The partial examples

```shell
  git filter-branch ... C..H
```
```shell
  git filter-branch ... C..H ^D
```
```shell
  git filter-branch ... D..H ^C
```

become

```shell
  git filter-repo ... --refs C..H
```
```shell
  git filter-repo ... --refs C..H ^D
```
```shell
  git filter-repo ... --refs D..H ^C
```

Note that filter-branch accepts `--not` among the revision specifiers,
but that appears to python to be a flag name which breaks parsing.
So, instead of e.g. `--not C` as we might use with filter-branch, we
can specify `^C` to filter-repo.

## Cheat Sheet: Additional conversion examples

### Running a code formatter or linter on each file with some extension

Running some program on a subset of files is relatively natural in
filter-branch:

```shell
  git filter-branch --tree-filter '
      git ls-files -z "*.c" \
          | xargs -0 -n 1 clang-format -style=file -i
      '
```

though it has the disadvantage of running on every c file for every
commit in history, even if some commits do not modify any c files.  This
means this kind of command can be excruciatingly slow.

The same functionality is slightly more involved in filter-repo for
two reasons:
  - fast-export and fast-import split file contents and file names into
    completely different data structures that aren't normally available
    together
  - to run a program on a file, you'll need to write the contents to the
    a file, execute the program on that file, and then read the contents
    of the file back in

```shell
  git filter-repo --file-info-callback '
    if not filename.endswith(b".c"):
      return (filename, mode, blob_id)  # no changes

    contents = value.get_contents_by_identifier(blob_id)
    tmpfile = os.path.basename(filename)
    with open(tmpfile, "wb") as f:
      f.write(contents)
    subprocess.check_call(["clang-format", "-style=file", "-i", filename])
    with open(filename, "rb") as f:
      contents = f.read()
    new_blob_id = value.insert_file_with_contents(contents)

    return (filename, mode, new_blob_id)
    '
```

However, one can write a script that uses filter-repo as a library to
simplify this, while also gaining filter-repo's automatic handling of
other concerns like rewriting commit IDs in commit messages or pruning
commits that become empty.  In fact, one of the [contrib
demos](../contrib/filter-repo-demos),
[lint-history](../contrib/filter-repo-demos/lint-history), was
specifically written to make this kind of case really easy:

```shell
  lint-history --relevant 'return filename.endswith(b".c")' \
      clang-format -style=file -i
```

---

## File: Documentation/examples-from-user-filed-issues.md

# Examples from user-filed issues

Lots of people have filed issues against git-filter-repo, and many times their
issue boils down into questions of "How do I?" or "Why doesn't this work?"

Below are a collection of example repository filterings in answer to their
questions, which may be of interest to others.

## Table of Contents

  * [Adding files to root commits](#adding-files-to-root-commits)
  * [Purge a large list of files](#purge-a-large-list-of-files)
  * [Extracting a library from a repo](#Extracting-a-library-from-a-repo)
  * [Replace words in all commit messages](#Replace-words-in-all-commit-messages)
  * [Only keep files from two branches](#Only-keep-files-from-two-branches)
  * [Renormalize end-of-line characters and add a .gitattributes](#Renormalize-end-of-line-characters-and-add-a-gitattributes)
  * [Remove spaces at the end of lines](#Remove-spaces-at-the-end-of-lines)
  * [Having both exclude and include rules for filenames](#Having-both-exclude-and-include-rules-for-filenames)
  * [Removing paths with a certain extension](#Removing-paths-with-a-certain-extension)
  * [Removing a directory](#Removing-a-directory)
  * [Convert from NFD filenames to NFC](#Convert-from-NFD-filenames-to-NFC)
  * [Set the committer of the last few commits to myself](#Set-the-committer-of-the-last-few-commits-to-myself)
  * [Handling special characters, e.g. accents in names](#Handling-special-characters-eg-accents-in-names)
  * [Handling repository corruption](#Handling-repository-corruption)
  * [Removing all files with a backslash in them](#Removing-all-files-with-a-backslash-in-them)
  * [Replace a binary blob in history](#Replace-a-binary-blob-in-history)
  * [Remove commits older than N days](#Remove-commits-older-than-N-days)
  * [Replacing pngs with compressed alternative](#Replacing-pngs-with-compressed-alternative)
  * [Updating submodule hashes](#Updating-submodule-hashes)
  * [Using multi-line strings in callbacks](#Using-multi-line-strings-in-callbacks)


## Adding files to root commits



Here's an example that will take `/path/to/existing/README.md` and
store it as `README.md` in the repository, and take
`/home/myusers/mymodule.gitignore` and store it as `src/.gitignore` in
the repository:

```
git filter-repo --commit-callback "if not commit.parents: commit.file_changes += [
    FileChange(b'M', b'README.md', b'$(git hash-object -w '/path/to/existing/README.md')', b'100644'), 
    FileChange(b'M', b'src/.gitignore', b'$(git hash-object -w '/home/myusers/mymodule.gitignore')', b'100644')]"
```

Alternatively, you could also use the [insert-beginning](../contrib/filter-repo-demos/insert-beginning) contrib script:

```
mv /path/to/existing/README.md README.md
mv /home/myusers/mymodule.gitignore src/.gitignore
insert-beginning --file README.md
insert-beginning --file src/.gitignore
```

## Purge a large list of files



Stick all the files in some file (one per line),
e.g. `../DELETED_FILENAMES.txt`, and then run

```
git filter-repo --invert-paths --paths-from-file ../DELETED_FILENAMES.txt
```

## Extracting a library from a repo



If you want to pick out some subdirectory to keep
(e.g. `src/some-filder/some-feature/`), but don't want it moved to the
repository root (so that --subdirectory-filter isn't applicable) but
instead want it to become some other higher level directory
(e.g. `src/`):

```
git filter-repo \
    --path src/some-folder/some-feature/ \
    --path-rename src/some-folder/some-feature/:src/
```

## Replace words in all commit messages



Replace "stuff" in any commit message with "task".

```
git filter-repo --message-callback 'return message.replace(b"stuff", b"task")'
```

## Only keep files from two branches



Let's say you know that the files currently present on two branches
are the only files that matter.  Files that used to exist in either of
these branches, or files that only exist on some other branch, should
all be deleted from all versions of history.  This can be accomplished
by getting a list of files from each branch, combining them, sorting
the list and picking out just the unique entries, then passing the
result to `--paths-from-file`:

```
git ls-tree -r ${BRANCH1} >../my-files
git ls-tree -r ${BRANCH2} >>../my-files
sort ../my-files | uniq >../my-relevant-files
git filter-repo --paths-from-file ../my-relevant-files
```

## Renormalize end-of-line characters and add a .gitattributes



```
contrib/filter-repo-demos/lint-history dos2unix
[edit .gitattributes]
contrib/filter-repo-demos/insert-beginning .gitattributes
```

## Remove spaces at the end of lines



Removing all spaces at the end of lines of non-binary files, including
converting CRLF to LF:

```
git filter-repo --replace-text <(echo 'regex:[\r\t ]+(\n|$)==>\n')
```

## Having both exclude and include rules for filenames



If you want to have rules to both include and exclude filenames, you
can simply invoke `git filter-repo` multiple times.  Alternatively,
you can do it in one run if you dispense with `--path` arguments and
instead use the more generic `--filename-callback`.  For example to
include all files under `src/` except for `src/README.md`:

```
git filter-repo --filename-callback '
    if filename == b"src/README.md":
        return None
    if filename.startswith(b"src/"):
        return filename
  return None'
```

## Removing paths with a certain extension



```
git filter-repo --invert-paths --path-glob '*.xsa'
```

or

```
git filter-repo --filename-callback '
    if filename.endswith(b".xsa"):
        return None
    return filename'
```

## Removing a directory



```
git filter-repo --path node_modules/electron/dist/ --invert-paths
```

## Convert from NFD filenames to NFC



Given that Mac does utf-8 normalization of filenames, and has
historically switched which kind of normalization it does, users may
have committed files with alternative normalizations to their
repository.  If someone wants to convert filenames in NFD form to NFC,
they could run

```
git filter-repo --filename-callback '
    try: 
        return subprocess.check_output("iconv -f utf-8-mac -t utf-8".split(),
                                       input=filename)
    except:
        return filename
'
```

or instead of relying on the system iconv utility and spawning separate
processes, doing it within python:

```
git filter-repo --filename-callback '
    import unicodedata
    try:
       return bytearray(unicodedata.normalize('NFC', filename.decode('utf-8')), 'utf-8')
    except:
      return filename
'
```
  
## Set the committer of the last few commits to myself



```
git filter-repo --refs main~5..main --commit-callback '
    commit.commiter_name = b"My Wonderful Self"
    commit.committer_email = b"my@self.org"
'
```

## Handling special characters, e.g. accents and umlauts in names



Since characters like ë and á are multi-byte characters and python
won't allow you to directly place those in a bytestring
(e.g. `b"Raphaël González"` would result in a `SyntaxError: bytes can
only contain ASCII literal characters` error from Python), you just
need to make a normal (UTF-8) string and then convert to a bytestring
to handle these.  For example, changing the author name and email
where the author email is currently `example@test.com`:

```
git filter-repo --refs main~5..main --commit-callback '
    if commit.author_email = b"example@test.com":
        commit.author_name = "Raphaël González".encode()
        commit.author_email = b"rgonzalez@test.com"
'
```

## Handling repository corruption



First, run fsck to get a list of the corrupt objects, e.g.:
```
$ git fsck --full
error in commit 166f57b3fbe31257100361ecaf735f305b533b21: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tree c15680eae81cc8539af7e7de766a8a7c13bd27df: duplicateEntries: contains duplicate file entries
Checking object directories: 100% (256/256), done.
```

Odds are you'll only see one type of corruption, but if you see
multiple, you can either do multiple filterings, or create replacement
objects for all the corrupt objects (both commits and trees), and then
do the filtering.  Since the method for handling corrupt commits and
corrupt tress is slightly different, I'll give examples below for each.

### Handling repository corruption -- commit objects

Print out the corrupt object literally to a temporary file:
```
$ git cat-file -p 166f57b3fbe31257100361ecaf735f305b533b21 >tmp
```

Taking a look at the file would show, for example:
```
$ cat tmp
tree e1d871155fce791680ec899fe7869067f2b4ffd2
author My Name <my@email.com>1673287380 -0800
committer My Name <my@email.com> 1673287380 -0800

Initial
```

Edit that file to fix the error (in this case, the missing space
between author email and author date).  In this case, it would look
like this after editing:

```
tree e1d871155fce791680ec899fe7869067f2b4ffd2
author My Name <my@email.com> 1673287380 -0800
committer My Name <my@email.com> 1673287380 -0800

Initial
```

Save the updated file, then use `git replace` to make a replace reference
for it.
```
$ git replace -f 166f57b3fbe31257100361ecaf735f305b533b21 $(git hash-object -t commit -w tmp)
```

Then remove the temporary file `tmp` and run `filter-repo` to consume
the replace reference and make it permanent:

```
$ rm tmp
$ git filter-repo --proceed
```

Note that if you have multiple corrupt objects, you need to create
replacements for all of them, and then run filter-repo.  Leaving any
corrupt object without a replacement is likely to cause the filter-repo run
to fail.

### Handling repository corruption -- tree objects



Print out the corrupt object literally to a temporary file:
```
$ git cat-file -p c15680eae81cc8539af7e7de766a8a7c13bd27df >tmp
```

Taking a look at the file would show, for example:
```
$ cat tmp
100644 blob cd5ded43e86f80bfd384702e3f4cc7ce42de49f9	.gitignore
100644 blob 226febfcc91ec2c166a5a06834fb47c3553ec469	README.md
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391	src
040000 tree df2b8fc99e1c1d4dbc0a854d9f72157f1d6ea078	src
040000 tree 99d732476808176bb9d73bcbfe2505e43d65cb4f	t
```

Edit that file to fix the error (in this case, removing either the `src`
file (blob) or the `src` directory (tree)).  In this case, it might look
like this after editing:

```
100644 blob cd5ded43e86f80bfd384702e3f4cc7ce42de49f9	.gitignore
100644 blob 226febfcc91ec2c166a5a06834fb47c3553ec469	README.md
040000 tree df2b8fc99e1c1d4dbc0a854d9f72157f1d6ea078	src
040000 tree 99d732476808176bb9d73bcbfe2505e43d65cb4f	t
```

Save the updated file, then use `git mktree` to turn it into an actual
tree object:
```
$ git mktree <tmp
ace04f50a5d13b43e94c12802d3d8a6c66a35b1d
```

Now use the output of that command to create a replacement object for
the original corrupt object:
```
git replace -f c15680eae81cc8539af7e7de766a8a7c13bd27df ace04f50a5d13b43e94c12802d3d8a6c66a35b1d
```

Then remove the temporary file `tmp` and run `filter-repo` to consume
the replace reference and make it permanent:

```
$ rm tmp
$ git filter-repo --proceed
```

As mentioned with corrupt commit objects, if you have multiple corrupt
objects, as long as you create all the replacements for those objects
first, you only need to run filter-repo once.


## Removing all files with a backslash in them



```
git filter-repo --filename-callback 'return None if b'\\' in filename else filename'
```

## Replace a binary blob in history



Let's say you committed a binary blob, perhaps an image file, with
sensitive data, and never modified it.  You want to replace it with
the contents of some alternate file, currently found at
`../alternative-file.jpg` (it can have a different filename than what
is stored in the repository).  Let's also say the hash of the old file
was `f4ede2e944868b9a08401dafeb2b944c7166fd0a`.  You can replace it
with either

```
git filter-repo --blob-callback '
    if blob.original_id == b"f4ede2e944868b9a08401dafeb2b944c7166fd0a":
        blob.data = open("../alternative-file.jpg", "rb").read()
'
```

or

```
git replace -f f4ede2e944868b9a08401dafeb2b944c7166fd0a $(git hash-object -w ../alternative-file.jpg)
git filter-repo --proceed
```

## Remove commits older than N days



This is such a bad usecase.  I'm tempted to leave it out, but it has
come up multiple times, and there are people who are totally fine with
changing every commit hash in their repository and throwing away
history periodically.  First, identify an ${OLD_COMMIT} that you want
to be a new root commit, then run:

```
git replace --graft ${OLD_COMMIT}
git filter-repo --proceed
```

(The trick here is that `git replace --graft` takes a commit to replace, and
a list of new parents for the commit.  Since ${OLD_COMMIT} is the final
positional argument, it means the list of new parents is an empty list, i.e.
we are turning it into a new root commit.)

## Replacing pngs with compressed alternative



Let's say you committed thousands of pngs that were poorly compressed,
but later aggressively recompressed the pngs and committed and pushed.
Unfortunately, clones are slow because they still contain the poorly
compressed pngs and you'd like to rewrite history to pretend that the
aggressively compressed versions were used when the files were first
introduced.

First, take a look at the commit that aggressively recompressed the pngs:

```
git log -1 --raw --no-abbrev ${COMMIT_WHERE_YOU_COMPRESSED_PNGS}
```

that will show output like
```
:100755 100755 edf570fde099c0705432a389b96cb86489beda09 9cce52ae0806d695956dcf662cd74b497eaa7b12 M      resources/foo.png
:100755 100755 644f7c55e1a88a29779dc86b9ff92f512bf9bc11 88b02e9e45c0a62db2f1751b6c065b0c2e538820 M      resources/bar.png
```

Use that to make a --file-info-callback to fix up the original versions:
```
git filter-repo --file-info-callback '
    if filename == b"resources/foo.png" and blob_id == b"edf570fde099c0705432a389b96cb86489beda09":
        blob_id = b"9cce52ae0806d695956dcf662cd74b497eaa7b12"
    if filename == b"resources/bar.png" and blob_id == b"644f7c55e1a88a29779dc86b9ff92f512bf9bc11":
        blob_id = b"88b02e9e45c0a62db2f1751b6c065b0c2e538820"
    return (filename, mode, blob_id)
'
```

## Updating submodule hashes



Let's say you have a repo with a submodule at src/my-submodule, and
that you feel the wrong commit-hashes of the submodule were committed
within your project and you want them updated according to the
following table:
```
old                                      new
edf570fde099c0705432a389b96cb86489beda09 9cce52ae0806d695956dcf662cd74b497eaa7b12
644f7c55e1a88a29779dc86b9ff92f512bf9bc11 88b02e9e45c0a62db2f1751b6c065b0c2e538820
```

You could do this as follows:
```
git filter-repo --file-info-callback '
    if filename == b"src/my-submodule" and blob_id == b"edf570fde099c0705432a389b96cb86489beda09":
        blob_id = b"9cce52ae0806d695956dcf662cd74b497eaa7b12"
    if filename == b"src/my-submodule" and blob_id == b"644f7c55e1a88a29779dc86b9ff92f512bf9bc11":
        blob_id = b"88b02e9e45c0a62db2f1751b6c065b0c2e538820"
    return (filename, mode, blob_id)
```

Yes, `blob_id` is kind of a misnomer here since the file's hash
actually refers to a commit from the sub-project.  But `blob_id` is
the name of the parameter passed to the --file-info-callback, so that
is what must be used.

## Using multi-line strings in callbacks



Since the text for callbacks have spaces inserted at the front of every
line, multi-line strings are normally munged.  For example, the command

```
git filter-repo --blob-callback '
  blob.data = bytes("""\
This is the new
file that I am
replacing every blob
with.  It is great.\n""", "utf-8")
'
```

would result in a file with extra spaces at the front of every line:
```
  This is the new
  file that I am
  replacing every blob
  with.  It is great.
```

The two spaces at the beginning of every-line were inserted into every
line of the callback when trying to compile it as a function.
However, you can use textwrap.dedent to fix this; in fact, using it
will even allow you to add more leading space so that it looks nicely
indented.  For example:

```
git filter-repo --blob-callback '
  import textwrap
  blob.data = bytes(textwrap.dedent("""\
    This is the new
    file that I am
    replacing every blob
    with.  It is great.\n"""), "utf-8")
'
```

That will result in a file with contents
```
This is the new
file that I am
replacing every blob
with.  It is great.
```

which has no leading spaces on any lines.

---

## File: Documentation/FAQ.md

# Frequently Answered Questions

## Table of Contents

  * [Why did `git-filter-repo` rewrite commit hashes?](#why-did-git-filter-repo-rewrite-commit-hashes)
  * [Why did `git-filter-repo` rewrite more commit hashes than I expected?](#why-did-git-filter-repo-rewrite-more-commit-hashes-than-i-expected)
  * [Why did `git-filter-repo` rewrite other branches too?](#why-did-git-filter-repo-rewrite-other-branches-too)
  * [How should paths be specified?](#How-should-paths-be-specified)
  * [Help! Can I recover or undo the filtering?](#help-can-i-recover-or-undo-the-filtering)
  * [Can you change `git-filter-repo` to allow future folks to recover from `--force`'d rewrites?](#can-you-change-git-filter-repo-to-allow-future-folks-to-recover-from---forced-rewrites)
  * [Can I use `git-filter-repo` to fix a repository with corruption?](#Can-I-use-git-filter-repo-to-fix-a-repository-with-corruption)
  * [What kinds of problems does `git-filter-repo` not try to solve?](#What-kinds-of-problems-does-git-filter-repo-not-try-to-solve)
    * [Filtering history but magically keeping the same commit IDs](#Filtering-history-but-magically-keeping-the-same-commit-IDs)
    * [Bidirectional development between a filtered and unfiltered repository](#Bidirectional-development-between-a-filtered-and-unfiltered-repository)
    * [Removing specific commits, or filtering based on the difference (a.k.a. patch or change) between commits](#Removing-specific-commits-or-filtering-based-on-the-difference-aka-patch-or-change-between-commits)
    * [Filtering two different clones of the same repository and getting the same new commit IDs](#Filtering-two-different-clones-of-the-same-repository-and-getting-the-same-new-commit-IDs)

## Why did `git-filter-repo` rewrite commit hashes?

This is fundamental to how Git operates.  In more detail...

Each commit in Git is a hash of its contents.  Those contents include
the commit message, the author (name, email, and time authored), the
committer (name, email and time committed), the toplevel tree hash,
and the parent(s) of the commit.  This means that if any of the commit
fields change, including the tree hash or the hash of the parent(s) of
the commit, then the hash for the commit will change.

(The same is true for files ("blobs") and trees stored in git as well;
each is a hash of its contents, so literally if anything changes, the
commit hash will change.)

If you attempt to write a commit (or tree or blob) object with an
incorrect hash, Git will reject it as corrupt.

## Why did `git-filter-repo` rewrite more commit hashes than I expected?

There are two aspects to this, or two possible underlying questions users
might be asking here:
  * Why did commits newer than the ones I expected have their hash change?
  * Why did commits older than the ones I expected have their hash change?

For the first question, see [why filter-repo rewrites commit
hashes](#why-did-git-filter-repo-rewrite-commit-hashes), and note that
if you modify some old commit, perhaps to remove a file, then obviously
that commit's hash must change.  Further, since that commit will have a
new hash, any other commit with that commit as a parent will need to
have a new hash.  That will need to chain all the way to the most recent
commits in history.  This is fundamental to Git and there is nothing you
can do to change this.

For the second question, there are two causes: (1) the filter you
specified applies to the older commits too, or (2) git-fast-export and
git-fast-import (both of which git-filter-repo uses) canonicalize
history in various ways.  The second cause means that even if you have
no filter, these tools sometimes change commit hashes.  This can happen
in any of these cases:

  * If you have signed commits, the signatures will be stripped
  * If you have commits with extended headers, the extended headers will
    be stripped (signed commits are actually a special case of this)
  * If you have commits in an encoding other than UTF-8, they will by
    default be re-encoded into UTF-8
  * If you have a commit without an author, one will be added that
    matches the committer.
  * If you have trees that are not canonical (e.g. incorrect sorting
    order), they will be canonicalized

If this affects you and you really only want to rewrite newer commits in
history, you can use the `--refs` argument to git-filter-repo to specify
a range of history that you want rewritten.

(For those attempting to be clever and use `--refs` for the first
question: Note that if you attempt to only rewrite a few old commits,
then all you'll succeed in is adding new commits that won't be part of
any branch and will be subject to garbage collection.  The branches will
still hold on to the unrewritten versions of the commits.  Thus, you
have to rewrite all the way to the branch tip for the rewrite to be
meaningful.  Said another way, the `--refs` trick is only useful for
restricting the rewrite to newer commits, never for restricting the
rewrite to older commits.)

## Why did `git-filter-repo` rewrite other branches too?

git-filter-repo's name is git-filter-**_repo_**.  Obviously it is going
to rewrite all branches by default.

`git-filter-repo` can restrict its rewriting to a subset of history,
such as a single branch, using the `--refs` option.  However, using that
comes with the risk that one branch now has a different version of some
commits than other branches do; usually, when you rewrite history, you
want all branches that depend on what you are rewriting to be updated.

## How should paths be specified?

Arguments to `--path` should be paths as Git would report them, when run
from the toplevel of the git repository (explained more below after some
examples).

**Good** path examples:
  * `README.md`
  * `Documentation/README.md`
  * `src/modules/flux/capacitor.rs`

You can find examples of valid path names from your repository by
running either `git diff --no-relative --name-only` or `git log
--no-relative --name-only --format=""`.

The following are basic rules about paths the way that Git reports and uses
them:
  * do not use absolute paths
  * always treats paths as relative to the toplevel of the repository
    (do not add a leading slash, and do not specify paths relative to some
     subdirectory of the repository even if that is your current working
     directory)
  * do not use the special directories `.` or `..` anywhere in your path
  * do not use `\`,  the Windows path separator, between directories and
    files; always use `/` regardless of platform.

**Erroneous** path examples (do **_NOT_** use any of these styles):
 * `/absolute/path/to/src/modules/program.c`
 * `/src/modules/program.c`
 * `src/docs/../modules/main.java`
 * `scripts/config/./update.sh`
 * `./tests/fixtures/image.jpg`
 * `../src/main.rs`
 * `C:\absolute\path\to\src\modules\program.c`
 * `src\modules\program.c`

## Help! Can I recover or undo the filtering?

Sure, _if_ you followed the instructions.  The instructions told you to
make a fresh clone before running git-filter-repo.  If you did that (and
didn't force push your rewritten history back over the original), you
can just throw away your clone with the flubbed rewrite, and make a new
clone.

If you didn't make a fresh clone, and you didn't run with `--force`, you
would have seen the following warning:
```
Aborting: Refusing to destructively overwrite repo history since
this does not look like a fresh clone.
[...]
Please operate on a fresh clone instead.  If you want to proceed
anyway, use --force.
```
If you then added `--force`, well, you were warned.

If you didn't make a fresh clone, and you started with `--force`, and you
didn't think to read the description of the `--force` option:
```
	Ignore fresh clone checks and rewrite history (an irreversible
	operation, especially since it by default ends with an
	immediate pruning of reflogs and old objects).
```
and you didn't read even the beginning of the manual
```
git-filter-repo destructively rewrites history
```
and you think it's okay to run a command with `--force` in it on
something you don't have a backup of, then now is the time to reassess
your life choices.  `--force` should be a pretty clear warning sign.
(If someone on the internet suggested `--force`, you can complain at
_them_, but either way you should learn to carefully vet commands
suggested by others on the internet.  Sadly, even sites like Stack
Overflow where someone really ought to be able to correct bad guidance
still unfortunately has a fair amount of this bad advice.)

See also the next question.

## Can you change `git-filter-repo` to allow future folks to recover from --force'd rewrites?

This will never be supported.

* Providing an alternate method to restore would require storing both
  the original history and the new history, meaning that those who are
  trying to shrink their repository size instead see it grow and have to
  figure out extra steps to expunge the old history to see the actual
  size savings.  Experience with other tools showed that this was
  frustrating and difficult to figure out for many users.

* Providing an alternate method to restore would mean that users who are
  trying to purge sensitive data from their repository still find the
  sensitive data after the rewrite because it hasn't actually been
  purged. In order to actually purge it, they have to take extra steps.
  Same as with the last bullet point, experience has shown that extra
  steps to purge the extra information is difficult and error-prone.
  This extra difficulty is particularly problematic when you're trying
  to expunge sensitive data.

* Providing an alternate method to restore would also mean trying to
  figure out what should be backed up and how. The obvious choices used
  by previous tools only actually provided partial backups (reflogs
  would be ignored for example, as would uncommitted changes whether
  staged or not). The more you try to carefully backup everything, the
  more difficult the restoration from backup will be.  The only backup
  mechanism I've found that seems reasonable, is making a separate
  clone.  That's expensive to do automatically for the user (especially
  if the filtering is done via multiple invocations of the tool).  Plus,
  it's not clear where the clone should be stored, especially to avoid
  the previous problems for size-reduction and sensitive-data-removal
  folks.

* Providing an alternate method to restore would also mean providing
  documentation on how to restore. Past methods by other tools in the
  history rewriting space suggested that it was rather difficult for
  users to figure out.  Difficult enough, in fact, that users simply
  didn't ever use them.  They instead made a separate clone before
  rewriting history and if they didn't like the rewrite, then they just
  blew it away and made a new clone to work with.  Since that was
  observed to be the easy restoration method, I simply enforced it with
  this tool, requiring users who look like they might not be operating
  on a fresh clone to use the --force flag.

But more than all that, if there were an alternate method to restore,
why would you have needed to specify the --force flag? Doesn't its
existence (and the wording of its documentation) make it pretty clear on
its own that there isn't going to be a way to restore?

## Can I use `git-filter-repo` to fix a repository with corruption?

Some kinds of corruption can be fixed, in conjunction with `git
replace`.  If `git fsck` reports warnings/errors for certain objects,
you can often [replace them and rewrite
history](examples-from-user-filed-issues.md#Handling-repository-corruption).

## What kinds of problems does `git-filter-repo` not try to solve?

This question is often asked in the form of "How do I..." or even
written as a statement such as "I found a bug with `git-filter-repo`;
the behavior I got was different than I expected..."  But if you're
trying to do one of the things below, then `git-filter-repo` is behaving
as designed and either there is no solution to your problem, or you need
to use a different tool to solve your problem.  The following subsections
address some of these common requests:

### Filtering history but magically keeping the same commit IDs

This is impossible.  If you modify commits, or the files contained in
them, then you change their commit IDs; this is [fundamental to
Git](#why-did-git-filter-repo-rewrite-commit-hashes).

However, _if_ you don't need to modify commits, but just don't want to
download everything, then look into one of the following:
  * [partial clones](https://git-scm.com/docs/partial-clone)
  * the ugly, retarded hack known as [shallow clones](https://git-scm.com/docs/shallow)
  * a massive hack like [cheap fake
    clones](https://github.com/newren/sequester-old-big-blobs) that at
    least let you put your evil overlord laugh to use

### Bidirectional development between a filtered and unfiltered repository

Some folks want to extract a subset of a repository, do development work
on it, then bring those changes back to the original repository, and
send further changes in both directions.  Such a tool can be written
using fast-export and fast-import, but would need to make very different
design decisions than `git-filter-repo` did.  Such a tool would be
capable of supporting this kind of development, but lose the ability
["to write arbitrary filters using a scripting
language"](https://josh-project.github.io/josh/#concept) and other
features that `git-filter-repo` has.

Such a tool exists; it's called [Josh](https://github.com/josh-project/josh).
Use it if this is your usecase.

### Removing specific commits, or filtering based on the difference (a.k.a. patch or change) between commits

You are probably looking for `git rebase`.  `git rebase` operates on the
difference between commits ("diff"), allowing you to e.g. drop or modify
the diff, but then runs the risk of conflicts as it attempts to apply
future diffs. If you tweak one diff in the middle, since it just applies
more diffs for the remaining patches, you'll still see your changes at
the end.

filter-repo, by contrast, uses fast-export and fast-import.  Those tools
treat every commit not as a diff but as a "use the same versions of most
files from the parent commit, but make these five files have these exact
contents". Since you don't have either the diff or ready access to the
version of files from the parent commit, that makes it hard to "undo"
part of the changes to some file.  Further, if you attempt to drop an
entire commit or tweak the contents of those new files in that commit,
those changes will be reverted by the next commit in the stream that
mentions that file because handling the next commit does not involve
applying a diff but a "make this file have these exact contents". So,
filter-repo works well for things like removing a file entirely, but if
you want to make any tweaks to any files you have to make the exact same
tweak over and over for every single commit that touches that file.

In short, `git rebase` is the tool you want for removing specific
commits or otherwise operating on the diff between commits.

### Filtering two different clones of the same repository and getting the same new commit IDs

Sometimes two co-workers have a clone of the same repository and they
run the same `git-filter-repo` command, and they expect to get the same
new commit IDs.  Often they do get the same new commit IDs, but
sometimes they don't.

When people get the same commit IDs, it is only by luck; not by design.
There are three reasons this is unsupported and will never be reliable:

  * Different Git versions used could cause differences in filtering

    Since `git fast-export` and `git fast-import` do various
    canonicalizations of history, and these could change over time,
    having different versions of Git installed can result in differences
    in filtering.

  * Different git-filter-repo versions used could cause differences in
    filtering

    Over time, `git-filter-repo` may include new filterings by default,
    or fix existing filterings, or make any other number of changes.  As
    such, having different versions of `git-filter-repo` installed can
    result in differences in filtering.

  * Different amounts of the repository cloned or differences in
    local-only commits can cause differences in filtering

    If the clones weren't made at the same time, one clone may have more
    commits than the other.  Also, both may have made local commits the
    other doesn't have.  These additional commits could cause history to
    be traversed in a different order, and filtering rules are allowed
    to have order-dependent rules for how they filter.  Further,
    filtering rules are allowed to depend upon what history exists in
    your clone.  As one example, filter-repo's default to update commit
    messages which refer to other commits by abbreviated hash, may be
    unable to find these other commits in your clone but find them in
    your coworkers' clone.  Relatedly, filter-repo's update of
    abbreviated hashes in commit messages only works for commits that
    have already been filtered, and thus depends on the order in which
    fast-export traverses the history.

`git-filter-repo` is designed as a _one_-shot history rewriting tool.
Once you have filtered one clone of the repository, you should not be
using it to filter other clones.  All other clones of the repository
should either be discarded and recloned, or [have all their history
rebased on top of the rewritten
history](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#_make_sure_other_copies_are_cleaned_up_clones_of_colleagues).