mirror of https://github.com/git/git.git
Browse Source
A mechanism to pack unreachable objects into a "cruft pack", instead of ejecting them into loose form to be reclaimed later, has been introduced. * tb/cruft-packs: sha1-file.c: don't freshen cruft packs builtin/gc.c: conditionally avoid pruning objects via loose builtin/repack.c: add cruft packs to MIDX during geometric repack builtin/repack.c: use named flags for existing_packs builtin/repack.c: allow configuring cruft pack generation builtin/repack.c: support generating a cruft pack builtin/pack-objects.c: --cruft with expiration reachable: report precise timestamps from objects in cruft packs reachable: add options to add_unseen_recent_objects_to_traversal builtin/pack-objects.c: --cruft without expiration builtin/pack-objects.c: return from create_object_entry() t/helper: add 'pack-mtimes' test-tool pack-mtimes: support writing pack .mtimes files chunk-format.h: extract oid_version() pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles' pack-mtimes: support reading .mtimes files Documentation/technical: add cruft-packs.txtpull/1274/head
Junio C Hamano
2 months ago
32 changed files with 1853 additions and 102 deletions
@ -0,0 +1,123 @@
|
||||
= Cruft packs |
||||
|
||||
The cruft packs feature offer an alternative to Git's traditional mechanism of |
||||
removing unreachable objects. This document provides an overview of Git's |
||||
pruning mechanism, and how a cruft pack can be used instead to accomplish the |
||||
same. |
||||
|
||||
== Background |
||||
|
||||
To remove unreachable objects from your repository, Git offers `git repack -Ad` |
||||
(see linkgit:git-repack[1]). Quoting from the documentation: |
||||
|
||||
[quote] |
||||
[...] unreachable objects in a previous pack become loose, unpacked objects, |
||||
instead of being left in the old pack. [...] loose unreachable objects will be |
||||
pruned according to normal expiry rules with the next 'git gc' invocation. |
||||
|
||||
Unreachable objects aren't removed immediately, since doing so could race with |
||||
an incoming push which may reference an object which is about to be deleted. |
||||
Instead, those unreachable objects are stored as loose objects and stay that way |
||||
until they are older than the expiration window, at which point they are removed |
||||
by linkgit:git-prune[1]. |
||||
|
||||
Git must store these unreachable objects loose in order to keep track of their |
||||
per-object mtimes. If these unreachable objects were written into one big pack, |
||||
then either freshening that pack (because an object contained within it was |
||||
re-written) or creating a new pack of unreachable objects would cause the pack's |
||||
mtime to get updated, and the objects within it would never leave the expiration |
||||
window. Instead, objects are stored loose in order to keep track of the |
||||
individual object mtimes and avoid a situation where all cruft objects are |
||||
freshened at once. |
||||
|
||||
This can lead to undesirable situations when a repository contains many |
||||
unreachable objects which have not yet left the grace period. Having large |
||||
directories in the shards of `.git/objects` can lead to decreased performance in |
||||
the repository. But given enough unreachable objects, this can lead to inode |
||||
starvation and degrade the performance of the whole system. Since we |
||||
can never pack those objects, these repositories often take up a large amount of |
||||
disk space, since we can only zlib compress them, but not store them in delta |
||||
chains. |
||||
|
||||
== Cruft packs |
||||
|
||||
A cruft pack eliminates the need for storing unreachable objects in a loose |
||||
state by including the per-object mtimes in a separate file alongside a single |
||||
pack containing all loose objects. |
||||
|
||||
A cruft pack is written by `git repack --cruft` when generating a new pack. |
||||
linkgit:git-pack-objects[1]'s `--cruft` option. Note that `git repack --cruft` |
||||
is a classic all-into-one repack, meaning that everything in the resulting pack is |
||||
reachable, and everything else is unreachable. Once written, the `--cruft` |
||||
option instructs `git repack` to generate another pack containing only objects |
||||
not packed in the previous step (which equates to packing all unreachable |
||||
objects together). This progresses as follows: |
||||
|
||||
1. Enumerate every object, marking any object which is (a) not contained in a |
||||
kept-pack, and (b) whose mtime is within the grace period as a traversal |
||||
tip. |
||||
|
||||
2. Perform a reachability traversal based on the tips gathered in the previous |
||||
step, adding every object along the way to the pack. |
||||
|
||||
3. Write the pack out, along with a `.mtimes` file that records the per-object |
||||
timestamps. |
||||
|
||||
This mode is invoked internally by linkgit:git-repack[1] when instructed to |
||||
write a cruft pack. Crucially, the set of in-core kept packs is exactly the set |
||||
of packs which will not be deleted by the repack; in other words, they contain |
||||
all of the repository's reachable objects. |
||||
|
||||
When a repository already has a cruft pack, `git repack --cruft` typically only |
||||
adds objects to it. An exception to this is when `git repack` is given the |
||||
`--cruft-expiration` option, which allows the generated cruft pack to omit |
||||
expired objects instead of waiting for linkgit:git-gc[1] to expire those objects |
||||
later on. |
||||
|
||||
It is linkgit:git-gc[1] that is typically responsible for removing expired |
||||
unreachable objects. |
||||
|
||||
== Caution for mixed-version environments |
||||
|
||||
Repositories that have cruft packs in them will continue to work with any older |
||||
version of Git. Note, however, that previous versions of Git which do not |
||||
understand the `.mtimes` file will use the cruft pack's mtime as the mtime for |
||||
all of the objects in it. In other words, do not expect older (pre-cruft pack) |
||||
versions of Git to interpret or even read the contents of the `.mtimes` file. |
||||
|
||||
Note that having mixed versions of Git GC-ing the same repository can lead to |
||||
unreachable objects never being completely pruned. This can happen under the |
||||
following circumstances: |
||||
|
||||
- An older version of Git running GC explodes the contents of an existing |
||||
cruft pack loose, using the cruft pack's mtime. |
||||
- A newer version running GC collects those loose objects into a cruft pack, |
||||
where the .mtime file reflects the loose object's actual mtimes, but the |
||||
cruft pack mtime is "now". |
||||
|
||||
Repeating this process will lead to unreachable objects not getting pruned as a |
||||
result of repeatedly resetting the objects' mtimes to the present time. |
||||
|
||||
If you are GC-ing repositories in a mixed version environment, consider omitting |
||||
the `--cruft` option when using linkgit:git-repack[1] and linkgit:git-gc[1], and |
||||
leaving the `gc.cruftPacks` configuration unset until all writers understand |
||||
cruft packs. |
||||
|
||||
== Alternatives |
||||
|
||||
Notable alternatives to this design include: |
||||
|
||||
- The location of the per-object mtime data, and |
||||
- Storing unreachable objects in multiple cruft packs. |
||||
|
||||
On the location of mtime data, a new auxiliary file tied to the pack was chosen |
||||
to avoid complicating the `.idx` format. If the `.idx` format were ever to gain |
||||
support for optional chunks of data, it may make sense to consolidate the |
||||
`.mtimes` format into the `.idx` itself. |
||||
|
||||
Storing unreachable objects among multiple cruft packs (e.g., creating a new |
||||
cruft pack during each repacking operation including only unreachable objects |
||||
which aren't already stored in an earlier cruft pack) is significantly more |
||||
complicated to construct, and so aren't pursued here. The obvious drawback to |
||||
the current implementation is that the entire cruft pack must be re-written from |
||||
scratch. |
||||