Thread by @ireneista

$ du -sh .git
2.6G .git

$ git gc
(currently running, we'll share output when it's done)

this is our local copy of the nixpkgs repo, which we keep on most of our machines because it's handy

we gave up on running git gc --aggressive on the machine that was low on disk space, but it occurred to us belatedly that we can run it on a larger machine and copy the result over

Enumerating objects: 2895079, done.
Counting objects: 100% (2895079/2895079), done.
Delta compression using up to 4 threads
Compressing objects: 100% (692280/692280), done.
Writing objects: 100% (2895079/2895079), done.

Total 2895079 (delta 2016256), reused 2870659 (delta 1991891), pack-reused 0
Checking connectivity: 2895097, done.
Expanding reachable commits in commit graph: 378544, done.

$ du -sh .git
2.1G .git

$ git gc --aggressive

if this saves even 100 MiB we'll consider it worthwhile, for our use-case. some of our smaller machines routinely run out of disk space by amounts that are, in modern terms, quite small.

(we can't upgrade this machine without replacing its enclosure and buying an extra board for it; it's an embedded ARM device and the eMMC is only so big.)

(yes we put our dev environment on embedded devices)

Enumerating objects: 2895079, done.
Counting objects: 100% (2895079/2895079), done.
Delta compression using up to 4 threads
error: pack-objects died of signal 1184171)4171)
fatal: failed to run repack

we're gonna create and map a swapfile and try again. this machine has plenty of space so it can't be disk that's the constraint, this must be the work of the OOM killer.

please note that the machine has 8 GiB physical RAM and 16 GiB swap in normal operation. we will add another 16 GiB.

for production you should use a swap partition rather than a file, it will be faster and you have more chance of making secure deletion actually work (not that it's anywhere close to sufficient)

for one-off tasks though, making a swap file is a nice trick to have around

$ sudo truncate --size=16GiB /root/swap
$ sudo mkswap /root/swap
mkswap: /root/swap: insecure permissions 0644, 0600 suggested.
Setting up swapspace version 1, size = 16 GiB (17179865088 bytes)
no label, UUID=[redacted]
$ sudo chmod 600 /root/swap

$ sudo swapon /root/swap
swapon: /root/swap: skipping - it appears to have holes.

oh. well, okay then. that explains why we didn't remember how to use truncate for this.

$ sudo dd if=/dev/zero of=/root/swap oflag=direct status=progress

43615232 bytes (44 MB, 42 MiB) copied, 34 s, 1.3 MB/s^C
$ sudo dd if=/dev/zero of=/root/swap oflag=direct bs=10M status=progress

we're pretty sure dd will stop at the existing size of the file. fingers crossed.

the block size of 10 MiB did help substantially.

$ sudo dd if=/dev/zero of=/root/swap oflag=direct bs=10M status=progress
22062039040 bytes (22 GB, 21 GiB) copied, 218 s, 101 MB/s^C[6~2C
2106+0 records in
2106+0 records out
22083010560 bytes (22 GB, 21 GiB) copied, 218.709 s, 101 MB/s

so it uh... did not stop at 16, we ^C'd it

*now* the truncate command is useful :)

$ sudo truncate --size=16GiB /root/swap
$ sudo mkswap /root/swap
$ sudo mkswap /root/swap
Setting up swapspace version 1, size = 16 GiB (17179865088 bytes)
no label, UUID=[redacted]
$ sudo swapon /root/swap

$ git gc --aggressive

fingers crossed :)

by the way, the reason we keep saying like GiB or MiB instead of abbreviating to G or M is that the binary units are important.

"megabyte" is a quantity that gets defined in, like, ten different ways, only two of which make any mathematical sense (10^6 bytes or 2^20). marketers have a strong interest in it never meaning anything unambiguous, and the confusion causes real tech problems.

"mebibyte" was defined, we think by the IEEE, in the late 90s to solve this problem. a mebibyte can only ever be exactly 2^20 bytes.

the abbreviations KiB, MiB, GiB, TiB, PiB, EiB, etc were defined at the same time to refer to them.

if you are making a command line tool that does low level disk stuff, please support the binary unit abbreviations! your users deserve not to have to quintuple-check your documentation to make sure you have the same understanding of "megs" as them.

we like to mention them in passing whenever we can, as part of normalizing and promoting their use. the personal is political, and all that.

besides, monologuing about this is more fun than watching the progress counter ;)

in another tab on the same machine we are running

$ watch "df -h; echo; free"

so that we can see when resources are getting low

almost done with the compressing phase, getting very near to where it failed last time.

it is indeed consuming ridiculous amounts of resources, but still 27 GiB of swap free (our entire new swapfile and about half of the swap partition)

also still over a hundred GiB of ordinary disk space available

hey, question to our followers! what's the largest amount of data you've ever had to query? we'll start: 160 PiB.

no, we absolutely are not going to talk about why. ;)

the "compressing objects" phase has been at 97% for at least 10 minutes now. it has consumed an additional gig of swap since our last report.

(we're reading that number off in decimal, so we don't actually know what it is in binary units and don't want to fool ourselves into thinking we do. the 1024 ~= 1000 approximation gets less accurate the larger the numbers are.)

hey it's at 98% now!

it's basically not consuming any significant amount of ordinary disk space past what it initially needed. apparently git gc is mostly memory-bound. cool.

it's also pegging all four logical cores (load average: 4.54, 4.25, 3.75), so it's hard to know whether more physical RAM would actually speed things up

99% compressed now. see, the thing about this though, is that it still has way more than 16 GiB free swap, so the OOM killer should have left it alone...

so we're not convinced we've correctly identified the cause of failure, unless the writing-objects phase gets even more RAM-heavy.

some people would try to make a blog post out of a task like this. we respect that choice, personally, but we also know from experience that when the post shows up on discussion forums people would question whether it was worth writing about.

it's only really worth writing about, to us, as tweets. no reason to polish this up into an essay.

Twitter has that nice informal quality. we do still worry about how we say things here, but - especially in unapologetically silly threads like this one - we aren't asking anyone to treat our words with reverent attention, and that's freeing.

25 GiB swap free.

ooh! and it died again. guess it wasn't the memory.

hmm

the previous attempt ended with:

error: pack-objects died of signal 1184171)4171)
fatal: failed to run repack

this one ended with

error: pack-objects died of signal 1184171)71)
fatal: failed to run repack

notice anything in common between those?

pretty sure that's just the total number of objects after compression though

so it's failing exactly at the end of the compression step, perhaps. that's at least a clue.

be nice to know what signal it was, but with the terminal output overwriting itself it's hard to be sure. could be 1 or 11. could even plausibly be 118 (signal numbers are 1 byte).

or perhaps that error message doesn't print the signal number

okay, the error message does print the signal number github.com/git/git/blob/e…

perhaps this error message should have \n prepended to it, given that it typically appears during a long-running task that's repeatedly overwriting the same line? if somebody reading this wants to drive that change, please do <3

(we have no idea what the git maintainer community is like socially, so caveat codor)

Linux signal 1 is SIGHUP, we know that offhand. we sure didn't kill this manually so we don't think that's likely to be it.

$ find /nix/store -name signal.h

it's slow but it'll do the job. we miss the convenience of global directories... ah well, it's worth it.

$ less /nix/store/[redacted]-linux-headers-4.4.10/include/asm/signal.h

(we refuse to search the web for something we can easily answer with reference to local files. we should never have put so much of our workflow into the cloud, fuck megacorps, fight the power.)

#define SIGSEGV 11

wow, we did not expect a segfault. have we hit a bug in git?????

hmmm, so searching around a bit,

1) quite possibly;
2) it looks like setting a pack size limit is likely to address it

those limits are options to git repack though, which is invoked under the hood by git gc but cannot simply be invoked by itself because we need the garbage collection behavior *first*

therefore,

$ git config --add pack.windowMemory 2g

we set that in the repo's .git/config because our ~/.gitconfig is on a read-only filesystem

we'll make sure to remove it when we're done, heh, we don't like stray things lying around that can fix and/or break things without us knowing

we could also consider setting pack.packSizeLimit, but that could result in suboptimal output

hmmm wait we can actually run repack by hand after we do the gc, that changes our plans

$ git config --add pack.windowMemory 100m
$ git config --add pack.packSizeLimit 256m

pack.windowMemory is per-thread, so wait... hmmm.... we picked 100m because we thought perhaps it was per-object or something.... four threads....

$ git config --add pack.windowMemory 4g

without actually reading the code it's hard to know exactly what this will do to the garbage collection, but anyway 4g falls within our machine's limits while being more than it really ought to need

$ git gc --aggressive

a friend suggested that the segfault could be malloc returning NULL. that makes sense, and we wouldn't really regard it as a git bug, it's a bug in the design of the POSIX C API.

97% through "compressing objects", 29 gigs of swap free.

died again. new thought: single-threaded operation.

hmmm the manpages suggest it isn't possible to do that. hope springs eternal, so we'll see if it exists as an undocumented option...

$ git gc --aggressive --threads=1

error: unknown option `threads=1'
usage: git gc [<options>]

-q, --quiet suppress progress reporting
--prune[=<date>] prune unreferenced objects
[etc]

$ git gc --aggressive --threads 1

nope. ah well. hmmmm....

hmmm ulimit seems to be telling us it can't actually limit threads on our machine

$ git gc --aggressive --prune=now

we're still operating under the belief that it's extremely likely that this will, in the end, save zero bytes. we just want to *know* that.

95%... should be less than ten minutes now. we'll refresh our beverage while we wait.

we took the liberty of hitting return while it works, to create a duplicate line of text so we can compare the final line to see which part is the error

sorry, terminal. your scrollback will contain evidence of our malfeasance, injecting our own thoughts into the stream of the program's output. at least until it hits the thousand-line scrollback limit and is washed away, like leaves on the water.

if you are now thinking: wtf that's so random, are Irenes a bizarre extradimensional anomaly? the answer is yes

Compressing objects: 95% (2567292/2684171)
error: pack-objects died of signal 1184171)

yep! definitely signal 11. cool.

if we were ever to report this upstream we'd want to be sure of that.

$ taskset -c 0 git gc --aggressive

Enumerating objects: 2895079, done.
Counting objects: 100% (2895079/2895079), done.
Delta compression using up to 4 threads

:(

load average: 2.51, 2.53, 2.45

this strongly suggests that it is running four threads *on one core*

hmm... well.... we're confident in that conclusion because think we did the taskset right, and because that isn't the way load average behaved before, but the more we think about it the less that really tells us

perhaps we can figure out what steps git gc is doing and invoke them manually, allowing us to specify options

hmmm begin with main()
github.com/git/git/blob/m…

need to know if it's a builtin or a separate binary. fortunately, code in this file informs us we can find out.

$ git --list-cmds=builtins | grep gc
gc

cool then

here's what we wanted

github.com/git/git/blob/m…

that single-core invocation is still running btw. we expect it to fail but it might as well churn away; it's not like it's running on the machine we're typing on.

ah. here's the top-level function, cmd_gc().

github.com/git/git/blob/m…

so (ignoring all the extraneous bookkeeping), it calls gc_before_repack();

then it does the equivalent of "git repack -d -l"

then "git prune --expire=now" (or whatever we provide there, defaulting to two weeks)

then "git worktree prune --expire=now"

then "git rerere gc" we swear we are not making this up

then it does some stuff we'll need to dig into, deferring to functions reprepare_packed_git() and clean_pack_garbage()

and it looks like by default it also does write_commit_graph_reachable() which is related to git-commit-graph in some way

plus it does lots of locking, for safety, which we won't be doing, so we need to remember to not mess with the repo while it's running

it takes several hundred lines of code to do all this, because this is C

so hmmmm, digging into gc_before_repack() it's pretty short. just invokes "git pack-refs --all --prune" followed by "git reflog expire --all"

reprepare_packed_git() is in another file so we'll mess with that later

clean_pack_garbage() is deleting some files that it considers unnecessary,based on an in-memory data structure. ah. a snag.

we think our strategy for this step is going to be to rerun git gc (without aggressive) after we've done everything. okay. now to look at reprepare_packed_git().

hey kids, if you know the words, sing along! github.com/git/git/blob/6…

this is doing something deep and intriguing based on in-memory data structures, without which these other steps may not accomplish anything.

we're gonna run the other steps anyway though, they seem at least relevant.

gonna let the single-core invocation keep going though, it's at 97% now. at the very least this was more productive than simply watching it run.

this code is very C, which is fair because it was written by oldschool C programmers during the era when C still ruled the world. if this were in a language that encourages minimizing global state, it would be a lot easier to figure out where these data structures come from.

also, we're an oldschool C programmer, so that's fine and all, we're just noting it.

Compressing objects: 97% (2614301/2684171)

it's nice to have its count of how many objects it thinks a fully gc'd repo would have. this is a couple hundred thousand fewer than the repo in its current state, as described a couple lines prior:

Counting objects: 100% (2895079/2895079), done.

therefore, we believe, this is a nonzero savings, and furthermore we can check whether whatever we end up doing gets the same savings that git gc --aggressive would have, by comparing the object count

Compressing objects: 98% (2644505/2684171)

a watched progress bar never boils, but we can't help it

a friend called this a candidate for geekiest thread of the year *before* we started reading git's source code, mind you

please note that at no point in the thread have we spent any time on justifying this exploration we're doing. the primary motivation is simply to know that we've done it.

okay the single-CPU invocation is almost at the point where we expect it to crash

the load average is still well under 3, which was definitely not the case with the multicore invocations, so we do think the taskset stuff worked the way it's supposed to

load average is a really weird number. we don't feel up to explaining how it's calculated right now, but anyway it doesn't mean very much in the scheme of things.

on uniprocessor architectures it meant a lot more but it was still kinda weird. we remember when multicore machines became common, and discussion around that.

these last couple hundred objects are taking a long time to compress. maybe it was that slow all along and we just weren't watching as closely.

holy shit

Writing objects: 14% (405312/2895079)

plenty of swap available, plenty of filesystem space, load average below 2.0

we really expected this invocation to simply fail, but it's now past where it died before

we're a bit disappointed that if this works we won't have a good excuse to try our hacky thing, lol

interestingly, the number of objects it's writing is more than the number it compressed. there must be some form of metadata that isn't part of the pack?

in fact the number of objects it's writing is the same as what it started with lol

we'll back up a bit in our pasting so you can see the full context of this invocation

$ taskset -c 0 git gc --aggressive
Enumerating objects: 2895079, done.
Counting objects: 100% (2895079/2895079), done.
Delta compression using up to 4 threads
Compressing objects: 99% (2683568/2684171)
Compressing objects: 100% (2684171/2684171), done.

Writing objects: 90% (2606507/2895079)
Writing objects: 100% (2895079/2895079), done.
Total 2895079 (delta 2121149), reused 719316 (delta 0), pack-reused 0
Checking connectivity: 2895097, done.
Expanding reachable commits in commit graph: 378544, done.

our prompt has timestamps in it, for exactly this reason, and they tell us that this invocation took 90 minutes to run.

$ du -sh .git
629M .git

WHAT

IT SAVED A GIG AND A HALF

we did NOT expect it to accomplish anything

there are several packs in .git/objects/pack, presumably due to our pack size limit, let us just clean up...

$ git config --unset pack.windowMemory
warning: pack.windowmemory has multiple values

$ cat .git/config
[... elided stuff ...]
[pack]
windowMemory = 2g
windowMemory = 100m
packSizeLimit = 256m
windowMemory = 4g

$ vi .git/config

good thing we have vi set up to show hard tab characters, it is easy to break this file

now to consolidate those packs

$ git gc

that completed quickly. now it's just one .pack and corresponding .idx in .git/objects/pack/.

the reason we needed to do that was that one of the options we'd set earlier in our attempt to save memory, caused it to create several smaller packs rather than one huge one.

we knew that would happen because we read the description of the option

the final gc pass to consolidate the packs even saved a couple additional megs:

$ du -sh .git
620M .git

so, amazingly and counter to our expectations, this investigation was well worth it

thanks for reading!

epilogue: we tarred it up and re-deployed it to our smolputers. one of the smolputers actually has two copies of the repo, a bare one and a working tree, but we used git clone --shared so it's only stored once.

everything's running smoothly :)