Thread by @wmitsuda

I was able to fit an entire functional mainnet archive into < 900 GB using Erigon + ZFS + compression (with some tradeoffs)

Yes, ➡️➡️➡️AN ENTIRE MAINNET ARCHIVE NODE⬅️⬅️⬅️

A short 🧵👇

Context: I've been running Erigon for a long time in my laptop using a portable 2TB Samsung T5 SSD.

Everything works fine, except last week I noticed current mainnet archive is taking +1.8TB, so it is time to migrate, you've done a good job Samsung ser, respect.

I already got a new 4TB drive, but wanted to experiment with something new that could give some extra life to the 2TB one.

I've seen some people talking about ZFS on Erigon discord, but it looks like they target Linux installations on cloud or home NUCs.

I use a mac, there is a package called OpenZFS on OSX and apparently no one has tried it yet, so why not? :)

openzfsonosx.org

The docs are not very friendly, it seems like a standard linux software website from the 00's, but there is a nice homebrew package and it works.

It seems macos Monterrey is not yet supported, but it apparently works #YOLO

https://twitter.com/openzfsonosx/status/1402801599987126274

After installing it, you'll probably want to dig into the standard openzfs/linux tutorials, nothing macos-specific, you create a zpool, enable compression, blablabla.

The magic is done by enabling filesystem level compression. According to some old messages on Erigon discord, lz4 works fine.

I moved my previous chaindata to the new FS after enabling compression, it takes 887GB with a 2.1x compression ratio.

The uncompressed size is 1.82TB.

Of course this doesn't come without tradeoffs. Sync is much more slower. Some olds numbers I found in Erigon discord says it can easily take double the time to sync from scratch.

I took an already synced node and moved it to the compressed FS.

Even so, it frequently goes a few blocks behind the tip, it is able to catch it eventually though (see timestamp 09:15:43).

Also CPU usage is constantly higher, I can hear the fans working, be sure to turn on the air conditioner in your room to max.

Next, I think this is a mac-specific issue: if you forget to explicitly unmount the FS using CLI, it messes everything and you can't mount it again unless you reboot your computer (I couldn't make it work other way).

I'm pretty satisfied overall, despite the unmount clunkyness, ZFS seems resilient even on accidental unplugs.

I think I'll keep it as my backup testing node for now and use the remaining space to fit all my testnet archives into it (WAGMI?) so I can free my secondary drive.

if you are a ZFS expert, feel free to suggest better configs, the reviews I found on Erigon discord are quite old, not sure if the numbers still apply.

Fin? :)

Addendum: (1) it seems it doesn't like if your mac sleeps, FS just hangs after that, need to reboot

(2) I don't recommend running Otterscan over a node configured this way, search relies on tracing multiple blocks simultaneously, performance really sucks on this scenario

Update: moved all my testnet archives into it and there is still plenty of space :)