Thread by @zettlr

The longer I think about the differences between Zettlr/The Archive and Roam/Obsidian/Notion, the more I believe it fundamentally concerns the question of databases vs. files, and, in turn, the question "Are files and databases actually something different?"

(A thread; 1/10)

I mean, yes, technically databases and files are something different, but fundamentally, similar principles apply. One may treat files as database records, and their ID are their unique paths. Similarly, you can also save lots of arbitrary data in a single database record. (2/10)

However, the way these two principles/concepts emerged back in the day are so different, that databases are up to 35 % faster for records smaller than 10kb (SQLite n.D.), and distributed file systems (HDFS/GDFS) run into what is known as the "small file problem" (3/10)

So in a way the current surge in database-based SaaS note taking applications pose some kind of an existential question: Where is the thin line between a file and a database record? Naturally, we assume that files are few and large, whereas records are many and small. (4/10)

But knowledge management like Zettelkästen fundamentally defy this simple demarcation: They use files, but so small they could also fit in a database. And this is why The Archive and Zkn3 are so similar, albeit both use fundamentally different storage mechanisms. (5/10)

Further, they raise the question if we shouldn't overthink the way we treat files and records, and begin applying database-ideas to files. Funfact here: The caching mechanism of Zettlr 1.7 uses sharding — a database technique – and applies it to files. And it works great. (6/10)

This also explains why I have a hard time with auto-changing files — automatic backlinks for instance. I hate the thought of an app modifying files without my knowledge, but concerning databases I accept this every time. So I'm increasingly asking myself: Is this right? (7/10)

But, looking at it from a different perspective: Files constitute a database that is transparent: We know how it works, and how to find information. A database, on the other hand, can be arbitrarily structured, and therefore becomes opaque — the root of "vendor lock-in" (8/10)

Evernote users migrating their stuff can tell a tale about this. Fundamentally, I think it all boils down to control: A database is something the application provider controls, the file system something the user controls. So in the end all apps have to make a tradeoff:

(9/10)

Either they use the file system, and get frustrated everytime a user does something to its files which the app didn't anticipate. Or they retain control and thereby create a power relation between their services and the user.

It's quick innovation vs. data sovereignty.

(10/10)

Source for the SQLite n.d.: sqlite.org/fasterthanfs.h…