Thread by @potetm

A short short story about how I'm an idiot and @datomic_team is an incredible piece of tech:

A week ago I accidentally reset the value of a column for all rows in the table.* Big oopsie. imaidiot.

* I'm translating here for all the folks more familiar with SQL. Datomic doesn't have tables or columns, but the idea is the same for this story.

This was part of a maintenance routine, so it didn't go through normal code paths. Were this SQL, I wouldn't even have `updated_at` to work with.

The values were just gone.

Not only that, but nobody noticed for TWO weeks.

That's two weeks of running on incorrect assumptions. Two weeks of incorrectly syncing data between this database and our primary.

To top it off, when I finally figured it out, it was at 5:00pm the day before my vacation day with the boys.

And it was the last day of the month.

The accountants would want to close the books on this accounting period tomorrow.

But [cue the hero theme]

Datomic is the database that doesn't forget.

I started by finding all of the rows whose values had been updated in that maintenance routine.

Luckily, "Find all updates to this column around this time" is a straightforward Datomic query.

In fact, I was able to find the exact transaction that was run AND the exact values that changed.

Datomic gives you access to the raw transaction log.

Hell yeah.

At this point it would have been easy to re-set those values to be what they were before.

But this is software. Software is messy.

One question is: Has anybody helpfully tried to fix these values in the interim? If so, I needed to investigate those.

It's possible someone set those values to something different than they were before. I didn't want to undo that work.

Luckily, "Find all updates to this column for these specific records since this specific transaction" is surprisingly easy in Datomic.

No one had tried to be helpful. That was good, actually. Now I could just reset the values to what they were before.

Easy.

End of story.

Except... this is software. And software is messy.

I was in the middle of notifying the people who brought this to my attention that it had been fixed, when I realized something: A whole bunch of incorrect transactions had been replayed into our primary database because of this error.

I had to fix the primary as well.

By now you can probably guess how this plays out:

"Find all sync events since the incorrect transaction that involved these specific rows" is a straightforward Datomic query.

From there I was able to reprocess these events properly into the primary.

I was able to have supper with my family, put the kids to bed, and still wrap up this entire tale by around 10:00pm.

Most importantly, I didn't think have to worry about this at all while kayaking with the boys the next day.

I want to point out that, had this been a SQL db, I would have been totally hosed. I would have been reduced to guess work and walking around hat in hand asking people what they happened to remember over the past 2 weeks.

The only thing that a SQL database could ape was finding sync events. If you were smart enough to design sync events as an immutable row with a `event_timestamp` column, you would have been able to find those.

HOWEVER

That is NOT the same as, "find all events since this specific db transaction."

Searching by `event_timestamp` is probably good enough for low-volume systems like mine, but it's not precise.

When it can even compete, SQL is 3 pigs in a trench coat by comparison.

I've not even discussed the development/design benefits of Datomic—which are very bigly. Massive.

This story was all just the maintenance power.

In short, software is messy.

Well-designed tech can make it manageable.

You should use Datomic.