Thread by @mipsytipsy

let's talk about two kinds of data that systems tend to generate: auditable and operational.

* auditable data: transaction logs, replication logs, billing/finance events etc
* operational data: telemetry, metrics and context that describe each request and system component

in the early days we tend to jam as much as possible into one data pipeline, toss it around using kafka/logstash, and sort it out someday.

but they have extraordinarily different characteristics and use cases, which become super fucking obvious and 💵expensive💵 as you scale.

(oops back)

with 'auditable logs' (better term?) losing data is very bad, and you should try hard to retain every record. because of this, you must be very disciplined in what you accept: strict schema, very compact rows. friction around changing schema is ok-- even desired.

you should always be able to assume, when querying e.g. your payment history or mysql replica, that the dataset is effectively complete. and the shape of it stays highly predictable so you can forecast costs.

cool.

operational datasets are the polar opposite of all this.

auditable: predictable, schema'd, compact, often contains PII/PHI/sensitive data

operational: messy, flexible, toss in whatever seems like it may be useful someday, spiky, unpredictable, oddly shaped, lots of numbers to do math on, ballooning write amplification #'s per request

you never want to incentivize your developers NOT to capture some little detail that may be useful for your team someday when trying to track down an unknown-unknown for the first time. this is *not* where you want people thinking conservatively about what to persist.

however, every request that comes in through your front door may generate tens or hundreds of events for your observability stack, esp if you're a microservices shop.

..and i'm not talking cheap little counters, i'm talking the gold standard: rich, wide, structured log events.

startups usually start with lots of these datasets mixed up together, in whatever kafka or logstash monstrosity pipeline they have cobbled together, but this is what always forces them apart: cost.

the infinite perfect retention cost model for operational data will break you

you cannot keep, and should not try to keep, all the operational spew that issues forth from your systems. dynamic sampling is your friend here, as are server-side limits to lop the top off the mountain when e.g. the site goes down and you get a billion operational msgs at once

common things are common, so keep a small % of them. rare things are rare, so keep all of them. everything exists on a spectrum.

sampling (& separating out any auditable data) is how you get to have your cake and eat it too (and not go broke because you bought the damn bakery)

honestly i recommend people start out by sampling from day one, just to get their developers out of the habit of expecting operational data to be lossless and just as accurate as their billing data. it's a terrible fucking habit to get into.

unless it's an auditable data source, you should never assume that.

you're never looking for "one event" in an operational dataset. you're looking for "some manifestation of a bug" or "some use case or pattern that looks like this" or "behavior that matches this report"

and i'm telling you, if the bug was tripped once out of a trillion requests, maybe that's not the bug you need to be focusing on.

if it is? ok just tune your sampling to catch 100% of that (user, whatever), and wait for it to happen again. it will.