Thread by @heinrichhartman

Live right now!

https://twitter.com/PrometheusIO/status/1192805576125898753

What works well in Prometheus Histograms:

- Percentage of requests in the last hour, with response time 100ms or less

- How many HTTP responses were larger 4kb between 2:30 .. 2:45.

You _can_ correctly aggregate Histograms, but you can not do that with percentiles, or other aggregates.

All this only works only in Prometheus if:

You have suitable / compatible buckets defined !

What does *not* work well:

Calculating percentiles with "histogram_quantile" function

b/c
- accuracy depends on bucket layout
- need lot's of buckets for high accuracy
- but buckets are expensive

Recommendation:

"Put buckets where you expect interesting values." e.g. SLA latency thresholds.

Also need compatible bucket layout...

So if the interesting values change, you need to change bucket layout... so you can't aggregate your histograms anymore.

All boils down to: Buckets are too expensive.

Can hit bad cases when histograms are partially scraped, e.g. when aggregating histograms from separate targets.

Current recommendation:
- partition request HTTP counts per status code
- collect only a single latency histogram for all status codes (because buckets are too expensive.)

Go to event processing system, if you need accurate results.

This get's increasingly annoying...

Can't answer questions like:

> Are we responding fast enough to 404 right now?

> Do the 501 and 503 requests take the same amount of time?

> How fast are my 200s while I have a 404 storm.

The Future of Prometheus Histograms, will evolve around two problems:

(1) Buckets are too expensive

(2) Ingestion Isolation

Ad (2): Fix isolation. See Brian's presentation 2years ago. Has just to get done.

Ad (1) "Histogram Buckets are too expensive."

Option 1: Do nothing.

... is becoming increasingly unpopular.

Prom "proverbs"

> Instrumentation first, ask questions later.

For histograms this does not hold b/c need to know interesting values, don't have partitions, etc.

Option 2: Make buckets a bit cheaper.

E.g. by improving the exposition format.Even with compression a single histogram with ~20 buckets takes ~300 bytes.
What can help:
- protobuf?
- more compact exposition format??

Option 2b: Change TSDB.

Problem: TSDB has 0-notion of histograms. Only groups of numeric metrics.

Worst bit: Bucket boundaries are in the metric name (string).

Lot's of room for improvement.

Option 3: Make buckets a lot cheaper. (yay!)

How about infinitely many buckets? each 5% larger than the next?

This has already been done in practice:

- HdrHistogram
- @circonus's Circllhist
- Datadog's DDSketch ("hot" but very simple)

Allows to handle sparseness!

@circonus Prometheus accumulates data going forward, creates problems with sparseness.

@circonus Option 4: Some kind of digest or sketch.

If buckets are expensive ...

... maybe we should not have buckets.

@circonus Huge body of research literature. Best candidate, Moment-Based Quantiles. Sadly, this does not seem to viable for Prometheus.

Problem: High percentiles are not accurate enough.

... this is what motivated DataDog to write their sketch, which really is a histogram.

@circonus Hitlist:

1. Master Sparseness Problem (with Histograms)
3. Make buckets a little cheaper
2. Digest/Sketches
4. Do nothing