Thread by @tleam

First talk up is @lizthegrey talking about Production Excellence.

#VelocityConf

If you haven’t read @lizthegrey’s opus on Sustainable Operations, go take a break after the keynotes and read it. Should be mandatory if you own a service or product.

infoq.com/articles/produ…

#VelocityConf

What runs our software? People. We need to focus on our people, culture and process.

ProdEx is about making our holistic systems more reliable and resilient. It takes *everyone* to make this happen.

#VelocityConf

Four things are critical:

1 Know when system is too broken
2 How do we resolve the breaks
3 How do we collaborate to make that happen
4 How we learn

#VelocityConf

How to measure too broken:

Service level indicators and objectives!

Key: these aren’t just numbers, they need to reflect customer experience. Work with client facing groups to understand user sentiment and what is healthy enough

#VelocityConf

@lizthegrey is talking about focusing alerts on those generated by your SLO. At the end of the day: if your customers are happy, why are you getting woken up in the middle of the night?

This parallels nicely with CASE alerting model:

onemogin.com/monitoring/cas…

#VelocityConf

To make this happen we need observability which provides context to events so that you can understand variance of healthy vs unhealthy.

SLOs - let you know too broken
Observability - let you know how it’s broken

#VelocityConf

Now, how do we learn from our production surprises? Sharing our learnings!

To do this we need:
1 Documentation
2 Incentivized sharing
3 Understanding of our risk portfolio (frequency & impact)

#VelocityConf