, 30 tweets, 13 min read Read on Twitter
Aaaand @mipsytipsy is here to lead off #ServerlessDaysBOS! She'll be talking about observability and teams owning their software all the way into production!
@mipsytipsy Remember to take backups! Her slides went missing overnight, sadface, but she did a partial recovery.

How many people don't consider themsleves software engineers? Only 3 out of the whole room.

So we need to care about systems, not just writing software. #ServerlessDaysBOS
We have to build them and tend to them. And make sure that you believe in the purpose of what you're doing!! ("Just a plug for quitting your job.")

Software is your enemy rather than your job; delete as much code as you can. It's a liability. #ServerlessDaysBOS
Write no code if you can. Reuse other peoples' code if you can. Only then should you consider writing your own.

We've gotten better at managing projects and people, but our systems are unmaintainable and undebuggable black boxes. #ServerlessDaysBOS
It's not good for us to be up late at night struggling to debug things all the time. $85B in GDP costs of bad software every year. citing stripe.com/reports/develo… #ServerlessDaysBOS
We need to care about the connective tissue, the users, the humans who take care of things... not just the software itself.

"It's people all the way down, unfortunately. I'm sorry." --@mipsytipsy #ServerlessDaysBOS
Engineering leadership is explaining our systems to the rest of the world. And people are *pissed* because we don't even understand our systems, let alone be able to explain their privacy etc. privacy implications.

It's way easier to build than understand. #ServerlessDaysBOS
"If you can solve your problems with a LAMP stack, please do." but most of us have complex sets of services.

Never run and support user code if you can help it. You'll have weird strange customer-visible outages while your dashboards are all green. #ServerlessDaysBOS
"It would be folly to spin up a staging copy of the national electrical grid." -- our distributed systems are hard to test.

It's hard to predict micro-failures in single points or systematic problems with all parts from a particular supplier... #ServerlessDaysBOS
"We used to have an illusion that we were in control and nothing was broken. But our systems are full of emergent behaviors and are broken all the time." --@mipsytipsy #ServerlessDaysBOS
Quoting me on Production Excellence: you need software ownership to achieve it. [ed: and vice versa].

Serverless is servers all the way down. But the best way to instrument any system, serverless or not is to pretend you don't control the underlying hardware. #ServerlessDaysBOS
Don't shit on your ops people. Operability requires ops, and dev, and business.

If you don't want to participate in production excellence, don't work on systems requiring 24x7 reliability.

You can't make it someone else's problem, that breaks feedback loop. #ServerlessDaysBOS
We have to have the feedback loop of paying the consequences for our own debt.

Oncall doesn't have to suck. Ops has a terrible culture of martyrdom and abuse. But it can be so much better. Don't let systems get to that state. It takes time to heal. #ServerlessDaysBOS
How do you achieve production excellence? One way is observability driven development. TDD stops at the edge of your laptop; it's hard to test network conditions, RPCs, etc. -- and you still miss things in staging without real user traffic. #ServerlessDaysBOS
Our graphing and monitoring tools don't speak our language. We don't care about uptime and free memory, we care about what's happening inside.

The difference between a frustrated and empowered team is whether their tools help them or hinder them. #ServerlessDaysBOS
Using the definition from control theory: can you understand the system by observing its outputs? You shouldn't need to ssh in and run strace, or re-deploy code to ask new questions. #ServerlessDaysBOS
Aggregating on the "wrong thing" shouldn't result in being unable to debug, having to re-instrument and re-deploy, and then hope it happens again.

Software engineers shouldn't have to care about the health of underlying systems. Abstraction matters. #ServerlessDaysBOS
It doesn't matter if servers are down (or even entire AZs), as long as user requests can execute end to end still. #ServerlessDaysBOS
When we used to be constrained on hardware resources, metrics had to throw away context in order to reduce things to just counters and tags with low cardinality. You can't keep the UUIDs.

[ed: furries may not id as human so species is high cardinality too!] #ServerlessDaysBOS
Logs aren't much better, because they're unstructured.

So instead we can get the best of both worlds by having events generated by requests as they travel through your system -- one event generated per request * service/function. #ServerlessDaysBOS
As much as we'd like to spend 5x as much on debuggability and observability than on our production environment, the people holding the purse strings don't like that.

But you don't have to overspend; you can dynamically downsample just the flood of HTTP 200s. #ServerlessDaysBOS
You want to get every 500 but you don't care about the majority of events in your system.

Don't just create new dashboard spam. You won't be able to find the dashboard you made for "the next time". Also it tricks you into being lazy & false hypotheses. #ServerlessDaysBOS
Dashboards make you jump straight to the answer without formulating the question. They're artifacts of past failures and you'd be better off just leaving a gravestone and moving on. #ServerlessDaysBOS
When you aggregate at write time and constrain to a fixed set of aggregate labels, you become unable to work your way back to the raw events. Don't smoosh away detail into one value. Don't average your averages. #ServerlessDaysBOS
Take your feedback loops a step further; work in production. Use feature flags, progressive deployment/delivery. Data can help you decide what's worth building.

Don't waste time and encourage sloppiness with staging environments. #ServerlessDaysBOS
Get used to working with production when it's not broken, rather than only when it's broken. There are so many more things you can do than fight fires.

There's an internalized fear of production that software engineers have. (sadly, often from being yelled at) #ServerlessDaysBOS
Delete all your pager rules *except* those that are for your service level indicators. And use observability, like @honeycombio for the rest. [fin]

[ed: @mipsytipsy and @djpiebob and I need to include in our article on sampling how/why aggregation != sampling] #ServerlessDaysBOS
Audience question about fragmented systems and instrumentation: @mipsytipsy answers that distributed tracing plus instrumenting your own application plus being able to look at histograms is critical. #ServerlessDaysBOS
Audience question about PCI/DSS, which I couldn't help jump in and answer: segregation of duties does *not* mean product developers can't access production telemetry; it only means they shouldn't be able to mutate prod/deploy their own code without approval. #ServerlessDaysBOS
and remember, if you have questions for @mipsytipsy or me, drop by the @honeycombio booth to leave a question in the bowl, or fill out goo.gl/forms/rhcvEQUQ…! #ServerlessDaysBOS
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Liz Fong-Jones (方禮真)
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!