We give our new devs and SREs the ability to screw up (incl. taking down an entire service) their first week. Most manage to accidentally do it. And that's great.
They understand blame isn't a thing, more about that system in a hurry, and that decisions aren't driven by fear.
They understand blame isn't a thing, more about that system in a hurry, and that decisions aren't driven by fear.
Most places have systems no one spends a lot of time on. A new person diving into those with a passion and wanting to take ownership it fantastic. If it starts with an oops, that's just fine. It's the cost of doing business and everyone's better for it. People are what matters.
We had an internal outage very few users saw this week.
What happened:
- Wrong server rebooted
- People learned:
- PTRs are in a bad state, don't trust them
- How we re-replicate Redis
- How to write a postmortem
- Lots of misc. around it
What didn't happen:
- Blame
What happened:
- Wrong server rebooted
- People learned:
- PTRs are in a bad state, don't trust them
- How we re-replicate Redis
- How to write a postmortem
- Lots of misc. around it
What didn't happen:
- Blame
I had zero problems with this happening. Zero. If people are afraid to touch something, it'll self-perpetuate into something fewer and fewer people are willing to look at because "it's been like that forever". That's not healthy. Gotta allow screw-ups and treat them as learning.
I haven't been a new employee in about a decade now, but I hope this is good. Systems should be in the state they are because that's how you want them to be. Not because people are afraid to touch them.
My boss has a great way of putting this:
"I don't mind us screwing up, as long as we try and not do it the same way over and over again - we should be finding new ways".
I've been thoroughly impressed with @teresadg all around - this has to be a mentality from the top to work.
"I don't mind us screwing up, as long as we try and not do it the same way over and over again - we should be finding new ways".
I've been thoroughly impressed with @teresadg all around - this has to be a mentality from the top to work.
In short, being afraid to make mistakes may benefit you in the short term with a stable, unchanged system...but it'll quickly leave you in a world of decay everywhere that everyone's afraid to touch. And we're in a marathon here.
