#devopsmsp @Mattstratton: Matt’s speaking at the DevOps Minnepolis meetup about incident response, etc.
#devopsmsp @Mattstratton: ”There’s a lot of paramilitary language in IT, but tech people would shit their Dockers if they were ever in combat.”
#devopsmsp @Mattstratton: Response.pagerduty.com has an incident response practices page that is tool-agnostic.
#devopsmsp @Mattstratton: PagerDuty just open sourced their security training!
#devopsmsp @Mattstratton: Talk has three parts: Before, during, and after.
#devopsmsp @Mattstratton: Before an incident: Have criteria for what generates an incident and how important response is.
#devopsmsp @Mattstratton: If you don’t know whether something is an incident, how can you respond to it? PagerDuty’s: Any unplanned disruption or degradation of service that is actively affecting customer’s ability to use the product.
#devopsmsp @Mattstratton: DO NOT litigate whether something is an incident DURING the incident.
#devopsmsp @Mattstratton: Monitor the BUSINESS criteria and act accordingly. It’s not about the metrics, it’s about user experience. (See @mipsytipsy: 9s don’t matter if your customers are unhappy.)
#devopsmsp @Mattstratton: What is the problem with responding to alerts? The fallible human meatbag between alerts and fixes.
#devopsmsp @Mattstratton: People are expensive — don’t have giant bridge calls to solve problems.
#devopsmsp @Mattstratton: Usually the list of things that cannot be automated is a lot shorter than you think it is.
#devopsmsp @Mattstratton: Practice makes perfect. Practice with incidents where you know the cause.
#devopsmsp @Mattstratton: Know your role and what the role is supposed to execute: Incident commander, deputy/scribe, SMEs.
#devopsmsp @Mattstratton: Have a clear understanding of who is supposed to be involved in each role. This gives people clear time off.
#devopsmsp @Mattstratton: During the incident: Incident Commander identifies themself clearly.
#devopsmsp @Mattstratton: The incident commander is the highest authority on the call. Yes, even higher than the CEO if they are on the call.
#devopsmsp @Mattstratton: The Incident Commander manages the flow of conversation and the timing of updates and communication.
#devopsmsp @Mattstratton: Bystander effect can really screw up an incident. Assign specific tasks to actual people.
#devopsmsp @Mattstratton: During a call, rally fast and disband faster. Tell people they can leave when not needed - no waiting around! You can always ask them back.
#devopsmsp @Mattstratton: Have a way to contribute information during the incident.
#devopsmsp @Mattstratton: Capture everything and then prioritize what should happen now vs. later.
#devopsmsp @Mattstratton: Make it very easy for people to object but make sure that there is one incident commander.
#devopsmsp @Mattstratton: Assign an owner at the end of an incident.
#devopsmsp @Mattstratton: After an incident: Run a blameless postmortem.
#devopsmsp @Mattstratton: “We might call them learning reviews if we’re feeling especially hugops-y.”
#devopsmsp @Mattstratton: The impact to people is part of your incident review as well. Make your org more resilient by identifying regular stess points.
#devopsmsp @Mattstratton: Record incident calls and review them afterwards, because we tend to forget things that happen in the middle.
#devopsmsp @Mattstratton: Regularly review the incident process itself.
#devopsmsp @Mattstratton: Listening to the Apollo 13 Flight Director audio loop as a really clear incident process. Also it’s making me stressed. ;)
#devopsmsp @Mattstratton: Don’t panic. Stay calm. Calm people stay alive. (or in business)
#devopsmsp @Mattstratton: “John Allspaw - if you don’t know who he is, he invented a lot of this devops crap....”
