Thread by @tasdikrahman

Started reading @Google 's #SRE book and some insights from it so far, will keep updating the thread as I finish the chapters. (1/n)

@Google The initial chapter touches upon the idea, that fixes being pushed with human-interruption need to scale linearly as the product grows/scale increases. Practicing the ideology of building systems which would in turn manage the hand holding which #syadmins do is radical (2/n)

@Google 100% uptime is probably never the right reliability target: not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice. Matching the profile of the service to the risk the business is willing to take. (3/n)

@Google Error budgets make it easier to decide the rate of releases and to effectively defuse discussions about #outages with stakeholders, and allows multiple teams to reach a consensus faster by making decisions by looking at data points from the #monitoring system. (4/n)

@Google This is in alignment with the #SRE team pushing for reliability & product teams for feature releases, both being important. The system being unreliable will affect customer experience, if there are no product features being released, may lead customers to other products. (5/n)

@Google Keeping a check on toil is something which has been repeatedly emphasized in this chapter, and it does make sense when you have observed this first hand. As the org grows, if toil is not checked, it will take up the whole team's bandwidth eventually landing.google.com/sre/sre-book/c… (6/n)

@Google It has been wonderfully phrased here, that "If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow" landing.google.com/sre/sre-book/c… (7/n)

@Google But what is toil? Chances are that work which is manual, repetitive, automatable, tactical, has no enduring value (system remains at the same state after you do it), O(n) with service growth has a high chance to be categorized as toil. (8/n)

@Google Too many pages? And the #oncall engineer will start second-guessing and even miss a real page that's masked by noise. More so, paging an employee is expensive as it breaks their workflow. Having minimal noise and good signal is the sign of a mature #monitoring system. (9/n)

@Google Rules that generate alerts for humans should be simple to understand and represent a clear failure. The what's broken indicates the symptom, the why indicates the possible cause. #SREBook #SRE (10/n)

@Google Doing automation thoughtlessly will only create as many problems as it will solve. Even though software based automation is better than manual, the sweet spot would be an autonomous system which self heals. #SREBook #SRE (11/n)

@Google What's the value of having automation? It adds consistency, a platform, faster repairs and faster action helping save time. #SREBook #SRE (12/n)

@Google Actions like changing the servers resolv.conf are repetitive & as the number of machines grow, chances of something going wrong in a manual task also go up. This inevitable lack of consistency leads to mistakes, issues with data quality, reliability problems. #SREBook #SRE (13/n)

@Google If designed and done properly, automatic systems also provide a platform on top of which you can build things/extend features on top of it, which can be shared among numerous teams. Bug once solved in this platform will also be fixed for the other teams. #SREBook #SRE (14/n)

@Google A very good example of a platform for automation is @kubernetesio, and in our own case of @gojektech , github.com/gojek/proctor on top of which we have numerous automation scripts for teams to use helping them achieve numerous things via automation. #SREBook #SRE (15/n)

@Google @kubernetesio @gojektech Faster recovery? Ideally, you would get paged & someone would ack it & do an analysis of what went wrong and then an action would be taken. In the case of an autonomous system which is self healing, the MTTD & MTTR would be lower than a human in most cases. #SREBook #SRE (16/n)

@Google @kubernetesio @gojektech On of the big reason for having automation(if done properly) is that it saves the human toil required to hand hold their systems, which would eat all your time as the scale increases along with the complexity. #SREBook #SRE (17/n)

@Google @kubernetesio @gojektech Automation of something is a gradual path, No automation->hacky script in ~/home -> script is shared among other devs -> improvements/gets added to your automation platform -> systems is self sufficient/autonomous/self-healing #SREBook #SRE (18/n)

@Google @kubernetesio @gojektech Automation for something specific should be owned by the owners of the service which use the script, reason? They have the highest incentive to maintain/fixing bugs/adding features to the automation. Otherwise if left with someone else, it will stagnate. #SREBook #SRE (19/n)

@Google @kubernetesio @gojektech If possible, the automation should be idempotent. If the automation fails in the middle for some reason, the automation should be able to pick up from where it left and redo what it has already done. Off the shelf tools like @ansible do try following that. #SREBook #SRE (20/n)

@Google @kubernetesio @gojektech @ansible Running reliable services require reliable release processes. The binaries and configurations require to be built in a reproducible, automated which would make releases repeatable and not unique snowflakes #SREBook #SRE (21/n)

@Google @kubernetesio @gojektech @ansible In order to scale your team, teams should be self sufficient which can be achieved with processes and tooling which enforce those policies. #SREBook #SRE (22/n)

@Google @kubernetesio @gojektech @ansible Ideally each team should be able to decide how they do their deployments depending on their service, user facing apps in most cases would require a gradual rollout while someone else might be ok with a full deployment. The deployment tool should be flexible. #SREBook #SRE (23/n)

@Google @kubernetesio @gojektech @ansible When equipped with the right tools, proper automation, and well-defined policies, developers and SREs shouldn’t have to worry about releasing software. Releases can be as painless as simply pressing a button. #SREBook #SRE (24/n)

@Google @kubernetesio @gojektech @ansible Software should be boring! It should not have unexpected surprises, but be predictable. A simpler/minimal API is also a hallmark of a well understood problem. Less is more in software. Dead code should be removed religiously, as it's a ticking time bomb. #SREBook #SRE (25/n)

@Google @kubernetesio @gojektech @ansible Lower the coupling between pieces of software, the more you can release fearlessly & with increased confidence. Measuring a single change you pushed is always easier than measuring what was the effect of pushing 100 changes. #SREBook #SRE (26/n)

@Google @kubernetesio @gojektech @ansible Having smaller changes would also make it easier to debug and pinpoint faster on what caused a regression. Hence software simplicity is a pre-requisite to reliability. #SREBook #SRE (27/n)