Profile picture
, 14 tweets, 6 min read
Kubernetes Borg/Omega history topic 11: PodDisruptionBudget. Google constantly performs software and hardware maintenance in its datacenters: firmware updates, kernel and image updates, disk repairs, switch updates, battery tests, etc. etc. More and more kinds over time.
Even though Borg tasks are designed to be resilient, this could get pretty disruptive. Rate-limiting maintenance tasks independently isn't efficient if you have dozens of them, and it's not always feasible to perform all types of maintenance at the same time.
Even if rate-limiting machine disruptions, it's also possible for the same task to get disrupted over and over again, like a can being kicked down the road. Hence, the Safe Removal Service (aka SRSly -- pronounced seriously) was developed by SRE. (SRE builds lots of automation.)
SRSly kept track of how often tasks of the same Borg Job were disrupted (aka evicted). Maintenance automation queried SRSly regarding all tasks scheduled on a machine before taking it out of service. This enabled Borg to provide a SLO on task disruption.
Borgmaster, however, did not know about SRSly. Instead, all critical/production workloads were changed to run with the same priority so that they wouldn't preempt each other. Doing that for every Borg Job in the company was extremely painful -- more on priority/preemption later
For Omega, we developed a model that could be applied both during task preemption to run a higher-priority task and eviction for maintenance -- disruption counters. There was a time dimension that ended up not being effective due to constant changes, so we dropped it in K8s
I think I first mentioned this in Kubernetes in my big scheduling braindump comment: github.com/kubernetes/kub…. It came up again when I proposed maxUnavailable to moderate concurrent disruptions caused by updates during the design of Deployment:
github.com/kubernetes/kub…
That discussion was forked into github.com/kubernetes/kub…. Around that time, Matt Liggett (github.com/kubernetes/kub…) joined the GKE team from Borg SRE (woo hoo!). One of the first things Matt worked on was improving node drains: github.com/kubernetes/kub…
Together with @davidopp and @erictune4, we folded disruption budgets into the rescheduling design proposal: github.com/kubernetes/com…. (Rescheduling deserves its own thread -- I'll do that one next.) Implementation began in github.com/kubernetes/kub… and github.com/kubernetes/kub…
PodDisruptionBudget is now documented: kubernetes.io/docs/concepts/… and kubernetes.io/docs/tasks/run…. Try it out and give us feedback on how well it works for you. We're looking to advance it from beta to GA: github.com/kubernetes/enh…
You can safely drain a node with kubetctl drain: kubernetes.io/docs/tasks/adm…. Node upgrades and the cluster autoscaler in Google Kubernetes Engine (GKE) also respect PodDisruptionBudget. The latter is documented here: cloud.google.com/kubernetes-eng…
Node upgrade behavior is documented here: cloud.google.com/kubernetes-eng…
And more about the Google SRE philosophy behind automation can be found in the SRE book: landing.google.com/sre/sre-book/c…
And the Safe Removal Service was also mentioned in Google's "VM Live Migration at Scale" paper in VEE 2018: dl.acm.org/citation.cfm?i…
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Brian Grant

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

More from @bgrant0607 see all

Profile picture
Brian Grant
@bgrant0607
Kubernetes Borg/Omega history topic 14: Computational Quality of Service (QoS) and oversubscription. What are they, why would you want them, and how is QoS different than priority? On the last point, it's distinguishing importance and urgency.
QoS is something that wouldn't matter if just one process were running per host system or if all the processes steadily used a constant amount of cpu, memory, and other resources. Because they vary, reserving max capacity needed for each would leave systems poorly utilized
Oversubscription mitigates that by packing more applications onto a system than could all fit at their peak requirements. It's kind of like a bank: not everyone can withdraw at the same time. The question is then: what happens when apps demand more resources than they can get?
Read 16 tweets
Profile picture
Brian Grant
@bgrant0607
Kubernetes Borg/Omega history topic 13: Priority and preemption. Some work is more important and/or urgent than other work. Borg represented this as an integer value: priority. A higher value meant a task was more important than a lower value, and should be able to displace it.
When choosing a machine for a task, the scheduler ignored lower-priority tasks for determining whether/where a task would fit, but considered the number of tasks that would have to be preempted as part of the ranking function for choosing the best machine.
Disruption budgets were never added to the scheduler, which would have been hard, but there were also concerns about performance and priority inversion. Higher-priority tasks could specify how long they would wait for lower-priority ones to gracefully terminate
Read 14 tweets
Profile picture
Brian Grant
@bgrant0607
Kubernetes Borg/Omega history topic 12: A follow-on to the PodDisruptionBudget topic: the descheduler (github.com/kubernetes-inc…). Descheduler is more appropriate than the original term "rescheduler", because its job is to decide which pods to kill, not to replace or schedule them
In Kubernetes, when running on a cloud provider such as in GKE, in the case of pending pods with no existing available space to be placed, either cluster autoscaling or even node autoprovisioning (github.com/kubernetes/aut…, cloud.google.com/kubernetes-eng…) can create new nodes for them
In Borg, the rescheduler was created to defragment nodes to make room. It selected tasks to evict so that the new tasks could schedule, while also ensuring the replacements for the evicted tasks could also find new homes so as not to just cause unnecessary churn
Read 9 tweets

Related threads

Profile picture
The Epoch Times
@EpochTimes
What if I told you that #BigTech and #SocialMedia platforms are manipulating you, steering you toward #Health information that they think is right, rather than letting you evaluate content for yourself?

Commentary by @MaryamHenein (Thread 👇) theepochtimes.com/google-censors…
Accredited professionals, meanwhile, who stand for #Health freedom and who criticize Big Anything, are losing posting privileges, getting banned, being buried, finding themselves deranked, and getting digitally assassinated.
Content is literally disappearing from the Internet along with our #Health choices.

It sounds conspiratorial because it is. Independent #Journalists and health experts are blowing the whistle. But it’s become a silent one.
Read 94 tweets
Profile picture
🚨 REDPilledCrasH
@Michaelcraddo16
#GOOGLE [ DARPA] says 'hackers' 🙄 put monitoring implants in I- phones for years #SurveillanceState
[DS] #Musk [DARPA]claims China is the Future In AI #BigTech #Building8

breitbart.com/tech/2019/08/2…
Thread on [DARPA] 👇👀
#NothingToSeeHere #Building8 #AI #Fbook #Google #23andMe
threadreaderapp.com/thread/1140699…
Read 7 tweets
Profile picture
MissMAGA2016
@MissMaga2016
#Google claims they’re not working with #China’s military, but they are through a top Chinese university that took major funding from the CCP to advance the Chinese military’s AI goals.

It’s part of China’s overall world domination goal, the AI goal to be completed by 2030.
China has utilized their “Thousand Talents” program to recruit people who’ve gained knowledge in specific industries in the west to help advance the Communist party’s objectives.
In this case, they’ve collaborated with Google’s head of AI, Jeff Dean, who is also a member of CCP-funded Tsinghua University’s computer science advisory committee.
Read 5 tweets
Profile picture
Kate Sheppard
@k8shep
The end of another semester of engaging students with @WikiEducation at @MissouriSandT @sandtcasb in #HistSCI means another tweet thread about this super fun project.
49 students, 32 articles, 40K words added, +285K views so far. I love their work! Take a look!
I'm going to group them. First group-- #WomenInSTEM #WomeninHistory
Students found that, uh oh, it's HARD to find reliable sources on women in science! This didn't stop them, though. They pushed through and learned some new research skills.
en.wikipedia.org/wiki/Annie_S._… is 1 product.
Tycho's sister, en.wikipedia.org/wiki/Sophia_Br… was another article that needed work. The women who worked on this one had to find all sources on Tycho then sift through for the 2-3 pages about Sophia. Thank goodness for good #librarians at S&T who were patient and helpful!
Read 29 tweets
Profile picture
Ajay
@R_Ajay_
#History/#Art thread – From my recent trip to Madurai/Tirunelveli. On Jain sites (some with a rich history dating back to at least the 3rd cent. BCE), early Pandyan rock-cut cave temples (7th-8th cent. CE) & the amazing Vettuvan koil.

Pic: Mahavira (~8 cent CE), Kizhakuyilkudi
Before I start on the places, the history around all this is fascinating & splits opinion. Is it apt to call the script Tamili/Tamil Brahmi? Did it develop independently in the Tamil country/come with the Jain monks from the north? Were 8,000 Jains actually impaled in Madurai?>
>Were the Saivites overaggressive in promoting their faith? Shall only be happy to discuss on these separately (unless it gets too long/irrational). And I’m not posting every inscription from the places here, despite their importance. Again, shall be happy to discuss them.
Read 41 tweets

Trending hashtags

#ElanKriegel #Google #HistTech #Dumbs #Kubernetes #NothingToSeeHere #IBOR #STS #Antibiotic #engineering #SearchEngine #china #advice #free #FederalReserve #AlternativeMedicine #BayerProcess #Medical #NBAChinaGames #GoogleTrends #Fascism #HistoryForgotten #GOOGLE #neuroscience #MindControl
Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!