Kubernetes Borg/Omega history topic 9: Scheduling constraints. I have volumes more to write about configuration, but will move on with history topics for now. Borg's set of constraints grew organically over time. It started with just required memory, before multicore and NPTL
Other resources were added: cpu, disks. Hard and soft constraints on key/value machine attributes, and “attribute limits” to limit the number of tasks per failure domain. Automatically injected anti-constraints were used to implement dedicated machines
In Omega (ai.google/research/pubs/…), we added the concepts of taints and tolerations in order to subsume a number of ad hoc means to prevent scheduling of most tasks and/or evict them from certain machines, and forgiveness to defer eviction
These scheduling features made their way pretty directly into Kubernetes: issues.k8s.io/168, issues.k8s.io/367, issues.k8s.io/1574, issues.k8s.io/17190. @davidopp, who was the TL of scheduling in Borg and Omega, worked on a number of these features in K8s too
A scheduling braindump I wrote in early 2015 (github.com/kubernetes/kub…) possibly helped to convince some that Google really was fully sharing its experience with the project. The scheduling design docs can be found in github.com/kubernetes/com…
These mechanisms can be used to manage how workloads are binpacked for efficiency, spread for availability, isolated from one another for performance or reliability or security, colocated with required resources, matched with desired configurations, and manage node drains
These scheduling primitives are pretty flexible, but if there are constraints or other policies or criteria that can’t be represented, users can use their own schedulers. In order to do that in Borg, one would have to add a constraint to a task to pin it to a specific machine
The Omega paper compared performance of 2-level scheduling with information hiding, but one issue it didn’t mention is that the lower-level scheduler needs to implement all of the same constraints as all the upper-level schedulers, or it may never satisfy their requirements
Anyway, while resource optimization is an important concern, there are many other considerations in decisions, such as whether container images already resident, which facilitates faster start time
