Thread by @kaslinfields

It's cool to hear the ways folks today can understand Borg, a technology like 15 years old, via papers about it from 6 years ago- by comparing it to their modern experience using its open source descendant, #Kubernetes today.

https://twitter.com/rothgar/status/1459209556525867009

The goals of borg and the ways they were accomplished are similar enough to k8s to be recognizable, but sometimes accomplished a bit differently. Similar naming conventions (borglets vs kubelets), BCL vs YAML, etc.

A difference is that the separate components of the Kubernetes control plane all seem to be part of a singular component, the Borgmaster. Which means Borgmasters could scale vertically, but not really horizontally - which is allowed more by the k8s component separation.

Borg approaches scaling, and related performance, differently from Kubernetes. k8s took a more pluggable approach to be able to work in more environments, whereas Borg clusters are generally a whole datacenter. So there are different assumptions about how hardware communicates.

Borg had 4 9's availability. That's about 1min of downtime per week. A lot of businesses couldn't allow that much downtime, but Google did (to some extent maybe does?) it. Very interesting perspective.

Tip from @rothgar when reading whitepapers - if you can't understand what a graph is trying to say, try imagining how you would visualize what the text is teaching you. Graphs are dense with information, so it can take some thinking to sort them out!

@rothgar Borg came about years before the popularization of containers. So their process isolation uses some components of containers (like cgroups), but it's much less isolated than modern containers. Which made sense for the time and use case.

@rothgar By the way, cgroups were created and contributed to the Linux kernel by Google. So containers in their modern form are built on this early work, so reading the borg papers gives insight into a moment of time with regard to the development of process isolation.

@rothgar An earlier point I didn't tweet was Borg's resource utilization. The system didn't (& I'm sure still doesn't) use its resources at 100% efficiency. They shoot for a percentage of overhead space, which is still a best practice today.

Kubernetes has a much higher focus on being extensible and flexible, whereas Borg was/is a much more limited use case (which is fine). For example, there's no concept of a Borg operator or custom resource in the Borg papers. Which is an important capability of Kubernetes.

The Borg papers just call all its workloads "jobs." They mention a limitation of needing abstractions to run microservices, several components of a single app. Kubernetes has several object types, but really still no abstraction of a whole app, which can be several deployments.

Thanks @rothgar for this! I, and several others, didn't get to read the paper but that didn't hinder our ability to learn from the discussion. Several folks asked great questions about limitations of Kubernetes that they wanted to compare to Borg, and I learned so much from that!

I should say learn and participate in the discussion! Anyone who wants to join the next one should! Even if you didn't read the paper.