,
20 tweets,
7 min read
Read on Twitter
Kubernetes Borg/Omega history topic 8: Declarative configuration and Apply. Inside Google, the most used configuration approach for Borg is the Turing-complete Borg Configuration Language (BCL). You can see a snippet of BCL on slide 7 in this deck: inf.ed.ac.uk/teaching/cours…
Millions of lines of BCL have been written. A fair amount of BCL was devoted to configuring application command-line flags, which was the most common way to figure server binaries, which is crazy IMO, but the practice sadly carried over to Kubernetes components
BCL was evaluated and instantiated using the borgcfg CLI, which supports commands like up, down, and update. Logic to diff and merge, perform rolling updates, and otherwise update the live state was embedded in the tool. Logic for common generation functions was written in BCL
This created a monolithic configuration and tool ecosystem. Even frameworks like mapreduce and services on top of Borg like BorgCron had to use BCL and borgcfg to interact with Borg. Getting-started tools generated BCL
A Python-based language was later developed, also. It interfaced with the update logic via a protobuf that wasn't quite the same as Borgmaster's. Other languages, such as Ruby, weren't used in Google. Several new Borg config languages were developed, but none were approved
Not specifically developed for Borg use, jsonnet.org/articles/desig… and github.com/cuelang/cue/ were inspired by BCL. aurora.apache.org/documentation/… and github.com/stripe/skycfg were inspired by the Python language.
Borgcfg didn't provide configuration packages. Shared templates were unversioned and directly imported from their homes in the monorepo, which inflicted churn on their consumers. There were also no "stacks" or lifecycle directives, so a number of imperative updates were needed
In Kubernetes, we wanted to decouple configuration authoring and generation from updates to the desired state via the API, so that users could express configuration using languages and tools familiar to them: Jinja, Python, Ruby, Javascript, Terraform, Ansible, whatever
I wrote about this in prs.k8s.io/1007. I also felt it needed to be possible for automation to write directly to the API and not need to update some arbitrary configuration language. To do that, we needed to be able to merge user intent and automated changes
My initial proposal, in issues.k8s.io/1178, was to maintain and merge 2 separate layers of desired state in the server. Resistance to that idea led to my client-side Apply proposal in issues.k8s.io/1702. We're finally getting server-side apply: github.com/kubernetes/enh…
A gotcha we ran into early in the Apply implementation was complex schema topology. Merging 2 flat maps is easy, but we unfortunately had associative lists: github.com/kubernetes/com…. And also sets and undiscriminated unions (being addressed: github.com/kubernetes/enh…).
Strategic merge patch was developed so that we could diff and merge two objects containing associative lists (non-ordinal lists with index keys in values of fields within list elements): github.com/kubernetes/com…
I wrote an overview of the motivation and principles for the configuration design in github.com/kubernetes/com…. The original draft of that also contained sketches of what became github.com/kubernetes-sig… and github.com/kubernetes-sig…
Whereas Apply facilitates collaborative config authoring between humans and machines (thanks to @originalavalamp for that description), kustomize enables collaboration among humans, by facilitating modification of unchanged base prototype/seed configurations.
The declarative API, Apply, and kustomize facilitate maintaining configuration as YAML or JSON or proto, amenable to manipulation by tools, rather than as YAML marked up with macros, complex configuration languages, or scripts written in general-purpose programming language.
On one hand, the ~100 tools that have been developed show that the decoupling of config format and the API has worked. OTOH, it shows there are still gaps. With work like diff and dry run (github.com/kubernetes/enh…) and prune (github.com/kubernetes/enh…), we're working to close them
A list of tools can be found here: docs.google.com/spreadsheets/d…. I just added another 20 or so that I've seen.
This thread is already the longest yet, so I'll start another later with configuration terminology: declarative vs intent, macros vs config languages, packages vs stacks, prototypes vs templates, whitebox vs blackbox, overlays, lifecycle directives, etc.
I've worked with @eric_brewer for several years at Google, including on configuration, between work Omega and Kubernetes. In the second half of this podcast, Eric also briefly discusses declarative configuration: softwareengineeringdaily.com/2019/04/26/clo…
BTW, eventually a "production database" called ProdSpec, did succeed and rolled out. Borg has converged towards a model similar to the Kubernetes Resource Model and "GitOps" (though our internal VCS isn't git), described here:
