Thread by @embano1

Did you ever ask yourself if cgroups (containers, pods, whathaveyou) can impact the @golang runtime? I did and here's what I learned... (thread) 👇#Docker #Kubernetes #Linux

So ~2y ago, while working with a lot of customers moving into containers and #Kubernetes, I naturally hit the (now well-known) Java/JVM cgroups issues (all you need to know -> lightbend.com/blog/cpu-consi… and ).

Now, @golang being the language of my (your?) choice, this made me curious and I started to ask questions:

1) Does the Go runtime consider cgroups settings, e.g. for CPU and memory, i.e. is it "container aware"?
2) How does the Go scheduler deal with CPU contention (cfs_shares) or cfs_limits, bc remember it's kinda two-level scheduling (multiplexing Goroutines onto OS threads)?

Note: If you're new to Go or only have a limited understanding of how the runtime scheduler works, look no further @kavya719 "The Scheduler Saga"

Still in 2017, I started to research (Google, Google Groups, Github <- e.g. searching for GOMAXPROCS through code bases), which did not get me very far, but provided some pointers in the right direction.

I found that @joshroppo had posted a similar question on Google groups asking for the effects when running in Kubernetes (groups.google.com/forum/#!search…)

Another comment from a Google engineer supported my suspicion: "Inside Google. we automatically update GOMAXPROCS when the job management system indicates that the container resources have changed." groups.google.com/forum/#!msg/go…

Yet another proof: Uber (of course, who else ? 😀) already had a library for dealing with this: github.com/uber-go/automa…

And then I found this issue by @monzo: "runtime: long GC STW pauses (≥80ms)" github.com/golang/go/issu… < great read and analysis how the resolved GC pauses impacting their Kubernetes workloads.

At least now I knew that there's something I had to be aware of when running (certain) Go workloads in containers, e.g. on Kubernetes. Now I was curious what the effects of CPU cgroups (CFS shares and limits) were on the runtime.

I wrote a simple CPU bound benchmark calculating prime numbers (sieve of eratosthenes) leveraging Go's concurrency strengths: github.com/embano1/gotuto…

I also found a nice benchmark (medium.com/@deckarep/the-…) on comparing some of Go's sync.<primitives> that came in handy for my cgroups tests.

You can find the analysis and results (based on Go v1.9 by then) here: github.com/uber-go/automa… Note: I just repeated them with v1.12 and the results are similar.

It boils down to cgroups (limits) can have a huge performance impact on your workloads, depending on your machine size, GOMAXPROCS (which defaults to NUM_CPUs in your OS) and applied cgroup limits (cfs_quota). This talk has details on quotas, etc.

Just like in the case of the JVM (and others), the lower the quota and the more threads in your cgroup, the faster they eat up the quota. Due to Go's multiplexing, e.g. 10000s of Goroutines on a much smaller number of OS threads, this can have a big impact.

Here's a trace (from 2y ago, but you'll find the same results for highly concurrent CPU bound workloads) that shows how multiple goroutines are starving on an 8-way machine.

Ok, you might say: well, I've not (yet?) seen this in my setup. Or the benchmarks (as always) are very synthetic and not reflecting real workloads. Fair points! First, if you listen to many failure stories (Kubecon), cgroups in production are always amongst them :)

Specific to this thread, scenarios like the one Monzo ran into show that this can be the cause of prod issues. Google and Uber aligning GOMAXPROCS with cgroup settings are a clear sign that it might be worth considering doing the same (or at least measure and compare in your env)

Thankfully, some very nice people from the cortex and @PrometheusIO communities did some validation and actually proved my findings. github.com/coreos/prometh… github.com/cortexproject/…

Note: I would be great if you could open an issue for your project and post your findings (whether there's impact or not) linking it to this discussion from @UberEng : github.com/uber-go/automa…

I talked a lot about quotas (limits) here, which you might be able to work around, e.g. disabling limits/quotas in your container/pod spec or on the kubelet. Actually this is what Google, Monzo, @ZalandoTech and others already apply as a best practice.

However, you might still run into the described problem: it depends on how you set your cfs_shares, or in Kubernetes speach "requests". If your machine is oversubscribed and processes contending for resources, the same effect will bite you.

Recent JVM versions translate cfs_shares into a number of threads. You might want to think about this (GOMAXPROCS=requests) if your workload is cpu bound at the likely expense of lower performance (i.e. when there's no contention). You certainly gain predictability.

If you're interested some further background reading on the Linux kernel (CFS) and cgroups (v1): kernel.org/doc/Documentat… helix979.github.io/jkoo/post/os-s…