Thread by @brunoborges

Our research of #Java on #Kubernetes is pointing us into looking at:

- ActiveProcessorCount
- Default GC
- Default Heap Sizing

Here are some facts: 1/n

ActiveProcessorCount currently matches cpu_quota: up to 1000m, 1 proc. 1001-2000m, 2 procs, and so on.

This sounds reasonable, but as we've learned, CFS control does not limit CPU count, only CPU time. Multiple native threads may execute in parallel, until the quota is reached.

JVM Ergonomics also selects G1GC by default only when the JVM sees more than 1792 MB of available memory, and 2 available processors (2000m+ in k8s). Otherwise it picks SerialGC.

Thus why so many devs force -XX:+UseG1GC in well constrained containers.

The heap size is by default 1/4 of available memory. And many developers set heap with Xmx to something between 50% to 80% of available RAM.

Ideally, devs should be using -XX:MaxRAMPercentage so the JVM scales along with the container in k8s.

We want to understand why the JVM behaves differently when sizing itself for more available processors even though it is being CPU throttled. Something to do with thread scheduling perhaps?

Also, how this impacts G1GC, and if more heap memory could reduce CPU throttled time.

Evidently though, developers are not sizing their Java workloads properly for Kubernetes, and without taking into consideration how the JVM works and it's need for parallel, multi-thread work such as JIT compilers and GC.

1000m doesn't seem a good minimum, and yet it's common.

And solving CPU Throttling with more replicas also is a common way to get around, but not the ideal solution, in my opinion.

Our hypothesis is still that for some workloads, less replicas with increased cpu limits, potentially with cost savings, may deliver better performance.

But finding that the JVM may behave better with low cpu limit, as long it is sizing itself for multiple processors, was an interesting experiment to day the least.

If you have other observations to share, please do! If you tried any of the changes so far shared here, let us know how that go with your systems.

There's a lot more to improve #Java on #Kubernetes.

Cheers all!

Oh and please,.do take this survey.

https://twitter.com/JavaAtMicrosoft/status/1364741339758219264?s=19