Our research of #Java on #Kubernetes is pointing us into looking at:

- ActiveProcessorCount
- Default GC
- Default Heap Sizing

Here are some facts: 1/n
ActiveProcessorCount currently matches cpu_quota: up to 1000m, 1 proc. 1001-2000m, 2 procs, and so on.

This sounds reasonable, but as we've learned, CFS control does not limit CPU count, only CPU time. Multiple native threads may execute in parallel, until the quota is reached.
JVM Ergonomics also selects G1GC by default only when the JVM sees more than 1792 MB of available memory, and 2 available processors (2000m+ in k8s). Otherwise it picks SerialGC.

Thus why so many devs force -XX:+UseG1GC in well constrained containers.
The heap size is by default 1/4 of available memory. And many developers set heap with Xmx to something between 50% to 80% of available RAM.

Ideally, devs should be using -XX:MaxRAMPercentage so the JVM scales along with the container in k8s.
We want to understand why the JVM behaves differently when sizing itself for more available processors even though it is being CPU throttled. Something to do with thread scheduling perhaps?

Also, how this impacts G1GC, and if more heap memory could reduce CPU throttled time.
Evidently though, developers are not sizing their Java workloads properly for Kubernetes, and without taking into consideration how the JVM works and it's need for parallel, multi-thread work such as JIT compilers and GC.

1000m doesn't seem a good minimum, and yet it's common.
And solving CPU Throttling with more replicas also is a common way to get around, but not the ideal solution, in my opinion.

Our hypothesis is still that for some workloads, less replicas with increased cpu limits, potentially with cost savings, may deliver better performance.
But finding that the JVM may behave better with low cpu limit, as long it is sizing itself for multiple processors, was an interesting experiment to day the least.
If you have other observations to share, please do! If you tried any of the changes so far shared here, let us know how that go with your systems.

There's a lot more to improve #Java on #Kubernetes.

Cheers all!

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Bruno Borges

Bruno Borges Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @brunoborges

24 Feb
Not only #Java developers are deploying their microservices to #Kubernetes with 1 vCPU only (1000m), there may be multiple pods on the same nodes since podAntiAffinity is unlikely to be common practice when scaling, to get an even distribution.
So, instead of having a bigger pod (2000m-4000m) on a node, there are two or more 1000m pods on a node.

For web apps, I think this setup is very unlikely to be reasonable.
I hope I am wrong and that my lame knowledge of Kubernetes followed by my hate towards YAML is as bad as this hypothesis.
Read 5 tweets
23 Jan 19
#Java's death will be due to SEO mess, lack of centralized docs, outdated tutorials & no one-stop-shop place to find it all

- go.java
- openjdk.java.net
- oracle.com/java
- java.oracle.com
- java.com
- docs.oracle.com/javase/
- go.java

This site was supposed to be the answer, but it only raises even more questions. The information here is mostly marketing-related, and has pointers to other online locations, outside go.java.

Potential: very high.
Reality: waste of TLD.
- openjdk.java.net

The Java source, no doubt. Yet, devs who want to just learn/code Java apps will only find binaries downloads. Nothing else serves the average Java software developer. "Developer's Guide" is for contributors.

Potential: high.
Reality: confusion.
Read 8 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!