Thread by @freebsdfrau

I am releasing details today. All information has been gathered, timelines analyzed, old code vetted, major flaw fixed, and results scrutinized. Strap-in boys and girls, this rollercoaster goes to 11! Sun Microsystems #GridEngine has been hiding an ace 🂡 up its sleeve for 16 yrs

https://twitter.com/freebsdfrau/status/1274627170573225985

Show you the code? OK ... github.com/FrauBSD/sge/

Still interested in how a decades-old job scheduler might have contained the necessary bits to dethrone even future contenders?

Stick around and we'll see how Sun #GridEngine may just surprise you

Schedulers: Airflow, Mesos, Torque, Univa, Open Grid, Sun Grid, Cook, Slurm, there's so many. For the sake of argument, let's say you need to schedule 10M jobs but you only have 100K cores. Does your scheduler break down like most of those listed at 10-30K pending?

Some schedulers thought they would out-perform decades-old Sun #GridEngine by using an optimization, but little did they know that SGE contained the same exact optimizations (off by default and unstable) more than 10 years earlier slurm.schedmd.com/SUG14/sched_tu…

Univa -- the ultimate successor-to, and based-on, Sun's code -- even spent time optimizing the scheduler and boasts a whole 6x improvement over SGE's scheduler. But, young grasshoppers, what Univa and friends do not know is that SGE has been hiding this feature I will show you

Hop in the time machine with me, and let's travel to the year 2004, a hair over 16 years ago. A Sun Microsystems employee by the name of Stephan Grell had been working on an experimental feature to improve the performance of #GridEngine dramatically and on July 30, he commits it!

Stephan Grell did this in 2 landmark commits (would have been one, but he forgot to add the documentation to his first commit, so soon thereafter came another)

github.com/gridengine/gri…

github.com/gridengine/gri…

We have the history thanks to Oracle and Univa for keeping it around

About 20 days later, Stephan Grell then updated the SGE tuning guide to advertise his new feature. My estimates put his optimization anywhere between 10x and 600x improved efficiency depending on real-world testing.

github.com/gridengine/gri…

gridscheduler.sourceforge.net/howto/tuning.h…

What was to happen next is an absolute tragedy (with a deferred happy ending albeit 16 years later). The feature sat idle (disabled by default) and likely became forgotten. 3 years 9 months later, Sun employee Roland Dittel motions to remove it github.com/gridengine/gri…

2 months after Roland decided that this experimental feature (now ~4yrs old) needed to go, he sent out one single e-mail to the #GridEngine mailing lists, asking if anyone uses it before he wields his axe arc.liv.ac.uk/pipermail/grid…

If it were not for Jeff Beadles [same-day] response to Dittel, I myself might never have learned of the feature. Spoiler alert, despite Jeff's plea to save the feature, Dittel marks it for death anyway, just 14 days after Beadles' plea github.com/gridengine/gri…

Through a VERY fortunate turn of events, we still have the feature to work with today. Less than 4 months after Roland Dittel marked the optimization for removal, IBM and Sun began to discuss a potential merger, on November 6th, 2008 web.archive.org/web/2013061518…

Another 4 months later, in March of 2009, talk of merger between IBM and Sun had stalled en.wikipedia.org/wiki/Sun_acqui…

The following month, on April 20, 2009, Oracle and Sun announced a definitive agreement. 9 months later (because that's just how long these things take ^_^), on January 27th, 2010, Oracle announces completion of its acquisition of Sun Microsystems en.wikipedia.org/wiki/Sun_acqui…

12 months after that, on January 18, 2011, Univa announced it had acquired from Oracle the core Sun #GridEngine developers including the CTO en.wikipedia.org/wiki/Univa_Gri…

2 months later, in March, 2011, Univa rebrands SGE as Univa Grid Engine primarily through these 3 commits

github.com/gridengine/gri…

github.com/gridengine/gri…

github.com/gridengine/gri…

2 weeks after they do that, they (Univa) release as the company's first initial offering, Univa Grid Engine 8.0, on April 12, 2011 en.wikipedia.org/wiki/Univa_Gri…

Univa continues to develop their "Open Core" publicly on GitHub for another 13.5 months github.com/gridengine/gri…

The feature created by Stephan Grell at Sun 16 years ago which got lost in the turmoil still exists in both the last version of the Univa Grid Engine Open Core, last updated June 1, 2012, as well as latest son-of-gridengine, updated June 15, 2018 (just 2 years ago)

You can see Stephan Grell's work, never actually removed, despite being on death-row for 12 years, still in the code here:

github.com/gridengine/gri…

github.com/son-of-grideng…

So, what do we do with features that were put on death-row by companies that don't control the code anymore and don't even exist anymore?

We give them a pardon! We buy them a nice suit, give a hot meal, and a let them lose in the World and see what they can do.

Technically speaking, this feature is old enough to legally work, so put it to work we did.

Immediately it had solved the problem (expertly described to Dittel by Beadles in 2008).

But we turned out backs, and by the 3rd day it had vomited all over the place!

And now we knew why it was marked experimental, given a death sentence, put on death-row, and awaited execution. However, ... the people making those prior decisions never had me look at it. Recognizing the raw performance improvement, I more than doubled-down. It would be fixed.

It was May 28th, 2020 @ 17:23:08 PST when we turned it on for the first time. We couldn't even get to June before it crashed. Super nasty. Would take over 15 minutes to come back up. Definitely not acceptable. It wouldn't be until July 29 @ 11:42:48 that we had the fixes in-place

Over a period of 2 months I would be wracked with highs when I thought it was fixed and lows when it proved the code had more to teach me. 2 months of doing battle in dark caves with no help and a dimly lit torch (the documentation). Weeks of abject horror followed by euphoria

The end result is that this feature, now with over 5M jobs under its belt and a full month of service, is looking like it has the mustard to dethrone every modern/maintained scheduler on the market today. I'm not joking. There's even a bakeoff happening in Amazon EC2 right now!

Here's some before/after photos of job-wait times for pending jobs on the cluster showing that in the department of getting your jobs onto the cluster, it's definitely winning. Red line represents when we turned the feature on

But let me put it into context of numbers.

With Slurm, you'll start to feel the pinch at 10K+ pending jobs. With SGE (without JC_FILTER by Stephan Grell), you'll feel the pinch at 30K+ pending jobs.

SGE + JC_FILTER ...

No pinch at 300k pending jobs and we're still going

So where do we get 600x optimization at the upper-end? With as little as 100K pending jobs, it can take the scheduler up to 30min to complete. Compared with just 3 seconds to complete with the same pending-job load and all you have to do is turn on JC_FILTER=1 using qconf

But, without this patch, you better not use qalter if you have JC_FILTER enabled github.com/FrauBSD/sge/co…