Thread by @LifeRemainsGood

’ve read this and I seem some real red flags in some of the interpretations in this blog. /1

lockdownsceptics.org/code-review-of…

Since the author laid out their experience, I’ll lay out mine: /2

I was the OS Technical lead on a real time operating system consisting of over 20 million lines of code. I was an OS Architect, OS technology manager, and OS consultant. /3

I’ve been an Operating System architect and have also provided expert consultation with companies such as Microsoft, Nokia, Toshiba, Fujitsu, Panasonic, Texas Instruments, Vodafone, France Telecom, Sony Ericsson, Motorola. /4

I also did Computer Simulation in my degree. It was a long time ago, but I don’t remember anyone saying that models needed to be deterministic. /5

But this is Stochastic Simulation, which is about guessing based on probability, and the randomness is dependent on what you want to model. /6

Can you adequately model Skyrmion Brownian Motion without true randomness? I suspect not. /7

I think it’s perfectly reasonable to question the model creators in terms of their use of randomness, but not to just claim it’s wrong just because it’s not deterministic. /8

Then the author does something strange, they point to a defect to claim that is an example of how results can’t be repeated because of “bugs”. This is not an accurate description. /9

They also claim that the “predictions varied by around 80,000 deaths after 80 days”. That’s not what the bug report claims. /10

The bug report says that the bug “mostly manifests as a time-offset”. Bring the lines back, and it’s the time the time things happen that are different on those graphs, the lines are broadly consistent. /11

My first red flag: Someone arguing against the lockdown who has represented a curve starting earlier as less deaths. I don’t believe this is dishonesty, but bias in interpretation. /12

Now, the Imperial response is, as claimed, to point to known variation, but I have been responsible for the output of a lot of people, this is not something that doesn’t happen in the professional world. /13

The next line after is the telling one, the reporters respond saying that if they continue to use the file it “will be identical”. Hang on, I thought there were bugs making it non-deterministic? /14

If you create the scenario in the first run and then look for normal distribution using the file, then this bug won’t lead to onerous results. The first run will just be a statistical anomaly. /15

But the author doesn’t seem to care about that, instead focusing on the fact this is a bug and they don’t seem to care about the initialisation randomisation, which they are not in a position to judge. /16

A valid question to Imperial would be “Based on this defect, how does this affect the results based on the way you’ve used the model?”

Not: “Bookem Danno. Let’s all go outside now. Case closed”. /17

The author also complains it’s single threaded “rather than the many cores that any video game would successfully use.” – I own a computer game company. We run in tight loops. /18

It’s not impossible for a game to be running another thread, but it’s a specific case, not “any video game”. /19

The big red flag for me in this article comes with issue #30, which is a floating point rounding error affecting the test run.

An AVX512 giving a different floating point rounding error? Get away! Really!? 🙃

/20

This is not a bug, but down to the way all computers work. I mean, they could try and change the code to make this not happen, but only if they wanted to take part in a colossus waste of time. /21

And now we can be in stitches, because the author writes: this new problem “will just add to the issues”. (The problem of having more accurate figures) /22

Again, I think if the author had looked carefully at what was being said, they’d have worked out what was going on. In fact they claim the test doesn’t work, and the defect says it does, just on other platforms. /23

Finally we’re pointed to a file which the code is pretty rank, but I have some experience here too. I used to maintain a language called PLK2600 which sought to replicate the language used in universities for fibre optic measurement. /24

Normally, I’d say “Badly written but very clever, that’s how it is”, but if that needed fixing and the engineers got sick, then nobody wants to start there. There are comments elsewhere, but that particular file is bad. /25

And I’m more worried about something that isn’t mentioned in the article. The code wasn’t written to be parallel and they are using OpenMP to make that happen. /26

Contrary to the article, I’m going to say get rid of it. Rewrite it if you really want parallelisation for that task, but what is going on there I really don’t like. /27

They have probably ended up with a race condition, which apparently being flagged as the end of the world, but that’s not how it works in the real world. /28

When a defect is identified it is evaluated and prioritised accordingly to how serious this is. This code is not multithreaded, and if it was serious, it wouldn’t actually be that hard to fix it by removing OpenMP statements. /29

I suspect that they've looked at it and concluded it's not an important issue in the scheme of what had to be done. It’s not that there isn’t a question to be asked, but the problem is, the article then gives the answers it finds politically convenient. /30

With my OS Tech lead hat on, if someone came to me with those bugs, I’d want to know about the file save issue first, because while there may be reasons it’s not a problem, there are reasons why it could be. /31

For the multithreading initialising bug, I’d speak to the engineer and ask him while giving him the respect he deserves as someone who is working in their specialist field on code they crafted over months. /32

And if someone came to me with a test failing because multiple floating point arithmetic was being done with a single rounding, I’d throw a party and assign it to the next release for that OS Tech Lead to “worry” about it. /33

And as for the Insurance industry doing this (and you know what I’m going to say ->) I’ve worked as a system tester in the Insurance industry way back in the mid-1990s. /34

I had to compare the company’s key feature documents generated by a dialup system called “The Exchange” against the internal system they were developing. /35

If anybody thinks a single person gave a damn about some rounding errors in a 10 or 20 year projection table, then think again. A 20 year projection of fund growth out by about a pound for a few years? /36

And that’s the point. Things can be wrong but that isn’t necessarily a big deal, the question should be: How do they effect what you’re trying to achieve? 37/37