Thread by @lzsthw

At the bottom of this:

computerweekly.com/news/252475310…

Is a timeline for the Horizon IT/UK Post Office bug that sent 550+ people to bankruptcy or prison for thefts they did NOT do. I'm going to try to figure out what was going one but someone might have to school me more:

First, apparently the UK the post office does a kind of franchise thing where in small towns someone can run a sub-postoffice that's kind of like a 7-11. In the US the post is run only by the government and only unionized employees handle mail. This is important to the story.

These sub-contractors are responsible for the money in every store, and every *device* that gathers money. For example, lottery tickets, ATM machines, etc. In the US there's nothing but mail, maybe a few government items, but no additional things like this.

All of these disparate computing systems that gather money are connected using a system called Horizon IT. From what I gather this is a system sold by Fujitsu, so I can already tell you one fact:

IT IS A MASSIVE PIECE OF CRAP RUNNING ON A MAINFRAME USING COBOL.

I'm totally guessing at this, but I'll bet someone a signed painting that there's a mainframe and COBOL at the heart of this debacle. How do I know?

A mainframe and COBOL contributed to the 2008 banking collapse, and I know 'cause I worked in IT at Bear Stearns then.

The reason COBOL and Mainframe is significant is because both of those technologies pre-dates the internet--or even the concept of interconnected computers--by a bout 20 years. They are clueless about other machines except for email and green screen terminals (to access email).

Apparently, this Horizon IT system would connect to the other Cash Computers (ATMs, vending machines, lotto machines), calculate the money that's been paid out, then look in the sub-contractors cash deposits, and figure out if there's a discrepancy.

You remember the COBOL?

So what's going on here? They won't say, but this article:

bbc.com/news/uk-291308…

Has this quote:

"equipment was outdated, and that power cuts and communication problems made things worse."

Ohhhhh ho ho ho what do we have here! As a programmer I can totally figure out what happened.

Remember that COBOL Mainframe? It was designed under the old idea of "network" where everything was in one giant building and constantly connected.

But, these are not.

I'm guessing, the power would go out, or there'd be some kind of connectivity issue to the lotto or ATM or vending machines. Maybe they get unplugged for a few minutes.

The Horizon IT system is coded based on the idea that this never happens, that connection is always there.

Now, somehow this causes a loss in transactions flowing from the lotto machines *into* the sub-contractor's cash register/accounting system. Now the computer comes back online, and reports its earnings to the central Horizon office, which records all of the transactions.

So, that might look like this:

20 sold -> sub-contractor
[[[power out/no connection]]]
30 sold
50 sold
[[[power on]]]
50 sold -> sub
50 sold -> sub

200 sold -> central office

That gives 80 missing in sub's system, and that's a lot of money.

This could be anything from power fluctuations, to bad wiring from the cash machines to the sub-contractor, to just bad configuration when it was installed. I mean, they could turn on a computer that used DHCP and stoll the lotto machine's IP address.

The report also mentions bad training, but you can actually read that as:

TERRIBLE COBOL MAINFRAME IS TERRIBLE TO USE.

I could do a whole thread on why people think mainframes are for money, but I'll stay on topic for now.

Here's some stories from people who were thrown in prison, bankrupted, and had their lives ruined by this:

bbc.com/news/uk-englan…

And interesting common thread is:

They're all way out in the middle of nowhere. Mostly remote small towns...with unreliable power and internet.

The Post Office maintains that trillions of transactions by 76k people were done successfully, and it was just a few idiots.

Problem is, I'm betting the problem was *way* worse, but only people in remote rural areas had massive losses and convictions.

Probably everyone using Horizon was hit with say a 1% or 3% loss due to missing transactions, and just paid it out of pocket since it'd be small.

These few people were hit with 10-100k losses and convicted of theft or fraud, served time, and had to pay it all back.

In fact, I believe now the sub-contractor systems should be considered the more accurate, and historical records of Horizon IT transactions should be compared with all of them to determine...

HAHAHAHA IT'S COBOL PEOPLE. THEY COULD NEVER WRITE THIS CODE!

Back in 2008 Bear Stearns had something like a 20000% increase in failed trades for two weeks leading to their collapse. Yes, it was something insane like that. The reason BS didn't know about this is because a mainframe running COBOL did all the processing.

In order to write the report that would have told them these failed trades has increased they would have had to write a massive stack of CICS code, that translated to a massiver stack of COBOL code, all of which would about 6 months.

That's to change 1 screen.

The idiot coders at Bear Stearns were ultra proud of their printouts of CICS code and their even bigger printouts of COBOL code. They acted like this was *real* code because it managed 400 billion dollars, but this real code was one big reason they collapsed.

This love of mainframes and COBOL in money operations is bizarre because it ignores the *insane* amount of money companies like Amazon and others make without using any mainframes.

I call it the Boomer Finance Guy Stack:

MAINFRAME
COBOL
CICS
TERRIBLE C#/JAVA WEBSITE BOLTED ON

So, Horizon IT is a system dealing with money; that can't handle computers not being connected; that had massive accounting losses without anyone know; that's created by Fujitsu, a big mainframe seller; that was insanely hard to use; which means it has to be the Boomer Stack.

Oh just thought of another way "connectivity issues" could cause this in the reverse:

If Horizon assumed constant connections then another possible error is when a lotto machine comes back online it doubles the previous transactions, that script might be:

1. Lotto gathers 20 transactions.
2. Power goes out, so lotto can't report to Horizon.
3. Power comes back on, lotto reports the 20 pending transactions.
4. On the next Horizon update cycle, horizon asks lotto for its transactions.
5. Horizon then *double accepts* the 20.

I'm sure you could just use the blockchain to fix this.

GET ON THAT COBOL BLOCK CHAIN WANTREPRENEURS.

(I joke but this would actually probably solve a lot of problems.)