Thread by @nickwallis

Good morning from the Salvation Army café in the shadow of St Paul's Cathedral. I am here to load up on caffeine before attending the final day of Phase 2 Week 2 of the Post Office Horizon inquiry at the IDRC. This photo was taken a little earlier today...

... this is the third day the inquiry has been sitting this week. Day 1 was on Tuesday during which the Inquiry's IT expert Charles Cipione started and finished the first part of his oral evidence during which he took the inquiry through the first part of his reporting into...

... the Post Office Horizon system. You can find that report here:

postofficehorizoninquiry.org.uk/sites/default/…

Yesterday was mainly taken up with former Royal Mail Group CEO John Roberts' evidence. RM group had Post Office as one of its three companies and Roberts was in charge during...

... the design and implementation of the Horizon system (between 1995 and 2002).

Roberts admitted Horizon was a fudge, but the best fudge, and recollects that despite being told there were problems with the system, it should be accepted by the PO and rolled out...

... because all the technical problems were manageable. He has no recollection of any discussion at RM board level as to whether Horizon evidence could, should or was going to be used in evidence in criminal prosecutions after its rollout. When shown evidence of...

... problems with Horizon's accounting integrity which were reported to the board, and not only persisted after rollout, but which continued to cause problems despite the extra software which was written to catch them, he said that it was up to the PO MD to bring it to...

... his attention if it was such a big problem it was going to affect the business.

As with many PO staff, Subpostmasters seemed to be little more than an afterthought.

You can watch his evidence and read the transcript here:

postofficehorizoninquiry.org.uk/hearings/phase…

Today we will hear the majority of the evidence from Paul Rich, who barely got out of the traps yesterday. Rich is a former development director of the Post Office. After he has finished today we'll hopefully hear from a former director of PA Consulting, Peter Copping...

PA Consulting were brought in to advise the PO and RM Group on whether Horizon would be fit for purpose. John Roberts twice refers to a representative of PA consulting advising the PO that Horizon was fit for purpose and they should accept it.

I'll live tweet from 10am as ...

... appropriate, but I'm not expecting fireworks.

Then I'll pull together a podcast looking back over the week with @Rebeccathomson_ and hopefully get a couple of blog posts up about the evidence we've heard this week and the Jenkins application for what...

@Rebeccathomson_ ... the journalist in me wants to call immunity from prosecution, but it's not really. The former Fujitsu engineer wants an undertaking from the Attorney General that any evidence he gives to the inquiry is not used against him in a future criminal prosecution...

... without this undertaking he could legitimately refuse to answer questions at the inquiry for fear of self-incrimination.

This has happened at the Grenfell Inquiry (where requests were partially grantsed) and the Manchester Arena (where, in the case of the bomber's brother...

... it wasn't granted).

Let's see what the day brings. Incidentally all of my reporting of the inquiry so far has been crowdfunded. If you can see your way to subscribing to my newsletter, for a single one-off payment, you can get the inside track on everything that's...

... happening, and fund some public-interest journalism.

The link and more info is here:

postofficescandal.uk/donate/

Thanks.

Oh and the hashtags I tend to use are #PostOfficeScandal and #PostOfficeinquiry - if you use these I've got a far better chance of seeing your tweet!

We are underway -

https://twitter.com/PostOffInquiry/status/1583382205577199617

Paul Rich is being cross-examined (xe'd) by Julian Blake (11kbw.com/barristers/jul…)

The inquiry building is in the foreground. St Paul’s (the camera is facing east) in the background.

We are right into the ancient history of the Post Office's Horizon IT system. This was, before launch, called "Pathway" and the company which won the contract was called ICL. ICL was a British company which by that stage was 80% owned by Fujitsu. Within months of...

... Horizon being rolled out in 1999/2000, ICL was completely swallowed up by Fujitsu.

So in broad terms:
Pathway = Horizon
ICL = Fujitsu

As we discovered yesterday, the three main contenders to win the contract to automate the PO were codenamed "Tom" (IBM), "Dick" (ICL/Fujitsu) and "Harry" (Cardlink). Dick won out.

Chat at the moment is very much about the management structure around the procurement of Horizon. As we know already, Pathway is seen as having the greatest technical risk.

Paul Rich describing the 19600 Post Offices when he was around:

"it was not a heterogeneous set - you had Crown offices with 20 counter positions and a front room in the Orkneys" says Fujitsu underestimated that this was not all covered by ISDN.

[btw nothing is a direct quote unless in "direct quotes" most of this tweeting will be paraphrasing and a summary of what's going on]

Rich: "we were acquiring a system not a piece of software and that system is about people and the human interface with that. That was always at the heart of our business, unlikely as it may sound now"

Rich says that the training provision was not adequate and they improved it after engagement with the NFSP and SPMs.

We were an unautomated business of 19,600 offices with 70k people needing training. That wasn't going to happen overnight without professional training.

Watching today and yesterday's evidence is like watching a car crash in slow motion hindsight. There was a political imperative, which came from Tony Blair, that Horizon could not be canned....

... this was not an unreasonable stance. Canning it would cost vast sums of money and do huge reputational damage to the government, the Post Office and Fujitsu. But...

... by going with the least worst option, the PO (after the Benefits Agency pulled out) and Fujitsu were left with a dog. It became imperative for Fujitsu to offload the dog onto the PO whereupon it became a golden goose.

The PO then offloaded the risk of the dog onto the Subpostmasters who were devoid of any influence bar that provided by the unions, the NFSP and the CWU.

It seems that no one at the CWU or NFSP saw the car crash that was coming and started drawing red lines...

... before endorsing the rollout of the system. This, I suspect, was down to a lack of any sophisticated technical knowledge within either organisation (or, at least, any that was focused on Horizon).

Even the PO, with it's greater (tho still limited) technical know-how, felt shut out of the delivery process by ICL/Fujitsu who told them that the terms of the PFI deal meant they should butt out.

We are seeing a lot of confidential memos up on the screens in the inquiry room which are also being shown on the youtube live channel. I've been told these documents will not be posted to the inquiry website until the end of Phase 2 of the inquiry so I'm screen-grabbing...

... them in sub-optimal quality as we go along. Documents of special interest may be made available on request.

At the moment its a run of documents which, to summarise, suggest:

There was lots of mission-creep from the clients (BA/DSS vs PO)

ICL/Fujitsu didn't seem to have the capability (managerially, technically or logistcally) to deliver the system as specced

The govt wanted PO automation to happen and it wanted it to happen sooner rather than later

Rich: "The delays were hurting every party financiall. There was no winner here. ICL weren't getting paid because the PFI structures. we were having our competitive position damaged.... but we had to hold our collective nerve to ensure what we put out was good enough."

We're going back to the PA consulting report of Oct 97 which found no "fundamental technical issues" with the proposed Horizon system.
Rich agrees says they took this to mean Horizon "could work... it wasn't technically intrinsically flawed"
[Reader, it was.]

[PA consulting have a witness up next]

Rich is taken to a letter he wrote in 1997 where he asks Peter Copping of PA consulting [the witness up next] if Horizon is or isn't "inherently unstable" and says his report is silent on this.

JB: "would it be fair to say that you're asking is it unreliabl even if it works?"

Rich: I don't think I had that in mind when I wrote it, is the truth

JB: were those sorts of questions being asked at the time?

Rich: we were all looking to the live trial. We had to have the live trial and see it worked before allowing the rollout. And see that problems that came out of the live trial were logged and dealt with.

JB: did you get an answer on "inherently unreliable"?

Rich: I think they concluded it wasn't "fundamentally flawed" and that I think answered my question.

We go to the live trial.

There were problems.

JB: If some SPMs were having some technical difficulties at theat stage when it was still a small project, would that have been a concern?
Rich: yes but my memory suggests these were not re balancing - it was screen freezes etc...

... you'll have to ask Dave Miller about it all, but I asked a Fed rep directly if he was happy about the system and he said directly he was.
JB would yo uhave expected issues to be raised by the NFSP
Rich the NFSP were not in the PO's pocket. They were vociferous...

... if they thought there was an issue and it had been substantive. I'm sure they would have been alerted to it and I a sure they would not have been slow in coming forward with those.
JB; what was the role of the NFSP as you saw it
Rich: getting SPMs together for reaction...

... the NFSP were involved early on in the genesis of horizon against the threat of ACT. They would have been representing their members fiercely to ministers.
JB did they have much of a say in what was going on
Rich. They were kept informed. They went to meetings with pathway...

... and pathway came to the conferences, but we didn't think about giving them a role on the board.

JB takes Rich to a minute of a meeting on 30 July 1993 where the automation of the PO was being discussed. The line "NFSP involvement would be kept to a minimum"

rich is surprised by this, but says "there had to be boundaries" "They were a trade organisation."

JB asks if people took a view that the NFSP shouldn't be getting involved in the technical side of things.

Rich says he's not sure they had any technical capability...

... but they were keen to get them involved in user testing.

We've moved from codename "Dick" to codename "Congo".

Congo appears to be the name for various testing iterations of early Horizon.

A report it brought up which shows some bugs causing balancing problems. One of the fixes includes a manual amendment which would have an effect on Post Office accounting.

JB so there's a manual amendment which would have an impact
Rich: it says Pathway are looking at amending

... I don't know if they did so, so I can't comment.

A note comes up from the "Counter Automation Steering Group" in 1998, chaired by yesterday's witness, John Roberts, RM CEO. One note says:

"work on EPOSS [the touchscreen till software which was bundled in with Horizon] was continuing...

... and Pathway had
indicated that whilst it could provide a system which
met the contract, its lack of robustness could generate
high level of errors within POCL...

... . This was being investigated although it was difficult to quantify how the system would work until after it had been installed and was operational."

Rich agrees this is serious, but demurs on whether Roberts should have escalated this to the RM Group board.

Another internal memo spells out the PO's dilemma:

"This means that we have to get the message across very clearly to officials/Ministers that we are not a pawn...

... in the game between DSS desire to exit and Treasury/DTI desire to prop up ICL and that if BA cease the payment Card we reserve our right to make our own decision on continuation/extension of the...

... contract or termination."

Rich being asked why a simpler system wasn't specced when the BA walked away.

He says there are all sorts of issues with that. What is an acceptable simpler system? What would we have expected to do with it going forward?

We get to a memo written within a year of Horizon's rollout in November 1998:

"There are major concerns about the test results... The results indicate that cash accounts and transaction data delivered to POCL's
downstream systems lack accounting integrity...

... all of which raises serious doubt about
Pathway's ability to enter into the next phase... testing
without some form of remedial action."

We then go to a letter from Bird and Bird solicitors who advised the PO in Dec 1998 on Horizon.

Bird and Bird had got a company called Project Mentors in to look at the Horizon system. This is what Bird and Bird say:

"Further to my Memorandum dated December 8th, I attach the full report of the
work by Andrew Davies and his team on requirements analysis. This fleshes out the
brief update from Andrew which...

... I sent you with my December 8th Memorandum. As
you will see, all three of Andrew's team are (I quote from Andrew's letter to me)
"deeply concerned that their findings show a serious problem with the way in which
ICL Pathway have developed the system. The impact...

... of this is likely to be that there will be failures to meet essential user requirements, causing the need for extensive re-work before the system can be accepted and, potentially, operational problems if the system is rolled out."

JB: is this serious?
Rich: it sounds serious.

Rich is poo-pooing the Project Mentors reports, suggest there is a bit of politicking going on as BA were trying to find an exit ramp from the whole project, not because it didn't work, but because they didn't want it.

Rich is surprised that Project Mentors previous report a year earlier had not flagged this.

JB were technical issues with Horizon being taken seriously enough in 1998/9

Rich we would have taken them seriously. We were not being expedient for the sake of speed or trying to shoehorn a solution that suited u strategically if we didn't think it could work.

Rich says he's "irritated" by the 1998 Project Mentors report. Clearly there was some considerable rancour between Benefits Agency and Post Office around that time.

Rich shown yet another document which states serious doubts about ICL/Fujitsu's ability to complete the project.

Rich notes that that concern was borne out by what happened.

Rich is finishing up - addresses the chair directly. Says he fully respects the inquiry has been reflecting on it. Wants to express his sadness on "the impact this has put on so many lives" - "I'm frankly baffled and shocked by how some of these people were treated...

... it's not a culture I personall recognise in terms of the ethos of what must have been happening at the time."

Rich is asked a final q. about PFI which he keeps saying is different.

Rich - contractually it meant the risk to design operate build and run was with the supplier not the procurer and this also icluded the risk around benefit fraud - not aware of ANY PFI in the world like that.

Also supplier ounly got paid on outcomes and outputs. Thy did not get paid till rollout. This was a learning experience because the PFI was unusial. We learned about the boundaries. ICL would say no under PFI you can't see the details of the design. Wait till we output test...

... them.

Also this was a project which straddled two governments.

Sam Stein KC on behalf of 150+ SPMs has some questions.

SS when giving your evidence you spoke about the Bird and Bird doc which "completely ignores teh fact it was under a PFI' - does that mean the burden for governance, running and maintencance of the project was on Fujitsu?

Rich: Primarily yes, but in relation to the supposed innovation they could bring in designing and developing the system.

SS you say you were baffled by what happened to SPMs - in your witness statement you say you were baffled by later professional advice and investigative...

... processes. What do you mean?

Rich; it would seem it was ill-advised in going ahead. The investigative processes were ones I don't recognised. Certainly non-disclosure would have been a real issue to me. The investigative processes around fraud... I can't see how a...

... board would not have noticed 700 people had been prosecuted cumulatively.

SS what control would put in place for investigative processes

Rich: I wasn't involved continually until 2002 - the processes were done separately and at arms length.

SS What do you mean...

... re bringing to the attention of the arms length process the existence of bugs and errors

Rich: didn't happen.

SS what was the system for bringing the existence of bugs and errors...

... to the attention of the investigation department?

RIch: don't know

SS: what should it have been

Rich: Who should have been in charge of it?

SS: Don't know. Can't help you.

[end of xe of Paul Rich. As he left the inquiry room I asked him if he would like to do an...

... interview.

"No." he said.]

[We've now broken for lunch. Back at 1350]

We're back after lunch. Mr Peter Copping formerly of PA Consulting is being sworn in. He's being cross-examined by the Inquiry's Catriona Hodge (5essexcourt.co.uk/our-people/pro…)

Copping (PC) is a chartered engineer.

He had been got involved with Hoizon in Summer 97 because there were concerns about delays, Pathway's ability to deliver and the Post Office's readiness to accept Horizon.
CH your review focused on business objectives, contracts, programme management processes and technical...

infrastructure.
PC yes
CH what's tech infrastructure
PC the hardware platform on which the software serviecs reside from the counter to the back end

For those who want to watch this afternoon:

https://twitter.com/PostOffInquiry/status/1583440334180450305

A cynic writes...

https://twitter.com/read292/status/1583440735667306496

PC currently talking about functions of the Pathway software which never got rolled out

CH moves to EPOS (the front end counter till point) and the APPS (automated payment service)
PC remembers nothing of this
CH asks him about his review which involved lots of interviews with stakeholders and then a document review...

[I think I will pause this live-tweeting until something germane comes up]

In the meantime, here's more on Gareth Jenkins' request for limited 'immunity' before giving evidence to the Post Office Horizon IT inquiry.

postofficescandal.uk/post/jenkins-w…

#PostOfficeInquiry
#PostOfficeScandal

PC asked about 98 report into Horizon which didn't look at H's tech problems.
CH "If you weren't making inquieties into the knowledge of faults and defects in Horizon how could you... make an accurated assessment of Pathway's ability to deliver this solution?"
PC "I don't...

... think... that's a difficult question to answer."

We go to the '98 report "The main architectural issues are scaleability and robustness. We are advised that a solution of this scale with so many different platforms and products has... as far as PA is aware... no precedent."

Report: "The project is probably the biggest of its kind and many of the component parts... are being used towards their current limits and scale. Pathway has recognised the risks and has in place the controls we would expect to see in a development project of this scale."

PA also concludes: "There is good evidence of future proofing at all levels. We have been satisfied that all reasonable steps have been taken to ensure robust sources of supply and compliance with industry standards in designing the architecture."

PA also identifies an absences of agreement - any agreement between the parties concerning the criteria and procedure for acceptance of the system.
CH - what were the areas of disagreement
PC - about the spec and criteria for acceptance and, as I recall, the basis on which....

... acceptance would be given.

[adds] I think all parties had resourced up and my recollection is that ther was still a concern that the pO were not ready to accept a system of this complexity - that's readiness in terms of preparing the network to live in a very...

... highly structured environment at a branch used to an unstructured environment.

PC adds there were concerns about PO managers being able to cope on a technical and managerial level with the size of business transformation Horizon represented.

PC then got involved again in Oct 1998 to look at feasibility of restructuring Horizon/Pathway project after the decision was made to press ahead with it.

[This was all before the benefits agency finally bailed on the Horizon project]

Copping's witness statement:

https://twitter.com/PostOffInquiry/status/1583452370704338944

Mrs Wallis has texted me to tell me that all power to Walton-on-Thames has been cut off. She has had to leave the cinema, and is currently at home playing board games with our son.
#howitbegins

PC's investigation in Oct 98 has found a serious "critical" issue affecting rollout which was a "failure to complete acceptance in planned timescales could cause one or more parties to resort to legal action and program [ie Horizon] could stop at end of 98 or before"

Pathway wanted acceptance for Horizon on a self-certification process which other parties weren't keen on AND wanted acceptance based on a "model office" (ie test) envirnoment, rather than "end to end" which would be a live system. Cheeky buggers.

Mrs Wallis texts to say power to parts of Walton has been restored. Hopefully no more #autumnblackouts

PC's memory is not as good as John Roberts or Paul Rich's. That's not a criticism. I only have the vaguest recollection of the late 90s.

PC wrote to the Horizon Prog Director with some private thoughts about breaking the impasse on acceptance of the Horizon project. We are waiting to see it on the screen...

Letter written 19 Oct 1998. Included a paper called "Acceptance testing: a framework for developing a new paradigm"

Suggests PO and BA give up termination rights rights on acceptance, in exchange for the option of being able to have more punitive service level agreements following the start of national rollout, should the system fail to meet acceptance criteria in Live Trial.

This proposal was not accepted.

We're back after a short break.

PC was involved in the acceptance threshold discussions into 1999 after the Benefits Agency withdrew.
An agreement on acceptance between PO and Fujitsu was made on 28 July 1999.

Basically says rollout will not happen if there are one or more high severity (category a) deficiencies or 20+ category b problems.

Parties agreed to appoint PC as an "expert" a to assist in resolving disputes relating to acceptance.

PC did not know about this. He was told his role was to work together to resolve conflict as to level of severity of each incident and resolution plan.

He only later found out he had the power to arbitrate (ie tell them what the solution was going to be), so this power was never exercised.

On 16 August 1999 - two months before rollout began, PC was given a "hotlist" of incidents.

PC says it was a list in which he had identified outstanding issues including those where there was a disagreement on severity.

CH looking at the list it looks as if there is disagreement on severity on almost every one.

PC agrees.

CH training course module - that's high severity, but ICL are saying that's closed...

[CH gives several more examples where Fujitsu say issue status is lower than PO thinks it is]

Note written to say contractually Fujitsu has three months to sort things out at its own expense. This is agreed as a "Limited Trial Period" to deal with the disputed faults. The workshop will be jointly chaired by Keith Bains from PO and Tony Oppenheim of ICL Pathway (Fujitsu)

In the agreement PC is involved to determine an issue between the parties, but otherwise he will be a facilitator and advisor to the parties to achieve a successful outcome. PC has not seen this document and was not told he could determine an intractable dispute.

He therefore just acted as a facilitator and advisor.

CH takes him back to the incident disputes and how they were resolved. There were three high severity incidents (as we have covered before). Shoddy training, accounting integrity problems...

... and system crashes/screen lock ups.

PC - "Pathway took the view this was pretty normal for IT - PCs lock up. PCs crash" PO took the view this would have a high impact on the business.

My recollection is that it was not resolved through our workshops. It was escalated.

PC discusses the "intermitted failures in the reconciliation process between the money in the till and what the system recorded and at the time I think it was believed the root cause was about a lack of integrity in transfers between Post Office and ICL Pathway systems."

CH where had this problem arisen?

PC at the TIP interface

CH what's that

PC the interface between the Post Office system for counting and the ICL Pathway interface and the database that recorded transactions in the system.

[And there we go, the basic problem -as 'Clint' the Fujitsu whistleblower told me for my book - was that Horizon COULD NOT ADD UP]

PC said this was "an inability on a consistent basis to reconcile Horizon data with cash data" [or - IT COULDN'T ADD UP]

CH was that a big problem?

PC yes

CH if a system wasn't producing accurate cash accounts, which appears to be what this incident was showing - did it not...

... call into account its very fitness for purpose.

PC Yes

CH How did Fujitsu propose to fix this problem

PC there were a number of proposals. I think the most worrying was that one of the fixes would not be implemented until 2000.

We go to a new doc which shows that Pathway are proposing a 3-level data integrity check to be implemented in Dec 1999. This needs to be "documented as a high level design including failure state analysis" PO to be involved in interactive walkthroughs during the development...

... of the design. PO does not want to accept rollout until this has been fixed.

Fujitsu/ICL/Pathway agree to come up with a rectifcation plan.

CH has been digging quite deep into Pathway's proposed solutions for fixing the accounting integrity problems which don't appear to work very well. PC is agreeing they are serious.

A viewer writes:

https://twitter.com/wierdhead/status/1583470493746634752

PC is asked his professional assessment of the robustness of the system

PC: "I think ... the Post Office... had accepted fuirther risk in agreeing to accept the system and release for rollout with the proposals from ICL. ... but I really don't know what happened after I left...

... [he left on 24 Sep 1999]" for further mitigation and bug testing.

CH has finished and Sam Stein KC has questions on behalf of some SPMs.

SS is taking him to a report PA consulting wrote which notes the skills required for many of the new senior posts are...

... in our opinion not thos we would have expexted to find esp re implementation management and contract and service management. Also points out there don't seem to be any plans to deal with this via external recruitment.

PC agrees this is a big problem.

[the review is from 97]

PC no doubt the PO had a shortfall in competencies at that time, but it did resource up - including the appointment of David Miller. But even in later stages they had a shortfall of technical competence that was capable of interrogating Pathway properly..

SS so you thought PO might not have the ability to understand what's happening with the system.

PC yes and also there was a readiness to accept the system within the PO which was not matched by an understanding...

... of how transformative the project was going to be.

[Flora Page for the SPMs has a q]

FP you mentioned PO was tkaing on more risk by taking acceptance of H in the way it was. Would you expect this risk to be registered in some way at management level?

PC yes but - I...

... don't know if it was.

FP should it have been?

PC I would have expected it to.

[And that concludes our session for today and the week]

A quick hot take:

We knew the car was going to crash, and we knew quite a bit about how it crashed, but we are now getting to watch it all happen in slow motion as the inquiry digs into the detail. The system did not work. It was a dog, but because the govt had...

... told the PO it had to happen, and because the PO wanted it to happen (or couldn't not let it happen), Fujitsu were able to offload the system into the PO network by coming up with enough "solutions" to the very major problems to allow the board to sign the project off.

Very little or no comment at all about how the Subpostmasters were going to be able to deal with this. And as they were the ones who shouldered a huge chunk of the risk with absolutely no direct representation at any level (save the unions who had very little visibility of the...

... problems before they manifested).

Right we're being kicked out. @Rebeccathomson_ and I are off to record a podcast about this week. We are powered by paid subscription to the "secret" email newsletter - please join the secret emailers if you can. Link in the next tweet...

@Rebeccathomson_ Here you go. There will be a newsletter round-up tonight or tomorrow - postofficescandal.uk/donate/

Thanks for reading. Have a good weekend. N

@threadreaderapp unroll please