Thread by @nickwallis

Good morning. I am at the #PostOfficeInquiry in London to watch Fujitsu whistleblower Richard Roll give evidence. I first heard Richard's voice on Alan Bates' cassette recorder in the Dobbie's garden centre café just off the A5 near Shrewsbury.

I first met Richard Roll in the Leathern Bottle pub just outside Wokingham in April 2015. What he told me was something the Post Office were explicitly denying - that Horizon errors could cause holes in Postmaster accounts, and that Fujitsu staff could go into those accounts...

... and change them at will.

By that stage the BBC had commission its first Panorama into the Post Office scandal. Mr Roll became a significant contributor to our programme.

His appearance and what he said in public, on the record, became a significant factor in...

... bringing the funders and legal teams together (added of course to the testimony and evidence of hundreds of Subpostmasters) to take on the Post Office in what became Bates v Post Office. Mr Roll was a witness for the Postmasters in that High Court litigation...

... and the judge (in finding for the Postmasters) called Mr Roll's evidence "very important".

You can read more about Mr Roll in my book The Great Post Office Scandal. He has been sworn in to the Inquiry, which means his witness statement is up:

postofficehorizoninquiry.org.uk/evidence/richa…

... and he is currently being asked about his early career.

Will tweet matters of interest in his testimony as we go.

You can watch him give evidence live here:

From Roll's witness statement re the Post Office Horizon system:

"My role in Fujitsu SSC was to provide third line support on the Horizon system. The system was hugely complex and included (amongst other things) Windows NT systems and Unix servers; asymmetric...

... digital subscriber line (ADSL), microwave and satellite communications systems; and software written in a variety of languages. Day-to-day tasks were many and varied - for example, we may have been asked to examine database transactions in...

... order to identify errors in the accounting system; examine computer programmes to try to identify bugs; investigate communications failures in the system; rectify failures in the overnight processing of the previous days...

... transactions; or liaise with engineers during the installation of new Post Office counters. We also generated reports for senior management and the Post Office, tested equipment, evaluated new hardware and software etc"

He goes on: "When I joined Fujitsu, I received the same basic training on the Horizon system as the SPM's [Subpostmasters], although as I didn't work in a Post Office I never became truly familiar with the system operation from a SPM's perspective....

... I don't recall there being any technical training as such —the system was so complex and the role so demanding that you had to be an expert in one or more fields when you started, you then picked it up as you went along."

Roll was at Fujitsu working on Horizon between 2001 and 2004:

"General security protocols were in place at the PO's. Secure passwords were required, which were not supposed to be shared but often were. There was a secure link from the...

... PO counter to Fujitsu's servers - the counters would only espond to requests originating from specific telephone numbers —and all data was encrypted. Additionally, the network was completely (logically) isolated form the internet, it had its own dedicated lines...

... and for resilience these were duplicated, with one to the east of the country and one to the west, I think (physically, the lines were probably shared with other BT customers). However, due to geographical and financial constraints, these lines merged in...

... London and both were routed along one of the Tube lines; I believe there was a minor train accident one day which unfortunately severed both cables and left the north of the country isolated from the south."

"We all had at least two PC's at Bracknell; one `open', which we used for emails, researching the internet etc, and one `secure' (completely isolated from the `open' system) for working on the Horizon system."

Mr Roll being asked about coming forward to raise the alarm. He told me several years ago it was the Inside Out broadcast in 2011, but he now can't remember how he became aware of it. Or who he contacted to offer his assistance (it was Alan Bates).

Roll does remember being on Panorama programme. Says he never thought of himself as a whistleblower until it was mentioned to him.

We are being taken to his witness statement to the High Court and his evidence about trying to find the source of discrepancies in Horizon code by downloading branch data. Sometimes scrolling through lines of code, sometimes printing the code off and using their own...

... filtering programmes they had written to strip out irrelevant data lines.

Having identified a problem, they would then look at the source code to see what might be causing the problem. If it was the source code...

... they would refer it to the developers and say "here's the problem, this is the source code. This is the source line. It's wrong. It should say minus this value when it says plus this value."

Says single errors were easy to identify, but multiple errors were really hard to spot and they would cause problems to "snowball".

Jason Beer KC is doing the questioning of Roll for the Inquiry. This is very different from the experience of watching Roll being cross-examined...

... at the High Court. It is in the spirit of inquiry, rather than an attempt to belittle or demolish his recollection and expertise, and is quite measured in its pace (given the High Court trials lasted five or six weeks and this Inquiry has as much time as it needs, that...

... is perhaps not surprising).

JB asks if he had to identify AND fix code problems.
RR says their "primary aim was to keep the system up and running so it worked, and so that Fujitsu didn't suffer any penalties"

"If we could identify problems in the code as we went along, that was a bonus."
JB asking about penalties for Fujitsu.
RR is hazy, but if a bank transfer didn't go through in 3 days there would be a small penalty, but can't remember if it was 10p or £10

Says the problem is when you've got 20-40,000 counters and a bug stops all transactions of a specific nature happening it magnifies the small fine into a massive one.
JB so you patched things up as you went along
RR It was widely accepted that the system was "crap" and needed...

... a re-write but that was never going to happen because of the money involved.

RR says that when an identification of an error was made it was passed to the developers, but he can't remember anyone in dev's name.

RR Says Subpostmasters might be told there was a fix going into the system to sort the problem affected their branch, but they were never specifically told it was a bug within Horizon.
JB pushes him on this
RR if we were talking to an SPM we'd say there was a problem...

... with their counter "data corruption" or something like that.
JB so it wasn't habitually fed back to them that there were coding errrors?
RR correct
JB was there an official line on this?
RR not sure
JB but the practice was not to tell them
RR correct

We go to RR's witness statement when he said:

"In my opinion the coding and development of the system did not meet my expectations of quality for a major software project; I considered it to be a very poor system that should never have been...

... deployed but I cannot be more specific than this."

JB asked if that related to his earlier "pithy" epithet he gave the Inquiry earlier (ie that it was "crap")
RR yes

JB takes RR to a section of his High Court WS "My recollection is that the software issues we were routinely encountering could, and did, cause financial discrepancies at branch level, including “shortfalls” being incorrectly shown on the Horizon system...

... If we were unable to find the cause of the discrepancy then this was reported up the chain and it was assumed that the postmaster was to blame."

RR struggles to remember examples.

Being asked about how problems came up. They could be raised by SPMs or they could find them themselves in SSC.

RR says he "thinks" Horizon improved whilst he was there in terms of coding standards and the documentation "but that's a distant memory"
JB asks why

... he has that recollection.
RR thinks people just became more professional
JB how reliable were teh Horizon cash accounts
RR "pretty ropey" I said "surely this should be rewritten" to my manager Mik Peach and he said "it's never going to happen"

JB and was that down to money as you said before
RR yes and we didn't have the staff
JB you say if you could not find a credible cause of a problem then it was assumed the SPM was to blame
RR correct
JB who?
RR PO management and Fujitsu

JB how did they come to this assumption? Who expressed it?
RR that was my feeling - if we couldn't find a problem in the code or the data then there is no problem. It must be the SPM.
JB did you understand action was being taken against SPMs.
RR no
JB were you aware people...

... were being prosecuted?
RR not at the time
JB were you aware anyone from SSC being asked to be a witness in a criminal prosecution at Kingston Crown Court - I am referring to Tracy Felstead.
RR no. can't remember.

JB you describe your work on Horizon as "firefighting"
RR yes it was quite hectic at times. "Sometimes there'd be a bit of a panic on... all hands on deck to try to fix a problem as soon as possible"

JB takes RR to his second High Court WS: "I do not recall Fujitsu carrying out any analysis of Transaction Corrections to try to identify if there may be an underlying software error....

... I also think it is wrong to say that software errors would occur uniformly across branches, as I explained in the example in para 10 above. My experience was that software errors occurred in very specific factual circumstances, which is why they...

... were challenging to identify and correct."

RR says he has no recollection of the PO ever asking SSC to analyse what they thought was a fault.

RR has not heard of Andy Dunks or knew of his job role. No liaison with him. Andy Dunks supplied the courts with witness statements

... and oral evidence in court about what RR's team had done in response to calls to the SSC by SPMs.
JB why was Dunks, a member of the customer service team, was selected to give evidence about what you and your team were doing at SSC
RR No
JB were you ever party to a discussion

... or did you ever hear why Dunks, the crypto key manager in the customer service team was giving evidence about what was happening in your team rather than someone from your team?
RR no
JB were you ever asked to give evidence
RR don't think so
JB if they had...

... would you have described all the problems you have told us about today
RR probably, yes
JB did you ever heard discussion about who had attended court on behalf of Fujitsu to give evidence about Horizon
RR no recollection of it

[so Fujitsu sent a useful idiot, Andy Dunks, along, to give evidence in court to vouch for Horizon, despite him knowing nothing of the problems SSC were dealing with. Feels a bit corrupt, that]

[in the same way the Post Office sent three useful idiots to Panorama to give an on the record statement about there being no remote access to Horizon, when it knew there was]

JB is now talking to RR about specific Horizon problems he raised in his High Court witness statements, including at least one where they attempted data recovery and could not retrieve it. And other occasion where RR found a hardware fault...

"An SPM seemed to be switching her laptop off before 6.00pm. Because the power was switched off the Laptop could not generate the end of day financial markers...

... consequently the accounts were not processed correctly which resulted in transaction data not being sent to the banks and utility companies so Fujitsu missed SLA deadlines. I was asked to investigate; the SPM (a very...

... experienced lady) insisted she was not turning the machine off but the log files on the counter showed that the laptop was being powered down. I arranged for the laptop to be swapped out and returned to Bracknell for testing and...

... found that when the screensaver button was pressed the power to the machine was switched off. When I disassembled the machine I discovered the fault —during the build the wires had been cross connected...

... I brought this to my manager's attention as I felt it should be investigated further; a few days later he called me over and informed me that the manager of the section that assembled the Laptops knew about the issue already as...

... one of his engineers had told him that he had inadvertently mis-wired several laptops that had been sent out to SPM's. I was told that no further action was to be taken and I was instructed to record the fault as no fault found or...

... something similar —the incident was hushed up without senior management or the Post Office being made aware of it. The faulty laptops remained in general circulation, but as none of the other SPM's used the screensaver...

... button regularly it did not cause a problem... however this raises questions regarding communication, honesty and transparency within and between departments within Fujitsu. For example, mistakes were made when releasing updates to the software and it is feasible...

... hat a programming error could have been rolled out to the estate and a fix rolled out a few days later, without anyone in the wider organisation being informed. In this scenario, if an SPM had problems with their accounts then...

... by the time SSC were asked to investigate the fault would have already been rectified so we would not have been able to duplicate the error. It would have seemed that the only logical explanation was that the SPM was to blame, with...

... potentially catastrophic consequences for that individual.'

JB asks RR who asked him to "hush up" the hardware issue.

RR "Mik Peach"

[we're on a break]

We're back

JB asks if he knows Anne Chambers.
RR no
JB recall her as working at SSC
RR yes.
JB her expertise?
RR v good on the accounting side and databases
JB did she have knowledge of the Horizon software or data integrity
RR can't remember

JB did you speak to her about giving evidence in court
RR no
JB do you remember if she was
RR no - I dont' remember anyone being selected

JB is being taken to Andy Dunks witness statement in the @CastletonLee case.

JB do you recognise the format of this call log?
RR no
JB were they printed or always on the screen
RR always on the screen

@CastletonLee JB says the call was opened on 25 Feb 2004 - within your time working in SSC. Underneath it says "Postmaster reporting that they are getting large discrepancies for the last few weeks"
JB wants RR's help re understanding the info...

@CastletonLee RR can't remember this type of data being set out like this.

JB "New call taken by Kujinder Bhachu: pm reporting that they are getting large discrepancies"

RR cannot remember much about this at all

@CastletonLee JB "NBSC have been in contact with the pm and can not find any user error"

RR can't remember NBSC

JB: "Checked tivoli events and health checked. Site is health checking okay"

RR says tivoli was a background monitor of activity

RR's memory of the time (18 to 19 years ago) not great here.

RR now discussing the audit trail when SSC engineers went into the code and how it could create error flags. The actual identifying codes they used for the engineers might cause errors. So they only used it when they knew it wouldn't cause problems. RR describes changing...

... Horizon data as it came into their servers and whilst it was on the servers to save going into branches. He also remembers blanking out SSC engineer identifiers, but can't remember why it was done - perhaps to change reference data parameters.

RR talking about hacking the Horizon system by creating a session in Riposte to insert transactional data. "We were doing it through the back door"
JB why describe it as a hack
RR it wasn't the way things were supposed to be done
JB why do it then
RR only way we could get the

... system back up and running. It was a workaround.
JB was it just you doing this?
RR no everyone was doing it. we had unrestricted access
JB did your managers know about this?
RR oh yes
JB how
RR my colleauges taught me how to do it
JB was this hack written down

RR don't know. I know there were problems when the auditors came in and found out we were doing it. At first nothing was written down. We were flying by the seat of our pants. It was a mess. Eventually the documentation was created.

RR taken to a document created by Fujitsu on 2 August 2002 which he wouldn't have seen. It's an audit report saying that SSC have unrestricted and un-audited privileged access to all systems including Post Office counter PCs.
JB is this true
RR yes
JB was it widely known

RR yes. within SSC.
JB but not outside
RR no - I doubt the PO would have known. It's only looking back I see how it's "pretty shocking" how much access we had

RR recalls hot fixes made to individual branches - sometimes they told a Subpostmaster to leave a computer on and not touch it. Other times they did it during lunch hours without telling SPMs and sometimes there was a way of logging on as an SPM (but he doesn't remember this...

... clearly and is concerned he might be confusing a memory with documentation he saw during the High Court trial).

[I've written a chapter on evidence and memory in my new book. Our memories are very unreliable fact-retention units and every time we retrieve a memory it changes without us knowing it]

[it is very easy for our memories to merge our direct experience of an event and a later experience of encountering the event through other means]

[we have stopped for lunch and will reconvene with Mr Roll at 1.45pm]

Okay we're back after lunch. Jason Beer KC [JB] asking the questions for the #PostOfficeInquiry and Richard Roll [RR], the Fujitsu whistleblower, answering. We have about an hour to go. Will tweet any items of interest. If you're new to this, please have a look at my website...

which is postofficescandal.uk - you can sign up to the blog posts (free) or the irregular secret email (£) newsletter if you want from there.

RR is being taken through an OCR form - Operational Correction Request (these came up in the Horizon trial). Now I know what an OCR is. This document creates a paper trail for changes to the Horizon system... the request, who signed it off, whether it was tested.

RR has no recollection of this OCR document.
JB did you have to fill something like this out when you made changes
RR no memory of this document before
[it's dated 2001]

JB takes RR through the 29 Jan 2001 protocols for making changes to Horizon system. They seem thorough and sensible (though, of course, I know nothing about this sort of thing).
JB calls it "very involved" and "complicated"
RR has no recollection of following these protocols.

He says he isn't sure whether that means he didn't - he just can't remember. His recollection is that he doesn't think they followed those protocols, but just can't remember for sure.

JB has finished his questions for the day. Inquiry breaks for 5 mins - there will be questions from barristers representing the Subpostmasters after the break. Looks like we'll be wrapped up by 2.30pm

It looks like Sam Stein KC will be first with the questions for RR.

SS you discussed with JB what can happen when a system used by an SPM has a dodgy on-off button. That could and did lead to loss of data integrity.
RR yes
SS can we assume losses of pwer would have the same...

... effect.
RR it would have that potential.
SS so potential for that when the power goes - and connectivity issues. Could that lose data?
RR yes but the data would hopefully still be on the Computer in the branch.

SS would it always
RR if there was a power failure there is always the potential hardware damage to the disk or boards. If taht made the computer inoperable. if you can't recover data from the disk, without a paper trial, you would not know what happened.

SS was this ever explained to SPMs.
RR I don't remember doing so and I'm pretty sure I didn't. I don't know if anyone else did.

Flora Page FP now asking questions.
FP wants to know about tivoli - were there any routine or processes around it
RR can't remember

JB has one last q

JB raises the third witness statement RR said he did for the High Court case.
JB thinks it might be an AMENDED second WS rather than a third. Thinks this might be the cause of some earlier confusion. Asking him about it now.

Particularly the hardware problem which caused a hard power off to the unit. Clarifies RR was told not to put details about the problem in his report at the time.
RR is thanked by Sir Wyn Williams for assisting the inquiry and the many years of assistance he has given...

... in previous years.

This is the last session until 27 April when compensation issues will once more be addressed, including, for the first time (if not then resolved) tax issues. I can't help but think this is a direct result of @danneidle's timely intervention on the matter.

@DanNeidle For more on this, have a listen to our most recent podcast: postofficescandal.uk/post/pushing-t…

There will be a podcast about this last three week section of the inquiry, but I can't record it till Monday. Hope to get a couple of guests on to chew it all over...

... until then, please do have a look through the postofficescandal.uk website.

Thanks for reading.

@threadreaderapp unroll pls