Thread by @nickwallis

Okay back in court 26 of the High Court’s Rolls Building for the final day of claimant witness cross-examination in the Horizon trial, part of the Bates and others v Post Office group litigation.

Follow this thread for live tweets.

I spent this morning updating the Horizon trial menu to ensure everything that happened yesterday is in the same place including my write-up, the transcript, witness statements and bonus material.
postofficetrial.com/2019/03/horizo…
#postofficetrial

Okay court is sitting. Richard Roll - former Fujitsu Horizon third tier support employee is in continuing his evidence in the box. This is he outside court yesterday.

PO QC (Anthony de Garr Robinson) is asking Richard Roll about some definititions - software, data, coding etc. He has asked RR to look at a witness statement from a Mr Godeseth.
RR what’s your questions
POQC Mr Godeseth is drawing a distinction between transactional data...

and data corruption. That’s not transaction data is it?
RR It could be transaction data that could be corrupted.
POQC I mean transaction data which could be recording a transaction data in branch. Not the data attached to the transaction. That data won’t affect transactions...

… in branch will they?
RR I don’t know.

They agree transaction data. And that the other sort of data is going to be called operational data.

POQC having established nomenclature let’s go to your second WS, WS2. p14...

"I do not believe that it is realistic to say that all software errors would have been picked up by
the processes which were in place, or that the likelihood of software errors staying disguised
as human errors …"

“… was very small (as Dr. Worden says at his paragraph 168). I believe there
were likely many cases where subpostmasters would have been held responsible for problems
which had not at the time been identified as …"

“… software errors, either because they could not
identify the problem and did not pursue these with Post Office or Fujitsu, or because when
they were raised we (Fujitsu) were ultimately unable to identify the problem at the time. .."

“… There
were, for example, bugs which created discrepancies such as transactions which subtracted
rather than added values, and it was only with experience and investigation …"

“… that we were able
to identify these types of problem affecting a particular set of accounts. …"

“…. I do not believe that
we were successful in identifying and correcting all problems. My experience of contacting
subpostmasters is that they were generally very frustrated by the time we called them because
…"

“… they had been unable to find the source of the problem and had become quite cross about the
perceived lack of support they had received (often by the time the SSC became involved, two
or three days had passed)."

In the time it’s taken me to paste that RR has accepted that what he wrote here is not borne out by experience and conceded that it is largely theoretical AND that cyclical redundancy checks would have picked it up anyway. We’ve move on...

Now being asked about WS1 point 8: "Any
errors made by sub-postmasters would be relatively easy to identify, and would normally
be picked up by 1st or 2nd line support. …"

“… If an error was referred to us then it was extremely
unlikely to be due to a mistake made by a postmaster; the vast majority of errors I dealt
with were due to coding errors or data corruption."

POQC shows him evidence from Mr Godeseth which contradicts this and asks if he accepts it?
RR from my memory no, but looking at this [Mr G’s WS] yes.

We move on to another part of RR’s WS "My recollection is that the software issues we were routinely encountering could, and did,
cause financial discrepancies at branch level, including “shortfalls” being incorrectly
shown on the Horizon system. .."

“… If we were unable to find the cause of the discrepancy
then this was reported up the chain and it was assumed that the postmaster was to
blame."

POQC is now looking at Mr Parker’s WS. He says that errors are referred up by first and second line support as Horizon errors, but proper investigation often shows they’re not. It’s just bad identification from 1st and 2nd line support. Do you accept that?
RR yes

POQC I am seeking to suggest to you as a matter of common sense that the majority of issues reported are user issues, not Horizon issues. Hence someone working in a support environment would always assum user error as first hypothesis.

POQC if it’s not then we look at the evidence to see what’s caused it.

POQC so Fujitsu work very carefully to find out exactly what happened.
RR usually - but I know there were times that the time constraints put on us to dismiss problems which made me feel uncomfortable.
POQC Remember any examples?
RR no
POQC I say that didn’t happen

RR I remember feeling it did, but it was a long time ago.
POQC everyone at SSC (Fujitsu Support Service Centre) including you did a professional job
RR we tried to

We go back to one of RR’s WS "I believe there
were likely many cases where subpostmasters would have been held responsible for problems
which had not at the time been identified as software errors, either…"

“… because they could not
identify the problem and did not pursue these with Post Office or Fujitsu, or because when
they were raised we (Fujitsu) were ultimately unable to identify the problem at the time."

POQC how can you say “many”? Was it many?
RR it wasn’t just me - it was comments from other people on the team about how busy they were
POQC speculation then?
RR yes
POQC your recollection is very hazy
RR yes
POQC particularly here

RR yes, but the feeling persists that somethings slipped though

POQC how long were you given to work on a solution? ten minutes?
RR might have been 8 or 10 hours - or a couple of days. So if we didn’t get a solution in a couple of days we might have to leave it
POQC did you bring your reservations to the attention of your manager?

RR to Mr PEach yes
POQC but not Mr PArker your deputy manager
RR no
POQC and Mr Peach is not giving evidence. Right.

[we got to par 15 of Mr Parker’s 2nd WS which I don’t have]

If there was a recovery problem and F cannot id the cause it was probably because the operator didn’t follow proper recovery procedures.
SSC investigated everything and even if we couldn’t determine the route cause

root cause, sorry. There would be documentation.

RR if we couldn’t determine root cause it would be put back on the SPM as their error.

POQC what about this “if there was a coding issue SSC would have found it” Do you accept that?
RR no - if we couldn’t find it...

… it would be passed onto development.
POQC but that would be very unlikely
RR unlikely
POQC so in most cases 3rd line support would spot the problem. If not it would be documented and then passed up to 4th line support who would give the prob a thorough investigation.

RR usually, except if we were under pressure to provide an answer to our manager who himself was under pressure

POQC I put it to you if there is a software bug it would be pursued by SSC either 3rd or 4th tier support until the suspicion had been fully exhausted - do you accept that?
RR no. If we couldnt find the problem it may have been closed. It would have been passed back to the PO..

… and I assume the blame would be put on the Postmaster.
POQC but you don’t know that.

POQC let’s say you think there’s a coding problem. You would not close down a PEAK or problem - you would pass it on, wouldn’t you?
RR yes
POQC always?
RR if we thought it was a coding...

… issues yes.
POQC any problem surely
RR not necessarily. some of the areas we had a lot of control over… don’t remember a lot about it - but there were times we coudn’t find a problem with the message store, so there were times it would be passed back...

… to the post office… I think I’m getting sidetracked.
POQC if you had a suspicion of a coding error you absolutely would never close it down and pass it back to the Subpostmaster
RR agrees

RR sometimes we would not have time to suspect a coding error and because we didn’t have time to investigate an error
POQC are you suggesting that sometimes you didn’t investigate properly?
RR yes
POQC that would be deeply unprofessional
RR I know. It happened.

POQC why wasn’t it in your WS
RR it is
POQC who told you to do this?
RR mr peach
POQC what is the exact nature of this problem?
RR explains the power boot problem which is in his WS
postofficetrial.com/2019/03/horizo…

RR says it was a build error they had been sent out to the estate realised their mistake but hadn’t told anyone. I pinpointed the problem, spoke to Mr Peach I was asked not to put any more info on this on the record and that it was being dealt with internally.

POQC I was going to ask you about this. But I asked are you if there was a problem which required further investigation and you were told to close it down. Do you remember that?
RR [long pause…] sorry what was it you said exactly before that.
J tells POQC where it is...

… in the transcript. We are about to read it:

“You say on occasion I was instructed to close a particular call which was not software related in order to close it down."

POQC my question was whether you needed to do further investigation but you were told not to. you said on occasion. [we are going back over the transcript]
POQC making the poin that the example he was given by RR was that he did do a thorough investigation.
RR agrees

RR gives a different example wher ehe was given until lunchtime to find something out and if not it would be closed down.
POQC but you are suggesting that investigations weren’’t completed. are you suggesting that now?
RR no
POQC and no team member would ever do that

RR the feeling that I have is that on occasion we could have done more. It’s my feeling from 19 years ago.
POQC could I suggest to you that feeling only happened on relatively rare occasions
RR agrees. but that is the feeling I had then.
[we move onto next part of Mr Parker’s WS]

[we can’t see this WS]
POQC do you agree analytical procedures at F were thorough. Do you agree?
RR generally yes

Going to RR’s WS2 par 9: "At paragraph 156, Dr. Worden describes zero sum baskets, other branch actions being zero
sum, and transactional integrity. I agree that the system was designed with these intentions
in mind, …"

“… but there were limitations and errors in the system. Data corruption and glitches
sometimes meant that transactions were not zero sum. I recall on more than one occasion
where subpostmasters had problems with a …"

“… deficit showing in their accounts, and then as a
result of working through a process to try to resolve it, the deficit doubled. .."

“… Sometimes we
found the source of the problem as a known bug (in the KEL) and we could resolve the
problem, but we were not always able to find or understand the cause."

POQC but transactions are never zero-sum are they? it’s the basket that’s zero-sum, correct?
RR I think you’re right there.

[POQC asks him to read PO IT Expert evidence which we don’t have yet - it’s the paragraph RR takes issue with.]

POQC having read the relevant paragraph you say you agree the system was designed with these intentions in mind… you then go on to give an example: “I recall..

[ he repeats the WS I’ve pasted recently]
POQC this is not an example of limitation or error in transactional...

… integrity.
RR No it’s not - I must have misread what Dr Worden said.
POQC so what Dr Worden said is correct and you don’t have any contra-examples to offer.
RR yes I misunderstood.

J how should I regard your evidence
RR I was trying to point out during an error at times it actually double the error
J carry on Mr dGR

[I think he’s scored one or two hits but generally RR is getting shredded]

We’re now on to par 10 of RR’s WS2. RR describes a serious and what he describes as “fairly common” problem which was “impossible” to fix. It was common knowledge and so Fujitsu had to establish workarounds. It’s a good paragraph worth reading. postofficetrial.com/2019/03/horizo…

POQC so you did develop a fix and it didn’t affect the branch balance accounts, would it?
RR no.

Now onto Par 11 of RR’s WS2. "I do recall that problems sometimes arose after subpostmasters used the recovery process
and that this was a not uncommon problem which affected even experienced subpostmasters."

"This might suggest that there was a problem with the recovery process itself, or at least that it
was not as straightforward as it should have been.“

POQC - “might?” - that’s a mealy-mouthed phrase. Do you believe there was a problem which affected branch accounts?

RR no
POQC would it be fair to say you are speculating as to the poss of a prob. you’re not saying there was likely to be one
RR no
POQC are you saying it should ahve been investigated?
RR No

POQC - so why are you even saying this?
RR there were problems with recovery that we got involved with. At times it was the number that came in. It made me feel there was something not quite right with it.

POQC so problems came into Fujitsu re recovery process and they were investigated and investigated thoroughly
RR yes
POQC and if there’s a problem it’s likely to be identified
RR yes

POQC when an investigation is carried out - your job at Fujitsu was looking for what could have caused a problem so for reconciliation etc when these investigations were done the investigators could not find a Horizon problem
RR yes
POQC so why now say there might have been one?

RR I was not happy with the way they were dealt with. I can’t remember, but I was uneasy with the way it was done.
POQC did you do all the investigations?
RR no

POQC are you saying your colleagues didn’t investigate properly
RR no
POQC so what are you saying?
RR that the time pressure we were under… I felt...

POQC corrects himself - he was calling them reconciliations not recovery procedure.

[we are still on recovery processes]

RR my recollection that the majority of the problems with recovery were found but sometimes we couldn’t find the problem
POQC so there was a very thorough investigation and you couldn’t figure out a way in which Horizon could have been responsible. Correct?
RR yes

[we move on to TCs in RR’s WS2]
"I do not recall Fujitsu carrying out any
analysis of Transaction Corrections to try to identify if there may be an underlying software
error."

POQC why are you talking about TCs when they didn’t exist when you were at Fujitsu?
RR I don’t know - wasn’t there?
POQC don’t you remember?
RR we corrected transactions
POQC at the time you worked at Fujitsu TCs did not exist. You had error notices.
RR accepts he...

… doesn’t know the specific definition of a TC.
RR I misread it.
POQC TCs are defined in Dr Worden’s report. Did you read the definition?
RR no
POQC we corrected transactions

J asks RR to finish his sentences.

Now onto par 19 of RR’s WS2 - postofficetrial.com/2019/03/horizo…
POQC asking if he was personally involved
RR yes in one or two cases
POQC but this is not being told not to work on something which was causing a problem in branch accounts
RR agrees

POQC asks for an example, even a vague one
RR can’t
POQC well what about the workflow?

RR we’d find out what the problem was, what the impact was, how easy it was to find the problem and that info would be passed up the chain to see if it warranted a software fix or if it would be faster and as reliable to come up with a...

POQC so you have a toss up between writing the code to stop the problem or coming up with a workaround to fix it
RR yep
J do both methods involve changing the code?
RR well solution one you change the Horizon code...

… solution two you fix the problme by writing a batch programme to sit on the server so the Horizon code isn’t changed but new programme is introduced.

[we are onto par 13 of RR’s WS1 about the pressure they were under to make fixes to software bugs or face financial penalties under their service level agreement]
POQC there’s a danger of a false impression being given by your evidence. You’re not talking about pressure...

… on areas of problems that could affect branch accounts.
RR not mainly no.
POQC my concern is the process by which the work that was done in Fujitsu might have an impact on branch accounts and so when one reads your WS, the natural reaction is to think that’s what youre talking

about. But I think you’re not saying there was a perception in Fujitsu that they had to do a quick job and not a proper job.
RR there was pressure if you couldn’t pinpoint the fault in the counter then there was pressure put on you you were asked how it was going and needed...

an answer.
POQC did the SLA stop Fujitsu investigating bugs which might affect branch accounts.
RR we had to investigate within the timescale and we knew there was an SLA
POQC was there a perception that the SLA stopped you from doing your job properly
RR I felt that.

POQC how strong are your recollections?
RR vague

POQC were you aware of any contractual pressure on Fujitsu to speed up the investigation procedure?
RR can’t remember

RR now basically agreeing with all the Post Office evidence after giving them ponderous thought.
He’s become quite a good witness for the Post Office!

Sorry evidence from Mr Parker and questions from the POQC.

Now agreeing one paragraph in his evidence was office gossip.

RR recollection is that the majority of the team were under pressure

POQC and how did that affect them?

RR I don’t know.

Now onto budget restrictions:

POQC what do you know about Fujitsu’s budget?
RR explains ICL Fujitsu merger and how the Horizon department was profitable.
POQC and because the H dept was profitable there were no redundancies when the merger happened

POQC and because the Horizon department worked so well it would be odd to cut corners there wouldn’t it? It would be worse in the long run to try to do so.
RR agrees

RR is now being asked about his statement "I would estimate
that I was involved with a hardware failure on average at least once a month. These problems
could and did affect branch accounts.”
POQCC how?
RR I can’t remember.

Sorry tweetdeck is slowing down. going to reboot

we are still talking about hardware failure.
POQC you say "I recall there were also PIN pad problems which caused issues in branches, and problems with…"

“…
other peripheral devices such as keyboards which only occurred intermittently, although I
cannot recall the specific detail of these now.”
POQC would these affect branch accounts?
RR no

We are going into a 4 March 2004 problem that RR dealt with. They have the PEAK and KEL of the case.

The problem is: evidence from event log shows power is being switched off shortly before the SPMR logs on.
This is RR’s example of the hardware rigs which were faulty.

[POQC is reading from RR’s report of the time which was discussed in court earlier]

12 days after RR discovered problem he asks for SPM Horizon terminal to be brought to him in Bracknell.

This happens.

Looking at the report - test carried out on screen power out switch - works fine - no further action required. This suggests it was fine?

RR no that’s not right. I fixed it first and then it worked, so then NFA.

POQC Okay and this was referred to the hardware team and that’s the point at which you found out this was a problem with a batch of laptops. Who told you this?
RR I can’t remember.

POQC and you were told that this was a known problem and not to do anything more about it.
RR yep
POQC how many laptops?
RR don’t know
POQC do you think that after this has been discovered by your team it would have been dealt with properly?
RR don’t know

POQC you are making a serious allegation that you were told by Mik Peach from Fujitsu not to do anything more about it. You are saying he took the view that the hardware team having let bad laptops go into circulation, he let that happen.
RR yep
POQC why? did he tell you why?

RR no
POQC why on earth would he do that?
RR he might have had a mate who was the manager of the hardware dept and it was his mate's shout to do something about it.
POQC is that speculation?

J you invited him to make the speculation by asking what conceivable reason Mr Peach might have to do this.

RR i was basing my presumption, which was all it was, was that recalling the laptops would expose the person, but actually the way you say it it would make more sense to get the laptops out of circulation rather than leave them in the estate.

We break for lunch

@threadreaderapp unroll pls