Thread by @nickwallis

Good morning. We are in court 26 of the High Court’s Rolls Building for the continuation of the Horizon trial, the second trial of the Bates and others v Post Office class action. Live tweet thread to follow. Here are the Post Office barristers arriving for work.
#postofficetrial

For background to this trial/class action please take a look at postofficetrial.com

Please note everything I tweet is a summary or description of what is happening/being said. Nothing is a direct quote unless it is in “direct quotes"

The judge is sitting and going through some admin. He says he is hoping to hand down what will be judgment no. 5 by the end of the day. It relates to his refusal to allow the Post Office to appeal the Common Issues trial judgment.

Jason Peter Coyne, the claimants’ IT expert is being sworn in. He is in the first minute of what will be four very long days in the witness box.

I’ve never met Jason Coyne, but I’ve seen him around during the first part of this trial. He’s got a loud, confident voice and if I had to guess I’d say he’s from somewhere in the North West of England.

My colleague on the press bench, @Karlfl from @ComputerWeekly thinks its closer to the Midlands.

This is not the most important issue at stake.

@Karlfl @ComputerWeekly In the public "gallery" - @RonRwarming (MD of Second Sight) and @JoHamil73963257 (claimant).

@JoHamil73963257 Mr Coyne is being cross-examined by Anthony de Garr Robinson (on the right in the photo at the top of this thread), reading from documents I don’t yet have. Mr Coyne has been more or less monosyllabic so far - he’s being asked to acknowledge things rather than give answers.

I am going to call Mr de Garr Robinson PO (he is the Post Office barrister) and Jason Coyne JC.

There is some back and forth between PO and JC about what the critical point of JC’s investigation is.

PO: important questions are what risk did a claimant face of a Horizon error causing a shortfall during a period when an accounting period is disputed.
JC disagrees - says he worked off the list of Horizon issues.
PO clarifies to say it’s the main reason we’re here.

JC: it’s a reason we are here
PO: but is the whole point of this for the court to make a decision as to the likelihood of it causing problems for Subpostmasters
JC: yes but I wasn’t focused on individual accounts
PO: but you didn’t look at the whole of Horizon’s lifetime, did you

JC: I disagree with that. I think you do have to look at the lifetime of H
PO: But the crucial question is to what extent would those bugs be likely to affect branch accounts
JC agree
PO: and countermeasures used to correct those bugs, either immediately or some time later

Judge bringing up “countermeasures” as it has technical meaning in IT world. asks PO what exactly he means by “countermeasures”.
PO goes beyond technical terms to suggest it uses human systems of countermeasures as well as coding ones.
JC understands

PO do you accept controls, measures or countermeasures stop bugs happening in the first place
JC explains how this would happen in a computer system - testing regimes to flush them out, defect triggers etc
PO impossible to have a testing regime that picks them all up?
JC agrees

… to a point. There are always going to be unknown bugs in a system that goes live. They might be discovered in days or weeks.
PO and there are ways of dealing with this - defensive programming
JC defensive progamming typically stops a bug from occurring - not done post hoc

PO but its also to stop a bug propogating
JC agrees
PO so we have measures to stop bugs going live, to stop them spreading and to correct them. but no system is perfect
JC agrees
PO and we also have systems in place to deal with bugs when they affect the system

JC agrees.
PO Many of those countermeasures pick up and correct user errors?
JC yes. you can design the software to correct user errors. gives example in H of high value cheques
PO so all of these measures help make a system robust.

JC yes. you want a system to fail safely.
PO so robustness is the very concept which underlies all the issues we’ve been discussing for the last half hour
JC yes
[we go to a joint statement between the two IT experts. it is an agreed definition of robustness]

Essentially we are all agreeing that robustness is not perfect, but that it has a capacity for dealing well with stuff when it goes wrong, including bugs in the system.

PO quoting from JC report in which JC says he agrees with PO IT expert that H is “relatively robust”.
more from the joint report: we agree robust does not mean infallible and so H will continue to suffer faults - robustness is how H reacts to those events.

PO continues to read - line about no countermeasures being perfect either.
JC agrees.
[there is a lot of agreeing going on]

PO reads Dr Worden [PO IT expert]’s definition of robustness in which he says JC’s definition is not adequate. Dr W extends the definition to several dimensions of robustness.
JC doesn’t get reference to dimensions. Says they aren’t defined.

PO critical q is how large remaining risks are in Horizon after countermeasures have been brought in?
JC - no it’s not the critical q - it’s important
PO so when determining whether a system is robust you don’t agree quantifying the remaining risk after countermeasures...

… is essential.
JC I’m saying you can’t quantify that risk.
PO doesn’t it follow that the ultimate question wrt to robustness is how well the risks in a system are guarded against.
JC yes and typically that analysis is conducted in the design process of the system from the start

JC that is different from having system already in place and looking back at it and trying to decide what its level of robustness was
PO so you can’t assess its success?
JC what I’m saying is that you assess it beforehand and then you build it. it is subjective. you can’t...

perfectly quantify risk.
PO you’re a Prince 2 practitioner, right?
JC yep
PO and you specifically quantify those risks - there are graphs and charts which measure risk
JC yes it’s too simplistic for what we’re talking about here

PO I put it to you it’s precisely the system that is employed here to measure risk
JC it’s too simplistic, you can’t scale it up and look at it retrospectively and then come to a conclusion
PO you’re talking about two concepts - measuring risk and retrospective review...

… asks him to agree it must be possible to assess risk
JC I’m not saying you can’t use Prince 2 to assess risk.
PO I’m saying that’s exactly what happens

JC says looking back at a project helps you look at the context going forward.
PO invariably you will be looking at past performance to inform you about a news system
JC yes
PO I’m very grateful.
[That appears to conclude this part of the cross examination]

I now have both of Jason Coyne’s reports. One is 243 pages long, the other is 273 pages long. I’ll upload them both to postofficetrial.com over lunch. There are lots of diagrams.

PO so your considered view of the Horizon system is that it is “relatively robust”
JC I said that in my first report, and then I took a view in my second report it was less robust than I thought
PO but overall you still say its relatively robust
JC yep

PO and you’ve considered it in the context of other systems
JC yes
PO I think you’re saying it has a good level of robustness compared with other systems
JC yes
PO top quartile?
JC no
PO top half?
JC I don’t want to be drawn

PO but if I say I go running with some people and compared to them I’m relatively fast, aren’t I saying I’m faster than most of them?
JC [laughs]. adds context to say yes we are talking about Horizon as it is today - to get to where we are today is quite a long journey.

PO I understand your point about that, but lets just deal with the issue of comparison. are you saying it is robust compared to other systems?
JC I’ve worked on military and healthcare systems which are life critical. H is nowhere near as robust as them. But it is robust...

… in terms of retail, some elements of banking, point of sale systems, stock control.
PO large businessess with lots of users?
JC yes - a similar scale.

So we have established that JC thinks Horizon is relatively robust compared to similar sized IT systems in the same sectors.

[we move on to the subject of acronyms and how well used the ones used by Dr Worden are in the wider IT industry]

PO notes JC points out in his report that some of Dr Worden's acronyms are not industry standard, even tho DrW notes this himself. Asks him why he felt the need to point this out when DrW had done so himself. Twice.
JC doesn’t know

PO asks him why he keeps chucking in the word “aspirations” to his evidence. Asks him if he’s subtly trying to introduce a theme here.
JC says he isn't

[we’re back to robustness]
PO a key element of the robustness question is how H limits the extent of impact on branch accounts
JC agrees
PO so informing a judgment on robustness - you see if it had an impact on branch accounts and then you look at how it can be fixed

JC no you need to look at whether it did not whether it would have
PO well in some cases it would be blindingly obvious. [Brings up example of remming in] - do you need to examine each individual case to see what the problem was
JC well not every time when there are typical fixes

JC but when there is a problem which isn’t typical you need to look at the specific information about what caused it.
[more chat]
PO but essentially your considered opinion is that Horizon is robust
JC relatively robust, yes.
[we move on]

PO asks about his judgment on robustness with regards to a section of his report
JC says that wrt reconciliation data which hasn’t be disclosed, he can’t close the loop. he says if it is disclosed it may help support his opinion that Horizon is relatively robust or not

PO that doesn’t make any sense. you have made a judgment about the robustness of H, are you now saying you can’t
JC I don’t see what’s wrong with what I said. There must be documents which exist which show what happens, it hasn’t been disclosed to us.

[it appears there is an issue here of disclosure. JC and apparently Fujitsu don’t know what the Post Office do wrt to issues of reconciliation]
PO but you don’t think the PO do nothing do you?
JC no I think they probably do do something. I just don’t know what it is.

PO which is how you form the view it is relatively robust.
JC yes
[judge rises to give transcribers a break]
@Karlfl accepts Mr Coyne is from the North West of England. @krug79 has pointed out his company is based in Preston.

@Karlfl @krug79 So in my preview on my blog (postofficetrial.com/2019/06/horizo…), I said we already knew the claimants thought H was “relatively robust” and PO though it was “robust”...

@Karlfl @krug79 … I suggested the discussion would be about what “relatively” means. so far all we have had is an hour of cross-examination to get to the point where we have established JC thinks H is "relatively robust”.

Hmm.

Also I don’t like calling De Garr Robinson PO as it confuses...

… when I start to talk about the PO itself, so I’ll call him DG from here on in.

Okay we’re back for the second session of the day.
Judge has a problem with his computer screens. He is soldiering on though.

DG now back on robustness and how banks and retailers, large private and public sector IT systems need it.
DG they can only afford for a tiny proportion to suffer from lasting errors - air traffic control etc can have no errors, but the kind of systems we’re...

… talking about still must have a low tolerance for errors.
JC yes - tiny fractions
DG so if there are lasting or serious errors it could be a threat to their very survival
JC yes
DG so when you say H is relatively robust you are saying in the overwhelming majority of cases...

… it works reliably?
JC yes
DG and when it doesn’t it’s only a tiny tiny proportion
JC yes
DG and even those get picked up?
JC not all of them.
DG that’s quite important
JC yes
DG well why didn’t you spell that out in your reports?

JC I thought I did
DG you tried to give the impression H is not reliable.
JC I’m not sure I did.
DG well let’s have a look at your reports, shall we…?
[we go to the report]

DG quotes: "The sheer volume of Known Error Logs and reconciliation reports confirm the wide-ranging extent of the impact of such bugs/errors/defects...

… This evidence demonstrates that such bugs/errors/defects would undermine the reliability of the Horizon system to accurately process and record transactions.”
DG is that fair
JC accepts his view changed as this statement was based on incomplete view

DG so the more evidence you got, the more you realised it was more robust
JC says no - the more he saw, the more defects were revealed - his concerns changed. there were large elements of unreliability, but that doesn’t detract from its overall relatively robustness

DG - one of us is confused and it might be me. I said in par 3.1 you gave an impression H was unreliable. Then you said no the more you learned the more unreliable you found it, and yet you then upgraded your view to it being relatively robust.

JC what I said there was correct - the errors we knew about would undermine its reliability.
DG let me quote this - “sheer volume” which have “branch impacts”? you appear to be saying there is a large volume of KELs (errors) which had branch impacts. Did you mean this?

JC you need to understand the difference between KELS and PEAKS. A KEL is a log of PEAKS. A PEAK is an error that sets up a KEL so the KEL can track PEAKS around a known problem. Of the PEAKS that had been examined there were those which had an impact on branch accounts.

DG let me ask my question again. you are referring to a large volume confirming a wide-ranging extent of bugs? was that the case?
JC yes
DG how many?
JC don’t know precisely. some listed in my report.
DG but you are talking about wide-ranging impact on branch accounts?
JC yes

DG in the whole of this report you refer to very few KELS which confirm the existence of branch impact
JC there’s a table at the back of the report which lists them.
DG where?
JC appendix G, p200
[judge gives electronic reference]

DG there are 29 bugs in that table. ID’d by Dr Worden or you.
JC Dr Worden ID’d the three the Post Office already disclosed.
DG I think he identified nine and that was agreed in the joint report.
JC don’t know
DG but Mr Coyne very few of the KELs you identified turned up in...

… your report. There’s no large volume here.
JC but the KELS don’t relate to the branch. The PEAKS do.
DG you just didn’t see the volume you say you did
JC it would be better if that paragraph referred to PEAKS and KELS
DG you’re not saying the pinpad errors ID’d had...

… a branch impact?
JC I think they did yes.
DG there’s no reference to it in your second report
JC what about the defect where the pinpad remembered the previous user’s account - the all pay defect?
DG well we can look at that later.
[takes him back to par 3.1]

DG wouldn’t it be fair to say a KEL will explain the impact of a problem - the practical consequence.
JC yes but not the specific impact
DG but generally speaking the KEL will tell you IF there is a branch impact
JC yes
DG so going back to 3.1 you are suggesting...

… there were a large number of reconciliation reports which caused branch impacts.
JC yes
DG what is a reconciliation report
JC [fumbles to an answer]
DG let me help - it’s when a transaction is recorded in branch and then transmitted by the TPS system - which does what?

JC decides where the payment goes.
DG isn’t it that it transmits it to the PO back office for their systems to decided how they are pushed around or in and out of the system
JC yes it harvest the data

[more discussion - I am currently lost in a large computer mainframe following hypothetical transactions]

DG so if they don’t reconcile that’s when there’s a problem - timestamps, values, a delay
JC yep basically something has gone wrong and something has to happen to correct...

… the process.
DG and it happens automatically
JC not every time
DG there are 3m transactions - human beings don’t do that
JC only when it’s working as it should
DG this generates a report of reconciliation exceptions
JC yes

DG and sometimes there will be some human involvement
JC yes
DG but the report will explain what has happened - automated
JC that’s the number I was referring to - many thousands
DG but why does that confirm the impact of many thousands of bugs

JC I’ve designed many systems and I’ve never seen tens of thousands of reconciliations that need human intervention. That suggests a lot is going wrong.
DG but there are many explanations for reconciliation exceptions which have nothing to do with problems causing an...

… impact on branch accounts.
JC correct
DG but it doesn’t suggest that any of these reconciliations have an impact, does it?
JC no but this level of manual reconciliation suggests something is going wrong
DG how do you quantify that

JC I’m comparing it to other systems I’ve worked on - I’ve never seen that many.
DG but you were saying in your report these reconciliation errors confirm an impact on branch accounts. It doesn’t do anything of the sort.
JC agrees. “it suggests, rather than confirms” he concedes

DG pulling apart his statement that there were wide-ranging bugs then says its relatively robust
JC robust doesn’t mean bugs won’t have a wide-ranging impact
DG weren’t you saying that there is only a tiny number to me earlier?
JC no I didnt’ use that word

DG really? earlier you said there was a tiny number
JC no I probably said a small fraction
[Patrick Green QC for the claimants stands up and says the evidence is a small fraction NOT tiny]
[crucial difference, obvs]

DG but the point is in this report you were saying there were all these problems and the next page you’re saying its relatively robust.
JC the two are not exclusive.

[we are now on a joint statement which I don’t have]
DG grilling him on an unequivocal judgment JC made in the joint statement about errors in the Horizon system based on limited data. Asks how he can be certain.
JC says it is a strong indication

DG you’d only seen the KELS which you say are helpful but incomplete, but you were able to make a judgment on H based on this?
JC yes because some of the KELS referred to PEAKS
DG then you got the PEAKS
JC some of the PEAKS
DG a large number - 250-odd thousand

DG and am I right in saying that you saw something in those PEAKS that made you change your mind
JC yes
DG so was your judgment here on p9 rather hasty?
JC yes - I should not have expressed that opinion there.
DG why didn’t you explain your change of mind in the report?

DG you see in Joint statement 1 you are negative about Horizon, then in your next statement you change your mind and say it is robust
JC relatively robust
DG but nowhere do you explain your change of mind.

JC I reject that
DG where do you explain it, then?
JC when reviewing the Fujitsu report and review process.
DG so Fujitsu had pretty good processes?
JC yes, but it changes over time. By the time it hit 3rd level support they had a reasonably good system.

JC but we often don’t know what happened before it hits Fujitsu and sometimes afterwards
DG so what you found when you read the PEAKS was that when a call came through to Fujitsu - they were quite good at problem spotting?
JC yes
DG and good at fixing it?
JC yes...

DG and good at informing the PO about it?
JC not sure about that bit
DG it’s fair to infer tho, that where there’s a bug which had an impact on branch accounts Fujitsu were good at identifying it
JC yep quite good, but they took a long time. eg Dalmellington weren’t asked...

… to get involved for a number of years. When they did they ID’d the problem quite quickly.
DG so they’re quite good at it?
JC yes
DG and quite good at informing the PO
JC that’s what we don’t know.

DG but from what you’ve seen Fujitsu are good at working out the branches and they give that information to the Post Office
JC I think so.

DG so you considered all of this and considered a lot of what Fujitsu did as quite good by 16 October. Why didn’t you think of putting this in your report? Make it more balanced?
JC doesn’t know. accepts he should probably have put it in and doesn’t know why he didn’t.

DG it’s because you were predeterminedly looking to find a negative
JC no, we were tasked with looking for problems. writing about the good stuff is not helpful
DG would it not have made the picture of the system more rounded?
JC I don’t see the value of it

DG well hang on what’s the value of making a second report in which your view has changed diametrically from an earlier report and not explaining why?
JC it’s not changed diametrically - my view is that it was more robust in the second report. On a spectrum.

DG let’s go back to what you mean by “high levels” of bugs?
JC certainly in the tens.
DG in the tens? we’re talking about a system which over 20 years was doing around 40m transactions a week and finding tens of bugs is a “high level"

JC starts to suggest (I think) he might have been extrapolating
DG did anyone ever say there were only 3 bugs in Horizon?
JC in a legal document it was said there were three?
DG no one has ever said ONLY three
JC that was my expectation when I started

DG then you read these KELS and saw there were more than three - tens.
JC there were a large number of KELS which were indicative
DG so they might be consistent with, but didn’t definitely
JC agrees
DG you are making a claim there is a high level of bugs which created...

… discrepancies in branch accounts. What were they?
JC they were in the tens
DG they were in the tens. in your table of 29 bugs. of the ones you identified, there are less than ten.

DG are you saying when you wrote this report you thought there were more bugs, then when you wrote your first joint statement, you thought there were less.
JC agrees the number of bugs he thought there were in Horizon went down from when he wrote his first report.

We have broken for lunch. Mr Coyne has confirmed to me he is from Preston. I am allowed to talk to him, just not about the trial. Right, I’ll work on getting his first two reports uploaded. Not sure whether I’ll get the joint statements today or not...

Okay Jason Coyne (independent IT expert contracted by the claimants in #postofficetrial) reports now up online. Happy reading:
postofficetrial.com/2019/06/horizo…

Court resumes at 2.05pm. I’m going back in...

Post-lunch session has begun. JC (Jason Coyne, independent IT expert) being cross-examined by DG (Anthony de Garr Robinson, QC for the Post Office).

JC explaining his team’s involvement in writing his reports.

DG would it be fair to say you were generally asking them to find problems - things that had gone wrong in Horizon?
JC yes. I’d ID’d a particular defect or theme I’d like them to look at. so they were acting under my instruction.

DG you focused on things going wrong?
JC yes and followed it through to see how it had gone wrong
DG did anyone else flag things up to you or speak to any individual claimants?
JC no

DG did you search across documents other than PEAKS and KELS?
JC yes - but only the ones disclosed
DG what is your role at IT group?
JC director
DG I’ve read your website it says you have a powerful disclosure tool for intelligent searching of documents

JC yes
DG and you would have used these techniques to go through the disclosed documents
JC yes
DG and your search tools are powerful, far more than mine
JC we didn’t employ artificial intelligence, just standard terms
DG but you can use all the tricks of the trade to...

… get the info you want out of large swathes of documents.
JC not sure what you mean by tricks of the trade, but we are quite advanced, yes.
DG what examples can you give me of the sort of techniques you used.
[JC does so]

DG so when your team would find a useful document they would bring it to your attention
JC they would tag a document so it would be a process of eliminating irrelevant documents
DG did you review everything they did?
JC they got better at spotting relevant and irrelevant docs

DG so the total number of docs reviewed by you and your team is larger than the ones you reviewed
JC yes
DG any idea how many?
JC no

DG points out in October 2018 he had received 218,367 PEAKS from Post Office and personally reviewed 1262 via intelligent search and his team did many more.
JC agrees
DG notes that later in the report he says: "Of these 5114, I have found that 163 contain PEAKs...

… that could be of significant interest and of these 76 are referred to in the report. The KELs disclose that there have been actual errors in data recorded within Horizon arising from transfer, processing of data in Horizon and data entry.”
DG asks if this is a subset...

… of the 1262 referred to earlier.
JC says yes it would be.

DG takes him to his second report where JC says: "I have now reviewed more of these records using text search criteria and filtering...

… This has enabled me to address some issues more thoroughly and has enabled a more in-depth analysis in relation to the extent of the Horizon Issues and the overall robustness of Horizon.”
DG was this more PEAKS and KELS?
JC yes

DG and other documents?
JC yes
DG could you give me examples of the other docs
JC [does so - includes the OCPs etc]
DG had you reviewed many more than you had in October
JC yes
DG any idea how many?
JC no
DG lots?
JC it would have been

[talking about how many docs JC reviewed]
DG was the main focus PEAKS and KELS
JC there were other things we were looking for
DG have you reviewed more docs since your second report in Feb?
JC yes
DG PEAKs and KELs?
JC yes
DG OCPs and OCRs?
JC yes
DG how many?
JC thousands

DG MSCs?
JC yes - they were a challenge as they were disclosed in three spreadsheets which seemed to bear no relation to each other containing millions of lines of code, so I have searched within them. So I would search for the word FAD, but not a FAD code.

DG what were you looking for?
JC the word Subpostmaster so we could see the advice given to or contact from Subpostmaster (and all the variations) were used as search terms.
DG Thanks. We now have a sense of what you looked at for your first and second report.

DG questions JC on a par in his first report: "With regards to Issue 3, whilst the present-day version of Horizon, supported by manual human support may now be considered as relatively robust in the spectrum of computer systems used...

… in businesses today it has undergone major modifications in its history. It is likely that in 1999 when it was first commissioned, and in 2010 when it was significantly upgraded (to Horizon Online), it was less robust. Horizon...

… comprises a hugely complex estate of hundreds of interfacing systems, each which exposes many potential points of error.”
DG why focus on these two areas?

JC think its relatively uncontroversial that there were problems during this time?
DG is your wording of that paragraph chosen quite carefully?
JC I’d like to think all my words are chosen carefully

DG but I note you’re not saying that Horizon is not robust
JC no that supports how I am trying to assist the court. It is not binary, the issue of robustness.
DG so you think that Horizon was relatively robust during that period
JC its a difficult question to answer as...

… you’d have to do the benchmarking across that specific time.
DG so you didn’t do any historical benchmarking?
JC well we looked at the PEAKS and KELs and there were definite spikes during the rollout of legacy Horizon and Horizon online and it was definitely less robust...

… during those periods.
DG what were the dates of this period of less robustness
JC it wasn’t a set date - it’s like a curve. as the problems get ironed out it curves downwards
DG this suggests your are visualising a graph? did you make one?
JC refers to p193 of his report .

… which refers to a graph he made about Horizon online and one which Dr Worden did on legacy Horizon.
DG so Horizon online stabilises in 2011
JC well you could argue it was in 2013.
[they try to find Dr Worden’s graph for legacy Horizon, the judge and Patrick Green QC get...

… involved. Notes are being passed from the bench behind DG]
[We have found it.]
DG Dr Worden’s report does not show a spike around legacy Horizon.
JC not in KELs
DG is your opinion based on your experience rather than your evidence?

[sorry the last two words of that tweet should say “the data”]
JC some is my experience, but there is evidence that Horizon problems spiked around the roll out of legacy Horizon and Horizon online.
DG so your evidence is that Horizon is robust and has been robust...

… apart from times around the launch of both iterations when it was less robust. Is that right?
JC correct
[we change tack]
DG why didn’t you explain the many good things about Horizon in your report?

JC I don’t think that was the point. The point was to work out the defects and errors and find out if they were causing discrepancies.
DG but you were meant to be doing a balanced report and you spent much more time talking about the bad things than the good things?

JC true
DG why?
JC because I was trying to find out things which had gone wrong and then whether or not they had caused a discrepancy - focusing on all the nice things it does is pointless
DG but the judgment of robustness is to look at in the round
JC yes but to do that you...

… have to first get to the bottom of the problem.
DG but you didn’t do that in your first or second report, did you?
JC I don’t believe that’s the case no. I spent a large amount of the second report looking at countermeasures.
DG and your view is that they are relatively good?

JC I don’t like looking at it too broadly as there are some I don’t like, but yes, if deployed correctly, they should work
DG so it follows that its part and parcel of your overall judgment that if H is relatively robust the countermeasures are relatively good too
JC corect

JC but you can’t disregard the fact the system has changed over the course of the last 20 years. Whatever the countermeasures are today, they will be different from legacy Horizon and different from other measures put in place.

DG is it your view that from 2012 - 2019 H was relatively robust?
JC yes but there were significant defects in that time
DG yes we agree that a definition of robustness is not perfection but overall you agree system and countermeasures around it were relatively robust
JC yes

DG and with legacy H - 2002 to 2010 it was relatively robust
JC yes
DG okay if you read your report into countermeasures there is no hint of your judgment that these countermeasures over this period were relatively good. Why?
JC not a conscious decision. also because I found...

… bugs errors and defects throughout that period it shows that no matter how good the countermeasure, it was overcome.
DG but NOWHERE do you say in your report that the countermeasures in Horizon are relatively good? Why?
JC I see no value to that.

JC I believe the court was asking if countermeasures had failed
DG so you regarded it to be your job to indicate to the court whether there were occasions where Horizon failed?
JC the court asked this specifically

DG but don’t you need to explain how well the countermeasures work? Dr Worden did, but in your second report you sat in your armchair and took potshots at his conclusions without making any judgment on whether he was right or not?
JC no I didn’t I looked at how the system...

… worked when it wasn’t working properly. Not when it was.

[they move on]

DG says we have agreed horizon is relatively robust, yes?
JC yes
DG so why do you say in your first report "In my position as an expert I am unable to estimate the level of the Horizon system’s robustness…

… Given the size and age of Horizon, I would however make the expert assumption (based upon systems of similar magnitude), that there are not many people who could. The sheer enormity of the task to garner a thorough understanding of the code...

.. which would be required to estimate robustness is, in my opinion, nearly impossible.”
JC Sorry - that meant I couldn’t put a number on it.
DG but why can’t you make an absolute judgment on robustness yet you can on relative robustness?

JC we can look at the rest of the industry - banking, retail etc and look at the type of defects tolerated in those industries and their impact and it is through that lens I have said it is relatively robust.

DG are you not, in your reports, by saying H is robust and then its not and then its relatively robust and then refusing to put a value on it… are you not trying to devalue your conclusion because of a preconceived idea?
JC no that’s not my intention.

[we are on a break]
a nice man who introduced himself as a lawyer came in and sat behind me for half an hour during that last session….

During the break he revealed himself as follower of the blog and a SECRET EMAILER! If you would like to be a secret emailer, please go here postofficetrial.com to find out more. You basically get secret emails. Here is yesterday’s: preview.mailerlite.com/g2x1e3/1170471…

He also told me that I shouldn’t refer to Sir Peter Fraser (the presiding judge in #postofficetrial) as J Fraser as I have been for the last seven months. It’s Fraser J. I knew this but have been unwilling to bite the bullet when I realised my mistake. In honour of...

… my mysterious friend, today marks the day. Fraser J it is.

Forgive me if I am being a little flippant. I know this is a deeply serious business. However today has been very heavy going (as you will see from the tweets) and it’s a natural inclination to let off steam during the breaks. I will rein it in.

Okay we’re back. DG is on his feet.
DG in your report you display more interest in pointing out problems you’ve found with Horizon, rather than a fair assessment of Horizon and I’m going to spend the rest of the afternoon asking you about it. Do you accept its incumbent to...

… on an expert to be scrupulously fair to assess all the documents you comes across so you don’t misrepresent them. Counsel doesn’t have time to xe you and a judge to a certain degree relies on your expertise.
JC yes
[we start to talk about cost-benefit analysis of fixing bugs]

DG you say that Fujitsu allowed bugs to remain on the system after deciding the cost of removing them was too steep
JC yes and there’s evidence that happened.
DG let’s have a look at your report…

DG quotes "There are a range of measures and controls existing in Horizon each designed to prevent, detect, identify, report and reduce the risk of several multifaceted errors. ...

…. It is likely that during the life of Horizon system that these measures and controls improved. It is also reasonably likely that in the majority of cases the measures and controls were successful.”
DG says he wishes to correct his earlier statement that there was...

… nothing positive about the countermeasures in his report. But he does undermine it with what comes next: "However, there is also evidence to indicate that a cost/benefit analysis was applied to the fixing of bugs/errors/defects…"

… and that the possibility of error was not reduced as far as possible.”

DG so you are undermining your own positive statement.
JC no I’m telling it like it is.

DG why do you repeat this statement throughout your second report?
JC because it was a question in the original Horizon issues. Was this done as far as possible and the answer is no. Cost/benefit shouldn’t have come into it.

DG points out he uses the word “often” when talking about cost/benefit decisions to let bugs lie on the system. What does he mean by often?
JC ten to 15 times
DG over 20 years and 49 bn transactions you say 10 - 15 is often?
JC we’re not talking about transactions tho are we?

DG often, tho?
JC accepts he should not have used the word often
DG well you use it more than once and we’ll be coming back to that.

[DG has just started a line of questioning about an Ernst and Young report into Horizon which recommends automating a manual process and the Post Office deciding to monitor it rather than enact the change.]
Fraser J has just stopped the cross-examination and sends...

… JC out to discuss a point of procedure with Mr de Garr Robinson.
Judge asks DG to explore documents that JC refers to so that their relevance can be discussed at the time rather than ignoring what he says and leaving it for Patrick Green QC to bring up in the 10 mins of...

… reexamination he has at 4.30pm on a Friday. DG agrees. JC is brought back in and we are all now trying to find the document JC referred to in passing. This is really tough on DG as now when JC makes a passing reference to a document, DG is being asked to stop his xe...

… and find the document. Which means DG needs to be aware of almost every document in a huge bundle. Judge tells him he doesn’t have to do it every time and if it can’t be quickly found they could move on, but it would be helpful for him as a judge to know what JC is referring..

… to. Judge says he doesn’t want to knock DG off his course. It’s a big ask, but understandable, I guess so that when the judge is going through the transcript he can see how the expert is attempting to counter the questions raised by the QC. DG has got back on course now, and..

… going back to JC’s comments on a decision made in the light of the Ernst and Young report referred to earlier. JC said in his report it was an example of a cost-benefit decision about a bug, when in fact its nothing of the sort. JC accepts has made a mistake in his report...

[we move on to another example]
DG asks if known issues are often dealt with on a cost benefit basis - there should be a significant number of PEAKS and KELs recording this being done?
JC yes but I don’t think Fujitsu were taking the decision it was Post Office...

DG but there is no evidence of this at all in the PEAKs and KELs. Or if there were you would show me.
You only have two documents, and one you have said is mistaken.
JC says the other one is quite strong.
DG but there would be PEAKs
JC possibly
DG I suggest there would be...

… if any existed. And you would put them in your report.
DG asks JC to go away tonight and come back tomorrow with any more examples he might have to stand up his assertion that bugs were often allowed to lie on the system for cost-benefit reasons. DG says there is only one...

… example he is aware of which they are discussing now.
[it is a document I cannot see]

DG explaining a low frequency problem and the risk involved in introducing a code fix and the advice that a KEL should be generated. Explains the cost of doing so. Do you say this is evidence that decisions are “often” made on a cost benefit basis?
JC well…
DG this one example?

JC it supports the other document which suggests decisions are generally made on a cost-benefit basis.
DG moves on but JC wants to show the judge how one of the possible solutions is to write a new line of code and insert it into Riposte. JC says this is relevant to other...

… matters.
DG has got him back to the KEL that was created on the back of this errors which Fujitsu decided not to fix on a cost-benefit analysis. DG is reading out from a doc I can’t see but it is the KEL which explains the bug.

DG notes that in the PEAK there is a direct reference to the cost, the KEL explains the risk. DG asks if JC thinks this is evidence that dealing with known issues was often dealt with on a cost benefit basis.
DG says it’s the only one he is aware of.
Patric Green stands up...

… “my lord I’ve tried to let this run, but…” he says in JC’s first report he references another PEAK in which the same thing happens.
Judge - what;s your point?
PG well transcript says it’s the only one - this has been going on for 20 mins - and it’s not. there’s...

… another one in his report which my learned friend should be aware of.
DG okay, let’s have a look at it then.
[DG now reading from the document - what the error was and how it was decided to be resolved]

DG is told by his bench that this is a testing peak. What is a testing PEAK?
JC it’s a defect which has arisen during testing not on a live system?
DG is this evidence of a defect which nothing is done about on a cost benefit basis?
JC certainly if nothing is done and it goes...

… live.

[they are all studying it]
JC accepts there is nothing in here which is to do with a cost benefit basis.
DG thank you.
We are finished xe-ing for the day.
Judge has some tips for JC which is to look at his hard copies as well as his screen and suggests to Mr Green...

… that he saves some of his points for re-examination before jumping to his feet in the way that he did just now [only to find the document he highlighted to the court and had everyone poring over was irrelevant]

The judge has risen. The Post Office team has left court.
There is going to be a very short write up about this tonight because I’m not really sure if there’s anything to say.
Time to unroll the tweets...

@threadreaderapp unroll pls