Please note everything I tweet is a summary or description of what is happening/being said. Nothing is a direct quote unless it is in “direct quotes"
There is some back and forth between PO and JC about what the critical point of JC’s investigation is.
JC disagrees - says he worked off the list of Horizon issues.
PO clarifies to say it’s the main reason we’re here.
PO: but is the whole point of this for the court to make a decision as to the likelihood of it causing problems for Subpostmasters
JC: yes but I wasn’t focused on individual accounts
PO: but you didn’t look at the whole of Horizon’s lifetime, did you
PO: But the crucial question is to what extent would those bugs be likely to affect branch accounts
PO: and countermeasures used to correct those bugs, either immediately or some time later
PO goes beyond technical terms to suggest it uses human systems of countermeasures as well as coding ones.
JC explains how this would happen in a computer system - testing regimes to flush them out, defect triggers etc
PO impossible to have a testing regime that picks them all up?
PO and there are ways of dealing with this - defensive programming
JC defensive progamming typically stops a bug from occurring - not done post hoc
PO so we have measures to stop bugs going live, to stop them spreading and to correct them. but no system is perfect
PO and we also have systems in place to deal with bugs when they affect the system
PO Many of those countermeasures pick up and correct user errors?
JC yes. you can design the software to correct user errors. gives example in H of high value cheques
PO so all of these measures help make a system robust.
PO so robustness is the very concept which underlies all the issues we’ve been discussing for the last half hour
[we go to a joint statement between the two IT experts. it is an agreed definition of robustness]
more from the joint report: we agree robust does not mean infallible and so H will continue to suffer faults - robustness is how H reacts to those events.
[there is a lot of agreeing going on]
JC doesn’t get reference to dimensions. Says they aren’t defined.
JC - no it’s not the critical q - it’s important
PO so when determining whether a system is robust you don’t agree quantifying the remaining risk after countermeasures...
JC I’m saying you can’t quantify that risk.
PO doesn’t it follow that the ultimate question wrt to robustness is how well the risks in a system are guarded against.
JC yes and typically that analysis is conducted in the design process of the system from the start
PO so you can’t assess its success?
JC what I’m saying is that you assess it beforehand and then you build it. it is subjective. you can’t...
PO you’re a Prince 2 practitioner, right?
PO and you specifically quantify those risks - there are graphs and charts which measure risk
JC yes it’s too simplistic for what we’re talking about here
JC it’s too simplistic, you can’t scale it up and look at it retrospectively and then come to a conclusion
PO you’re talking about two concepts - measuring risk and retrospective review...
JC I’m not saying you can’t use Prince 2 to assess risk.
PO I’m saying that’s exactly what happens
PO invariably you will be looking at past performance to inform you about a news system
PO I’m very grateful.
[That appears to conclude this part of the cross examination]
JC I said that in my first report, and then I took a view in my second report it was less robust than I thought
PO but overall you still say its relatively robust
PO I think you’re saying it has a good level of robustness compared with other systems
PO top quartile?
PO top half?
JC I don’t want to be drawn
JC [laughs]. adds context to say yes we are talking about Horizon as it is today - to get to where we are today is quite a long journey.
JC I’ve worked on military and healthcare systems which are life critical. H is nowhere near as robust as them. But it is robust...
PO large businessess with lots of users?
JC yes - a similar scale.
[we move on to the subject of acronyms and how well used the ones used by Dr Worden are in the wider IT industry]
JC doesn’t know
JC says he isn't
PO a key element of the robustness question is how H limits the extent of impact on branch accounts
PO so informing a judgment on robustness - you see if it had an impact on branch accounts and then you look at how it can be fixed
PO well in some cases it would be blindingly obvious. [Brings up example of remming in] - do you need to examine each individual case to see what the problem was
JC well not every time when there are typical fixes
PO but essentially your considered opinion is that Horizon is robust
JC relatively robust, yes.
[we move on]
JC says that wrt reconciliation data which hasn’t be disclosed, he can’t close the loop. he says if it is disclosed it may help support his opinion that Horizon is relatively robust or not
JC I don’t see what’s wrong with what I said. There must be documents which exist which show what happens, it hasn’t been disclosed to us.
PO but you don’t think the PO do nothing do you?
JC no I think they probably do do something. I just don’t know what it is.
Judge has a problem with his computer screens. He is soldiering on though.
DG they can only afford for a tiny proportion to suffer from lasting errors - air traffic control etc can have no errors, but the kind of systems we’re...
JC yes - tiny fractions
DG so if there are lasting or serious errors it could be a threat to their very survival
DG so when you say H is relatively robust you are saying in the overwhelming majority of cases...
DG and when it doesn’t it’s only a tiny tiny proportion
DG and even those get picked up?
JC not all of them.
DG that’s quite important
DG well why didn’t you spell that out in your reports?
DG you tried to give the impression H is not reliable.
JC I’m not sure I did.
DG well let’s have a look at your reports, shall we…?
[we go to the report]
DG is that fair
JC accepts his view changed as this statement was based on incomplete view
JC says no - the more he saw, the more defects were revealed - his concerns changed. there were large elements of unreliability, but that doesn’t detract from its overall relatively robustness
DG let me quote this - “sheer volume” which have “branch impacts”? you appear to be saying there is a large volume of KELs (errors) which had branch impacts. Did you mean this?
DG how many?
JC don’t know precisely. some listed in my report.
DG but you are talking about wide-ranging impact on branch accounts?
JC there’s a table at the back of the report which lists them.
JC appendix G, p200
[judge gives electronic reference]
JC Dr Worden ID’d the three the Post Office already disclosed.
DG I think he identified nine and that was agreed in the joint report.
JC don’t know
DG but Mr Coyne very few of the KELs you identified turned up in...
JC but the KELS don’t relate to the branch. The PEAKS do.
DG you just didn’t see the volume you say you did
JC it would be better if that paragraph referred to PEAKS and KELS
DG you’re not saying the pinpad errors ID’d had...
JC I think they did yes.
DG there’s no reference to it in your second report
JC what about the defect where the pinpad remembered the previous user’s account - the all pay defect?
DG well we can look at that later.
[takes him back to par 3.1]
JC yes but not the specific impact
DG but generally speaking the KEL will tell you IF there is a branch impact
DG so going back to 3.1 you are suggesting...
DG what is a reconciliation report
JC [fumbles to an answer]
DG let me help - it’s when a transaction is recorded in branch and then transmitted by the TPS system - which does what?
DG isn’t it that it transmits it to the PO back office for their systems to decided how they are pushed around or in and out of the system
JC yes it harvest the data
DG so if they don’t reconcile that’s when there’s a problem - timestamps, values, a delay
JC yep basically something has gone wrong and something has to happen to correct...
DG and it happens automatically
JC not every time
DG there are 3m transactions - human beings don’t do that
JC only when it’s working as it should
DG this generates a report of reconciliation exceptions
DG but the report will explain what has happened - automated
JC that’s the number I was referring to - many thousands
DG but why does that confirm the impact of many thousands of bugs
DG but there are many explanations for reconciliation exceptions which have nothing to do with problems causing an...
DG but it doesn’t suggest that any of these reconciliations have an impact, does it?
JC no but this level of manual reconciliation suggests something is going wrong
DG how do you quantify that
DG but you were saying in your report these reconciliation errors confirm an impact on branch accounts. It doesn’t do anything of the sort.
JC agrees. “it suggests, rather than confirms” he concedes
JC robust doesn’t mean bugs won’t have a wide-ranging impact
DG weren’t you saying that there is only a tiny number to me earlier?
JC no I didnt’ use that word
JC no I probably said a small fraction
[Patrick Green QC for the claimants stands up and says the evidence is a small fraction NOT tiny]
[crucial difference, obvs]
JC the two are not exclusive.
DG grilling him on an unequivocal judgment JC made in the joint statement about errors in the Horizon system based on limited data. Asks how he can be certain.
JC says it is a strong indication
JC yes because some of the KELS referred to PEAKS
DG then you got the PEAKS
JC some of the PEAKS
DG a large number - 250-odd thousand
DG so was your judgment here on p9 rather hasty?
JC yes - I should not have expressed that opinion there.
DG why didn’t you explain your change of mind in the report?
JC relatively robust
DG but nowhere do you explain your change of mind.
DG where do you explain it, then?
JC when reviewing the Fujitsu report and review process.
DG so Fujitsu had pretty good processes?
JC yes, but it changes over time. By the time it hit 3rd level support they had a reasonably good system.
DG so what you found when you read the PEAKS was that when a call came through to Fujitsu - they were quite good at problem spotting?
DG and good at fixing it?
JC not sure about that bit
DG it’s fair to infer tho, that where there’s a bug which had an impact on branch accounts Fujitsu were good at identifying it
JC yep quite good, but they took a long time. eg Dalmellington weren’t asked...
DG so they’re quite good at it?
DG and quite good at informing the PO
JC that’s what we don’t know.
JC I think so.
JC doesn’t know. accepts he should probably have put it in and doesn’t know why he didn’t.
JC no, we were tasked with looking for problems. writing about the good stuff is not helpful
DG would it not have made the picture of the system more rounded?
JC I don’t see the value of it
JC it’s not changed diametrically - my view is that it was more robust in the second report. On a spectrum.
JC certainly in the tens.
DG in the tens? we’re talking about a system which over 20 years was doing around 40m transactions a week and finding tens of bugs is a “high level"
DG did anyone ever say there were only 3 bugs in Horizon?
JC in a legal document it was said there were three?
DG no one has ever said ONLY three
JC that was my expectation when I started
JC there were a large number of KELS which were indicative
DG so they might be consistent with, but didn’t definitely
DG you are making a claim there is a high level of bugs which created...
JC they were in the tens
DG they were in the tens. in your table of 29 bugs. of the ones you identified, there are less than ten.
JC agrees the number of bugs he thought there were in Horizon went down from when he wrote his first report.
JC explaining his team’s involvement in writing his reports.
JC yes. I’d ID’d a particular defect or theme I’d like them to look at. so they were acting under my instruction.
JC yes and followed it through to see how it had gone wrong
DG did anyone else flag things up to you or speak to any individual claimants?
JC yes - but only the ones disclosed
DG what is your role at IT group?
DG I’ve read your website it says you have a powerful disclosure tool for intelligent searching of documents
DG and you would have used these techniques to go through the disclosed documents
DG and your search tools are powerful, far more than mine
JC we didn’t employ artificial intelligence, just standard terms
DG but you can use all the tricks of the trade to...
JC not sure what you mean by tricks of the trade, but we are quite advanced, yes.
DG what examples can you give me of the sort of techniques you used.
[JC does so]
JC they would tag a document so it would be a process of eliminating irrelevant documents
DG did you review everything they did?
JC they got better at spotting relevant and irrelevant docs
DG any idea how many?
DG notes that later in the report he says: "Of these 5114, I have found that 163 contain PEAKs...
DG asks if this is a subset...
JC says yes it would be.
DG was this more PEAKS and KELS?
DG could you give me examples of the other docs
JC [does so - includes the OCPs etc]
DG had you reviewed many more than you had in October
DG any idea how many?
JC it would have been
DG was the main focus PEAKS and KELS
JC there were other things we were looking for
DG have you reviewed more docs since your second report in Feb?
DG PEAKs and KELs?
DG OCPs and OCRs?
DG how many?
JC yes - they were a challenge as they were disclosed in three spreadsheets which seemed to bear no relation to each other containing millions of lines of code, so I have searched within them. So I would search for the word FAD, but not a FAD code.
JC the word Subpostmaster so we could see the advice given to or contact from Subpostmaster (and all the variations) were used as search terms.
DG Thanks. We now have a sense of what you looked at for your first and second report.
DG why focus on these two areas?
DG is your wording of that paragraph chosen quite carefully?
JC I’d like to think all my words are chosen carefully
JC no that supports how I am trying to assist the court. It is not binary, the issue of robustness.
DG so you think that Horizon was relatively robust during that period
JC its a difficult question to answer as...
DG so you didn’t do any historical benchmarking?
JC well we looked at the PEAKS and KELs and there were definite spikes during the rollout of legacy Horizon and Horizon online and it was definitely less robust...
DG what were the dates of this period of less robustness
JC it wasn’t a set date - it’s like a curve. as the problems get ironed out it curves downwards
DG this suggests your are visualising a graph? did you make one?
JC refers to p193 of his report .
DG so Horizon online stabilises in 2011
JC well you could argue it was in 2013.
[they try to find Dr Worden’s graph for legacy Horizon, the judge and Patrick Green QC get...
[We have found it.]
DG Dr Worden’s report does not show a spike around legacy Horizon.
JC not in KELs
DG is your opinion based on your experience rather than your evidence?
JC some is my experience, but there is evidence that Horizon problems spiked around the roll out of legacy Horizon and Horizon online.
DG so your evidence is that Horizon is robust and has been robust...
[we change tack]
DG why didn’t you explain the many good things about Horizon in your report?
DG but you were meant to be doing a balanced report and you spent much more time talking about the bad things than the good things?
JC because I was trying to find out things which had gone wrong and then whether or not they had caused a discrepancy - focusing on all the nice things it does is pointless
DG but the judgment of robustness is to look at in the round
JC yes but to do that you...
DG but you didn’t do that in your first or second report, did you?
JC I don’t believe that’s the case no. I spent a large amount of the second report looking at countermeasures.
DG and your view is that they are relatively good?
DG so it follows that its part and parcel of your overall judgment that if H is relatively robust the countermeasures are relatively good too
JC yes but there were significant defects in that time
DG yes we agree that a definition of robustness is not perfection but overall you agree system and countermeasures around it were relatively robust
DG okay if you read your report into countermeasures there is no hint of your judgment that these countermeasures over this period were relatively good. Why?
JC not a conscious decision. also because I found...
DG but NOWHERE do you say in your report that the countermeasures in Horizon are relatively good? Why?
JC I see no value to that.
DG so you regarded it to be your job to indicate to the court whether there were occasions where Horizon failed?
JC the court asked this specifically
JC no I didn’t I looked at how the system...
[they move on]
DG so why do you say in your first report "In my position as an expert I am unable to estimate the level of the Horizon system’s robustness…
JC Sorry - that meant I couldn’t put a number on it.
DG but why can’t you make an absolute judgment on robustness yet you can on relative robustness?
JC no that’s not my intention.
a nice man who introduced himself as a lawyer came in and sat behind me for half an hour during that last session….
DG in your report you display more interest in pointing out problems you’ve found with Horizon, rather than a fair assessment of Horizon and I’m going to spend the rest of the afternoon asking you about it. Do you accept its incumbent to...
[we start to talk about cost-benefit analysis of fixing bugs]
JC yes and there’s evidence that happened.
DG let’s have a look at your report…
DG says he wishes to correct his earlier statement that there was...
DG so you are undermining your own positive statement.
JC no I’m telling it like it is.
JC because it was a question in the original Horizon issues. Was this done as far as possible and the answer is no. Cost/benefit shouldn’t have come into it.
JC ten to 15 times
DG over 20 years and 49 bn transactions you say 10 - 15 is often?
JC we’re not talking about transactions tho are we?
JC accepts he should not have used the word often
DG well you use it more than once and we’ll be coming back to that.
Fraser J has just stopped the cross-examination and sends...
Judge asks DG to explore documents that JC refers to so that their relevance can be discussed at the time rather than ignoring what he says and leaving it for Patrick Green QC to bring up in the 10 mins of...
DG asks if known issues are often dealt with on a cost benefit basis - there should be a significant number of PEAKS and KELs recording this being done?
JC yes but I don’t think Fujitsu were taking the decision it was Post Office...
You only have two documents, and one you have said is mistaken.
JC says the other one is quite strong.
DG but there would be PEAKs
DG I suggest there would be...
DG asks JC to go away tonight and come back tomorrow with any more examples he might have to stand up his assertion that bugs were often allowed to lie on the system for cost-benefit reasons. DG says there is only one...
[it is a document I cannot see]
DG this one example?
DG moves on but JC wants to show the judge how one of the possible solutions is to write a new line of code and insert it into Riposte. JC says this is relevant to other...
DG has got him back to the KEL that was created on the back of this errors which Fujitsu decided not to fix on a cost-benefit analysis. DG is reading out from a doc I can’t see but it is the KEL which explains the bug.
DG says it’s the only one he is aware of.
Patric Green stands up...
Judge - what;s your point?
PG well transcript says it’s the only one - this has been going on for 20 mins - and it’s not. there’s...
DG okay, let’s have a look at it then.
[DG now reading from the document - what the error was and how it was decided to be resolved]
JC it’s a defect which has arisen during testing not on a live system?
DG is this evidence of a defect which nothing is done about on a cost benefit basis?
JC certainly if nothing is done and it goes...
JC accepts there is nothing in here which is to do with a cost benefit basis.
DG thank you.
We are finished xe-ing for the day.
Judge has some tips for JC which is to look at his hard copies as well as his screen and suggests to Mr Green...
There is going to be a very short write up about this tonight because I’m not really sure if there’s anything to say.
Time to unroll the tweets...