We are having a debate about what a transaction on Horizon is.
PG you got the stats on transactions from angela van den bogerd’s witness statements
DW got them from a range of sources
PG on Tuesday you said you got the 48m figures from AvdB witness statement
PG it was her first witness statement, the figure was 47m and JC used 47m
[we got to AvdB’s second witness statement]
DW it’s still my opinion that a transaction in a customer session has to be recovered.
DW that’s not the case
[we move on]
H = Horizon
H1 (etc) = Horizon Issue 1 (etc)
DW I did.
H1b = "(1) To what extent was it possible or likely for bugs, errors or defects to have the potential to:
(b) undermine the reliability of Horizon accurately to process and to record transactions."
PG takes him to where JC was challenged on this
PG notes cost of picking out ARQ logs for error correction via Fujistu costs around £250 a pop and there is a limit of 720 free ones under the contract per year...
PG notes a document [which I can’t see] which notes the cost of ARQ is £450 a pop.
DW yes there are two figures - I imagine one relates to the bulk order of 720...
[ah so the first 720 aren’t free.sorry. that was me trying to paraphrase PG. incorrectly]
DW technically yes
PG but also potentially misleading
DW my knowledge of the price of ARQs is not very precise
PG that’s the point
PG you say...
DW No I do have some idea
J intervenes and asks DW what he thinks the costs for ARQ requests are.
[J = judge]
PG takes DW to 2010 internal PO email re Pam Stubbs branch where an auditor had come up...
DW It looks like they’re preparing to do so.
PG so auditor has problematic figures, Pam Stubbs is asking detailed questions, it would seem ARQ data would be useful here, yes?
DW it was not a consideration of mine
PG cos here in Mrs Stubbs’ case the Post Office recognises there is a need to get ARQ data...
PG reads from later in the chain “we don’t hand over logs to a Subpostmaster” Why would they do that?
DW don’t know
PG “it needs an expert to understand what it says”...
DW well here they’re saying they usually charge the cost to the defence lawyers so it won’t cost them anything but I need to read this email properly
QC for PO intervenes
J takes his point. Asks PG to reformulate
PG asks if it makes his inference in his report in par 270 is correct in the light of these emails.
DW that was a technical inference based on quality of data
PG and you sought to explain that away by saying that other sources were good enough
DW I wouldn’t say explain away
PG do you accept that cost was now a factor
PG this isn’t about TCs in Pam Stubbs case. do you accept cost was a factor here
[we move on]
PG PO had negotiated themselves a situation where they couldn’t read the ARQ data in real time or access it without Fujitsu?
PG and they were paying £11m a year for the archiving/data centre
DW Write Once, Read Many
PG now when it is put to JC the ARQ data has to be “cracked open” from a “sealed” store he says that’s not the right way of looking at it - it could be WORM. It’s quite common that [WORM] isn’t it?
PG do you have any experience of designing audit stores?
DW not this sort
PG JC says he does. and he agrees with you they are often easily accessible
DW agrees with me?
PG do you agree audit stores can be WORM if you design them that way?
PG You state in your report "I have not had much personal involvement in building secure kernel software (SEK), or computer security techniques (SEC)...
PG focusing on the ARQ data in the data store, you describe it as the “gold standard”
PG and its accuracy matters when its being looked at when its extracted, because if it’s not the same as it is in the data store, you’ve got a problem
PG had you seen this doc?
PG this page?
DW no maybe not this page
PG this adds a more complete picture as to whether audit data is the gold standard, yes?
PG fair inference is that gaps and duplicates occur
DW no - it’s that if they do occur you should be concerned
PG so if you’d seen this
PG gaps and duplicates could arise on the journey, whilst being extracted or some bug, error, defect or remote access that could make the underlying data unreliable?
DW what do you mean?
DW I would expect it to appear in the audit store
PG if piggybacking itself did not leave a trace in transactions, that would NOT show up in the audit store
PG what about gaps
DW they shouldn’t be there
DW don’t accept - we both looked for bugs and I did lots of testing
J this doc on the screen - duplicates may occur and you say this might be because of...
DW sign that something’s wrong
J and what about on the way in?
DW maybe from marooned transactions?
[we move on]
PG continues: Fujitsu aware, a fix in, and column will be added...
[we go to the Helen Rose report]
PG you know this report?
PG Mr Jenkins features a lot in this - did you speak to him about this when you had your phone conversation?
PG Mr Jenkins notes putting a column in about transaction reversals
DW no idea
PG so the H system does not record whether one of the parties involved in a transaction agrees a figure is correct or no
DW disputes are between SPM and PO over what happened.
PG there’s no dispute button on the screen
PG were you aware a dispute button was considered in 2008 and rejected
PG you know a disputed item below £150 can’t even be settled centrally
PG and there could be more than one of these in a trading period
DW in a trading period if it goes over £150 it CAN...
PG okay. But H can record a figure which is actually in dispute. And because of your theory about TCs - those transaction corrections are correcting figures that could be wrong
PG and it might be the SPM knows or believes the figure to be wrong...
PG but it is still a figure on H which is wrong
DW yes - and a TC puts it right.
PG - picking up from Tuesday re TCs and the time they took to issue… [we go to a document called WORKING AGREEMENT - finance services centre and network - I can’t see it] did you know anything about it…
DW I rather formed the impression...
[PG is whizzing through the doc]
PO QC stands up - what is this doc?
J explain it
PG it is a PO document dated 10 March 2015 it is headed “working agreement” between PO financial services centre and PO network and internal working agreement.
PG You said you hadn’t seen this. did you know of guidance like this from any other doc?
PG this is a request by JC asking for resolution average of transaction corrections. you did not support that...
DW no. what date was it?
PG 4 June 2018. You didn’t support it, did you?
PG it might have helped learn about TCs
DW I didn’t think supporting requests made them happen any quicker
PG that’s why you didn’t support so many requests?
DW they were not...
PG but they relate to two H issues
DW not explicitly
PG would you accept they are related?
DW isnt’ sure.
DW it relates to human processes
PG so not part of the H system that you looked at
PG and this includes the process of issuing transaction corrections
[DW says this was out of scope so didn’t look at it]
PG do you agree with it?
[so JC and DW agree on the above points, just that DW felt they were out of scope]
[PG goes to section 9 - calculates upper limit of likely failures due to erroneous TCs]
DW yes because my exercise goes to robustness.
JC let’s have a look at volumes of TC. when you were eyeballing that - anything leap out?
DW 2015 seems odd
PG did you read [PO’s] Mr Smiths evidence?
PG in fact you premised your calculations on Mr Smith’s evidence. Were you in court for his cross-examination
PG did you read the transcript
DW yes and it was changed
PG did that shake your confidence in what he was talking about? You specifically relied on his figures for Camelot, didn’t you?
DW yes but when you look at them...
PG let’s have a look at them [they do] so there weren’t any figures for TCs disputed or Bank of Ireland - so they were estimated - did you pick up on that when relying on them for your report
PG you assumed they’d been
DW well on people knowing what they were talking about.
PG relating to some kind of fact?
PG let’s have a look at the transcript
PG reads from Mr Smith’s xe transcript - TCs issued is accurate, TC corrections is essentially a guess.
PG takes him to a TC summary used in DW’s report - volume of Tcs by origin 2011/2012 and the source document from which you took that - it’s an excel spreadsheet
DW lots of tabs
PG yes and we’re going to look at one in a moment...
DW I poked around it quite a bit
PG you poked around… quite important to look at it carefully. all those tabs…
[spreadsheet is so huge it is taking forever to load so PG moves to a DW’s report]
DW yes and then I think I went back to £2
PG the effect of the figures of the claim by a claimant being correct is extraordinarily low.
PG and you weren’t asked in any of the H issues by the court to discern the likelihood of the claimants being right in disputing a TC.
DW no but rightly or wrongly I thought it was important.
So here you are reaching an opinion.
PG takes him to Naushad Abdulla’s case.
DW feels right
J to two decimal places
PG what are the chances of it happening two months in a row?
DW depends on correlation factors
PG if there were no correlating...
DW yes - consecutive months might have correlations
PG and you haven’t factored any concentrations or correlations into any of your calculations, have you?
PG do you accept looking at what happened might be...
DW but then that opens up
PG so when reality departs from your theoretical model, you haven’t considered it,
DW in future trials we might look at that.
PG your model does not really fit with Mr Abdulla’s case, does it
PG no you haven’t, that’s why I’m asking you to do it now
DW I don’t know enough about Mr A’s case to do that
PG you have reached a conclusion about the claimants without considering any of their cases
DW if I did that I might reach...
PG you’ve relied on a false premise to reach a conclusion.
PG quotes @sjmurdoch’s tweets about lottery winners. Says you walk into a room and meet a load of lottery winners and you’re really surprised until you see a sign on the door saying...
DW I did a statistical analysis of the whole cohort. I can’t do an analysis on the claimants case because you have to assume the claimants are right. They are self-selected. They have decided post-hoc that H was raining bugs on them
PG that’s comical
PG no one is suggesting it can. How many claimants would join an action NOT thinking they had suffered at Horizon
DW don’t know but unlikely
PG but just because you join an action we’re not saying that causes you to go with the professor...
DW that’s exactly what I am saying
PG but all these people have joined this group because they believe they have suffered shortfalls and you’re ignoring them
DW they weren’t at the time - you’ve got your causation...
J I’m bringing this section of cross-examination to an end
PG okay if I could look at the approach that DW did do…
PG starts quoting from Tuesday’s transcript which DW says he has applied advanced statistical techniques
DW I applied elementary techniques
DW the post office lawyers told me to keep it simple
PG did you come up with the idea of an even distribution or did someone else?
DW I did.
[ we move on ]
(4) To what extent has there been potential for errors in data recorded within Horizon to arise in
(a) data entry,
(b) transfer or
(c) processing of data in Horizon?”
PG point (a) included potential for miskeying, yes
PG so having a screen with having a 1st class stamp button on it is considered helpful.
PG have you had a look at the screen layouts?
DW not my department
PG not your expertise?
PG in 224 you say "The Claimants have drawn attention to the user error of 'mis-keying', to the question of how well Horizon prevented..
and in 226 you say: "The first challenge is the very large volume of transactions, which (as I describe in section 8.5) is of the order of 6 million transactions...
PG did you actually read this document you wrote about here?
PG and did you write that section in 226 or did someone else?
PG let’s have a look then “a very large miskeyed transaction could put the viability of the branch in doubt”
DW that didn’t register with me.
DW it seemed to me basically that miskeying can lead to errors and costs to CORRECT those errors and...
PG but the report says directly the opposite to what you said it did.
DW the general tenor of the document is more about the costs to the Post Office than the costs to the branch.
PG is that fair?
DG I admit I didn’t regard...
PG so are you being fair
DW but it also talks about dealing with the error so you can assume they would be dealt with
PG did you know PO had not acted on the recommendations in this document when you wrote...
DW no. I don’t want to be an armchair critic of someone’s user interface design and what steps they took to improve it. The virtue of changing vs improvement.
PG despite having doen a lot of work on user interfaces you didn’t try to investigate how user-friendly..
PG branches have to manually key in the value of remmed in bags of cash and this document shows that the amount of cash on H is wrong because of manual keying in. Is this a H...
DW it’s something in H which is causing the SPMs to key things in wrong.
[we move on]
PG this doc goes on to say when entering docs in manually NEVER put in a decimal point followed by two zeroes. Yet that’s quite a common way of doing things.
PG would you have designed it like this?
DW I’m very afraid of designing a user-interface but it does seem strange
DW yes. there must be a reason for this.
PG now this communication to the branches doesn’t specifically mention the decimal point issue.
PG [looking at a specific case] branches who called this in told to declare...
is that a defect?
DW accepts [eventually]
PG but not necessarily fixed
DW I’d have to read the whole thing
PG let’s go back to the PO communication which doesn’t tell the SPMs about the decimal point - is that the way you’d expect...
DW [wants to go back and look] 8 Feb says don’t do this, 18 Feb comm says if there’s a problem, do this
PG well if you look at the way it’s phrased - it says we have become aware of some problems with miskeying they don’t say...
and it doesn’t refer to the problem with the decimal point.
DW the 8 Feb one does
PG this one doesn’t
DW no it says what you need to do
PG and if you want them to
DW i demur from that. you don’t want to explain too much as you get info overload.
PG was the communication about the decimal point right or wrong
PG was the communication NOT telling them about the decimal...
DW I don’t know.
PG you reproduced the former table in your report, not the latter. Why?
DW don’t know - perhaps should have done
PG mistake or deliberate
DW not deliberate
PG surely TC which arise...
DW well it depends on the proportion - in my first report I didn’t spot that for instance Santander transactions were small in volume but HUGE in value so I corrected that.
DW yes it does seem an interesting figure but you have to assume they are correct and so won’t be imputed on a branch
PG but don’t you think that 4,000+...
[sorry - missed his answer - you’ll have to wait for the transcript]
[we move on]
PG you looked at Horizon from a top down perspective
PG and we agreed that if it actually impacted branch accounts it also had the potential to do so, and if had the potential to do so, it doesn’t necessarily follow that it did?
PG takes him to his report par 650: "The Receipts/Payments Mismatch Issue
650. This issue is cited in paragraph 5.6 of Mr Coyne's report. It involved a bug in Horizon which was triggered by a rare circumstance...
PG you then cite your sources. Did your conversation with Gareth Jenkins...
DW it did help clarify
PG what did you learn that is not in the report
DW certain operations in the rollover were not double-entry - final month figure etc. I realised that double-entry accounting was applied in H in more complicated ways than I understood.
DW par 654.2 says it all "Because the operation involved was apparently not a double-entry operation on the BRDB, the countermeasure of checking the double-entry constraint DEA did not catch it."
PG did Mr Jenkins give you any detail of how, exactly when and indeed whether anyone affected by receipts/payments mismatch bug was compensated?
DW no we didn’t touch on that
DW no - neither did Fujitsu - I saw it as my business to find them...
PG this was a failure in double entry accounting (DEA) not an absence of DEA.
PG so the stock would disappear from the sending stock unit and not reappear in the receiving stock unit.
PG so H was...
So a failure of DEA?
PG because the initial failure was not evident to the SPM.
PG so let’s look at 664: "While the failure was not immediately visible to the Subpostmaster at the time of the stock transfer...
PG see that?
PG and "It was also, as Mr Godeseth says at paragraph 13.7 of his Second Witness Statement, soon visible to Fujitsu in two different ways (a flag from overnight...
and 665: "So in the normal course of events, the Subpostmaster would see a discrepancy of some large and easily identifiable sum ...
PG so it basically doesn’t sound too bad.
DW I’d expect...
PG reads 667: "At paragraph 15 of Mr Godeseth's Second Witness Statement, he says that this bug had impact on branch accounts in 20 cases…
PG have you see any evidence of SPMs being compensated.
DW no but I’ve seen them being carefully tabulated.
PG well lets have a look at the receipts/payments mismatch...
DW I agree that’s different from other cases
DW “it does a bit, actually”
PG “I’m grateful.”
[ten minute break]
reconvene at 1503 for the full charge towards 1630 close.
PG just picking up on the CS bug and payments/mismatch your understanding is that it gets corrected. just to clarify your position [we go to the document we were looking at before the break]
PG so that’s a fundamental point
PG branches will be out of sync with our own systems
DW so RDS would pick it up
DW yes but you have to make a decision about communicating with users because you don’t want to confuse those who won’t have a problem
PG and your opinion about PO’s approach here… worries about...
DW they don’t have a correct bearing on UEC [user error correction?]
PG they’re listing the impacts because that’s what they’re thinking about
"Many software bugs can have the same effects as a user error (as illustrated, for instance, by the Dalmellington bug, which produced a remming error).”
PG you’ve agreed many bugs have the same effects as user error
DW irrespective - both will get picked up by countermeasures
[PG takes him to a letter from WBD to Freeths in Jan 2017]
PG they say CS bug only affected..
PG and if we look at how the story developed...
PG see “unable to find relevant KEL” Fay at NBSC [PO H helpline] has investigated for 2 hours. Asking for help…. at the top "SU cash amounts vary on counters"… and in fifth box “due to Riposte errors” messages
DW it’s a little
PG the approach you’ve taken in your report is that neither SPMs or PO want SPMs to suffer from bugs in H. Now, the latter is right, but if PO was gaining money at the expense of SPMs - that is not an incentive to give money back
DW that’s about PO motivations...
PG well if they’ve been told to pay up and they do, then its a double win - reduces admin costs and you don’t have to reimburse
PG are you aware of PO...
[PG takes him to Second Sight pt2 report - which I’ve also published somewhere…]
PG The report notes that substantial credits have been made to PO profit and loss accounts out of unreconciled balances in its suspense account. Did you spot...
PG would you have liked to known about it
DW it would have been useful to investigate to see the sums of money Second Sight are writing about.
DW my view has not changed.
PG takes him to the way CS bug was actually handled.
PG here we have issue of SPM having trouble getting authorisation to shift sums of disputed cash into suspense account
DW for that branch?
PG yes… reading from the PEAK...
DW not specifically
PG in his xe Mr Godeseth says “a horrible position to be in” a fair summary?
DW not sure why he couldn’t get the authorisation to put it into suspense...
DW these are stock transfers which went in twice
PG reading - SPM claims some sort of H bug and she’s put things through again - so H system is not showing correctly what has happened
DW yep - it’s a Riposte problem
DW in some cases...
PG you’re not aware of a huge amount of what happened in many of these bugs are you
DW no I don’t know about the procedural side of things
PG but you in your reports made a view on whether these would be lasting discrepancy or not
PG without knowing the detail of what actually happened.
DW yes that seems to be the case
PG Did you know about this?
[we move on to a new doc - internal PO email about SPM getting problems again]
PG is this H working correctly?
PG asks DW - does this fit in with your theory about H working as it should.
[PG takes him to Anne Chambers email about Riposte Lock problem - hits some branches most weeks]
PG did you have any conception this was an
DW the Riposte Lock problem which is the source of the chain happens more often than the chain.
PG you were not aware how widespread this was
DW I was aware of some investigations...
PG the KEL was raised in 2002. it was updated in 2010. Let’s look at it.
DW this is an example of it causing different types of problems
PG 2005 - problems occuring several times a week…
DW that might be an SPM noticing and calling up...
PG 2010 - problem still occurring
DW well it might be a different manifestation of the same problem or it might be a different problem
PG it is a list of a majority of problems which “came to PEAK” but not all. This is not evidence of a systematic problem management process automatically happening.
DW that’s a rather different document imo
DW she’s doing the best that can be done
PG so fingers crossed?
DW she’s a competent woman
PG and because she said this list of branches probably didn’t have losses that’s why you’ve limited your figure to 20
PG is was first identified in 2000
PG and it was still going in 2006
PG do you regard this as a significant failing
PG Riposte was part of H
PG an important part, central
DW but they were corrected by TCs or in branch
PG leave that to one side, let’s look at the failure
PG so its important to look at how many branches are affected.
DW yes because it was detected and corrected and didn’t need a PEAK.
PG and you say in appendix D3 to your first report you’re dealing with KELs referred to by Jason Coyne. And you didn’t recognise it and you hadn’t been told about it.
PG but you didn’t recognise it either. you say in this case any error in this case will be corrected. so you conclude there’s no permanent effect on branch accounts.
DW there are ...
PG you say the PEAK implies the problem may have been around for some time. it’s not referenced in your table of PEAKs
DW that’s an oversight.
PG so where did you get your information from that it was done.
DW I would expect it
PG so you inferred from a KEL that didn’t say it that TCs would be issued.
DW you have to make inferences
PG now mr coyne found this bug by himself, didn’t he?
PG he concludes the fact that ATOS made changes to the H system is consistent with their being a H bug...
DW well I was looking for bugs which had LASTING impact on branch accounts so we had slightly different criteria.
PG it is an email re a blog by @Jusmasel2015 it says it is...
DW who is this email from?
PG it’s clear through 2015/6 there was a lot of investigation into this. was your attention drawn to this at all ay any point...
PG was the first time you came across it when JC alerted you to it?
DW think so
PG you expressly relied on Mr Godeseth’s evidence on those three bugs
DW it didn’t really add to my knowledge of what I had already thought about lasting...
[PG now takes him to his expert report]
[I was about to say we were whizzing around and J has just asked PG to slow down]
PG short point is that you don’t touch on Dalmellington in your report - why?
PG isn’t the scale of it important?
DW Dalmellington never came to Fujitsu - for good reason.
PG [takes him to a Fujitsu presentation] Branch Outreach Issue (initial findings)
6 incidents - 2011 - 0 calls raised
9 incidents - 2012 - 0 calls raised
7 incidents - 2013 - 0 calls raised
DW these were, as we have heard, resolved
PG do you not think that given the scale and effect of this bug it should have featured in your report?
DW my opinion is and remains that remming errors get corrected
PG at 1.02pm today we got the missing ARQ figures - 2013 to 2014 and it turns out there were more than the allowance of 720 used in that year.
J that’s about the extraction of data in more usable form
PG correct it was disclosed in separate spreadsheets. it was then disclosed in a new and different form and I cross-examined in the difference between the two.
J that’s a point for submission and argument and for me not xe
Mr De garr robinson rises - and says he was responsible for the initial format of the initial MSC submission - he asked for it for his xe
J accepts that this was said in the letter he has received but is grateful for the repetition
Mr de Garr just at pains to point out: There is no conspiracy here.
PG says we were...
[now on to late disclosure of 1000s of documents]
J this much, this late in proceedings needs explanation
DG can I do so now?
DG Fujitsu found thousands on an old database and in a matter of days steps were taken to make the court aware of it.
J okay well no one has quite explained this in the way you just have
Okay that’s you’re lot for today. I’ll unroll the tweets and get a report and the transcript up on postofficetrial.com as soon as possible.
Contributors who put in more than £20 will get added to the secret email list which means you get extra (usually quite gossipy...
Right - we’re back at 10.30am tomorrow for Dr Worden’s last day of cross-examination.
And it’s Friday, thank goodness.