Thread by @nickwallis

We are back in court 26 for Day 19 of the Horizon trial. Claimants’ barristers arriving at the Rolls Building this morning. Patrick Green QC (left) putting the questions to IT expert for the Post Office, Dr Worden. Live tweets to follow.

Usual disclaimer: I am paraphrasing and summarising what is being said and happening in court. Nothing is a direct quote unless it is in “direct quotes”.

If you know nothing about this litigation, you can find everything you need at postofficetrial.com
The hashtag is #postofficetrial

Very thin in the public “gallery” today. Claimant Subpostmaster Jo Hamilton is here @JoHamil73963257 and @Karlfl from @ComputerWeekly. ...

@JoHamil73963257 @Karlfl @ComputerWeekly Spotted @JoshuaRozenberg coming into the building this morning, but I think he’s on another trial (pretty sure that’s a friendly wave, rather than a "no paps” hand).

@JoshuaRozenberg Patrick Green (PG) QC for the claimants is on his feet. Doctor Worden, independent IT expert for the claimants is on his second day of cross-examination.
We are having a debate about what a transaction on Horizon is.

DW it’s my opinon that stats on customer sessions are synonymous with transactions
PG you got the stats on transactions from angela van den bogerd’s witness statements
DW got them from a range of sources
PG on Tuesday you said you got the 48m figures from AvdB witness statement

PG and that Jason Coyne (JC - independent IT expert for the claimants) used the same figure.
PG it was her first witness statement, the figure was 47m and JC used 47m
[we got to AvdB’s second witness statement]

PG reads: "if there is a payment due to or from the customer, the session (ie transaction or transactions)" - that is the witness from whom you took the figures as to the number of transactions
DW it’s still my opinion that a transaction in a customer session has to be recovered.

PG you were embarrassed by what you said on Tuesday and today is an answer of convenience to cover up your embarrassment
DW that’s not the case
[we move on]

[we are looking at Horizon issues 1b - if you would like to have the Horizon Issues to hand I have posted them up in a very easy-to-read format here postofficetrial.com/2019/06/horizo…]
H = Horizon
H1 (etc) = Horizon Issue 1 (etc)

PG you chose to deal with H1b separately in your report.
DW I did.
H1b = "(1) To what extent was it possible or likely for bugs, errors or defects to have the potential to:

(b) undermine the reliability of Horizon accurately to process and to record transactions."

PG now takes him to par 270 of DW’s 1st report "the evidence that I have seen in KELs indicates that use of the audit database was a backstop, and rarely used - because other comparisons of data were usually sufficient to diagnose the problem"

PG you know this is a matter of dispute between the parties
DW yes
PG takes him to where JC was challenged on this
PG notes cost of picking out ARQ logs for error correction via Fujistu costs around £250 a pop and there is a limit of 720 free ones under the contract per year...

PG notes there’s no evidence PO has been charged for ARQ retrieval. Or that anyone has been discouraged for asking.
PG notes a document [which I can’t see] which notes the cost of ARQ is £450 a pop.
DW yes there are two figures - I imagine one relates to the bulk order of 720...

… and the higher amount for more than 720
[ah so the first 720 aren’t free.sorry. that was me trying to paraphrase PG. incorrectly]

PG what was put to JC was that getting an ARQ data request was “over” £200. so is that consistent with it actually costing £450
DW technically yes
PG but also potentially misleading
DW my knowledge of the price of ARQs is not very precise
PG that’s the point
PG you say...

… there’s a distinction between the cost of the first 720 and the remainder. And you say its £450 for the first 720 and around £200-£250 after. But you have no idea.
DW No I do have some idea
J intervenes and asks DW what he thinks the costs for ARQ requests are.

DW says £450 up to 720 and then “ballpark” £250 after
[J = judge]

PG reads from PO internal doc in 2012 noting cost of £436 per ARQ inquiry. Continues to read. PO doc author notes they could change the contract to get the ARQ data much more cheaply.
PG takes DW to 2010 internal PO email re Pam Stubbs branch where an auditor had come up...

… with proof of an odd discrepancy after visiting Pam Stubbs’ branch.

PG setting the scene. This is a classic case where ARQ data might be helpful, yes?
DW It looks like they’re preparing to do so.
PG so auditor has problematic figures, Pam Stubbs is asking detailed questions, it would seem ARQ data would be useful here, yes?
DW yes.

PG continues reading from email which suggests getting ARQ data as there is growing interest from MPs and media. Were you aware of that?
DW it was not a consideration of mine
PG cos here in Mrs Stubbs’ case the Post Office recognises there is a need to get ARQ data...

… because of the situation, but further motivated to do so because of interest from MPs and media.
DW accepts
PG reads from later in the chain “we don’t hand over logs to a Subpostmaster” Why would they do that?
DW don’t know
PG “it needs an expert to understand what it says”...

PG “is this for our benefit? as there’s a cost attached” cost was a disincentive for PO based on this email
DW well here they’re saying they usually charge the cost to the defence lawyers so it won’t cost them anything but I need to read this email properly
QC for PO intervenes

QC PO says it is not a proper line of questioning for an IT expert
J takes his point. Asks PG to reformulate
PG asks if it makes his inference in his report in par 270 is correct in the light of these emails.
DW that was a technical inference based on quality of data

PG you and JC found the audit data was very rarely referred to
DW yes
PG and you sought to explain that away by saying that other sources were good enough
DW I wouldn’t say explain away
PG do you accept that cost was now a factor

DW I’ve always said cost was a disincentive in TCs
PG this isn’t about TCs in Pam Stubbs case. do you accept cost was a factor here
DW accepts
[we move on]

PG you haven’t looked at number of ARQ requests in any particular year.
DW no
PG PO had negotiated themselves a situation where they couldn’t read the ARQ data in real time or access it without Fujitsu?
DW accepts
PG and they were paying £11m a year for the archiving/data centre

…. to Fujitsu. Now it was put to JC that it was too expensive to read ARQ or audit data. But that was a consequence of the way it had been set up that it was expensive
DW accepts

PG you’ve heard of the acronym WORM haven’t you
DW Write Once, Read Many
PG now when it is put to JC the ARQ data has to be “cracked open” from a “sealed” store he says that’s not the right way of looking at it - it could be WORM. It’s quite common that [WORM] isn’t it?

DW accepts
PG do you have any experience of designing audit stores?
DW not this sort
PG JC says he does. and he agrees with you they are often easily accessible
DW agrees with me?
PG do you agree audit stores can be WORM if you design them that way?
DW yes

PG and if you don’t design them that way, they’re not?
DW yes
PG You state in your report "I have not had much personal involvement in building secure kernel software (SEK), or computer security techniques (SEC)...

… although I am familiar with the underlying mathematical specification methods.”
PG focusing on the ARQ data in the data store, you describe it as the “gold standard”
DW yes

PG it’s accuracy matters within the data store
DW yes
PG and its accuracy matters when its being looked at when its extracted, because if it’s not the same as it is in the data store, you’ve got a problem
DW yes

PG takes him to doc: audit store data retrieval client user manual. Describes the filtering process when extracting data from the audit store. shows an example - lists gaps and duplicates (shown in red and blue). Did anyone tell you about this?
DW no
PG had you seen this doc?

DW yes
PG this page?
DW no maybe not this page
PG this adds a more complete picture as to whether audit data is the gold standard, yes?
DW yes
PG fair inference is that gaps and duplicates occur
DW no - it’s that if they do occur you should be concerned
PG so if you’d seen this

PG it wouldn’t change your view that it was the “gold standard”
DW no
PG gaps and duplicates could arise on the journey, whilst being extracted or some bug, error, defect or remote access that could make the underlying data unreliable?
DW what do you mean?

PG well if someone had piggybacked onto someone’s login to change some data - would it show?
DW I would expect it to appear in the audit store
PG if piggybacking itself did not leave a trace in transactions, that would NOT show up in the audit store
DW correct

PG Mr Dunks could not explain why gaps might appear [we are now talking about duplicates in ARQ data the seema misra case and how they were noticed, and according to Gareth Jenkins they were rectified through a semi-automated process]
PG what about gaps
DW they shouldn’t be there

PG your report has been written from the point of design aspiration and JC looked at what actually happened
DW don’t accept - we both looked for bugs and I did lots of testing
J this doc on the screen - duplicates may occur and you say this might be because of...

… info being extracted from two correspondence servers.
DW yes
J gaps?
DW sign that something’s wrong
J and what about on the way in?
DW maybe from marooned transactions?
[we move on]

PG [new doc] it says ARQ data does not record transactions reversals [if I heard that right - seems odd if it doesn’t]

PG reads from a Fujitsu doc: spreadsheets supplied by the prosecution team mis out an indication as to whether transactions are reversed - what’s being spoken about is the transaction process
DW yes
PG continues: Fujitsu aware, a fix in, and column will be added...

… don’t need a KEL, but must get this out asap as otherwise prosecution data will be incomplete. So the filtering of ARQ data can have a huge bearing on the info you get.
DW yes
[we go to the Helen Rose report]

[which I published here: postofficetrial.com/2019/03/the-he…]

PG you know this report?
DW yes
PG Mr Jenkins features a lot in this - did you speak to him about this when you had your phone conversation?
DW no
PG Mr Jenkins notes putting a column in about transaction reversals

… saying it would be relatively simple thing to do. Yet Angela van den Bogerd said it hadn’t been done. Why - given it is simple and required?
DW no idea

PG you agree with JC that H does not record disputes
DW yes
PG so the H system does not record whether one of the parties involved in a transaction agrees a figure is correct or no
DW disputes are between SPM and PO over what happened.
PG there’s no dispute button on the screen

DW no
PG were you aware a dispute button was considered in 2008 and rejected
DW no
PG you know a disputed item below £150 can’t even be settled centrally
DW yes
PG and there could be more than one of these in a trading period
DW in a trading period if it goes over £150 it CAN...

… be settled centrally.
PG okay. But H can record a figure which is actually in dispute. And because of your theory about TCs - those transaction corrections are correcting figures that could be wrong
DW yes
PG and it might be the SPM knows or believes the figure to be wrong...

… it might be PO or Fujitsu think the figures are wrong
DW yes
PG but it is still a figure on H which is wrong
DW yes - and a TC puts it right.

[short break]
PG - picking up from Tuesday re TCs and the time they took to issue… [we go to a document called WORKING AGREEMENT - finance services centre and network - I can’t see it] did you know anything about it…
DW I rather formed the impression...

PG did you know any facts?
DW no
[PG is whizzing through the doc]
PO QC stands up - what is this doc?
J explain it
PG it is a PO document dated 10 March 2015 it is headed “working agreement” between PO financial services centre and PO network and internal working agreement.

PG at par 3.1 it sets out standard timescales for the issuing of TCs.
PG You said you hadn’t seen this. did you know of guidance like this from any other doc?
DW no
PG this is a request by JC asking for resolution average of transaction corrections. you did not support that...

… request did you.
DW no. what date was it?
PG 4 June 2018. You didn’t support it, did you?
DW no
PG it might have helped learn about TCs
DW I didn’t think supporting requests made them happen any quicker
PG that’s why you didn’t support so many requests?
DW they were not...

… my main focus.
PG but they relate to two H issues
DW not explicitly
PG would you accept they are related?
DW isnt’ sure.

PG if we look at what JC says in his expert report. "The reconciliation process ultimately presents the possibility of further error within the Horizon system whereby an inappropriate method of fix was selected, and/or the corrective fixes may have been carried out erroneously."

PG do you agree with that?
DW it relates to human processes
PG so not part of the H system that you looked at
DW yes
PG and this includes the process of issuing transaction corrections

PG goes to more JC: "Transaction Corrections can be issued to either rectify an error or discrepancy deemed as a fault by the Subpostmaster (or clerk), or when branch transaction data does not align with Post Office, or a Post Office...

… external client which may not be an error on the Subpostmasters’ part. It is also possible that Transaction Corrections were issued as a result of error in Horizon transaction data processing.”
[DW says this was out of scope so didn’t look at it]
PG do you agree with it?

DW yes
[so JC and DW agree on the above points, just that DW felt they were out of scope]
[PG goes to section 9 - calculates upper limit of likely failures due to erroneous TCs]

[sorry section 9 of DW’s report, not JC’s]

PG so you calculate portion of disputes upheld in relation to claimant branches but you think JC’s exercise it out of scope
DW yes because my exercise goes to robustness.
JC let’s have a look at volumes of TC. when you were eyeballing that - anything leap out?
DW 2015 seems odd

PG did you pursue it?
DW no
PG did you read [PO’s] Mr Smiths evidence?
DW yes
PG in fact you premised your calculations on Mr Smith’s evidence. Were you in court for his cross-examination
DW no
PG did you read the transcript
DW yes

PG Mr Smith presented his figures for Santander in a way that misled you and JC
DW yes and it was changed
PG did that shake your confidence in what he was talking about? You specifically relied on his figures for Camelot, didn’t you?
DW yes but when you look at them...

… they’re not very big towards the overall total.
PG let’s have a look at them [they do] so there weren’t any figures for TCs disputed or Bank of Ireland - so they were estimated - did you pick up on that when relying on them for your report
DW yes
PG you assumed they’d been

… based on some kind of fact?
DW well on people knowing what they were talking about.
PG relating to some kind of fact?
DW yes
PG let’s have a look at the transcript
PG reads from Mr Smith’s xe transcript - TCs issued is accurate, TC corrections is essentially a guess.

[Mr Smith says he has no confidence in them]
PG takes him to a TC summary used in DW’s report - volume of Tcs by origin 2011/2012 and the source document from which you took that - it’s an excel spreadsheet
DW lots of tabs
PG yes and we’re going to look at one in a moment...

PG … when it comes up. did you look at it carefully
DW I poked around it quite a bit
PG you poked around… quite important to look at it carefully. all those tabs…
[spreadsheet is so huge it is taking forever to load so PG moves to a DW’s report]

PG quotes from DW "If there were 2% of TCs issued in error30, which were resolved incorrectly against the branch, the net effect on branch accounts would be £6 per branch per month...

… As described above, in my opinion this is a conservative upper limit on the magnitude inaccuracies introduced into branch accounts - which could be of either sign.”

"However, because, as described in section 8.5, Claimants' branches were on average three times smaller than typical Post Office branches (as measured by number of customer transactions per day), one would expect the number of TCs issued to Claimants' branches, and the...

… number issued in error, to be three times smaller than the average for all branches - and therefore to be approximately £2 per month. This again is a conservative upper limit on the magnitude of discrepancies, which may have been of either sign."

PG and in a later report you change that to £1.50 a month. So that 50p change is important based on your engineering approach.
DW yes and then I think I went back to £2
PG the effect of the figures of the claim by a claimant being correct is extraordinarily low.

DW yes
PG and you weren’t asked in any of the H issues by the court to discern the likelihood of the claimants being right in disputing a TC.
DW no but rightly or wrongly I thought it was important.

PG goes to par 32 of the DW’s supplementary report which confirms £1.50 correction “so TC corrections cannot be part of the claimed shortfalls”.
So here you are reaching an opinion.
PG takes him to Naushad Abdulla’s case.

PG you say chances of a TC correction being wrong is 1% on average for £200, so for £1000 it would be 0.14%?
DW feels right
J to two decimal places
PG what are the chances of it happening two months in a row?
DW depends on correlation factors
PG if there were no correlating...

… factors its 0.14 x 0.14 = around 1 in 500,000 chance.
DW yes - consecutive months might have correlations
PG and you haven’t factored any concentrations or correlations into any of your calculations, have you?
DW no
PG do you accept looking at what happened might be...

… a good way of testing your theoretical model?
DW but then that opens up
PG so when reality departs from your theoretical model, you haven’t considered it,
DW in future trials we might look at that.
PG your model does not really fit with Mr Abdulla’s case, does it

DW I haven’t looked at Mr Abdulla’s case
PG no you haven’t, that’s why I’m asking you to do it now
DW I don’t know enough about Mr A’s case to do that
PG you have reached a conclusion about the claimants without considering any of their cases
DW if I did that I might reach...

… different conclusions
PG you’ve relied on a false premise to reach a conclusion.
DW no
PG quotes @sjmurdoch’s tweets about lottery winners. Says you walk into a room and meet a load of lottery winners and you’re really surprised until you see a sign on the door saying...

@sjmurdoch … lottery winners only
DW I did a statistical analysis of the whole cohort. I can’t do an analysis on the claimants case because you have to assume the claimants are right. They are self-selected. They have decided post-hoc that H was raining bugs on them
PG that’s comical

DW - causation cannot go back in time
PG no one is suggesting it can. How many claimants would join an action NOT thinking they had suffered at Horizon
DW don’t know but unlikely
PG but just because you join an action we’re not saying that causes you to go with the professor...

… Back to the Future and start finding that H causes bugs!
DW that’s exactly what I am saying
PG but all these people have joined this group because they believe they have suffered shortfalls and you’re ignoring them
DW they weren’t at the time - you’ve got your causation...

… back to front.
J I’m bringing this section of cross-examination to an end
PG okay if I could look at the approach that DW did do…
PG starts quoting from Tuesday’s transcript which DW says he has applied advanced statistical techniques
DW I applied elementary techniques

PG who told you that
DW the post office lawyers told me to keep it simple
PG did you come up with the idea of an even distribution or did someone else?
DW I did.
[ we move on ]

[we got to H4:

(4) To what extent has there been potential for errors in data recorded within Horizon to arise in

(a) data entry,
(b) transfer or
(c) processing of data in Horizon?”

PG point (a) included potential for miskeying, yes
DW yes

PG you refer in par 222 "In the design of the Horizon counter user interface, there are large numbers of measures to prevent user errors. Many of these measures have by now become common practice in the design of user interfaces...

… such as the use of menus and buttons, rather than free text input”
PG so having a screen with having a 1st class stamp button on it is considered helpful.
DW yes
PG have you had a look at the screen layouts?
DW not my department
PG not your expertise?

DW well I did run a team that looked at air-traffic control and what I learned was that the only way to get it right was extensive user-testing
PG in 224 you say "The Claimants have drawn attention to the user error of 'mis-keying', to the question of how well Horizon prevented..

… mis-keying, and whether Horizon might have prevented it more effectively.”

and in 226 you say: "The first challenge is the very large volume of transactions, which (as I describe in section 8.5) is of the order of 6 million transactions...

… per day across Post Office branch network. If even a tiny fraction of these were in error through mis-keying, and those errors involved Post Office central support costs in correcting the errors, then the central support costs would be significant.”

[PG takes him to 8.5 - a document about the costs of miskeying]

PG did you actually read this document you wrote about here?

DW yes

PG and did you write that section in 226 or did someone else?

DW I read it and wrote it.

PG let’s have a look then “a very large miskeyed transaction could put the viability of the branch in doubt”
DW that didn’t register with me.
DW it seemed to me basically that miskeying can lead to errors and costs to CORRECT those errors and...

… large errors seem very likely to b corrected.

PG but the report says directly the opposite to what you said it did.

DW the general tenor of the document is more about the costs to the Post Office than the costs to the branch.

PG is that fair?
DG I admit I didn’t regard...

… that final sentence about a mistake having an impact on the branch.
PG so are you being fair
DW but it also talks about dealing with the error so you can assume they would be dealt with
PG did you know PO had not acted on the recommendations in this document when you wrote...

… your report.
DW no. I don’t want to be an armchair critic of someone’s user interface design and what steps they took to improve it. The virtue of changing vs improvement.
PG despite having doen a lot of work on user interfaces you didn’t try to investigate how user-friendly..

…. H is, did you?
DW no.

[J says he normally does this offline, but can’t resist on this occasion - notes that in the transcript when PG was waxing lyrical about Back to the Future the transcribers called the car a DeLoray-yum. He asks if counsel can ensure the correct spelling is supplied. He rises.]

Lunch. Or “the short adjournment” as the transcribers obliquely describe it. I suppose the idea of something so vulgar as eating shouldn’t be mentioned.

NEW: The Post Office has lodged its request for permission to appeal the Common Issues trial judgment with the Court of Appeal. I was pointed to it in court.
“What - that big yellow file?”
“No those two boxes each containing six big yellow files.”
#litigation
#postofficetrial

[we’re back for the post lunch cross-examination of Dr Worden (DW) independent IT expert for the Post Office, and Patrick Green (PG), QC for the claimants]

[we’re looking at a document showing over-stated cash positions due to keying errors in a branch]
PG branches have to manually key in the value of remmed in bags of cash and this document shows that the amount of cash on H is wrong because of manual keying in. Is this a H...

… system error or an SPM error?
DW it’s something in H which is causing the SPMs to key things in wrong.
[we move on]
PG this doc goes on to say when entering docs in manually NEVER put in a decimal point followed by two zeroes. Yet that’s quite a common way of doing things.

DW yes it does seem a bit odd
PG would you have designed it like this?
DW I’m very afraid of designing a user-interface but it does seem strange
PG bizarre
DW yes. there must be a reason for this.

PG if we go over the page please… you can see 5 March 2019 - this communication was sent as there has been a higher than average or forex remittances not populating. We have seen this causes a lot of errors - “a general reminder all branches to take extra care when manually ...

…. entering totals into Horizon.”
PG now this communication to the branches doesn’t specifically mention the decimal point issue.
DW agrees.

PG so this has to be corrected by a TC - if you can’t reverse a rem in, it has to be corrected by a transaction. If you can reverse it, you have an opportunity to correct it in branch.
DW accepts
PG [looking at a specific case] branches who called this in told to declare...

… cash and that it would be corrected by TC. So we have a bug which causes the SPM to have to manually key in and then we have a defect which causes sums to be out by a factor of 10 or 100 if they use a decimal point when keying in.
is that a defect?
DW accepts [eventually]

[PG reads from PO report - this needs to be proactively managed in future]
PG but not necessarily fixed
DW I’d have to read the whole thing
PG let’s go back to the PO communication which doesn’t tell the SPMs about the decimal point - is that the way you’d expect...

… essential communications to be done.
DW [wants to go back and look] 8 Feb says don’t do this, 18 Feb comm says if there’s a problem, do this
PG well if you look at the way it’s phrased - it says we have become aware of some problems with miskeying they don’t say...

… it’s because of a system failure, and then they say some branches have keyed in the wrong amounts…
and it doesn’t refer to the problem with the decimal point.
DW the 8 Feb one does
PG this one doesn’t
DW no it says what you need to do
PG and if you want them to

… do the right thing you need to tell them what’s wrong
DW i demur from that. you don’t want to explain too much as you get info overload.
PG was the communication about the decimal point right or wrong
DW right
PG was the communication NOT telling them about the decimal...

… point right or wrong?
DW I don’t know.

[we move on to a table in a spreadsheet containing tables called branch errors and other errors]
PG you reproduced the former table in your report, not the latter. Why?
DW don’t know - perhaps should have done
PG mistake or deliberate
DW not deliberate
PG surely TC which arise...

…. from what are not branch errors might be highly relevant?
DW well it depends on the proportion - in my first report I didn’t spot that for instance Santander transactions were small in volume but HUGE in value so I corrected that.

PG but don’t you see how TC’s which aren’t branch errors are relevant to your analysis of how the system works?
DW yes it does seem an interesting figure but you have to assume they are correct and so won’t be imputed on a branch
PG but don’t you think that 4,000+...

…. errors outside of the branches is worth looking at?
[sorry - missed his answer - you’ll have to wait for the transcript]
[we move on]
PG you looked at Horizon from a top down perspective
DW yes

PG but if you look at a particular bug it’s helpful to know how many branches it affected
DW yes
PG and we agreed that if it actually impacted branch accounts it also had the potential to do so, and if had the potential to do so, it doesn’t necessarily follow that it did?

DW agrees
PG takes him to his report par 650: "The Receipts/Payments Mismatch Issue
650. This issue is cited in paragraph 5.6 of Mr Coyne's report. It involved a bug in Horizon which was triggered by a rare circumstance...

… (which one would not expect to be exercised in testing) and which had an effect on branch accounts. If Mr Godeseth's evidence about this bug is not accepted, I shall revise my opinions accordingly.”

PG you then cite your sources. Did your conversation with Gareth Jenkins...

… help?
DW it did help clarify
PG what did you learn that is not in the report
DW certain operations in the rollover were not double-entry - final month figure etc. I realised that double-entry accounting was applied in H in more complicated ways than I understood.

PG did you set this out in your report
DW par 654.2 says it all "Because the operation involved was apparently not a double-entry operation on the BRDB, the countermeasure of checking the double-entry constraint DEA did not catch it."

PG what other operations were not double-entry accounting, can you remember?
DW no
PG did Mr Jenkins give you any detail of how, exactly when and indeed whether anyone affected by receipts/payments mismatch bug was compensated?
DW no we didn’t touch on that

PG [we move on to Callender Square bug.] so this was one of the three bugs acknowledged by PO - Suspense account, Callender Square (CS) and receipts payments mismatch. Did they tell you of any others.
DW no - neither did Fujitsu - I saw it as my business to find them...

PG [we go to DW’s first report par 663] on CS "The result was typically that the stock would disappear from the sending stock unit, and not reappear in the receiving stock unit - a failure of double entry accounting (DEA) which...

… was not evident to the Subpostmaster at the time. At paragraph 13.6 of his Second Witness Statement, Mr Godeseth comments on this failure of double entry accounting.”
PG this was a failure in double entry accounting (DEA) not an absence of DEA.

PG quotes par 663: "In my opinion, under the later Horizon Online software this failure of DEA might have been immediately manifest as a failure to send a zero-sum basket to the BRDB. ...

… But in Old Horizon, apparently it was not immediately detected, so in this respect Old Horizon was possibly less robust than Horizon Online.”
PG so the stock would disappear from the sending stock unit and not reappear in the receiving stock unit.
DW yes
PG so H was...

… recording information incorrectly.
So a failure of DEA?
DW yes
PG because the initial failure was not evident to the SPM.
DW yes
PG so let’s look at 664: "While the failure was not immediately visible to the Subpostmaster at the time of the stock transfer...

… , it would always be visible later when balancing stock units.”
PG see that?
DW yes
PG and "It was also, as Mr Godeseth says at paragraph 13.7 of his Second Witness Statement, soon visible to Fujitsu in two different ways (a flag from overnight...

… processing, and a system event). If this is correct, it was robustness through the countermeasure RDS.”
and 665: "So in the normal course of events, the Subpostmaster would see a discrepancy of some large and easily identifiable sum ...

… (because stock unit transfers generally involve larger sums than customer transactions) and would know, since he had not made any mistake, to call the help desk. This was countermeasure MID. As is shown by the Peaks...

… , the presence of the Riposte error was easily identifiable from system logs, so the help desk would know it was not a user error and Post Office could correct any discrepancy if it arose in the branch accounts.”
PG so it basically doesn’t sound too bad.
DW I’d expect...

… most cases to be sorted out.
PG reads 667: "At paragraph 15 of Mr Godeseth's Second Witness Statement, he says that this bug had impact on branch accounts in 20 cases…

… For the receipts/payments mismatch bug, there is evidence that affected branches were compensated. Because of this evidence, and because Fujitsu could always spot any occurrence of the bug in event logs, and because neither Post Office or Subpostmaster...

… wanted Subpostmasters to suffer shortfalls from bugs in Horizon, I would expect the Subpostmaster to be left with a shortfall (i.e. not compensated) in only a small minority of cases, if any cases. So in my opinion the net shortfall ...

… caused by all its occurrences would be possibly zero, and in any event at most a few thousand pounds.”
PG have you see any evidence of SPMs being compensated.
DW no but I’ve seen them being carefully tabulated.
PG well lets have a look at the receipts/payments mismatch...

… notes [from an internal PO doc] - affecting 40 branches with a £20K loss. Note the branch will not get a prompt - branch will believe it has balanced correctly. Now that is not the system alerting the SPM, is it?
DW I agree that’s different from other cases

PG “honestly Dr Worden, does that take you by surprise?”
DW “it does a bit, actually”
PG “I’m grateful.”
[ten minute break]
reconvene at 1503 for the full charge towards 1630 close.

[we’re back]
PG just picking up on the CS bug and payments/mismatch your understanding is that it gets corrected. just to clarify your position [we go to the document we were looking at before the break]

PG under impact “the branch has appeared to balance” so it wouldn’t be apparent would it
DW no
PG so that’s a fundamental point
DW yep
PG branches will be out of sync with our own systems
DW so RDS would pick it up

PG if you want your users to pick up on problems is being candid important
DW yes but you have to make a decision about communicating with users because you don’t want to confuse those who won’t have a problem
PG and your opinion about PO’s approach here… worries about...

ongoing legal cases, belief in the integrity in the H system - does that affect effective countermeasures?

DW they don’t have a correct bearing on UEC [user error correction?]

PG they’re listing the impacts because that’s what they’re thinking about

[by the way I think I published this document because it was discussed at length in the common issues trial - let me have a look]

Yep here it is: postofficetrial.com/2018/11/two-im…

[it’s clear DW is seeing this for the first time. I published it on 12 Nov last year]

[he is being taken through it line by line - why not call it up and do the same where you are. They're on the solutions options list atmo]

PG the short point is the correction of what had gone wrong is not automatic, it is discretionary as to how and whether they did it.
DW accepts.

[we go to one of the IT experts joint statements]
"Many software bugs can have the same effects as a user error (as illustrated, for instance, by the Dalmellington bug, which produced a remming error).”
DW yes…
PG you’ve agreed many bugs have the same effects as user error

PG now Angela van den Bogerd agreed the PO suffers from User Error Bias. so SPMs are more likely to blamed for bugs than bugs
DW irrespective - both will get picked up by countermeasures
[PG takes him to a letter from WBD to Freeths in Jan 2017]
PG they say CS bug only affected..

… one branch and you have identifed it affected 20 branches.
DW yes
PG and if we look at how the story developed...

… [DW is given a hard copy of a PEAK - PEAK is a Fujitsu error log]
PG see “unable to find relevant KEL” Fay at NBSC [PO H helpline] has investigated for 2 hours. Asking for help…. at the top "SU cash amounts vary on counters"… and in fifth box “due to Riposte errors” messages

…. were not replicated [explains how] this resulted in a loss of £3K+… phoned SPM to explain loss and he is worried about duplicate transactions and advised him to go to NBSC as this is a business issue. we know this is a bug, but he’s been told to go to NBSC.
DW it’s a little

… confusing.
PG the approach you’ve taken in your report is that neither SPMs or PO want SPMs to suffer from bugs in H. Now, the latter is right, but if PO was gaining money at the expense of SPMs - that is not an incentive to give money back
DW that’s about PO motivations...

DW and I think their motivation is to keep down costs including admin costs from SPMs complaining about costs
PG well if they’ve been told to pay up and they do, then its a double win - reduces admin costs and you don’t have to reimburse
DW yep
PG are you aware of PO...

… suspense accounts.
DW yes

[PG takes him to Second Sight pt2 report - which I’ve also published somewhere…]

PG The report notes that substantial credits have been made to PO profit and loss accounts out of unreconciled balances in its suspense account. Did you spot...

… that?
DW no.
PG would you have liked to known about it
DW it would have been useful to investigate to see the sums of money Second Sight are writing about.

PG now bringing him back to CS bug and asks if what he has seen over the course of the last 90 mins would alter his conclusion about the way the CS bug/ mismatch was handled
DW my view has not changed.
PG takes him to the way CS bug was actually handled.

[PG looks at transcript of xe of Fujitsu chief architect Torstein Godeseth]
PG here we have issue of SPM having trouble getting authorisation to shift sums of disputed cash into suspense account
DW for that branch?
PG yes… reading from the PEAK...

… "“this office has severe problems balancing” causing a discrepancy of £6K+ - can you remember this?
DW not specifically
PG in his xe Mr Godeseth says “a horrible position to be in” a fair summary?
DW not sure why he couldn’t get the authorisation to put it into suspense...

[PG takes him to area manager intervention log]
DW these are stock transfers which went in twice
PG reading - SPM claims some sort of H bug and she’s put things through again - so H system is not showing correctly what has happened
DW yep - it’s a Riposte problem

PG reading again - other Giro deposits also not coming up on the system so that was re-keyed, doubled up transfers, SPM told to carry on balancing. Is that what in your expectation of how thi sprocess should work - is that what you expect should happen?
DW in some cases...

…. NBSC tell people not to rollover and in other cases they tell them they should and I’m not quite sure of the rationale in each case.

PG did you know about this happening at CS
DW no
PG you’re not aware of a huge amount of what happened in many of these bugs are you
DW no I don’t know about the procedural side of things
PG but you in your reports made a view on whether these would be lasting discrepancy or not

DW I formed a view that in most cases it would all eventually be resolved
PG without knowing the detail of what actually happened.
DW agrees

PG let’s have a look at what actually happened… [goes to area intervention manager report] SPMR is now concerned that he has now made a fraudulent entry on his account. See that?
DW yes that seems to be the case
PG Did you know about this?
DW no

PG and we look at what he is eventually told, which is that there is no system error and he has to report this to NBSC. and then we see the manager saying he’s told the SPM to log every call to NBSC about the alleged fault as it will help him build a case to get his kit...

… replaced.
[we move on to a new doc - internal PO email about SPM getting problems again]
PG is this H working correctly?
DW no

PG reads: since then it appears to have happened again tho Fujitsu says no flaw has been detected I’m worried about the branch being fundamentally misconfigured.
PG asks DW - does this fit in with your theory about H working as it should.

DW I would expect false steps on the way, but eventually I’d expect them all to be sorted as people got to the bottom of the problem.
[PG takes him to Anne Chambers email about Riposte Lock problem - hits some branches most weeks]
PG did you have any conception this was an

issue affecting a number of sites, most weeks FOR YEARS, when you were forming your reports.
DW the Riposte Lock problem which is the source of the chain happens more often than the chain.
PG you were not aware how widespread this was
DW I was aware of some investigations...

… by Anne Chambers and it still is my opinion the RL problem occurs more frequently than the chain which causes the storm which causes the problems.
PG the KEL was raised in 2002. it was updated in 2010. Let’s look at it.

PG reads from an entry in 2003 when the Riposte Lock was causing problems several times a week.
DW this is an example of it causing different types of problems
PG 2005 - problems occuring several times a week…
DW that might be an SPM noticing and calling up...

PG notes advice to reboot before balancing or there is a strong possibility that it will create a misbalance
DW yes
PG 2010 - problem still occurring
DW well it might be a different manifestation of the same problem or it might be a different problem

[we go to another doc by Anne Chambers in 2015]
PG it is a list of a majority of problems which “came to PEAK” but not all. This is not evidence of a systematic problem management process automatically happening.
DW that’s a rather different document imo

PG this is her doing it manually
DW she’s doing the best that can be done
PG so fingers crossed?
DW she’s a competent woman
PG and because she said this list of branches probably didn’t have losses that’s why you’ve limited your figure to 20
DW yes

PG and you say in your report it would have been picked up by batch processing overnight. This is not going to get picked up overnight, is it
DW no
PG is was first identified in 2000
DW yes
PG and it was still going in 2006
DW yes

PG and it may well have been going still in 2010
DW yes
PG do you regard this as a significant failing
DW yes
PG Riposte was part of H
DW yes
PG an important part, central
DW yes

PG now the Dalmellington bug affected 112 branches so it’s significant
DW but they were corrected by TCs or in branch
PG leave that to one side, let’s look at the failure
DW okay
PG so its important to look at how many branches are affected.

PG why did you not mention the occurrences or how many were affected. was there a reason?
DW yes because it was detected and corrected and didn’t need a PEAK.

PG in trying to assess whether there was a bug in H it’s worth looking at Dalmellington
DW yes
PG and you say in appendix D3 to your first report you’re dealing with KELs referred to by Jason Coyne. And you didn’t recognise it and you hadn’t been told about it.

DW there were three instances of it in JC’s report and they were unconnected with this KEL
PG but you didn’t recognise it either. you say in this case any error in this case will be corrected. so you conclude there’s no permanent effect on branch accounts.
DW there are ...

… three safety nets here. Notices at the time, notices and reverses or notices at monthly balancing or gets a TC.
PG you say the PEAK implies the problem may have been around for some time. it’s not referenced in your table of PEAKs
DW that’s an oversight.

PG now we’ve seen the KEL for this and it doesn’t say TC will be sent out.
DW no
PG so where did you get your information from that it was done.
DW I would expect it
PG so you inferred from a KEL that didn’t say it that TCs would be issued.
DW you have to make inferences

PG and you did that on many occasions across various KELs
DW yes
PG now mr coyne found this bug by himself, didn’t he?
DW yes
PG he concludes the fact that ATOS made changes to the H system is consistent with their being a H bug...

PG So JC was trying to identify whether there had been a bug which impacted branch accounts. He seemed to think it was relevant.
DW well I was looking for bugs which had LASTING impact on branch accounts so we had slightly different criteria.

[PG onto new doc re Oct 2015 - “very strange” glitch but TC can’t be issued to outreach as there is no unique number… he moves onto another doc which is an email.. looking into the Dalmellington error in Horizon]
PG it is an email re a blog by @Jusmasel2015 it says it is...

@Jusmasel2015 … different from Sparrow… and raises issue of lax handling of 24K… urgent review ordered and then an order to stand down.
DW who is this email from?
PG it’s clear through 2015/6 there was a lot of investigation into this. was your attention drawn to this at all ay any point...

… in your instructions?
DW no
PG was the first time you came across it when JC alerted you to it?
DW think so

[we go to Torstein Godeseth;s 2nd witness statement where he talks about CS, receipts/payments mismatch and Dalmellington]
PG you expressly relied on Mr Godeseth’s evidence on those three bugs
DW it didn’t really add to my knowledge of what I had already thought about lasting...

… impact.
[PG now takes him to his expert report]
[I was about to say we were whizzing around and J has just asked PG to slow down]
PG short point is that you don’t touch on Dalmellington in your report - why?

DW because it didn’t have a lasting impact on branch accounts. It’s not an important bug
PG isn’t the scale of it important?
DW Dalmellington never came to Fujitsu - for good reason.
PG [takes him to a Fujitsu presentation] Branch Outreach Issue (initial findings)

PG two issues in play (listed) clear findings of 112 branches affected and then on p7 we see detailed preliminary findings
6 incidents - 2011 - 0 calls raised
9 incidents - 2012 - 0 calls raised
7 incidents - 2013 - 0 calls raised

PG did you notice the £24k and £2.5K?
DW these were, as we have heard, resolved
PG do you not think that given the scale and effect of this bug it should have featured in your report?
DW my opinion is and remains that remming errors get corrected

[DW asked to stand down. evidence finished for the day. we are now going to get into the issue of disclosure]
PG at 1.02pm today we got the missing ARQ figures - 2013 to 2014 and it turns out there were more than the allowance of 720 used in that year.

PG and the matter I raised this morning…
J that’s about the extraction of data in more usable form
PG correct it was disclosed in separate spreadsheets. it was then disclosed in a new and different form and I cross-examined in the difference between the two.

PG This is a new doc and we rely on the difference
J that’s a point for submission and argument and for me not xe
Mr De garr robinson rises - and says he was responsible for the initial format of the initial MSC submission - he asked for it for his xe

but didn’t get a chance to use it. He thought it would be helpful in the trial bundle.
J accepts that this was said in the letter he has received but is grateful for the repetition
Mr de Garr just at pains to point out: There is no conspiracy here.
PG says we were...

… seeing documents which extracted the good things from a source document which left all the bad things out.
[now on to late disclosure of 1000s of documents]
J this much, this late in proceedings needs explanation
DG can I do so now?

J well I’m going to ask for a witness statement, but anyway you can
DG Fujitsu found thousands on an old database and in a matter of days steps were taken to make the court aware of it.
J okay well no one has quite explained this in the way you just have

J but I am still going to ask for a witness statement by 12 noon next Wed. With names of the individuals involved at Fujitsu who found these documents and how.

J adds that he needs a discussion about how the rest of the year is going to pan out after closing submissions on 2 July.

Which suggests to me that there could be some date slippage (or head-cracking to ensure no date slippage) with the rest of this litigation. Next trial is scheduled for November. Currently.

[judge rises]
Okay that’s you’re lot for today. I’ll unroll the tweets and get a report and the transcript up on postofficetrial.com as soon as possible.

If you liked reading any of this and you haven’t already, please consider a small donation. There’s a tip jar on the website: postofficetrial.com
Contributors who put in more than £20 will get added to the secret email list which means you get extra (usually quite gossipy...

…) information about this litigation.

Right - we’re back at 10.30am tomorrow for Dr Worden’s last day of cross-examination.

And it’s Friday, thank goodness.

@threadreaderapp unroll pls

@threadreaderapp unroll pls