PG = Patrick Green QC
H = Horizon, the PO’s IT system
PO = Post Office
JC = Jason Coyne, the claimants’ IT expert
DG = Anthony de Garr Robinson QC for the Post Office
H1, H2 etc = the Horizon Issues
(what this trial is all about: postofficetrial.com/2019/06/horizo…)
PG that very title suggests just one event.
PG takes him to a line which says: "Post Office and Fujitsu were concerned that Horizon should not induce the user to make such an error, even in rare circumstances...
"To make this point more clearly: in my opinion, for many KELs, it would have been obvious to Fujitsu that any error in branch accounts...
PG saying how he goes from the specific re Dalmellington to the general re error-fixing and then back to specific re Dalmellington.
PG points out that nowhere...
PG There are two reasons for this. You didn’t know or you did know and you chose not to mention it.
DW the latter - over the whole sequence of several years...
PG would it have been helpful to note errors which had potential to have an impact on branch accounts
DW they did have a TRANSIENT effect, but not a LASTING one
PG could have mentioned...
DW I could have mentioned the 112 errors which happened over the years compared to the 20,000 manual errors made every year, but if I didn’t I’m sorry
J where did you get 20,000 from
DW the table which says there are 100,000 errors a year and approx 20,000 are...
[DW is much less chuckly old prof which he has been over the last two days and quite grumpy today - I prefer him like this. Spikier]
[we move on]
So, there were no lasting effects on branch accounts.”
PG takes him to appendix of his first report...
we are still on the Remming in bug
PG asks if DW has looked at the PEAK behind the KEL behind the bug he cites in his table
DW says he hasn’t read the PEAK
PG it shows an error of £20K, which suggests its v serious.
PG did you look at the other PEAK in this KEL
[we look at it]
“this is not another example of the duplicate rem system which we have seen in the past"
PG if that is the case - if GJ was right - that then was a bug in the system.
PG if he was wrong, it was a defect for allowing...
DW i see - something has gone wrong yes
PG so it’s a fair question
PG and it affects branch accounts
DW you get a transient error
[PG lets that hang - this is the battleground - DW introduced the idea of LASTING impact on branch accounts...
[the public “gallery” was empty until now. Sue Knight has just walked in]
PG this was a lasting bug
DW but they took a decision based on its frequency that it was not worth addressing as it would get picked up in the remming process
PG it was a lasting bug
DW it might have got fixed in the next release
DW yes you’re right and I shouldn’t have done that.
DG stands up - could m’learned friend distinguish between a lasting bug and a bug with lasting effect?
PG I think DW knew what I meant
DW I did.
PG did you read that KEL?
PG let’s have a look - 33 in a couple of weeks of testing
DW but potentially more
PG you hadn’t spotted that?
PG did you look at the PEAK for this bug
DW doubt it
[PG goes to the PEAK]
PG and that is an example of User Error Bias
PG you didn’t see this, did you?
[PG reads on] lots of incidences etc… you didn’t read this?
DW I read the KEL and formed the opinion it was a transient effect so I didn’t drill down.
[we are looking at a KEL with more than 2000 PEAKS in it from 2010 to 2018]
PG if something is handled by NBSC and sorted straight away - it doesn’t make it to F support?
DW no - you can fix yourself, and go to NBSC and yes then get to F support.
DW not sure - some go via MSC…
PG well let’s look at JC’s report - 2472 PEAKs which are potentially listing a discrepancy in branch accounts.
DW its about temporary discrepancies...
DW this issue between the experts is that JC believes a lot of the failed recovery can be automated and I don’t - it’s very delicate…
[we move on to reversals]
PSteed2847N (the KEL associated to the PEAK record (see Supporting Evidence column)) records: “the PM should be contacted to say that the problem is due to a software error and that they should ask the NBSC for balancing procedures”.
[they have a chat about this]
DW They copied my report
PG so where did you get the info about this bug from then
DW from the KEL or PEAK
[we move on]
PG reading F engineer report: so the mystery is how a top up of £30 appeared in a basket without being credited to anyone. We’ve heard about this before...
PG breaks to say that we’ve seen evidence before of F engineers closing calls when they don’t have evidence
DW yes a black mark against support for that
PG continues reading:
PG so we’ve got actually keystrokes listed here
DW I don’t think they are, I think they’re event logs
PG so you think, somewhere in the system you can find these keystrokes
DW pretty sure. yes
PG this was also disclosed on 30 May so you may not have seen this before…
J what’s the date on it?
PG this doc is april 2018...
PG this document - having identified the bug and the fix. Is dated...
DW so afterwards
PG yes and we see that whoever was writing the PEAK [Henk Bakker? sounds like] - the bug is a mystery to him.
PG so there’s been a detailed fix in April
DW it’s a different branch. it seems a similar problem
PG it’s the same script in both docs
DW did we look at that
PG we did - well we looked at the name. it’s the same
DW it is
DW well fixes do go wrong. That does happen.
[I’m playing catch up. I was distracted by the coffee cup]
[okay we’re going through DW’s statistical calculations.]
PG in terms of number of affected branches by particular bugs… receipts and payments mismatch - 60 to 62, Callendar Sq 30
DW doesn’t accept Dalmellington
PG okay accepting that - if we were to look at the branches affected by bugs…
PG so 148 bugs affecting 48 branches each - so that 6960 bugs, multiply that by 258 and we have 30+K, or if we go to favour the PO, we have...
PG now there are only 550 claimants. On actual incidents of bugs affecting branch accounts - that is totally consistent, is it not?
PG we’re back to Penny Black
DW no we’re not -we’re doing maths - 32 x 50/3 = 1 occurrence of a bug to each claimant during their tenure
PG thank you
[! - DW just seems to have accepted the maths...
[so this trial really is about error-correction - countermeasures - suspect claimants will say they are not as “robust” as DW thinks they are or
[we have moved on]
PG There were no service audits of H before 2012
DW non disclosed
PG there were the 2010 and 2011 E&Y audits… let’s go to the 2010 audit...
PG they say: "we continue to face difficulties getting accurate info from Fujitsu…” had you read this?
PG reads: in relation to control observations POL has made significant changes to its IT environment including bringing Credence into scope for the first time - so this suggests Credence was never part of any audit if any were taking place.
PG reads: so we established a key person at F to deal with the difficulties of auditing H, but we had real problems obtaining information
PG now - if you are record-keeping properly you should be doing so in real time
PG and they weren't
PG reading out level of top level access to H system - which had been going on for some years - is dramatically stopped.
DW yes I don’t know about dramatically
PG so you’re trying to think up excuses for Fujitsu
DW no I’m just trying to think about what might have been going on
So they’re expressing...
DW yes about governance (ie what control PO have over their own system) and control environment which is access…
PG you heard there were a number of concerns expressed in relation to generic accounts etc - let’s go to the PEAK on APPSUP
PG we’ve seen this several times before. This starts in the progress narrative 1 Feb 2011 - SSC (F support) database users do not have the correct permissions.
DW no I didn’t
PG and it’s a very powerful role
PG JC identified it and if you look under Dave...
PG and your definition of robustness requires a strong policy which is carried out
DW yes - but it’s proportionate
PG if you have an acceptable security policy...
PG and if the hole in the security policy is a role which is extremely powerful - it magnifies the risk
DW yes but the F lot were pretty trustworthy
PG you haven’t met them
DW but I’ve read loads of PEAKs and KELs...
PG reads down - so there’s a tiny tidy up task script produced by dev, but the people in SSC were using the APPSUP role so they...
PG so there’s a discussion lower down and it states that SSC users have ALWAYS been APPSUP.
DW either things we’re not supposed to do, or not doing what we usually do
PG neither of those two categories are hermetically sealed?
PG now- you hadn’t picked up on APPSUP as a role in either of your reports
DW “standing” MSC?
PG were you aware of what that was?
DW this is recapping what we’ve been through
[PG reading from PEAK about this security issue]
PG it looks at this stage there is tidying up going on with users, but there is a release PEAK which stops NEW people having access to APPSUP role
DW don’t know enough...
PG traces a line about continuing cocerns re security access up to 2015. So the security policy was still being breach up to 2015.
PG and you hadn’t picked up on it because you hadn’t read that PEAK?
PG and you hadn’t reported on APPSUP either
DW isn’t quite sure how the table they are reading from marries up
DW so all of this is saying PO need to be more involved or formally need to record stuff
PG there was no internal control to authorise fixes - increased risk..
J asks DW how would you described script to a layman
DW a sequence of steps to execute a computer business process
J line of code?
DW yes a line of code
[the short adjournment]
Sue sees this as a human act of magnaminity amid the madness of her prosecution. I see it as something rather different. She carries that man’s card in her purse as a reminder that good people exist and as she showed me it...
Sue travelled six hours to get here today and has to leave early to travel six hours to get back tonight. She felt it important to bear witness to these court proceedings...
Sue and her partner live in rented accommodation. They were forced to sell their house because after losing the income from her Post Office it became unaffordable...
Some of the claimants’ situations are pretty bleak. They are the only ones in this courtroom not getting paid...
Anyway. Thought I’d balance up my usual flippant tweets with something else for changes.
And obviously I haven’t checked Sue’s anecdote with the man who came and shook her...
But now I have a photograph of his card, I will.
Dr Worden has just admitted something PG has put to him has made him change his opinion on something. I’ll make sure I’ll look at this when I get the transcript tonight. Sorry.
PG is still talking to DW about security controls and RR’s assessment of them.
PG has pointed out that DW disagrees with them based on...
PG points out the earliest services audit is 2012 and RR left F in 2004.
DW good point.
PG an apology due to RR there?
DW I’m sorry I shouldn’t have put that in my report.
PG when you say visible - would these be available to SPMs?
DW oh no.
PG so let’s have a look at some logs
PG I wasn’t talking about those
PG are you talking about ARQ logs then?
PG what are you talking about
DW there are tools available and lots of people spend their time looking at them - you can pull them out on the day
PG on the day? and would this include keystrokes…?
[we go to the issue of remote access]
PG you agree with JC that PO could do more or less anything...
PG and you looked at the issue of access at the counter - [reads from doc] - it is difficult for experts to say x or y never happened because of the complexity of the system
PG that pertains to what you were saying about plumbing the depths and it being a swamp...
PG okay let’s explore this.
PG this is about F access to H. Did F have the ability to insert or inject and the answer is yes and you explain why
Fixes is yes and rebuilds is yes
b) without knowledge of SPM and you say “no"
PG Can we focus on this?
DW I think it’s been superseded by the joint statement
PG so you can’t exclude it
DW don’t know
[weirdly the public gallery is now rammed - a bunch of people have just walked in and plonked themselves next to Jo and Sue, and there are loads on the PO side. Both were virtually empty this morning]
PG is difficult
PG so even if PO gave the SPM ARQ data
DW I don’t think they should be given ARQ data
PG Okay so if PO requested ARQ data to show and explain to the SPM, the SPM would have to have remote access...
[ we go to the Post Office response to the Panorama programme I co-produced on this story]
PG reads: there is no evidence of transactions recorded by branches being altered remotely
DW: didn’t see the programme
DW I didn’t...
J what about the statement that branch transactions can’t be changed
DW the DVA can do anything and we can’t say what can’t be done - it could be but there’d be a discrepancy between the audit store and the branch
J okay. Mr Green - I don’t see much value to me in questioning an IT expert on the Post Office’s public position on this subject.
[PG talking about logs of F access to horizon and the data trail - they ...
PG and you looked at the post-2015 access logs to see what more info there was
DW I didn’t get very far with it because of the way the info was set out
PG goes to TG’s WS which states the level of access post-2015...
DW agree - I didn’t go into privileged access logs much
PG and you say the same about the Managed service user logs
DW yes, but I”m less sure about that now having...
PG change approvals are meant to be in the OCPs and OCRs
DW not sure how MSCs, OCPs and OCRs overlap
PG they ought to be somewhere in there?
PG no one id’d the APPSUP role to JC - he had to find it himself.
DW yes they do
PG and in response TG said to JCs findings - he confirms the APPSUP role - is privileged user access to BRDB but there is no evidence
PG goes to Mr Parker’s WS and he comments on APPSUP - it’s not a new type of remote access and refers across to TG - those logs suggest the APPSUP role has been used 2175 times to make emergency amendments to the BRDB
PG now neither Mr Parker or TG looked at the logs, and you’ve agreed they’re hard to look at, whereas JC has and got this answer
DW yes but Mr Parker works there and would know how SSC works better
PG you agree that if these logs are hard to read...
DW on this disclosed evidence, agrees.
[we now go to OCPs and OCRs]
PG notes the four eyes protocol...
PG notes that the overseer of this engineer’s work in this particular document was himself - he’s making a change to hundreds of branches and he’s monitoring himself
DW yep, looks like it
[okay we’re back]
PG takes DW to an OCP 2 March 2009.
[then goes to another one] and we see raised AND...
So it’s not recorded, the four eyes, always, in the way you would expect
[we move on]
PG in a well-maintained system, people shouldn’t be working from draft docs which haven’t been approved
DW yes, but it happens much more nowadays - phases overlap...
PG so - there was a perceived need for the tool, that it was a powerful tool, there was a need for protections, clear guidance and a table to which it writes so its use can be audited.
PG we’ve seen from AC about wanting to go off-piste...
PG takes him to a 2010 PEAK to amend the TC tool
DW to amend the templates, more precisely
PG if there was no...
PG the TC tool was carefully designed to meet a need. it’s common ground it was only used once and there were occasions where it could have been used and the APPSUP tool...
DW got to be careful, because F only would go off-piste if they really needed to
PG how do you know. the APPSUP tool gave them such freedom that they didn’t need the TC tool
DW but they only did it if they really had to.
PG let’s go to JC’s report...
DW I think most inserts were not TCs
DW the tables it has been designed for are not the ones which could affect transactions
PG but you could use it to affect branch transactions
DW it was only used once for balancing transactions - it may have been used in other ways
DW is pretty sure he knows
PG but we know F engineers very much valued having the APPSUP role
DW they needed it sometimes
PG this is a doc dated 24 May 2019 so absolutely no complaint about it being disclosed on 30 May.
PG actions taken - 32 people who had already left business were removed
PG next steps - no current owner of the process, but the no one has owned data integrity for several years
DW no something’s fallen between the cracks here
DW F’s preferred method is to fix things quickly, so if things are fixed quickly we won’t see much documentation
DW no I was pointed towards it
[n further questions]
[reexaminations by DG for the PO]
DG you were asked questions about the gaps in the...
DG it was put to you you’d put false evidence about that later and the premise on which that was put to you was that...
DW i remember
DG it is a surprising suggestion and I wish to explore it. The info you got this from was an Angela van den Bogerd witness statement…
[ he does - I thought this was a...
DW F is not the column to use and that is blindingly obvious
[we move on to ARQ data requests]
DG it was suggested to you that there were various costs attached to ARQ requests.
DW the point was that if you’re doing 100,000 ARQ requests a year charging £200 a go is ludicrous.
[DG says the line of qs...
DW I agreed the cause had been around since 2000
DG the impression may have been given that it was undetected until 2005 and I want to explore this
[reads from PEAK and says…]
DG what should we infer from that
DW well we know F knew about it and they were trying to get Escher to fix it and had been doing so for five years.
DW loads for all sorts of issues
DG but large numbers by the time of your first report?
DG how many
DW a couple of dozen
DW saying manifestations of the CS bug would have been double entry accounting which would have got picked up by the back end systems
DG what does this PEAK show us about CS?
DW that the Riposte problem would be easily picked up. Easily.
DW yes CS as a manifestation of the Riposte problem too
[we move on to “grey” power problem]
DG you were asked some extraordinary questions about this today - your comments on JC’s first report about KELs… if we go to your analysis...
DW I imagine it was the KEL
DG well let’s have a look - under the heading May 2010 - there is a discussion about black and white comms
DW grey is not in...
DW I would possibly have made up that word myself [chuckles]
[no further questions]
J you were asked about privileged user logs - you said they were unclear - you said you couldn’t get to the bottom of it in the time you had - how long did you spend on them?
J relatively recently you served your third report on the court. I am not going to ask you any question on its contents, but how it happened.
J the docs you relied on are PEAKS KELS OCRS OCPs etc - when did you first get access to those docs
J a while
J have you ever served one of your expert reports on the court directly before
DW no I’ve had it done to me but I haven’t, no
J you sent an email to my clerk
J by whom
DW the Post Office lawyers
J and did you draft the covering email yourself?
J did you show it to anyone or have it approved?
J okay thank you. your evidence is finished.
oral closing is 1 and 2 July - a day each
J 1st and 2nd of July I look forward to seeing you all then.
You’ll find both here postofficetrial.com where you’ll also find a wealth of reports, docs and info on this trial.
This means new stories going up on postofficetrial.com even though court...