Thread by @nickwallis

We’re back for the afternoon session of Day 7 of the Horizon trial - part of the Bates v Post Office litigation.
#postofficetrial
Mr Godeseth from Fujitsu continues to being cross-examined.

Follow this thread for live tweets from court.

Anything tweeted is a summary and paraphrase of what is happening in court. Nothing is a direct quote unless it is in “direct quotes"

Patrick Green QC (PG) for the claimants is cross-examining Torstein Godeseth (TG)

they are discussing the early days of Horizon. It was a difficult birth back in the late 90s. DSS and Post Office were clients. DSS pulled out.

ICL Pathway (later taken over by Fujitsu) nearly blamed for the disaster (big waste of money) ICL blamed having two clients and changing brief for delays and changes.

This is not new news - it has been documented at length in parliamentary documents.

publications.parliament.uk/pa/cm199899/cm…

PG brings up document talking about pilot rollout of legacy Horizon. Gareth Jenkins is mentioned.
TG says he doesn’t have a firm memory of this as he was not working directly on the project.

PG by the time you were involved in rolling out Horizon online you were aware of the many problems with Legacy Horizon
TG Horizon online was about refreshing the system. I don’t remember it being about a whole load of PEAKS
PG did you have an awareness of any problems with...

… legacy Horizon by the time you got involved.
TG there will have been problems but I can’t remember.
PG important PEAKS would have been brought to your atention under legacy Horizon
TG I would have thought so, yep

PG we got to a document “whilst PO has considerably increased expectations on cost-reductions” the business was more open to a more retail system “improved capabilties”. but the new business case is almost entirely based on cost-reductions. Is that right
TG not going to argue

[so this is about designing and migrating to Horizon online (HOL) on a cost-reduction basis]

PG in this doc it seems PO is not getting involved in redesigning HOL for improvements but cost-reduction…
[the doc mentions keeping a lot of legacy systems in place]

PG explaining the big change in Horizon to take transactional data off the counter terminal and store it in a remote data store. This would or should improve data security and crucially allow realtime transactions between clients and customers.

PG asks about phase 2 of the rationalisation process
TG lists what it would have been and says it was never implemented.
PG quotes from report existing architecture of Horizon as being robust but not suitable for current business and technology drivers in use today.

PG is that fair?
TG yep

PG goes to a later part of the document he’s reading from which is a joint Fujitsu report describing the history of Horizon online and how the initial plan to implement it in 2005 was abandoned.

PG original Horizon online was called HNG. The solution which was implemented was called HNGx. HNGx was a much cheaper solution, no hardware upgrades etc
...

…
Moves to a later report - Horizon online, legacy architecture - slow and expensive to use - 13 year old technology - difficult to change… Horizon is a system “wrapped up in barbed wire” hard to make changes to
PG is that fair?
TG I think changes could be made

[I think we can all see the “long point” PG is trying to make. slowly. but if you want to read the techy guts of the stuff being discussed here I will post the full transcript around 6pm today]

PG goes to a new doc called "Horizon Online Programme update” by a Mark Burley (TG’s old boss)
PG do you see “high volume pilot suspended” and “NFSP raised concerned but remain supportive” - do you remember this?
TG no I was working on a different project.

PG talks about “Red Alert” on Horizon
TG knows about all this - Fujitsu was in a state of Red Alert when I went across
PG it sounds serious
TG it was. there was a problem with Oracle
PG any others?
TG can’t remember I was focused on the Oracle problem

PG reads out a PEAK generated by a SPM who has lost £1000 cash withdrawal from the Horizon system without it showing up on the records. Would you know about that?
TG no I was back in the PO at that time
PG did any of this come to your knowledge
TG no

PG reads out another part of the PEAK - numbers randomly doubling. any familiarity with this?
TG no this is a problem with Horizon online and I wasn’t working on that specific part of the business then.

PG there are quite a number of apparent problems they are wishing to raise in this PEAK, aren’t there.
TG this looks to me like it is the BT PEAK isn’t it - there certainly were issues with the software in the early days of HNGx (Horizon online)
PG that doesn’t leap out of your..

… WS does it?
TG no
PG and you know that’s what his Lordship is here to do - determine if there were issues
TG yes fair enoug
PG so why not mention any problems then
TG I am giving an overview of the system

PG in you WS you talk about migration from Horizon to Horizon online - this was the biggest overhaul of the infrastructure. Now - components remained the same and lots of additions made over its life. yes?
TG yes
PG is Horizon online an end of life rather than ...

a reinvigouration.
TG completely disagrees - moving off ISDN [lists many other changes]

TG the legacy version of H was far more suscepitble to communication glitches.
PG so HOL saw the quality of comms infrastructure improved. Yes?
TG yes

PG moving to a new doc. “the HNGx system is end of life, running on unsupported software and therefore needs replacing” is that right?
TG yes
PG and that was written in 2017
TG yes
PG “branch technology is aging and unreliable” fair?
TG yes

TG NT was released in 1994 and any technologist would tell you that was too old but it continued to work surprisingly well
J that’s Windows NT yes?
TG yes
[we are now talking about the migration from Windows NT to Windows 10]
PG there’s nothing in your WS

about the state of the technology we’ve seen from these reports.
TG that’s a fair point.

PG now talking about Computer Centre entering the fray. Fujitsu provide the software to CC, CC send it on to the branches. This happened in 2017.
TG yes

PG now reading about data sources from TG’s WS "The vast majority of transactions are manually entered by a user in branch at the counter, by pressing icons on the touchscreen, keying …"

….in the transaction on the keyboard, scanning a barcode, scanning a magnetic card or some other manual interaction with the system. These are referred to as "counter transactions".

TG admitting it doesn’t know a huge amount about that. He is asked to explain a footnote in his WS: “TCs are incepted in Post Office's POLSAP system before being communicated to Horizon, via TPS to the BRDB."

TG he can’t. but he would have been able to work it out from what it said
PG so you got it from someone else
TG yes
PG why was there no dispute button on Horizon
TG PO decision
PG why did they decide

TG to get the speed of data flowing through - the inference would be if you had a dispute you would call up the helpline and register your dispute.

TG the principle was there to say this is the correct figure, accept it and THEN argue the case for disputing it outside the system

[we move on]
PG describe to us how a scratchcard goes through the system. A scratchcard is activated on the lottery terminal. That is sent from Camelot to Fujitsu
TG with Camelot it goes to Credence
PG okay so from Camelot to PO Credence system which then goes to Fujitsu...

TG and we send it down to the branch
PG so it goes via a third party
TG not via - the first Horizon knows about is when we sent to branch
PG via credence
TG and because it’s come in via credence it gets stored in the branch database

PG so in the morning they get some TAs and they have to be accepted
TG yep
PG then they enter the branch accounts and that is the data THEN captured by the audit system
TG yes
PG and that’s true for other bits of equipment in paragraph C

TG broadly yes
PG and that is the data that goes into the audit store
TG yes at the counter when they accept the TA it goes into the store
PG when we look at the data we’re concerned about in 17.2a in your WS you’re looking about touchscreen input.

PG so we’re talking about the fact of a button press and the significance of that in the data table is something pops up in the basket. eg you hit stamp and the cost of a stamp appears in the basket
TG yes
PG so in the transaction data there is what the SPM has done...

… in terms of keystroke
TG I don’t see it in those terms, we see it in terms of a basket and we put that basket into the audit trail.
PG sorry the result we see is the fruit of two different pieces of data. The fact of a button press and the data transferred.

TG yes - some are much more complicated - weigh scales, lots of screens and button presses etc
PG indeed and that’s captured in the audit store and the database
TG simultaneously. the audit store is inside the database.

We are on a break.

Taking over after the break is Kim - who will be subbing for me in April. Be nice to Kim.

Kim has been shadowing me for the afternoon. I have explained most things about tweetdeck, twitter and all the usual stuff, but she doesn’t usually work off Macs so she is just getting used to the interface. We also have a duty to be...

… accurate in our reporting as a priority (these are, after all High Court proceedings) so Kim HAS to be accurate before she can be FAST. If that means you don’t get as many tweets, please br understanding. I will still be in court with KIM.
Say hello to KIM everybody.

And wish her luck!

Hi everyone - yes as was pointed out to me earlier, Nick IS the fast tweeter in the known Universe, and I have big shoes to fill. I will do my very best to do this justice though.

Resumed. PG Counters held data in the message store and the correspondence server also had a message store?
TG Riposte was responsible for replicating data. The audit server is outside of riposte

TG the audit data stored was in Riposte attribute language. As simple as a copy of the message store.
TG Escher (?) provided Riposte software. Provided support for the software.
Conscious of PEAKS but not aware at the time.

[long discussion re transaction data] issue with two commits that shouldn’t have happened, duplicate entries. PG were you aware of any problems with items with a duplicate JSN number being committed to the database?
TG No

re data integrity checks revealing issues. Were you aware of problems in the Misra case?
TG No

TG says he was asked to put information in his witness statement.
PG when you were shown the info to put into your ws you must have been aware it only went back to 2014 you must have had in mind what this ws was supposed to deal with
TG I’m afraid I didn't do my job in that case

ie the statement didn't go back to 2010

TG taken to a document he says he hasn’t seen before.
A document re supporting delivery of HNGX. 'A problem is defined as the unknown underlying root cause of one of more incidents’. PG Important to make informed assessments of where problems lie on the information available
Yes

PG and important to be able to track and analyse
Yes
Discussing Mr Coyne's report (Claimant’s independent IT expert). …PO should be aware of all recorded bugs - seems to have inferred that problem management process had been implemented and there would

be feedback to Fujitsu.
Agreed.

Were you aware as a key person at Fujitsu that Mr Coyne had asked for key docs to be provided based on what he though would exist as part of the problem management data? PO response to request ...

… - PO objects to this request - data not recorded without detailed analysis at disproportionate effort and cost.

PO document saying that records referred to and requested by Mr. Coyne do not exist … (re problem management data).
Talking about internal policy reviews of requests for this info …

J taking witness to page 1 of this document re who had authority to sign it off.
PG Re Legacy Horizon reporting system and following the previous reporting methodology - witness says he doesn’t know what that reporting methodology is - I got that from Steve

PG Wouldn't PO want to know the sort of information conveyed from that sort of reporting procedure?
TG I would have thought so.

J says having difficulty with following this information - does the witness mean these metrics are only available for the years in his statement? TG says he doesn’t know. TG says he spoke to Steve about whether problem management was implemented and he said no.

PG … this (doc) appears to be talking about a review of problems and ...

on the face of it it looks as if there are considerations about for example in specific problems that have occurred, as if there is a problem management review doc with some actual problems in it although it seems to be issued for approval. Do you know about this? TG No.

Talking about problems with First Rate transactions (didn’t catch it but will look at transcript later because from memory this could be important).

A fix developed for duplicate when remming in. Report doesn’t capture how many branches affected or how long it took to find or sums involved.
TG No
PG Not a robust way of recording the problem
TG Fair comment

Please check the blog later once we’ve got the transcript to make better sense of some of these tweets!

Talking about a document re transaction correction journalling. A tool being developed for Horizon Online. Allowing FSC to make changes and audit changes. Warning! Use of this powerful tool has inherent risks … causing serious problems in branch database.

PG Are you familiar with this?
TG I’ve never seen it used. I’ve had lengthy conversations about it.
PG It would allow FSC to insert balancing records, auditing but no updating or deleting of records.

[PG lists a bunch of techy things related to this tool]
ie where one half of a transaction is missing and the other half can be inserted.

[discussion re Fujitsu data re corrections and actions taken which it says would have no affect on branch accounts]
TG this would have been written by others but I’m fully aware of it. Transcript at 16.07 for those among you that can understand this info.

Talking about whether it would or would not affect branch accounts. PG It’s clear from this design that there would be no updating/deleting of records in the branch database.
TG Yes it says that

PG this tool is designed to insert?
TG yes
PG the use of the tool has gone way beyond its original design?
TG agrees
J asks for clarification about the Oracle term ‘insert into’.

PG Re balancing transactions and adding a transaction to a branch accounts - letter says they are rare and only been used once. Letter in 2016 can you remember being asked about this?
TG No

PG Re system crashing 10 March 2010 Cheryl Card suggested correction to negate the duplicate £4k transfer out. Using the transaction correction tool. Did you hear anyone asking for authorisation for this to be done? No
Did you know PO’s approval had been sought for this ...

… transaction?
TG No
PG PEAK raised re transfer out doubling up from £4k to £8k. The whole point of JSN entries is they should not duplicate, yes?
TG Yes

PG re a PEAK and Cheryl Card and templates so scripts don’t have errors when they’re deployed. OCP - Operational Change Process. OCR - Operational Change Request. One for front end and one for back end?
TG Don’t know.
Transfer in details incorrectly doubled up when written

to BDRP - this looks like one use of this tool?
TG Used if it’s a one sided transaction.
'Used TRT to insert the missing data’.
TG Inserting into a totally different database, not the branch database. Using the transaction repair tool not the transaction correction tool.

PG [technical data re half the data missing in the basket] asks for witness to be provided with a copy of this PEAK overnight so he can consider this information. Discussion re how is that going to happen when everything is digital?

Court is going to rise now, when it will be goodnight from me, and goodnight from him. Thanks for your forbearance. Nick will make sense of my tweets in the blog.

@threadreaderapp unroll pls

Thanks all. I will pass on your compliments to Kim.