Thread by @JonHall_

I've been working through the recent independent report into a huge IT migration failure at TSB bank in the UK. The full report is here, and there's a lot to address, but a walk through the executive summary is revealing, almost jaw dropping at times.
tsb.co.uk/news-releases/…

Important background: the bank was demerged from another UK bank post financial crisis, and acquired by a Spanish company, Sabadell. For a while they rented their spot on the old bank's platform. The acquiring bank's tech function, SABIS, handled the migration and a new platform.

A walk through the executive summary with a few drill downs is very interesting.

Note to screen reader users: this is a scanned PDF so difficult to copy text to the ALT text, but they're all pictures of text so I hope something can be done with that. Apologies if not.

Right from the start, the decision was made to do a "big bang" migration and there was a lack of consideration of other options. #TSBReport

It seems like the piloting work that was done was inadequate and probably a waste of effort. #TSBReport

Both the bank and the supplier (SABIS) seemed to be keen to settle on the same target date. The date was defined without up-front understanding of *how* they'd hit it. It was just assumed that they could. #TSBReport.

The arbitrary date selection seems to have been based on assumptions driven by past projects undertaken by SABIS. (The platform was built in 2000 in part to handle acquisitions and migrations). But they didn't account for TSB's particular requirements. #TSBReport

This was in part due to underestimating the need to customise for TSB. The board didn’t understand the extent of or SABIS's capability to deliver. The report asserts they’d have behaved differently if they’d known (although I can't see much basis for that conclusion!). #TSBReport

Whatever protections were put in place didn't prevent the very simple issue of trying to migrate before things were ready. #TSBReport

Functional testing suffered significant overrun, leading to unplanned parallel functional and non-functional testing. #TSBReport

Sept-October 2017, delay was agreed from the original November 2017 target date, but replanning wasn’t comprehensive. No lessons learned were carried into the new plan.
#TSBReport

Internal audit of the project seems to have badly failed. #TSBReport

The board also failed to heed the warnings that were very obvious that the project was overstretched and the platform wasn't ready. #TSBReport

Threaded through this seems to have been a situation in which deliverables had been overcommitted, and people did not take responsibility for communicating the lack of readiness. #TSBReport

An obvious and damaging impact of these compressed timelines was that testing was shortened, over-parallelised, and incomplete. #TSBReport

And again, people seemed not to speak up about those issues with testing. #TSBReport

Non-functional testing (particularly performance) was badly inadequate, perhaps primarily due to the timescale compression. Performance testing was not carried out across both datacentres despite there being material differences in configuration between the two. #TSBReport

This one is one of those jaw-drop moments. When performance tests failed, the targets were downgraded, which allowed the tests to pass… at a level lower than the actual production load at go-live.
#TSBReport

The board missed key chances prior to go-live to reflect on the bad situation. But it was also misled about the scale of issues. #TSBReport

I was curious about that defect count, and drilled into the main body of the report. The internal communication was obfuscating and misleading. It implied that most of a total 85,000 defects had been fixed, and hence only 1% of defects were being carried forward...
#TSBReport

…but actually, the numbers were not equivalent. There were 800 pieces of functionality each of which had one or more defects.
#TSBReport

SABIS's lack of readiness to support the platform was a key issue identified in the report. They had already struggled through smaller preliminary go-lives. #TSBReport

SABIS did not have sufficient support and remediation capacity in place, but this came out only after the event in reports by both SABIS and TSB. #TSBReport

Jaws ready again. *Prior* to go-live, however, it seems TSB had simply relied on a written assurance by SABIS’s country MD that they were ready. They didn’t receive any evidence to back this up but instead proceeded apparently on the basis of this letter.
#TSBReport

Here's that letter. #TSBReport

There were major warning signs but the supplier appeared to get a soft ride from both TSB and auditors. For example, 15.13 notes a report from KPMG Spain just two months before the major failure. A lack of capacity management controls was brushed off as minor.
#TSBReport

And is this our old ITSM friend, the CMDB without purpose?
#TSBReport

The bank didn’t perform adequate upfront due diligence or exercise the extent of their rights to audit ongoing work by SABIS.
#TSBReport

The report suggests the bank was over-integrated with the supplier… directing operations more than a typical “customer” of such a relationship. I’m not sure about this one - was the relationship too close or too distant? The report seems to be suggesting both.
#TSBReport

Although this was a project of unprecedented complexity for both TSB and in the context of UK banking as a whole, the bank didn’t seek sufficient independent advice to validate the capability and work of its supplier and its own executives.
#TSBReport

Finally, the report asserts that independent advice might have led to different decisions on supplier, platform, approach, and also would have prevented the project carrying on regardless when it was failing. #TSBReport

If nothing else, this is a fine piece of evidence for the "there is no single root cause" camp.

I'm also sorely tempted to point out the many similarities to Brexit, but I'd better not.

My feelings on this: it seems like a classic failure of a big waterfall project. Maybe circumstance forced the decision to structure it that way, but it's a good example of why not to do it. Probably had a big human cost along the way in terms of horrible stresses and workloads.

For those unfamiliar with the story or wanting to know more, here's a good Guardian article about it... theguardian.com/business/2018/…