Thread by @dritsyk

1/ 🎯Bland-Altman analysis is the common way to evaluate AI in #EchoFirst. However, BA analyses can be tricky to interpret. 🧵On the paradox of BA analysis for AI in #EchoFirst with examples from our studies on #AutoMAPSE 📜
#Foamed #Cardiotwitter rdcu.be/dH8aR

2/ BA analysis is an important approach used for comparing 2⃣measuement methods for measuring the same unit, like #MAPSE (mm) and #AutoMAPSE (mm). ‼️Correlation (r) is inappropriate yet commonly used because it is seductive. Read the reasoning in their📜: doi.org/10.1016/S0140-…

3/ The output in BA analysis is 1⃣bias &2⃣limits of agreement (LOA). Bias is the avg difference across all paired measurements of manual MAPSE and autoMAPSE. LOA=+/- 2SD of the bias in the study. The bias and LOA is presented in a BA plot from 📜: rdcu.be/dH0mY

4/ Ideally, the bias is zero. Two scenarios can give zero bias. 1⃣: The 2 methods measure EXACTLY the same in all cases - this is never the case. 2⃣: The 2 methods measure differently, but the differences are random and therefore cancel out in the study - this is common and OK.

5/ If the bias is very different from zero (defined clinically), inspect the BA plot. If the data is evenly spread around the bias, then the bias is systematic and can be calibrated by subtraction the bias. If not, then the two methods may be different: doi.org/10.1034/j.1399…

6/ The LOA is "the 95% CI of the bias". Because two methods NEVER measure EXACTLY the same, LOA is never +\- zero. How wide LOA is acceptable requires judgement, and there are two common approaches for this: the usual way and the AI-in-#EchoFirst way.

7/ The usual way is by asking: "Does differences in measurement by one-sided LOA cause clinical issues?" If no, then the two methods are interchangeable. This approach works when the one of the methods is the gold standard as the differences are from errors in the new method.

8/ Unfortunately, there is no gold standard in #EchoFirst studies. In our studies on #AutoMAPSE, humans are both the reference method AND the method to beimproved. This is common but makes interpretation of LOA difficult; the LOA can be due to human error or errors in #AutoMAPSE.

9/ Error in either method of BA analysis will widen the LOA. If errors in the reference is unrecognised, the AI-method will be discarded despite being a promising method. This Editorial in @BJAJournals show how error in the refrence method affects LOA: doi.org/10.1093/bja/ae…

10/ How to overcome this issue of human error? One popular solution is to assess how much human error to expect by assessing interobs variability. If LOA between #AutoMAPSE and manual #MAPSE is similar for two humans, then #AutoMAPSE is probably OK.

11/ Another solution is a test-retest study design to assess precision. If precision of #AutoMAPSE is good and bias is zero, then a wide LOA is likely human error. We tried test-retest for our study, but the study design was suboptimal. Better results are coming soon.

12/ A third solution is to assess the two methods for predicting an outcome. If they are equal, but the AI is faster n' easier, then use the AI irrespective of LOA. This is also how a new method can eventually replace the current standard: doi.org/10.7326/0003-4…

13/ So, back to #AutoMAPSE vs. manual #MAPSE. The bias was near-zero and LOA from this study was -3.7 to 4.5 mm; too wide if the reference was a gold standard. However, it was human measurements. We must determine how much human error to expect.
rdcu.be/dH8aR

14/ Interobs variability was assessed a priori and the LOA was similar to our results (-4.7 to 3.0 mm): 📜. Thus, the LOA between #AutoMAPSE and manual #MAPSE seems OK, and we concluded that #AutoMAPSE provides valid measurements overall. doi.org/10.1093/ehjimp…

15/ However, we reassessed the interobs between another pair of observers, and the LOA was narrower...this emphasises the variability of human measurements and the challenges of using BA analysis alone. More studies on #AutoMAPSE with data other than BA analysis is coming soon!

unroll @threadreaderapp