I'm not a PhD statistician but I do have a stat degree.
My experience over the past 30 years has been analysts violating assumptions all over the place (I'm sure I've done it too) yielding poor results posing as "data-driven outcomes" ... or nobody testing anything.
My experience over the past 30 years has been analysts violating assumptions all over the place (I'm sure I've done it too) yielding poor results posing as "data-driven outcomes" ... or nobody testing anything.
2 - I once supervised a PhD statistician. This individual was responsible for measuring all print campaigns.
He used A/B tests within customer segments to measure print campaigns. So far, so good.
One problem.
He used A/B tests within customer segments to measure print campaigns. So far, so good.
One problem.
3 - For whatever the reason, he had a blind spot ... he didn't know how many customers had to be in his test/control groups to obtain meaningful results.
For whatever the reason, he just picked small quantities ... 1,500 in a cell instead of the 200,000 required.
For whatever the reason, he just picked small quantities ... 1,500 in a cell instead of the 200,000 required.
4 - Then he'd look at the mailed group ($10.00 average on 1,500 cases) against the no-mail group ($8.00 average on 1,500 cases), and say that the result wasn't statistically significant.
Then he made decisions that were very bad for business.
Then he made decisions that were very bad for business.
5 - He'd say that because there wasn't a statistically significant difference, he would say that we shouldn't mail the segment next year ... because you had no incremental sales but you had the incremental $0.50 mailing cost, yielding a loss.
6 - On his watch, circulation kept getting smaller and smaller. And nobody questioned him ... he was a PhD statistician.
About 2 weeks into my new role as a VP in his area, I called out his error.
He fought the error for about 90 seconds ... and then he just got quiet.
About 2 weeks into my new role as a VP in his area, I called out his error.
He fought the error for about 90 seconds ... and then he just got quiet.
7 - His methodology cost my company millions in annual profit.
He was applying a data-driven approach to an extreme, and was using horrific test/control sample size design, and he cost my company a fortune.
He was applying a data-driven approach to an extreme, and was using horrific test/control sample size design, and he cost my company a fortune.
8 - And he was a PhD Statistician.
Now imagine all of the foibles created by analysts who have zero statistical training but have software that allows 'em to do just about anything.
Now imagine all of the foibles created by analysts who have zero statistical training but have software that allows 'em to do just about anything.
9 - You have countless arguments out here from analysts lambasting co-workers for using "gut feel" and not adhering to a data-driven thesis.
And yet, the analysts are making soooo many mistakes applying statistical methodologies, costing their companies a fortune.
And yet, the analysts are making soooo many mistakes applying statistical methodologies, costing their companies a fortune.
10 - Business is hard, and we all make mistakes. Every single one of us.
11 - Trusting statistical methods just because they have some element of rigor isn't necessarily right, just like making decisions via "gut feel" isn't necessarily right either.
Try hard to understand what your analytical blind spots are. We all have 'em.
Questions?
KH
Try hard to understand what your analytical blind spots are. We all have 'em.
Questions?
KH
