In addition to being a professor, I also do freelance statistical consulting (my rent is too damn high). I thought it would be interesting to go over the way that people think about statistics in this hidden little corner of the internet.
To be specific, I consult on research and analysis design, and also do coding work. Most clients are small businesses or independent. A few bigger organizations, and the odd student rich enough to drop a few hundo on getting a prof to critique their term paper. Rarely, academia.
The first big thing you notice is that a lot of people tend to think of statistics as a tool that serves a function, rather than as a method of analysis.
What I mean by this is that it's common to think of statistical analysis almost like an enterprise product. If I buy this "statistics" doodad, it will help me make my point. If it doesn't do that, it serves no purpose.
It's extremely common that I will get requests to "fix" the work if it doesn't give them the result they want. I've had people demand refunds. Plenty of job ads that tell the worker the results they're supposed to get. I don't respond to those ones.
Tempting to think of this as fraudulent, but in the conversations I have with people I think it's more a deep misunderstanding of the purpose of statistics and empiricism.
It's extremely difficult to even get across the point of why this is a bad thing. And believe me, I've tried this conversation many times. They want good statistics and "best practices." They just literally think that this means doing whatever gives you the "right" result.
Related, but p-hacking is just an incredibly natural temptation. I'd say that a good 50% of the insignificant results I return get a response asking how they can be changed. The most common phrasing is asking how they can be "fixed." No concept of that being bad.
I say "natural" from academic experience. I've never seen a stat/metrics class that teaches p-hacking as good. Many explicitly warn against it. But when I see those students in the follow-up class, for many it's just the obvious thing to do. Of COURSE you "fix" insignificance.
(I have one client at the moment who I keep telling that his results long ago stopped meaning anything as we round the corner on maybe 2000 different specifications, but he keeps adding 'em and looking for significance! The desire to beat the stock market is a strong one.)
The second thing that I see is that statistics is seen as a real push-button operation. You pour in the numbers, the smart person pulls the right lever, and the result pops out toasty, warm, and ready to eat.
There's very little concept that context is important, or that there's some sort of underlying model. I've on occasion gotten shocked surprise when I've told clients that I need to know what the variables in the data set actually represent before I build an analysis plan.
Also common to just have a dataset dropped on my lap so I can go do the statistics to it. Very hard to coax a research question or even a preferred dependent variable out of a client who doesn't understand why you need such a thing. I have the data don't I? Just run regressions.
Similarly, the (usually) students want robustness checks, but in a very cargo cult way. They see them done elsewhere and they want them too, but they're not sure why or what purpose they serve. Just add all the robustness checks, please.
There's a real desire for statistical analysis to be a completely objective, theory-free, and context-free affair divorced from the reality of where the data came from. And I get where they get that, that's probably what I'd assume statistics was if I didn't know more about it.
I should point out that many of my clients fall into none of these holes and grok this stuff well. And the ones that do have these issues aren't bad or unintelligent. They're just not statisticians or econometricians (if they were they wouldn't need me!)
It's valuable to know what the folk understanding of statistical analysis is. Even if by "folk" I mean people who have enough stats knowledge to hire a consultant. And especially if this is a group of people who are connected, by volume, to a huge chunk of produced analysis.
Whether you're teaching, consulting, or reading the analysis of others, you want to have a good sense of how people think, and the assumptions and attitudes they're coming to the table with. /fin.
