1/ I’ve believe Data Scientists fall into two groups. A) CS (and similar) backgrounds that build ML models which can be effective but less interpretable; B) Those trained in statistical methods & build simpler models that are very interpretable. Former gets most attention.
2/ Here @CircleUp we care about both. For the later we recruit backgrounds in econ, econometrics, quant marketing, cognitive psychology (and other data-heavy behavioral sciences). Former = CS, physics, and applied math. Not exhaustive examples.
3/ Both backgrounds are valuable but suited for different problems. There are some data science problems where interpretability is critical (e.g. investment decision), and some jobs to be done where precision matters much more than interpretability (e.g. movie recommendation).
5/ Think of your product and the job to be done. If you’re suggesting another type of toothpaste to consumer via an online marketplace (low AOV, low risk if wrong), the “why” doesn’t matter that much. I’m not going to dissect the decision- I just want it right (high precision).
6/ For @waymo, I want to be safe. I don’t need to dissect the infinite # of decisions being made as I am driven. High precision.
7/ However for other decisions, i.e. prediction on whether a CPG company will be successful- we find that interpretability matters a lot.
8/ In those cases, the user- perhaps a buyer at a large retailer that wants to work with the best brands, or an investor- typically thinks they know better than the models. Thus a “black box” approach gets rejected by the host.
9/ Additionally, optimization objectives such as error minimization in models fail to capture more complex real-life goals. A rich set of examples can be found in medicine, criminal justice etc. Would you really want to prescribe, implicate etc. without understanding why?
10/ Strong criminal justice example shows that when racial bias is built into the input data, it will be part of the output as well. Some things need to be interpretable as well in order to pick up the mistakes/bias.
propublica.org/article/how-we…
propublica.org/article/how-we…
11/ We see a mix of approaches to be effective. Entity resolution (ability to distinguish between brands and cluster products), has involved black box ML. In other cases simple regressions- built on top of some terrific and proprietary data, is darn effective AND interpretable.
12/ Last year at a CEO conference I gave a talk about this topic and the CEO of a known tech company disagreed with my pts here and said that the CTO should make these decisions in isolation - I disagreed.
13/ I think as you’re building out a DS team, the CTO can’t be expected to know the nuances above and how they apply for your particular product. If your CTO does congrats- that’s awesome. But if you don't understand, could lead to hiring the wrong person for the job
14/ To quote @arvind5813vg “Find people who can focus on explaining and rationalizing up front rather than focusing exclusively on predictive power.”
