Thread by @duncan3ross

Many years ago my then employer @Teradata let me create a set of #Analytics cards. @JudyBayer and @edouardss helped hone them.

So here's a thread that some of you in #DataScience might find useful.

1/n

The idea was simple. If you are doing some cool Data Science every now and then just draw a card, and see if you can answer it for your project. If not... well.

2/n

So over the next 52 days (plus Joker days) I'm going to draw a card and explain why it's important. At #random. Because in data random is usually best.

3/n

Day 1: King of Spades - What do you do now?

This is always a good question to ask. It's rare that you're doing data science in a vacuum. So ask yourself - what are we doing now, and why?

4/n

It's pretty likely that whatever is being done now has lots of useful insight built into it - not necessarily on display, but buried in the latent rules. Perhaps that includes lessons that were learnt the hard way.

5/n

And if you don't know what your company (or organisation) is doing about this problem now, perhaps you better go and find out. Because you won't do good data science unless you can do better than that...

6/n

Day 2: 4 of Diamonds - Who will do the analysis?

Now obviously this can be read simply - who is doing the work, do they have the right skills etc...

7/n

But one thing that has become obvious when thinking about #AIEthics is that analysts also bring their biases with them.

8/n

So think about this one more deeply. What might the analyst be unaware of because of who they are? What - or who - are they not considering?

9/n

Today we have 9 of Hearts: What data haven’t you used?

This suggests two things to me - firstly is there other data that I could have used?

10/n

I know that I tend to go back to the same trusted data whenever I can, but things change. Data gets stale, or superseded - new and better datasets come along!

11/n

But the second thought is perhaps stronger: why didn’t you use the other data? This may reflect your own biases. Try swapping some data around. What effect does that have?

12/n

And, of course, document this (no, I don’t either). Your reasons will be useful another day

13/n

Tonight’s card is King of Hearts - Did anything in your data happen after the event you are predicting?

14/n

I wish I could say I’ve never fallen for this. But of course I have. Why is it a problem? Well essentially you’re building a model that can’t work when deployed. Which is a big problem.

15/n

Fortunately there is usually an easy way to notice.

Is your model 99% accurate? Then either you aren’t solving a real world problem or something has gone wrong - and it might be this.

16/n

Of course it won’t be the easiest thing to find, but find it you must...

17/n

And on to the 8 of Hearts - what should the data look like?

Perhaps that sounds a bit odd. Data is data. How can it look like anything?

18/n

And yet this is one of those cards that holds a more widely applicable truth, and which embodies an approach to data that is pretty central... too esoteric? Then let me explain...

19/n

When I hire a data scientist one of the key things I want is a feel for data. What looks right. More importantly, what looks wrong. Are these numbers plausible? What about those outliers? And those zeros? (Especially those zeros)

20/n

And this is something we can all apply when reading the latest conspiracy theory. Do those numbers look right? Really? Because then that would mean x. Plausibility doesn’t guarantee accuracy, but data literate plausibility is pretty damn important...

21/n

So: what should the data look like, and does this data look like that, and if not then why?

22/n

Nine of Spades - how sure are you about the question?

23/n

At the end of the day all analysis is about trying to answer a question. So how confident are you that it is the right question?

24/n

For example: your question might be “can I predict if someone will leave?” It sounds like a good question.

But are you sure? Some people are going to leave whatever you do. Some people might leave and take others with them. Can you ask the question in a different way?

25/n

Take the time. Think about how you might ask the question differently so that it is a better, stronger question. It might even lead you to conclude that this question is better not asked!

26/n

The two of Hearts - Is your answer what people want to hear?

Should it matter? Well, no, ideally it shouldn't. But data doesn't exist in a vacuum... all analysis sits within a social frame (even science, if in doubt on that try reading Kuhn!)

27/n

If your answer isn't what people will want to hear, then firstly I would double check your answer. Is it right? There is often a good reason why people are expecting to get a particular result from analysis. Have you misunderstood the question?

28/n

Is there something else going on that you haven't taken into account (or that isn't in your data)?

But let's try it from the other angle too. Just because you produce the answer that people wanted, is it right?

29/n

There is often enormous pressure to produce a particular result - that's why you get p-hacking.

(And here's a great example from XKCD)

30/n

So it's a good idea to be sure of your answers, but then to be willing to stand by them - even if they aren't what people wanted to hear.

Of course it's also good to remind them that data scientists are human too at this point.

31/n

Six of Spades - What effort will it take to put it into production?

Sometimes we just do analysis for insight. But a lot of the time in order to see the benefit of our models we need to actually put them into production somehow...

32/n

And if we're going to need to do that then the sooner we think about the challenges involved the better.

There are a series of interesting technical challenges, of course. But a huge part of that comes back to our favourite challenge: data.

33/n

Wait a minute! Why should data be a problem?

Anyone who works with data will tell you that at least 60% (and if you said 90% I wouldn't contradict you) of their time is spent cleaning and manipulating data.

34/n

It's a pain, but it has to be done to get onto the sexy bit of building the model itself. And it's far, far easier when you're doing this with static data from historical sources.

When you need to do it "live" - in production - you hit all kinds of interesting problems.

35/n

One of which is the difference between data that (in our nice training set) is related to a particular date and time, and when that data actually becomes available for analysis.

Yes, I have seen analysis fail when data from day X actually only became available on day X+2

36/n

So think about production early, not late. And don't forget the most important part of production: the people who will need to use your model. I suspect that's on a card out there somewhere...

37/n

Today's card is the four of Clubs - How does this relate to strategic goals?

Just as you can do analytics for insight rather than deployment, you can do analysis for tactical reasons rather than strategic ones.

But it helps.

38/n

And even tactical stuff should relate to strategy.

Now, of course, you may be working in an organisation that doesn't have a well defined strategy. That's tough. Or goals.

But if you do then your analysis and it's outputs should be related to them.

39/n

So your analysis could positively support your goals. But it should also not work against them - either intentionally or unintentionally.

That may take some working out. Prediction is the kind of thing that can change behaviours. It may well be intended to.

40/n

What will that mean in the long run?

Remember 0% interest credit cards? You moved a balance onto them and paid 0% for a year. And then clever people predicted who would close the account after a year and started offering them another credit card with a 0% interest rate.

41/n

The result was tactically great. Customers were retained.

But in terms of a bank's strategic objectives (very bluntly making money) it sucked. And worse still, it taught people to expect this kind of deal...

42/n

With the 10 of Hearts we ask: How long will the data remain relevant?

Models are built at a point in time. Usually with data that is static (unless you're doing something clever with self-learning. But if you think that get's you out of this one, think again!)

43/n

But the world changes. And in many cases you want the world to change - after all you want your model to have an impact.

When the world changes the data that you are using changes. Distributions shift. Models do not like this.

44/n

Why? Think of the data-space your model inhabits. It performs differently at different points (unless you have a very weird model). As your data shifts that performance changes unpredictably, especially at the edges.

45/n

And some data will just cease to be relevant. Or become a source of noise. And that will defeat the clever self-learning models too, eventually*.

So plan for this. Build in performance monitoring of both your output AND your data.

46/n

When your model goes outside those boundaries it's time to rebuild it.

Good news! There will still be employment for you :)

47/n

Back to the Clubs! The 10, and If you couldn't do this, what would you do instead?

48/n

How should we think about this? Well it's an unblocker card. When we get stuck in an analytical process it can be great to try to rethink the whole thing.

What else could we do?

49/n

This isn't really asking you to swap out regression for a neural net - it's asking you to think about a totally different way of approaching the problem.

How about clustering instead of predicting?

What about changing the target variable?

50/n

What about (and this may be heresy) not doing the analysis at all? Are there better analyses that you could be doing instead?

I think I will stop before I suggest a career change though!

51/n

The Eight of Diamonds: Do different errors have different costs?

There's a naive assumption in modelling that costs are equal. Which is odd, because in life we know that different decisions have very different outcomes. There is a whole subgenre of #romcoms based on that!

52/n

Formally there are different types of error Type I (false positives) and Type II (false negatives). Type III is where you get Type I and II confused.

53/n

It's always worth thinking about the different costs, because they may change what you're looking for in the model.

Take a scenario from marketing. You're using Twitter DMs to market Ferraris. I'm not suggesting it's a good idea.

54/n

A false positive means your model thinks that this DM will result in a sale. It won't. Cost to you? Very little.

55/n

A false negative means that the model tells you not to send this DM. Unfortunately the recipient has just won the lottery and is looking to buy a shiny red car. Cost to you? £263,098

carpages.co.uk/ferrari

56/n

Sometimes costs are hard to quantify. Take a more important decision. Do you release someone on bail?

The social cost of keeping someone in jail wrongly against the potential damage to society (or flight risk) of releasing someone when you shouldn't.

57/n

But in either case we might want to think carefully about how our model performed in very different circumstances. Where would we be willing to sacrifice accuracy?

Is it better that ten guilty persons escape than that one innocent suffer?

en.wikipedia.org/wiki/Blackston…

58/n

And a nice one to end the working week. Five of Spades: Are your visualisations misleading?

59/n

When I was doing Using Data For Evil talks at @strataconf with @fhr (who definitely deserves credit for the cards too), we used to have a lot of fun with bad #visualisations.

60/n

For example, the infamous Florida Gun Deaths chart.

61/n

Or there are the many, many @LibDems "XXX Can't Win Here" leaflets (this one collected by @Londonist but I guess the copyright belongs to Jill Fraser)

62/n

So yes, your visualisations can be misleading. But does it matter (beyond being a question of ethics).

The point of analysis is to change things, and much of the time that means convincing people, not just doing maths.

63/n

What happens if people take the wrong message from your visualisations? They won't understand your analysis. They may even do exactly the wrong thing.

64/n

How can you spot your own bad visualisations? Spotting other people's is pretty easy, of course.

Well, try showing them to people who aren't involved in the analysis. Better yet who aren't doing any analysis at all! If they can't understand them then maybe try again...

65/n

Of course then we get into the whole issue of how to make your visualisations more effective (stand up @EdwardTufte) - but start by at least not making them misleading!

66/n

By special request (@RuthArnold) the Three of Hearts: What surprises are there in the data?

There aren’t always surprises, but they do happen an awful lot!

67/n

Sometimes they can derail your analysis, but just as often they challenge your preconceptions!

People (and therefore data) behave oddly.

68/n

Surprises can come from the way data is collected too - and that’s always worth thinking about.

When @DataKindUK was exploring UK beneficial ownership data there was a surprise - hundreds of versions of “British” in the nationality field

68/n

The system designer had left it as free text.

In a bank there were strange values cropping up in a balance field (always subtracted out later) because it was easier to reuse a field than get a new one added.

69/n

The best ones though are when the data shows that people are doing things that are unexpected. Now you just need to find out why.

70/n