Imagine a model to predict customers who will unsubscribe from your service.
You want to incentivize them with $10 because they will cost you $100 if they churn.
Look at the attached confusion matrix showing that the model is only 77% accurate.
Is this model good enough?
You want to incentivize them with $10 because they will cost you $100 if they churn.
Look at the attached confusion matrix showing that the model is only 77% accurate.
Is this model good enough?
I love this question because it puts a couple of things in perspective:
1. A model that doesn't look too good by the numbers.
2. A business case that can use a less-than-ideal solution to solve the problem.
1. A model that doesn't look too good by the numbers.
2. A business case that can use a less-than-ideal solution to solve the problem.
There's only one question about this problem: how many people will not churn if they get the incentive?
We don't know, but we can play out different scenarios and see what happens.
We don't know, but we can play out different scenarios and see what happens.
Assume everyone who takes the incentive sticks around.
That means that we will be spending $190 on incentives.
Churn will cost us $1,300 (13 people will churn — the false negatives.)
Total cost: $1,490.
If we did nothing, we would have $2,200 in costs.
The model is useful!
That means that we will be spending $190 on incentives.
Churn will cost us $1,300 (13 people will churn — the false negatives.)
Total cost: $1,490.
If we did nothing, we would have $2,200 in costs.
The model is useful!
Assume that half the people who take the incentive stay.
We will still spend $190 on incentives, but churn will be ~(13 + 4) * $100 = $1,700.
The total cost is $1,890 which is still less than $2,200.
Model is still useful with the incentive at a 50% success rate.
We will still spend $190 on incentives, but churn will be ~(13 + 4) * $100 = $1,700.
The total cost is $1,890 which is still less than $2,200.
Model is still useful with the incentive at a 50% success rate.
This model wouldn't make sense if we can only get a single person (or none of them) to stay after getting the incentive.
In every other case, the model will be useful.
In every other case, the model will be useful.
Here is the summary of the story:
In a business context, performance metrics aren't the whole story. The economy of things plays a more prominent role.
(And of course, in this example we are assuming we created the model for free... but it's just an example.)
In a business context, performance metrics aren't the whole story. The economy of things plays a more prominent role.
(And of course, in this example we are assuming we created the model for free... but it's just an example.)
@AlejandroPiad I remember we had a conversation a while ago about measuring whether a model is useful or not. We discussed ROI that day.
This model is one example.
This model is one example.
Even better yet, assuming that the incentive is 100% effective, then giving $10 to everyone will save more money than using the model in the first place.
It all comes down to the assumptions we make about the effectiveness of that incentive.
It all comes down to the assumptions we make about the effectiveness of that incentive.
https://twitter.com/whereisravi/status/1362045164668080129?s=20
• • •
Missing some Tweet in this thread? You can try to
force a refresh
