Data Science Applications @Shopify
In this thread I'll highlight some important pieces from a variety @ShopifyData & @ShopifyEng blogs where they discuss applications they've built that I think would benefit Data Scientists.
#DataScience
In this thread I'll highlight some important pieces from a variety @ShopifyData & @ShopifyEng blogs where they discuss applications they've built that I think would benefit Data Scientists.
#DataScience
Contents:
1. How Shopify Capital Uses Quantile Regression To Help Merchants Succeed
2. How to Build an Experiment Pipeline from Scratch
3. How to Use Quasi-experiments and Counterfactuals to Build Great Products
4. Categorizing Products at Scale
1. How Shopify Capital Uses Quantile Regression To Help Merchants Succeed
2. How to Build an Experiment Pipeline from Scratch
3. How to Use Quasi-experiments and Counterfactuals to Build Great Products
4. Categorizing Products at Scale
Other Threads On Shopify DS Applications:
- How Shopify Uses Recommender Systems to Empower Entrepreneurs:
- Measuring Churn and CLV:
- The Evolution of Kit: Automating Marketing Using Machine Learning:
- How Shopify Uses Recommender Systems to Empower Entrepreneurs:
https://twitter.com/parker_brydon/status/1359961158208401419
- Measuring Churn and CLV:
https://twitter.com/parker_brydon/status/1360020820664410113
- The Evolution of Kit: Automating Marketing Using Machine Learning:
https://twitter.com/parker_brydon/status/1362892866696134659
1 Quantile | How Shopify Capital Uses Quantile Regression To Help Merchants Succeed
by Kyle Tate
"Shopify Capital provides funding to help merchants on Shopify grow their businesses. But how does Shopify Capital award these merchant cash advances?"
shopify.engineering/how-shopify-us…
by Kyle Tate
"Shopify Capital provides funding to help merchants on Shopify grow their businesses. But how does Shopify Capital award these merchant cash advances?"
shopify.engineering/how-shopify-us…
1.1 Quantile | Motivation
When giving out a loan the most important thing to consider is the probability that it will be paid back.
To determine that, you want to know the merchant's future sales.
The problem with regression for this problem is it won't account for uncertainty
When giving out a loan the most important thing to consider is the probability that it will be paid back.
To determine that, you want to know the merchant's future sales.
The problem with regression for this problem is it won't account for uncertainty
1.2 Quantile | Motivation
Expected sales at $10k +- $1k is very different than
Expected sales at $10k +- $10k
This is where Quantile regression can help, by predicting the exact quantile of interest.
For ex: predict such that merchant would have a 90% chance of being above.
Expected sales at $10k +- $1k is very different than
Expected sales at $10k +- $10k
This is where Quantile regression can help, by predicting the exact quantile of interest.
For ex: predict such that merchant would have a 90% chance of being above.
1.3 Quantile | Implementation
A lot of boosting models have this option built in, for ex sklearn: scikit-learn.org/stable/auto_ex…
In a neural net it's just a simple change to the loss function.
More details here:
A lot of boosting models have this option built in, for ex sklearn: scikit-learn.org/stable/auto_ex…
In a neural net it's just a simple change to the loss function.
More details here:
https://twitter.com/parker_brydon/status/1325114785785065473
1.4 Quantile | Understanding
Notice that unlike a standard prediction interval which would be derived as a static above and below your predictions.
The band size can change appropriately based on the input features, for ex if seasonality causes changes to the variance of errors
Notice that unlike a standard prediction interval which would be derived as a static above and below your predictions.
The band size can change appropriately based on the input features, for ex if seasonality causes changes to the variance of errors
1.5 Quantile | Extension
In addition to using this when you want to represent a specific quantile of interest (like above).
This can also be useful to predict the prediction bands around your forecasts for a variety of applications.
In addition to using this when you want to represent a specific quantile of interest (like above).
This can also be useful to predict the prediction bands around your forecasts for a variety of applications.
2 XP | How to Build an Experiment Pipeline from Scratch
"One of the most compelling ways to prove the value of any decision or intervention ... is to run an A/B test. But what if that wasn’t an option on your current stack?"
shopify.engineering/build-experime…
by Mojan Hamed
"One of the most compelling ways to prove the value of any decision or intervention ... is to run an A/B test. But what if that wasn’t an option on your current stack?"
shopify.engineering/build-experime…
by Mojan Hamed
2 XP | How to Build an Experiment Pipeline from Scratch
This post is great walkthrough on steps to build a robust piece of software from scratch that is going to satisfy all the business requirements.
This post is great walkthrough on steps to build a robust piece of software from scratch that is going to satisfy all the business requirements.
2.1 XP | Intro
There was no experimentation platform for email experiments
So the DS manually managed them, which had problems:
1. Local storage
2. Didn't account for: user unsubscription or many-many relations between emails and shops
3. Experiments can leak into each other
There was no experimentation platform for email experiments
So the DS manually managed them, which had problems:
1. Local storage
2. Didn't account for: user unsubscription or many-many relations between emails and shops
3. Experiments can leak into each other
2.2 XP | Problem
"We define the problem as: given a list of visitors, we want to randomize so that each person is limited to one experiment at a time, and the experiment subjects can be fairly split among data scientists who want to test on a portion of the visitor pool."
"We define the problem as: given a list of visitors, we want to randomize so that each person is limited to one experiment at a time, and the experiment subjects can be fairly split among data scientists who want to test on a portion of the visitor pool."
2.3 XP | Diagram
2.4 XP | Plan Ideal Output
She built a table w 1 row per: email/shop/experiment + additional info around timing and theme.
Once v0 ideal output is made she considered example use cases and queried the fake table, to identify gaps.
She built a table w 1 row per: email/shop/experiment + additional info around timing and theme.
Once v0 ideal output is made she considered example use cases and queried the fake table, to identify gaps.
2.5 XP | Build Requirements
She sat down w stakeholders in a guided exercise where she got them to query her fake table to ensure the structure can support their needs.
She sat down w stakeholders in a guided exercise where she got them to query her fake table to ensure the structure can support their needs.
2.5 XP | Build Requirements
Requirements that resulted:
1. Exclude subjects from other experiments
2. Include exp tags
3. Exclude linked shops: shops linked to a given email get excluded
4. On-going randomization: assign new users as they qualify over time
5. Backfill past exp
Requirements that resulted:
1. Exclude subjects from other experiments
2. Include exp tags
3. Exclude linked shops: shops linked to a given email get excluded
4. On-going randomization: assign new users as they qualify over time
5. Backfill past exp
2.6 XP | Tech Planning
phase1: "create an experiment definition file that defines the criteria for candidates in the form of a SQL query"
ph2: "many-to-one transform stage that consolidates all incoming experiments into a single output"
ph3: filter down to satisfy requirements
phase1: "create an experiment definition file that defines the criteria for candidates in the form of a SQL query"
ph2: "many-to-one transform stage that consolidates all incoming experiments into a single output"
ph3: filter down to satisfy requirements
2.7 XP | Key Takeaways
3 Cause | How to Use Quasi-experiments and Counterfactuals to Build Great Products
"At Shopify, we believe that understanding causality is the key to unlocking maximum business value."
shopify.engineering/using-quasi-ex…
"At Shopify, we believe that understanding causality is the key to unlocking maximum business value."
shopify.engineering/using-quasi-ex…
3.1 Cause | Intro
"We aim to identify insights that actually indicate why we see things in the data, since causal insights can validate (or invalidate) entire business strategies. Below I’ll discuss different causal inference methods and how to use them to build great products."
"We aim to identify insights that actually indicate why we see things in the data, since causal insights can validate (or invalidate) entire business strategies. Below I’ll discuss different causal inference methods and how to use them to build great products."
3.2 Cause | Levels of Evidence
3.3 Cause | A/B Tests
The gold standard for causal inference.
The environment each group is placed in needs to be identical besides one parameter: the treatment.
Gotchas: self-selection to participate in test
The gold standard for causal inference.
The environment each group is placed in needs to be identical besides one parameter: the treatment.
Gotchas: self-selection to participate in test
3.4 Cause | A/B Tests From Scratch
You'll need:
1. way to randomly assign units to the right group
2. tracking mechanism to collect data for all relevant metrics
3. to analyze those metrics and associated stats to compute effect sizes and validate causal effects
You'll need:
1. way to randomly assign units to the right group
2. tracking mechanism to collect data for all relevant metrics
3. to analyze those metrics and associated stats to compute effect sizes and validate causal effects
I've run out of room :)
I will be sending out the second half of this thread in the coming days.
I will be sending out the second half of this thread in the coming days.
3.4 Cause | Can't Use A/B
Sometimes it's not possible to do an A/B test.
For ex:
1. Lack of tooling
2. Lack of time
3. Ethical concerns (not fair to leave some merchants out)
4. Not possible (ex: want to compare to a historical launch)
Sometimes it's not possible to do an A/B test.
For ex:
1. Lack of tooling
2. Lack of time
3. Ethical concerns (not fair to leave some merchants out)
4. Not possible (ex: want to compare to a historical launch)
3.5 Cause | Quasi-Experiments
(2nd best)
"treatment and control group are divided by a natural process that isn’t truly random, but are considered close enough to compute estimates"
Very common in product companies.
(2nd best)
"treatment and control group are divided by a natural process that isn’t truly random, but are considered close enough to compute estimates"
Very common in product companies.
3.5 Cause | Quasi-Experiment Ex 1
"feature rollout happens at different dates in different countries"
You can use a quasi-experiment tool to evaluate the expected lift from the feature, by comparing to the sales in the country w out the feature
"feature rollout happens at different dates in different countries"
You can use a quasi-experiment tool to evaluate the expected lift from the feature, by comparing to the sales in the country w out the feature
3.5 Cause | Quasi-Experiment Ex 2
"new feature is dependent on the behaviour of other features (like in the case of a deprecation)"
Where your treatment is going to get this feature (bc of their behaviour) and the control group is not
But you need to account for their behaviour
"new feature is dependent on the behaviour of other features (like in the case of a deprecation)"
Where your treatment is going to get this feature (bc of their behaviour) and the control group is not
But you need to account for their behaviour
3.6 Cause | 2 Quasi-XP Methods
(Both used at Shopify)
1. linear regression w fixed effects
- Assumes: we have the data on all factors that divide the individuals between treatment and control
- Then: lin reg on metric of interest controlling for these factors
(Both used at Shopify)
1. linear regression w fixed effects
- Assumes: we have the data on all factors that divide the individuals between treatment and control
- Then: lin reg on metric of interest controlling for these factors
3.6 Cause | 2 Quasi-XP Methods
2. diff in diffs (very popular)
- Assumes: have a control group that shows a trend parallel to your treatment on the metric of interest, prior to treatment
- Then: after treatment assume the diff in these diffs is from treatment
2. diff in diffs (very popular)
- Assumes: have a control group that shows a trend parallel to your treatment on the metric of interest, prior to treatment
- Then: after treatment assume the diff in these diffs is from treatment
3.6 Cause | Counterfactuals
(3rd best)
"want to try to detect causal factors from data that only consists of observations of the treatment.
A classic example in tech is estimating the effect of a new feature that was released to all the user base at once"
(3rd best)
"want to try to detect causal factors from data that only consists of observations of the treatment.
A classic example in tech is estimating the effect of a new feature that was released to all the user base at once"
3.6 Cause | Counterfactuals Approach
"create a model that allows you to compute a counterfactual control group.
In other words, you estimate what would happen had this feature not existed."
Difficult part is developing a robust model about your users.
"create a model that allows you to compute a counterfactual control group.
In other words, you estimate what would happen had this feature not existed."
Difficult part is developing a robust model about your users.
3.6 Cause | Counterfactuals Ex
Security updated introduced friction for users.
Wanted to see if this decreased usage.
Built time-series model to estimate usage of the updated feature using info around features not affected by update and global trends around activity.
Security updated introduced friction for users.
Wanted to see if this decreased usage.
Built time-series model to estimate usage of the updated feature using info around features not affected by update and global trends around activity.
3.7 Cause | Robustness
Quasi-xp and counterfactuals make it much harder to compute sensible confidence intervals, they increase uncertainty and risks of FPs.
Robustness checks can help falling into FP traps.
Quasi-xp and counterfactuals make it much harder to compute sensible confidence intervals, they increase uncertainty and risks of FPs.
Robustness checks can help falling into FP traps.
3.8 Cause | Robustness Checks
Gradually relaxing each assumption your model/data relies on and seeing if your results still hold.
If your finding drastically change due to a single variable you should be skeptical, especially if that variable is subject to noise.
Gradually relaxing each assumption your model/data relies on and seeing if your results still hold.
If your finding drastically change due to a single variable you should be skeptical, especially if that variable is subject to noise.
3.9 Cause | Robustness DAGs
"Direct Acyclic Graphs (DAGs) are a great tool for checking robustness. They help you clearly spell out assumptions and hypotheses in the context of causal inference."
Great post on causal DAGs by @Cmrn_DP: dataorigami.net/blogs/napkin-f…
"Direct Acyclic Graphs (DAGs) are a great tool for checking robustness. They help you clearly spell out assumptions and hypotheses in the context of causal inference."
Great post on causal DAGs by @Cmrn_DP: dataorigami.net/blogs/napkin-f…
3.10 Cause | Robustness Dagitty
"In a nutshell, when you draw an assumed chain of causal events in Dagitty, it provides you with robustness checks on your data, like certain conditional correlations that should vanish."
Dagitty: dagitty.net
"In a nutshell, when you draw an assumed chain of causal events in Dagitty, it provides you with robustness checks on your data, like certain conditional correlations that should vanish."
Dagitty: dagitty.net
3.11 Cause | 3 Points About Causal Inference
1. A/B tests should go in every DS toolbox
2. When A/B not possible look for natural experiments to replace them
3. If no natural experiments can be found counterfactuals can be useful, but you shouldn't expect to detect weak signals
1. A/B tests should go in every DS toolbox
2. When A/B not possible look for natural experiments to replace them
3. If no natural experiments can be found counterfactuals can be useful, but you shouldn't expect to detect weak signals
3 Cause | Attribution
And forgot to mention above that Antoine Rebecq was the one to author this excellent post around causality.
And forgot to mention above that Antoine Rebecq was the one to author this excellent post around causality.
2.8 XP | References
Also want to say thanks to @MojanBenham for the great post:
How to Build an Experiment Pipeline from Scratch
Great example on how to build a robust pipeline from scratch, and gives great insights into what should be considered when building 1 4 experiments
Also want to say thanks to @MojanBenham for the great post:
How to Build an Experiment Pipeline from Scratch
Great example on how to build a robust pipeline from scratch, and gives great insights into what should be considered when building 1 4 experiments
• • •
Missing some Tweet in this thread? You can try to
force a refresh
