,
17 tweets,
6 min read
Read on Twitter
8.1/n Continuing here the twitter summary of my keynote talk ("Best practices of using machine learning in businesses") at the @BudapestBI data science conference. Tip #7: Do *online* validation of ML models (on live data, using the same pipeline as you would use in production) 
8.2/n Besides the evaluation you do when you develop/train a model (evaluation on separate test/hold-out dataset / using cross validation) you must also do evaluation of the online model (deployed model/scoring on live data and using the same data pipeline as used in production)
8.3/n There are several reasons for this: debugging (especially of the online feature engineering "reimplementation" / or generated code if using a tool), detecting changes due to non-stationarity etc. While every single ML textbook talks about train-test split etc., most ...
8.4./n ... most textbooks ignore to mention the need to do online evaluation. Graphlab (renamed Dato, renamed Turi, then bought by Apple and RIP) had a couple of nice blog posts about this.
8.5/n Tip #8: Monitor your models. You do online evaluation before and right after you deploy your models. You do monitoring continuously after you deploy your models. (2 different things) Unfortunately, both are done less than they should be (that is always).
8.6/n Monitoring should consist in dashboards and text/email/message alerts for significant changes monitoring the distribution of scores, important quantiles, effect of actions taken based on scores (upstream), characteristics of input features (downstream data pipeline) etc.
9.1/n Tip #9: Focus on business value. Seek business value in the project, measure the impact, sell your work to the business.
9.2/n Again most ML textbooks focus on evaluation based on accuracy, log-loss, ROC, precision-recall, AUC etc. Your boss/business executives don't understand/don't give a shit about this. Your have to show them the money. Money is the ultimate data science metric. 💰
9.3/n There is not much literature on this, but here is a nice blogpost (assign $ value to true positive, FP, TN,... and then compute utility/$ vs thresholds base on that and optimize it):
10.1/n Tip #10: Make your work reproducible.
10.2/n There has been a reproducibility crisis in science, especially in social science or the medical field. People had hard time to reproduce earlier studies, because of poor procedures in keeping a track of records, data, procedures, software versions etc.
10.3/n Keeping a good track of your work (data, methods, software etc) ensure trust and makes debugging much more easy
10.4/n If you use tools for that, they also help with automation (a script can be run with different parameters or at different time/regularly) and it also makes you more productive
10.5/n Tools that help with reproducibility are Jupyter notebook / R markdown (e.g. within Rstudio) in the R/python world, git (e.g. via github or gitlab) with version control (keeping track of changes) etc.
10.6/n Ultimately the whole ML workflow should be kept reproducible/automatized (it should be a few clicks to rerun the whole process of feature extraction, training, tuning, evaluation, deploying the models etc.)
10.7/n That's the 10 tips I discussed in the keynote. Why 10? because it's a nice round number. I have more tips, but that's what it could be fit reasonably in a 45-minute talk. So that's all for now, hope you enjoyed and learned something new.
10.8/n In case you got to this thread somewhere from the middle, here is the beginning:
