Correlation is the skill that has singlehandedly benefitted me the most in my career.
In 3 minutes I'll compress what I learned in 10 years of using correlation to solve business problems.
Let's go!
1. Correlation: Correlation is a statistical measure that describes the extent to which two variables change together. It can indicate whether and how strongly pairs of variables are related.
2. Types of correlation: There are several types of correlation used in statistics to measure the strength and direction of the relationship between variables. The three most common types are: Pearson, Spearman Rank, and Kendall's Tau. We'll focus on Pearson since that is 95% of the time what I use.
For years, I was hyperparameter tuning XGBoost models wrong.
In 3 minutes, I'll share one secret that took me 3 years to figure out.
When I did, it cut my training time 10X. Let's dive in:
1. XGBoost:
XGBoost (eXtreme Gradient Boosting) is a popular machine learning algorithm, especially for structured (tabular) data. It's claim to fame is winning tons of Kaggle Competitions. But more importantly, it's fast, accurate, and easy to use. But it's also easy to screw it up.
2. Hyperparameter Tuning:
To stabilize your XGBoost models, you need to perform hyperparameter tuning. Otherwise XGBoost can overfit your data causing predictions to be horribly wrong on out of sample data.