I’m not saying you need to be an expert in advanced calculus to do machine learning…
BUT, there is a big difference between someone that does vs someone that does NOT have a good foundation in stats when it comes to getting & explaining business results.
My thought process back in the day was to obtain a great foundation in stats and machine learning at the same time.
So here’s what helped me. I read a ton of books.
Here are the 3 books that helped me learn data science the most...
1. R for Data Science (Wickham & Grolemund) r4ds.had.co.nz
But all it takes is mastering 1 technique: time series decomposition.
Here's why:
1. What is Time Series Decomposition?
A statistical method used to deconstruct a time series into several components, each representing underlying patterns in the data.
There are 3 key components: Trend, Seasonal, and Residual. Let's break them down.
2. Trend:
Trend is the long-term movement of the series. Typically, we use a smoother (LOESS, LOWESS) or moving average to calculate the trend. The key is that it removes the seasonal variation from the time series.
Merlion is a Python library for time series intelligence.
It provides an end-to-end machine learning framework that includes loading and transforming data, building and training models, post-processing model outputs, and evaluating model performance.
2. GUI Dashboard
Merlion makes time series analysis easier with a web-based dashboard.
This dashboard provides a great way to quickly experiment with many models on your own custom datasets.
A Python Library for Time Series using Hidden Markov Models.
Let me introduce you to hmmlearn.
1. Hidden Markov Models
A Hidden Markov Model (HMM) is a statistical model that describes a sequence of observable events where the underlying process generating those events is not directly visible, meaning there are "hidden states" that influence the observed data, but you can only see the results of those states, not the states themselves
2. HMM for Time Series with hmmlearn
hmmlearn implements the Hidden Markov Models (HMMs).
We can use HMM for time series. Example: Using HMM to understand earthquakes.
Understanding probability is essential in data science.
In 4 minutes, I'll demolish your confusion.
Let's go!
1. Statistical Distributions:
There are 100s of distributions to choose from when modeling data. Choices seem endless. Use this as a guide to simplify the choice.
2. Discrete Distributions:
Discrete distributions are used when the data can take on only specific, distinct values. These values are often integers, like the number of sales calls made or the number of customers that converted.
Forecasting time series is what made me stand out as a data scientist.
But it took me 1 year to master ARIMA.
In 1 minute, I'll evaporate your confusion. Let's go.
1. Autoregressive Forecast Models
ARIMA and SARIMA are both statistical models used for forecasting time series data, where the goal is to predict future points in the series. The implement a concept called Autoregression.
2. ARIMA Decomposed:
AR-I-MA stands for Autoregressive (AR), Integrated (I), Moving Average (MA).
🚨BREAKING: New Python library for agentic data processing and ETL with AI
Introducing DocETL.
Here's what you need to know:
1. What is DocETL?
It's a tool for creating and executing data processing pipelines, especially suited for complex document processing tasks.
It offers:
- An interactive UI playground
- A Python package for running production pipelines
2. DocWrangler
DocWrangler helps you iteratively develop your pipeline:
- Experiment with different prompts and see results in real-time
- Build your pipeline step by step
- Export your finalized pipeline configuration for production use