Thread by @ammaryh92

The ABSOLUTE ESSENTIALS of Linear Regression

🧵This thread will cover:
- Intuition behind linear regression.
- Different implementations.
- Comparisons in terms of computational complexity.

Introduction

A linear regression model aims to use input features to predict a target numeric value.

For example, training a linear model to predict the price of a set of features such as area, location, number of rooms, etc.

The intuition behind "Linear"

A linear model assumes that the relationship between the input features and the target value is essentially linear.

So the goal of training a linear model is to find the line that best describes this relationship (The line of best fit).

Types of Linear Regression

Linear regression can be divided into different types depending on:

A. The number of input features
1. Simple Regression
Uses a single input feature to make a prediction.

2. Multiple Regression
Uses multiple input features to make a prediction.

B. The number of target values
1. Univariate Regression
Predicts a single target value.

2. Multivariate Regression
Predicts multiple target values.

Note that these categories can be combined in different ways such as a "multiple univariate regression model" which uses multiple input features (explanatory variables) to predict a single output value (response variable).

Training a Linear Model

The goal of training any machine learning model is to find the optimal parameters that best describe the relationship between the input features and the target values.

Read my article on this topic for a more detailed explanation.
ammarblogs.hashnode.dev/model-training…

Training Methods

1. Scikit-learn model (SVD method)

This method does not use gradient descent but rather a linear algebra technique called Singular Value Decomposition (SVD) to calculate the model parameters.

The above SVD method has a computational complexity of O(n**2), where "n" is the number of features.

This means if you double the number of input features, the computation time will be multiplied by 4.

2. Gradient Descent
In gradient descent, you start by defining a cost function that represents how bad your model is doing. Next, you search for the parameters that minimize that cost function.

Types of Gradient Descent Algorithm

a. Batch Gradient Descent
Uses the whole training set to make a single gradient descent step.

The cost function decreases smoothly with each iteration.

The training becomes slower when you have a large number of training instances.

b. Stochastic Gradient Descent
Uses a single randomly chosen example to make a single gradient descent step.

Because a random example is chosen in each iteration, the cost function bounces up and down, decreasing only on average.

Because of its oscillating behavior, stochastic gradient descent is a better approach if your algorithm got stuck in a bad local minimum.

c. Mini-Batch Gradient Descent
Uses a small random set of instances called 'mini-batches' in each iteration.

This approach takes advance of the performance boost from hardware optimization for vectorized operations.

Now, which approach should you use?
1. If you have a relatively low number of features per example, then the SVD method is a good choice.

The SVD method can handle huge training sets really well as long as each instance contains a small number of input features.

2. Avoid using the SVD method if each instance is made up of a large number of features (10,000 for example).

3. If you have a huge number of features per example, then gradient descent is the way to go.

4. Batch gradient descent is very slow when you have a huge number of training examples.

In that case, you can choose stochastic or mini-batch gradient descent as they process only one instance (stochastic) or a small number of instances (mini-batch) each iteration.

5. If your cost function has local minima, stochastic gradient descent is your best bet because of its oscillating behavior.

6. When using Gradient descent, input features must be scaled using sklearn's StandardScaler.

That's it for this thread.

If you enjoyed this thread, make sure to follow me @ammaryh92 for more contact about data science and Python programming.

Thanks a lot for your time and support.