Regression

Introduction

Regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables (or 'predictors'). More specifically, regression analysis helps one understand how the typical value of the dependent variable (or 'criterion variable') changes when any one of the independent variables is varied, while the other independent variables are held fixed.

A photograph of a statistical data analysis on a computer screen.

Types of Regression

There are several types of regression analysis. Some of the most commonly used include:

Linear Regression

Linear regression is a basic and commonly used type of predictive analysis. The overall idea of regression is to examine two things: (1) does a set of predictor variables do a good job in predicting an outcome (dependent) variable? (2) Which variables in particular are significant predictors of the outcome variable, and in what way do they–indicated by the magnitude and sign of the beta estimates–impact the outcome variable?

Logistic Regression

Logistic regression is another type of regression analysis used for prediction of outcome of a categorical dependent variable. Unlike linear regression, logistic regression can model the probability of a certain class or event existing such as pass/fail, win/lose, alive/dead or healthy/sick.

Polynomial Regression

Polynomial regression is a form of regression analysis in which the relationship between the independent variable x and the dependent variable y is modeled as an nth degree polynomial in x. Polynomial regression fits a nonlinear relationship between the value of x and the corresponding conditional mean of y.

Ridge Regression

Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where independent variables are highly correlated. Unlike least squares regression, ridge regression shrinks the coefficient estimates towards zero, which helps to reduce model complexity and multicollinearity.

Lasso Regression

Lasso regression is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the statistical model it produces.

Assumptions of Regression Analysis

There are several assumptions that are made during the process of regression analysis. These include:

1. There is a linear relationship between the dependent variables and the regressors, meaning the model you are creating actually fits the data, 2. The errors or residuals of the data are normally distributed and independent from each other, 3. There is minimal multicollinearity between explanatory variables, 4. Homoscedasticity. This means the variance around the regression line is the same for all values of the predictor variable.

In practice, these assumptions are not always met. Violations of these assumptions may lead to inaccurate and misleading results. Therefore, checking for these assumptions should be a key part of any regression analysis.

Applications of Regression Analysis

Regression analysis is widely used for prediction and forecasting, where its use has substantial overlap with the field of machine learning. Regression analysis is also used to understand which among the independent variables are related to the dependent variable, and to explore the forms of these relationships.

In restricted circumstances, regression analysis can be used to infer causal relationships between the independent and dependent variables. However this can lead to illusions or false relationships, so caution is advisable.

References