Regression Metrics: MSE, R2 and Adjusted R2 Score

Table of Contents

1. Introduction
2. MSE & RMSE
   • Mean Squared Error
   • Root Mean Squared Error
   • Importance
3. R-squared (R²) and Adjusted R²
   • R²
   • Adjusted R-Square
   • Importance
4. Comparison of MSE, RMSE, R², and Adjusted R²
5. Where to use which?
6. Conclusion

Introduction

Evaluating the performance of linear regression models is essential to understand how well the model predicts the target variable. Here are some common evaluation metrics used for linear regression:

1. Root Mean Squared Error (RMSE) / Mean Squared Error (MSE)

Root Mean Squared Error (RMSE) and Mean Squared Error (MSE) are two closely related metrics used to evaluate the accuracy of regression models. Both metrics focus on the residuals (the difference between the actual and predicted values), but they handle them in slightly different ways.

Mean Squared Error (MSE):

MSE calculates the average of the squared differences between the actual values and the predicted values. The squaring of differences ensures that large errors are penalized more heavily than small ones, making MSE sensitive to outliers.

$$MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i – \hat{y_i})^2$$

Where:

• yi is the actual value,
• yi^ is the predicted value,
• n is the number of data points.

Root Mean Squared Error (RMSE):

RMSE is simply the square root of the MSE. It gives a more interpretable metric by returning the error in the same units as the target variable. Since the error is expressed in the original units of the data, RMSE is often easier to understand and communicate.

$$RMSE = \sqrt{MSE}$$

Importance:

• MSE (Mean Squared Error) measures model performance by calculating the average of the squared differences between predicted and actual values. Squaring the errors ensures larger mistakes have a bigger impact, making it effective for highlighting significant deviations.
• RMSE is often preferred when we want to understand the magnitude of error in terms of the original scale of the data, making it more interpretable.

2. R-squared (R²) and Adjusted R-squared

R² and Adjusted R² are metrics used to assess model performance by comparing it to a baseline model that predicts the mean value for all data points. Both metrics measure how well the model explains the variance in the target variable, with values closer to 1 indicating better performance. R² can sometimes increase with the addition of irrelevant predictors, while Adjusted R² addresses this issue by penalizing features that contribute little to the model, making it a more reliable metric in such cases. While MSE is used to assess the accuracy, R² and Adjusted R² are used to evaluate the performance of the model.

R-squared (R²):

R² is a statistical measure that explains how well the regression model fits the data. It represents the proportion of the variance in the dependent variable that is explained by the independent variables in the model. R² ranges between 0 and 1:

• R² = 1: Perfect model fit (the model explains all the variance in the target variable).
• R² = 0: The model does not explain any of the variance, i.e., it performs no better than just predicting the mean value of the target variable.

The formula for R² is:

$$R2 = 1 – \frac{\sum_{i=1}^{n} (y_i – \hat{y_i})^2}{\sum_{i=1}^{n} (y_i – \bar{y})^2}$$

Where:

• yi is the actual value,
• yi^ is the predicted value,
• yˉ is the mean of the actual values.

Adjusted R-squared:

The Adjusted R-squared modifies the R² value to account for the number of predictors in the model. R² tends to increase when more predictors (independent variables) are added to the model, even if these predictors do not add any significant value to the model. Adjusted R² corrects this issue by penalizing the inclusion of unnecessary predictors.

The formula for Adjusted R² is:

$$Adjusted R2 = 1 – \left(\frac{(1 – R^2) \cdot (n – 1)}{n – p – 1}\right)$$

Where:

• n is the number of observations (data points),
• p is the number of predictors (independent variables),
• R^2 is the R-squared value.

Importance of R² vs. Adjusted R²:

• R² is a good measure of how well the model fits the data, but it can be misleading when more predictors are added to the model, even if they do not contribute significantly to the prediction.
• Adjusted R² is preferred when you have multiple predictors, as it penalizes the inclusion of irrelevant features and gives a more reliable measure of the model’s true explanatory power.

3. Comparing MSE, R², and Adjusted R²:

Metric	Purpose	Value Range	Best Use
MSE (Mean Squared Error)	Measures the average squared error between predicted and actual values	Use to evaluate the model’s overall prediction accuracy. Lower values indicate better performance.	Use to evaluate model’s overall prediction accuracy. Lower values indicate better performance.
RMSE (Root Mean Squared Error)	Provides an interpretable version of MSE (same units as the target)	0 to ∞	Use when you want to understand errors in the same units as the target variable.
R² (R-squared)	Measures the proportion of variance explained by the model	Use it to understand how well the model fits the data. Higher values indicate better performance.	Use it to understand how well the model fits the data i.e. evaluating model performance not accuracy. Higher values indicate better performance.
Adjusted R²	Adjusts R² for the number of predictors in the model	Negative values to 1	Use when comparing models with different numbers of predictors (independent features) to avoid misleading interpretations of R².

4. Practical Use of These Metrics:

• RMSE/MSE is often the preferred metric for evaluating linear regression models when the goal is to measure the accuracy of predictions. It tells us how far the model’s predictions are, on average, from the true values.
• R² is useful for understanding the proportion of variance/details explained by the model, but it can be misleading when comparing models with different numbers of predictors.
• Adjusted R² is more reliable when comparing models with different numbers of features because it penalizes overfitting.

5. Conclusion:

• MSE and RMSE are commonly used to evaluate the accuracy of the regression model.
• To evaluate the fit of the model (i.e., how well the model explains the variance in the target), R² and Adjusted R² are used.
• Adjusted R² is particularly useful for model comparison, especially when the number of features varies.

Together, these metrics help us assess both the accuracy and the fit of the regression model, assisting in the selection of the best model.