Regression Evaluation Metrics

Introduction

Let’s start with some motivation, evaluation metrics are a crucial part of applied machine learning, without them, you wouldn’t be able to choose the best model for your specific problems or know if what you’re doing is making the results better or worse.

To refresh your memory let's briefly talk a bit about what regression problems actually are.

Regression models a target value based on a set of independent variables, some examples include predicting the price of a house after a model was formed on the independent variables such as the number of bedrooms and location.

Now let’s see what evaluation metrics we cover here.

Contents

• R squared
• Mean square error (MSE)
• Mean absolute error (MAE)

R Squared

This metric simply tells you how well your model fits the data. Where a value of 1.0 means you have a perfect model.

More formally it “is the proportion of the variance in the dependent variable that is predictable from the independent variable(s)”.

Even though this is a popular metric it has some negatives:

• R squared cannot predict weather predictions are biased.
• A high R squared score does not automatically mean you have a good model, the same goes for a low score, it doesn’t always mean that the model is bad.

This is a great explanation for why multiple evaluation metrics should be used and also that you need to understand the problem well so that you can set optimal evaluation targets.

Here is a great link to learn more about the ups and downs of this metric.

https://blog.minitab.com/en/adventures-in-statistics-2/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit

Mean Squared Error (MSE)

MSE will make outliers stand out more and it should be used when a prediction that is 5% off is more than double as bad as a prediction that is 10% off (Another reason to gain a good understanding of the problem domain).

The MSE is calculated as the mean of the squared differences between predicted and expected target values in a dataset.

The units of MSE are squared units which can be sometimes confusing when reporting results. For example, if you’re trying to predict the price of a stock in dollars the unit for MSE will be squared dollars which is why you should also consider using a variant of this called root mean squared error (RMSE).

https://en.wikipedia.org/wiki/Mean_squared_error

Mean Absolute Error (MAE)

This is another common metric (similar to MSE) where all errors are on the same scale, for example, if the actual value is 50 then a prediction of 48 is the same error as predicting 52.

In a nutshell, MAE calculates the average magnitude of errors without taking into consideration their directions. The following formula will calculate MAE however it is much easier to use the sklearn library (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html)

Here is a great article on MAE and a comparison with RMSE:

Performance measures: RMSE and MAE - The Data Scientist

Summary

This post combined with my earlier post on evaluation metrics for classification models will give you a solid foundation to help you dive deeper into this topic and also have a quick reference to what these metrics actually do.

I highly recommend experimenting with these and looking into some of the articles linked to really push you forward with your applied machine learning skills.