Skip to content

Lesson 5 of 6

Evaluate your model on unseen data

You compute the MSE on the test set using the line fitted on training data, and interpret what the error means for future predictions.

4 minMachine Learning

After this lesson, you will be able to evaluate a fitted line on unseen data, interpret the mean squared error on a test set, and explain why this error is a better guide to future performance than the training error.

The test set is unseen data

In the previous lesson, you split the dataset into a training set and a test set. The training set is used to find the best slope and intercept. The test set is kept aside and not used for fitting. This makes the test set unseen data for the model. Evaluating on unseen data gives a fair idea of how well the model will work on new examples.

You already have the code to split the data and fit a line on the training set. Here is a script that does the full evaluation:

# evaluate_model.py
from data import hours, scores
from split_data import split_data
from least_squares import least_squares
from mse import mean_squared_error

train_hours, train_scores, test_hours, test_scores = split_data(hours, scores, 0.4)
slope, intercept = least_squares(train_hours, train_scores)

mse_train = mean_squared_error(train_hours, train_scores, slope, intercept)
mse_test = mean_squared_error(test_hours, test_scores, slope, intercept)

print('Slope:', slope)
print('Intercept:', intercept)
print('MSE on training set:', mse_train)
print('MSE on test set:', mse_test)

Run this script. The numbers will change each time because the split is random. Usually, the test MSE is larger than the training MSE. That is expected: the model was chosen to fit the training data well, so it often does a little worse on new data.

What the test MSE means

The mean squared error is the average of the squared differences between the actual scores and the predicted scores. For the test set, it tells you how far off the predictions are, on average, for data the model has not seen. A smaller test MSE means the model generalises better to new examples.

The test MSE is in squared score units. To get an idea of the typical error in score points, you can take the square root. This is called the root mean squared error, or RMSE. Add this to the script:

import math

rmse_test = math.sqrt(mse_test)
print('RMSE on test set:', rmse_test)

The RMSE is in the same units as the scores. For example, if the RMSE is 5, the predictions are typically off by about 5 score points. That is a useful way to judge whether the model is good enough for your purpose.

Why not just use the training error?

The training MSE measures how well the line fits the data it was fitted on. It can be too optimistic. A model that is too complex could fit the training data almost perfectly but fail on new data. With a simple line, the difference is usually small, but the principle matters: always report the test error when you want to know how the model will perform in practice.

In this course, the model is a straight line, so it cannot overfit much. But in real machine learning, models can be very flexible. The test set is your protection against being fooled by a model that only memorises the training data.

A stable evaluation

Because the random split changes the test set, the test MSE changes from run to run. To get a more stable number, you can repeat the split and fitting several times and average the test MSE. Here is a script that does that:

# average_test_mse.py
from data import hours, scores
from split_data import split_data
from least_squares import least_squares
from mse import mean_squared_error

test_mses = []
for _ in range(100):
    train_hours, train_scores, test_hours, test_scores = split_data(hours, scores, 0.4)
    slope, intercept = least_squares(train_hours, train_scores)
    mse_test = mean_squared_error(test_hours, test_scores, slope, intercept)
    test_mses.append(mse_test)

average_test_mse = sum(test_mses) / len(test_mses)
print('Average test MSE over 100 splits:', average_test_mse)

Run this script. It takes a moment because it fits the line 100 times. The average test MSE is more stable than a single value. You can use this number to compare different models or different ways of preparing the data.

Summary

The test set is data the model has not seen during fitting. The test MSE measures how well the model generalises. The RMSE gives the typical error in the original units. Always evaluate on unseen data to get a realistic picture of future performance.

Key takeaways

  • The test set is unseen data used only for evaluation, never for fitting.
  • The test MSE estimates how well the model will perform on new examples.
  • The RMSE is the square root of the MSE and is in the same units as the target.
  • The training MSE is usually smaller than the test MSE because the model was fitted to the training data.
  • Repeating the random split and averaging the test MSE gives a more stable estimate.

Try it

Write a new Python script called my_evaluation.py. In it, import the necessary functions from data.py, split_data.py, least_squares.py, and mse.py. Split the data with a test size of 0.3. Fit the line on the training set, then compute and print the training MSE, test MSE, and test RMSE. Run the script five times. Check that the test RMSE is a positive number and that the test MSE is usually larger than the training MSE. A good result is a script that runs without errors and clearly shows the three evaluation numbers for each run.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.