Skip to content

Lesson 4 of 6

Split data to test fairly

You divide the dataset randomly into a training set and a test set, fit the line on training data only, and see why evaluating on the same data is misleading.

5 minMachine Learning

After this lesson, you will be able to split a dataset into a training set and a test set, fit a line on the training set only, and explain why evaluating on the same data you trained on gives a misleadingly good result.

Why split the data?

In the earlier lessons, you fitted a line to all five data points and measured the error on those same points. The line was chosen to be as close as possible to those points, so the error you measured is the smallest possible for that data. But that does not tell you how well the line will do on new data, such as a new student's score. To get a fair idea of how the model will perform in the future, you should test it on data it has not seen during training.

The usual way is to divide the dataset into two parts: a training set, used to fit the model, and a test set, used only to evaluate the model. The model never sees the test set during training, so the test error is a more honest measure of how it will do on new data.

Split the data randomly

You will use the same data.py file from the earlier lessons. It contains two lists: hours and scores. To split the data, you need to choose which points go into the training set and which into the test set. A random split is common because it avoids any bias in how you choose the points.

Python's standard library has a module called random that can help. You will use random.shuffle to shuffle the indices of the data points, then take the first part for training and the rest for testing.

Create a new file called split_data.py with this code:

# split_data.py
import random

def split_data(hours, scores, test_size):
    """Split data into training and test sets.

    test_size is the fraction of data to use for testing, between 0 and 1.
    Returns four lists: train_hours, train_scores, test_hours, test_scores.
    """
    n = len(hours)
    indices = list(range(n))
    random.shuffle(indices)
    test_count = int(n * test_size)
    test_indices = indices[:test_count]
    train_indices = indices[test_count:]
    train_hours = [hours[i] for i in train_indices]
    train_scores = [scores[i] for i in train_indices]
    test_hours = [hours[i] for i in test_indices]
    test_scores = [scores[i] for i in test_indices]
    return train_hours, train_scores, test_hours, test_scores

The function takes the full hours and scores lists and a test_size fraction. It shuffles the indices, then uses the first test_count indices for the test set and the rest for training. The data points are returned as four separate lists.

Now try it out. Create a script called try_split.py:

# try_split.py
from data import hours, scores
from split_data import split_data

train_hours, train_scores, test_hours, test_scores = split_data(hours, scores, 0.4)
print('Training hours:', train_hours)
print('Training scores:', train_scores)
print('Test hours:', test_hours)
print('Test scores:', test_scores)

Run the script. You will see the data split into two groups. Because the split is random, the output will be different each time you run it. The training set will have three points and the test set two points, since 40 percent of five points is two points.

Fit on training data only

Now you will fit the line using only the training set. You already have the least_squares function from lesson 3 in least_squares.py. It takes two lists and returns the best slope and intercept.

Create a script called fit_train.py:

# fit_train.py
from data import hours, scores
from split_data import split_data
from least_squares import least_squares

train_hours, train_scores, test_hours, test_scores = split_data(hours, scores, 0.4)
slope, intercept = least_squares(train_hours, train_scores)
print('Slope fitted on training data:', slope)
print('Intercept fitted on training data:', intercept)

Run this script. The slope and intercept will be different from the ones you got in lesson 3 when you used all the data. That is because the line is now fitted to only three points instead of five.

Why evaluating on the same data is misleading

In lesson 2, you computed the mean squared error (MSE) of a line on the same data used to create it. That error is called the training error. It is often smaller than the error on new data because the line was chosen to minimise that particular error.

To see this, compute the MSE on both the training set and the test set using the line fitted on the training set. Add these lines to fit_train.py:

from mse import mean_squared_error

mse_train = mean_squared_error(train_hours, train_scores, slope, intercept)
mse_test = mean_squared_error(test_hours, test_scores, slope, intercept)
print('MSE on training set:', mse_train)
print('MSE on test set:', mse_test)

Run the script again. You will see two numbers. The training MSE is usually smaller than the test MSE. That is expected: the line was fitted to the training data, so it fits those points better than the unseen test points. If you had evaluated on all the data, you would have mixed training and test points, and the error would be somewhere in between, giving a false sense of how well the model generalises.

The test MSE is a better estimate of how the model will perform on new data. In the next lesson, you will use this test MSE to evaluate your model properly.

Key takeaways

  • Split your data into a training set and a test set before fitting a model.
  • Fit the model only on the training set, never on the test set.
  • Evaluating on the same data you trained on gives a misleadingly low error.
  • The test error is a more honest estimate of performance on new data.
  • Use a random split to avoid bias in choosing which points go where.

Try it

Write a new Python script called my_split.py. In it, import the hours and scores lists from data.py, the split_data function from split_data.py, the least_squares function from least_squares.py, and the mean_squared_error function from mse.py. Split the data with a test size of 0.2. Fit the line on the training set, then compute and print the MSE on both the training set and the test set. Run the script several times. Each run will give different numbers because the split is random. Check that the training MSE is usually smaller than the test MSE. A good result is a script that runs without errors and clearly shows the training and test MSE for each random split.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.