Skip to content

Lesson 4 of 6

Day 4: Test It Against What Really Happened

Rebuild the list as it would have been four weeks ago, count who went on to cancel, compare that with chance, and report the result with its limits.

6 minForward Deployed Engineering

After this lesson you will be able to test a prediction against what really happened, compare it with chance, and report the result without overstating it.

The idea of a back-test

The pass test in the brief says: of the members on the list, at least one in three cancel within four weeks, measured on a period the list was not built from.

You cannot wait four weeks to find out. But the export already contains the answer for an earlier day. Go back to 30 March, four weeks before the export ends. Make the list using only what was known then. Then look at the four weeks that followed, which the export also holds, and count who cancelled.

This is called a back-test. It is the nearest thing to trying the list for real that one week allows.

Nothing may leak from the future

A back-test is only worth something if the list for 30 March uses nothing from after 30 March. This is easy to get wrong without noticing. If the rule counted a member's visits in April, it would be using the very thing it is trying to predict, since members who cancel stop visiting. The result would look excellent and mean nothing.

That is why call_list in lesson 3 takes the day as an argument and ignores every visit on or after it, and why one of the six tests checks exactly that. The work was done on day 2 so that day 4 could be trusted.

The back-test script

Create backtest.py.

"""Test the call list against what really happened.

Make the list as it would have been four weeks before the export ends,
then count who cancelled in those four weeks.

Usage: python backtest.py [list size]
"""

import sys
from datetime import timedelta

from loader import load_members, load_visits
from score import EXPORT_END, LIST_SIZE, WINDOW, call_list, visit_counts


def measure(size=LIST_SIZE):
    as_of = EXPORT_END - timedelta(days=WINDOW)
    members = load_members()
    visits = load_visits(members)
    current = [m for m, member in members.items()
               if not (member["cancelled_on"] and member["cancelled_on"] < as_of)]
    left = [m for m in current if members[m]["cancelled_on"]]
    listed = [row["member_id"] for row in call_list(members, visits, as_of, size)]
    caught = [m for m in listed if m in left]
    return {
        "as_of": as_of,
        "members": len(current),
        "left": left,
        "listed": listed,
        "caught": caught,
        "missed": {m: visit_counts(visits[m], as_of) for m in left if m not in listed},
        "by_chance": len(listed) * len(left) / len(current),
    }


def main(size):
    result = measure(size)
    members, left, listed, caught = result["members"], result["left"], result["listed"], result["caught"]
    print(f"As of {result['as_of']}: {members} members, {len(listed)} on the call list")
    print(f"Cancelled in the next 4 weeks: {len(left)} ({len(left) / members:.0%} of members)")
    print(f"On the list and cancelled: {len(caught)} of {len(listed)} ({len(caught) / len(listed):.0%})")
    print(f"Cancellations the list caught: {len(caught)} of {len(left)} ({len(caught) / len(left):.0%})")
    print(f"A list of {len(listed)} picked by chance would hold about {result['by_chance']:.1f}")
    print("Missed, with visits before and after:")
    for member_id, (before, recent) in result["missed"].items():
        print(f"  {member_id}  {before} then {recent}")


if __name__ == "__main__":
    main(int(sys.argv[1]) if len(sys.argv) > 1 else LIST_SIZE)

measure does three things. It works out who was a member on 30 March. It asks call_list for the list as of that day. And it finds who cancelled afterwards, which it can see because their cancellation date is in the export.

The line by_chance is the comparison. If members were put on the list with no information at all, the share of them who cancelled would be the same as the share among all members.

Run it

python backtest.py
As of 2026-03-30: 106 members, 13 on the call list
Cancelled in the next 4 weeks: 14 (13% of members)
On the list and cancelled: 9 of 13 (69%)
Cancellations the list caught: 9 of 14 (64%)
A list of 13 picked by chance would hold about 1.7
Missed, with visits before and after:
  M010  9 then 5
  M020  7 then 4
  M065  8 then 7
  M089  10 then 9
  M102  7 then 5

To see the list itself as it stood that day, run python score.py 2026-03-30.

Read the result

Take the lines in order.

On 30 March there were 106 members, and the rule put 13 on the list. In the next four weeks 14 members cancelled, which is 13% of them, close to the one in eight the head of membership quoted.

Of the 13 on the list, 9 cancelled. That is 69%, against a pass test of one in three. A list of 13 chosen by chance would have held between one and two. The list is about five times better than chance.

The list caught 9 of the 14 cancellations. The team would have had the chance to call nearly two thirds of the members who left, a few weeks before they did.

So the answer to the question in the brief is yes, on this evidence.

Read the misses

Now look at what went wrong, because the customer will ask, and because it shows what the method cannot do.

Five members cancelled without being on the list. Two of them, M010 and M020, were coming less often: nine visits then five, seven then four. They fell just short of the rule's "half". The exercise looks at what loosening the rule would do.

The other three gave no sign at all: eight visits then seven, ten then nine. No rule based on visits could have found them. People also leave because they move house or change jobs, and the turnstile knows nothing about that.

Four members on the list did not cancel in the four weeks. That does not make them mistakes. Some may cancel in week five, and some were away. A caller would find out which.

How much the result is worth

Be exact about the limits when you report this. They do not weaken the result. They are part of it.

  • The numbers are small. Nine of 13 comes from one month and one sample. With another month it might be 7 or 11. It would be very unlikely to be 2, which is what chance gives, so the conclusion holds, but "69%" should not be quoted as if it were precise.
  • It was tested once. The rule was written before the back-test was run and has not been adjusted to improve the score. Keep it that way. Every adjustment made after seeing a result makes the result less honest.
  • It shows who is likely to leave. It does not show that a phone call keeps them. That was left out in the brief, and it is the next question.

What day 4 produced

You have a number, a comparison and a list of misses, and all three will go in front of the customer tomorrow. If the back-test had failed, you would still have a result: a clear "no, not from visits alone", found in four days and not four months. A proof of concept that answers no has done its job too.

Key takeaways

  • Test on a period the list was not built from: make the list as of a past day, then look at what happened next.
  • Nothing from after that day may leak into the list, or the test proves nothing.
  • Compare with what chance would give, because a number means little without it.
  • Read the misses one by one; they show what the method cannot see.
  • Say how small the numbers are when you report them.

Try it

Loosen the rule from a fall of a half to a fall of a third. In score.py, change recent * 2 > before to recent * 3 > before * 2, then run python backtest.py and python backtest.py 25. A good result reports 9 of 15 for the first run and 11 of 17 for the second, which catches 11 of the 14 cancellations, and explains two things: why the first run caught no more than before, and what the second would cost a team that can make 15 calls a week. Put the rule back afterwards.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.