Skip to content

Lesson 3 of 6

Measure It: An Evaluation Script with a Pass Mark and a Hard Rule

Run every case through the system, count the results by queue, apply the two rules agreed with the customer, and learn to read what the numbers hide.

5 minForward Deployed Engineering

After this lesson you will be able to write a script that runs an evaluation set, reports the result in a form a customer can read, and decides pass or fail by rules agreed in advance.

The stand-in for the model

Create classifier.py. Its one function, classify, takes a message and returns a queue and a confidence.

"""Sort a customer's message into a queue.

`classify` stands in for the call to an AI model, so that this course runs with no account and
no key. A real version sends the message and the list of queues to a model the customer has
approved and reads back a queue and a confidence. Nothing else in the course depends on which
of the two it is.
"""

QUEUES = ["emergency", "billing", "meter", "moving", "complaint"]
KEYWORDS = {
    "emergency": ["smell gas", "smell of gas", "gas leak"],
    "billing": ["bill", "charge", "payment", "refund", "direct debit"],
    "meter": ["meter", "reading"],
    "moving": ["moving", "moved", "move out", "new address"],
    "complaint": ["complaint", "unacceptable", "rude", "still waiting"],
}


def classify(text):
    """Return (queue, confidence). Confidence runs from 0 to 1."""
    text = text.lower()
    hits = {queue: sum(1 for word in KEYWORDS[queue] if word in text) for queue in QUEUES}
    total = sum(hits.values())
    if total == 0:
        return "billing", 0.2  # nothing recognised: the busiest queue, with low confidence
    best = max(QUEUES, key=lambda queue: hits[queue])
    return best, round(hits[best] / (total + 0.25), 2)

In a real deployment, classify would send the message and the list of queues to an AI model and read back the answer. Here it counts key phrases, so that the course runs on any machine with no account and gives everyone the same results. It makes the kinds of mistake a real model makes: it handles familiar wording well and unfamiliar wording badly.

What matters is the shape. Text goes in, and a queue and a confidence come out. Everything else you build in this course only uses that shape, so it would work unchanged around a real model.

The evaluation script

Create evaluate.py.

"""Run every case in the evaluation set and decide whether the result is good enough to ship.

Usage: python evaluate.py messages.csv [label]
With a label, the result of every case is also saved to results_<label>.csv.
"""

import csv
import sys

from classifier import QUEUES, classify

PASS_MARK = 0.85  # agreed with the customer: the share of all cases that must be right
NEVER_MISS = "emergency"  # agreed with the customer: no case in this queue may go anywhere else


def run(path):
    """Classify every case. Each result holds the case, what came back and whether it was right."""
    with open(path, newline="", encoding="utf-8") as handle:
        cases = list(csv.DictReader(handle))
    results = []
    for case in cases:
        got, confidence = classify(case["text"])
        results.append({**case, "got": got, "confidence": confidence, "right": got == case["queue"]})
    return results


def report(results):
    """Print the results and return True if both of the customer's rules are met."""
    correct = sum(result["right"] for result in results)
    share = correct / len(results)
    print(f"{len(results)} cases, {correct} right ({share:.0%})")
    print(f"{'queue':<10}{'cases':>6}{'right':>6}")
    for queue in QUEUES:
        rows = [result for result in results if result["queue"] == queue]
        print(f"{queue:<10}{len(rows):>6}{sum(row['right'] for row in rows):>6}")
    wrong = [result for result in results if not result["right"]]
    if wrong:
        print("Wrong:")
    for result in wrong:
        print(f"  {result['id']} {result['queue']} -> {result['got']} ({result['confidence']}): {result['text']}")
    missed = sum(1 for result in wrong if result["queue"] == NEVER_MISS)
    print(f"{NEVER_MISS} cases missed: {missed} (must be 0)")
    print(f"pass mark {PASS_MARK:.0%}: {'met' if share >= PASS_MARK else 'not met'}")
    return missed == 0 and share >= PASS_MARK


def save(results, label):
    with open(f"results_{label}.csv", "w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(["id", "queue", "got", "right"])
        for result in results:
            writer.writerow([result["id"], result["queue"], result["got"], int(result["right"])])


if __name__ == "__main__":
    results = run(sys.argv[1])
    passed = report(results)
    if len(sys.argv) > 2:
        save(results, sys.argv[2])
    print("RESULT:", "PASS" if passed else "FAIL")
    sys.exit(0 if passed else 1)

The two rules from the deployment sheet are at the top, as named values: the pass mark, and the queue that must never be missed. Anyone who opens the file can see what was agreed.

run puts every case through classify and records what came back. report prints the counts and applies the rules. The last lines turn the outcome into the word PASS or FAIL and into an exit status, which is how another program, such as a release pipeline, can refuse to ship a failing version.

Run it

Give the run a label, v1. The script then also saves each case's result to a file, which lesson 4 uses.

python evaluate.py messages.csv v1
24 cases, 19 right (79%)
queue      cases right
emergency      4     2
billing        7     7
meter          5     5
moving         4     3
complaint      4     2
Wrong:
  M15 moving -> billing (0.44): Please close my account. I move out on Friday and need a final bill.
  M19 complaint -> billing (0.44): This is unacceptable. Three payments taken and nobody replies.
  M20 complaint -> billing (0.2): I have been passed between four people and I am fed up.
  M23 emergency -> billing (0.2): My carbon monoxide alarm is going off.
  M24 emergency -> meter (0.8): There are sparks coming from the meter box.
emergency cases missed: 2 (must be 0)
pass mark 85%: not met
RESULT: FAIL

Read the result

Start with the first line: 19 of 24 right, which is 79%. On its own that number sounds close to the pass mark of 85%, as if a little tuning would finish the job. Now read the rest.

The table shows where the mistakes are. Billing and meter are perfect. Emergency, the one queue that must be perfect, has two right out of four.

The list of wrong cases shows what they are. M23 is a carbon monoxide alarm, sent to billing. M24 is sparks from a meter box, sent to the meter team. Both would have waited a day.

This is why the hard rule is checked on its own line. A system could reach 96% by getting every case right except one emergency, and it would still be unsafe to ship. One overall percentage treats every mistake as equal, and the customer told you in lesson 1 that they are not.

Confidence is not safety

Look at the number in brackets after each wrong case. It is the confidence.

M23 went to billing with a confidence of 0.2. The system was unsure, and a rule such as "send unsure messages to a person" would have caught it.

M24 went to the meter team with a confidence of 0.8. The system was sure and wrong. The message contains the word "meter", and nothing it recognises as danger. No rule based on confidence would have caught that one.

Real models do the same. They can give a wrong answer in a certain tone, and their stated confidence can be high when they are wrong. So confidence is useful for deciding which messages a person should look at, and it is never enough to protect against the mistake that must not happen. Lesson 5 adds a protection that does not depend on the model at all.

What to tell the customer

Report this result as it is, the same day. A good message to the head of customer service is short.

"First test on 24 cases: 19 right. Both rules fail. Two of the four emergency cases went to the wrong queue, because they describe danger without mentioning gas. We are fixing that first and will send the comparison."

It gives the count, says which rule failed and why, and says what happens next. A customer who sees an honest failing result early will believe the passing result later.

Key takeaways

  • One overall percentage hides the mistake that matters; always break the result down and list the wrong cases.
  • The hard rule is checked separately from the pass mark, and failing either one fails the run.
  • A confident answer can be a wrong answer, so confidence alone is never the safety measure.
  • End the script with a pass or a fail that another program can read.

Try it

Before you change anything, predict the answer: if the pass mark were 75% and not 85%, would this run pass? Then change PASS_MARK to 0.75 in evaluate.py and run it again. A good result explains, in one sentence, why the last line still says FAIL although the pass mark is now met. Change the pass mark back to 0.85 afterwards.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.