Lesson 4 of 6
Change, Re-run, Compare: Fix Failures Without Breaking What Worked
Make one change at a time, re-run the whole evaluation set after each, and compare versions case by case so that a fix that breaks something else is caught at once.
9 minForward Deployed Engineering
After this lesson you will be able to improve an AI system in controlled steps: change one thing, re-run every case, and compare the new version with the old one case by case.
You need the files from lesson 3, and the file results_v1.csv that the last run saved.
First fix: the missed emergencies
The first version missed two emergencies because it knew only one way of describing danger. In classifier.py, add three phrases to the emergency list, so that KEYWORDS reads as follows. Change nothing else.
KEYWORDS = {
"emergency": ["smell gas", "smell of gas", "gas leak", "carbon monoxide", "alarm", "sparks"],
"billing": ["bill", "charge", "payment", "refund", "direct debit"],
"meter": ["meter", "reading"],
"moving": ["moving", "moved", "move out", "new address"],
"complaint": ["complaint", "unacceptable", "rude", "still waiting"],
}With a real model, the same step would be a change to its instructions: a sentence saying that carbon monoxide, alarms and sparks are emergencies too. The discipline that follows is identical.
Run the whole set again, as v2:
python evaluate.py messages.csv v224 cases, 20 right (83%)
queue cases right
emergency 4 4
billing 7 7
meter 5 4
moving 4 3
complaint 4 2
Wrong:
M10 meter -> emergency (0.44): My smart meter alarm keeps beeping about low credit.
M15 moving -> billing (0.44): Please close my account. I move out on Friday and need a final bill.
M19 complaint -> billing (0.44): This is unacceptable. Three payments taken and nobody replies.
M20 complaint -> billing (0.2): I have been passed between four people and I am fed up.
emergency cases missed: 0 (must be 0)
pass mark 85%: not met
RESULT: FAILAll four emergencies are now right, so the hard rule is met. The score rose from 19 to 20. But two were fixed, and 19 plus 2 is 21. Something else changed.
Compare case by case
Create compare.py. It reads two saved runs and lists what the change fixed, what it broke, and what is still wrong.
"""Compare two saved evaluation runs: what the change fixed, and what it broke.
Usage: python compare.py v1 v2
"""
import csv
import sys
def load(label):
with open(f"results_{label}.csv", newline="", encoding="utf-8") as handle:
return {row["id"]: row["right"] == "1" for row in csv.DictReader(handle)}
def main(before_label, after_label):
before, after = load(before_label), load(after_label)
groups = {
"fixed": [case for case in after if after[case] and not before.get(case, True)],
"broken": [case for case in after if not after[case] and before.get(case, False)],
"still wrong": [case for case in after if not after[case] and not before.get(case, True)],
}
for name, cases in groups.items():
print(f"{name} ({len(cases)}): {', '.join(cases) or 'none'}")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])python compare.py v1 v2fixed (2): M23, M24
broken (1): M10
still wrong (3): M15, M19, M20M10 was right and is now wrong. It is a customer whose smart meter alarm beeps about low credit. The new phrase "alarm" pulled it into the emergency queue.
Without the comparison you would have seen only that the score went up. This is the most common way an AI system gets worse over time: each change is tested on the case it was meant to fix, the fix works, and nobody looks at the cases that used to work.
Second fix, and one more
The phrase "alarm" is too broad, and "carbon monoxide" already covers the case it was added for. Remove "alarm". At the same time, deal with M20: add "fed up" to the complaint list.
KEYWORDS = {
"emergency": ["smell gas", "smell of gas", "gas leak", "carbon monoxide", "sparks"],
"billing": ["bill", "charge", "payment", "refund", "direct debit"],
"meter": ["meter", "reading"],
"moving": ["moving", "moved", "move out", "new address"],
"complaint": ["complaint", "unacceptable", "rude", "still waiting", "fed up"],
}Those are two changes in one step, which is acceptable here only because they touch different queues. As a rule, change one thing at a time. When two changes go in together and a case breaks, you have to work out which of them did it.
python evaluate.py messages.csv v324 cases, 22 right (92%)
queue cases right
emergency 4 4
billing 7 7
meter 5 5
moving 4 3
complaint 4 3
Wrong:
M15 moving -> billing (0.44): Please close my account. I move out on Friday and need a final bill.
M19 complaint -> billing (0.44): This is unacceptable. Three payments taken and nobody replies.
emergency cases missed: 0 (must be 0)
pass mark 85%: met
RESULT: PASSpython compare.py v2 v3fixed (2): M10, M20
broken (0): none
still wrong (2): M15, M19Two fixed, none broken. Both rules are met: no emergency missed, and 92% right against a pass mark of 85%.
What the passing score does not tell you
Before you send that result to the customer, be clear with yourself about what it means.
You changed the system three times while looking at these 24 cases. The system now fits them. It reached 92% on messages you had studied, and the phrase "fed up" is in the list because one of those messages used it. Next week's messages will use other words.
So 92% is the highest score you should expect, and the real figure will be lower. The honest measure is a set of cases the system has never been adjusted against. In a real project you would keep part of the evaluation set aside from the start and look at it only at the end. In this course the unseen cases arrive in lesson 5, from a week of live messages.
The cases that are still wrong
Two cases remain wrong, and you should show them to the customer and not hide them inside the percentage.
M15 is the customer who is leaving and wants a final bill. M19 is a complaint that mentions payments. Both are about two things at once, and both went to billing with a confidence of 0.44. The system was unsure, and the number says so.
That is the acceptable kind of mistake. In lesson 5, messages with a low confidence go to the triage desk, so a person will sort both of these.
Keep the record
The files results_v1.csv, results_v2.csv and results_v3.csv are now the history of the system. Keep them with the code, along with a line for each saying what changed.
The same routine applies to every later change, whoever makes it: a new sentence in the model's instructions, a different model, a new version of the same model from its provider. Run the whole set, compare with the last version, and read the broken list before anything else.
Key takeaways
- Re-run the whole set after every change, because a fix for one case can break another.
- Compare versions case by case; two versions with the same score can be wrong about different cases.
- Change one thing at a time, so that you know what caused each difference.
- A score on cases you tuned against is flattering; only unseen cases give an honest number.
- Show the customer the cases that are still wrong, with the reason.
Try it
Try a tempting fix for M19. In classifier.py, change the order in QUEUES so that complaint comes before billing, which makes a tie go to complaint. Run python evaluate.py messages.csv v4 and then python compare.py v3 v4. A good result reports that the score is still 22 of 24, that M19 is fixed and that M04 is now broken, and adds one sentence you would say to the triage desk lead about which of the two is the better mistake to make. Put the order back afterwards and run the evaluation as v3 once more.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.