Lesson 2 of 6
Build an Evaluation Set from the Customer's Real Cases
Collect real cases with the right answer for each, chosen to include the ordinary, the awkward and the dangerous, and check the set before you rely on it.
6 minForward Deployed Engineering
After this lesson you will be able to build an evaluation set from a customer's real cases, decide which cases belong in it, and check it before you use it.
What an evaluation set is
An evaluation set is a list of real inputs, each with the right answer. For Tessel it is a list of messages, each with the queue it should go to. You run the whole list through the system, count how many answers match, and look at the ones that do not.
It replaces opinion with a count. Without it, every discussion about whether the model is good enough turns on the last example somebody happened to see. With it, you can say that 19 of 24 were right, show the five that were wrong, and show the same count again after a change.
The right answers must come from the customer's own people. You do not know whether a chased refund is a billing matter or a complaint at Tessel. The triage desk does.
Which cases to include
Do not take a random handful. Choose in three groups.
- Ordinary cases, in roughly the proportions that really arrive. If a third of all messages are about bills, about a third of these should be.
- Awkward cases: messages about two things at once, messages with unusual wording, very short ones. Ask the triage desk which messages make them stop and think.
- Every form of the costly case. For Tessel that means emergencies, and not only the obvious "I can smell gas". Ask the emergency team for the ways real people have described danger.
The third group matters most and is the easiest to get wrong. A set with two emergencies, both worded the same way, cannot tell you whether the hard rule holds.
Who decides the right answer
Ask two people from the triage desk to label each case separately, without seeing each other's answers. Then compare.
Where they agree, you have an answer. Where they disagree, you have found something more useful: a case with no agreed rule. Take it to the triage desk lead and get a rule in writing. At Tessel two rules came out of this. A customer who is chasing a refund belongs in billing, even if they sound annoyed. And a customer who is leaving a home belongs in moving, even if they also ask for a final bill.
Those rules will go into the instructions the model receives, and they explain two of the cases below.
Protect the people in the messages
Real messages contain names, addresses, phone numbers and account numbers. Remove or replace them before a message leaves the customer's systems, and keep the set where the customer's security lead has agreed it may be kept. The messages in this course are invented for the same reason.
The set for this course
In your deploy-ai folder, create messages.csv with exactly this content. Each line has a case number, the queue the triage desk says is right, and the message.
id,queue,text
M01,billing,My bill this month is twice what I normally pay.
M02,billing,I was charged twice for the same payment.
M03,billing,Can I change the date of my direct debit?
M04,billing,I am still waiting for the refund you promised in March.
M05,billing,Why has my tariff gone up without any letter?
M06,billing,I cannot afford my payment this month. Can I pay in parts?
M07,billing,Please send me a copy of my last bill.
M08,meter,Here is my meter reading: 45213.
M09,meter,My smart meter display is blank.
M10,meter,My smart meter alarm keeps beeping about low credit.
M11,meter,When will someone come to fit the new meter?
M12,meter,The reading you used is an estimate and it is too high.
M13,moving,I am moving house on the 14th. What do I need to do?
M14,moving,We have moved in to a new flat and need to set up an account.
M15,moving,Please close my account. I move out on Friday and need a final bill.
M16,moving,My new address is 4 Hart Close. Please update your records.
M17,complaint,I want to make a complaint about how my call was handled.
M18,complaint,Your engineer was rude and left mud on my carpet.
M19,complaint,This is unacceptable. Three payments taken and nobody replies.
M20,complaint,I have been passed between four people and I am fed up.
M21,emergency,I can smell gas in my kitchen.
M22,emergency,There is a smell of gas by the front door and I feel dizzy.
M23,emergency,My carbon monoxide alarm is going off.
M24,emergency,There are sparks coming from the meter box.Read it as the triage desk would. M04 is the chased refund, and M15 is the customer who is leaving. M19 is a complaint that mentions payments. M23 and M24 are emergencies that never use the word "gas".
Twenty-four cases are enough to learn with. For a real deployment, aim for a few hundred, and for at least twenty or thirty of the case that must never be missed.
Check the set before you use it
A set can be wrong in dull ways: a queue name mistyped, one queue with hardly any cases, the same message pasted twice. Create check_set.py to catch those.
"""Check an evaluation set before using it: enough cases in every queue, and no case twice.
Usage: python check_set.py messages.csv
"""
import csv
import sys
from collections import Counter
QUEUES = ["emergency", "billing", "meter", "moving", "complaint"]
FEWEST = 4
def main(path):
with open(path, newline="", encoding="utf-8") as handle:
cases = list(csv.DictReader(handle))
per_queue = Counter(case["queue"] for case in cases)
print(f"{len(cases)} cases")
for queue in QUEUES:
note = "" if per_queue[queue] >= FEWEST else f" <- fewer than {FEWEST}"
print(f" {queue:<10}{per_queue[queue]:>3}{note}")
unknown = sorted(set(per_queue) - set(QUEUES))
if unknown:
print("Unknown queues:", ", ".join(unknown))
texts = Counter(case["text"].strip().lower() for case in cases)
repeated = [text for text, times in texts.items() if times > 1]
print(f"Repeated messages: {len(repeated)}")
if __name__ == "__main__":
main(sys.argv[1])Run it:
python check_set.py messages.csv24 cases
emergency 4
billing 7
meter 5
moving 4
complaint 4
Repeated messages: 0Every queue has at least four cases, no queue name is unknown, and no message appears twice.
Keep it apart, and keep it growing
Two habits keep an evaluation set honest.
First, do not show the model the cases you test it with. If you put example messages into the model's instructions, use different ones. A model that has seen the test will pass it and prove nothing.
Second, the set is never finished. Every mistake found later, in testing or in live use, becomes a new case. Lesson 6 comes back to this.
Key takeaways
- An evaluation set is a list of real inputs with the answer the customer's own people say is right.
- Choose cases on purpose: ordinary ones, awkward ones, and every form of the mistake that must never happen.
- When two people label a case differently, settle it with a written rule; that rule is worth more than the case.
- Remove personal details before a case leaves the customer's systems.
- Check the set itself for thin spots and repeats before you measure anything with it.
Try it
Add four cases of your own to messages.csv, with the next free numbers. Make one of them a message that is about two things at once, and decide which queue it belongs in and why. Run python check_set.py messages.csv. Then add one line that repeats an existing message word for word and run it again. A good result shows 28 cases after the first step and "Repeated messages: 1" after the second. Remove the four cases and the repeat afterwards, so that your numbers match the next lessons.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.