Lesson 3 of 6
Check the Evidence: Measure the Problem in a Sample of Data
Use a short Python script on a sample of the customer's orders to turn claims into numbers, and learn what a small sample can and cannot tell you.
7 minForward Deployed Engineering
After this lesson you will be able to take a sample of a customer's data, measure the problem in it with a short script, check the result by hand, and say what the numbers do and do not show.
You need Python 3.9 or later. The script uses only the standard library, so there is nothing to install. Where this course writes python, use python3 if that is the command on your machine.
Why measure before you plan
Lesson 2 ended with one claim and several unknowns. The operations director believes email orders cause the late deliveries. Nobody knows how many orders miss the 12:00 cut-off, how many are corrected after entry, or how much of the email comes from a few large customers.
A plan built on those unknowns is a guess. An hour with a sample of data can settle most of them, and it changes what you propose. If email orders turn out to be entered as fast as the others, the problem is somewhere else and the requested agent would fix nothing.
The sample
The IT contractor sends an export of two days of orders, a Monday and a Tuesday. The real file has about six hundred rows. This lesson uses twenty-four of them, so that you can check every number by hand.
In your scoping folder, create orders_sample.csv with exactly this content:
order_id,customer,channel,received,entered,corrected
O-3001,Brightwell Homes,email,2026-03-02 07:10,2026-03-02 09:40,no
O-3002,Castle Roofing,email,2026-03-02 07:25,2026-03-02 10:05,yes
O-3003,Ives Decorators,portal,2026-03-02 07:40,2026-03-02 07:40,no
O-3004,Brightwell Homes,email,2026-03-02 07:55,2026-03-02 10:50,no
O-3005,Elm Joinery,email,2026-03-02 08:15,2026-03-02 11:30,yes
O-3006,Dunmore Builders,email,2026-03-02 08:30,2026-03-02 12:20,no
O-3007,Marsh Glazing,phone,2026-03-02 08:45,2026-03-02 08:52,no
O-3008,Jarrow Paving,portal,2026-03-02 09:05,2026-03-02 09:05,no
O-3009,Castle Roofing,email,2026-03-02 09:20,2026-03-02 12:45,no
O-3010,Brightwell Homes,email,2026-03-02 09:50,2026-03-02 13:10,yes
O-3011,Fenwick Plastering,email,2026-03-02 10:30,2026-03-02 13:40,no
O-3012,Nash Electrical,phone,2026-03-02 11:15,2026-03-02 11:21,yes
O-3013,Kestrel Lofts,portal,2026-03-02 13:20,2026-03-02 13:20,no
O-3014,Brightwell Homes,email,2026-03-02 14:05,2026-03-02 15:00,no
O-3015,Ives Decorators,portal,2026-03-03 07:30,2026-03-03 07:30,no
O-3016,Castle Roofing,email,2026-03-03 07:45,2026-03-03 08:50,no
O-3017,Dunmore Builders,email,2026-03-03 08:10,2026-03-03 09:25,yes
O-3018,Garrow Kitchens,email,2026-03-03 08:40,2026-03-03 10:10,no
O-3019,Orchard Bathrooms,phone,2026-03-03 09:15,2026-03-03 09:24,no
O-3020,Brightwell Homes,email,2026-03-03 09:35,2026-03-03 10:20,no
O-3021,Jarrow Paving,portal,2026-03-03 10:00,2026-03-03 10:00,no
O-3022,Hale Flooring,email,2026-03-03 10:45,2026-03-03 12:15,yes
O-3023,Lowe Fencing,portal,2026-03-03 11:30,2026-03-03 11:30,no
O-3024,Pike Scaffolding,phone,2026-03-03 14:40,2026-03-03 14:48,noEach row is one order. channel says how it arrived. received is when the customer sent it, entered is when it reached the order system, and corrected says whether somebody had to fix the order after it was entered.
Read the rows before you write any code. You can already see that portal orders have the same time in both columns, and that the email orders on Monday morning wait a long time.
The script
Decide what to measure before you write the script. Four numbers would change the plan: how much of the volume each channel carries, how long an order waits to be entered, how many orders arrive in time for same-day delivery and are entered too late for it, and how many are corrected. One more matters because of what the order desk said: who sends the email orders.
Create profile_orders.py in the same folder:
"""Profile a sample of orders: where they come from, how long entry takes and how often it goes wrong.
Usage: python profile_orders.py orders_sample.csv
"""
import csv
import sys
from collections import Counter
from datetime import datetime
from statistics import median
CUT_OFF_HOUR = 12 # an order entered before noon goes out the same day
def read_rows(path):
with open(path, newline="", encoding="utf-8") as handle:
rows = list(csv.DictReader(handle))
for row in rows:
row["received"] = datetime.strptime(row["received"], "%Y-%m-%d %H:%M")
row["entered"] = datetime.strptime(row["entered"], "%Y-%m-%d %H:%M")
return rows
def minutes_to_enter(row):
return (row["entered"] - row["received"]).total_seconds() / 60
def missed_cut_off(row):
"""Received in time for same-day delivery, but entered too late for it."""
return row["received"].hour < CUT_OFF_HOUR <= row["entered"].hour
def profile(rows):
"""Return one summary for each channel, the busiest channel first."""
by_channel = {}
for row in rows:
by_channel.setdefault(row["channel"], []).append(row)
summaries = []
for channel, group in sorted(by_channel.items(), key=lambda item: -len(item[1])):
summaries.append(
{
"channel": channel,
"orders": len(group),
"share": round(100 * len(group) / len(rows)),
"median_minutes": round(median(minutes_to_enter(row) for row in group)),
"missed": sum(1 for row in group if missed_cut_off(row)),
"corrected": sum(1 for row in group if row["corrected"] == "yes"),
}
)
return summaries
def biggest_senders(rows, channel, count=3):
senders = Counter(row["customer"] for row in rows if row["channel"] == channel)
return senders.most_common(count)
def main():
if len(sys.argv) != 2:
sys.exit("Usage: python profile_orders.py orders_sample.csv")
rows = read_rows(sys.argv[1])
print(f"{len(rows)} orders")
print("channel orders share median minutes missed cut-off corrected")
for s in profile(rows):
print(
f"{s['channel']:9}{s['orders']:6}{s['share']:6}%{s['median_minutes']:16}"
f"{s['missed']:16}{s['corrected']:11}"
)
print("Biggest senders by email:")
for customer, orders in biggest_senders(rows, "email"):
print(f" {customer}: {orders}")
if __name__ == "__main__":
main()Three things in this script are worth noticing. missed_cut_off puts the customer's own rule into one small function with a name, so that anyone reading the script can see which rule was applied. The delay is reported as a median and not as an average, because one order that waited all day would pull an average up and hide what a typical order experiences. And profile returns its results without printing them, so the numbers can be tested or reused.
Run it and read the numbers
In a terminal, in the scoping folder, run:
python profile_orders.py orders_sample.csvYou should see:
24 orders
channel orders share median minutes missed cut-off corrected
email 14 58% 155 5 5
portal 6 25% 0 0 0
phone 4 17% 8 0 1
Biggest senders by email:
Brightwell Homes: 5
Castle Roofing: 3
Dunmore Builders: 2Now read it as answers to the unknowns from lesson 2.
- Email carries more than half of the orders: 14 of 24.
- A typical email order waits 155 minutes to be entered, which is more than two and a half hours. A phone order waits 8 minutes and a portal order does not wait.
- Five orders arrived in time for same-day delivery and were entered too late for it. All five came by email.
- Five of the 14 email orders were corrected after entry. One of the 4 phone orders was, and no portal order.
- Three customers sent 10 of the 14 email orders.
The director's claim holds up in this sample. The last line is something the director did not say, and it will shape the plan: most of the email comes from three customers.
Check one number by hand
Before you repeat a number to a customer, check it against the rows. Take the five missed orders. Look down the file for email orders received before 12:00 and entered at 12:00 or later. You should find O-3006, O-3009, O-3010 and O-3011 on the Monday and O-3022 on the Tuesday. That is five, and the script agrees.
The check takes two minutes. It protects you from the worst moment in customer work, which is a customer finding the mistake in your number during the meeting where you present it.
What a sample can and cannot tell you
This sample is twenty-four orders from two days, and one of the days is a Monday, which the order desk said is the worst morning. It can tell you that the problem is real and roughly what shape it has. It cannot tell you how large the problem is across a month, or whether the same three customers always send the most.
So say what the sample is whenever you quote from it, and ask for more before you promise anything. A month of orders from the same export would turn "5 of 13 in a two-day sample" into a figure the director can plan around. Add that request to plan.md under "What we need from the customer".
Turn the numbers into sentences
The customer will not read a table of medians. Write what you found as sentences a director could repeat to a colleague, and put them in plan.md under "Evidence":
- In a two-day sample, 13 email orders arrived before 12:00, and 5 of them were entered too late to go out the same day. No portal or phone order missed the cut-off.
- A typical email order waited about two and a half hours to be typed in.
- About a third of the email orders had to be corrected after entry.
- Three customers sent 10 of the 14 email orders.
Each sentence says what was counted and out of how many. A reader can disagree with what the numbers mean, and cannot be confused about what they are.
Key takeaways
- A claim from discovery becomes a fact, or is overturned, when you count it in the customer's data.
- Measure the things the build would change: how many, how long, how often wrong, and who sends them.
- Check at least one number by hand against the rows before you repeat it to anyone.
- Say how big the sample was and what it leaves out every time you quote a number from it.
- Give the customer sentences, not tables.
Try it
The three biggest senders matter to the plan, so print their share. Change only the main function of profile_orders.py so that it ends with one more line, in this form: Three biggest senders: 10 of 14 email orders (71%). The script must work the numbers out from the rows and not have them typed in. Predict the line first from the output you already have, then run the script. A good result prints everything it printed before, followed by that line, and still gives a sensible line if you delete a few rows from the sample.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.