Skip to content

Lesson 2 of 6

Day 1: Get the Data and Find Out What Is Wrong with It

Look at a customer's export before you build on it: count it, find what is missing, repeated or odd, ask the customer what each oddity means, and write down a decision for each.

7 minForward Deployed Engineering

After this lesson you will be able to profile a customer's data export, find its problems on the first day, and record a decision about each one.

Get the export

The brief asked for two files by day 1: the members, and their visits for the last 16 weeks. In a real proof of concept the customer's systems administrator sends them. For this course a script stands in for that person and writes the two files, with the same contents for everyone.

Create make_data.py in your poc-week folder. You do not need to study it. It plays the part of the customer's systems.

"""Stand-in for the customer's export: writes members.csv and visits.csv.

In a real proof of concept these two files come from the customer.
The fixed seed gives everybody the same data, so your results match the course.
"""

import csv
import random
from datetime import date, timedelta

START = date(2026, 1, 5)  # a Monday; the export covers the 16 weeks from here
WEEKS = 16
rng = random.Random(23)


def pick(low, high):
    """A whole number from low to high. Uses only random(), which is the same on every Python."""
    return low + int(rng.random() * (high - low + 1))


members, visits = [], []
for number in range(1, 121):
    member_id = f"M{number:03d}"
    kind = rng.random()
    joined = START - timedelta(days=pick(30, 900))
    plan = ["gym", "swim", "gym and swim"][pick(0, 2)]
    usual = pick(2, 3) if kind < 0.8 else 1          # visits in a normal week
    fade_from = pick(3, 14) if kind < 0.3 else None  # the week this member starts coming less
    leaves = None                                    # the week this member cancels
    if fade_from and rng.random() < 0.75:
        leaves = fade_from + pick(3, 6)
    elif not fade_from and rng.random() < 0.07:
        leaves = pick(2, 16)
    cancelled_on = ""
    for week in range(1, WEEKS + 1):
        monday = START + timedelta(weeks=week - 1)
        if leaves == week:
            cancelled_on = (monday + timedelta(days=pick(0, 4))).isoformat()
            break
        count = usual if rng.random() < 0.8 else usual - 1
        if fade_from and week >= fade_from:
            count = 1 if week < fade_from + 2 and rng.random() < 0.6 else 0
        for day in sorted({pick(0, 6) for _ in range(count)}):
            visits.append([member_id, (monday + timedelta(days=day)).isoformat()])
    members.append([member_id, joined.isoformat(), plan, cancelled_on])

# The untidy parts that every real export has.
members.insert(40, list(members[16]))                # one member appears twice
members[22][1] = members[87][1] = ""                 # two joining dates are missing
for week in range(0, WEEKS, 2):                      # the staff test card
    visits.append(["M999", (START + timedelta(weeks=week, days=2)).isoformat()])
early = [row for row in members if row[3] and row[3] < "2026-04-01"][:3]
for row in early:                                    # three visits after the member cancelled
    visits.append([row[0], (date.fromisoformat(row[3]) + timedelta(days=5)).isoformat()])
visits.sort(key=lambda row: (row[1], row[0]))
for row in visits[::40]:                             # one site writes dates the other way round
    year, month, day = row[1].split("-")
    row[1] = f"{day}/{month}/{year}"

for name, header, rows in (("members.csv", ["member_id", "joined", "plan", "cancelled_on"], members),
                           ("visits.csv", ["member_id", "date"], visits)):
    with open(name, "w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(header)
        writer.writerows(rows)
    print(f"wrote {name}: {len(rows)} rows")

Run it once:

python make_data.py
wrote members.csv: 121 rows
wrote visits.csv: 2709 rows

A real export would hold thousands of members. This one holds a sample small enough to read.

Before you go further, note how you were given the data. A real export contains real people. Keep it only where the customer agreed, do not copy it to other machines, and agree when it will be deleted. This export identifies members by number only, which is what you should ask for.

Look before you build

It is tempting to start on the interesting part. Do not. Anything you build today rests on what these files mean, and the cheapest day to find a problem is the first.

Create profile_data.py. It reads both files, changes nothing, and prints what is there.

"""Look at the export before trusting it. Changes nothing.

Usage: python profile_data.py
"""

import csv
from collections import Counter

with open("members.csv", newline="", encoding="utf-8") as handle:
    members = list(csv.DictReader(handle))
with open("visits.csv", newline="", encoding="utf-8") as handle:
    visits = list(csv.DictReader(handle))

ids = Counter(row["member_id"] for row in members)
print(f"members.csv: {len(members)} rows, {len(ids)} different members")
print("  appear more than once:", ", ".join(m for m, n in ids.items() if n > 1) or "none")
for column in ("joined", "plan", "cancelled_on"):
    print(f"  blank {column}: {sum(1 for row in members if not row[column])}")
print("  plans:", dict(Counter(row["plan"] for row in members)))

print(f"visits.csv: {len(visits)} rows")
slashed = [row["date"] for row in visits if "/" in row["date"]]
dashed = sorted(row["date"] for row in visits if "-" in row["date"])
print(f"  dates like 2026-03-30: {len(dashed)}, from {dashed[0]} to {dashed[-1]}")
print(f"  dates like 30/03/2026: {len(slashed)}, for example {slashed[0]}")
unknown = Counter(row["member_id"] for row in visits if row["member_id"] not in ids)
print("  visits by someone not in members.csv:", dict(unknown) or "none")
python profile_data.py
members.csv: 121 rows, 120 different members
  appear more than once: M017
  blank joined: 2
  blank plan: 0
  blank cancelled_on: 92
  plans: {'gym and swim': 43, 'gym': 38, 'swim': 40}
visits.csv: 2709 rows
  dates like 2026-03-30: 2641, from 2026-01-05 to 2026-04-26
  dates like 30/03/2026: 68, for example 05/01/2026
  visits by someone not in members.csv: {'M999': 8}

Read the profile

Go through the output a line at a time and mark anything you cannot explain.

  • There are 121 rows and only 120 different members. Member M017 appears twice.
  • Two members have no joining date.
  • 92 members have no cancellation date. That probably means they are still members, but it is a guess until someone confirms it.
  • Most visit dates are written year first. 68 are written day first.
  • Eight visits belong to M999, who is not in the members file.

None of this is unusual. An export that looks perfectly clean is rarer, and more worrying, than one like this.

Ask what it means

For each oddity there is a wrong move, which is to decide by yourself what it means, and a right one, which is to ask. Send one short message the same day, with the evidence.

"Five questions about the export. One: M017 is in the members file twice with the same details. Is that one person? Two: two members have no joining date. Three: does a blank cancellation date mean the person is still a member, and is the date the day they told you or the day the membership ended? Four: some visit dates are written day first. Five: who is M999?"

The systems administrator answered within the hour. M017 is one person, exported twice. The blank joining dates came from an older system. A blank cancellation date means a current member, and the date is the day the member gave notice. One centre's turnstiles write the day first. M999 is a test card the staff use to check the gates.

The third answer matters more than it looks. If the date had been the last day of the membership, a month after notice, every result in this course would have been a month out.

Write the data note

Record each decision where the next person will find it. This note goes into the write-up, and whoever builds the real system will start from it.

# Data note: Brightwell export, 16 weeks to 26 April 2026

1. M017 appears twice. Same person. Keep one row.
2. Two members have no joining date. Not needed for the question.
   Keep the members.
3. Blank cancelled_on means a current member. The date is the day
   the member gave notice. Confirmed by the systems administrator.
4. Visit dates come in two forms, year first and day first.
   Same meaning. Read both.
5. M999 is a staff test card. Leave its visits out.

Not checked: whether every visit is logged. A member who is let in
without scanning a card does not appear.

The last line is as useful as the rest. A proof of concept cannot check everything, and saying what you did not check keeps you honest later.

What day 1 produced

At the end of the first day you have no feature to show. You have something more valuable at this stage: you know the data is good enough to answer the question, you know its five faults, and the customer has seen you find them. The exercise finds a sixth.

Key takeaways

  • Look at the data before you write anything that depends on it.
  • Every export has problems; the first day is for finding them, not for being surprised by them later.
  • Ask the customer what an oddity means; do not guess.
  • Record each decision in a short data note, because the write-up and the real build will need it.
  • Check what each column means, especially dates.

Try it

Add one more check to profile_data.py: count the visits dated after the day the same member cancelled. Dates written like 2026-03-30 can be compared as text, and all the visits you are looking for are written that way. A good result prints 3. If you get 4, you have counted the member who appears twice in members.csv. Then write the question you would send to Brightwell about these visits.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.