Lesson 4 of 6
Map Their Data to Yours: Field Mapping, Cleaning and Rejects
Write the mapping from the customer's fields to your own as a table first and as a tested function second, reject records you cannot use, and leave out personal data you do not need.
6 minForward Deployed Engineering
After this lesson you will be able to turn a customer's record into the shape your own product needs, decide what to clean and what to reject, and prove each rule with a test.
Write the mapping down first
Fernhill's system describes a repair in its own words. Your product's job list uses different words and stricter values. The mapping between them is a set of decisions, and most of them belong to the customer. So write it as a table before you write it as code, and send it to the repairs manager to check.
refbecomesjob_ref, unchanged. It is the key that links your copy to their record.addrbecomesaddress, unchanged.postcodebecomespostcode, in capitals with single spaces. A repair with no postcode is rejected.categorybecomestrade: PLUMB is a plumber, ELEC an electrician, HEAT a heating engineer, ROOF a roofer. Any other code is rejected.prioritybecomesurgent: 1 for P1, otherwise 0.reportedbecomesreported_on, the date only, read day first.statusbecomesstatus, in lower case with no spaces around it. It must be open, closed or cancelled.updatedbecomesupdated_at, unchanged.tenant_nameandtenant_phoneare not copied.
The repairs manager's answers are in that table. Only four categories go to this contractor. P1 means an emergency. And cancelled repairs stay in the list with that status, so the contractor can see that a job was called off.
The mapping in code
Create mapping.py. It has one function, which takes a repair as the API gives it and returns either a job or the reason there is no job.
"""Turn a repair, as Fernhill's API gives it, into a job, as the contractor's list needs it."""
from datetime import datetime
TRADES = {"PLUMB": "plumber", "ELEC": "electrician", "HEAT": "heating engineer", "ROOF": "roofer"}
STATUSES = {"open", "closed", "cancelled"}
def to_job(repair):
"""Return (job, None) for a repair the contractor can use, or (None, reason) for one it cannot."""
ref = repair["ref"]
trade = TRADES.get(repair["category"])
if trade is None:
return None, f"{ref}: unknown category {repair['category']}"
postcode = " ".join(repair["postcode"].upper().split())
if not postcode:
return None, f"{ref}: no postcode"
status = repair["status"].strip().lower()
if status not in STATUSES:
return None, f"{ref}: unknown status {repair['status']}"
reported = datetime.strptime(repair["reported"], "%d/%m/%Y %H:%M")
job = {
"job_ref": ref,
"address": repair["addr"],
"postcode": postcode,
"trade": trade,
"urgent": 1 if repair["priority"] == "P1" else 0,
"reported_on": reported.date().isoformat(),
"status": status,
"updated_at": repair["updated"],
}
return job, NoneThe function does not fetch anything and does not save anything. It only turns one dictionary into another. That makes it easy to test, and it means the rules live in one place where you can read them next to the table.
Clean what is obvious, reject the rest
There are two kinds of untidy value, and they need different treatment.
Some have exactly one sensible meaning. fh1 4cd is the postcode FH1 4CD, and open is the status open. The function tidies these without comment, because no information is lost and nobody would decide differently.
Others have no certain meaning. A repair with no postcode might be at an address you could look up, or might be a mistake. A category of PEST might belong to a different contractor, or might be a new code nobody told you about. Here the function does not guess. It returns no job, and a reason that names the repair and the problem.
Guessing looks helpful and is not. A guessed trade sends the wrong person to somebody's home, and nobody can tell later which values were guessed. A reject list is something the customer can act on: the contact centre can add the missing postcode, and the repairs manager can tell you what PEST means.
The date needs the same care. 02/03/2026 is the second of March at Fernhill. The format in the code, day then month, is a decision confirmed with the repairs manager. If a date arrives in a form the function does not expect, strptime raises an error and the run stops. That is the right outcome for something that should never happen: it is better to stop than to store a wrong date.
Leave out what you do not need
Look at how the job is built. Every field is copied by name. Nothing in the function says "copy the whole repair", so the tenant's name and phone number are never copied, and neither is any field Fernhill adds to its API next year.
The opposite habit, copying everything and removing what you do not want, fails in the worst way. On the day a new personal field appears in the API, it flows straight into your product without anyone deciding that it should.
Test the rules
Each rule came from something real in the data, so each gets a test. Create test_mapping.py:
import unittest
from mapping import to_job
def repair(**changes):
base = {
"ref": "R-1", "addr": "1 Test Street", "postcode": "FH1 1AA", "category": "PLUMB", "priority": "P2",
"reported": "02/03/2026 08:14", "status": "Open", "updated": "2026-03-02T08:14:00",
"tenant_name": "A. Tenant", "tenant_phone": "01632 960000",
}
base.update(changes)
return base
class ToJobTests(unittest.TestCase):
def test_a_clean_repair_becomes_a_job(self):
job, reason = to_job(repair())
self.assertIsNone(reason)
self.assertEqual(job["trade"], "plumber")
self.assertEqual(job["reported_on"], "2026-03-02")
self.assertEqual(job["urgent"], 0)
def test_the_date_is_read_as_day_then_month(self):
job, _ = to_job(repair(reported="4/3/2026 12:10"))
self.assertEqual(job["reported_on"], "2026-03-04")
def test_status_and_postcode_are_tidied(self):
job, _ = to_job(repair(status="open ", postcode="fh1 4cd"))
self.assertEqual((job["status"], job["postcode"]), ("open", "FH1 4CD"))
def test_priority_one_is_urgent(self):
job, _ = to_job(repair(priority="P1"))
self.assertEqual(job["urgent"], 1)
def test_the_tenant_is_not_copied(self):
job, _ = to_job(repair())
self.assertNotIn("A. Tenant", job.values())
self.assertNotIn("01632 960000", job.values())
def test_a_repair_with_no_postcode_is_rejected(self):
self.assertEqual(to_job(repair(postcode="")), (None, "R-1: no postcode"))
def test_an_unknown_category_is_rejected(self):
self.assertEqual(to_job(repair(category="PEST")), (None, "R-1: unknown category PEST"))
if __name__ == "__main__":
unittest.main()Run the tests from the integration folder. They do not need the practice server.
python -m unittestThe last lines of the output are these. The time will differ on your machine.
Ran 7 tests in 0.001s
OKRead the names of the tests as a list. Together they state the mapping almost as clearly as the table does: a clean repair becomes a job, the date is read day first, status and postcode are tidied, priority one is urgent, the tenant is not copied, and two kinds of repair are rejected. When the customer changes a rule, you change one line and one test.
The test called test_the_tenant_is_not_copied deserves a word. It protects a promise made to the data protection lead. If somebody later adds a line that copies the phone number, this test fails before the change reaches the customer.
Key takeaways
- Write the mapping as a table the customer can check before you write it as code.
- Clean only what has one obvious meaning; reject everything else with a reason.
- A rejected record with a clear reason is useful to the customer; a guessed value is a hidden error.
- Copy fields by naming them, so that personal data you did not ask for never arrives by accident.
- Each rule in the mapping gets a test, written from a real oddity in the data.
Try it
The repairs manager tells you that priorities are always P1, P2 or P3, and that anything else is a typing mistake. Add a rule to to_job that rejects any other priority with a reason in the same style as the others, for example R-1: unknown priority P9. Add one test for it to test_mapping.py. A good result runs eight tests, all passing, and the seven earlier tests are unchanged.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.