Skip to content

Lesson 4 of 6

Map Their Data to Yours: Field Mapping, Cleaning and Rejects

Write the mapping from the customer's fields to your own as a table first and as a tested function second, reject records you cannot use, and leave out personal data you do not need.

6 minForward Deployed Engineering

After this lesson you will be able to turn a customer's record into the shape your own product needs, decide what to clean and what to reject, and prove each rule with a test.

Write the mapping down first

Fernhill's system describes a repair in its own words. Your product's job list uses different words and stricter values. The mapping between them is a set of decisions, and most of them belong to the customer. So write it as a table before you write it as code, and send it to the repairs manager to check.

  • ref becomes job_ref, unchanged. It is the key that links your copy to their record.
  • addr becomes address, unchanged.
  • postcode becomes postcode, in capitals with single spaces. A repair with no postcode is rejected.
  • category becomes trade: PLUMB is a plumber, ELEC an electrician, HEAT a heating engineer, ROOF a roofer. Any other code is rejected.
  • priority becomes urgent: 1 for P1, otherwise 0.
  • reported becomes reported_on, the date only, read day first.
  • status becomes status, in lower case with no spaces around it. It must be open, closed or cancelled.
  • updated becomes updated_at, unchanged.
  • tenant_name and tenant_phone are not copied.

The repairs manager's answers are in that table. Only four categories go to this contractor. P1 means an emergency. And cancelled repairs stay in the list with that status, so the contractor can see that a job was called off.

The mapping in code

Create mapping.py. It has one function, which takes a repair as the API gives it and returns either a job or the reason there is no job.

"""Turn a repair, as Fernhill's API gives it, into a job, as the contractor's list needs it."""

from datetime import datetime

TRADES = {"PLUMB": "plumber", "ELEC": "electrician", "HEAT": "heating engineer", "ROOF": "roofer"}
STATUSES = {"open", "closed", "cancelled"}


def to_job(repair):
    """Return (job, None) for a repair the contractor can use, or (None, reason) for one it cannot."""
    ref = repair["ref"]
    trade = TRADES.get(repair["category"])
    if trade is None:
        return None, f"{ref}: unknown category {repair['category']}"
    postcode = " ".join(repair["postcode"].upper().split())
    if not postcode:
        return None, f"{ref}: no postcode"
    status = repair["status"].strip().lower()
    if status not in STATUSES:
        return None, f"{ref}: unknown status {repair['status']}"
    reported = datetime.strptime(repair["reported"], "%d/%m/%Y %H:%M")
    job = {
        "job_ref": ref,
        "address": repair["addr"],
        "postcode": postcode,
        "trade": trade,
        "urgent": 1 if repair["priority"] == "P1" else 0,
        "reported_on": reported.date().isoformat(),
        "status": status,
        "updated_at": repair["updated"],
    }
    return job, None

The function does not fetch anything and does not save anything. It only turns one dictionary into another. That makes it easy to test, and it means the rules live in one place where you can read them next to the table.

Clean what is obvious, reject the rest

There are two kinds of untidy value, and they need different treatment.

Some have exactly one sensible meaning. fh1 4cd is the postcode FH1 4CD, and open is the status open. The function tidies these without comment, because no information is lost and nobody would decide differently.

Others have no certain meaning. A repair with no postcode might be at an address you could look up, or might be a mistake. A category of PEST might belong to a different contractor, or might be a new code nobody told you about. Here the function does not guess. It returns no job, and a reason that names the repair and the problem.

Guessing looks helpful and is not. A guessed trade sends the wrong person to somebody's home, and nobody can tell later which values were guessed. A reject list is something the customer can act on: the contact centre can add the missing postcode, and the repairs manager can tell you what PEST means.

The date needs the same care. 02/03/2026 is the second of March at Fernhill. The format in the code, day then month, is a decision confirmed with the repairs manager. If a date arrives in a form the function does not expect, strptime raises an error and the run stops. That is the right outcome for something that should never happen: it is better to stop than to store a wrong date.

Leave out what you do not need

Look at how the job is built. Every field is copied by name. Nothing in the function says "copy the whole repair", so the tenant's name and phone number are never copied, and neither is any field Fernhill adds to its API next year.

The opposite habit, copying everything and removing what you do not want, fails in the worst way. On the day a new personal field appears in the API, it flows straight into your product without anyone deciding that it should.

Test the rules

Each rule came from something real in the data, so each gets a test. Create test_mapping.py:

import unittest

from mapping import to_job


def repair(**changes):
    base = {
        "ref": "R-1", "addr": "1 Test Street", "postcode": "FH1 1AA", "category": "PLUMB", "priority": "P2",
        "reported": "02/03/2026 08:14", "status": "Open", "updated": "2026-03-02T08:14:00",
        "tenant_name": "A. Tenant", "tenant_phone": "01632 960000",
    }
    base.update(changes)
    return base


class ToJobTests(unittest.TestCase):
    def test_a_clean_repair_becomes_a_job(self):
        job, reason = to_job(repair())
        self.assertIsNone(reason)
        self.assertEqual(job["trade"], "plumber")
        self.assertEqual(job["reported_on"], "2026-03-02")
        self.assertEqual(job["urgent"], 0)

    def test_the_date_is_read_as_day_then_month(self):
        job, _ = to_job(repair(reported="4/3/2026 12:10"))
        self.assertEqual(job["reported_on"], "2026-03-04")

    def test_status_and_postcode_are_tidied(self):
        job, _ = to_job(repair(status="open ", postcode="fh1  4cd"))
        self.assertEqual((job["status"], job["postcode"]), ("open", "FH1 4CD"))

    def test_priority_one_is_urgent(self):
        job, _ = to_job(repair(priority="P1"))
        self.assertEqual(job["urgent"], 1)

    def test_the_tenant_is_not_copied(self):
        job, _ = to_job(repair())
        self.assertNotIn("A. Tenant", job.values())
        self.assertNotIn("01632 960000", job.values())

    def test_a_repair_with_no_postcode_is_rejected(self):
        self.assertEqual(to_job(repair(postcode="")), (None, "R-1: no postcode"))

    def test_an_unknown_category_is_rejected(self):
        self.assertEqual(to_job(repair(category="PEST")), (None, "R-1: unknown category PEST"))


if __name__ == "__main__":
    unittest.main()

Run the tests from the integration folder. They do not need the practice server.

python -m unittest

The last lines of the output are these. The time will differ on your machine.

Ran 7 tests in 0.001s

OK

Read the names of the tests as a list. Together they state the mapping almost as clearly as the table does: a clean repair becomes a job, the date is read day first, status and postcode are tidied, priority one is urgent, the tenant is not copied, and two kinds of repair are rejected. When the customer changes a rule, you change one line and one test.

The test called test_the_tenant_is_not_copied deserves a word. It protects a promise made to the data protection lead. If somebody later adds a line that copies the phone number, this test fails before the change reaches the customer.

Key takeaways

  • Write the mapping as a table the customer can check before you write it as code.
  • Clean only what has one obvious meaning; reject everything else with a reason.
  • A rejected record with a clear reason is useful to the customer; a guessed value is a hidden error.
  • Copy fields by naming them, so that personal data you did not ask for never arrives by accident.
  • Each rule in the mapping gets a test, written from a real oddity in the data.

Try it

The repairs manager tells you that priorities are always P1, P2 or P3, and that anything else is a typing mistake. Add a rule to to_job that rejects any other priority with a reason in the same style as the others, for example R-1: unknown priority P9. Add one test for it to test_mapping.py. A good result runs eight tests, all passing, and the seven earlier tests are unchanged.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.