Skip to content

Lesson 5 of 6

Sync Safely: Run Twice, Pick Up Changes and Try Without Writing

Join the client and the mapping into a sync that can be run again at any time, fetches only what changed since the last run, and has a dry run that writes nothing.

7 minForward Deployed Engineering

After this lesson you will be able to write a sync that is safe to run again, that fetches only what changed since the last run, and that you can try against a real system without writing anything.

Keep the practice server running, and REPAIRS_TOKEN set, as before. You need client.py from lesson 3 and mapping.py from lesson 4.

What goes wrong with a simple copy

The simplest sync fetches everything and adds it to the job list. Run it twice and every job is there twice. Let it fail half way and nobody knows which jobs were saved. Run it every fifteen minutes and it asks Fernhill's system for every repair it has ever held, ninety-six times a day.

Three properties fix this. The sync is safe to repeat, it asks only for what has changed, and it can be run without writing.

The sync

Create sync.py. The job list is a small SQLite database in a file called jobs.db, standing in for your product. SQLite comes with Python.

"""Copy repairs from Fernhill's API into the contractor's job list. Safe to run again at any time.

Usage: python sync.py [--full] [--dry-run]
"""

import os
import sqlite3
import sys

from client import fetch_all
from mapping import to_job

COLUMNS = ("job_ref", "address", "postcode", "trade", "urgent", "reported_on", "status", "updated_at")


def open_db(path):
    db = sqlite3.connect(path)
    db.execute(f"CREATE TABLE IF NOT EXISTS jobs ({', '.join(COLUMNS)}, PRIMARY KEY (job_ref))")
    db.execute("CREATE TABLE IF NOT EXISTS sync_state (name PRIMARY KEY, value)")
    return db


def last_seen(db):
    """The newest `updated` time saved by an earlier run, or an empty string on the first run."""
    row = db.execute("SELECT value FROM sync_state WHERE name = 'updated_since'").fetchone()
    return row[0] if row else ""


def save(db, repairs, dry_run=False):
    """Write the repairs that map to jobs. Return how many were saved and why the others were not."""
    saved, rejected, newest = 0, [], last_seen(db)
    marks = ", ".join(f":{column}" for column in COLUMNS)
    for repair in repairs:
        newest = max(newest, repair["updated"])
        job, reason = to_job(repair)
        if job is None:
            rejected.append(reason)
            continue
        if not dry_run:
            db.execute(f"INSERT OR REPLACE INTO jobs VALUES ({marks})", job)
        saved += 1
    if not dry_run:
        db.execute("INSERT OR REPLACE INTO sync_state VALUES ('updated_since', ?)", (newest,))
        db.commit()
    return saved, rejected


def main():
    full, dry_run = "--full" in sys.argv, "--dry-run" in sys.argv
    db = open_db("jobs.db")
    since = "" if full else last_seen(db)
    repairs = fetch_all(os.environ["REPAIRS_TOKEN"], since)
    saved, rejected = save(db, repairs, dry_run)
    print(f"{len(repairs)} fetched since {since or 'the beginning'}: {saved} saved, {len(rejected)} rejected")
    for reason in rejected:
        print(f"  rejected {reason}")
    total = db.execute("SELECT COUNT(*) FROM jobs").fetchone()[0]
    urgent = db.execute("SELECT COUNT(*) FROM jobs WHERE urgent = 1 AND status = 'open'").fetchone()[0]
    print(f"job list now holds {total} jobs, {urgent} of them urgent and open" + (" (dry run)" if dry_run else ""))


if __name__ == "__main__":
    main()

Try it without writing

Run it first with --dry-run:

python sync.py --dry-run
  got 429 for page 2, waiting 1s, then trying again
10 fetched since the beginning: 8 saved, 2 rejected
  rejected R-2204: no postcode
  rejected R-2207: unknown category PEST
job list now holds 0 jobs, 0 of them urgent and open (dry run)

A dry run fetches and maps exactly as a real run does, and skips only the writing. The line "8 saved" says what would be saved, and the job list still holds nothing. You have learned how many records there are, how many your rules accept and which are refused, at no risk.

Make this a habit. The first run against any real system is a dry run, and you read its numbers with the customer before the second.

Run it for real, twice

python sync.py
  got 429 for page 2, waiting 1s, then trying again
10 fetched since the beginning: 8 saved, 2 rejected
  rejected R-2204: no postcode
  rejected R-2207: unknown category PEST
job list now holds 8 jobs, 2 of them urgent and open

Now run the same command again:

0 fetched since 2026-03-05T15:30:00: 0 saved, 0 rejected
job list now holds 8 jobs, 2 of them urgent and open

The second run asked Fernhill's system one small question, received nothing and changed nothing. Two things in the code made that happen.

Safe to repeat

Each job is saved with INSERT OR REPLACE, and the table's key is job_ref, which is Fernhill's own reference. If a job with that reference is already there, it is replaced. If not, it is added. Saving the same repair a hundred times leaves one job.

This is the property to look for in every sync. It matters most on a bad day. If a run stops half way, because the network failed or the machine restarted, the cure is to run it again, and you can do that without thinking about what was already saved.

Notice also where db.commit() is. The jobs and the remembered time are saved together at the end. If the run fails before that, neither is saved, and the next run starts from the same place.

Only what changed

Every repair carries the time it was last changed. After a run, the sync stores the newest such time it has seen, in the sync_state table. The next run sends that time as updated_since and receives only repairs changed after it. That is why the second run fetched nothing.

When the contact centre adds the missing postcode to R-2204, that repair's time of last change moves forward. The next run fetches it, the mapping accepts it and it joins the job list. Nobody has to tell the sync. A repair that was rejected for a fault in the data mends itself once the data is corrected.

When to run a full sync

A repair rejected because of your own rules is different. R-2207 was refused because the mapping does not know PEST. If you later add PEST to the mapping, the repair itself has not changed, so an ordinary run will never fetch it again.

That is what --full is for. It ignores the remembered time and fetches everything. Because saving is safe to repeat, a full sync does no harm to the jobs already there. Run one whenever you change the mapping, and on a schedule the customer agrees, for example once a night, as a safety net.

What this sync cannot see

Be honest with the customer about the limits. There are three.

  • If Fernhill deletes a repair outright, the API simply stops listing it, and the job stays in your list. Ask how deletions happen. At Fernhill repairs are cancelled and not deleted, and a cancelled repair arrives as a change.
  • If two repairs change in the same second and a run ends between them, the second could be missed until the next full sync. The usual cure is to ask for changes from a few minutes before the remembered time, which is safe because repeats are harmless.
  • The sync trusts Fernhill's clock and its updated field. If some kind of change does not move that time forward, the sync will not see it. Ask the system's owner whether every change does.

Write these on the integration sheet. Each one is a small risk that the customer should know about, and the nightly full sync covers the first two.

Key takeaways

  • A sync must be safe to run twice: saving by the customer's own reference means a repeat changes nothing.
  • Remember the time of the newest change you have seen, and ask only for what changed after it.
  • A dry run does everything except write, and it is the first thing to run against a real system.
  • Report every run in numbers: fetched, saved, rejected and why.
  • Keep a full sync for the days when your own rules change, and know what your sync cannot see.

Try it

The repairs manager confirms that PEST repairs do go to this contractor, as work for a pest controller. Add that category to TRADES in mapping.py. Predict what python sync.py will print, then run it. Then run python sync.py --full. A good result explains why the first run fetched nothing, and ends after the full run with 9 saved, 1 rejected and a job list of 9 jobs, 2 of them urgent and open.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.

    Sync Safely: Run Twice, Pick Up Changes and Try Without Writing | Integrate with a Customer's Systems: APIs, Data and Access | Rolebound