Skip to content

Lesson 6 of 6

Go Live and Hand Over: Least Access, Monitoring and the Runbook

Move the sync from the practice server to the customer's real system in careful steps, make sure somebody learns when it breaks, and leave a runbook the customer can follow without you.

6 minForward Deployed Engineering

After this lesson you will be able to take an integration from a practice server to a customer's real system, arrange for somebody to be told when it stops working, and hand it over with a runbook.

From the practice server to the real one

The code is ready to point at Fernhill's real system, and nothing in it needs to change. The address comes from the REPAIRS_API setting and the token from REPAIRS_TOKEN, so going live is a change to two settings.

Do it in steps, and stop after each to look.

  • Make one request, with first_call.py. It proves that the address, the token and the network path all work. If Fernhill only accepts requests from known network addresses, this is where you find out.
  • Run python sync.py --dry-run. Nothing is written. You get the counts for the real data.
  • Compare those counts with a figure the customer trusts. Ask the repairs manager how many repairs their own screen shows, and how many are open emergencies. If your numbers differ, find out why before going further.
  • Read the reject list with the repairs manager. The real data will hold categories, statuses and oddities that ten practice records did not. Each one is a question for the customer and then a line in the mapping and a test.
  • Run the real sync once by hand, and look at the job list with somebody from the contractor.
  • Only then put it on a schedule.

Expect the real data to surprise you. The practice server has ten tidy records and the real system has years of history. If the first dry run rejects a fifth of the repairs, the steps have worked: you found out before anything was written.

Least access, checked

In lesson 1 you asked for a token that can only read repairs. Check that you got one. Ask the IT lead to confirm what the account can do, or try a request it should not be allowed to make and confirm that it is refused.

Then settle where things run and who holds what.

  • The token lives in a secrets store on the machine that runs the sync, and nowhere else. Remove it from your own machine once the handover is done.
  • The expiry date goes into a calendar that somebody will still be reading in a year, with the name of the person who renews it. An expired token is one of the most common reasons an integration stops, and it is entirely predictable.
  • The sync runs on a schedule inside the agreed limit: every fifteen minutes, with a full sync once a night.

Know when it breaks

Every integration breaks at some point. The customer's system is upgraded, a token expires, or a new category appears. What matters is whether somebody finds out within minutes or the contractor finds out days later from a missed emergency.

Decide what counts as broken, and for each case who is told and how.

  • A run fails. The program ends with an error, so the scheduler can see it and send an alert.
  • No run has succeeded for an hour. This catches the case where the scheduler itself has stopped.
  • Nothing has been fetched for a whole working day. Each run "succeeded", and still no day at Fernhill passes without a repair being changed, so something is wrong.
  • The share of rejected repairs jumps. A new code or a changed format usually shows up here first.

The third case is the one people forget. A sync that runs happily and fetches nothing looks healthy on every dashboard that only counts errors.

Logs without personal data

The lines that sync.py prints are its log: how many repairs were fetched, since when, how many were saved, and which were rejected and why. Keep them. When somebody asks why a job was missing on Tuesday, those lines are the answer.

Look at what the lines do not contain. They name a repair by its reference and never print the address, the tenant or the token. Logs are copied into tickets and read by people who have no right to the data, so the rule is the same as for the mapping: a log holds references, counts and reasons.

The runbook

A runbook is a short page for the person who is on duty when the sync fails and who did not write it. It says what the thing is, how to check it, what to do for each known failure, and who to call. Write it for a reader who is capable and has never seen the code.

# Runbook: Fernhill repairs sync

## What it does
Copies repairs from Fernhill's housing system into the job list.
Runs every 15 minutes. A full sync runs at 02:00.

## Is it healthy?
The last line of the latest run shows the size of the job list.
Healthy: a run in the last 15 minutes, and repairs fetched at some
point today.

## If a run fails
Status 401: the token has expired or been replaced. Ask Fernhill's
  IT lead for a new one. Put it in the secrets store. Never send it
  by email or chat.
Status 429, or 500 and above: Fernhill's system is busy or down.
  The sync retries by itself. If it fails for an hour, tell the
  IT lead the time and the status.
An error about a date or a missing field: the data has changed
  shape. Do not edit the data. Call the owning team.

## If repairs are rejected
"no postcode": pass the reference to Fernhill's contact centre.
"unknown category": ask the repairs manager what the code means.
  After the mapping is changed, run: python sync.py --full

## Safe to do at any time
python sync.py --dry-run   (writes nothing)
python sync.py             (repeats are harmless)
python sync.py --full      (fetches everything again, harmless)

## Dates to remember
Token expires: 12 months after issue. Renew a month early.

## Contacts
Fernhill IT lead: access and faults in their system.
Fernhill repairs manager: what the data means.
Owning team: changes to the code.

Test the runbook the way you would test code. Give it to a colleague, stop the practice server and ask them to work out what is wrong using only the page.

When their system changes

The customer's system will change, and the change will not be planned around you. A supplier's upgrade can rename a field or add a status. You asked in lesson 1 how much notice you would get, and the honest answer is often none.

So the integration defends itself. The mapping rejects values it does not know and does not guess. An unexpected date format stops the run. The reject count is watched. Each of those turns a silent change into a visible one on the same day.

Hand over

The work is finished when the customer can run it without you. That means four things exist and have been read by the people who need them: the integration sheet from lesson 1, brought up to date; the mapping table from lesson 4, agreed by the repairs manager; the runbook; and the list of what the sync cannot see, from lesson 5.

Walk the IT lead and the contractor's planner through one real run and one deliberate failure. Then look back at what you built. The code is under two hundred lines. What makes it an integration the customer can trust is the rest: the least access that works, rules the customer has checked, numbers on every run, and a page that tells the next person what to do.

Key takeaways

  • The first run against the real system is a dry run, and its numbers are checked against a figure the customer trusts.
  • An integration that fails silently is worse than none, so decide what counts as a failure and who is told.
  • Logs record references, counts and reasons, and never tokens or personal data.
  • Write down when the token expires and who renews it, on the day you receive it.
  • A runbook lets somebody who has never seen the code check, restart and escalate.

Try it

Make sync.py report failure in a way a scheduler can see. At the end of main, if anything was fetched and more than a quarter of it was rejected, print a line that says so and call sys.exit(1). Run python sync.py --full --dry-run and check the exit status (echo $? on macOS or Linux, echo $LASTEXITCODE in PowerShell): it should be 0, because no more than 2 of the 10 repairs are rejected, and that is not more than a quarter. Then temporarily remove HEAT and ROOF from TRADES in mapping.py and run the same command. A good result now shows at least 4 rejected and an exit status of 1. Put both back afterwards.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.