Lesson 5 of 6
Make the Smallest Safe Change
Weigh the possible fixes by who owns them and what they hide, make the smallest change on your own side behind a switch, try it without sending anything, and write the plan the customer approves.
9 minForward Deployed Engineering
After this lesson you will be able to choose between possible fixes in a system you only partly own, make the change in a way that can be undone in a minute, and get it approved.
You have a cause and a failing test. Three owners each hold a part of the fault: the gateway vendor, whose system answers slowly; the customer, who moved the batch; and your company, whose code repeats a message it cannot be sure has failed.
The options, and who owns each
Write every option down, including the ones you will not take. The customer will think of them too, and it is better that they hear your reasons first.
Option Owner What it does
---------------------------- ---------- -----------------------------
Move the batch back to 07:00 Quillfield avoids the slow quarter-hour;
the fault returns whenever the
gateway is slow at another time
Make the gateway faster vendor removes the slowness; not in
our hands or the customer's
Wait longer than 5 seconds us fewer repeats on most days;
a slower day brings them back
Stop trying again us no repeats; a reminder that
really failed is never sent
Send a message id us a repeat is recognised by the
gateway and not deliveredThe first two are not yours to make. You can recommend them, and lesson 6 does. Of the three on your side, two only move the problem. A longer wait would have covered 12 May, when the slowest answer took a little over eight seconds, and it would fail on the first day an answer took longer. Not trying again swaps repeated reminders for missing ones, and a missed appointment costs a practice more than a text.
The last option removes the cause on your side. If every attempt for one reminder carries the same identifier, and the gateway ignores an identifier it has already seen, then trying again becomes harmless. It is also the smallest change: one extra value on a call the service already makes.
Confirm before you depend on it
The whole change rests on belief 2 from lesson 4: that the gateway does not send a message whose id it has already seen. So far that comes from the vendor's documentation. Documentation can be out of date, and "ignored" can hide conditions, such as how long an id is remembered.
Ask through the IT lead, since the contract is Quillfield's. The vendor confirms that an id is remembered for 24 hours and that a repeat within that time is acknowledged and not delivered. Where a vendor offers a test line, go one step further with the customer's agreement: send the same id twice to your own phone and count the texts. In this course the stand-in plays the gateway, so that step cannot be shown here. On a real engagement, do not skip it.
Make the change behind a switch
Replace sender.py with this version.
"""The part of the reminder service that sends one text. This is the code we own."""
import hashlib
import json
TIMEOUT = 5.0 # seconds to wait for the gateway's answer
ATTEMPTS = 3
class GatewayTimeout(Exception):
pass
class Rejected(Exception):
pass
def load_settings(path="settings.json"):
"""Read the switches. A missing file means every switch is off."""
try:
with open(path, encoding="utf-8") as handle:
return json.load(handle)
except FileNotFoundError:
return {}
def send_reminder(gateway, appt, to, text, log=print, settings=None):
"""Send one reminder. Return True when the gateway confirms it, or would have been asked."""
settings = load_settings() if settings is None else settings
extra = {}
if settings.get("use_message_id"):
# Made once, from the message itself: every retry carries the same id,
# and a different number or wording gives a different one.
digest = hashlib.sha256(f"{to}|{text}".encode("utf-8")).hexdigest()[:8]
extra["message_id"] = f"{appt}-{digest}"
note = f" message_id={extra['message_id']}" if extra else ""
if settings.get("dry_run"):
log(f"would send appt={appt}{note}")
return True
for attempt in range(1, ATTEMPTS + 1):
log(f"send appt={appt} attempt={attempt}{note}")
try:
gateway.send(to=to, text=text, timeout=TIMEOUT, **extra)
except GatewayTimeout:
log(f"timeout appt={appt} after={TIMEOUT}s attempt={attempt}")
continue
except Rejected as error:
log(f"rejected appt={appt} reason={error}, not retried")
return False
log(f"sent appt={appt} attempt={attempt}")
return True
log(f"gave up appt={appt} attempts={ATTEMPTS}")
return FalseThree things have changed, and each is deliberate.
- The switch.
use_message_idis read from a settings file, and a missing file or a missing value means off. With the switch off the function behaves exactly as before. - The id. It is made once, before the loop, so every attempt carries the same one. It is built from the appointment, the phone number and the wording. A retry therefore has the same id. A new reminder for the same appointment, sent because the time was moved, has a different wording and so a different id, and it is delivered. Getting this wrong in the other direction would silently drop real reminders.
- The dry run. With
dry_runon, the function logs what it would send and calls nothing. It is for a first trial on a test machine. It must never be left on where real reminders are due, because nothing is sent.
The settings are read on every call, so changing the file takes effect at the next reminder with no restart. Save the file as settings.json, with both switches off.
{
"use_message_id": false,
"dry_run": false
}Test both positions of the switch
Replace test_sender.py with this version. The tests pass their settings in directly, so they do not depend on the file.
import unittest
from fake_gateway import FakeGateway
from sender import send_reminder
NUMBER = "+440000000000"
TEXT = "Your appointment is tomorrow."
OFF = {}
ON = {"use_message_id": True}
def quiet(line):
pass
class SendReminderTests(unittest.TestCase):
def test_a_quick_answer_means_one_text(self):
for settings in (OFF, ON):
gateway = FakeGateway(reply_times=[0.7])
self.assertTrue(send_reminder(gateway, "A1", NUMBER, TEXT, log=quiet, settings=settings))
self.assertEqual(len(gateway.delivered), 1)
def test_a_slow_answer_still_means_one_text(self):
gateway = FakeGateway(reply_times=[6.8, 0.9])
self.assertTrue(send_reminder(gateway, "A1", NUMBER, TEXT, log=quiet, settings=ON))
self.assertEqual(len(gateway.delivered), 1)
def test_with_the_switch_off_nothing_has_changed(self):
gateway = FakeGateway(reply_times=[6.8, 0.9])
self.assertTrue(send_reminder(gateway, "A1", NUMBER, TEXT, log=quiet, settings=OFF))
self.assertEqual(len(gateway.delivered), 2)
def test_two_different_reminders_are_both_sent(self):
gateway = FakeGateway()
send_reminder(gateway, "A1", NUMBER, TEXT, log=quiet, settings=ON)
send_reminder(gateway, "A2", NUMBER, TEXT, log=quiet, settings=ON)
self.assertEqual(len(gateway.delivered), 2)
def test_a_changed_message_for_the_same_appointment_is_sent(self):
gateway = FakeGateway()
send_reminder(gateway, "A1", NUMBER, "Your appointment is tomorrow at 9:00.", log=quiet, settings=ON)
send_reminder(gateway, "A1", NUMBER, "Your appointment is tomorrow at 14:30.", log=quiet, settings=ON)
self.assertEqual(len(gateway.delivered), 2)
def test_a_bad_number_is_not_tried_again(self):
gateway = FakeGateway(bad_numbers=[NUMBER])
self.assertFalse(send_reminder(gateway, "A1", NUMBER, TEXT, log=quiet, settings=ON))
self.assertEqual(len(gateway.delivered), 0)
def test_a_dry_run_sends_nothing(self):
gateway = FakeGateway()
lines = []
settings = {"use_message_id": True, "dry_run": True}
send_reminder(gateway, "A1", NUMBER, TEXT, log=lines.append, settings=settings)
self.assertEqual(len(gateway.delivered), 0)
self.assertTrue(lines[0].startswith("would send appt=A1 message_id=A1-"))
if __name__ == "__main__":
unittest.main()python -m unittestAll seven tests pass. Look at what they cover between them. The failing test from lesson 4 now passes with the switch on. With the switch off, a slow answer still gives two texts, which proves that installing the new file changes nothing until the switch is turned. A changed message for the same appointment still goes out. A dry run delivers nothing.
A test checks one reminder. To see a whole morning, save this as try_change.py.
"""Run one pretend morning against the stand-in gateway, with each setting.
Usage: python try_change.py
"""
from fake_gateway import FakeGateway
from sender import send_reminder
# Five reminders. The gateway answers the second one slowly once, and the fourth slowly twice.
REPLY_TIMES = [0.6, 7.1, 0.8, 0.5, 6.3, 8.0, 0.9, 0.7]
def morning(name, settings):
gateway = FakeGateway(reply_times=REPLY_TIMES)
lines = []
for n in range(1, 6):
send_reminder(gateway, f"A{n}", "+440000000000", "Your appointment is tomorrow.",
log=lines.append, settings=settings)
timeouts = sum(1 for line in lines if line.startswith("timeout"))
would = sum(1 for line in lines if line.startswith("would send"))
print(f"{name:<19} reminders 5, timeouts {timeouts}, would send {would}, texts delivered {len(gateway.delivered)}")
morning("switch off", {})
morning("switch on", {"use_message_id": True})
morning("switch on, dry run", {"use_message_id": True, "dry_run": True})python try_change.pyswitch off reminders 5, timeouts 3, would send 0, texts delivered 8
switch on reminders 5, timeouts 3, would send 0, texts delivered 5
switch on, dry run reminders 5, timeouts 0, would send 5, texts delivered 0The same five reminders and the same three late answers give eight texts with the switch off and five with it on. The timeouts are still there in both. The change does not make the gateway faster. It makes a late answer harmless.
The plan the customer approves
Nothing is installed on the customer's server on your say-so. Write the change down in a form the IT lead can approve or refuse in two minutes.
Change request: send a message id with each reminder
For approval by: IT lead, Quillfield Dental
What changes The reminder service sends the same id with every
attempt for one reminder. The gateway does not deliver
an id it has already seen (confirmed by the vendor).
What does not Batch time, wording of texts, wait of 5 seconds,
number of attempts.
Tested 7 tests pass. In a trial morning: 8 texts with the
switch off, 5 with it on, for 5 reminders.
Step 1 Wed 13 May after 18:00: install the new file with the
switch off. Behaviour unchanged.
Step 2 Same evening: turn the switch on in settings.json.
Watch Thu 14 May 08:00 batch. I will count repeated texts
from that morning's logs and report by 10:00.
Way back Set use_message_id to false. Takes effect at the next
reminder. No reinstall, no restart.
Not fixed The gateway is still slow between 08:05 and 08:20.
Timeouts will still appear in the log.Look at the shape of it. It is made of two steps, each small and each undoable, at a time when no batch is running. It says what to watch and when you will report. It gives a way back that the IT lead could carry out without you. And it says plainly what the change will not fix, so that nobody is surprised by timeouts in Thursday's log.
The IT lead replies "approved" by email on Wednesday afternoon, and you add the time to your note. The next lesson is Thursday morning.
Key takeaways
- List every possible fix with its owner. You may only make the ones on your own side.
- Prefer the change that removes the cause over the one that makes the symptom rarer.
- Put the change behind a switch that is off by default, so installing it and turning it on are two separate, small steps.
- The way back should be one setting, with no reinstall.
- Confirm the other system's behaviour with its owner before your change depends on it.
- The customer approves the change in writing before it is installed.
Try it
Repeat the exercise from lesson 4 with the new code and the switch on. Make FakeGateway(reply_times=[6.0, 6.0, 6.0]), call send_reminder with settings={"use_message_id": True}, then print what the call returned and len(gateway.delivered). A good result prints False and 1. The patient now gets one text, but the service still reports a failure for a reminder that was delivered. Write one sentence saying why that still matters; lesson 6 comes back to it.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.