Lesson 4 of 6
Reproduce It Safely, Away from the Live System
Read the code you own, build a small stand-in for the system you do not own, and capture the fault as a failing test that sends nothing to anybody.
7 minForward Deployed Engineering
After this lesson you will be able to reproduce a fault from a customer's live system on your own machine, in a form that reaches no real person and can be run as often as you like.
The logs gave a cause: the gateway sometimes answers late, and the reminder service sends again. So far that is a reading of two files. Before anything is changed, you want to see the fault happen on demand.
Why not just try it on the live system?
On your own system you might send a test reminder and watch. Here that is not available to you, for three reasons.
- Every message through the live service goes to a real patient's phone, and each one is charged to the customer.
- You cannot make the gateway slow when you want to. You would have to wait for quarter past eight and hope.
- The server is not yours. Running experiments on it was not in the access you asked for.
The safe place is your own machine, with the one part you own, the service's sending code, and a stand-in for the part you do not.
Read the code you own
The reminder service is your company's code, so you can read it. This is the part that sends one text, trimmed to what matters here. Save it as sender.py.
"""The part of the reminder service that sends one text. This is the code we own."""
TIMEOUT = 5.0 # seconds to wait for the gateway's answer
ATTEMPTS = 3
class GatewayTimeout(Exception):
pass
class Rejected(Exception):
pass
def send_reminder(gateway, appt, to, text, log=print):
"""Send one reminder. Return True when the gateway confirms it."""
for attempt in range(1, ATTEMPTS + 1):
log(f"send appt={appt} attempt={attempt}")
try:
gateway.send(to=to, text=text, timeout=TIMEOUT)
except GatewayTimeout:
log(f"timeout appt={appt} after={TIMEOUT}s attempt={attempt}")
continue
except Rejected as error:
log(f"rejected appt={appt} reason={error}, not retried")
return False
log(f"sent appt={appt} attempt={attempt}")
return True
log(f"gave up appt={appt} attempts={ATTEMPTS}")
return FalseRead it with the trace from lesson 3 beside you. The log lines match: send, timeout, sent, gave up. Now find the place where the code's belief and the facts part. It is the two lines under except GatewayTimeout. The code logs the timeout and goes round the loop again. In doing so it treats "I did not hear an answer" as "the message was not sent". The gateway's log showed that this is false whenever the gateway is only slow.
The docstring makes a careful promise: the function returns True when the gateway confirms. It says nothing about what has happened when it returns False, and the batch that calls it assumes the worst.
A stand-in for the gateway
You need something that behaves like the gateway in the one way that matters: it takes the message first and answers afterwards, sometimes too late. Save this as fake_gateway.py.
"""A stand-in for the text gateway, for tests. It sends nothing to anybody."""
from sender import GatewayTimeout, Rejected
class FakeGateway:
def __init__(self, reply_times=(), bad_numbers=()):
self.reply_times = list(reply_times) # seconds each answer takes, in order
self.bad_numbers = set(bad_numbers)
self.delivered = [] # texts that would reach a phone
self.seen_ids = set()
def send(self, to, text, timeout, message_id=None):
took = self.reply_times.pop(0) if self.reply_times else 0.5
if to in self.bad_numbers:
raise Rejected("invalid_number")
if message_id is None or message_id not in self.seen_ids:
self.delivered.append((to, text)) # taken for delivery before the answer goes back
if message_id is not None:
self.seen_ids.add(message_id)
if took > timeout:
raise GatewayTimeout()The stand-in is given a list of reply times in seconds. On each call it takes the next one. It records the text as delivered before it decides whether its answer is late, which is the order of events the real logs showed. If the reply time is longer than the caller is willing to wait, it raises the same timeout the service's real gateway client raises. It opens no connection and waits for nothing, so a test takes no time at all.
One part of it is there for the next lesson. The vendor's documentation says a message may carry a message_id, and that a message whose id has been seen before is acknowledged and not sent again. The stand-in copies that rule. The present sender.py never passes an id, so for now the rule does nothing.
Capture the fault as a test
Write down what should be true, as tests, and let the code show you where it is not. Save this as test_sender.py.
import unittest
from fake_gateway import FakeGateway
from sender import send_reminder
NUMBER = "+440000000000"
def quiet(line):
pass
class SendReminderTests(unittest.TestCase):
def test_a_quick_answer_means_one_text(self):
gateway = FakeGateway(reply_times=[0.7])
self.assertTrue(send_reminder(gateway, "A1", NUMBER, "Your appointment is tomorrow.", log=quiet))
self.assertEqual(len(gateway.delivered), 1)
def test_a_slow_answer_still_means_one_text(self):
gateway = FakeGateway(reply_times=[6.8, 0.9])
self.assertTrue(send_reminder(gateway, "A1", NUMBER, "Your appointment is tomorrow.", log=quiet))
self.assertEqual(len(gateway.delivered), 1)
def test_a_bad_number_is_not_tried_again(self):
gateway = FakeGateway(bad_numbers=[NUMBER])
self.assertFalse(send_reminder(gateway, "A1", NUMBER, "Your appointment is tomorrow.", log=quiet))
self.assertEqual(len(gateway.delivered), 0)
if __name__ == "__main__":
unittest.main()The second test is the fault in one screen. The gateway answers the first attempt after 6.8 seconds, which is too late, and the second after 0.9 seconds. One reminder was asked for, so one text should be delivered.
python -m unittestTwo tests pass and one fails. The failing test is test_a_slow_answer_still_means_one_text, and the reason given is AssertionError: 2 != 1: two texts were delivered where one was expected. The run ends with FAILED (failures=1).
This failure is good news. It is the same fault as appointment A2050, produced on your machine in a fraction of a second, with no patient and no bill. It can be run a hundred times. And when a change makes this test pass, you will know the fault is gone without waiting for tomorrow's batch.
What the stand-in proves, and what it does not
Be exact about what you now have, because you will be asked.
- It proves that the service's code, as written, sends a second text whenever the gateway takes the message and answers late. That is a fact about your own code.
- It does not prove anything about the real gateway. The stand-in is your belief about the gateway, built from one day of its log and its documentation.
A belief about somebody else's system should be checked with its owner before you rely on it. Two beliefs matter here. The first, that the gateway delivers a message whose answer came late, is supported by the log: 49 extra accepted lines. The second, that the gateway ignores a repeated message_id, rests so far on the documentation alone, and the next lesson depends on it. Add both to the note, with where each comes from.
Wed 13 May
11:30 Fault reproduced on my machine as a failing test. No live
system touched.
Belief 1: late answer still means delivered. From gateway.log.
Belief 2: repeated message_id is not sent again. From vendor's
documentation only. To confirm with the vendor through IT lead
before any change relies on it.With the fault in a box, you can now try a change and see at once whether it works. Choosing which change to make, in a system where most of the parts are not yours, is the next lesson.
Key takeaways
- Never reproduce a fault on the live system when doing so reaches real users.
- Read the code you own with the logs beside you, and find the line where its belief and the facts part.
- A stand-in only needs the one behaviour that matters, taken from the logs and the owner's documentation.
- A failing test is the best reproduction: it is exact, it is repeatable, and it tells you when the fault is gone.
- A stand-in is your belief about the other system. Say so, and check the belief with the owner.
Try it
The gave up reminders in lesson 3 had three slow answers in a row. Reproduce that case. In a short script, make FakeGateway(reply_times=[6.0, 6.0, 6.0]), call send_reminder with it, then print what the call returned and len(gateway.delivered). Write down what you expect before you run it. A good result prints False and 3: the service reports a failure, and the patient has three texts, the same as appointment A2062 in the real logs.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.