Lesson 6 of 6
Check It Worked, Then Hand It Back
Measure the change with the same tools that found the fault, separate what is fixed from what is not, write the incident note, pass each remaining item to its owner and give back your access.
9 minForward Deployed Engineering
After this lesson you will be able to prove that a change worked, say honestly what it did not fix, and leave a customer's system in a state they can run without you.
The change was installed on Wednesday evening with the switch off, and the switch was turned on an hour later. It is now Thursday 14 May, and the 08:00 batch has run. The IT lead sends the two logs for the morning, as agreed.
Measure it the same way
Your stand-in for the export writes Thursday's logs when it is run with the word after.
python make_logs.py afterwrote reminder_after.log: 742 lines
wrote gateway_after.log: 316 linesOn Tuesday the fault was measured with duplicates.py. Use the same script on Thursday's logs, so the two numbers mean the same thing.
python duplicates.py gateway_after.log reminder_after.loggateway_after.log: 265 reminders accepted
accepted 1 time: 265
Texts sent that should not have been: 0
Repeats the gateway ignored: 46
The service logged "gave up" for 3 reminders.
A2092: accepted by the gateway 1 time
A2109: accepted by the gateway 1 time
A2145: accepted by the gateway 1 timeEvery reminder was accepted once. No text was sent that should not have been, against 49 on Tuesday. The gateway recognised 46 repeats and did not deliver them. Those 46 are the texts that patients would otherwise have received a second or third time.
Look at one of them, to be sure the mechanism is the one you intended and not a quiet morning.
python trace.py A2049 reminder_after.log gateway_after.logA2049, times in UTC
07:05:13.842 service send practice=westbrook attempt=1 message_id=A2049-c757d74e
07:05:13.892 gateway accepted took_ms=7895 message_id=A2049-c757d74e
07:05:18.842 service timeout attempt=1 after=5.0s
07:05:18.852 service send practice=westbrook attempt=2 message_id=A2049-c757d74e
07:05:18.902 gateway duplicate_ignored took_ms=983 message_id=A2049-c757d74e
07:05:19.835 service sent attempt=2 took=0.98s
The gateway accepted this reminder 1 time.The gateway took nearly eight seconds to answer the first attempt. The service timed out and sent again, with the same id. The gateway answered duplicate_ignored in under a second. One text reached the patient. This is the same sequence as A2050 on Tuesday, with a different ending.
What is fixed and what is not
Now summarise the service's log for Thursday, as you did in lesson 2.
python summarise.py reminder_after.logreminder_after.log: 742 lines, from 00:00 to 23:30
By level: INFO 637, WARN 54, ERROR 51
level what count first last hours
WARN timeout 49 08:05 08:19 1
ERROR metrics push failed: connection refused 48 00:00 23:30 24
WARN rejected 5 08:11 14:26 2
ERROR gave up 3 08:10 08:15 1Read this beside Tuesday's summary. The 49 timeouts are still there, in the same quarter of an hour. The metrics error is unchanged. And three reminders are still logged as gave up.
This is what the change request said would happen, and it is worth being exact about it when you report.
- Fixed: patients no longer receive repeated texts. Measured: 0 extra texts on Thursday, 49 on Tuesday.
- Not fixed: the gateway is still slow between 08:05 and 08:20. The change made the slowness harmless to patients. It did not remove it.
- Not fixed, and ours: the three
gave upreminders were each delivered once, as the output ofduplicates.pyshows. The service still records a failure for a patient who was reminded.
Reporting only the first line would be true and would mislead. A customer who later finds timeouts in the log, after being told the problem was fixed, will trust the next report less.
Give each open item to its owner
Each thing still open belongs to somebody. Name the owner and give them what they need to act, so that nothing is left as a general worry.
Open item Owner What they need
------------------------------- --------------- -----------------------
Gateway slow 08:05-08:20 vendor, through times and counts from
Quillfield the gateway's own log
Whether to keep the 08:00 batch Quillfield the fact that 07:00 was
outside the slow period
"gave up" logged for delivered our company a ticket: on a final
reminders timeout, record "not
confirmed", not failed
Metrics push refused all day Quillfield IT the log line and its
times
Rejected numbers (3 Tue, 5 Thu) practice the appointment
managers referencesTwo points about this table. The slowness is raised with the vendor by Quillfield, because the contract is theirs, and you supply the evidence in a form they can forward. The exercise prepares it. And the item that belongs to your company goes into your own team's list of work with the same seriousness as the customer's items. It is tempting to leave it out because nobody has complained about it.
The incident note
Your running note from lesson 1 already holds everything. The incident note is the same material, turned round so that the conclusion comes first. One page is enough.
Incident: repeated reminder texts, Quillfield Dental
Reported Tue 12 May. Change made Wed 13 May. Confirmed Thu 14 May.
What happened
Since the batch moved to 08:00 on 27 April, some reminders were
delivered two or three times. On 12 May: 39 of 267 reminders,
49 extra texts.
Cause
Between 08:05 and 08:20 the gateway accepts a message but can take
more than 5 seconds to answer. The reminder service waited 5
seconds, then sent again. Each attempt was delivered.
What was changed
The service now sends the same id with every attempt for one
reminder, and the gateway does not deliver an id twice. Approved
by the IT lead on 13 May.
Result
14 May: 0 extra texts. 46 repeats recognised and not delivered.
How to undo
Set use_message_id to false in settings.json.
Still open
See the table of open items, each with an owner.
Evidence
Logs for 12 and 14 May, and the scripts used to read them, in the
case folder.A reader who stops after the first three lines knows what happened. A reader who needs to undo the change at night can find how in ten seconds. Nothing in it depends on having been there.
Give the system back
The investigation used access and data that were lent to you. Return them on the day you said you would, without being asked.
- Ask the IT lead to remove your read access on Friday 15 May, the date in your request, and confirm that it is gone.
- Delete the copies of the logs from the case folder once the incident note is accepted, and say when you have done it. They hold patients' appointment references and masked numbers.
- Leave the scripts and the note with the customer.
duplicates.pyis now their check: they can run it on any day's logs without you. - Tell the people who use the service what changed, in one paragraph, including the name of the switch.
What you practised
Look back over the three days. You began by finding out who owned what, and asked for two logs and nothing else. You summarised before reading, and ruled out the loudest error with a reason. You followed one message across two systems and two clocks, and then counted all of them. You reproduced the fault where it could reach nobody. You chose the smallest change on your own side, put it behind a switch, and had it approved. And you measured the result with the tool that had measured the fault, and said what was still open.
None of the code was difficult. The skill an FDE brings to a fault in somebody else's system is restraint: looking before changing, taking only the access the work needs, being exact about what is known, and leaving every part with its owner.
Key takeaways
- Measure the fix with the same tool that measured the fault, so the two numbers can be compared.
- A fix can work while the underlying problem remains. Report both.
- Each remaining item goes to its owner by name, with the evidence they need to act.
- An incident note says what happened, what was changed, how to undo it and what is still open.
- Give back access and delete copies of data on the date you promised, and say that you have done it.
Try it
Gather the evidence for the vendor. Using read_gateway from logs.py, count the answers in gateway_after.log that took more than 5000 milliseconds, and find the first time, the last time and the slowest answer. A good result finds 49 slow answers, between 07:05:13 and 07:19:52 UTC, the slowest 8461 milliseconds. Then write the three sentences Quillfield could send to the vendor: what was seen, on which day and between which times, and what you are asking for.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.