Skip to content

Lesson 3 of 6

Follow One Message Across Two Systems

Line up two logs that use different formats and different clocks, follow one reminder from the service to the gateway and back, and measure how large the fault is.

9 minForward Deployed Engineering

After this lesson you will be able to follow one message through two systems, put their logs on one clock, and say how often a fault happens.

The reminder service's log showed 53 timeouts on Tuesday 12 May, all between 08:05 and 08:19. It could not show what happened to those messages afterwards. The text gateway's log can. Both files are in your course folder from lesson 2.

Two logs, two formats

Take the first timeout in the service's log. It concerns appointment A2050. These are all the service's lines for it.

2026-05-12 08:05:21.895 INFO  send appt=A2050 practice=westbrook attempt=1
2026-05-12 08:05:26.895 WARN  timeout appt=A2050 after=5.0s attempt=1
2026-05-12 08:05:26.905 INFO  send appt=A2050 practice=westbrook attempt=2
2026-05-12 08:05:28.011 INFO  sent appt=A2050 attempt=2 took=1.11s

The service sent the reminder, waited five seconds, logged a timeout, sent it again and was told it had been sent. From the service's side this looks like a small delay that put itself right.

Now the gateway. Its log has one JSON object on each line, and it calls the appointment reference ref. The obvious first move is to look at the same time of day. Search gateway.log for 08:05 and you find nothing at all. Search for the reference instead.

{"ts": "2026-05-12T07:05:21.945Z", "event": "accepted", "ref": "A2050", "to": "+44*******639", "message_id": null, "took_ms": 5809}
{"ts": "2026-05-12T07:05:26.955Z", "event": "accepted", "ref": "A2050", "to": "+44*******639", "message_id": null, "took_ms": 1105}

There are two lines, both accepted, and both an hour earlier than the service's lines.

Two clocks

The gateway's times end in Z, which marks them as UTC. The service's times carry no marker, the question noted in lesson 2. In May the United Kingdom is one hour ahead of UTC, so a server keeping local time would write 08:05 when the gateway writes 07:05. That is a likely explanation. Check it before relying on it.

  • Compare a message where nothing went wrong. The first reminder of the batch was sent at 08:00:02.605 by the service's clock and accepted at 07:00:02.655 by the gateway's. That is one hour and a twentieth of a second, exactly what a clean one-hour offset would give.
  • Ask the owner. The IT lead confirms that the reminder server keeps UK local time.

Without this step you would have compared 08:05 with 08:05, found a quiet gateway, and concluded that the gateway had nothing to do with it. Mismatched clocks are one of the commonest ways an investigation across two systems goes wrong.

Put the rule into code once, so that nothing after this has to think about it. Save this as logs.py.

"""Read both logs onto one clock.

The reminder service writes UK local time with no marker. In May that is one
hour ahead of UTC. The gateway writes UTC. Everything returned here is UTC.
"""

import json
from datetime import datetime, timedelta

SERVICE_AHEAD_OF_UTC = timedelta(hours=1)


def read_service(path="reminder.log"):
    events = []
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            day, clock, level, text = line.rstrip("\n").split(None, 3)
            words = text.replace(",", "").split()
            values = dict(word.split("=", 1) for word in words if "=" in word)
            kind = []
            for word in words:
                if "=" in word:
                    break
                kind.append(word)
            when = datetime.strptime(f"{day} {clock}", "%Y-%m-%d %H:%M:%S.%f") - SERVICE_AHEAD_OF_UTC
            events.append({"when": when, "source": "service", "level": level,
                           "kind": " ".join(kind), **values})
    return events


def read_gateway(path="gateway.log"):
    events = []
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            row = json.loads(line)
            when = datetime.strptime(row["ts"], "%Y-%m-%dT%H:%M:%S.%fZ")
            events.append({"when": when, "source": "gateway", "kind": row["event"],
                           "appt": row["ref"], "took_ms": row["took_ms"],
                           "message_id": row["message_id"]})
    return events

Both functions return a list of events with the same fields: when it happened in UTC, which system wrote it, what kind of event it was, and the appointment. The one-hour offset lives in one named line. If this investigation ran in winter, that line would change and nothing else would.

Follow one message

Save this as trace.py. It merges the two logs for one appointment and prints the events in the order they happened.

"""Follow one reminder through both logs, in time order, on one clock.

Usage: python trace.py APPOINTMENT [reminder.log gateway.log]
"""

import sys

from logs import read_gateway, read_service

SHOW = ("practice", "attempt", "attempts", "after", "took", "took_ms", "reason", "message_id")


def main(appt, service_path="reminder.log", gateway_path="gateway.log"):
    events = [event for event in read_service(service_path) + read_gateway(gateway_path)
              if event.get("appt") == appt]
    events.sort(key=lambda event: event["when"])
    print(f"{appt}, times in UTC")
    for event in events:
        details = " ".join(f"{key}={event[key]}" for key in SHOW if event.get(key) is not None)
        print(f"{event['when']:%H:%M:%S.%f}"[:-3], f"{event['source']:<8}", f"{event['kind']:<17}", details)
    accepted = sum(1 for event in events if event["kind"] == "accepted")
    print(f"The gateway accepted this reminder {accepted} time{'s' if accepted != 1 else ''}.")


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit(__doc__)
    main(*sys.argv[1:4])
python trace.py A2050
A2050, times in UTC
07:05:21.895 service  send              practice=westbrook attempt=1
07:05:21.945 gateway  accepted          took_ms=5809
07:05:26.895 service  timeout           attempt=1 after=5.0s
07:05:26.905 service  send              practice=westbrook attempt=2
07:05:26.955 gateway  accepted          took_ms=1105
07:05:28.011 service  sent              attempt=2 took=1.11s
The gateway accepted this reminder 2 times.

Read it line by line. At 07:05:21.895 the service sends the reminder. A twentieth of a second later the gateway accepts it. The gateway's documentation, which the IT lead forwarded, says the time on an accepted line is the moment the message was taken for delivery, and took_ms is how long the gateway then took to send its answer back. This answer took 5809 milliseconds.

The service waits five seconds. At 07:05:26.895 it stops waiting and logs a timeout. The message is already on its way to the patient's phone, and the service does not know that. It sends the reminder again. The gateway accepts that as a new message, answers in about a second, and the service logs sent.

One reminder, two texts. That is the fault in the screenshots.

The sentence to remember is this: a timeout tells you that you did not hear the answer. It does not tell you what the answer was.

How big is it?

One traced message is an explanation. The customer also needs a number. Save this as duplicates.py.

"""Count the reminders the gateway accepted more than once.

Usage: python duplicates.py [gateway.log reminder.log]
"""

import sys
from collections import Counter

from logs import read_gateway, read_service


def main(gateway_path="gateway.log", service_path="reminder.log"):
    gateway = read_gateway(gateway_path)
    accepted = Counter(event["appt"] for event in gateway if event["kind"] == "accepted")
    ignored = sum(1 for event in gateway if event["kind"] == "duplicate_ignored")
    repeats = sorted(event["when"] for event in gateway
                     if event["kind"] == "accepted" and accepted[event["appt"]] > 1)
    copies = Counter(accepted.values())
    print(f"{gateway_path}: {len(accepted)} reminders accepted")
    for times in sorted(copies):
        print(f"  accepted {times} time{'s' if times != 1 else ''}: {copies[times]}")
    print("Texts sent that should not have been:", sum(accepted.values()) - len(accepted))
    print("Repeats the gateway ignored:", ignored)
    if repeats:
        print(f"Repeated reminders were accepted between {repeats[0]:%H:%M} and {repeats[-1]:%H:%M} UTC")
    gave_up = [event["appt"] for event in read_service(service_path) if event["kind"] == "gave up"]
    print(f'The service logged "gave up" for {len(gave_up)} reminders.')
    for appt in gave_up:
        print(f"  {appt}: accepted by the gateway {accepted[appt]} time{'s' if accepted[appt] != 1 else ''}")


if __name__ == "__main__":
    main(*sys.argv[1:3])
python duplicates.py
gateway.log: 267 reminders accepted
  accepted 1 time: 228
  accepted 2 times: 29
  accepted 3 times: 10
Texts sent that should not have been: 49
Repeats the gateway ignored: 0
Repeated reminders were accepted between 07:05 and 07:19 UTC
The service logged "gave up" for 4 reminders.
  A2062: accepted by the gateway 3 times
  A2111: accepted by the gateway 3 times
  A2149: accepted by the gateway 3 times
  A2153: accepted by the gateway 3 times

Of 267 reminders, 39 were accepted more than once: 29 twice and 10 three times. That is 49 texts that should not have been sent, close to one extra text for every five reminders, which matches the higher bill. All the repeats fall between 07:05 and 07:19 UTC, the same quarter of an hour as the timeouts.

The end of the output holds a second finding. The service logged gave up for four reminders. Anyone reading the service's log would take those four patients to have had no reminder. The gateway accepted each of them three times. The service's own record of what it did is wrong in the worst direction: its failures were the patients who were texted most.

Say what is known and what is not

You can now replace the symptom from lesson 1 with a statement of cause. Keep the same habit of separating what you have seen from what you have not.

Cause, as far as the logs show (Tue 12 May)
Between 08:05 and 08:20 UK time the gateway takes a message for
delivery at once but sometimes needs more than 5 seconds to answer.
The reminder service waits 5 seconds, treats no answer as "not sent"
and sends again, up to 3 times. Each attempt is delivered.

Size: 39 of 267 reminders repeated, 49 extra texts.
Also: 4 reminders logged as "gave up" were each delivered 3 times.

Known from: both logs for 12 May, lined up on UTC.

Not known
- why the gateway is slow in that quarter of an hour. It is the
  vendor's system. Our batch sends one message every few seconds
  and the first five minutes are fast, so our own load is unlikely.
- whether other days look the same. One day of logs so far.

Notice where the fault sits. The gateway is slow, and that belongs to the vendor. The service repeats a message it cannot be sure has failed, and that belongs to your company. The customer moved the batch into the slow quarter of an hour, which was a reasonable decision made without this knowledge. Three owners each hold a part, and only one part is yours to change. Before changing it, you need to see the fault happen somewhere safe. That is the next lesson.

Key takeaways

  • Follow one message by its identifier, not by the time on the clock.
  • Find out which time zone each log is written in before you compare them, and check it on a message where nothing went wrong.
  • A timeout means you did not hear the answer. It does not mean the other side did nothing.
  • Once one case is understood, count them all, so the customer learns the size of the fault.
  • A log records what a system believed, and that can differ from what happened.

Try it

Run python trace.py A2062. This is the first reminder for which the service logged gave up. Write down how many texts the patient received and how long each of the gateway's answers took. A good result finds that the gateway accepted the reminder three times, and that its answers took 6823, 5661 and 5879 milliseconds, each longer than the five seconds the service waits. A patient the service's log counts as not reminded received three texts.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.

    Follow One Message Across Two Systems | Debug a System You Do Not Own: Logs, Access and Safe Changes | Rolebound