Skip to content

Lesson 2 of 6

Read the Logs: Find the Pattern Before the Cause

Get one day's logs, summarise them before reading a single line, set aside the loudest error with a reason, and write down the pattern the real problem follows.

8 minForward Deployed Engineering

After this lesson you will be able to take a log you have never seen, summarise it in one screen, and say what pattern the fault follows before you know its cause.

The IT lead at Quillfield Dental has exported two logs for Tuesday 12 May: the reminder service's own log, and the text gateway's delivery log. This lesson uses the first. The second waits for lesson 3.

Get the logs

You have no customer to send you an export, so a script stands in for one. Make a new folder for the course and save this as make_logs.py. You do not need to study it. Treat what it writes the way you would treat files a customer sent you.

"""Stand-in for the customer's log export. Writes one day's logs from two systems.

Usage: python make_logs.py          the day of the complaint
       python make_logs.py after    a day after the change made in lesson 5
"""

import hashlib
import json
import random
import sys
from datetime import datetime, timedelta

AFTER = sys.argv[1:] == ["after"]
DAY = datetime(2026, 5, 14 if AFTER else 12)  # midnight, UK local time
rng = random.Random(22 if AFTER else 13)
PRACTICES = ["hale", "marsh-lane", "northgate", "riverside", "stonebridge", "westbrook"]
TIMEOUT = 5.0
service, gateway = [], []


def pick(low, high):
    return low + (high - low) * rng.random()


def say(seconds, level, text):
    stamp = (DAY + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
    service.append((seconds, f"{stamp} {level:<5} {text}"))


def gate(seconds, **fields):
    stamp = (DAY + timedelta(seconds=seconds - 3600)).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
    gateway.append((seconds, json.dumps({"ts": stamp, **fields})))


def send(appt, practice, start):
    number = f"+44*******{int(pick(100, 1000))}"
    message_id = f"{appt}-{hashlib.sha256(appt.encode()).hexdigest()[:8]}" if AFTER else None
    if rng.random() < 0.015:
        say(start, "INFO", f"send appt={appt} practice={practice} attempt=1")
        gate(start + 0.05, event="rejected", ref=appt, to=number, message_id=message_id,
             reason="invalid_number", took_ms=180)
        say(start + 0.23, "WARN", f"rejected appt={appt} reason=invalid_number, not retried")
        return
    now, note = start, f" message_id={message_id}" if AFTER else ""
    for attempt in (1, 2, 3):
        busy = 8 * 3600 + 300 <= now <= 8 * 3600 + 1200  # 08:05 to 08:20
        took = pick(2.5, 8.5) if busy and rng.random() < 0.5 else pick(0.3, 1.6)
        say(now, "INFO", f"send appt={appt} practice={practice} attempt={attempt}{note}")
        event = "duplicate_ignored" if AFTER and attempt > 1 else "accepted"
        gate(now + 0.05, event=event, ref=appt, to=number, message_id=message_id,
             took_ms=int(took * 1000))
        if took <= TIMEOUT:
            say(now + took, "INFO", f"sent appt={appt} attempt={attempt} took={took:.2f}s")
            return
        say(now + TIMEOUT, "WARN", f"timeout appt={appt} after={TIMEOUT}s attempt={attempt}")
        now += TIMEOUT + 0.01
    say(now, "ERROR", f"gave up appt={appt} attempts=3")


say(6 * 3600 + 2.5, "INFO", "config loaded timeout=5.0s attempts=3 batch_at=08:00")
for minute in range(0, 24 * 60, 30):
    say(minute * 60 + pick(10, 20), "ERROR", "metrics push failed: connection refused host=metrics.internal")
for minute in range(6 * 60 + 5, 20 * 60, 15):
    say(minute * 60 + pick(1, 5), "INFO", f"sync ok appointments={int(pick(900, 990))}")
say(8 * 3600, "INFO", "batch start reminders=240")
for n in range(240):
    send(f"A{2001 + n}", PRACTICES[int(pick(0, 6))], 8 * 3600 + 1 + n * 6.5 + pick(0, 3))
say(8 * 3600 + 1640, "INFO", "batch end")
for n in range(30):  # reminders for appointments changed during the day
    send(f"A{2301 + n}", PRACTICES[int(pick(0, 6))], pick(9 * 3600, 17 * 3600))

suffix = "_after" if AFTER else ""
for name, lines in ((f"reminder{suffix}.log", service), (f"gateway{suffix}.log", gateway)):
    lines.sort()
    with open(name, "w", encoding="utf-8", newline="\n") as handle:
        handle.writelines(text + "\n" for _, text in lines)
    print(f"wrote {name}: {len(lines)} lines")

Run it.

python make_logs.py
wrote reminder.log: 749 lines
wrote gateway.log: 319 lines

Open reminder.log in an editor and look at how a line is built before you look at what it says. These four lines are from eight o'clock, when the morning batch begins.

2026-05-12 08:00:00.000 INFO  batch start reminders=240
2026-05-12 08:00:02.605 INFO  send appt=A2001 practice=hale attempt=1
2026-05-12 08:00:03.324 INFO  sent appt=A2001 attempt=1 took=0.72s
2026-05-12 08:00:09.136 INFO  send appt=A2002 practice=westbrook attempt=1

Each line has a date, a time, a level and a message. The message starts with a few words saying what happened, followed by details written as name=value. One thing is missing: nothing says which time zone the clock is in. Write that in your note as a question. It matters in lesson 3.

Summarise before you read

The log has 749 lines. Reading from the top is slow, and it lets the first alarming line you meet decide what you believe. Count first. Save this as summarise.py.

"""Summarise the reminder service's log before reading it line by line.

Usage: python summarise.py [reminder.log]
"""

import sys
from collections import Counter, defaultdict


def parse(line):
    day, clock, level, text = line.rstrip("\n").split(None, 3)
    words = text.split()
    kind = []
    for word in words:
        if "=" in word:
            break
        kind.append(word)
    return {"clock": clock, "level": level, "kind": " ".join(kind), "text": text}


def main(path):
    with open(path, encoding="utf-8") as handle:
        rows = [parse(line) for line in handle if line.strip()]
    print(f"{path}: {len(rows)} lines, from {rows[0]['clock'][:5]} to {rows[-1]['clock'][:5]}")
    levels = Counter(row["level"] for row in rows)
    print("By level:", ", ".join(f"{level} {levels[level]}" for level in ("INFO", "WARN", "ERROR")))
    groups = defaultdict(list)
    for row in rows:
        if row["level"] != "INFO":
            groups[(row["level"], row["kind"])].append(row["clock"])
    print()
    print(f"{'level':<6}{'what':<42}{'count':>5}  {'first':<6} {'last':<6} hours")
    for (level, kind), clocks in sorted(groups.items(), key=lambda item: -len(item[1])):
        hours = len({clock[:2] for clock in clocks})
        print(f"{level:<6}{kind:<42}{len(clocks):>5}  {clocks[0][:5]:<6} {clocks[-1][:5]:<6} {hours}")


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "reminder.log")

The script takes the words before the first name=value as the kind of line, so timeout appt=A2050 ... and timeout appt=A2054 ... are counted together. For every kind of warning and error it prints how many there were, the first and last time, and how many different hours of the day they fell in.

python summarise.py
reminder.log: 749 lines, from 00:00 to 23:30
By level: INFO 641, WARN 56, ERROR 52

level what                                      count  first  last   hours
WARN  timeout                                      53  08:05  08:19  1
ERROR metrics push failed: connection refused      48  00:00  23:30  24
ERROR gave up                                       4  08:06  08:16  1
WARN  rejected                                      3  08:00  08:02  1

Do not chase the loudest error

There are 52 errors, and 48 of them are the same line: the service failed to push its metrics. It is tempting to start there. It is the commonest error by far, and "connection refused" sounds serious.

Check it against the symptom before you spend an hour on it. The symptom is confined to the morning, at about quarter past eight. The metrics error runs from midnight to half past eleven at night and appears in all 24 hours, twice an hour, including the hours when no reminders are sent at all. Something that happens evenly all day cannot explain something that happens in one quarter of an hour.

So set it aside, and do it properly.

  • Write in your note that you saw it, and why you ruled it out.
  • Tell the IT lead. The metrics host refuses connections, and whoever owns that host should know.
  • Do not fix it now. An unrelated change in the middle of an investigation makes every later observation harder to read.

The pattern

Now look at what does fit the symptom. All 53 timeouts fall between 08:05 and 08:19. They are warnings, not errors, which is why a search for errors alone would have missed them. The four gave up errors fall inside the same window. The three rejected lines come just after eight; the full lines in the log give the reason as an invalid number and say they were not retried, so they are a different matter.

You can now state a pattern without knowing its cause.

Pattern, from reminder.log for Tue 12 May
- 53 timeouts, all between 08:05 and 08:19. None at any other time,
  although reminders are also sent through the day.
- 4 reminders given up on after three attempts, in the same window.
- The batch starts at 08:00, so its first five minutes are clean.

Ruled out
- "metrics push failed" (48 a day, evenly, all 24 hours): does not
  follow the pattern. Reported to IT lead.

Still not known
- what happens to a reminder after a timeout
- whether a timeout produces a repeated text
- why 08:05 to 08:19

This is the discipline of the whole lesson: pattern first, cause second. A cause proposed before the pattern is known is a guess, and guesses in somebody else's system lead to changes that should never have been made.

Ask what changed

The complaint said the trouble began "about two weeks ago". A system that worked and then stopped has usually had something changed, and the customer often knows what it was without knowing that it matters. Ask plainly: "Did anything change around the end of April? Settings, timings, suppliers, anything."

The IT lead looks and replies. On Monday 27 April, at the request of the practices, the morning batch was moved from 07:00 to 08:00, because patients did not want a text at seven. The log agrees with the new time. Its line from six in the morning, when the service loads its settings, ends with batch_at=08:00.

This is a lead and not yet a cause. It fits: the timeouts sit inside the hour the batch moved into. It explains nothing yet about repeated texts, and a batch at 08:00 is a perfectly reasonable thing for a customer to want. Write it down as it is.

Wed 13 May
09:15  Both logs received from IT lead.
09:40  reminder.log summarised. 53 timeouts 08:05-08:19, 4 "gave up".
09:55  Metrics error ruled out (all day, all hours). Told IT lead.
10:20  IT lead: batch moved from 07:00 to 08:00 on Mon 27 April.
       Lead only. Need the gateway's side to see what a timeout does.

The reminder service's log tells you what the service believed. To find out what really reached patients, you need the other side of the conversation. That is the next lesson.

Key takeaways

  • Summarise a log before you read it: how many lines, of what kinds, and at what times.
  • The loudest error is not always the cause. Check whether it follows the same pattern as the symptom.
  • An error you set aside still gets written down and passed to its owner.
  • Look for the pattern first, in time and in place, and only then ask what causes it.
  • Ask the customer what changed. The answer is a lead to check, not yet a proof.

Try it

Find out whether the problem belongs to one practice. Each send line in reminder.log names a practice and an attempt number. Count the lines with attempt=2 for each practice, with a few lines of Python or with the tools in your terminal. A good result finds 39 second attempts spread over all six practices, between 3 and 8 at each, so no single practice is the cause.

Official documentation

Ready to use this in a job?

Rolebound applies to matching roles for you.

Start now

Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.

    Cookies on Rolebound

    We use necessary cookies to sign you in and keep the site working — those cannot be turned off. We would also like to set analytics and advertising cookies so we can see which pages help people get interviews and which adverts brought them here. Nothing optional is set unless you say yes, and you can change your mind at any time from “Cookie settings” in the footer. Read our Privacy Policy.