Lesson 3 of 6
Fetch Everything Reliably: Pages, Timeouts and Retries
Write a small client that reads every page of the customer's API, gives up on a request that hangs, and retries the failures that are worth retrying.
5 minForward Deployed Engineering
After this lesson you will be able to write a client that fetches every record from a paged API, does not hang when the other system goes quiet, and retries a failed request only when retrying can help.
Keep the practice server from lesson 2 running in its own terminal, and make sure REPAIRS_TOKEN is set in the terminal you work in.
One page is not the data
In lesson 2 the answer said "page 1 of 3". An API hands out records a page at a time so that no single request is large or slow. The cost is that your code must ask again and again until it has everything.
APIs page in different ways. Some give a page number and a page count, as this one does. Some give a token to send back for the next page. Some give the address of the next page. The supplier's documentation says which, and the loop is the same in each case: ask, keep what you get, and stop when the API says there is no more.
The mistake to avoid is treating the first page as the whole answer. Nothing fails when you do that. The numbers are just wrong, and they stay wrong until somebody at the customer notices a missing repair.
The client
Create client.py. It has one function that makes a single request and one that walks through the pages.
"""Fetch every repair from the API: page by page, with a timeout and patient retries."""
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
BASE = os.environ.get("REPAIRS_API", "http://127.0.0.1:8765")
RETRY_STATUSES = {429, 500, 502, 503, 504}
def get_json(path, params, token, attempts=4):
"""One GET request. A busy or failing server is tried again; any other error is raised."""
url = f"{BASE}{path}?{urllib.parse.urlencode(params)}"
request = urllib.request.Request(url, headers={"Authorization": f"Bearer {token}"})
for attempt in range(1, attempts + 1):
try:
with urllib.request.urlopen(request, timeout=10) as response:
return json.load(response)
except urllib.error.HTTPError as error:
if error.code not in RETRY_STATUSES or attempt == attempts:
raise
wait = float(error.headers.get("Retry-After") or 2**attempt)
print(f" got {error.code} for page {params['page']}, waiting {wait:g}s, then trying again")
time.sleep(wait)
def fetch_all(token, updated_since=""):
"""Every repair changed after `updated_since`, or every repair when it is empty."""
items, page = [], 1
while True:
params = {"page": page}
if updated_since:
params["updated_since"] = updated_since
body = get_json("/repairs", params, token)
items.extend(body["items"])
if page >= body["pages"]:
return items
page += 1
if __name__ == "__main__":
repairs = fetch_all(os.environ["REPAIRS_TOKEN"])
print(f"{len(repairs)} repairs fetched, from {repairs[0]['ref']} to {repairs[-1]['ref']}")fetch_all is the paging loop. It also accepts a time and passes it on as updated_since, which asks the API for only the repairs changed after that time. Lesson 5 uses it.
get_json makes one request. It is where the timeout and the retries live, so that every request in the program gets both.
Run it
python client.pyIt prints this:
got 429 for page 2, waiting 1s, then trying again
10 repairs fetched, from R-2201 to R-2210The practice server refuses the first request for page 2 on purpose, with status 429, so that you can watch the retry. The client waited the one second that the server asked for, tried again and carried on. All ten repairs arrived.
Give every request a timeout
The timeout=10 in get_json means: if nothing has come back after ten seconds, stop waiting and raise an error. Without it, a request to a system that has gone quiet can wait for ever. A sync that runs every fifteen minutes would then stop, with no error and no message, and you would learn about it from the customer.
Choose the timeout from what is normal for that system. If a page usually takes one second, ten seconds is generous. If the customer's reports take a minute, ten seconds would fail every time.
Retry only what waiting can fix
Lesson 2 listed the failure statuses. They fall into two groups.
Waiting can fix a 429 and the statuses from 500 to 504: the system is busy or has a fault, and may be fine in a moment. Those are the ones in RETRY_STATUSES.
Waiting cannot fix a 401, a 403 or a 404. The token is wrong, the permission is missing or the address is wrong, and it will be just as wrong in ten seconds. Retrying only delays the error and fills the customer's logs with refused requests. For these the client raises the error at once.
Three more rules are in the code. If the server sends a Retry-After header, wait that long, because the server knows best when it will be ready. Otherwise wait longer after each failure: 2 seconds, then 4, then 8. And stop after a fixed number of attempts and raise the error, because a program that retries for ever hides the fault from everybody.
One caution for later in your work: this client only reads, and repeating a read is harmless. Repeating a request that creates or changes something is not harmless, because the first attempt may have succeeded before the answer was lost. Before you retry a write, find out whether the API can recognise a repeated request.
Be a good guest
The customer's system has its own users, and they come first. Fernhill's IT lead asked for at most one request a second, and that limit is on the integration sheet.
The client keeps to it in a simple way: it asks for one page, waits for the answer and only then asks for the next. Do not fetch pages in parallel to save time unless the owner has agreed to it. A sync that finishes in three seconds and not in ten is worth nothing if it slows the contact centre's screens.
If you are ever told to slow down, by a 429 or by a person, slow down first and discuss it afterwards.
Key takeaways
- One request rarely returns everything; follow the pages until the API says there are no more.
- Every request needs a timeout, or one silent failure stops the whole program for good.
- Retry only what waiting can fix: a busy server or a server fault, never a refused token.
- Wait as long as the server asks, limit the number of attempts, and then fail loudly.
- You are a guest on the customer's system: stay inside the agreed limits and fetch one page at a time.
Try it
Stop the practice server with Ctrl+C and run python client.py. It fails at once with a "connection refused" error, which is what happens when the customer's system is restarting. Change get_json so that a failed connection is also tried again, with the same limit on attempts and a wait of 2**attempt seconds, and prints a line each time it waits. Take care with the order of your except clauses: HTTPError is a kind of URLError. A good result prints three waiting lines over about fourteen seconds with the server stopped and then fails with the original error, and still fetches all ten repairs once the server is running again.
Official documentation
Written with AI and checked automatically. Nobody has run these steps by hand, and tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.