lesson 151 HTTP & API Clients · advanced

HTTP requests: build a request, read the response

A program asks a web API for data by sending an HTTP request: a method, a URL with a query string, and headers. The answer has a status code, headers and a body of bytes. urllib.request, part of Python, does both halves.

Simulated: this runs against results recorded on real Python.

You'll be able to

Name the parts of a request (method, URL, query string, headers) and of a response (status, headers, body)
Build a query string with urlencode and a Request with headers
Open a request with a timeout and read status, headers and the body as bytes

The idea

Web APIs answer questions over HTTP. The USGS publishes every earthquake it records; to ask for yesterday's large ones, a program sends a request: the method GET ("give me"), a URL whose query string after ? carries the parameters (starttime=2025-01-01&minmagnitude=4.5), and headers such as Accept: application/json (the format it wants) and User-Agent (who is asking).

The server sends back a response: a status code (200 OK; 4xx means the request was wrong, for example 404 Not Found or 400 Bad Request; 5xx means the server failed), headers such as Content-Type, and a body of bytes.

Python's urllib.request module builds and sends requests. (Many projects install requests or httpx, which wrap the same ideas in a shorter API.) Parameters must be encoded: a space, a comma or an & inside a value would otherwise break the URL. urlencode does that.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Build a request

from urllib.parse import urlencode
from urllib.request import Request

params = {"format": "geojson", "starttime": "2025-01-01", "endtime": "2025-01-02", "minmagnitude": 4.5}
url = "https://earthquake.usgs.gov/fdsnws/event/1/query?" + urlencode(params)
request = Request(url, headers={"Accept": "application/json", "User-Agent": "quake-report/1.0 (ana@example.com)"})

print(request.get_method(), request.host)
print(request.selector)
print(request.get_header("User-agent"))
print(urlencode({"search": "Rabaul, Papua New Guinea", "page": 2}))

Sending it is one call: urllib.request.urlopen(request, timeout=10), or opener.open(request, timeout=10) on an opener made by build_opener(). Always pass timeout= (seconds): without it, a server that never answers makes your program wait forever.

About this page: Python running in a browser tab has no network connections of its own. So in the HTTP & API Clients module the opener is built with one extra piece, a small urllib handler called Replay, which answers each request from saved copies of real responses (from the USGS and OpenAlex APIs, trimmed to a few fields) instead of from the server. urllib lets you plug in handlers exactly like this; testing tools use the same trick. Everything else is the real urllib: the Request, open, the status, the headers and the bytes. On your computer, delete the handler and call build_opener() (or urlopen), and the same code talks to the live API.

example-2.py · Send it and read the response

import io
import json
from email.message import Message
from http import HTTPStatus
from urllib.request import BaseHandler, Request, build_opener
from urllib.response import addinfourl

class Replay(BaseHandler):
    """Answers https requests from saved copies of real API responses."""
    handler_order = 100

    def __init__(self, saved):
        self.saved = saved

    def https_open(self, request):
        status, body = self.saved.get(request.full_url, (404, {"error": "Not Found"}))
        headers = Message()
        headers["Content-Type"] = "application/json"
        response = addinfourl(io.BytesIO(json.dumps(body).encode()), headers, request.full_url, status)
        response.msg = HTTPStatus(status).phrase
        return response

url = "https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime=2025-01-01&endtime=2025-01-02&minmagnitude=4.5"
saved = {url: (200, {"type": "FeatureCollection", "metadata": {"count": 1}, "features": [
    {"id": "us6000pgsd", "properties": {"mag": 5.5, "place": "153 km S of Isangel, Vanuatu"}},
]})}
opener = build_opener(Replay(saved))

request = Request(url, headers={"Accept": "application/json"})
with opener.open(request, timeout=10) as response:
    print(response.status, response.headers["Content-Type"])
    body = response.read()
print(type(body).__name__, len(body))
print(body[:30])

When to use it: any time a program needs data another service owns: weather, prices, publications. Mistakes to avoid: gluing values into the URL with + or f-strings (a space or & in a value corrupts the query: use urlencode); leaving out timeout=; and forgetting that the body is bytes. Many APIs also ask you to identify yourself in User-Agent (OpenAlex asks for an email in a mailto parameter), so they can contact you instead of blocking you.

Review

A request is a method, a URL with an encoded query string, and headers; a response is a status, headers and a body of bytes.
Build query strings with urlencode, and always pass timeout= when you open a request.
In this course the opener answers from saved copies of real responses; on a computer build_opener() talks to the server.
Next: turning a JSON body into Python data, and handling the requests that fail.

Practise this

Build API URLs safely