HTTP response disk cache as the outermost RoundTripper layer

byob-http-client.6 httpstorage

Problem: a CLI that ingests from slow, rate-limited public APIs (multi-MB exports, query endpoints that take tens of seconds) re-downloads identical data on every run. Sprinkling app-level caching at call sites duplicates the logic per source and misses new sources by default. HTTP already has the vocabulary for this — freshness windows, ETag/Last-Modified validators, 304s — the client just has to use it.

Idea: a caching RoundTripper at the outermost position of the middleware chain (byob-http-client.1), backed by a directory under the user cache dir (byob-runtime-directories.1 — regenerable, safe to delete):

Cache → UserAgent → Retry → Logging → transport

Outermost so a cache hit skips retry, UA stamping, and wire logging entirely — a hit does no network work at all. Mechanics:

Key: sha256(method + "\n" + url + "\n" + body). Folding the body in matters for read-shaped POSTs (query APIs whose requests are too large for a URL) — same URL, different query, different entry. Buffer and restore req.Body (and reset GetBody / ContentLength) before passing the request inward, since reading consumes it.
Freshness: entries inside the TTL are served from disk. Past the TTL, if the stored response carried ETag/Last-Modified, send a conditional request — a 304 refreshes the timestamp without re-downloading the body, which is the cheap path on repeat runs.
Bypass: TTL = 0 disables reads entirely (no validators sent, no cached body served). A per-request context key (cache.WithBypass(ctx)) gives commands a --force that skips the read path but still writes the fresh response back.
Entry layout: two files — a JSON meta record (status, headers, timestamp) and the raw body. Write the body first and treat the meta write as the commit record: each file is written atomically (see the atomic-rename-samedir memory), but the pair isn't transactional — if meta committed first, a crash in between would pair a fresh timestamp and new validators with the old body and serve it as a hit forever. Body-then-meta means a failure leaves the previous consistent pair intact and the entry simply re-fetches.
Discipline: cache only 200s. Cap the body read with an io.LimitReader (a hostile or broken server shouldn't fill the disk) — and read one byte past the cap so an over-limit body fails loudly instead of being silently truncated into the cache as a valid entry. Cache writes are best-effort — a failed write must never fail the request.

Tradeoffs: this is for slow-changing, unauthenticated upstream data (public geodata, package indexes, published datasets) — never for authenticated APIs, where cached bodies are user-scoped state on shared disk. Disk usage is unbounded without a sweeper; acceptable for a CLI whose cache dir is documented and deletable, but say so. Stale-within-TTL is the deal you signed — pick a TTL (24h is a sane default) that matches how often the upstream actually changes, and give users the bypass knob.

Design

type CachingTransport struct {
    Wrapped http.RoundTripper
    Dir     string
    TTL     time.Duration
}

func (t *CachingTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    // Only read-shaped methods. POST is included because query APIs
    // send reads as POST when the query outgrows a URL.
    if req.Method != http.MethodGet && req.Method != http.MethodPost {
        return t.Wrapped.RoundTrip(req)
    }

    bodyBytes, err := readAndRestoreBody(req) // buffer + reset Body/GetBody/ContentLength
    if err != nil { return nil, err }

    key := cacheKey(req.Method, req.URL.String(), bodyBytes)
    meta, cachedBody, haveCache := t.readCache(key) // skipped when TTL == 0
    if bypassRequested(req.Context()) {
        haveCache = false // --force: ignore entry; fresh response still written back
    }
    if haveCache && time.Since(meta.Timestamp) < t.TTL {
        return buildResponse(req, meta, cachedBody), nil
    }
    if haveCache { // past TTL: try a cheap 304 revalidation
        req = withConditionalValidators(req, meta) // clones; RoundTrip must not mutate
    }

    resp, err := t.Wrapped.RoundTrip(req)
    if err != nil { return nil, err }

    if resp.StatusCode == http.StatusNotModified && haveCache {
        _ = resp.Body.Close()
        meta.Timestamp = time.Now()
        t.writeMeta(key, meta) // bump timestamp; body untouched
        return buildResponse(req, meta, cachedBody), nil
    }

    body, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes))
    _ = resp.Body.Close()
    if err != nil { return nil, err }
    if resp.StatusCode == http.StatusOK {
        t.writeCache(key, resp, body) // body first, then meta; both best-effort
    }
    resp.Body = io.NopCloser(bytes.NewReader(body))
    return resp, nil
}

func cacheKey(method, url string, body []byte) string {
    h := sha256.New()
    h.Write([]byte(method)); h.Write([]byte{'\n'})
    h.Write([]byte(url));    h.Write([]byte{'\n'})
    h.Write(body)
    return hex.EncodeToString(h.Sum(nil))
}

The Factory wires it in the lazy HTTPClient closure (byob-http-client.2), with the cache dir resolved through the Factory's Paths (byob-runtime-directories.1/.2) and the TTL as a constructor argument so tests can pass 0.