Paginate on the server's continuation signal, never a short-page heuristic

byob-http-client.7 http

Problem: the intuitive pagination loop — "request N rows; if fewer than N come back, that was the last page" — silently drops data. Real servers clamp every response to their own page-size cap regardless of the size you requested, so a "short" page routinely has more pages behind it. (Observed in the wild: a server with a 2000-row cap answering 2000-row pages to a client requesting 5000 — the heuristic stopped after page one and dropped 3780 of 5780 rows, with no error anywhere.) Two more traps hide in the same loop:

Offset drift. If the client filters rows after parsing (dropping null geometries, malformed records), advancing the offset by the filtered count stalls the cursor and re-fetches or skips rows. The offset must advance by the raw row count the server returned.
Unstable order. Offset pagination is only coherent if the row order is pinned. Servers that don't guarantee order across requests can reorder between pages, silently duplicating some rows and skipping others. Always send an explicit sort (ORDER BY id / orderByFields=OBJECTID / equivalent).

Idea: loop on the server's explicit continuation signal — a boolean flag (exceededTransferLimit), a cursor token, a Link: rel="next" header — as the authoritative "more rows remain". Layer the guards in this order:

Continuation signal present → keep going, whatever the page size.
Short-page check only as a fallback for servers that omit the signal.
Empty page → stop unconditionally (guards against a server that sets the flag but returns no rows — infinite loop otherwise).
Hard page cap as a runaway backstop, failing loudly with the count fetched so far — never returning a silently truncated result as success.

When the API offers a total-count query, a cheap cross-check (fetched == reported total) turns any residual pagination bug from silent data loss into a visible error. For a source with no continuation signal at all, note that fallback (2) is exactly the heuristic the problem statement opens with — against such a server the cross-check isn't optional hygiene, it's the only defense left; treat it as required there.

Tradeoffs: parsing the continuation signal is per-API work — some put the flag at the top level, some nest it, some only give a next link. That's ten lines per source against a bug class whose only symptom is numbers that are quietly too small.

Design

const maxRecords = 5000
const maxPages = 200 // runaway backstop, not a tuning knob

var all []Row
offset := 0
for page := 0; ; page++ {
    if page >= maxPages {
        return nil, fmt.Errorf("exceeded %d pages (%d rows fetched), aborting", maxPages, len(all))
    }
    rows, rawCount, more, err := fetchPage(ctx, client, endpoint, offset)
    if err != nil { return nil, err }
    all = append(all, rows...) // rows = post-filter; rawCount = as-returned

    // Continuation signal is authoritative; short-page is only the
    // fallback for servers that omit it; empty page always stops.
    if rawCount == 0 || (!more && rawCount < maxRecords) {
        break
    }
    offset += rawCount // raw count, NOT len(rows) — filtered rows still advance the cursor
}

fetchPage sends the page-size and offset params plus an explicit sort field, and extracts the continuation flag defensively (some deployments emit it at the top level, others nested):

func exceededLimit(body []byte) bool {
    var env struct {
        ExceededTransferLimit bool `json:"exceededTransferLimit"`
        Properties struct {
            ExceededTransferLimit bool `json:"exceededTransferLimit"`
        } `json:"properties"`
    }
    if err := json.Unmarshal(body, &env); err != nil {
        return false // malformed bodies error out in the row parser instead
    }
    return env.ExceededTransferLimit || env.Properties.ExceededTransferLimit
}