Append-only JSONL run history when the workload is an event log

byob-storage.8 statestorage

Problem: a CLI that orchestrates long runs (a build loop, a sync daemon, an agent driver) accumulates run/event history that is single-writer, append-heavy, replayed in order, and rarely queried. Reaching for the sqlite Store (byob-storage.1) buys migrations (byob-storage.3) and a contract suite (byob-storage.6) that this workload never earns — and a crash mid-iteration must not corrupt what's already recorded.

Idea: a per-run directory under the state dir, holding two kinds of file with two write disciplines:

manifest.json — the run's summary (start/end, outcome, counters), written atomically via temp+fsync+rename (byob-runtime-directories.3) and carrying a schema version with refuse-newer semantics (byob-runtime-directories.4).
*.jsonl event streams — one JSON object per line, appended through a shared mutex-guarded O_APPEND writer so concurrent goroutines in the one process never tear a line.

Run IDs are RFC3339-flat UTC timestamps, so the filesystem is the index: listing is ReadDir + lexicographic sort, "latest" is the last entry, and there is no registry file to drift out of sync.

Tradeoffs:

No queries — readers scan. That's fine for "show this run" / "tail the log"; the moment you need cross-run aggregation, upserts, or filtering, graduate to the byob-storage.1 sqlite Store.
Single-writer by design. The in-process mutex serializes goroutines; concurrent processes are excluded by the process lock (byob-runtime-directories.5), not by the writer.
A crash can tear the final line of a .jsonl stream. Readers tolerate exactly that case — a torn final line, detectable because the file doesn't end in \n — and treat the file as everything before it. Corruption on an earlier line is a hard error, not a skip: silently dropping mid-file records would mask real bugs.
Records that embed free text outgrow bufio.Scanner's 64KB default token cap and surface as bufio.ErrTooLong. Ship one shared scanner constructor with a raised cap so every read-side command inherits the fix.

Design

// internal/log/jsonl.go — one writer per stream, shared across goroutines.
type JSONL struct {
    f  *os.File
    mu sync.Mutex
}

func OpenJSONL(path string) (*JSONL, error) {
    if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil { return nil, err }
    f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
    if err != nil { return nil, err }
    return &JSONL{f: f}, nil
}

func (j *JSONL) Write(v any) error {
    b, err := json.Marshal(v)
    if err != nil { return err }
    b = append(b, '\n')
    j.mu.Lock()
    defer j.mu.Unlock()
    _, err = j.f.Write(b) // one write() per line: no torn interleaving
    return err
}

// internal/log/scanner.go — shared read-side setup with a raised cap.
const lineMax = 4 * 1024 * 1024

func NewLineScanner(r io.Reader) *bufio.Scanner {
    sc := bufio.NewScanner(r)
    sc.Buffer(make([]byte, 64*1024), lineMax)
    return sc
}

// internal/runs/runs.go — the filesystem is the index.
// <state>/runs/<20260513T150405Z>/manifest.json     (atomic, versioned)
// <state>/runs/<20260513T150405Z>/events.jsonl      (appended)
const SchemaVersion = 1

var ErrSchemaTooNew = errors.New("runs: schema newer than this binary supports")

func List(root string) ([]Meta, error) {
    ents, err := os.ReadDir(filepath.Join(root, "runs"))
    if err != nil { ... }
    // run IDs sort lexicographically == chronologically
    sort.Slice(metas, func(i, j int) bool { return metas[i].ID < metas[j].ID })
    return metas, nil
}