PID lockfile with stale takeover for single-orchestrator CLIs

byob-runtime-directories.5 concurrencystate

Problem: a CLI whose long-running mode owns shared state (a work loop, a sync, a migration) must ensure only one orchestrator runs per repo/state-dir at a time. flock (byob-runtime-directories.3) excludes correctly but is invisible: nothing on disk tells an operator who holds the lock, sibling commands can't report "a run is already active as PID N", and a lock held by a hung process looks identical to no lock at all.

Idea: a PID lockfile with stale takeover. Acquire writes the current PID to a temp file in the lock's directory and links it into place with os.Link — the file appears in the namespace atomically, so a reader never sees a half-written PID, and Link failing with ErrExist is the contention signal. On contention, read the recorded PID and probe liveness (kill(pid, 0); treat EPERM as alive — better to refuse than to clobber). A dead PID means a stale file: remove it and retry inside a bounded attempt loop (a racing peer may win — losers either see a live PID or retry). A live PID returns a sentinel ErrHeld whose message includes the PID. A separate read-only ActivePID(path) helper lets status/gc commands report or avoid the active run without taking the lock.

Tradeoffs:

vs. flock: flock auto-releases on crash (no stale handling) but is invisible to ops and other commands. The pidfile is inspectable (cat pid.lock) and powers "already running as PID N" UX, at the cost of the stale-detection code, the bounded retry loop, and a theoretical PID-reuse race (an unrelated process recycled the recorded PID and the lock reads as held — rare enough in practice; the failure mode is refusing to start, not corruption).
Single-host only, like flock. State on a network filesystem needs a different scheme entirely.
Release deletes the file and is idempotent; a crashed holder simply leaves a stale file for the next Acquire to take over — no cleanup daemon needed.

Design

var ErrHeld = errors.New("lock: held by another process")

func Acquire(path string) (*Lock, error) {
    const maxAttempts = 16
    for attempt := 0; attempt < maxAttempts; attempt++ {
        l, err := tryCreate(path) // temp-in-same-dir + os.Link → atomic appearance
        if err == nil { return l, nil }
        if !errors.Is(err, os.ErrExist) { return nil, err }

        pid, readErr := readPID(path)
        if errors.Is(readErr, os.ErrNotExist) { continue } // peer released; retry
        if readErr != nil { return nil, readErr }
        if processAlive(pid) {
            return nil, fmt.Errorf("%w: pid %d", ErrHeld, pid)
        }
        // Stale — remove and retry.
        if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
            return nil, err
        }
    }
    return nil, fmt.Errorf("lock: could not acquire %s", path)
}

// ActivePID reports the recorded PID and its liveness without taking
// the lock. Missing file → (0, false, nil): nothing is running.
func ActivePID(path string) (pid int, alive bool, err error)

func processAlive(pid int) bool {
    p, err := os.FindProcess(pid)
    if err != nil { return false }
    err = p.Signal(syscall.Signal(0))
    return err == nil || errors.Is(err, syscall.EPERM)
}

Callers surface ErrHeld with a hint (byob-errors.2): "another run is active as pid N; wait for it or stop it".