tl;dr: Ten weeks and 2,593 Ralph iterations later, Rust is the default symbolic execution engine and Python is the fallback. Along the way I asked Claude for moonshots, pruned its memory store from 1,296 entries down to 414 and watched it grow back, and settled into a rhythm of auditing on weekends so Ralph has something to chew on all week.

Recap

I have been exploring agentic engineering and have been using Claude to create a Rust Symbolic Execution engine for Angr based on inspiration from a blog post. In April, I wrote about how the port was going, and in May I wrote about rebuilding the automation that drives it – Ralph, a Go tool that loops Claude over a backlog using a clean/dirty state machine. This post covers the ten weeks since then.

Much like my previous post, this entry documents how I have kept pushing on the Rust Symex without a clear stopping point.

Moonshots

Given the solid foundation for the Rust Symex, I asked Claude to be more ambitious and explore how to extend the engine beyond what the Python engine does today. Claude came up with a list of moonshots:

  1. Crack the Z3 wall – pre-classify solver queries to skip expensive Z3 calls
  2. Deterministic, reportable results – identical found-sets/models across runs
  3. Real-binary practicality – directed search + state merging on real binaries
  4. Hybrid concolic engine – pair the icicle fuzzer with symex, Driller/QSYM-style
  5. Parallel find-all payoff – does parallelism actually pay off on real workloads?
  6. Flip the default engine to Rust – make Rust the default, Python the fallback

These all seemed interesting so I let Claude plan a roadmap for each, including creating go/no-go decisions. The roadmap was stored in beads (over 90 issues) so that I could use Ralph to work through them.

Here is how each one landed:

  1. NO-GO – a structural pre-check only classified 2.2% of queries as cheap, well short of the 30% bar needed to justify building it.
  2. GO – canonicalizing constraint and witness ordering made results reproducible, even though raw Z3 wall-clock timing stayed irreducibly variable.
  3. Deferred – the implementation shipped, but it is still waiting on a real binary with the right control-flow-divergent shape to actually validate against.
  4. Partial GO – the fuzzer-to-symex handoff was proven sound in principle, but full validation needs an x86 pypcode/sleigh environment that my sandbox does not have yet. Fixable, just not done.
  5. Deferred – same problem as #3, no real “wide and slow” target to measure the speedup on yet, though chasing one surfaced and fixed several parallel-drain bugs.
  6. GO – every child issue shipped and default_engine="rust" is live, backed by nightly parity CI.

The one that matters most is #6: Rust is now the default engine, with Python as the fallback. The port is no longer an experiment and is hopefully getting close to being ready to submit a PR upstream.

I still plan to write a post on the state of the engine itself but I am waiting for the benchmarking results to be something truly worth writing up.

So Many Memories

Since early April (even before switching Ralph to a FSM), Claude has been accumulating memories in beads via bd remember. Specifically, in my Ralph prompt footer:

10. **MANDATORY — save what you learned.** Run `bd remember` for each that applies:
    - Root cause was surprising or non-obvious? `--key <topic>-root-cause`
    - An approach failed before the one that worked? `--key avoid-<thing>`
    - A constraint/invariant future work must respect? `--key invariant-<thing>`
    - Profiling showed bottleneck was NOT where expected? `--key <topic>-bottleneck`
    - A benchmark number changed significantly? `--key benchmark-<topic>`

    Do NOT skip this step. Context dies between sessions; memories are the only
    bridge. **Anchor references to symbol names (fn/struct/method) instead of
    raw line numbers** — line refs drift 10-600 lines across refactors while
    symbol anchors stay resolvable (see `refactor-memory-sweep-rule`).

At the beginning of June, I decided to check to see how many memories it had made and was surprised to find almost 1,300 memories. That seemed like a lot, especially since memories appear in every context window via bd prime, so I decided to ask Claude what to do with all of them. Specifically, I was interested in what to convert into “long-term” memories (a.k.a., documentation) versus what was fleeting, and whether any of the memories were in conflict.

The first “round” of pruning used some heuristics and light agent review which reduced the number of memories from 1,296 to 832 – 464 forgotten. This still felt like a lot of memories so I had Claude do a more aggressive prune, reviewing the body of each memory, which dropped another 418 to land at 414. I did not have a chance to look through the 882 memories forgotten across those two rounds so I decided to save them and review them later.

A month or so later, I checked again and the number of memories was back up to 934. I had Claude prune them yet again, and this process repeated several times until I decided to spend a few sessions in the repo with the memories to characterize what was forgotten and then adapt the Ralph footer to restrict what gets created. I will need to do an update in a few weeks to see if the rate of memory creation has decreased. While I was working with the memories, I made a site so that I could look through them, copying a lot of the design from byob-go-cli.

Some of this is a tooling gap rather than a prompting one. Beads memories currently carry no per-key timestamps, so bd prime can only order them alphabetically when it hits its cap and there is no way to audit which guidance has gone stale. Recency-aware selection would make a large store much less dangerous.

Just Keep Auditing

My typical workflow has been to audit the codebase for bugs and then use Ralph to address the findings. My goal has been to find enough bugs on the weekends that I can let it iterate on through the week until I hit my weekly limit. This process seems to work reasonably well with minimal oversight but it is not very fast or efficient. The last three bug audits (from scratch, not incremental) have produced 239, 149, and 50 issues, respectively. We seem to be trending in the right direction but we are not there yet.

After each audit, I have been using a few sessions to understand the types of bugs that were discovered and then apply fixes to the root causes. One of the current themes is that the agent decided to recreate code instead of extracting a shared helper. That is near-miss cloning – copies that drift as each site gets patched independently, so a fix in one place silently misses the others. Detecting code clones was a large focus in my dissertation and it makes me want to point those techniques at agent-generated code.

Ralph Stats

Ralph has been the workhorse of my agentic engineering experiments. It works surprisingly well and has clocked a huge number of iterations over the last ten weeks on this project.

State transitions – the FSM edges Ralph took, where clean and dirty describe the state of the working tree at the start of an iteration and done/failed/revert describe how a task ended. These are transitions rather than iterations, so they do not sum to the iteration count below:

  • clean: 2287
  • dirty: 233
  • done: 26
  • failed: 5
  • revert: 50

Cost:

  • iters: 2593
  • wallclock: 303h56m35s (~12.7 days)
  • cost: $4,549.37 (API-equivalent cost)

Fortunately, the monthly subscription costs far less.

Systemd

I can now check on Ralph from my phone. Claude recently added “server mode” to remote control – claude remote-control starts a server that can be used to spawn sessions. In the past, checking in on Ralph meant keeping a long-running remote control session running which eventually accumulated a lot of context. Now, I can spawn a new session whenever I feel like it.

One problem with this approach was that I was also using these sessions to start audits and Ralph itself which sometimes OOM’d during cargo commands. This never took down the machine – it killed my Claude session – but it still left me restarting the server by hand, so I had Claude create a systemd service. The two lines that matter are Restart=always, which brings the server back without me, and MemoryMax=4G, which caps the service cgroup so a runaway cargo has a ceiling:

[Unit]
Description=Claude Code Remote Control (angr repo)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu/repos/angr
Environment=HOME=/home/ubuntu
Environment=PATH=/home/ubuntu/.local/bin:/home/ubuntu/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/home/ubuntu/.local/bin/claude remote-control
Restart=always
RestartSec=5
MemoryAccounting=true
MemoryMax=4G
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

I have been thinking about doing something similar for Ralph – I added a --wait-for-beads bd ready flag so that instead of exiting on done{idle} or done{queue_empty}, Ralph parks polling bd ready and resumes when the queue changes. With that, I could probably make it so that Ralph is always on standby.

Lessons Learned

  • Pruning a memory store is treating the symptom. I pruned several times before I did the thing that actually worked, which was editing the prompt that creates the memories. The store regrew at roughly the same rate after each prune because I had not addressed the root cause.
  • Trading machine time for my time is the whole point. Finding bugs on the weekend and letting the loop grind through them all week costs me very little time. It costs a lot of machine time – 12.7 days of wallclock, about $1.75 an iteration – but that is the resource I have in surplus.
  • Agents duplicate rather than abstract. Left alone, the loop consistently recreated code instead of extracting a shared helper. It is a bias worth naming explicitly in the prompt, because it does not show up as a failing test – it shows up months later as three copies that have drifted apart.

Summary

It has been interesting to continue to explore how to build tools in the agentic era. This has been the biggest project by far – the existing Rust Symex is over 160K LoC. Are we there yet? Not quite – the engine is good enough to be the default but there is still work to do to eliminate bugs. I am hoping to get this process refined enough that I can finally make progress on all the various side projects that I have thought about over the years.