Just keep auditing

Recap

I have been trying to use Claude to create a Rust Symbolic Execution engine for angr based on inspiration from a blog post since late January. In August, I wrote about letting Ralph grind away for 10 weeks, chasing moonshots, and three waves of bug audits. This post covers many more audits, an exploration of the bugs that the audits found, and the latest benchmark results.

Just Keep Auditing

I was somewhat hopeful based on the trend from my first three bug audits that we were converging towards a codebase that was suitable to share with others. Those audits produced 217, 138, and 95 findings, respectively, categorized into bugs, tasks, and chores. Each of these audits was done from scratch and the trend was looking great. To my dismay, the next audit produced 83 issues. Not to worry, another Ralph session can fix those and then maybe the trend will resume? Nope. I proceeded to run the audit/ralph a total of eight times over a two-month period, producing a total of 855 findings (yikes). This table shows the number of findings (and their priorities) for each audit:

Audit findings per round with priority

These are based on Claude’s self-assigned priorities (assigned by a finder agent and confirmed by a verifier agent). P1 is the highest priority and P4 is the lowest. The number of P1s has generally decreased over time. I tried to hold the audit workflow consistent across rounds (specifically, asking Claude to do the N+1th comprehensive audit of the codebase) but the workflow ended up drifting over time. The audits all started with the same goal: auditing native/angr/src from multiple angles including correctness, DRY, and organization. The way the codebase was split varied slightly for each audit. Later rounds added a new angle for overflow and width-boundary bugs based on bugs that were discovered in previous audits.

During the remediation phase, Ralph actually found about 20 more bugs that the audit rounds missed and filed them as additional findings (and a bunch of non-bug findings) under the same epics used to track the audit rounds. The bugs fell into categories like “fixed one site, then checked for the same shape elsewhere” and “writing tests for one bug surfaced another”. The reuse of the top-level epic confused Claude (and myself) when trying to document how many findings each audit identified (including an error in my previous post). It would have been better to snapshot the beads database after each audit to avoid this confusion.

Hidden in the table above is the breakdown of bugs, tasks, and chores:

Audit findings per round broken down by type

Again, these are based on Claude’s self-assigned categorizations for bug versus task versus chore. In round 6, it decided to make everything a task (specifically, the bead type, which we use for counting). It did put the intended category in the bead titles (e.g., “chore:”). Repo conventions (if they were actually enforced) could have improved our ability to compare audits.

Most of the audits were conducted with Opus. The last audit was done with Sonnet 5 and it still found plenty of bugs. I am curious to go in the opposite direction to see what an audit with Fable finds. Hopefully Fable 5.5 comes out soon and is token-efficient enough to make this feasible.

Bug Investigation

To better understand the bugs produced by these audits, I asked Claude to write me a script to tabulate, for each bug:

• When was it added?
• What model added it?
• What subcomponent was it in?

The script traces the lineage of each bug to identify when it was introduced via git blame and some extra tracing. The extra tracing was due to a couple of resets early on. Once the bug and its introducing commit have been identified, the model that wrote the code can be extracted from the Co-Authored-By line that Claude likes to append. The audits started at commit 2e6cac6f5 (2026-07-30) and the analysis snapshot was at commit 8c157f891. The bugs introduced by model are as follows:

Bugs already in the code at the audit baseline (2026-07-30)

It is surprising to see just how many models have contributed to the project. Opus 5.5 is too new to have written code before the beginning of the auditing. I usually switch to newer models as they become available (newer is always better) and was curious which models wrote most of the code. Perhaps models with many bugs just wrote more code:

Code at the audit baseline, with bugs per 1k production lines

Opus 4.8 wrote the most code (and bugs) but had the lowest bugs per 1k production lines. Sonnet 5 broke the measure by writing code with no bugs found (yet). There is a lot of noise in these measurements; hopefully some good papers on bugs/LoC across models come out soon.

Lastly, I wanted to understand what we actually did during all the fixing passes. This summarizes the changes in lines of code per subsystem:

What changed during the audit/fix loop, by subsystem (2026-07-30 → HEAD)

The tests grew significantly in comparison to the production code. This was due to the prompt asking for a test case that failed on the bug and went green once it was fixed. It would be interesting to see if one model wrote more of a given subsystem than another (and whether that explains why some subsystems changed more than others). I also need to revisit these numbers with cloc.

Finally, I had Claude review the bugs that it had found so far and gave it some rough guidance to create a rubric for the project. At some point, it would be interesting to do a retrospective for all audit findings against this rubric and see how things change.

Towards Merging

I doubt that the angr developers want to merge this any time soon. There are a lot of minor things that we have found along the way that I have started to carve out and submit PRs for:

• libc: add getopt, getopt_long and getopt_long_only merged
• libc: add str*_chk and __fprintf_chk, fix _chk prototypes merged
• libc: add stpncpy, memrchr, rawmemchr, strchrnul, asprintf and getline merged
• format_parser, strtol: fix printf/scanf conversions merged
• glibc.json: fix __fprintf_chk and __sprintf_chk prototypes

There has also been a ton of movement in the upstream repo that I have had Claude update the rust-symex branch to match. Funnily enough, upgrading all the dependencies and checking the outputs uncovered another eight bugs in the Rust engine (five of which are P1s based on the new rubric).

Latest Benchmark

It has been a while since I benchmarked the Rust engine against the latest upstream code. Inside my little VM with 8 CPUs and 8 GB memory, I had Claude benchmark our Rust engine (commit 940333138) against the upstream engine (commit d1e9ec806) using angr-examples (commit c98733f):

There is definitely a speedup, but the two biggest speedups come from subtle differences in the engines:

Lessons Learned

A few lessons learned from this process:

• Define a rubric (e.g., severity, categorization) for Claude to follow when filing bugs in your repo.
• Do not expect one audit to find all the bugs (or even a significant fraction of them).
• Expect fixing bugs to introduce new bugs.

Summary

At this point, I think the Rust symex engine is in decent shape. It would be neat to discuss with folks at SynSec but I cannot attend this time. Hopefully there will be another similar conference next year.

Links

• rust-symex branch
• Triage rubric
• angr-examples
• Ralph