Recap

I have written many times about using Claude to fix bugs and add features to Forge, a tiling window manager for GNOME, that I use daily. Last month, I wrote about adding end-to-end (E2E) tests that run in GitHub CI/CD to catch regressions. This post picks up where that one left off – what happened when I pointed that safety net at the whole extension and could not stop pulling on loose threads.

Chasing Perfection

After implementing E2E testing for Forge, I was fairly confident that I was nearly “done” with the extension. Instead of accepting that there might still be bugs lurking and fixing them as they appeared, Claude made it too easy to preemptively try to find (and fix) them all. Chasing this perfection led to 170 more commits and 209 issues closed (140+ bugs) over the last 5 weeks…

Importing GitHub Issues

The first set of bugs that I decided to chase was low-hanging fruit – those that had already been reported on the upstream GitHub repo. It had been a few months since I last triaged the issue board so I decided that it made sense to triage everything again from scratch to make sure that I did not miss anything. The process for this was to use gh to download issues in JSON format and then ask Claude to triage them against our fork to identify which ones were fixed, not fixed, and not applicable. In total, this process identified 48 new issues which were all filed as beads and then fixed over several plan-implement cycles, resulting in roughly 28 commits.

Test-suite improvements

As I worked to expand the E2E test infrastructure, I repeated a mistake that I had made before when generating unit tests for Forge – I gave Claude too much freedom which allowed it to create tests that were not useful. Some tests were trivial:

  it("should create node with default TILE mode", () => {
    const newNode = ctx.tree.createNode(monitor.nodeValue, NODE_TYPES.CON, new St.Bin());
    // CON nodes don't have mode set, but WINDOW nodes would
    expect(newNode).toBeDefined();   // never checks mode === TILE
  });

Others could not fail (e.g., no expect()):

  it("should not add duplicate actorBin to window_group", () => {
    global.display.get_n_monitors.mockReturnValue(1);
    const existingBin = {};
    global.window_group._children.push(existingBin);
    global.window_group.contains.mockReturnValue(true);

    monitorManager.addMonitor(0);
    // add_child should still be called (for the new bin)
    // but contains check prevents duplicate
  });

Others created inline implementations of the functions that they were supposed to test:

  it("should find sibling window when focused window is removed", () => {
    // ...builds container + node1, node2...
    const siblings = container.childNodes.filter((n) => n !== node1);   // the "logic" -- inlined
    expect(siblings.length).toBe(1);
    expect(siblings[0]).toBe(node2);
  });

I learned a new word in this process, vacuous (meaning devoid of substance or meaning), which I now include whenever prompting for tests as something to avoid.

Around the time that Fable 5 was first available, I decided to use it to audit the E2E tests and look for vacuous tests. This burned through a lot of tokens across several sessions but I figured it would be worth it. I had Fable turn its findings (39 test/CI/E2E-infrastrcture issues) into beads so that I could fix them in later sessions.

Fuzzing

After improving the test suite, I felt pretty confident that we would surface most regressions for future changes through CI/CD. When I did research on Android (a long time ago), I used a monkey to test my apps using random inputs. It did not seem like a big lift given the existing test infrastructure so I asked Claude to build a similar monkey for Forge. This yielded an early win: a GNOME Shell crash caused by a keyboard swap re-parenting a window’s title tab mid-move. Encouraged, I had Claude expand the fuzzer to more actions and windows, which surfaced a second crash: closing a window in a stacked or tabbed container touched an already-disposed tab actor. I let this cycle continue and Claude ran several fuzzing campaigns but no other crashes occurred. This was probably worth the effort given that crashing the shell is pretty serious, but I do wonder if more careful code auditing could have uncovered the bugs instead.

Fable

In parallel to the test suite audit, I decided to also use Fable to audit the extension itself to see if it could find any more bugs. I did this twice – once in mid-June and once in early-July. The first audit surfaced around 40 bugs and the second another 28. Again, these all ended up in beads as tasks, which we worked through over the following weeks in follow-up sessions (primarily with Opus).

Other Improvements

As fixing bugs all the time can get boring (even with Claude doing all the heavy lifting), I also used a few sessions to work on a few additional features. These were mostly minor (e.g., golden-ratio resize, keybinding conflict detection) but I did want to promote several “experimental” features to auto-enabled. Given the robust test suites, this seemed appropriate.

I also started working towards a more polished release – adding a signed release artifact and cleaning up the documentation. The v49-90-beta.2 pre-release is now available for testing.

Life of a bug

At this point, Claude has rewritten significant portions of the Forge extension which made me curious – where did all these bugs come from? Did they already exist in the code or were they from the code that Claude added? Naturally, I asked Claude to help me figure this out.

Fixed product bugs by area and origin. Blue is inherited (already in the code); amber is self-inflicted (introduced during this effort). Nearly every area is dominated by pre-existing bugs – cheatsheet, a small area, is the lone exception.

Fixed product bugs by area and origin. Blue is inherited (already in the code); amber is self-inflicted (introduced during this effort). Nearly every area is dominated by pre-existing bugs – cheatsheet, a small area, is the lone exception.

Lessons Learned

At this point, I have clearly hit the point of diminishing returns. Claude makes it too easy to chase perfection. Much like the slot machine where one more prompt will get the code to work, one more session will be the last one that finds bugs. I want to see this project through to completion which includes merging these changes upstream. But, for now, I have tagged a beta release and decided to enjoy the extension for a few weeks before iterating further.