Self Healing Codebases

As we work more and more with AI agents, and as we increase the speed at which we ship, one concern is always there: does it really work? It’s not possible, I believe, to increase the speed significantly without the agents being able to run the application and test and see everything that it does. Once they do, though, once we see them clicking links and entering data and checking browser errors and all that results, a thing happens: we gain confidence.

Confidence that the agents do a fairly correct thing.

It’s not everything we want, but it’s a lot. Enter verify-openhands and control-openhands.

How It Works

I gave my agent a link to poteto’s gist about her verification skills, and told him to analyze everything in OpenHands, understand all user-facing scenarios embedded in the codebase, then make a CLI to drive OpenHands as a user would. I also asked it to write, in the same time, the full feature map of OpenHands.

I had a few more prompts in that conversation, handling decisions like how deep should it go or whether the CLI should support concurrent runs and how. Then there it was: a verify-openhands skill.

First, the skill contains a CLI for agents: any agent can use it to run OpenHands in a browser. It’s called control-openhands. This CLI sets things up for real-world tests, and it has commands helping it to do things like check for errors easier or clean up and start from scratch in different environments.

Second, the skill has a feature map, for all behaviors that a user may want to initiate when using OpenHands. Start a conversation, add a backend, switch the backend, switch the LLM during a conversation, restore, filter in the left panel, you name it. What a user can do in the UI, is what the agent can do too.

With these two main helpers, agents can run everything.

First Run

Then I ran it all. And I mostly just watched, as far as I recall (though there was some messing around, I can’t remember why) I watched how it went through all of them, it filed issues for the bugs it found, then it asked me: should it also fix them? I said yes. Then I found myself with 70+ PRs on my hands, nice and green and ready for review.

On a side note, this reminds me of my first attempt to make a Kanban board from my OpenHands conversations: I initially thought it was such a great idea, until I tried it. Then… in less than an hour, all conversations ended up in ready-for-review column. Review was, of course, the bottleneck. Of course! I remember my thought at the time was “oops, actually, maybe Kanban is not a good UI for working with agents…”.

Well, who knows what is a good UI for agents. The important thing, it seems to me, is that they became relatively fairly easy to review when I asked the agents to attach their full verify-openhands screenshots and videos to each PR.

As poteto put it somewhere, what you need is to do whatever it’s missing until you can trust your agents.

I think this is exactly right. Sure, sure, they still implement stuff in the wrong place and are perfectly able to break the code design, but you can judge that. And you have to judge that, anyway.

Daily Self-Healing

I wrote some instructions for daily runs, and set up an automation to run daily. The automation looks at the PRs merged the previous day, works out what they were trying to achieve, writes the missing features or adapts the behaviors in the feature map accordingly, then runs them.

If the agents end up with failures, they run those scenarios twice to be sure, then file the bug; when the run is done, they start new agents to fix all that was newly filed. When I wake up, I merge the fixes.

Self-healing software. It became daily routine: self-healing OpenHands finds and fixes some 1-4 issues almost every day.

We live in interesting times. Yes, it’s a blessing… or a blessing too!