In nine days, the main-branch test run for RoughPatch, the resume tool we’re building at TwelveApart, went from under 5 minutes to about 18. Merges passed 40 on a single day, and more than a third of main-branch runs failed. Several coding agents build RoughPatch in parallel, each on its own issue and branch, and every merge to main runs the full suite, from type checks to browser tests against a production build. We could make changes faster than ever, and we were waiting longer and longer to learn whether the combined system still worked.
That’s the part of agentic engineering I think gets too little attention. Scaling coding agents also means scaling the system that verifies their work. Agents raise the rate of change, the product grows, and every new behavior is one more thing the suite has to check. Verification costs time and compute, so implementation can outrun it. You need that verification to stay confident in what the agents ship. The engineering problem is keeping that confidence while the feedback stays fast enough to keep building.
So I moved main-branch integration off GitHub’s hosted runners and onto the machine that already runs the agents. Here’s why, how, and what the first day showed.
Our ideas outran the build
I’d seen this coming on other projects. As the models get better, agents make more of their own decisions, and with a good model you can diagnose a million things that went wrong. But every fix turns up a few more things worth improving, each of those becomes an issue, and each issue spawns more. Mandem, my local-first agent orchestration project, stalled for a couple of months that way: the work piled up and the platform work piled up. We’ll probably release it when the code is done.
Compare that with a v0 prototype in the browser. Those builds go extremely fast, and I get something I can use and iterate on almost right away. I wanted that pace on my own machine, for a real product. RoughPatch needs it more than most, because resume optimization is deeply technical and also full of feel, and the only way to get the feel right is to use it every day.
Kai and I work on it nearly every day, and we come up with three to five feature ideas a day. Kai turns ideas, research and bug reports into specs in our living product requirements document. My job is an engineering system that can build, test and release those specs fast enough for both of us to try them, and then the loop starts again. How do we ship each idea in the smallest increment possible?
Ponytail, a skill that pushes coding agents to write the least code that works, helped: our agents started making smaller changes. But the loop as a whole was still slow, and the delivery metrics showed where.
The delivery metrics showed where
At WeWork, as a Senior Staff Engineer on product and platforms, I learned to manage engineering through delivery metrics. Randy Shoup, then our VP of Engineering, pushed the staff engineers to track the four DORA metrics: lead time for changes, deployment frequency, change failure rate and time to restore service. We each got a copy of Will Larson’s Staff Engineer. Mine came off the shelf again for this.
With the agents in parallel, the project board looked great: issues moved, pull requests merged and “done” kept filling up. The four numbers told a different story. “Green main” here means the main branch with the full suite passing, and I used local stand-ins for two of the metrics: merges to main for deployment frequency, and failed main runs for change failure rate. Over the nine days before the change:
| DORA metric | Main branch, hosted runners |
|---|---|
| Lead time, merge to green main | about 15 min typical, about 90 min for the slowest tenth |
| Deployment frequency | up to 40-plus merges a day |
| Change failure rate | more than a third of runs |
| Time to restore, red main to green | about 30 min typical, over 2 h at worst |
Main-branch runs also used more than half of the month’s free hosted minutes in under two weeks. We’d been here before: a couple of months earlier, our agent work used up the month’s Actions minutes, and none of our repositories could run CI until the allowance reset.
Run length drives two of those numbers. Lead time includes at least one full run, because nothing on main counts as delivered until the suite passes. Time to restore is roughly the number of repair attempts times the run length, so a run that grows fourfold makes every incident about four times slower to close. And the agents don’t stop while main is red, so every extra minute adds more changes to sort through when you diagnose the failure. Writing code got faster. Getting a verified result got slower.
Does a faster runner fix the failure rate? No. That rate mixes real regressions, flaky tests and infrastructure failures, and sorting them is separate work. But a faster suite makes each failure cheaper to recover from while that work happens.
Integration has to happen in one place
WeWork also taught me what happens when integration gets expensive. It was remarkably good at giving small teams agency: a technology group of about 1,000 people across four hubs worked in pods, roughly Spotify’s model. But all that independent, generative work still had to come together into one app and one platform, and making sure it did was a large part of a staff engineer’s job. Without that, you crawl to a halt. I’d already seen the halt at OkCupid, where we stopped integrating new back-end code for product releases for almost six months just to un-f*ck everything.
Agents are the new pods: many of them, each acting on its own. That leads to the first principle here. Work from many independent agents has to be integrated and verified somewhere, and it can’t happen inside each agent, because checking everything against everything is too costly for any one of them. It has to happen in one place. Two changes can each pass on their own branch and still break each other once they’re combined, which is exactly what the main-branch suite is there to catch. Ours checks three things: the code and its architecture (lint, types, module boundaries, unit tests), the services and their contracts, and the product in a browser, from component tests to full runs against a production build.
Finding what’s wrong now happens at light speed. Integrating the fix is the expensive part, and it’s a big reason large companies move slowly: an idea has to clear every check before it reaches a customer. Apple’s releases take an awfully long time, and a friend at Google describes the same thing. In agentic building, by the time you’ve cleared every check, you’re already behind.
That’s why I think old-school DevOps and platform discipline (checks, verification, architectural rules) matters more now than ever. More people than ever can build software without knowing how it works, and without feedback from integration, never mind code coverage, you shoot off in a direction you didn’t mean to go.
TwelveApart’s working thesis is that the bottleneck in agentic work isn’t the agents: as agents produce more, the limit moves to how quickly whoever supervises them can notice drift and correct it. We call that alignment bandwidth (I’ve been building a supervisory layer to measure it), and CI is where that limit shows up first. Nobody can correct a problem until a run reports it, so run time sets how fast we learn that something drifted.
Moving integration onto the rig
We were running the tests twice. Every agent already runs them on the rig for its own feedback loop, since pushing code and waiting to see what happens is the slowest feedback an engineer can get. Then CI ran them again on GitHub’s standard private-repository runner, a 2-vCPU machine that reinstalls every tool and package on each run. A passing suite means the same thing wherever it runs, so why pay for the second run? The tests weren’t the waste; running them twice was. Queue time on the hosted runners was only a few seconds, so the run itself was where the minutes went, and GitHub bills bigger hosted runners by the minute. Caching alone still leaves the suite on 2 vCPUs.
Besides, I already had a better machine. The rig that runs our agents is on all day, with 32 cores and plenty of free disk. I set it up as an agent rig from code: one Ansible playbook in my configuration repository installs everything from the shell and terminal to the coding agents, and the CI runner is just one more role in it (that setup deserves its own post). I moved main-branch integration onto the rig and gave each job 6 CPUs and 16 GiB of memory. Pull-request checks are short, so they stay on GitHub’s runners.
Doesn’t moving CI onto my machine break the one-place principle? No. I moved the responsibility somewhere cheaper, but integration still happens in one place, with one runner and one consistent suite. That consistency is what lets the agents move fast without me worrying that they’ll break things.
The design choices that matter if you try this:
- One job per throwaway container. Each job gets a fresh container that registers with GitHub, runs one job and is deleted along with its disk volume.
- Limit what the job can reach. The container runs as a non-root user on a read-only filesystem, with CPU, memory and process limits, no host folders and no Docker access.
- Keep the long-lived credential out of the job. A dedicated system account holds a token scoped to one repository and trades it for a registration token that expires within an hour. Only the short token enters the container.
- Start the time limit when a job arrives. A clock that starts while the runner sits idle kills late jobs.
- Plan for concurrency. Feedback time is queue time plus run time, and one runner takes one job at a time, so at peak volume a faster run can lose its gain to the queue. A shorter run isn’t more throughput. I’m sizing a second runner against measured memory use, because the agents share the machine.
Agents read issues, documentation and dependencies written by strangers, so their code can carry outside instructions into the repository. Each CI job is partly trusted: it can run its tests, but it can’t reach the host or the credential.
What changed
On the first day, eight passing main-branch runs on the rig took about 11 minutes each, against 14 to 18 for the last hosted runs. Three more failed: two on test failures in new code and one on a bug in my runner, since fixed. That’s an encouraging comparison, not a controlled benchmark. The suite kept growing and the hardware changed, and across the whole hosted period, on a smaller suite, the median run was about 8 minutes. The runs used none of the hosted allowance, and GitHub’s billing documentation lists self-hosted runner usage as free.
So the first day shows faster runs. It doesn’t yet show shorter lead times or faster restores. Our experiment record tracks lead time and time to restore against the hosted baseline as local runs accumulate.
Where I think hosted CI is going
This is opinion, not measurement. The know-how to run builds on your own hardware is spreading, and agents make it easier to set up. Agents can now rebuild a lot of software from scratch, too. REA (Reverse Engineer Anything), which points agents at an app’s behavior and works down to its native binaries, has drawn hundreds of thousands of GitHub stars since April, and I’d like to write about it separately. Tools like that mean more competition for every service built on Git, which is already an open protocol. Plenty of tools can run a workflow, so what’s left to compete on is where the compute runs.
I’m bullish on local compute, and for in-house work I’m not sure hosted CI earns its keep in the long run. In December 2025 GitHub announced a per-minute fee for runners on your own machines, then postponed it (check before you plan around it). I read that as GitHub seeing the same thing coming. Charge for it and teams leave; don’t, and hosted minutes stop selling. Damned if you do, damned if you don’t.
When not to do this
- Your suite is already fast and your merge rate is low.
- You have no fallback. The rig becomes the gate for main, and when it’s down, main goes unverified.
- You haven’t measured how agents and CI compete for CPU and memory.
- The repository is public or takes pull requests from forks. Containers share the host’s kernel, and the job network can reach your local network.
- Your agents run on a laptop that sleeps.
What’s next
Next come a second runner, running the suite’s stages in parallel, caching what doesn’t change, test selection that reliably includes every check a change needs, and batching merges when the queue grows. We’ll judge each by lead time and time to restore: how long a change takes to reach a passing main, and how long a broken main stays broken.
The machine is the least interesting part of this. As agents increase the capacity to make changes, the system around them has to increase the capacity to verify those changes, reliably and promptly. Treat verification as something you design and scale on purpose, the same way you scale the agents.
About RoughPatch
RoughPatch compares your resume with a job description, line by line, and shows which of the employer’s requirements your experience already answers and where it doesn’t say so yet. Kai and I started it at a hackathon built around AdaL, SylphAI’s coding agent, and we thank Li Yin and the AdaL team for the event that started it. We kept building because we want to put it in people’s hands, and we’re getting close. We’ll write more about how we’re building it.
