AI News

SpaceXSI Engineer Poteto on How Grok Bot Merged 11,000 PRs in One Month

Shao MengUpdated Nov 15 21:44

AI summary

SpaceXSI Principal Engineer Lauren Tan (@poteto) spoke at AI Engineer Code Summit 2026 on "All Code Is Slop (Except Mine)," describing how the Grok Bot team merged about 11,000 pull requests into production last month, with the relative rollback rate holding steady at about 0.5%. She advocates that every time an agent is corrected, it should first be turned into a lint rule, directory convention, or interface constraint rather than adding explanations to AGENTS.md, and she bans about 99% of comments; code review then shifts toward recordings, screenshots, and formal verification results, with both her and her colleague Cindy spending most of their tokens on verification. She emphasizes first making the codebase and verification trustworthy, then scaling up outer-loop automation such as automatic merging.

Why read this

AI score 74

She shares the concrete sequence for writing corrections into lint and architecture and shifting review toward verifying artifacts, which can be transferred to your own agent workflow.

Topics

Tags

Body · AI translation

SpaceXSI Principal Engineer Lauren Tan @poteto gave this 17-minute keynote on the main stage at the AI Engineer Code Summit 2026 in San Francisco, titled "All Code Is Slop (Except Mine)." For an overall introduction to the AI Engineer Code Summit 2026, you can first read my analysis article:

She opened by saying that last month she merged 3,000 pull requests into production, many of them while she was asleep. It took about six months to get to this point. What she really set out to argue was not the volume itself, but that the work of software engineers is becoming: maintaining or even improving code quality while greatly increasing output. Some people call this path a software factory; she prefers to frame it as trust: not blindly trusting agents, but investing time in the agents' working environment so that people have reason to believe they will do the right thing.

Where to spend time

She uses an inverted pyramid to allocate effort. The bottom two layers are deterministic, and both live in the code itself: one layer is interface design, architecture, and refactoring; the other is lint rules, the type system, the compiler, and tests. Together these two layers constrain what agents can write and force them by default onto the right path. Above that is verification, including runtime verification and formal verification, used to find defects, check properties that should always hold in the system, and judge whether agents have truly understood the goal and intent. At the very top she leaves only a little written documentation: rules, Skills, AGENTS.md, reserved for things that still require judgment and cannot yet be written as checks.

The corrective approach she repeatedly emphasized was: every time you correct an agent, first think about whether it can be turned into a repeatable check, rather than adding another line to AGENTS.md. Constraints written into code are easier to reproduce and more worthy of trust than sentences written in documentation.

The codebase is the largest input, and a garden that spreads

An agent's output depends on its input, and the codebase is often the largest such input. In 2024 she still remembers reviewing three pull requests written by colleagues at work; now it is closer to 300. Each engineer is surrounded by hundreds or even thousands of agents, code pours in almost continuously, and it grows outward like something alive. She compares the codebase to a garden: the constraints and guardrails people establish are the concrete foundation that new code will grow along. As a result, the engineer's job increasingly resembles gardening.

This also has a very practical consequence. Workarounds, outdated patterns, and technical debt all look to an agent like "the customary way of doing things here," and the next agent will very likely copy them. Weeds spread on their own. So she advocates keeping the codebase as clean as possible, so that agents are always copying the good parts.

Use frameworks and checks to remove entire classes of errors from review

To this end, the Grok Bot team built a client framework called Dune. She said it is not particularly novel technically and can be roughly understood as Next.js for Electron applications; what is really interesting is the large number of constraints and lint rules accumulated from building desktop apps that are written into it.

One principle is: any interface that requires a lot of judgment to use well tends to produce low-quality code. Her example is React's useEffect. It has its appropriate uses, but misuse is very common, and agents do not use it well either. Their approach is not to keep watching for it in review, but to ban it in product code, allowing it only inside some of Dune's interfaces. This class of problem thus disappeared from code review.

Another approach is to make conventions clear through directory structure. She mentioned that in Cursor, also an Electron-based application, agents often import main-process code into the renderer process, causing long tasks and jank. In Grok Bot and Dune, the top level is split into directories named electron-renderer and electron-main, and the dependency analyzer and lint rules forbid cross-boundary imports. The rules themselves are not complicated, but there is one less thing to keep watching for during review.

Further down the line, there will be fewer and fewer things to watch for in review. She said that at a certain point you find there is almost nothing that must be looked at by a human, and most pull requests can be left to agents to merge themselves.

Comments and documentation may become excuses for bypassing the right way

A more surprising observation was: agents treat Markdown as a place to justify themselves, including docs, code comments, rules, and Skills. They will write there "why this time it's okay not to do it the right way"; they may also interpret a small offhand correction from you as a hard rule that can never be broken.

She gave the example from her time at Cursor. At the time, the agent loved using imperative timers like setTimeout, requestAnimationFrame, and requestIdleCallback as a stopgap to patch over the architectural conflict between VS Code's imperative style and React's declarative style. A large number of timers being rapidly created and destroyed increased garbage collection, which made the UI stutter. Worse, the agent also wrote lots of comments explaining why this stopgap was justified. The next round, the agent saw these comments and treated them as established practice, so timers spread everywhere and application performance noticeably degraded. Her judgment was: the architectural choices at the time made both the client and the agent perform worse.

By the time of Grok Bot, they chose to ban about 99% of comments. The reason was not just that comments easily go stale and people rarely read them carefully, but more importantly that comments cannot be enforced: you think you're telling the next agent "don't do this," but in reality nothing stops it. They switched to lint errors with clear explanations, requiring the agent to fix the real root cause rather than patching around it. The general principle is: every correction should either become a deterministic check, or be eliminated at the root through better interfaces and architecture.

The object of review shifts from the diff to verification artifacts

The other half of trust is verification. She said code review has now narrowed to mainly looking at verification artifacts: recordings, screenshots, charts, and formal verification results.

The most basic is runtime verification—put simply, letting the agent run the program itself and use the debugging tools developers commonly use, such as the Chrome DevTools Protocol. Their runtime verification skill includes a feature map, essentially a sitemap listing, from the user's perspective, what features exist in the application. That way, when you say "that thing in the left sidebar," the agent doesn't have to guess, and both sides share common names. The skill also comes with reusable command-line tools and drivers, which saves tokens and can also be incorporated into tests.

They are still experimenting with formal verification, but already use it for some critical paths. They use TLA+ for model checking, for example the notification flow across devices and platforms; they use Lean to prove certain algorithms, such as their own virtualization list library, to rule out flickering and jitter. She said they have already found quite a few defects, and it looks promising, but it is by no means a panacea—just another layer of tooling in the environment. On the UI side, they have also started using a library called Bombadil for property-based testing, which has already caught some UI defects.

She mentioned the change in Cindy: six months ago, Cindy did not dare trust her own agent because of the subtle defects that got merged in; six months later, she says 70% of her tokens each day are spent on verification. Tan said this matched his own experience—most of his tokens are also spent on verification. These methods themselves are not new; they were just too expensive to write and maintain before, and now agents have brought the cost down. She therefore said that software testing and verification are entering a golden age—not because there is less verification, but because verification can be broader and deeper. She has also built a skill called statechart in her own plugin pstack (potato stack), used to draw state machines from the user's perspective and review them alongside recordings and screenshots, which is clearer than just looking at a screenshot of one successful state.

First make the inner loop reliable, then talk about the software factory

As a return on these investments, she gave a set of team-reported numbers: last month the Grok Bot team merged about 11,000 pull requests into production; the client cuts a stable release roughly every day; in October about 227 people contributed at least one pull request, including casual contributors without much context. The relative rollback rate not only did not rise, but stayed stable at about 0.5%. She thinks what is more noteworthy is: at this speed and with this many people, Grok Bot's startup, input, opening a new conversation, and switching conversations are all faster than at release. It is still an Electron app, so its performance has a ceiling, but she believes it is already faster than quite a few other agent interfaces.

She laid out the order very clearly. Don't start by building a software factory and automating everything that can be automated. Every time you correct the agent, first change the codebase: first add lint to block the new wrong way of writing, then clean up the old ones; at the same time, make verification solid. These are the prerequisites for automation in the first place. If the inner loop itself is producing low-quality code, automating the loop will only produce more low-quality code, faster and more expensively.

Once the foundation is laid, the outer loop becomes easy. She gave an example: now every time a new version is cut, Grok Bot automatically spins up a Cursor project, fuzzes the entire application and build, operating it like a real user, and after finding defects opens fix pull requests separately and merges them automatically. It sounds bold, but the premise is that the codebase is already good enough to "pass by default," and most product code no longer needs deep human review.

A few thoughts after reading

This talk places great emphasis on: where corrections should be written. When the agent repeatedly makes the same kind of mistake, the easiest reaction is to add another line to AGENTS.md; Tan wants people to move that line out of the documentation and write it as an interface, a directory convention, or a check that throws an error. Comments and Skills documentation are re-evaluated here: what they preserve is often an excuse, not a constraint.

Another layer equally worth learning from is that verification is becoming the main cost. Both she and Cindy spend most of their tokens on verification, and formal methods and property-based testing have become worthwhile again because agents have become cheaper. This is not the same thing as "making agents write faster," and may even be the opposite: to merge with confidence, you have to spend more computation proving that it was done right.

These practices can reach automatic merging partly because Grok Bot controls the entire client tech stack itself. Banning useEffect, banning most comments. What is transferable is the order: first make the codebase and verification trustworthy, then scale up automation. If the inner loop is unreliable, the outer loop will only amplify the problems.

This sharing session strongly emphasizes: where corrections should be written. When the agent repeatedly makes the same kind of mistake, the easiest reaction is to add another line to AGENTS.md; Tan wants people to move that line out of the documentation and write it as an interface, a directory convention, or a check that throws an error. Comments and Skills documentation are re-evaluated here: what they preserve is often an excuse, not a constraint.

Another layer equally worth learning from is that verification is becoming the main cost. Both she and Cindy spend most of their tokens on verification, and formal methods and property-based testing have become worthwhile again because agents have become cheaper. This is not the same thing as "making agents write faster," and may even be the opposite: to merge with confidence, you have to spend more computation proving that it was done right.

These practices can reach automatic merging partly because Grok Bot controls the entire client tech stack itself. Banning useEffect, banning most comments. What is transferable is the order: first make the codebase and verification trustworthy, then scale up automation. If the inner loop is unreliable, the outer loop will only amplify the problems.

Source

Shao Meng