Testkube CTO: Why Testing Must Be Built into the AI Software Delivery Assembly Line from the Start
Testkube co-founder and CTO Ole Lensmar argues that testing should be an independent stage in agentic software delivery workflows, rather than an add-on at the end of CI/CD. DORA reports show that AI adoption is associated with higher delivery throughput but lower delivery stability, while related research has also found that agents may modify or delete failing tests. The article recommends restricting agents from altering existing tests, running tests early in real testing environments, and centrally storing results for use by both humans and agents.
Why read this
AI score 63
Drawing on developer surveys and research into agent testing, the article explains why rising code output can exacerbate delivery bottlenecks and proposes a practical path centered on an independent testing control plane.
Topics
Body
I’ve been building testing tools for more than twenty years, and for most of that time my conversations with engineering teams sounded fairly similar. Testers felt like second-class citizens in the software lifecycle. Testing happened late, often squeezed into whatever time was left before a release, and injected into a CI pipeline that was built for something else. AI has now transformed how we write code – and the impact on testing is definitely there – for better and worse. In a lot of ways, AI has made the challenge more visible.
Over the past few months, I’ve been at a handful of testing and cloud-native conferences (LeadDev, CloudX, StarWest), and nearly every team I spoke with is working on some version of the same project: an AI-driven software delivery lifecycle, where agents take work from a spec to something running in production with the right amount of human involvement along the way. I picture it as an assembly line. Agents write the code, other agents review it, and pipelines build and deploy it. Humans keep an eye on things. What I hear much less about is where testing sits on that line.
Right now, it’s a bit of a Wild West. New agent capabilities come out every few weeks, the large AI labs keep releasing new models, and teams and vendors alike are throwing things at the wall to see what sticks.
Some of the data suggests that gaps are already showing. Google’s most recent DORA report on AI-assisted software development, which surveyed nearly 5,000 technology professionals, found that AI adoption is linked to higher delivery throughput, which is good news. It’s also still linked to lower delivery stability. The researchers’ explanation is that AI speeds up development and exposes weaknesses further downstream, and they point to strong automated testing as one of the controls that stop more change from turning into more instability.
I don’t find that surprising. If you push a lot more code through a process that was designed for much less of it, it slows down somewhere, or it breaks somewhere.
When people imagine what goes wrong with AI-generated code, they tend to picture an agent pushing something terrible into production. I’m sure that happens – and when it does, the consequences aren’t great: bugs, unhappy customers, maybe a security incident. But TBH, it isn’t what I run into most of the time.
The 2026 Stack Overflow Developer Survey shows where AI use is concentrated today. Developers mostly use it to generate code in areas they already know (67%) and to debug (61%), while only 20% use it to deploy, operate or troubleshoot production systems. So far, AI has mostly sped up writing code. Getting that code verified and into production still works largely the way it did before.
This lines up with what I’m seeing, too: teams producing a lot more code without releasing any faster. An individual developer or team might feel like they’re building twice as many features, but those features get stuck downstream, waiting for review, waiting for tests, waiting for an environment to test them in. All that extra code clogs the pipeline, and in many cases it even slows down teams’ overall delivery.
While the concern about AI-induced delivery bottlenecks is very tangible, it isn’t the only thing teams ask us about. Developers mostly want to know how to write better code faster. The questions about guardrails come from engineering managers and the people setting up these agent-driven workflows. How do we make sure the agents don’t go rogue? How do we know they aren’t introducing regressions, or quietly skipping work they were supposed to do? How do we make sure the tests they write actually (and truthfully!) validate what they’re supposed to?
I think that’s because they’re looking at different things. A developer looks at one pull request at a time. An engineering manager is responsible for everything that gets released across many agents and many people, and is the one who has to explain it when something breaks in production.
And they have reason to be careful. Researchers built a benchmark called ImpossibleBench using coding tasks where the unit tests contradict the specification, so the only way to pass is to cheat. Leading models often did exactly that, for example by editing or deleting the failing tests instead of fixing the code (!!). A separate study of agent-authored pull requests found gaps in how well agents test their own changes, and it cites earlier research showing that nearly a third of merged agent pull requests needed follow-up fixes or refactoring.
My takeaway is that guardrails can’t live only inside the agent’s own environment, and they can’t only come after code ships. If the agent that wrote the code also decides whether it passed, the result doesn’t tell you much. That doesn’t mean agents shouldn’t write tests. It means the verdict has to come from somewhere the agent can’t adjust. Testing needs to be its own station on the line, one that every change goes through, whether a person or an agent wrote it.
What That Station Needs
The answer isn’t a brand-new kind of testing. Most of what’s needed is what good QA teams have asked for all along. It just matters more now.
Tests need to be reviewed for accuracy against requirements – just like the code they test. Tests also need to run in the environment the software runs in. A test that passes in an agent’s sandbox doesn’t tell you much about staging or production, and it can hide which dependencies were mocked out – or how. A recent study of nearly 180,000 agent-generated test files found that agents covered more edge cases than human-written tests, but their tests showed more signs of being flaky, largely because the agents didn’t know enough about the environment the tests would run in. Running tests against real infrastructure early catches that.
Tests and their results need to stick around. If an agent writes a test, it should have existing tests and their results as context so it can learn from both coverage and potential flakiness. Tests need to be reviewed and versioned like any other code, so they run again on the next change instead of disappearing when the session ends.
A passing result should mean the same thing no matter who asked for it, whether that’s a pipeline, a developer or an agent. That means the same tests, running on the same infrastructure, with the results in one place.
And people should keep their judgment. I’m happy to let agents handle the boilerplate and the wiring. Deciding what’s worth testing, and what a failure means, is still a human job (for now at least…).
This also means testing can’t live only inside CI/CD (if you remember one thing from this article, make it this!). CI pipelines were designed to build and ship code; they were never built for running automated tests at scale. As we shift toward an agentic SDLC and the amount of code keeps growing, CI becomes an even poorer fit for running tests. Testing needs to be a first-class citizen of how software gets delivered instead of a step attached to the end of a pipeline. That means orchestrating it as its own system, across every tool and environment a team uses, regardless of who is driving – an agent or a human.
Where to Start
Most teams already have tests they trust in Playwright, Cypress, k6, Postman or whatever they’ve standardized on, and that’s a great place to begin. A few things I’d suggest:
Draw out how a change gets from a prompt to production today, and mark every place a test runs. If there’s only one, near the end, I’d start there.
Be careful about letting agents quietly change existing tests. The ImpossibleBench researchers found that restricting agents’ access to tests reduced cheating. When an agent does need to change a test, the change should come through as a pull request that a person reviews, like any other code.
Run your tests in (preferably real) test environments rather than agent sandboxes, as early as you can, even if it’s only a small smoke suite against staging for every agent-generated change. If you use AI to write tests, don’t accept one until it has run there.
Centralize your test results in one control plane – be it a reporting tool, your APM solution or a dedicated testing platform. Make sure these results are available to both humans and agents.
Measure delivery rather than activity. The number of pull requests says very little. DORA’s delivery metrics, such as change lead time and change fail rate, tell you much more about whether AI is helping.
Let agents use the same testing infrastructure your people use, so their work is checked against the same standard.
Optimistic, with Conditions
I’m a glass-half-full person when it comes to AI, and I think it will make software teams far more capable and, ultimately, more productive. But to get there, teams need to trust what their agents produce enough to ship it, and they need a holistic approach to testing in the agentic SDLC – which is where that trust is either built or lost.
If you’re striving for an agentic SDLC, where does testing sit in it today?
AI Trends