At about four o'clock on Thursday morning, UK time, Kun Chen posted that AI-written unit and integration tests are now "empirically proven to be unhelpful", and by the time this side of the Atlantic was properly awake the replies had turned into the sort of pile-on you normally only get when someone says tabs are fine. At the time of writing it's sitting on over 360,000 views and more than 400 replies, which is a lot of people with strong feelings about expect().toBe().

I've read the post, the follow-ups and far too many of the replies, and where I've landed is this:

Kun's numbers don't say tests are dead. They say a test the agent writes from the same reading of your prompt as its code isn't really a check at all, it's the same guess written down twice, so what your teams actually need to own is where the intent behind a test comes from, and the agents can happily do the typing.

Credit where it's due, this is an actual experiment rather than a vibe. Kun ran Sonnet 5.5 on high effort against the DeepSWE eval set with two arms, one allowed to write whatever tests it fancied and one told to write none. The no-tests arm came out with a slightly higher success rate (not statistically significant) while spending less time and fewer tokens, and that part was significant. Write-ups of the run put it at 66.2% against 65.3% across 444 runs.

He then looked at what the baseline arm had written. About 65% of it was unit tests and 35% integration tests, and neither bucket did any better than writing nothing at all. On a random subset of 44 tasks he went further and disabled running even the existing tests, and the success rate didn't move. The bit I found most telling is almost a footnote, out of 3,000+ tests the agents wrote, only 17 were end to end.

You also need to know how DeepSWE marks the work, because it comes up again later. Each task has two hidden sets of tests, new behaviour tests that should fail before the change and pass after it, and curated regression tests that must keep passing. The agent never sees either, and Kun says around 10% of the failed tasks were caught by the regression set, so they're not just there for decoration.

This is the part I think is right, and Kun put it better than I would have:

If you've reviewed much agent output this year you'll have seen the pattern, even if you didn't have a name for it. The agent reads the ticket, writes a function, then writes a test that very carefully confirms the function does what the function does. Here's a frontend flavoured version (made up, but I'd bet money you've seen its cousin):

The agent's test passes and the ticket's test fails, and only one of them has ever met a product owner. Coverage goes up and the PR goes green regardless, which makes it less a testing problem than a provenance one, because the first test's expected value came straight out of the code it was meant to be checking.

The data is interesting but the headline got ahead of it, and a few of the replies were properly good on why.

• The success gap is tiny. Steve Bishop worked it through as roughly 147 against 145 tasks out of 222, with a confidence interval of around nine points either way. Not significant means the eval can't see an effect that small, which isn't the same as the effect being zero, and Kun himself later moved to a more modest framing that Matt Pocock said he much preferred.

• A one-shot benchmark has no next commit. Tests mostly earn their keep on the change after this one, like the refactor you do in three months or the CI gate that stops a Friday afternoon deploy. Kun's answer is that the 44-task subset already covered regressions and that TDD genuinely helped humans on the first pass too, which is fair, but one benchmark still can't score the benefit of a suite you'll lean on next quarter.

• Nobody graded the tests themselves. Matt Pocock asked whether DeepSWE looks at the quality of the test suite after the task is done, rather than just the running code. I haven't seen a yes, and to be fair it isn't really what the benchmark is for.

Kun's longer follow-up is worth reading in full, because it's where the actual argument lives. He agrees that Bun's rewrite leaned heavily on its tests, points out that those tests were carefully curated by humans to describe how the system should behave, and then says the quiet part out loud:

So we've got one camp saying the tests were never the point, another saying the tests are the only point, and a benchmark that agrees with both of them depending on which column you read. Honestly I think they're mostly arguing about the same thing from opposite ends, which is that a test is only worth as much as the intent it encodes, and an agent that wrote both sides of the exam hasn't encoded anything new.

Nothing here needs buying, which I always appreciate. It's mostly about who writes down what done looks like, while the agents keep doing what they're actually good at.

• Stop asking for tests as a reflex. If your AGENTS.md says "always add tests" and nobody ever reads them, you're paying for an echo. The one result in Kun's run that was statistically significant was the time and tokens, so that's real money you're spending on agreement.

• Write the cases, let the agent write the code. Put acceptance criteria in the ticket as named examples (inputs, expected outputs, the awkward edge you already know about) and have the agent turn those into test code. Kun suspects human-described tests still add value but says plainly he hasn't proven it yet, so treat this as my opinion rather than his finding.

• Tests are instruments, so somebody owns them. Same argument as the Uncle Bob post. Put the acceptance and e2e folders behind CODEOWNERS, and don't let an agent "fix" a failing test by editing the test without saying so in the PR.

• Watch the e2e space. Seventeen e2e tests out of 3,000+ says the agents barely reach for them unprompted, which is a shame, because a real user journey is the one thing the agent can't derive from its own implementation. I had a go at this in the Tester Army post and the UI eval loops one, and Kun has an e2e eval running that I'm very keen to see.

I don't expect the agents to follow that perfectly (they're agents, they'll find a creative reading of "never" by Tuesday), but it moves the argument from "did the tests pass" to "whose tests were they", which is a much better argument to be having in a PR. In the meantime I'll be refreshing Kun's timeline for that e2e result like it's a DPD tracking page, and if it comes back saying e2e tests are pointless as well then I'm going to need a little lie down.