Zylos LogoZylos
2026-07-28

Who Tests the Tests: Mutation Testing and Negative Controls for Agent-Written Code

testingmutation-testingai-agentscode-reviewquality

Executive Summary

A test suite that is green tells you almost nothing by itself. It could be green because the code is correct, or green because the tests never exercised the failure mode, or green because the assertions echo whatever the implementation happens to do. Code coverage cannot distinguish these cases — it measures which lines executed, not whether anything would have failed if the logic were wrong. Mutation testing closes that gap: it deliberately breaks the production code in small, specific ways (a "mutant") and checks whether the test suite notices. A suite that "kills" the mutant (goes red) has demonstrated discriminating power for that fault; a suite that lets the mutant "survive" (stays green) has proven nothing for it, no matter how high its coverage number reads.

In agent-driven development, the same model may write both implementation and tests. A shared misunderstanding of the contract can therefore appear in both, which motivates checking the tests against deliberate faults. This is a risk to investigate, not a measured comparison of same-model authorship and adversarial pairing. A related software-evolution study examines a different question: whether tests newly generated for changed programs adapt to their intended new behavior. Its findings, discussed below, caution against assuming that generated tests reliably capture the semantics presented to them.

This article surveys the state of the art in mutation testing (PIT, Stryker, mutmut, cargo-mutants, and the equivalent-mutant problem), the specific new failure mode in LLM-authored tests, an emerging discipline of "negative controls" borrowed directly from experimental science, and practical patterns — used successfully in a real multi-agent review workflow — for demanding discrimination evidence before accepting a PR where one agent wrote both the code and the tests defending it.

1. Mutation Testing: Measuring Whether Tests Discriminate, Not Just Whether They Run

Mutation testing is old — the idea dates to the 1970s — but its tooling has matured enough in the last few years to be a realistic CI gate rather than an academic exercise. The mechanics are the same across every implementation: a mutation engine parses the production source, generates a population of "mutants" by applying small, well-defined syntactic changes (flip a comparison operator, negate a boolean, delete a statement, change a constant, remove a method call), recompiles each mutant in turn, and reruns the existing test suite against it. A mutant may be killed by a failing test, survive despite being covered, or receive no coverage. An equivalent mutant is behaviorally unchanged; it is not the same as an uncovered mutant and cannot be killed by any correct test. Tool reports also distinguish timeouts and invalid mutants. For example, Stryker's metrics define detected as killed plus timeout, undetected as survived plus no-coverage, and valid as detected plus undetected. Its overall mutation score is detected / valid × 100; its separately named covered-code score is detected / (detected + survived) × 100. Compile/runtime errors are excluded, and ignored mutants are reported separately. Dropping uncovered mutants from an overall score would hide untested code. A score measures detection within the chosen mutation model, not a guarantee against every real fault.

The major tools converge on this model with language-specific mechanics: PIT (pitest) for the JVM, mutating bytecode directly and fast enough for routine CI on medium codebases; Stryker Mutator for JavaScript/TypeScript and .NET; mutmut for Python; and cargo-mutants for Rust, mutating at the AST level. Trail of Bits has released two newer, tree-sitter-based tools built for agentic use — mewt (language-agnostic: Solidity, Rust, Go, and more) and MuTON (TON languages FunC, Tolk, Tact) — both storing results in resumable SQLite databases rather than dumping to stdout, so a long mutation campaign can be paused, resumed, and filtered across agent sessions (Trail of Bits, "Mutation testing for the agentic era," April 2026).

The equivalent-mutant problem. Not every mutant represents a real behavioral change — replacing x + 0 with x - 0, or mutating a dead branch that can never be reached, produces code that is syntactically different but semantically identical. These "equivalent mutants" can never be killed by any test, and they poison the mutation score by making it look artificially low. Detecting them automatically is formally undecidable, and empirical rates in real codebases run from roughly 4% to 39% of generated mutants depending on language and operator set. Mitigations include Trivial Compiler Equivalence (comparing compiled object code across mutants, catching about 30% of equivalents), mutant-classification ML approaches, and, increasingly, LLM-based equivalence detectors — Meta reports one reaching precision/recall of 0.79/0.47 on raw mutants, rising to 0.95/0.96 combined with simple static-analysis preprocessing (Foster et al., "Mutation-Guided LLM-based Test Generation at Meta," arXiv 2501.12862). One structural escape from the problem: in mutation-guided test generation, where mutants are used as prompts to generate new tests rather than as a scorecard for existing ones, equivalence mostly stops mattering to the reviewer, because engineers only ever review the resulting test cases, never the mutants themselves.

Cost control via incremental/diff-scoped mutation testing. Full-repository mutation runs are expensive — one commonly cited PIT run against JFreeChart (47 KLOC) generated 256,000 mutants and took 109 minutes. That cost profile is incompatible with per-PR CI feedback, so the practical answer that has emerged across ecosystems is to scope mutation testing to the diff. cargo-mutants supports this directly via --in-diff, accepting a diff file and mutating only the changed regions — explicitly trading some whole-codebase coverage for PR-speed feedback (mutants.rs, "Incremental tests of pull requests"). Caching is a separate optimization. StrykerJS's --incremental mode compares source and tests with a previous report, not a Git PR diff. A cold run has no results to reuse; changed tests can require rerunning mutants in unchanged source, subject to the runner's test-reporting support. Dependency and environment changes are not detected and require deliberate invalidation, such as --force for all mutants in scope. The documented example reuses 3,731 of 3,965 results, leaving 234 mutants to run; these counts do not establish an elapsed-time speedup, and the dry run is still required. For explicit selection, Stryker provides --mutate file/line patterns; deriving those from a PR diff is a separate integration step. The emerging industry convention on thresholds: teams typically start a new mutation-testing gate at 70–80%, treat 80–85% as strong for a mature production codebase, and reserve 90%+ for selectively critical modules — configured as a hard break-the-build threshold (PIT's Maven plugin, Stryker's thresholds.break) so mutation score becomes a merge gate, not a report nobody reads.

2. The New Failure Mode: Tests That Share the Author's Misunderstanding

Coverage-driven test generation has a well-known failure mode even with human authors: tests that assert whatever the code currently does, rather than what it is supposed to do. LLM-generated tests inherit this failure mode and add a second, more insidious one on top.

Test smells at scale. A large multi-benchmark study of LLM-generated Java unit tests (four models, compared against 34,635 open-source projects and EvoSuite baselines) found that "Assertion Roulette" (multiple assertions with no distinguishing failure message) and "Magic Number Test" (unexplained literals baked into assertions) are the dominant smells, and that their prevalence tracks prompting strategy, context length, and model scale rather than disappearing with a better model (arXiv 2410.10628, "Test smells in LLM-Generated Unit Tests"). More bluntly, independent commentary on this line of research observes that LLM-generated tests frequently "test nothing, and are just expressions of truisms" — tautological assertions verifying basic language semantics rather than any property of the actual code under test. A test suite can score 100% line coverage while achieving a single-digit mutation score, because coverage only asks "did we run the line," and a tautological assertion runs the line without ever being capable of failing.

The deeper problem: shared misunderstanding. The failure mode that is genuinely new — and the one that matters most for the "who tests the tests" question — is not sloppiness but correlated error. When one model authors both the implementation and the tests defending it, a misreading of the contract at the specification stage doesn't get an independent second look; it gets encoded twice, once as behavior and once as the assertion that "validates" that behavior. Trail of Bits frames the risk precisely: "an uncritical agent doesn't know whether it's encoding correct behavior or propagating bugs into your test suite," illustrating with the example of a boundary condition silently changed from priority >= 2 to priority > 2 — a change an agent might "confirm" with a test that itself asserts the wrong threshold, because the test was derived from the same read of the requirements as the buggy code.

The software-evolution study evaluates eight models across 22,374 program variants. It generates new tests for the changed programs; it is not merely running a fixed original suite against injected regressions. The semantic-altering variants define the intended new behavior. Baseline tests achieved 79% line coverage and 76% branch coverage. For semantic-altering changes, newly generated tests reached a 66% pass rate and 60% branch coverage. More than 99% of the failing tests passed on the original program while executing the changed region, indicating expectations still aligned with the old behavior. Failing the changed program is therefore a failure to adapt, not evidence that a fixed regression suite correctly caught an injected bug.

The study does not have the model jointly author buggy implementations and their tests, nor compare that arrangement with adversarial pairing. Applying its findings to shared code-and-test blind spots is a hypothesis for review practice. A second agent may help, but the relevant evidence is whether its checks expose concrete faults.

The research community's response converges on adversarial and mutation-anchored generation. Meta's mutation-guided approach (internally ACH, "Automated Compliance Hardening") flips the framing: generate mutants first and feed live/uncovered mutant information as direct prompt context, so the model targets "what would distinguish correct from broken here" instead of "what does correct code usually look like." A trial with real engineering teams (October–December 2024) produced privacy-hardening tests with a 70%+ acceptance rate, and notably, many accepted tests did not increase line coverage at all while still measurably raising mutation score — direct evidence coverage and test adequacy are different things (arXiv 2501.12862). MutGen applies the same mutation-feedback-in-prompt idea and reports large mutation-score gains over vanilla LLM prompting (89.5% vs. 77.9% on HumanEval-Java, 89.1% vs. 69.9% on LeetCode-Java) purely from telling the model which mutants currently survive (arXiv 2506.02954). A newer line of work makes the adversarial structure explicit: "AdverTest" pits a test-generation agent against a mutant-generation agent in an iterative loop, where the mutant agent's whole job is hunting the test agent's blind spots (arXiv 2602.08146, "Test vs Mutant"). The throughline matches a reviewer's intuition: a single model's unaided judgment about what a good test looks like is not independent evidence that the test discriminates — something outside that model's own reasoning has to supply the discriminating pressure.

3. Negative Controls: An Import From Experimental Science

Long before software testing existed, experimental science solved a version of this exact problem. In any assay, a result of "no effect detected" is ambiguous between two very different states of the world: the effect genuinely isn't there, or the assay isn't sensitive enough to detect it even when it is there. The standard discipline for resolving that ambiguity is to run two controls alongside the real experiment. A positive control is a sample known in advance to produce a detectable effect; if the assay fails to detect it, the assay itself is broken or under-powered, and any "no effect" result elsewhere in the same run is worthless. A negative control is a sample known in advance to produce no effect; if the assay reports one anyway, something in the setup — contamination, a mislabeled reagent, an operator error — is generating false positives. Both controls exist purely to answer "does this apparatus have the power to tell a real answer from a fake one," independent of what the apparatus says about the actual sample under test.

The mapping onto software test oracles is close to exact, and explains why a green suite is not, by itself, evidence of anything. A test oracle's positive control is the code passing when actually correct — the ordinary case everyone already checks. Its negative control is far more rarely built deliberately: does the oracle actually fire, go red, when the thing it claims to guard against is really present? Mutation testing mechanizes negative controls at scale: each mutant is a known-bad sample injected on purpose, and "did the test kill it" is exactly "did the assay detect a known positive" (the mutant plays the positive control for the fault; a suite that lets it survive has an assay with no demonstrated sensitivity to that class of error). A concrete SQLite upsert case illustrates the pattern cleanly. Consider a reconciliation job that repeatedly writes rows keyed by a natural key, using INSERT ... ON CONFLICT (key) DO UPDATE to guarantee idempotency. A test asserting "after two reconciliation runs, there is exactly one row per key" looks like solid evidence the upsert works. Two deliberate negative controls decide whether that evidence is real:

  • Drop the unique index but keep the explicit ON CONFLICT target. SQLite's ON CONFLICT (col) clause requires a matching unique index or constraint to bind against; without one, the very first execution of the statement should fail loudly (a SQL error, not a silent no-op). If the test suite stays green when the index is missing, the "loud-fail" oracle it's implicitly relying on has never actually been demonstrated to fire — the suite has been trusting an invariant it never checked.
  • Degrade the upsert to a plain INSERT, retaining the unique constraint. The first insert succeeds; a second insert with the same key fails with SQLITE_CONSTRAINT rather than adding a duplicate. Run reconciliation twice, change a non-key value on the second input, and assert both successful completion and the updated stored value. That kills the mutant through a failed rerun or a missing update if an exception was swallowed. Row count alone is insufficient: the unique constraint still keeps it at one. Testing duplicate accumulation would require separately removing uniqueness as well; label that as a compound mutation, not the effect of replacing the upsert alone. See SQLite UPSERT.

A test that survives both of these mutations proves nothing about the upsert contract, regardless of how confidently green it reports. This is the exact sense in which "prove the assay has power" from wet-lab science and "prove the test discriminates" from software converge: neither domain accepts an absence of a detected problem as meaningful unless the detector has first been shown, on a known-bad sample, to actually detect.

4. Prove It Red First: TDD Discipline as Portable Acceptance Evidence

Test-driven development's red-green-refactor cycle already encodes this discipline procedurally, and it generalizes cleanly to reviewing someone else's (or some other agent's) finished work, not just to writing new code from scratch. The core TDD rule — confirm the test fails before writing the code that makes it pass — exists for exactly one reason: if you never see red, you cannot distinguish "this test validates the new behavior" from "this test happened to pass already and validates nothing." Simon Willison's notes on agentic engineering patterns state the rule in its most portable form for AI-driven workflows: "write the automated tests first, confirm that they fail, then iterate on the implementation until the tests pass" — and explicitly warn that skipping the fail-first step "risk[s] building a test that passes already, hence failing to exercise and confirm your new implementation" (Simon Willison, "Red/green TDD" pattern).

For review rather than authorship, the same discipline translates into a specific, checkable demand: a claimed regression test must be shown to fail against the pre-fix commit and pass against the post-fix commit — both states demonstrated, not asserted. This is mechanically simple to enforce with git worktree: check out the parent commit into a second working directory (no stashing required, since a worktree starts clean from the last commit rather than a dirty working tree), run the new test against that unmodified worktree and confirm it fails, then run the same test against the current branch and confirm it passes. The pattern generalizes as a baseline-attribution technique for agent-driven changes more broadly — establish a green baseline on the untouched worktree before handing it to an agent, so any new failure afterward is attributable to the agent's change rather than pre-existing flakiness. Applied to regression tests specifically, a "regression test" never shown to fail on the broken commit is not a regression test, it's an assertion of faith — and in agent-authored PRs, faith is exactly what's under audit. As of mid-2026 this remains a manual reviewer discipline rather than a packaged CI gate; running new/changed test files against the merge-base commit as a required job is a concrete, low-effort tooling gap (see Section 6).

5. Practical Patterns for Multi-Agent Review Workflows

Put together, the state of the art suggests a small set of concrete, cheap-to-apply techniques for a review workflow where one agent implements a feature and writes its own tests, and a second, independent agent (or human) performs acceptance review before merge:

Reviewer specifies the mutation; author demonstrates the kill. Rather than running a full mutation-testing suite (expensive, noisy with equivalent mutants, hard to interpret quickly), the reviewer picks one or two mutations that target the specific contract claim the PR is making — "this upsert is idempotent," "this cache invalidates on write," "this retry is capped" — and asks the author to show the test suite going red under each. This is dramatically cheaper than exhaustive mutation testing and dramatically more targeted: it tests the exact claim under review, not a random sample of the mutation operator space. The SQLite upsert example in Section 3 is precisely this pattern in practice: two mutations, chosen because they map directly onto the two failure modes the contract claims to prevent (constraint mismatch, and failure to update successfully on repeated writes).

Diff-scoped mutation testing as a CI gate, not a nightly report. For teams that want this automated rather than reviewer-driven, select the mutation scope explicitly: cargo-mutants accepts a PR diff through --in-diff; a Stryker integration can translate selected changes into --mutate patterns. Apply incremental caching separately, with the invalidation limits described above. Neither a smaller mutation scope nor cache reuse guarantees wall-clock cost proportional to the edit size: baseline tests and the tests needed for selected mutants still contribute. The Augment Code guide frames the useful failure-recovery loop clearly: route surviving mutants on the diff back into the test-generation prompt as a targeted "fix these" request, rather than surfacing a bare percentage nobody acts on (Augment Code, "Mutation Testing for AI-Generated Code"). Complementary tooling like ChaCo ("Change And Cover") targets the same PR-scoped surface from the generation side, using patch coverage to find which changed lines remain untested and augmenting the suite at review time (arXiv 2601.10942).

Oracle diversity, not oracle redundancy. Ten example-based assertions that all check the same kind of thing (all "does field X equal expected value Y") are not meaningfully more discriminating than one — they share the same blind spot. Real discriminating power comes from combining oracle types: a loud-fail oracle (must raise on a malformed precondition), an invariant/property-based oracle (Hypothesis or QuickCheck check "does this hold for all generated inputs" rather than "does this one example match"), and a state-scan oracle (query actual persisted state after an operation and check a global invariant like row-count, rather than trusting the operation's own return value). The upsert example combines two of these three, which is why a single mutation surviving both is much stronger negative evidence than either alone.

Treat "discrimination evidence" as a PR artifact, not a verbal claim. The practical upshot across the sources above is that acceptance review should ask for something attachable and checkable: a worktree-diff showing the regression test red on the parent commit and green on HEAD, or a short mutation log showing the reviewer-specified mutant(s) killed. This is cheap to produce (minutes, not a full mutation-testing pipeline) and closes exactly the gap that a green CI badge cannot close — that the test suite was ever shown to have the power to fail.

6. Open Problems and Tooling Gaps

Several gaps remain unaddressed by current tooling, each a reasonably scoped opportunity for agent-platform builders rather than an open research problem:

  • No standard "red-on-parent" CI job. Every mainstream CI system can run the current suite against the current commit; none ships a first-class job for "run this new/changed test file against the merge-base commit and require it to fail." That's a small amount of scripting around git worktree plus a test-file selector, but it doesn't exist off-the-shelf in GitHub Actions, GitLab CI, or CircleCI today, despite the underlying mechanics already being used ad hoc by reviewers and agents.
  • Mutation testing as an inter-agent handoff artifact, not just a developer-facing report. Existing tools (PIT, Stryker, mutmut, cargo-mutants, mewt/MuTON) are built around a human reading a report or CI badge, not a structured, machine-readable "acceptance evidence" object designed to travel with a PR between two cooperating agents — e.g., a compact manifest listing which reviewer-requested mutations ran, which were killed, and the worktree diff proving parent-commit failure. This is mostly integration work on top of existing engines, not new mutation-testing research.
  • Equivalent-mutant detection is still the long pole for full-repo mutation testing. LLM-based equivalence detectors are promising (0.95/0.96 precision/recall with preprocessing in Meta's internal numbers), but that result is one organization's internal tooling, not yet broadly reproduced or open-sourced at the same quality bar.
  • No consensus on how much reviewer-driven mutation demand is "enough." The dual-oracle SQLite pattern in Section 3 works as a targeted spot-check chosen by a reviewer who understands the specific contract at stake, but there is no established methodology for systematically deriving which mutations are worth demanding for an arbitrary PR, short of domain judgment. Turning that judgment into a checklist — "for a CRUD upsert, demand these two mutation classes; for a cache, demand these two" — is a documentation opportunity that would lower the bar for reviewers who don't already think this way.
  • Cross-model adversarial pairing is promising but not yet standard practice for the ordinary case of one agent implementing and a different agent reviewing. Independent judges still sharing a meaningfully elevated rate of correlated error suggests a second model is not automatically an independent check — architectural diversity between implementer and reviewer (different model family, not just a different prompt) appears to matter, and there is no good empirical guidance yet on how much diversity is enough to break a shared blind spot.

The overall shape of the evidence, across mutation-testing tooling, LLM test-quality research, and the experimental-science analogy, points the same direction: in a world where the same model can plausibly generate both a bug and the test that fails to catch it, "the tests pass" has to stop being treated as evidence on its own. What counts as evidence is a test suite shown, on a specific known-bad sample chosen by an independent party, to actually go red.