SlopCodeBench measures how coding agents behave when they repeatedly extend software they previously wrote. The May 2026 paper evaluates 15 coding agents on 36 problems containing 196 checkpoints. Each checkpoint adds requirements to the agent's existing workspace, exposing how early design choices affect later work.
No evaluated agent completed every checkpoint of any problem. GPT-5.5 achieved the highest strict checkpoint solve rate at 14.8%. The paper also reports that its two targeted code-quality measures deteriorated during most trajectories while average cost per checkpoint rose as projects progressed.
What SlopCodeBench measures
Each problem begins with an empty workspace. The agent receives a specification, implements it, and carries that implementation into the next checkpoint. Later requirements can reward a flexible initial design or expose shortcuts embedded in earlier code. Problems contain three to eight checkpoints.
One example begins as a command-line source search tool with exact and regular-expression matching. Later checkpoints add languages, structural pattern matching, selectors, and automatic fixes. An early architecture organized around one language and one matching mode becomes part of the next checkpoint's starting state.
Specifications describe observable command-line or API behavior. They do not prescribe internal interfaces, structure, or architecture. Test suites remain hidden, including held-out cases beyond the examples in the specification. The task design is language-agnostic, although the paper evaluates only Python implementations because of experimental cost.
How correctness is scored
A strict solve requires the workspace to pass the current checkpoint's tests and regression tests from earlier checkpoints. An isolated solve excludes earlier regression tests, helping distinguish failure on the new requirement from damage inherited from prior work. A core solve counts behavior explicitly described or demonstrated by the specification.
| Measure | Reported result |
|---|---|
| Problems completed end to end | 0 of 36 |
| Best strict checkpoint solve rate | 14.8%, GPT-5.5 |
| Best isolated checkpoint solve rate | 28.1%, GPT-5.5 |
| Core pass rate, early to late | 64.6% to 35.5% |
| Mean cost per checkpoint, early to late | 2.2× increase |
Across the evaluated configurations, core correctness fell from 64.6% near the beginning of a problem to 35.5% at the end. Error-handling correctness fell from 80.1% to 62.2%. Mean cost per checkpoint increased 2.2 times, while the proportion of lines changed declined from 97.4% early in a project to 29.5% late in the project.
This combination describes agents spending more while making increasingly localized changes to larger inherited workspaces. It does not by itself identify the cause of each failure, but it shows why a single final pass rate misses important trajectory behavior.
How the paper measures code quality
The authors define two static measures for following code changes across checkpoints: structural erosion and verbosity.
- Structural erosion is the share of a codebase's cyclomatic-complexity mass concentrated in functions whose complexity exceeds 10. The score rises when more control-flow complexity accumulates inside already-complex functions.
- Verbosity is the proportion of source lines affected by structural duplication or one of 137 targeted AST rules. The rules identify patterns such as unnecessary intermediates, redundant checks, and avoidable constructions.
These metrics cover concentrated complexity, duplication, and particular redundant patterns. They are not complete measures of maintainability, architecture, readability, or long-term development cost. A high score does not prove that software is unusable, and a low score does not prove that its design is sound.
Quality deteriorates during iteration
Structural erosion increased in 77% of agent trajectories and verbosity increased in 75.5%. The average number of functions with cyclomatic complexity of at least 10 rose from 3.6 to 23.7. Mean maximum cyclomatic complexity increased from 27.5 to 69.0. Structural duplication grew by 96%, while the density of other AST-rule violations changed by only 0.3%.
The duplication result suggests that much of the measured verbosity came from copying and extending existing structures rather than introducing new categories of violation. Iteration made small structural choices accumulate.
Comparison with open-source Python repositories
The paper calibrates its measures against 473 open-source Python repositories and 13,667 sampled commits. Agent checkpoints averaged 0.44 verbosity and 0.68 erosion. The repository panel averaged 0.19 verbosity and 0.34 erosion. Under the paper's definitions, agent code was 2.3 times more verbose and 2.0 times more eroded.
The trajectory comparison produced a larger gap. Agent verbosity grew about seven times faster per checkpoint than the median rate in the repository histories, while erosion grew about five times faster.
Prompting improves the starting point
The researchers tested an anti-slop prompt and a plan-first prompt on GPT-5.3 Codex, GPT-5.4, and GPT-5.5. The anti-slop prompt reduced average verbosity by 27.5% to 35.6%, depending on the model, and reduced erosion by 34.3% to 57.6%. The improvement usually changed the level at which code started, not the rate at which it degraded across checkpoints.
The interventions introduced trade-offs. Anti-slop prompting reduced average strict correctness by 2.4 percentage points, while plan-first prompting reduced it by 3.6 points. Across the tested configurations, the modified prompts raised average cost per checkpoint by 12.1%.
Limits on the result
SlopCodeBench evaluates one implementation language, a fixed set of hand-authored problems, and a particular snapshot of models and native agent harnesses. Results can change with model updates, tools, context management, or different checkpoint designs. Hidden tests measure only the behavior encoded by the benchmark authors.
The quality measures are targeted static indicators, and the open-source calibration is not a matched maintenance study. These limits do not erase the observed trajectory, but they constrain claims about production codebases or maintainability as a whole.
How to interpret SlopCodeBench
SlopCodeBench identifies a gap between satisfying a current specification and preserving a codebase's capacity to absorb later changes. Current agents can pass individual checkpoints, but their prior design choices often become liabilities as requirements accumulate.
The benchmark supports evaluating coding agents as maintainers of evolving software, with correctness, cost, regression behavior, and structural change considered together. It does not establish a universal limit on autonomous development. It provides evidence that iterative degradation remains a distinct weakness under these long-horizon conditions.
For a complementary view of project-scale completion, read the MirrorCode benchmark summary. MirrorCode asks whether an agent can reproduce a complete program under strict behavioral tests, while SlopCodeBench asks what happens as the agent repeatedly changes the code it chose to build.