The Code They Leave Behind

The Code They Leave Behind

Stephen Benjamin · September 4, 2026

I’ve been thinking a lot about what we’re actually trying to measure with coding benchmarks. Almost every major leaderboard evaluates models on a simple binary: did the model complete the task? That pass/fail metric hides the qualitative properties of the solution. A single score doesn't tell you how the code was built, how fragile it is, or whether you would ever want to maintain it.

Opus 5 is a great example of this gap. On paper, it looks like a massive improvement: on Terminal Bench 4.0, Opus 4.8 scored 23.6%, while Opus 5 jumped to 51.8%, surpassing Fable 5. But once developers actually started using it, many called the model practically unusable.

We see this cycle constantly: a new model lands, benchmarks hit record highs, and real-world utility fails to match the headline.

One thing I think we are missing is subjective experience with a model. What happens after the task is solved? Does it produce clean, readable code, or does it dump a fragile, tangled mess into your codebase? Does it write clearly, or is reading its rationale a painful slog?

Because many benchmarks, including Terminal Bench, are open source, we don't have to treat model outputs as a black box. We can inspect the exact artifacts these models leave behind. While evaluating code quality involves subjective elements best judged by humans, static analysis and programmatic tooling already offer a far richer picture than a binary pass/fail score.

To move beyond single scores, I've settled on five programmatic views of model behavior:

  • ReliabilityHow consistently does the model solve the same task?
  • VerbosityHow much code does it generate to achieve a working solution?
  • ComplexityHow structurally expensive, deeply nested, or fragile is that code?
  • SpecializationWhere does the model disproportionately excel or struggle across domains?
  • WritingHow clearly does it structure its code, comment its work, and explain its reasoning?

Not every Terminal Bench task can be evaluated this way, but I’ve selected 27 that produce measurable code either greenfield solutions or tasks involving debugging and repair.

View the 27 measured tasks

I’m by no means an expert in this domain, and others will likely find more insightful ways to analyze code quality, but I think it's a discussion worth starting. Feel free to critique or poke holes in what I've done here. I've made the code used to generate these graphs available in a Git repository here.

  • Fable 5.1 is more reliable in solving the same task
  • Opus 5 creates the most cognitive and cyclomatically complex code
  • GPT models excel in Science; fall behind on hardware (CAD, RTL)
  • Opus 5 produces the heaviest prose, Grok 4.6 the lightest

Reliability

When a model solves a task, does it solve it consistently?

Repeatability is the pass rate calculated only across tasks where that model recorded at least one success. Breadth counts those tasks once. Five attempts were run per model and task.

Verbosity

How much code does a successful solution cost?

Economy inverts each task-relative volume index: the most compact model on a task scores 100, and larger solutions score proportionally lower. Model values average those economy scores over tasks solved. Higher and farther right is more compact; failures remain in the solve-rate axis.

Complexity

Does the model produce structurally expensive code?

Both axes are task-relative economy scores over the same complexity-supported rows: task-best compactness is 100 and higher is better. The dashed line is an ordinary least-squares fit across the plotted model means. Unsupported languages are omitted.

Speciality

Does the model specialize in specific Terminal Bench categories?

Adjusted lift compares actual category performance with a simple expectation: model overall rate + cohort category rate − cohort overall rate. It is an exploratory normalization, not a significance test. Raw mode shows the underlying pass rates.

Writing

Does the model produce heavy prose, and is it easily understandable?

Prose economy is 100 for the lightest model mean and falls proportionally as comment burden grows. Volume uses every successful artifact; density is total comment words per 100 final-side lexical code tokens. ARI and Flesch use artifacts with at least 100 prose words and five punctuated sentences; lower ARI and higher Flesch indicate easier prose, not better technical explanations.