Field report · · AgentConn Team
Same Model, 17x the Cost: How Your Harness Decides Who Wins
FrontierHarness tested one model across 9 harnesses and found 17x cost variance. Here is how to read agent benchmarks without getting fooled.
Same Model, 17x the Cost: How Your Harness Decides Who Wins
Take one model — Kimi K3 — and run it through nine different agent harnesses on the same 30 coding tasks, with the same fresh checkpoint, the same vCPU, the same memory. The cost per successful pass ranges from $1.05 to $18.34. That is a 17x spread. Not from switching models. Not from changing prompts. From the harness alone.
That finding comes from FrontierHarness Eval, a Show HN project by Runta that dropped yesterday and immediately surfaced the uncomfortable truth the agent ecosystem has been circling for months: the infrastructure wrapping your model is a bigger variable than the model itself, and most benchmarks do not disclose it.
Yesterday we covered how GitHub Trending has become a skills marketplace — the land grab for the layer above the model. Today we cover the layer below the leaderboard: the measurement infrastructure that decides who “wins” and who “loses,” and why those results may be telling you less than you think.
The FrontierHarness Numbers
FrontierHarness ran 360 trials across 12 configurations of 9 harnesses, burning through 2 billion tokens on the same Kimi K3 model. Every trial started from an identical fresh checkpoint restore. The results:
| Harness | Pass Rate | Median Cost/Task | Cost/Successful Task |
|---|---|---|---|
| Codex | 66.7% | Mid-range | Mid-range |
| Claude Code | 63.3% | $18.34 | $0.288 |
| DSH Creator | 63.3% | — | — |
| Pi | 60.0% | $2.43 | Low |
| DSH Standard | 60.0% | — | — |
| Kimi Code | 56.7% | — | — |
| Exo Harness | 53.3% | $1.05 | — |
| OpenCode | 50.0% | Low | $0.062 |
| Hermes | 50.0% | — | — |
Codex led on raw pass rate at 66.7%. But Claude Code, while scoring a comparable 63.3%, cost $18.34 per task — the most expensive option by a wide margin. Exo Harness cost just $1.05 but passed only 53.3% of tasks. OpenCode had the lowest cost per successful task at $0.062, but its 50% pass rate means half your tasks fail entirely.
The headline “17x cost difference” is not an edge case. It is the spread between the cheapest and most expensive harness running the exact same model on the exact same tasks.
The Academic Case Is Already Made
FrontierHarness is not the first to find this. Two academic papers published in May 2026 formalized what practitioners were discovering empirically.
Harness-Bench (arxiv 2605.27922) ran 5,194 execution trajectories across 106 sandboxed tasks and multiple model-harness pairings. Their conclusion: “agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone.” The same GPT-5.3-Codex ranged from 64.7% to 78.4% across different harnesses. Claude Opus 4.6 ranged from 58.0% to 76.4%. Those are not rounding errors — they are 14- to 18-point spreads driven entirely by the harness.
“Stop Comparing LLM Agents Without Disclosing the Harness” (arxiv 2605.23950) went further, introducing the Binding Constraint Thesis: for long-horizon tasks evaluated across models with comparable frontier capability, the harness is a stronger determinant of performance than the model it wraps. The paper documents ranking reversals — cases where Model A beats Model B under Harness X, but Model B beats Model A under Harness Y. Until harness specifications are disclosed, they argue, leaderboard comparisons should be treated as “incomplete and potentially misleading.”
The Commit Log’s “Rethinking AI Agents” video captures the shift in framing: the model is no longer the unit of comparison. The model-harness pair is.
Industry Data Confirms the Pattern
This is not just an academic finding. Every major industry benchmark published in 2026 converges on the same conclusion.
Databricks benchmarked coding agents on their own multi-million-line codebase in July 2026. Same model, different harness: cost per task varied more than 2x at identical quality. Opus 4.8 via Pi cost approximately $0.74/task versus $1.94/task through the native harness. Their finding: “The token price of a model is a poor indicator of actual costs incurred on end-to-end tasks.”
Paddy Srinivasan distilled the Databricks finding into the sharpest formulation: “The harness is the runtime. Same model. Same codebase. Very different cost/perf. Because the harness controls: repo search, context hygiene, file selection, tool orchestration, test/repair loops, state across turns.”
Composio ran DeepSeek V4 Flash through 4 agent harnesses on 30 agentic tasks. A different harness won on each metric — success rate, cost, and speed. Across 240 runs, only 6 of 30 workflows were completed successfully by every harness. The median time per task ranged from 122.7 seconds (Claude Code) to 272.4 seconds (Oh My Pi).
Artificial Analysis launched the Coding Agent Index in May 2026 — the first public benchmark that evaluates full agent stacks (model + harness) rather than models in isolation. Their data shows the same model, wrapped in two different harnesses, can generate a bill that differs by 32x.
And the cost amplification compounds. HuggingFace’s analysis of evaluation costs found that “small scaffold choices can multiply costs 10x” and documented a 33x cost spread on identical tasks across different scaffolds. The Holistic Agent Leaderboard spent $40,000 on 21,730 rollouts — and adding statistical credibility via 8 reruns per cell would push that to $320,000.
What the Community Is Saying
The FrontierHarness Show HN thread surfaced the tension between the finding’s importance and its methodological limitations.
On the core finding, HN commenter netcyrax put it bluntly: “As models become commodities, the harness will be the next optimizing game.” This tracks with the broader convergence signal — Polymarket now prices “best agent” as moving faster than “best model,” and GitHub Trending shows the infrastructure land grab already underway.
On methodology, the thread raised serious objections. yorwba pointed out that “with just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap.” joshheitzman flagged that using median cost instead of mean “understates actual billing.” kaishin criticized the missing harness code: “Putting only the tasks and results in the repo is a poor decision. These conclusions would be far more credible if anyone could re-run the benchmark.”
On home-field advantage, vidarh raised perhaps the most important critique: testing against Kimi K3 specifically skews results because Kimi has “quirks requiring specific behaviors.” Harnesses built around Anthropic or OpenAI models may underperform not because they are worse, but because they were optimized for a different model’s tendencies. The benchmark author acknowledged this and committed to a full harness-model matrix in v1.1.
The harness-over-model narrative has a convenient beneficiary: harness vendors. If the model is a commodity and the harness is the moat, then Runta (which sells agent runtimes), Composio (which sells harness tooling), and every “harness engineering” consultant has a financial incentive to amplify the signal.
That does not make the finding wrong — the data is clear. But consider vidarh’s critique about home-field advantage: much of the observed variance may come from model-harness co-optimization, not raw harness quality. Codex was built FOR OpenAI models. Claude Code was built FOR Claude. Testing both on Kimi K3 measures them outside their optimization envelope. The 17x spread is real, but some fraction of it is misattribution — the harness is not the only variable, even when the model is held constant, because harness quality is model-conditional.
The practical implication: do not blindly trust a harness benchmark that tested on a model your team does not use. The harness that wins on Kimi may lose on Claude.
Cole Medin’s “Harness Engineering” deep-dive captures why this matters for practitioners: the most valuable engineering skill in 2026 is not prompt writing or model selection — it is assembling and tuning the harness layer that sits between the model and your codebase. We covered this shift two weeks ago in Stop Chasing Models. Fix Your Harness. — the FrontierHarness data now puts hard numbers on the thesis.
How to Read Agent Benchmarks Without Getting Fooled
The harness-variance finding does not mean benchmarks are useless. It means they require a different literacy. Here is the framework:
1. Check Whether the Harness Is Disclosed
If a benchmark ranks models without specifying the exact harness, prompt template, retry budget, context window strategy, and tool configuration used for each, the ranking is noise. The “Stop Comparing” paper calls this the minimum disclosure standard. Most current leaderboards fail it.
2. Ask: Was the Model Tested in Its Native Harness?
Codex tested in Codex’s harness will look different from Codex tested in Pi’s harness. Harness-Bench showed 14-18 point accuracy spreads driven by this mismatch. A model underperforming in someone else’s harness does not mean the model is weak — it means the pairing is suboptimal.
3. Look for Cost-Per-Success, Not Cost-Per-Task
FrontierHarness found OpenCode was cheapest per task but had a 50% pass rate. The real metric is cost-per-successful-completion. Exo Harness was cheapest per task ($1.05), but OpenCode was cheapest per successful task ($0.062). These tell different stories — and your budget cares about the second one.
4. Demand Statistical Significance
With 30 tasks, confidence intervals are enormous. yorwba’s critique on the HN thread is correct: overlapping confidence intervals at these sample sizes mean you cannot reliably rank harnesses that are within 10 percentage points of each other. Look for benchmarks with 100+ tasks or multiple model backends.
5. Budget for Harness Evaluation
Databricks found the right harness could cut costs 2x at identical quality. At scale, that is the difference between a viable product and a money pit. Budget for testing at least 3 harness configurations before committing to production.
What This Means for You
The “best model” question was a 2024 question. In 2026, the question is: what is the best model-harness pair for my specific workload, at a cost I can sustain?
Three shifts are already underway:
Leaderboards are splitting. Artificial Analysis now benchmarks model+harness pairs, not models alone. Expect every serious benchmark to follow. The single-model leaderboard is going the way of single-core CPU benchmarks — technically still published, practically useless for purchase decisions.
Harness engineering is becoming a job title. GitHub Copilot Day coined “loop engineering” and “harness engineering” as distinct specializations. The awesome-harness-engineering repo on GitHub tracks the emerging discipline. This is not hype — it is the logical consequence of the moat moving from model to infrastructure.
Eval costs are the next bottleneck. When running one leaderboard costs $40,000 and statistical credibility costs $320,000, evaluation itself becomes gated infrastructure. HuggingFace argues that evals are becoming the new compute bottleneck. The teams that can afford to evaluate rigorously will ship better agents — not because they have better models, but because they measured accurately.
The 17x cost spread is not a curiosity. It is the measurement layer’s way of telling you that the question you are asking — “which model is best?” — is the wrong question. The right question is: which harness makes the model you already have perform like the model you wish you had?
The agent benchmarking landscape is moving fast. Follow AgentConn for ongoing analysis of what the numbers actually mean.