AGENTCONN

Field report · · AgentConn Team

Stop Fine-Tuning. Engineer the Harness.

Clay, Harness-Bench, and GitHub trending all say the same thing: model weights aren't your bottleneck. The harness is.

AI AgentsAgent HarnessFine-TuningAgent InfrastructureHarness EngineeringDeveloper Tools2026

Stop Fine-Tuning. Engineer the Harness.

A glowing forge anvil with streams of code flowing through tooling infrastructure around a central AI model core, symbolizing harness engineering over model fine-tuning

Here is the most expensive lesson of 2026: the model is not your bottleneck. Clay spent months post-training a custom model and found it barely matched what Sonnet and Opus could do out of the box. The Harness-Bench paper ran the same model across different harnesses and watched scores swing from 52.4 to 76.2 — a 45% gap from tooling alone. Meanwhile, 7 of the top 15 repos on GitHub trending are agent infrastructure, not model weights.

The moat moved. If you are still spending engineering hours on fine-tuning before you have exhausted your harness improvements, you are optimizing the wrong layer.

Naval Ravikant tweet: From vibe code to verified code... — 3.3K likes, 262 retweets

View original post on X →

Naval’s two-word arc — “vibe code to verified code” — captures the shift in five syllables. The industry moved past prompting tricks. The engineers who are shipping reliable agents in November 2026 are building verification loops, tool infrastructure, and execution harnesses. Not training runs.

The Evidence: Three Data Points That Closed the Debate

1. Clay: Post-Training Hit Diminishing Returns

Jeff Barg, Head of AI at Clay, told LangChain that the company post-trained a Qwen 3.6 35B model as a router for their agent system. The goal was to match frontier-model quality at lower cost. The result: auto-mode cost 47% less than routing every task to the most capable model — but the quality gains came from the routing harness, not the custom weights. Clay’s real innovation was building a harness around the Vercel AI SDK integrated with LangSmith for observability, running 300 million agent invocations per month.

The lesson is not that post-training failed. It is that post-training solved the wrong problem. Clay’s agent harness — the tool definitions, the eval loops, the routing logic — delivered the production reliability that weight changes could not.

2. Harness-Bench: Same Model, 45% Score Swing

The Harness-Bench paper from Qihoo360 introduced 106 realistic end-to-end agent tasks, each run in a sandboxed environment. The critical finding: holding the model fixed while changing only the harness, scores ranged from 52.4 to 76.2. A related position paper (arXiv:2605.23950) found that on SWE-bench Pro, Claude Opus 4.5 reached 45.9% under a standardized scaffold but 55.4% under Claude Code — same weights, different harness, 20% improvement.

This is not a marginal effect. Cross-harness variance is larger than cross-model variance at the frontier. If you are benchmarking models without controlling for the harness, your numbers are noise.

Harness-Bench by the Numbers

  • 106 realistic agent tasks in sandboxed environments
  • 52.4 to 76.2 — score range for the same model across different harnesses
  • 45.9% vs 55.4% — Claude Opus 4.5 under standardized scaffold vs Claude Code on SWE-bench Pro
  • 0.86 vs 0.87 — Nemotron 3 Ultra (harness-tuned) vs Opus 4.8 on Deep Agents suite, at ~10x lower cost

3. LangChain’s Nemotron Playbook: 99% of Frontier at 10% of Cost

LangChain published a case study showing that harness tuning alone took Nemotron 3 Ultra to 0.86 on the Deep Agents suite — nearly matching Opus 4.8’s best of 0.87. The cost difference: roughly 10x lower per run. No weight changes. No training cluster. Just systematic improvements to prompts, tool descriptions, and middleware.

The case study also identified the ceiling: when harness changes stop moving the score, the remaining failures trace to capability gaps in the weights. That is when post-training earns its keep. But for most teams, that plateau is months of harness work away.

Today’s GitHub trending page reads like a venture capital thesis on agent infrastructure. Of the top 15 repositories, 7 are agent harness and orchestration tools:

RepoStarsWhat It Does
superpowers322KAgentic skills framework — the most-starred agent repo in history
firecrawl204KKnowledge infrastructure for agent retrieval
CLI-Anything60KMakes ALL software agent-native via CLI wrappers
diagram-design69KEditorial diagram design for coding agents
paper-orchestra5KGoogle’s multi-agent orchestration framework
LoopForge4KTencent’s self-hosted coding-agent delivery workflows
opencode228KOpen-source coding agent harness

Not a single fine-tuning repo in the top 15. Not one LoRA adapter. Not one training framework. The open-source community is voting with commits: the harness is the product.

This is not a November anomaly. We tracked the same pattern in our October survey: 12 of 15 trending repos were agent infrastructure. The software factory thesis we published yesterday documents what happens when companies build on this foundation — Cognition merged 40x more PRs with 2.7x the engineers.

Theo’s Signal: When the Best Engineers Stop Typing

Meanwhile, the practitioners are speaking with their keyboards — by not using them. Theo (t3.gg) argued this week that AI writes over 90% of his code, calling Anthropic’s own “90% AI-written” claim conservative. The key data point: Lauren Tan, former React core team member, now builds exclusively with agents. When one of the best human frontend engineers in the world voluntarily stops typing, the question is not whether agents work. The question is what “engineering” means when you spend 100% of your time on the system around the agent.

The answer, increasingly, is harness engineering. Defining tools. Writing evals. Building verification loops. Designing the constraints that turn a probabilistic model into a reliable system.

Fabian Hedin, CTO of Lovable, reinforced this at the Code Summit: Lovable nearly failed after building a sophisticated multi-agent framework. What saved them was stripping it back to durable agent runtimes with Git-inspired execution — simpler harness, better results. The lesson is as old as engineering: the complex solution is rarely the right one. The harness that works is the one you can reason about.

The Contrarian Corner: When Fine-Tuning Actually Works

The “never fine-tune” absolutism is as wrong as the “always fine-tune” default it replaced.

Clement Delangue dropped a data point this week that complicates the narrative. HuggingFace adapted Qwen3.8-27B specifically for the pi coding agent using reinforcement learning and a LoRA adapter. The results:

  • SWE-bench Pro (Hard 51): 29.4% → 62.7% — more than doubled
  • Terminal-Bench 2.1: 39.5% → 55.8%

Clement Delangue tweet announcing Qwen3.8-27B-pi-adapted with RL and LoRA — SWE-bench Pro jumping from 29.4% to 62.7%

View original post on X →

That is not a marginal improvement. That is a capability unlock. So when does fine-tuning earn its keep?

The pattern from every 2026 case study is consistent:

  1. The task must be narrow and well-defined (SWE-bench, not “general coding”)
  2. The training data must come from the target harness (HuggingFace trained on pi-agent trajectories, not generic code)
  3. The harness must already be optimized (the 29.4% baseline was already the pi harness; fine-tuning improved within that system)

Fine-tuning works when it is the last mile, not the first. The Arize analysis put it precisely: iteration has split. A tiny fraction run continuous RL against their own environments (Cursor, Cognition). Everyone else moved the iteration loop out of the model and into the harness. OpenAI’s decision to wind down its self-serve fine-tuning platform in May 2026 did not cause this split — but it made it impossible to ignore.

The Decision Framework: Where to Spend Your First 100 Hours

If you are building an agent system today, here is the empirical priority order. This is not theory — it is the sequence that Clay, Lovable, and the Nemotron case study all converged on independently:

Hour 0-20: Retrieval and Context

The highest-leverage harness work is getting the right information to the model at the right time. Addy Osmani’s harness engineering guide recommends keeping tool output under 2,000 lines, offloading the rest to the filesystem. HumanLayer keeps its AGENTS.md under 60 lines. The rule: ten focused tools outperform fifty overlapping ones.

Concrete steps:

  • Audit your tool definitions. Remove overlap. Sharpen descriptions.
  • Implement context windowing — load skills progressively, not all at startup.
  • Add filesystem offload for large tool outputs.

Hour 20-50: Verification Loops

The gap between demo and production is verification. Naval’s “vibe code to verified code” is not poetry — it is architecture. Every agent action should produce evidence that it succeeded.

Concrete steps:

  • Add post-edit hooks: typecheck, lint, test after every code change.
  • Implement a separate evaluator — agents grade their own work too generously.
  • Build rollback capability via Git branching for every agent experiment.

Hour 50-80: Tool Design and Execution Sandboxes

The agent can only be as good as the tools you give it. Lovable’s near-failure came from complex multi-agent orchestration; their recovery came from durable runtimes with simple tool interfaces.

Concrete steps:

  • Sandbox all code execution (allow-lists, network isolation, disposable environments).
  • Design tools that are self-documenting — the tool description enters the prompt.
  • Vet MCP servers carefully; a sloppy tool description is a prompt injection vector.

Hour 80-100: Observability and Evals

You cannot improve what you cannot measure. Clay integrated LangSmith for observability across 300 million monthly invocations. The pattern is clear: traces → failed-span inspection → targeted harness changes is the iteration loop that ships.

Concrete steps:

  • Instrument every agent run with structured traces.
  • Build domain-specific evals — not benchmarks, your benchmarks.
  • Feed production failures back into harness rules (the “ratchet” principle from Osmani).

Hour 100+: Consider Fine-Tuning (Maybe)

Only after the harness work plateaus — and you have the traces to prove it — should you consider post-training. When you do:

  • Train on trajectories from your target harness, not generic data
  • Start with LoRA on an open-weight model (cost: hundreds, not thousands)
  • Measure against your eval suite, not public benchmarks

The Diagnostic Checklist

Before spending a dollar on fine-tuning, answer these:

  1. Have you audited your tool definitions in the last 30 days?
  2. Do you have post-action verification hooks on every agent step?
  3. Can you trace a failure to the specific tool call that caused it?
  4. Is your context window under 60% utilization at task start?
  5. Do you have domain-specific evals that run on every PR?

If any answer is “no,” your bottleneck is the harness, not the model.

What This Means for Builders

The fine-tuning-vs-harness debate is over. The empirical evidence from 2026 is unambiguous:

  • Harness variance > model variance at the frontier (Harness-Bench)
  • Harness tuning closes 99% of the gap at 10% of the cost (Nemotron)
  • Capital allocation follows the harness (GitHub trending, 7 of 15 repos)
  • The best engineers are becoming harness engineers (Theo, Lauren Tan, Clay)

The teams that figured this out in Q1 — Clay, Cognition, Lovable — are six months ahead. The teams still budgeting for training runs are optimizing the wrong variable.

The harness is the product. The model is a dependency. Build accordingly.

Hacker News discussion thread debating whether harness engineering is real engineering

View on Hacker News →

The debate on HN is telling. Some practitioners report concrete wins from harness tuning — reduced tool count, better routing — while others question whether “harness engineering” is engineering or empirical tinkering. Both camps agree on one thing: the model is not where they spend most of their time anymore.

Hacker News thread: Building agents without harness engineering — 16 comments debating agent development approaches

View on Hacker News →

Arize blog post: The end of fine-tuning — Why evals, context, and traces matter more

Read the full analysis on Arize →


The harness-first thesis is the foundation AgentConn was built on. For the temporal survey data behind this shift, see 80% Use Agents Daily, 49% Ship Them to Prod. For the organizational implications, see The Software Factory Is Here. For the technical deep-dive on what makes a harness production-grade, see The Harness Is the Moat.

The AgentConn Weekly

Weekly digest of new AI agent releases, framework comparisons, and deployment guides. Built for builders.

Weekly. Unsubscribe anytime.

Explore AI Agents

Discover the best AI agents for your workflow in our directory.

Browse Directory