ZeroNoise Logo zeronoise
Post
Verification is the bottleneck: Claude Code's 90% self-authorship, Huntley's Antithesis loops, and LangChain's agent cost controls
•
6 min read
• 122 docs
Anthropic credits verification for letting Claude write about 90% of Claude Code's code, and Huntley is putting Overalls through two weeks of autonomous verification. Also covered: LangChain's method for controlling coding-agent spend, Huntley's composable inference proxies, credential hygiene, and smaller tool updates.

Verification, not review, is what lets agents write most of the code

Anthropic says Claude writes about 90% of Claude Code's code. From June to September, merged PRs more than doubled while the share of PRs reverted fell by more than 40% . Addy Osmani puts this down to one practice: give the agent a way to verify its work. When code is produced faster than people can review it, the bottleneck becomes proving that it works .

His recipe:

  • Replace the single reviewer with layers. One person reading line by line can no longer be the gate. Use a multi-layered system with fast, deterministic feedback loops for agents: end-to-end tests, feature maps (which poteto advocates), property tests and multi-agent adversarial reviews .
  • Move humans to judgement. People should focus on product intent, assessing risk and weighing evidence. Leave code checking to automated tests, scale approvals to risk, and turn test failures into living evaluations .

Anthropic's write-up is here . One caution from the field: in a Reddit thread where a developer said Opus 5.5 cut their tickets to 1–2 hours a day, a commenter said manual review on their team still catches "significant issues, including data loss and security issues" .

Geoffrey Huntley is testing the same idea under stress. His starting point: just because an application compiles doesn't mean it works. Overalls has gone into Antithesis's "torture chamber", where he will run autonomous discovery-and-rectification loops for two weeks until his New York talk. He wants to see what classes of problems turn up and whether agents with "the right level of back pressure" can fix failing properties on their own . He also flagged a slide deck on formal verification of LLM-generated code .

He also proposed using thinking traces to check whether a skill or AGENTS.md is working: whether the agent considers the skill, and when it decides not to use it. He notes this isn't possible with Claude .

Controlling coding-agent spend: LangChain's three steps

Harrison Chase guesses LangChain spends roughly 30% of its engineering salaries on coding tools. That share has fallen over the last two months even though token use went up . His steps:

  1. Visibility. Log all coding-agent usage to LangSmith so you can see who spends what, and on what .
  2. Basic controls. Set budgets with a way to escalate: the VP of engineering almost always raises the limit, but the budget guards against agents stuck in bad loops .
  3. Own the harness. Their in-house cloud agent, which Chase starts from Slack, has prompt caching and model routing built in . Providers now let you add tools mid-run, which used to break the cache .

The example that justifies tracking: a plugin sent a DOM snapshot back on every agent turn during frontend work, and that one loop cost about $40k in a week .

Prompt caching is also behind Theo's claim that Gemini is about 50% more expensive than it should be. His test showed a 27% cache-miss rate , using the same basic agentic-work test on all three APIs . A respondent said their own miss rates are much lower . Theo says the test is about 10 lines of code and published his methodology , so you can check your own workload.

Huntley's composable inference proxies

Huntley has lost time to "model is at capacity" errors that start when Europe wakes up, so he released three pieces of software that can be used separately or together :

  • Underclass pools ChatGPT subscriptions behind a proxy, claims usage resets automatically and handles the prompt cache key so usage isn't wasted. He calls it stable, runs it with six ChatGPT 500 accounts, and it's on GitHub and available as a Nix Flake .
  • Preflight is a local proxy that scans requests and attachments and redacts secrets before they reach the provider. Point Underclass at Preflight to use both .
  • geoclass is an experiment. Tag VMs around the world on a Tailscale network, point Underclass at geoclass, and requests get routed across regions. "In theory" the rest just works .

Early results are anecdotal. Claude found no header in the Codex source or OpenAI docs that pins requests to a region . After routing out of Germany via Australia, Huntley says OpenAI inference felt much better, but he hasn't measured it and it was a Saturday afternoon .

Keep secrets where agents can use them but not read them

A user blamed a $90,000 Google bill on a leaked key, and said Opus 5.5 found that Codex had published their .env to Cloudflare Pages. They say Codex itself flagged the chat as a cybersecurity risk . This is their account and hasn't been verified. Kent C. Dodds said he feels safer because he uses cloud agents and keeps secrets and OAuth tokens where agents can use them but not retrieve them . He also restricts tokens to specific actions, for example Gmail drafts only . Preflight, above, is another way to do this.

The opposite complaint also surfaced: auto mode in dots and Claude Projects refuses or misclassifies basic tasks, such as merging PRs into a repo set up for that purpose. This blocks work overnight or demands approvals you can't give remotely . Riley Brown agreed.

Smaller items

  • Claude Code latency: it now gzips the conversation it sends each turn, and cloud sessions start up as soon as you begin typing . Theo says responses now start almost immediately, and the difference is most noticeable when steering, because you can watch the model correct itself .
  • cc-stretch is an MIT-licensed Claude Code plugin. Replies start compressed and you expand sections as needed (it claims 83% fewer words read), and it tells Claude which sections you expanded. Install with /plugin install cc-stretch --marketplace pro-vi/cc-stretch and run /tui fullscreen .
  • Consistent harness runs: OpenHands looks stable mainly because it hardcodes temperature 0. Normalizing tool outputs fixes much of the variance in other harnesses .
  • Haiku 5.5 costs $0.10/$0.50 for prompts up to 100K tokens and is reported at 39.2% on Terminal-Bench 4.0 at max effort .
  • Linus Torvalds used an LLM to refactor hard-to-follow associative-array code into smaller functions. Userspace tests confirmed the reported bug and found three more, all dating to 2013, according to an Italian video recap . In the same video, Salvatore Sanfilippo says AI helped him write Redis patches that cut RAM use on large sorted sets by 50% .
  • Email as an agent channel: Ben Tossell's bot opens BB app threads for coding work, emails updates, and updates a shared task list and work log . He says email threads work well for agent chats .
  • Riley Brown edited a recorded video in one prompt with Opus 5.5, Blender and a skill he had worked on for a day. He rates it about 3% as good as it could be .
  • tsc-ts from Vercel Labs ports TypeScript back from Go to TypeScript, compiles it with scriptc, and claims 2–4x faster than tsgo . Theo says it beats his tsc-rs port in most things .
Verification is the bottleneck: Claude Code's 90% self-authorship, Huntley's Antithesis loops, and LangChain's agent cost controls
Salvatore Sanfilippo
Profile
  • Linus Torvalds used Claude Code to draft a Linux kernel feature. For a hard-to-follow associative-array library bug, he used an LLM to understand and refactor complex code into smaller functions; userspace tests confirmed the reported bug and found three more, all dating to the library’s 2013 introduction.
  • In his Redis work, Salvatore Sanfilippo says he used AI as leverage to develop memory-saving patches for large sets. He reports a 50% RAM reduction, outperforming both Valkey and a manually written Redis patch.
Linus Torvalds promuove l'AI sempre più
Harrison Chase
Profile
  • Harrison Chase estimated LangChain’s coding-tool spend at roughly 30% of engineering salaries—a guesstimate—and said that share had fallen over the prior two months even as token use rose. His cost-control sequence: track usage in LangSmith, set budgets with an escalation path, then improve the harness.
  • LangChain’s in-house cloud coding agent, “open suite,” can be launched from Slack; engineers can still choose their own IDE or terminal-based agents. The custom harness uses prompt caching and model routing, and dynamically adding tools mid-run can avoid cache breaks and added costs.
  • Usage tracking surfaced a costly failure: a plugin repeatedly sent DOM snapshots on every agent turn, creating a loop that cost $40,000 in a week. Chase described catching this kind of unintended usage as worthwhile.
The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO
Geoffrey Huntley
  • Huntley uses Underclass, a proxy that pools ChatGPT subscriptions, automatically claims resets, and handles prompt-cache keys. He describes it as stable, says he runs six ChatGPT 500 accounts with no real upper limit on configured accounts, and notes it is available on GitHub and as a Nix Flake.
  • For secret protection, run one Preflight local LLM proxy and one Underclass instance, then point Underclass to Preflight; Preflight scans requests and attachments and scrubs or redacts secrets before sending them for inference.
  • After losing agent time to inference-capacity problems—including performance drops as Europe woke up while he worked from Germany—Huntley released GeoClass as an experimental regional-routing prototype. The proposed setup uses tagged VMs in a Tailscale network and points Underclass to GeoClass; he says it remains uncertain whether this will solve his inference problems.
pooling OpenAI subscriptions and bypassing regional inference capacity limits
Latent.Space
  • Claude reportedly writes about 90% of Claude Code’s code; merged PRs doubled from June to September while the reverted share fell by more than 40%. Claude Code also now gzips conversation payloads on every turn to reduce latency.
  • In one developer’s account, Opus 5.5 reduced assigned tickets and tech-debt work to 1–2 hours per day; a commenter cautioned that human review remains important, citing significant data-loss and security issues caught by their team.
  • The MIT-licensed cc-stretch Claude Code plugin claims to reduce words read by 83%: replies start compressed, users expand or shrink sections interactively, and expansion behavior is fed back to Claude. Install with /plugin install cc-stretch --marketplace pro-vi/cc-stretch, then use /tui fullscreen.
  • For more consistent agent runs, the roundup says OpenHands’ stability is largely attributable to hard-coded temperature 0; normalizing tool outputs fixes much of the variance in other harnesses.
  • Cursor’s Environments API supports environments with up to 500 repositories and reportedly makes startup for large environments 5× faster.
[AINews] High Taste Testers Ep. 1
Ben Tossell

Ben Tossell says Pi with Opus 5.5 emails a bot and that email threads work well for agent chats; he also notes that agents email politely.

agents email politely btw pi with opus 5.5 emailing [@bot](https://x.com/bot) email threads are great for agent chats ![](https://pbs.twi…
Addy Osmani
  • Addy Osmani’s key workflow advice is to give coding agents a way to verify their work; he says this can substantially improve output quality. He reports Claude Code wrote 90%+ of its code itself with this process, while Anthropic reported that merged PRs more than doubled from June to September and the share reverted fell by more than 40%.
  • Replace line-by-line human review as the sole gate with layered verification and fast feedback loops: end-to-end tests, feature maps, property tests, and multi-agent adversarial reviews. Humans should focus on product intent, risk, and evidence; automate code checks, scale approvals to risk, and turn test failures into living evaluations.
Give your agent a way to verify its work. This can dramatically improve the quality of output. With this process in place, Claude Code wa… Claude writes about 90% of Claude Code's code. From June to September, merged PRs more than doubled while the share reverted fell by more…
Ben Tossell

Ben’s bot opens threads in get_bb_app for “proper” coding work; thread activity emails bot updates and updates a shared task list and work log, tying coding conversations to project tracking. He is assembling a Linear-style Superhuman-and-todos all-in-one, with email built on Executor; the bot can be tagged in comments, and the hub links local repositories and tracks domains, expiry dates, and hosts.

spent all day on my agent setup and paying off [@bot](https://x.com/bot) now opens threads in [@get_bb_app](https://x.com/get_bb_app) for… vibe-slopped superhuman + todos, linear style my [@bot](https://x.com/bot) uses this too, I can tag it in comments email built on [@RhysS…
Kent C. Dodds 🐨

Kent C. Dodds says he feels safer using cloud agents and storing secrets/OAuth tokens somewhere agents can use them but cannot retrieve them; he also limits tokens to intended actions, such as allowing Gmail drafts only . This advice came in response to a user’s report of a $90,000 Google bill and claim that Codex published their .env to Cloudflare Pages; that account is an allegation in the quoted post, not independently verified here .

This stinks. I feel much safer about this since I started to 1. Use cloud agents 2. Stored my secrets/oauth tokens somewhere agents can u… Oh, and: 3. Made it so agents can ONLY use my tokens the way I want them to (for example [https://kody.codes/docs/locked-gmail-drafts](ht… FINALLY found how my key got leaked and got the $90,000 bill from google Opus 5.5 said codex had PUBLISHED MY \*\*\*\*ING .ENV TO CLOUDFA…
ThePrimeagen

Claude Code now gzips the conversation sent on every turn so it reaches the model faster, and cloud sessions start up as soon as the user begins typing; the announcement gives no quantified performance gain. ThePrimeagen called the change “rediscovering good practices from first principles.”

We keep improving performance across Claude, and you may have noticed it starts responding sooner! Claude Code now gzips the conversation… rediscovering good practices from first principles impressive [https://x.com/ClaudeDevs/status/2111209208012210409](https://x.com/ClaudeD…
geoff

For a large evidence set, the post proposes OCRing everything, running loops over the material, then additional loops to verify summaries and requesting citations to specific pages . It presents this as a plausible alternative to manual work by a lawyer and paralegal team, not as a reported experiment .

so there’s this scene in the netflix series called inventing anna. the context is the defence lawyer asked the prosecutor for all of the …
geoff

Geoffrey Huntley says Mic92 has already built the Nix tooling he needs, pointing to tribuchet and nix-grpc-store, and describes the idea as “rbe but for nix.”

everytime i need something for nix mic92 already built it 1. [https://github.com/Mic92/tribuchet](https://github.com/Mic92/tribuchet) 2. …
Theo - t3.gg

Theo reported that a basic cache test simulating normal agentic work found Gemini had a 27% cache-miss rate, which he said made it 50% more expensive than expected; he said the same test was run across three APIs . A respondent said the miss rate was very low compared with their workloads, a caveat to generalizing the result. Theo said the test was about 10 lines of code to recreate and shared an agent-written methodology page to clarify it .

Gemini is 50% more expensive than it should be because their cache doesn't work. 27% cache miss rate. Insane. ![](https://pbs.twimg.com/m… [@_mohansolo](https://x.com/_mohansolo) This was a very basic cache test simulating normal agentic work. Exact same test for all 3 APIs. [@theo](https://x.com/theo) Not what we're seeing on our workloads, that is very low. What are you running here? [@_philschmid](https://x.com/_philschmid) [@_mohansolo](https://x.com/_mohansolo) This should be very very simple to recreate. It was lik…
geoff

Geoffrey Huntley described connecting @bot to iPhone mirroring so it could handle an incorrect Uber Eats delivery issue—a practical example of delegating a real-world task through a mirrored phone interface.

new fun trick with [@bot](https://x.com/bot). connect it to iphone mirroring and have it sort out an incorrect uber eats delivery drama. …
geoff

Geoffrey Huntley says OpenAI inference has felt “so much better” since he routed it from Germany via Australia, but describes this as unmeasured and flags the time of day as a possible factor; he gives no setup details or coding-agent-specific results.

idk, perhaps because it's 2pm in london on a saturday right now but on complete vibes (unmeasured for now) inferencing for openai has bee…
geoff

Geoffrey Huntley is testing Overalls in @antithesishq’s “torture chamber,” running autonomous discovery-and-rectification loops for two weeks to explore verifying large amounts of code and having agents auto-resolve failing properties with the right level of back pressure. His practical warning: an application compiling does not mean it works.

Here's the fun thing, just because an applications compiles doesn't mean it actually works. In other news, [http://overalls.dev](http://o…
geoff

Huntley is testing overalls.dev in Antithesis, planning two weeks of autonomous discovery-and-rectification loops ahead of a New York talk to explore verifying large amounts of code; his premise is that successful compilation does not establish that an application works. He is investigating whether agents given sufficient “back pressure” can automatically resolve failing properties; the post mentions a 15-minute run but provides no textual findings from it.

Here's the fun thing, just because an applications compiles doesn't mean it actually works. In other news, [http://overalls.dev](http://o…
geoff

Geoffrey Huntley praised @namespacelabs for the “blazing-fast CI” powering overalls.dev, calling it a “game changer.”

Big shoutout to [@namespacelabs](https://x.com/namespacelabs) for the blazing-fast CI powering [https://overalls.dev](https://overalls.de…
geoff

Geoff proposed using agent thinking traces to assess skill effectiveness, including whether a skill is considered and when the agent decides not to use it, and to evaluate an agent’s MD; he said this was not possible with Anthropic or Claude and planned to explore it further.

just had a random idea. Has anyone ever used the thinking traces to evaluate whether a skill is being effective? Whether it's coming in f…
geoff

Geoffrey Huntley announced geoclass, an experimental router for requests to different inference regions, framing it as a way to pool OpenAI subscriptions and bypass regional inference-capacity limits. To try it, deploy virtual machines in different regions on a Tailscale network, tag them, and point underclass to geoclass; Huntley caveated that “the rest just works” only “in theory.”

🗞️ pooling OpenAI subscriptions and bypassing regional inference capacity limits [http://ghuntley.com/underclass](http://ghuntley.com/unde…
geoff

Geoffrey Huntley is testing whether OpenAI inference-capacity errors are region-specific: after seeing errors from Europe in his own usage over the previous 30 days, he plans to route Underclass through Australia instead of Germany; he has not reported the test result yet. He also floated a global geographic router that follows the sun and routes requests to the region with the least usage, such as where users are sleeping.

this is going to be interesting. I've got the last 30 days of inference capacity errors for OpenAI from the Europe region, from my own pe… like how cooked would it be to have a geographic request router running in different regions around the world, following the sun and rout…