ZeroNoise Logo zeronoise
Post
Claude Code team's verification playbook, and practitioners split on GPT-6.2 Astra
•
5 min read
• 212 docs
How the Claude Code team verifies agent-written code: a repo-local verify skill, a three-round stop rule, and checks that trigger on the diff. Plus a backlash from early testers against GPT-6.2 Astra, Sol's quieter reception, and new tools for coordinating agents from Cursor and Cognition.

The Claude Code team's verification rules

In an article Addy Osmani shared, the Claude Code team says about 90% of its code is written and reviewed by Claude. It argues that without verification loops you can supervise only one or two agents, because you check everything yourself, and with them you can run "dozens or hundreds in parallel" . The practices:

  • A verify skill. Put it in .claude/skills/, as close to the relevant code as possible. It should boot and drive the real app: the Claude Chrome extension for web, an iOS/Android MCP for mobile, computer use for desktop, or a local service for backend. Claude uses it before committing. The team treats it as the smoke-test harness and asks Claude to stress-test it until it is fast and reliable .
  • Test types worth investing in: end-to-end first, because they're "difficult to fool" . Then property tests with fast-check, Hypothesis or proptest, checking round-trips, invariants, idempotency and equivalence . For migrations, differential tests run the old and new systems on random or sampled production inputs and keep a list of approved differences . Flaky tests teach the agent to rerun rather than fix .
  • Three-round stop rule. If a check still fails after three fix attempts, Claude stops and brings in a human .
  • Checks that trigger on the diff. A 39-item reliability checklist records, for each item, a trigger readable from a diff, a check, and the incidents or PRs behind it. Claude runs only the matching items, usually 3–10 .
  • Gates the change can't edit. REVIEW.md is read from the base branch, not from the PR . Tests, build and linter run before a PR opens, enforced with a hook or a /goal . To stop repeat failures, add a hook or a CLAUDE.md rule .

Two other items this period support the property-testing advice. In a paper shared by @eulerfx covering 25 agent-built apps, properties derived from the spec found 56 bugs that properties derived from the code missed, and the code-derived properties found 43 the spec-derived ones missed. Spec-derived properties mostly caught state, lifecycle and logic errors; code-derived ones mostly caught input handling . Caveats: the bugs were judged by the same model that wrote the tests, the prompts differed between the two approaches, and the method needs a good spec . In a Bend video, ThePrimeagen recommends writing the laws yourself and having the agent write the proofs. He notes that agents often change unit tests along with the code, which "doesn't give me confidence." His suggested target is a job-ticket state machine .

GPT-6.2 Astra and Sol: cheaper Astra, mixed reviews

The models are in the API as gpt-6.2-astra and gpt-6.2-sol. Astra costs $8/$40 per million input/output tokens, and cached input drops to $0.40 from $1, which Simon Willison calls good news for long agentic sessions. Sol stays at $2/$10 . Willison supports both in LLM 0.38 , and Armin Ronacher says Pi users can get them with pi update --models . On Willison's Three.js/OpenStreetMap mech test at Ultra effort in Codex CLI, neither model matched Claude Fable 5.2 or Gemini 4 Carbon .

Theo, who tested Astra early, would still pick Opus 5.5. He credits Astra with strong review, 3D/game work, computer use and subagent prompts, plus a reverse-proxy design he says will save $25k a month . His failure reports are useful for anyone running long jobs:

  • It blocks itself on questions you can't answer. Two hours into a /goal, it stopped to ask about a reviewer's clip filenames. He saw the same in Codex and now runs agents on crons to check whether it has blocked again .
  • Its summaries are hard to read. He switches to Opus 5.5 at the end of a long run just to get a readable summary . He also uses Opus as an "auditor and translator" for Astra through a T3 Code feature .
  • It runs without progress. On his ts-rust benchmark, Opus finished in under two weeks for about $20k in tokens. Astra ran seven days, landed no improvements and made the compiler 56% slower, then ran four more days with no changes after he redirected it .

Other views: Tibo (OpenAI) says early testers' videos predate "a lot of changes in the last 3-4 days." He recommends Sol ultrafast, now on Pro 200 plans as well as Pro 500 . Theo has heard, secondhand, that many OpenAI engineers use Sol Ultrafast rather than Astra . swyx disagrees with the negative reviews: he calls Astra the best workflow-automation model he has used, "no longer getting stuck in horrible loops," and says it runs much of Latent Space's publishing pipeline .

Coordinators and persistent agents

Cursor Projects. You talk to one coordinator, which starts workers, for example on the data layer, the API and the UI. When one worker changes something like a schema, the coordinator pushes the change to the others . In multiplayer projects, coordinators message each other to split work and avoid conflicts. Cursor says this makes it 5–10x more productive internally . The speaker's setup advice:

  • Connect docs, the task tracker, Slack and monitoring data .
  • Require a demo video in the PR for UI changes, and have the agent watch CI and fix Bugbot and security-review comments .
  • Configure a cloud environment so the agent can actually use what it built. The speaker calls this the single biggest lever on dev velocity .
  • Route reviews by risk: schema changes need a human, flagged UI changes may not .
  • Projects work best with 3–8 people and a medium-term scope .

Devin Bots give each project its own persistent identity and memory, replacing short-lived sessions . Cognition's Steven K says this means you should delegate "goals and responsibilities," not tasks. He adds that the change was "almost entirely an orchestration change" that could probably have been built a year ago . After a week of trials, swyx handed over QA, cross-platform app porting and video editing. His advice is to expect agent managers that own outcomes, not "AI Employees" .

Also notable

  • Claude Code Code Mode (experimental): Claude writes scripts that chain your tools together and can reply while long commands are still running. Opt in with CLAUDE_CODE_EXPERIMENTAL_CODE_MODE=1 .
  • Sandbox placement: LangChain says calling the sandbox as a tool lets you update the agent without rebuilding an image and keeps API keys outside the sandbox. The cost is network latency . Only microVM-style kernel isolation stops a compromised agent from exploiting kernel bugs .
Claude Code team's verification playbook, and practitioners split on GPT-6.2 Astra
Simon Willison’s Newsletter
  • Prototype-to-production: Simon used Claude Opus 4.8 to generate a May research spike for Datasette Lite, but says he shipped it only in November, after becoming familiar enough with the system while working with Claude Fable 5.2.
  • Prompt pattern: Simon built most of the Datasette MapLibre plugin with one Claude Fable 5.2 prompt: inspect two existing plugins and a local demo, combine their features, and brainstorm beyond them. The creative instruction produced SVG previews of GeoJSON shapes, with WebGL previews loaded only when scrolled into view.
  • Agent-code sandboxing: The external-sandbox pattern keeps the coding agent outside a dedicated sandbox where its code runs. Three implementations use v86 with Alpine Linux in WebAssembly, Alpine inside Apple Containers, or Microsoft Quicksand/QEMU; all expose the same localhost API for Pi, Claude Code, and Codex. Optional HTTP/HTTPS allow-listing and credential replacement control network access and let code make authenticated calls without giving the agent the secret.
  • Reported capability: Anthropic co-founder Tom Brown said the Mythos model brought Claude up on AMD in three days with almost no human intervention, porting or writing nearly all the kernels; he contrasted this with previous chip-platform onboarding that took dozens of engineers months.
  • Model caveat: Assessments of GPT-6.2 Astra were mixed: Theo Browne said it could coordinate subagents on long tasks but stalled two hours into an Outer Wilds level task to ask about an obscure filename choice. In Simon’s Codex CLI Three.js mech-animation test, Astra and Sol also did not match Claude Fable 5.2 or Gemini 4 Carbon.
Running the full server-side Datasette Python app in the browser with service workers and Pyodide
Addy Osmani
  • The Claude Code team says about 90% of its code is written and reviewed by Claude; it argues evidence-based verification loops make it possible to delegate from one or two agents to dozens or hundreds. In internal structured-file-generation tests, a separate rubric-based grader improved task success by up to 10 points over standard prompting, with the largest gains on harder tasks.
  • Prioritize end-to-end checks of real user flows: they provide clearer evidence that a feature works than unit tests alone. Give the agent access to run and inspect the real app (for example, through the Claude Chrome extension, mobile MCP, desktop computer use, or a local backend), and create a fast, reliable repo-local verify skill under .claude/skills/ near the relevant code for it to use before committing.
  • For edge cases, use property tests to check invariants such as round-trip behavior, idempotency, or equivalence, with libraries such as fast-check, Hypothesis, or proptest. For migrations where behavior should be preserved, compare old and new systems on the same randomly generated or sampled production inputs; normalize irrelevant differences and document approved changes.
  • Have Claude run tests, the build, and the linter before opening a PR; if checks still fail after three repair rounds, stop and involve a human. The team also uses a 39-item reliability checklist whose diff-based triggers usually select 3–10 checks, and keeps review rules on the base/default branch so the change being reviewed cannot modify its own gate.
How the Claude Code team thinks about verification
Simon Willison
Profile
  • Willison dates the coding-agent inflection to November 2025, when Claude Opus 4.5 and GPT-5.1 became reliable enough to use daily rather than as occasional novelties.
  • To locate an intermittent beep in a room of about 40 boxes, he gave an agent access to two iPads, two laptops, and a phone as web-audio microphones connected over WebSockets. The agent found a 2007 algorithm for unreliable microphones, built a beep locator, and pinpointed the source within 41 cm; it was an old carbon monoxide detector.
  • When his command-and-conquer-style game felt dull, he asked Opus 5.5 to make it more fun. Claude ran for 12 hours, simulated 1,000 battles, and added objectives, explosion feedback, and achievements; Willison said the resulting game was fun.
  • Willison describes Opus 5.5 as his daily driver and “cheap Fable,” saying it comes close to or sometimes beats the original Fable 5 on benchmarks at a fraction of the price.
2026 in LLMs — Simon Willison, Datasette
swyx
  • Devin Bots replace ephemeral sessions with project-specific bots that have persistent identities, goals, and memory surfaced across sessions. Teams can provide high-level project context and delegate goals or responsibilities rather than repeat task-by-task setup; the explanation attributes much of the change to memory persistence and orchestration, not model improvements alone.
  • After a week-long trial, swyx says he handed bots AEO, QA, cross-platform app porting, personal-admin automation, and video editing. He highlights always-on, ROI-tuned agents that own outcomes, manage and review subagents, work through channels such as text and Slack, and report through interactive artifacts rather than long chats; his workflow was cloud-based.
  • Swyx cautions against treating agents as AI employees or overestimating their capabilities: delegate outcome ownership at the level above individual tasks.
Bots are super cool and if you haven't used them yet, you absolutely must try them. Create a bot to manage a goal or a project that you'l… devin bots are multi-agent teams done right: - always on, proactive (but tuned for ROI) - own outcomes not tasks (but can collab w others…
Simon Willison's Weblog
  • Datasette Lite’s service-worker architecture began as an AI research spike generated with Claude Opus 4.8 in May. Willison says he waited until November, when he felt familiar enough with the system to ship it, and used Claude Fable 5.2 to generate explanatory SVG diagrams.
  • After building WebAssembly wheels for 163 SQLite versions, Willison used Claude Fable 5.2 to build a tool for comparing the same SQL query across selected SQLite versions—a concrete agent-assisted debugging tool.
  • Willison says coding agents can compile C and Rust extensions into Pyodide-compatible WebAssembly, helping remove a former blocker to running Python apps in browsers; PostgreSQL and MySQL in WebAssembly remain less-trodden.
Running the full server-side Datasette Python app in the browser with service workers and Pyodide
Simon Willison's Weblog
  • OpenAI released GPT-6.2 Astra and Sol in the API, and Simon Willison added support for both in LLM 0.38. Astra costs $8/$40 per million input/output tokens, down from GPT-6 Astra’s $10/$50; its cached-input price is $0.40 per million tokens versus $1 previously. Willison called the lower cached price good news for longer agentic sessions. Sol remains $2/$10, half the price of GPT-5.6 Sol.
  • In a hands-on comparison using Ultra effort in Codex CLI, Willison gave both models the same Three.js/OpenStreetMap task: animate Yonkers standing up as a mech and stomping around New York City. Neither handled the mech transformation as well as Claude Fable 5.2 or Gemini 4 Carbon on that prompt.
  • For a video-editing task in the Codex desktop app, Willison gave Astra two animation URLs and specified the split-screen layout, labels, square 2000×2000 output, and an initial 10-second duration, then followed up with “OK make it 20s.”
  • A caveat in Theo Browne’s pre-release feedback: Astra could manage many subagents on a long task, but then block to ask the user an obscure implementation question they could not reasonably answer; Browne said an Outer Wilds level task was blocked two hours in over a detail about reviewer output filenames.
GPT‑6.2 Astra and GPT‑6.2 Sol
ThePrimeTime
  • In a Bend game prototype, he encoded behavior as laws rather than relying only on example cases: a non-movement key must leave the game state unchanged, ground color and decorations must stay consistent as the player moves, and wall influence must follow a specified algorithm. He reports checking these properties across game inputs and states; deleting a wall made the proof fail, and reverting the change made it pass again.
  • His agent workflow is to write the laws himself and delegate writing the proofs to an agent; he stresses that formal proofs supplement, not replace, unit tests. He suggests applying this approach to job-ticket state machines with permitted states and transitions, and notes that agent-driven changes to unit tests can make test results less reassuring.
finally a language for vibecoders
Simon Willison's Weblog
  • Simon Willison and Prime Radiant introduced an external-sandbox pattern for coding agents: the agent stays outside a dedicated sandbox while its executed code runs inside, reducing the risk of damaging actions from mistakes or prompt injection. Three implementations are available: Node.js with v86 running Alpine Linux in WebAssembly; Python with Alpine Linux in Apple Containers; and Python with Microsoft Quicksand, which bundles QEMU and Alpine Linux.
  • All three expose the same localhost API for configuring Pi, Claude Code, or Codex to send code-execution requests to the sandbox. An optional HTTP/HTTPS allow-list proxy restricts which hosts code can contact, and credential replacement enables authenticated API calls without giving the agent the secret.
External sandboxes for coding agents
swyx

Swyx says Astra is his daily-driven workflow-automation model and the best he has used; he reports it avoids getting stuck in loops and learns efficiently through compactions and skill writing. He uses it in the Latent Space and AI Engineer content-publishing pipelines, and credits it as contributing to the bootstrapped business’s revenue efficiency of more than $1 million per employee.

the fact that nobody with decisionmaking power saw this coming a mile away and vetoed the 6.2 astra ad is my primary critique of the astr…
geoff

Combine spec-driven and code-driven property-based tests for agent-built apps: have the agent write plain-English invariants from the requirements, translate them into property tests after inspecting the code, and keep code-only-derived properties as a separate approach. In a study of 25 agent-built apps, each approach found bugs the other missed (56 spec-driven-only and 43 code-driven-only; the post reports 59% of bugs were found by only one approach); spec-driven tests mostly caught state, lifecycle, and logic errors, while code-driven tests mostly caught input-handling errors. The evidence has caveats: the prompts differed, the method requires a good spec, and the same model that wrote the tests judged the bugs, although 25 were manually sampled and reportedly held up.

Property-based tests generated from the spec and from the code catch different bugs. Use both. In a new paper, across 25 agent-built apps…
ThePrimeagen

OpenAI announced GPT-6.2 Astra and Sol as available, describing them as “super efficient” and built to perform in the real world; the announcement provides no coding-agent-specific capabilities or workflow details.

Some of you have been asking very politely. Some of you have just been asking. GPT-6.2 Astra and Sol are now available. Super efficient a…
Kent C. Dodds 🐨

Kent C. Dodds posted “Works great with Kody.codes!” alongside a link to Cognition’s Devin Bots announcement, but gave no details about the setup or what worked. Cognition presents Devin Bots as a bot that leads a team of agents, with tasks ranging from writing firmware to merging thousands of PRs per week.

Works great with [https://Kody.codes](https://Kody.codes)! [https://x.com/cognition/status/2122737511684587551](https://x.com/cognition/s… Meet Devin Bots. From writing firmware to merging thousands of PRs a week, a Devin Bot leads a team of agents to get the job done. [![Vid…
Ben Tossell

Ben Tossell reports that @droid with Opus 5.5 Fast did a good job building a usage tracker in one shot, though he gives no implementation details.

everyone builds a usage tracker at some point, right? [@droid](https://x.com/droid) w opus 5.5 fast did a sweet job one-shot ![](https://…
Riley Brown

Riley Brown reports that Claude built a two-minute 4K Blender film from one Fable 5.2 prompt in about 4.5 hours: it wrote Python for the assets, seven sets, and camera moves and animation across 52 shots, and started seven copies of itself to build sets in parallel. The work totaled 12,500 lines of code; rendering 3,220 frames used 89 rented RunPod GPUs, with 36 running at once, and cost a little over $20. Brown says a similar prompt on a computer without his RunPod skill setup produced lower quality, and that Claude Code’s awareness of available cloud compute made it more ambitious.

The Fable of Clawd A 2 minute 4K film made entirely in Blender, from one prompt on Fable 5.2. It took about 4.5 hours. Claude built every…
Riley Brown

Riley Brown’s software-factory pattern is to give agents a clear specification, have several AI workers build different parts in parallel, use other workers to test, send broken work back for correction, and turn mistakes into rules to prevent recurrence; he says the factory improves with use. He casts the developer as a director coordinating agents, a token budget, and the software goal, stepping aside wherever AI can do the work. His agent-first tool examples are Claude Code and Codex, with an IDE available for cases where it is needed.

Content Factories and the Rise of Agentic Editing
Kent C. Dodds 🐨

Kent C. Dodds says he likes seeing companies use “MCP Server” as a selling point, calling it validating .

Man I just love hearing companies using "MCP Server" as a selling point. Feels freaking validating.
Riley Brown

Brown describes a software-factory workflow: give agents a clear specification, have them build different parts in parallel, use other agents to test the results and send failures back for repair, then turn mistakes into rules so the system improves with use. The human operator acts as a director, coordinating the agent team, token budget, and software to build while delegating wherever possible; Brown names Claude Code and Codex as agent-first tools, with the IDE opened when needed.

Content Factories and the Rise of Agentic Editing
ThePrimeagen

ThePrimeagen recommends using one AI chat at a time and thinking through the work rather than running many tasks in parallel; he argues that juggling small changes without stepping back to choose the best path can produce “slop.”

most developers would do much better just using one AI chat a time and actually think about what they are doing too many of y'all larp people are all trying to figure out what i mean by this Its quite simple: There is a bunch of whatever work that goes on (hey move this b…
Mckay Wrigley

Mckay Wrigley said Astra 6.2 might benchmark better but felt like a regression; he found the initial GPT Astra release had more “magic” than subsequent updates and only speculated that compute constraints might explain the change.

astra 6.2 is a pretty good ad for anthropic’s ipo maybe it benchmarks better, but it feels like a regression in every way. initial gpt astra release had a lot of magic that i feel like su…