ZeroNoise Logo zeronoise
Post
Production Traces Are Becoming the Coding Agent’s Regression Suite
1 day ago
5 min read
155 docs
LangSmith Engine’s production trace-to-fix loop, reinforced by Rippling’s layered eval pipeline, points to a new coding-agent operating pattern: mine real failures, turn them into regression tests, and keep watching after deployment.

🔥 TOP SIGNAL

The most actionable shift is from launch-time checks to continuous production repair: LangChain’s LangSmith Engine reviewed 20,000 traces from a go-to-market agent, grouped a disqualified-prospect failure across seven runs, traced it to disqualified: true being overridden by an “always send” instruction, proposed a tool-level confirmation gate plus a prompt exception, turned failing runs into eval examples, and continued monitoring for recurrence. Rippling’s reported stack makes the deployment version explicit: offline mocks and fixtures on every commit, 300–400 sandbox queries after merge, about 10 real-system scenarios that block deployment, and multiple daily runs against production data.

⚡ TRY THIS

  • Turn production traces into a regression gate. After a failure: (1) cluster recurring traces, (2) write the expected behavior as criteria, (3) turn failed inputs into dataset examples, (4) harden both the tool contract and the prompt, (5) run the proposed branch against the new examples plus existing evals, and (6) keep the issue under watch so recurrence reopens it. LangSmith’s concrete fix required a confirmed disqualified flag and returned needs confirmation when absent, rather than deleting the broader “don’t be timid” instruction.

  • Use a planner → implementer → reviewer chain. DHH’s current routing is Fable for planning/review, another model such as Opus 5 for implementation, then Codex xHigh—and sometimes Grok—as an independent checker. His prompt shape for a real port was: make a dependency-free Rust version as a single executable, keep it pixel-perfect frame by frame, perform a full analysis, and “don’t stop until you’re finished”; Fable’s detailed eight-step plan let Opus take over when Fable ran out of tokens. Pass that plan as the handoff artifact. In DHH’s run, Fable finished in just under 45 minutes; Sol and Grok completed the same task at about $46 and $55 of estimated token cost, while DeepSeek Pro took 2h45 at about $23 and Luna plus DeepSeek Flash failed.

  • Put visual UI changes behind an isolated branch. Riley Brown’s reproducible setup is: connect a GrokBot developer bot to Cursor with the same account; ask for the redesign, specify the style, request screenshots, and name the model—his example ends Please use Claude Opus 5. Cursor then creates a separate branch and cloud computer, runs the app, opens a PR, and returns web/mobile screenshots; query status while it runs.

  • Keep agent instructions portable. Romain Huet says OpenAI contributed AGENTS.md to the Agentic AI Foundation so instructions, skills, and plugins can travel across tools; DHH reports using a minimal CLAUDE.md that points to AGENTS.md because Claude Code otherwise expects its own instruction and skills locations. Keep the canonical policy in AGENTS.md, add thin harness-specific adapters, and avoid maintaining divergent copies.

📡 WHAT SHIPPED

  • LangSmith Engine — production-agent maintenance tooling. It can open and merge a PR directly, expose an issue through the CLI for another coding agent, derive dataset examples and expected-behavior criteria from failing production runs, test the proposed branch, and reopen a resolved issue when the failure recurs.

  • OpenWiki 0.4.0 — cross-harness integrations, plus WikiBench. Choose a harness after npm install -g openwiki@latest, then run openwiki integrations install claude, ... codex, or ... opencode and ask the agent to create or update the repo wiki. WikiBench evaluates that context layer on a pinned repository with a reader agent, coverage/retrieval questions, and separate fact-presence and grounding judges; wiki plus source produced the highest mean score at lower cost than raw source alone, while wiki-only performed much worse.

  • Google Antigravity — richer agent I/O. Interactive Generative UI Artifacts can render dynamic data visualizations and 3D explanatory simulations inline and in the artifacts panel. Gemini 3.5 Transcribe adds voice interaction that, with permission, uses screen context and chat history to improve transcription of file names, agent thoughts, and active documents.

  • Grok Bot — access expansion, but check the entitlement. The @bot account says access is for SuperGrok and Cursor Pro subscribers, while Michael Truell says it is available to anyone with a standard Grok or Cursor subscription; the two announcements do not establish one consistent tier.

  • Codex compatibility watch. Theo reports that Codex 0.150 introduced a breaking change that regressed T3 Code; he cut another stable release to fix it and said it should be out within 15 minutes. Separately, swyx advises avoiding Codex “locked use” on macOS after two keychain lockouts in one week, citing an Apple-recognized known bug and saying cloud use is not ready as a fallback.

🎬 GO DEEPER

  • ThePrimeTime — “Protecting Your Energy”. The useful section is an agent-testing design critique: inspect what each trial did and how much damage it caused, launch experiments, generate tasks, run tests, and graph results—but recognize that a control surface can expose everything requested while still producing a workflow that is not useful. The close gives a practical human boundary: reserve the first hour for one personally meaningful feature and use prompting for surrounding setup.
  • Study OpenWiki with WikiBench. Its pinned-commit environment, reader-agent verifier, repository-grounded questions, and separate fact/grounding judges are a concrete reference architecture for testing whether agent memory helps rather than merely adding context.

Editorial take: The durable advantage is a feedback system—not a single “best” model: production failures become evals, model roles are separated, and execution stays isolated.

Production Traces Are Becoming the Coding Agent’s Regression Suite
Research extraction
ThePrimeTime

Exact timestamp ranges are unavailable in the supplied transcript; use these transcript-line windows instead.

  1. Testing-agent setup and bug hunting — L16-L20. The speaker describes a testing framework in which multiple agents play the game, search for bugs, and generate alerts . He contrasts this with the slower coding process in which implementation itself provides tactile design feedback and helps shape the product .

  2. Testing queue and experiment loop — L25-L30. This is the strongest operational workflow segment: inspect what each agent did and how much damage occurred, review trial names and running/finalizing states, launch queues, generate gameplay-improvement tasks, run tests, and graph results . The speaker adds an important caveat: the system can expose all the requested controls while still producing an interface that is not useful or compelling .

  3. Closest match to the one-hour manual-coding boundary — L58-L62. The source does not state a formal manual-coding boundary; its closest recommendation is to reserve an hour first thing in the morning for programming on one personally owned feature . It then suggests using prompts for surrounding setup while retaining control over the interface and design , and rejects the idea that developers must become “agent herders,” while warning that the speaker’s own testing framework may primarily burn tokens .

  4. Scope and deep-work guardrail — L47-L52. The speaker recommends narrowing ambition to one or two things and going deeper where there is a moat, rather than continually producing generic projects . He connects that approach with taking time, sleeping, and building a more intuitive, cohesive product instead of chasing a supposed deadline .

Protecting Your Energy
David Heinemeier Hansson (DHH)
Profile
  • DHH’s current routing SOP is to use Fable for planning and review, another model such as Opus 5 for implementation, and Codex xHigh for the final review; he has also started testing Grok and says Copilot now catches legitimately broken code. He ranks Fable first and Opus 5 second, while using Claude Code as his main harness because of its multi-agent view and OpenCode for open-weight models through Fireworks. In a concrete benchmark, he prompted an agent to port a Python library to dependency-free Rust as a single executable, with pixel-perfect frame-by-frame behavior and exhaustive validation. DHH reports that Fable finished in just under 45 minutes, reducing startup time from 86ms to 2ms, execution time by about 9.6×, and producing a 3MB executable; Fable’s detailed plan also let Opus 5 continue when Fable tokens ran out. In reruns, he estimated Fable’s per-token cost at about $550, versus roughly $46 for Sol, $55 for Grok 4.6, and $23 for DeepSeek V4 Pro; Sol and Grok completed the task at much lower cost, while DeepSeek took longer, and GPT Luna plus DeepSeek V4 Flash failed.
  • To stay productive while agents work, DHH moved from one session to parallel CLI sessions in tmux tabs, then adopted Herdr, which adds completion notifications and status tracking. He connects additional machines over Tailscale/WireGuard, uses GL.iNet Comet KVMs for remote control, and reports a current ceiling of roughly four to five machines and 16 threads at full acceleration. In Omarchy, agents filter wrong, duplicate, and low-quality PRs, validate fixes in VMs, and send him a summary for the human merge decision; he says he merged more than 1,000 PRs in three months while roughly 400 remained unmerged. Omarchy Quattro’s extension system ships agent “skills” for creating OS plugins, and DHH reports 330 marketplace plugins appearing within three days. He is also automating the queue with an Omarchy bot that processes pending to-dos, PRs, and issues on a schedule and emails him a final-decision digest via Hey CLI; at Basecamp, he assigns agents to to-dos and cards as asynchronous coworkers rather than making users wait in chat. His Amabot architecture separates a model-running coordinator (“brain”) from isolated VM workers (“hands”) that execute untrusted PR or issue code, treating test output as external data so malicious payloads cannot contaminate the coordinator.
  • DHH’s prompting advice is outcome-first: provide enough high-level vision to manifest a first version, then use the software and iterate instead of prescribing the implementation upfront. For design evaluation, he has agents produce three alternatives and relies on rapid human selection rather than exhaustive up-front specification. He relays Boris’s secondhand observation that Opus 5’s shipped system prompt became 80% smaller because overly prescriptive human instructions could damage the agent’s work. In his own projects, he places a CLAUDE.md containing a pointer to AGENTS.md, because he says Claude Code otherwise does not read agents.md or .agent/skills. For early design, he records 10–20-minute stream-of-consciousness prompts with Plaud, transcribes them with ElevenLabs, and uses a codebase-aware dictionary and LLM cleanup pass to preserve technical terms while avoiding overspecification. When review exposes overengineering, his practical feedback is simply “make it simpler”; he reports that agents often respond by cutting the implementation in half. The approach still has an architectural boundary: DHH says Quattro was entirely agent-written but he reviewed its shape and critical model-layer code, whereas an early Basecamp 5 sprint in which designers “vibed” produced individually plausible PRs that collectively damaged the architecture and required manual cleanup; he considers programming judgment important on substantial existing codebases.
DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501
Riley Brown
Profile
  • Grokbot-to-Cursor coding workflow (firsthand): Riley Brown says he used Grokbot across his business for the prior week. He connected a developer bot to Cursor using the same account, then prompted it: “The landing page we are working on the other day, I need you to redesign it… in the same style as Grok… simple black and white… Create a new version… take screenshots… Please use Claude Opus 5.” The coordinator delegated to Cursor, which created a separate branch and cloud computer so the production app was not directly changed, then produced a pull request, ran the app, and returned web/mobile screenshots; Brown checked progress with status or by opening the task in Cursor.
  • Transferable orchestration pattern: Brown uses a general agent as a coordinator that messages specialist agents, gathers their recaps, and synthesizes prioritized next actions; agents can also collaborate in group chats.
  • Computer-use caveat: Although the agent’s cloud computer can be taught tasks through demonstrations, Brown says he has found it of limited practical use so far because the IP gets blocked by bot-resistant websites and he has not identified many strong use cases yet.
11 Insane Things Cursor’s NEW GrokBot Can Do
ThePrimeTime
  • Firsthand Cursor Agent workflow (ThePrimeTime): On a long-running game project he describes as roughly 30,000–35,000 lines, ThePrimeTime runs a testing framework with multiple agents playing the game and searching for bugs, producing alerts. The queue lets him inspect what each trial did and the damage caused, see active and finalizing runs, launch jobs under different “personalities,” generate tasks to make the agent play better, run tests, and graph results. He says the system became “everything I asked for, but nothing I actually wanted”: it exposed many controls but was barely useful because he had not taken enough time to design a compelling workflow.

  • Preserve deliberate human design time: ThePrimeTime argues that implementation used to provide feedback on data shape, UI, and interaction design, whereas rapid AI/vibe coding compresses that loop into myopic “make a decision now” iterations. His concrete countermeasure is a first-thing-in-the-morning, one-hour manual-coding block for a personally meaningful feature; use prompting for surrounding setup, such as shaping the interface or generating a .d.ts file, while retaining hands-on control of the feature.

  • Relayed account — Aaron Francis (secondhand in this transcript): Francis is introduced as Laravel’s VP of marketing and community, a developer, and a product builder; he reports staying up until midnight or 2 a.m. exploring AI possibilities despite having a wife and four children. He changed strategy to scope work to one or two projects and increase vertical ambition, moving from broad experimentation to depth in an area where he has a moat rather than broadly building to-do lists or Screen Studio ripoffs.

Protecting Your Energy
Salvatore Sanfilippo
Profile
  • Salvatore Sanfilippo’s firsthand local-inference testing suggests the M5 Ultra is a poor-value platform for coding-agent workloads: its 1.2 TB/s memory bandwidth is only modestly above the M3 Ultra’s 850 GB/s, while GPU compute is the limiting factor for prefill and speculative decoding. He expects fast token generation but weak prefill, and says the machine is not well suited to large mixture-of-experts models with roughly 800 GB of weights.
  • For a multi-user local coding-agent service, Sanfilippo reports that an optimized Dwarf Star deployment on a DGX Station achieved about 23,000 tokens/second of aggregate prefill, around 200 tokens/second for one session, and thousands of tokens/second of aggregate generation. He describes this as enough for roughly 10–20 users to submit long files to an API-like system while a coding agent continues responding.
  • His practical next step is a two-DGX-Station RDMA setup for more serious Dwarf Star development and testing: mixed RAM/VRAM inference, alternate kernels, thermal-throttling behavior, and RDMA-latency reduction. He cautions that current remote access has stability and crash issues, and emphasizes that vendor CUDA kernels plus vLLM or SGLang optimization can materially affect model performance.
M5 Ultra. Non mi entusiasma.
Latent.Space
  • Lovable’s hosted MCP capability architecture: Lovable can expose selected functions from a published app as tools through a hosted MCP server, giving the same application both a conventional human UI and an agent interface callable from ChatGPT, Claude, and other MCP-compatible clients. For coding-agent builders, the reusable pattern is to treat application functions as stable capabilities orchestrated through one agent; Lovable CTO Fabian Hedin argues that connecting capabilities correctly and reliably is harder than orchestration itself.
  • Asynchronous agent loop: Lovable’s agent can schedule itself to resume later—for example, to check a deployment or monitor a recurring process—and return the result to the same conversation.
  • Permission-safe tool integration: Lovable’s “app user connector” preserves the user’s identity and source-system permissions, keeps encrypted credentials server-side in its connector gateway, and gives the generated app only a short-lived key bound to that user. Lovable explicitly separates external-system connections from application code so the application never receives those credentials. Relevant resources: Lovable Connectors and How Lovable secures connected data.
Lovable CTO: The Future of SaaS Is Apps That Agents Can Use
DHH

Local-model agent workflow (firsthand report from Justin Starner, amplified by DHH): Qwen 3.8 27B ran locally on an RTX 5090 and, through a new CLI tool, triaged a HEY Imbox and screener, helping clear 300+ emails that had accumulated over time. The concrete signal is a local 27B model handling a high-volume tool-use task; the reported use case is email triage rather than coding, so it is not evidence of coding-agent performance.

I am happy to report Qwen 3.8 27B running locally on my RTX 5090 just successfully triaged my HEY Imbox and screener using the new CLI to…
Matthew Berman
  • MCP connector workflow: Higsfield MCP can extend a Claude agent used for research and planning so it can also generate videos and images. The setup described is: open Claude → Settings → Connectors → Add custom connector, enter Higsfield, paste the MCP URL shown in the video, then connect and sign in.
  • Evaluate models by completed-task cost, not token price alone: The video compares “Kimmy K3” with “GPT 5.6 Soul”: although Kimmy K3 is described as roughly half the price per token, the total cost of the example task is nearly the same—$0.84 versus $0.96—because it uses more tokens.
  • Deployment heuristic for agent workloads: Model selection should weigh cost alongside privacy and product fit; open-weight models can be self-hosted, fine-tuned on internal data, and served through competing inference providers, providing more control over data, deployment, and customization.
The Most Important Chart In AI Right Now
Riley Brown
  • Firsthand architecture and agent hygiene: Riley Brown reports using Grokbot across his business for a week. Each agent has its own session and cloud computer, while skills and plugins are shared across agents; the agent description is reread on every run to define behavior. Brown recommends simple, single-purpose bots rather than a new bot for every conversation, with one pinned general bot for unclassified work and quick questions.

  • Persistent monitoring loop: His “monitor the situation” bot wakes every five minutes on weekdays, checks an explicit list of active situations across Slack, email, calendar, and other connected sources, and sends desktop/mobile updates when something changes. Completed situations are removed from the monitoring list.

  • Concrete coding-agent workflow: Brown connects a Grokbot developer bot to Cursor using the same account, then delegates a task such as: “Redesign [the] landing page… Create a new version… take screenshots… Please use Claude Opus 5.” Grokbot improves and forwards the prompt to Cursor; Cursor creates an isolated branch and cloud computer, runs the app, opens a PR, and returns web/mobile screenshots for review. Brown can request progress with “status” or inspect the job directly in Cursor.

  • Multi-agent orchestration: Brown asks a general Grokbot to message every other agent for a weekly recap and prioritized next steps, or uses group chats so several agents can respond together. When a recurring project emerges, he has the general agent transfer the relevant context into a dedicated project bot and attaches a scheduled routine; his example sends a weekday 8:00 a.m. prompt to identify interesting topics and Twitter bookmarks for short-form scripts.

  • Event-driven automation pattern: Grokbot supports both time-based routines and action/trigger-based routines. Brown describes triggers for Slack, GitHub, Microsoft Teams, Linear, Sentry, and PagerDuty, with webhooks available for custom triggers; his example creates a Slack-watching agent that reacts to any message and performs a specified follow-up.

  • Context management and safety: Brown cautions that each plugin increases both available context and security exposure. For Notion, his custom “add context” skill writes supplemental information into a collapsible callout/dropdown instead of changing the original outline, preserving the source document while allowing the agent to enrich it.

  • Useful control-loop caveat: An agent can send an email, poll for a response on a schedule, notify the user when it arrives, and delete the polling routine automatically. Brown also reports that browser computer-use is currently limited by bot-blocking IP addresses, although the agent can be taught workflows through demonstrations.

11 Insane Things Cursor’s NEW GrokBot Can Do
swyx
  • Codex safety warning: The post advises avoiding Codex “locked use” capabilities after unstable macOS features completely locked the author out of their macOS Keychain twice in one week. The author points to Apple Developer Forums acknowledging this as a “known bug” and says cloud use is not yet sufficient.
PSA: do not use codex "locked use" capabilities right now. it is currently relying on unstable mac features and has completely locked me …
Simon Willison's Weblog
  • Paul Dix’s verification-loop pattern: AI reportedly wrote 1 million lines of code, then refined them over several months into reliable software running on millions of developer machines. Dix argues that building an objective verification system—or “oracle”—and providing clear direction lets AI iteratively improve complex software until it works.
Quoting Paul Dix
Ben Tossell

@bentossell reported that his most-used prompt over the previous 24 hours was: “do all with new luna threads.” The post provides no explanation of what Luna Threads is, what “all” refers to, or how the workflow works, so it does not establish a replicable coding-agent technique.

most used prompts over the last 24 hours "do all with new luna threads."
Addy Osmani

Addy Osmani offers a contrarian caution for AI-assisted coding: tools that remove friction can also remove the friction through which engineers develop taste and mastery. He recommends this perspective to engineering leaders and early-career engineers via the linked article.

Friction is what builds taste and mastery. The tools that remove it also remove what made us good enough to use them well. Worth reading …
Kent C. Dodds 🐨
  • Kent Dodds is building bot-character agents and says pairing them with Kody (@kodykoala) gives the bots access to his resources; he considers this more powerful and customizable than built-in plugins and says the same pattern can use other agents.
  • In the described workflow, Cole finished documentation, used Kody for GitHub access, and then handled a Cloudflare domain redirect. After encountering a Worker upload limit, Cole deployed a temporary Worker in the KCD Cloudflare account; epicsystem.dev and www were configured to 301-redirect to epicproduct.engineer while preserving paths and query strings.
Making [@bot](https://x.com/bot) characters. Combining this with Kody feels like a super-power because I can use [@kodykoala](https://x.c…
Kent C. Dodds 🐨

Kent C. Dodds says he built Kody.exchange “just last week,” in response to a proposal for an agent-to-agent communication protocol that would let agents exchange messages directly instead of requiring humans to copy Slack messages between them.

I built this just last week 😆 [https://kody.exchange](https://kody.exchange) [https://x.com/coleywoleyyy/status/2092722918828679385](http… there needs to be a communication protocol between agents I just want your agent to talk to my agent. not your agent send me a slack dm t…
Kent C. Dodds 🐨
  • Grok Bot is available to SuperGrok and Cursor Pro subscribers, with weekly usage limits reset for all users.
  • Kent C. Dodds strongly endorses Grok Bot and says it can be combined with KodyKoala so other agents can participate and help make Grok Bot more efficient; he also considers it useful on its own.
All SuperGrok and Cursor Pro subscribers now have access to Grok Bot. We're also resetting weekly usage limits for all users. Enjoy! My friends, [@bot](https://x.com/bot) is just so good. [https://x.com/bot/status/2092664183334846840](https://x.com/bot/status/2092664183… [@bot](https://x.com/bot) Combine it with [@kodykoala](https://x.com/kodykoala) so other agents you use can play too and you can make [@b…
ThePrimeagen

ThePrimeagen humorously presents a caution about AI-generated code: code written by Fable was reviewed by “Sol, opus 5, and Grok,” yet looked “horrible” when he inspected it himself—his “Schrödinger’s code” framing implies that multi-model review does not eliminate the need for human inspection.

Schrödinger's code 1. fable wrote it, it must be great. in fact, i had Sol, opus 5, and Grok all review it. 2. if i look at it, its horri…
LangChain
  • LangChain’s open-source OpenWiki generates and maintains codebase documentation, while its WikiBench benchmark tests both wiki quality and whether the documentation improves coding-agent answers using questions grounded in the repository.
  • Replicable evaluation workflow: pin a repository commit, initialize OpenWiki to generate the wiki, have a reader agent answer automatically generated coverage and retrieval questions, attach JSON rubrics listing expected facts, use separate LLM judges for fact presence and source grounding, score each answer by the fraction of expected facts recovered, and average across questions. Coverage prompts explicitly ask where a subsystem lives, what it interacts with, and how to verify a change did not break it.
  • The practical context-management takeaway is to use generated documentation as an index alongside source code, not as a replacement: wiki-plus-source produced the highest mean answer score at lower cost than raw source alone, while wiki-only performed substantially worse.
  • LangChain reports substantial model cost/time variation: DeepSeek Flash was roughly one-sixth the cost of GLM 5.2; runs ranged from $0.44 for Luna to $9.18 for GLM 5.2 and from 11 minutes for Terra to 50 minutes for GLM 5.2. Higher-cost models read about three times as many files but produced only 1.2–1.5 times as many pages, indicating that much of the extra work went into repository understanding rather than output volume.
Evaluating OpenWiki with WikiBench
Anthropic
  • Anthropic reports that Stanford’s SALT Lab, Oxford’s HIP Lab, and METR designed independent studies analyzing aggregated outputs from 250,000 Claude.ai or Claude Code conversations from April–May 2026; METR’s study specifically aims to estimate real-world productivity gains from coding agents, with results still forthcoming.
  • The initiative gives external researchers access to real, privacy-preserved Claude usage data for studying AI’s impacts; details are available at Anthropic’s research page. This is a measurement/research lead rather than a reported benchmark or replicable developer workflow yet.
Three research groups—Stanford’s Social and Language Technologies lab, Oxford’s Human Information Processing Lab, and METR—designed indep… The other two studies are ongoing: HIP Lab is studying how Claude's behavior relates to how people feel when using AI, while METR is esti… For the first time, we’ve given external researchers a way to study AI’s impacts using real, privacy-preserved Claude usage data. To date…
Kent C. Dodds 🐨

Kent C. Dodds reports a low-friction coding-agent workflow: keep one long-running Cursor cloud-agent session dedicated to simple fixes discovered while using Kody Koala, such as small UX “paper cuts,” and let the agent auto-merge and deploy the completed changes.

I've got a long-running session with a single [@cursor_ai](https://x.com/cursor_ai) cloud agent for handling simple tasks that I think of…