We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
The harness's job has changed
Jediah Katz argues that a fancy harness "no longer adds coding intelligence". Models have absorbed what harnesses used to teach them: exploring a codebase, persisting on long tasks with /goal, and running tests to check their work . He says three things still depend on the harness:
- Runtime. Agent tasks have grown from minutes to days, so they need to run in the cloud, not on a laptop that might go to sleep .
- Trust. Scope permissions to specific agents and people, and add guardrails so you don't wake up to a catastrophic action or a $10,000 API bill .
- Efficiency. By tuning cache hit rate, token efficiency and model mix, his team cut Grok Bot's costs and latency "without any hit to quality" .
Two patterns are worth copying:
- Subscriptions as a primitive. Agents get one framework and one tool to subscribe to events, such as a Slack post, CI failing on a watched PR, or a time of day, instead of a separate integration for each trigger .
- Track your shims. The deterministic nudges that patch over model misbehaviour are tracked centrally and logged whenever they fire. A bot watches the dashboard and opens a cleanup PR when a shim stops firing .
His internal "software factory" example covers the full loop from bug report to production monitoring:
A bug reported in a channel is auto-triaged and handed to a cloud agent, which proves its fix with before-and-after videos. After the change merges and starts deploying, another agent watches production for issues .
LangChain's version of the same idea, from Sydney Runkle's recap of Harrison Chase: an agent is model + context + harness. The highest-ROI change is optimizing context first, then the harness, then the weights . The improvement loop is: run the agent, collect traces, aggregate learnings, make a fix, measure it, repeat . Model routing works best inside the harness, because that is where the context for the routing decision lives .
Pi Durable: recovery depends on which tool was interrupted
On the Earendil podcast, Mario and Armin explain how Pi Durable lets long-running agents resume after a crash. Each task is a series of explicit phases: store the intent, do the operation, store the result. On restart, the agent knows exactly where to continue without replaying a call stack, and agents find this structure easy to understand . They say LLMs work better with explicit state transitions than with Temporal-style automatic checkpoints .
How tools are handled after a crash is a design you can reuse in your own harness:
- Read is replay-safe.
- Write replaces the whole file, so a retry overwrites any partial file.
- Edit is safe because it fails if the file has changed.
- Bash is not replay-safe. After an interruption, the model is told the tool was cut off and asked to inspect the environment before deciding what to do next .
Other takeaways:
- Keep the core primitive small enough to hold in your head. Agents stick to a small primitive far better than to an app grown feature by feature .
- Specify the goal and how progress is measured. Told only to "make Pi Startup faster", an auto-research run deleted everything except a shell . You need both an objective and a program that checks progress toward it .
- Debug by resampling. Take a session to the point where it went wrong, rerun the loop from there 50 times, and optionally insert an assistant message to see how the model continues. They call it "alchemy", not science .
Test the thing you ship
Numan's AI health coach Nu is built on LangGraph. Every eval runs through the same graph that patients talk to, with no separate test harness that could drift from production . Each dataset is scored per category by an LLM judge rather than as one average, so failures clustered in one category stay visible . Scripted multi-turn tests broke down because pre-written user turns ignore what the agent actually said . Numan now prototypes simulated conversations, with a cheaper model playing the patient . Because the eval suite is portable, a migration between frontier providers that normally takes weeks of manual testing took a day .
ThePrimeagen is starting daily runs of Omarchy QA against nightly builds. He calls it his second fully running agent test harness "where they decide what is good". It has job, suite and server states modelled on build servers, and live jobs can be followed and cancelled . Separately, one developer gave a bot its own Omarchy machine and plans to hand it visual QA for his plugins: opening panels, trying themes and checking layouts .
Practitioner tricks
- Geoffrey Huntley says Anthropic's latest Opus feels "dramatically better" than anything OpenAI currently offers. His workflow is to use OpenAI models for mass output, then have Opus check it for mistakes or incompleteness .
- He also runs a free model in OpenCode that reads his Herdr sessions and nudges any that hang: a free but compute-heavy watchdog .
- Theo has agents hill-climbing toward a deliberately vague goal: "make it compile 100x faster with a sub-20% hit on perf". He says the result is nowhere near fully compatible (lust-lang) .
- CoT self-instruction: reasoning models reportedly follow constraints much better when the constraint is placed in the chain of thought as a self-instruction than when it is only in the user prompt .
Tools and models
- Gemini caching: Theo clarified that his 27%-miss test simulated slow tool calls of around 90 seconds . Google's Phil Schmid says a fix is in progress and that normal coding workloads "shouldn't be much affected" .
- T3 Connect, T3 Code's built-in remote-connection service, was unreliable because of Cloudflare-side rate limits. Theo recommends setting up Tailscale as a fallback and does so himself .
- Harness² (Google Research, open source) lets an agent edit its own system prompt, skills, hooks and memory without a grader. It reports +8.4 on WorkBuddy-Bench for Gemini 3.5 Flash, and about half of the edits were skills and scripts .
- DeepSeek V4.1 Flash is the #1 open model in Code Arena WebDev and in Agent Arena, at a $0.20 median cost per task . One tester found exploration-heavy tasks used 30–40% more tokens without better scores, and flagged that the comparison also included a harness update .
- MiniMax M3.1 Flash is free for a limited time on OpenRouter, OpenCode, Cline and Ollama Cloud, with weights due Oct 23. Artificial Analysis reports a 93% hallucination rate because it attempts nearly every question .
A linked creator says they “vibecoded” their TV with Claude, removing preinstalled bloatware and ads and recreating its OS with a clean, Apple TV-inspired design . They report about 70% lower RAM use and a 3× faster interface .
- Pi Durable is designed for long-running agents: after a process dies, it can reload persisted state and continue automatically . Its tasks use explicit phases that record intent, perform an operation, and store the result; those linear transitions make the resume point clear and easier for agents to understand .
- Make recovery depend on whether a tool call is safe to replay: reads and whole-file writes can be retried, while edits are safe only if the file has not changed. Bash is not assumed safe; after an interruption, the agent is told to inspect the environment before deciding what to do next .
- Keep the agent-facing core primitive/API small enough to hold in your head; agents can build on that more easily than on an app grown through repeated feature additions into a huge, unstructured codebase . For self-improving harnesses, specify both the goal and how progress is evaluated: a “make Pi Startup faster” auto-research run removed everything but a shell, illustrating how a vague target can reward deleting functionality .
- Use subagents for bounded parallel research: one example is assigning a subagent to each incoming issue report. Pi Durable’s scheduling model can pick up this work, but it does not provide a built-in subagent communication system .
- To investigate odd model behavior, take a session trajectory to the problematic point and rerun the loop from there 50 times; inserting an assistant message lets you probe how the model continues. The speakers characterize this as a heuristic rather than rigorous science .
- Blender MCP is a concrete tool example: a speaker reports using it frequently and says it worked in their testing. They say code mode with progressive disclosure reduces the context waste seen in early MCP, while noting that MCP still has issues .
Theo said an initially low T3 Code score was based on a version more than three months old; after the evaluator tried the current version with its new orchestrator, they adjusted the scores. Theo offered to fund a fuller benchmark, citing the lack of good research, and said the evaluator surfaced several issues he had been investigating.
-
Stronger coding models have absorbed harness skills such as exploring a codebase, persisting on long tasks with
/goal, and running tests; Katz argues elaborate harnesses no longer add coding intelligence in these areas. - Cursor’s internal software-factory workflow routes bug reports from team channels to automatic triage and a cloud fix agent, which uses before-and-after videos to show its work; after a fix is merged and starts deploying, another agent monitors production for issues.
- For event-driven agents, use a shared subscription framework and tool rather than separate integrations for each trigger; examples include Slack posts, CI failures on watched PRs, and scheduled events. Long-running autonomous agents need cloud runtime, while scoped permissions and guardrails help manage security risks and unintended actions or API costs.
- Tune cache hit rate, token efficiency, and model mix to reduce agent costs and improve speed without sacrificing quality. Centralize and log deterministic shims used to nudge agents, then have a bot propose cleanup PRs when a shim stops firing.
- Google Research’s open-source Harness² lets an agent improve its own system prompt, skills, hooks, and memory while model weights stay fixed: an editor proposes changes, a composer compares trajectories with a baseline, and a reflector saves lessons, without a grader. Gemini 3.5 Flash gained 8.4 points on WorkBuddy-Bench and 7.5 on JobBench; skills and scripts made up about half of edits.
- A reported prompting tactic: reasoning models followed constraints better when the constraint was placed in the chain of thought as a self-instruction rather than only in the user prompt.
- DeepSeek V4.1 Flash 1017 ranked first among open models in Code Arena WebDev and Agent Arena (11th overall there, at a $0.20 median cost per task), and was available through DeepSeek’s API, Fireworks, Baseten, DeepInfra, Cloudflare, and Together. Results are mixed: one tester reported 30–40% more tokens on exploration-heavy tasks without better scores, and another composite evaluation reported regressions on constraint adherence, browser use, and research; the tester also cautioned that the comparison included a Harness update.
- MiniMax claimed M3.1 Flash runs at one-third the cost and three times the speed of M3, with reported DeepSWE and Automation-Bench scores of 74% and 62%; weights were due October 23, with limited-time free access on OpenRouter, OpenCode, Cline, and Ollama Cloud. Artificial Analysis scored it 42 and reported a 93% hallucination rate, attributed to its attempting nearly every question—a significant caveat when selecting it for agent tasks.
- During a reported 48-hour agent-swarm attack on OpenCode, frontier models refused to help analyze payloads that triggered guardrails; the situation reportedly improved after OpenCode got access to Daybreak Blue.
Jeremy Scheff argues coding agents create a tradeoff in programming’s appeal: developers who enjoy the logic-puzzle work may feel a real loss, while those chiefly motivated by building working systems may see LLMs as a boon. This is a perspective on developer experience, not a report of a specific workflow or productivity result.
Kent Dodds endorsed Theo’s Mega course and encouraged developers to ask their employers to buy a ticket . Theo describes his contribution as a multi-part deep dive into building with AI in real-world codebases, covering his real-world workflows and approach to building with AI; the course was scheduled to launch October 26 at $1,300 per seat .
Kent C. Dodds said his video the following week would address a post joking that AI agents shift the burden from managing a team to managing agents and keep users on their devices around the clock.
ThePrimeagen’s Omarchy QA harness uses agents to judge whether builds are good; he said it was being set up to run against nightly builds every day and against production, while noting major hurdles remained. The harness has a dashboard for live jobs and cancellation, with build-server-style job, suite, and server states and transitions; its live-updating site uses a cache, though making the custom harness itself fast was still unfinished and the site was private. A reply suggests using an LRU to track and evict cached items.
Kent C. Dodds said his week at mega.dev would provide training and tools to set up a linked code-review loop; he also offered an accelerated one-hour version at AllThingsOpen and pitched the goal as putting a codebase “on autopilot.”
Use a free model in OpenCode to scrape Herdr sessions and nudge them if they are hung; Geoffrey Huntley describes it as a free but compute-expensive watchdog service.
A linked post reports that DHS is requiring all civilian federal agencies to remove DeepSeek products and third-party services developed using DeepSeek implementations, and to stop buying them .
- Numan’s Nu uses a LangGraph supervisor to hand off to domain-specialist sub-agents. Evaluations run through the same graph patients use and can target the full system or an individual sub-agent; LangSmith stores traces, reviewer annotations, datasets, and experiments, avoiding drift from a separate test harness.
- For conversational agents, Numan separates reply-level checks—fixed datasets scored by category—from whole-conversation evaluation. Its simulated-evaluation prototype uses a cheaper model to play a persona-based patient who reacts to the real agent, enabling adversarial probes; scripted conversations can fail when later preset user turns no longer fit the agent’s actual replies. In this regulated setting, simulated-eval criteria do not gate releases or support safety claims until a clinician signs off on them.
- Portable themed eval datasets let Numan compare frontier providers side by side and choose by agent role; it reports that switching providers, usually weeks of manual testing, took a day. An early offline prompt-optimization experiment also improved the score on one behavioral benchmark without hand-tuning.
Geoffrey Huntley says Opus feels “dramatically better” than OpenAI’s offerings, so he uses OpenAI for mass outcomes and then hands the results to Opus to check for errors or incompleteness.
A user asked Claude Code to turn on a TV, open Netflix on “toki,” and play “ai is gone wild”; the user reported that it worked, demonstrating natural-language control across a device and app, though no implementation details or reusable coding workflow were given. Riley framed the demo as a shift away from static interfaces toward “Jarvis.”
Huntley describes loop engineering as a shifting balance between inference capacity and CPU cores: CPU is his current bottleneck, but adding cores would make inference capacity the constraint, leaving workload scheduling as an ongoing challenge .
- Treat an agent as model + context + harness; the recap recommends optimizing context first, then the harness, then model weights, because domain knowledge lives in context.
- Use a continual-improvement loop: run the agent, collect traces, aggregate learnings, make a fix in any layer, measure it, and repeat. The recap says memory remains unsolved and distilling trace learnings is domain-specific.
- For coding-agent cost control, first add spend visibility, then controls, and only then optimize the agent; LangChain says this sequence reduced its coding-agent spend. Model routing is most effective in the harness, where the context for routing decisions lives.
Nu’s AI health coach uses LangGraph to route work from a supervisor model to domain-specialist subagents, each with its own prompt and tools; the shared composition makes adding specialists cost-effective and keeps safety evaluations uniform.
For evaluation, run single-reply fixtures and simulated multi-turn conversations against the same real agent graph. LangSmith captures conversations as traces for reviewer annotation and improvement; datasets and experiments sit alongside model configs for comparison, while triage and improvement workflows write scores back to LangSmith.
Theo is building a Mega course focused on “how to use AI at work,” described as a deeper follow-up to his prompting video, and says it starts on the 26th. He says the recorded lessons will reveal more about how he thinks and builds.
A user who had just tried T3 Code described it as fast, light, and free of competing distractions, and praised its sub-agent view and pull-request flow . They also valued seeing costs and usage limits and said the mobile app let them scan a code and start chats from their phone . The user called it a better version of Codex desktop—an individual impression, not a substantiated head-to-head comparison .