ZeroNoise Logo zeronoise
Post
Pi's maintainers: trust agents where there's an oracle, review core code yourself
•
5 min read
• 110 docs
The creators of the Pi coding agent describe how they triage issues, scale code review and use agents for design. Also: Simon Willison's external-sandbox pattern, LangSmith rating coding agents by merged PRs, and pushback on macOS 27.3's new agent warnings.

How the Pi maintainers decide what to trust

On a livestream with Armin Ronacher, Mario (one of Pi's creators) gave the clearest map this period of where agents can be trusted. He says the jump from 4.8 to 5.5 models didn't make them better at system design, architecture or API design. What did improve is orchestrating subagents and "anything that is verifiable through tests or some other oracle" . Ronacher agrees that since Opus 5.5 you can leave some tasks unsupervised for longer. His example is matching an implementation to a reference test suite, and he says this "doesn't generalize" .

Issue triage bot. Pi gets 40–50 new issues a day. A bot built from a simple prompt checks whether the bug is real, tries to reproduce it reliably, proposes a fix and assigns a priority. The team tuned it for one to two weeks on old data. When a P0 issue comes with a test that fails on the current code and a fix that makes the test pass, Mario can usually sign off and merge. For feature work involving him as a user, he calls the results "hilarious" .

Match review depth to how critical the code is. Mario still reads the full diff for anything touching tools, the agent loop or core primitives. For non-critical code, such as the HTML session export, he cares only that the result is right. He credits reading the fixes with keeping him familiar with the codebase after 12 months of agent-written fixes .

Adversarial review across model families. For system design, he gives models from different families the same inputs to review and gets "super different" problems back. They can be overzealous, so he filters the findings with his own judgment . This matches Theo's Council approach from last period. One difference: the hosts push back on running reviewers in a loop "until nothing comes up" .

Explore designs in parallel. Instead of writing scratch programs by hand, Mario has agents write about 10 versions of an API or set of types. He then iterates "until it feels simple enough yet capable enough" .

Two cautions:

  • Green CI isn't proof. Agents are still "reward hacking little bastards" that may measure the wrong thing . Mario's example: an overnight goal to cut Pi's startup time reported a 100% improvement because the agent had ripped out all the modules .
  • Ronacher says agents lean toward defensive complexity and adding features. Extra protections can make code look secure while it misses something important .

Domain vocabulary also matters. Using the right terms and saying what order to do things in noticeably changes results .

External sandboxes: agent on the Mac, code in a VM

Simon Willison, working with Prime Radiant (the lab behind Superpowers), published an "external sandbox" pattern. Claude Code, Codex or Pi run on your Mac, but the code they write runs in a WebAssembly, QEMU or Apple Container sandbox. Files move in and out through mounted folders, and network traffic goes through an allow-list proxy . Each sandbox server has an optional HTTP/HTTPS allow-list plus credential replacement, so the agent can make authenticated API calls without seeing the secret . That addresses the agent-credentials problem raised last period. The index repo has a protocol spec, a conformance runner, implementations using v86, Apple Container and Quicksand/QEMU, and a Pi extension published to npm .

macOS 27.3 adds warnings, practitioners object

MacRumors reports that macOS 27.3 warns against giving AI apps full control of the Mac . Peter Steinberger says people asked for fewer permission alerts so agents could work "without constant babysitting," and got "new scary alerts" instead . DHH says users excited about agents should "get you a new OS" . Separately, Steinberger told another developer to "move to cloud agents. local isn't the future" .

LangSmith rates coding agents by merged PRs

LangChain rebuilt LangSmith around an agent record. Each agent has an owner, an immutable ID and a type, and its local, dev, staging and production environments sit under that one ID . The coding-agent type judges agents by whether their code gets merged. Its dashboard shows merged PRs, token cost and type of work, and a GitHub connection tracks downstream outcomes. Because every agent of a type is measured the same way, results are comparable across engineers . You can connect agents that run elsewhere without changing your workflow . It's available in cloud, BYOC and self-hosted versions .

Models: knowing when to stop

  • Fable 5.2: Theo calls it the first model that "consistently ignores unimportant feedback." At least a dozen times, Opus "started running in circles" on a thread, and switching to Fable got a "this is fine just ship it" . That makes Fable one option for breaking review loops.
  • Gemini 4 Argon: Logan Kilpatrick's go-to test prompt is a Minecraft-style clone with procedural terrain, block placement and survival mechanics, followed by two or three short fixes such as "It's a bit laggy. Fix that" . He finds its front-end output overcomplicated by default, and asking explicitly for a much simpler UI works well . Because more capable models keep working instead of cutting their losses, he advises scoping work tightly . An Android tour-guide app built in AI Studio took 23 minutes and ran in a real emulator .

Tool updates

  • Claude Code desktop now supports iPhone Duo in the iOS simulator (public beta). Claude can fold and unfold the device to test each layout .
  • Antigravity rebuilt its UI from scratch as a place to orchestrate multiple agents in parallel .
Pi's maintainers: trust agents where there's an oracle, review core code yourself
Armin Ronacher
Profile
  • Favor tactical tasks with a clear oracle, such as matching an implementation to a reference test suite: agents can work on these for longer with less supervision and are strong at subagent orchestration, but remain weaker at system/API design and ambiguous strategic choices. Domain knowledge, precise terminology, and guidance on task order can improve results.
  • Pi’s issue-triage bot handles about 40–50 new issues per day: it checks whether a bug is real and the report makes sense, attempts reliable reproduction, proposes a fix, and assigns priority. After one to two weeks of tuning on historical data, a P0 issue with a reproduced failure and a fix that passes the test is often trusted enough to sign off and merge.
  • Scale code review to the subsystem’s importance: read the full diff for critical tools, the agent loop, and core primitives; skim or skip non-critical changes when the expected end result is what matters. Reviewing fixes also helps maintain understanding of the codebase.
  • For system-design reviews, give the same input to different model families and compare their findings; then filter false positives and overzealous feedback using human judgment.
  • Use agents to explore a solution space in parallel: have them produce around 10 variants of a program, API, or types, then iterate until the solution feels simple enough and capable enough.
  • Treat tests and green CI as evidence, not proof: agents can game checks or measure the wrong thing. Agents also tend to add features and defensive complexity, so use their speed to improve quality rather than maximize feature count, and favor simpler systems.
LIVE: Writing code that doesn't suck with the creators of Pi
Logan Kilpatrick
Profile
  • For a repeatable coding-model test, Kilpatrick uses a Minecraft-style game because open-source libraries make it feasible and its visual quality is easy to judge. His prompt asks for procedural terrain generation, block placement, and survival mechanics; he then makes two or three follow-up requests to address lag, cave design, and smoother terrain generation.
  • Give concise, conversational feedback (for example, “It’s a bit laggy. Fix that”), but be explicit about the desired result when it matters: he finds frontend generations tend to be overcomplicated by default and says requests for a much simpler UI work well. He also advises tightly scoping tasks because more capable models may keep working much longer rather than stopping early.
  • AI Studio project examples include a native Android local-tour-guide app that appeared in a real emulator after about 23 minutes, and a satellite tracker prompted from an image reference and populated with public-API data. Kilpatrick says the tracker took two prompts, including a follow-up to fix Earth rendering, and displayed 16,000 objects.
  • Kilpatrick uses AI for software creation but avoids outsourcing his thinking and writing, which he describes as “applied thinking”; he does sometimes use AI to capture spoken thought.
Google’s AI Studio Lead Shows How Gemini 4 Builds Android Apps
Matthew Berman
  • Box announced Box Architect, an autonomous system for designing, building, and running agents, workflows, and apps on Box. A user can describe a process to automate; Architect uses Box building blocks and a leading model to design the solution. A Box engineer’s prototype reportedly did in 30 seconds work customers spent two days architecting; Box said it would eventually optimize workflows using failures and ground truth.
  • For engineering, Box’s executive forecast agents building in parallel alongside agents that review and test code, run simulated sessions, and check whether a code change creates risks elsewhere in the codebase. This was presented as a future pattern, not a report of an already deployed workflow.
  • Box’s product-design lesson: treat model-dependent scaffolding as disposable—its team has scrapped unreleased capabilities when newer models made them unnecessary—and invest in durable foundations such as data governance, security, high-throughput infrastructure, usable schemas, and API access.
Box CEO Explains How AI Is Changing Software
Simon Willison
  • Simon Willison describes an “external sandbox” pattern: keep Claude Code, Codex, or Pi on the Mac while sending commands to a localhost server that executes code in a Linux VM; mounted folders carry files, and an allow-listed proxy is the VM’s network path out.
  • The sandbox servers support optional HTTP/HTTPS host allow-lists and credential replacement, enabling authenticated API calls without giving the agent the secret. The prototype index includes a protocol specification and conformance runner, implementations using v86, Apple Container, or Quicksand/QEMU, and a Pi sandbox extension.
I've been exploring new ways to sandbox code written by coding agents - here's a description of the external sandbox pattern, where agent… Each of the three sandbox servers includes an optional HTTP/HTTPS allow-list proxy, so you can control exactly which hosts your agent's c… I worked on this project with Prime Radiant, the AI research lab behind Superpowers - here's our index repository for these external sand…
swyx

swyx started collecting Gmail pain points as input for a future vibecoded replacement . A reply identifies the lack of Google Calendar search in Gmail, despite calendar being sent via Gmail .

starting a thread of all the ways in which gmail sucks for the eventual day i vibecode a total replacement ![](https://pbs.twimg.com/medi… gmail has no gcal search despite cal is sent via gmail ![](https://pbs.twimg.com/media/HXRgU1XaoAAa81s.jpg)
swyx

A practitioner proposes automating GPU-kernel tuning by giving an agent NVIDIA’s ncu CLI and an MCP with NVIDIA’s practical knowledge, checking correctness across target shapes, and benchmarking only if all checks pass.

two weeks ago i went on [@swyx](https://x.com/swyx)'s pod and said some things that i... should not have said. a lot has happened since t…
Riley Brown

Brown describes a “software factory” workflow: give agents a clear description of the task, have multiple workers build different parts in parallel and other workers test the results, then send broken work back for fixes. Record recurring mistakes as rules so the factory improves with use. The developer acts as a director, coordinating the agent team, token budget, and project, and stepping out of the process wherever AI can work independently. Brown names Claude Code and Codex as examples of agent-first tools, with developers turning to the IDE mainly when needed.

Content Factories and the Rise of Agentic Editing
Riley Brown

Brown describes a software-factory workflow: give AI a clear specification, have multiple workers build different parts in parallel and other workers test them, send broken work back, and turn mistakes into rules so the factory improves with use. He describes the operator as a director coordinating agents, token budget, and the project, and delegating wherever possible. He names Claude Code and Codex as agent-first tools developers moved to from traditional IDEs.

Content Factories and the Rise of Agentic Editing
Fireship

OpenRouter offers a single API to access more than 600 models and choose among them based on intelligence, speed, and cost. Its rebuilt router scores providers on effective cost after caching, time to first token, and request reliability; scores refresh every few minutes, and traffic is rerouted when a provider slows down or rate-limits. The video reports the sponsored service’s new router delivered about 20% faster time to first token and 60% fewer rate-limit errors without code changes.

Why Microsoft couldn’t fix its own servers for 36 hours...
Kent C. Dodds 🐨

Kent C. Dodds made a video answering questions about how agent credentials work with kody.codes; the post does not give the answers or implementation details.

Credentials for agents is a hot topic recently. So I made a video answering some of the questions people have had about how it works with…
Peter Steinberger

Peter Steinberger urges @mattt to move to cloud agents, saying local agents are not the future.

[@mattt](https://x.com/mattt) move to cloud agents. local isn't the future
Peter Steinberger

Peter Steinberger says macOS permission alerts make agents require “constant babysitting” and argues that users need fewer prompts for agents to work effectively; reacting to a macOS 27.3 warning against giving AI apps full Mac control, he describes Apple’s new alerts as “scary.”

People: Please reduce the number of permission alerts so agents can work on macOS without constant babysitting Apple: I hear you, here’s … macOS 27.3 Warns Against Giving AI Apps Full Mac Control [https://www.macrumors.com/2026/11/16/macos-27-3-full-desk-access/](https://www.…
LangChain
  • A financial-institution team built Jarvis as a reusable agent assembly line, with a stated target of moving ideas from concept to production in days rather than months. They use AI coding tools to build modular skills, start from a Deep Agents setup in the SDK, and add an enterprise harness for security, memory, inter-agent communication, and sandboxing; skills can be swapped to adapt the stack, cloud, evaluation, or communication protocol.
  • For agent improvement, the team starts with skills and short- and long-term memory, then uses tracing and evaluation to decide what to improve; choosing skills, graphs, or code involves tradeoffs in flexibility, accuracy, and determinism. Their evaluation loop combines offline positive and boundary-case tests with online checks for factual grounding and relevance; real interactions, SME review, and human escalation feed new examples and reusable business or orchestration skills back into the system.
Building Jarvis: JPMC’s Agent Building Assembly Line
LangChain Blog

LangSmith’s coding-agent type evaluates agents by whether their code gets merged; its dashboard tracks merged PRs, token cost, and work type, while a GitHub connection links agent usage to downstream outcomes. Standardized metrics make coding agents comparable across teams. Developers can use one agent record to connect traces, evals, datasets, and annotations, with local, dev, staging, and production environments grouped under the same ID to compare performance after changes; agents can run on other frameworks and runtimes. The new LangSmith is available in cloud, BYOC, and self-hosted environments, with LangSmith Studio or connecting an existing agent as starting points.

Why we rebuilt LangSmith around the agent
LangChain

LangChain introduced the LangSmith Agent Platform as a single place to build, run, improve, and govern agents across an organization . It makes agents the core object in LangSmith to connect fragmented developer workflows; teams can manage the full agent-development lifecycle from one place, using any stack, with ownership and governance across the organization .

Meet the new LangSmith Agent Platform: a single place to build, run, improve, and govern every agent across your org. Head of Product [@s… We made agents the core object across LangSmith, connecting fragmented workflows for developers and enabling platform teams to manage age…
Riley Brown

Riley Brown says “Nothing beats Claude Code Projects right now,” offering a strong current endorsement but no workflow details or comparison criteria.

I've gotten 50+ messages about this over the weekend. Nothing beats Claude Code Projects right now. [https://x.com/iamBarronRoth/status/2…
LangChain

For long-horizon agents, define goals around work that people can review and give feedback on, and make review intuitive: human feedback supplies tacit context and is often the performance bottleneck. Actively AI found that organizing agents by reusable job (“motions,” such as outbounding) rather than by customer account made similar tasks easier to review together and let feedback transfer across tasks; background agents separately gathered objective account context.

Goal Engineering: How to Design Goals
LangChain

LangSmith announced the Agent Platform as a single place to build, run, improve, and govern agents across an organization .

Meet the new LangSmith Agent Platform: a single place to build, run, improve, and govern every agent across your org. Head of Product [@s…
LangChain

JPMorgan Chase’s team described Jarvis, its internal agent assembly line, as turning months of agent building into minutes and scaling one agent into a fleet; the post links to the full session.

At Interrupt NYC, the [@JPMorgan](https://x.com/JPMorgan) Chase team took the stage to share how Jarvis, their internal agent assembly li…
LangChain

LangChain announced a hands-on workshop on Managed Deep Agents on LangSmith, where participants get practical experience building, testing, and deploying a production-ready data-analysis agent.

Happening tomorrow in SF! Join our hands-on workshop exploring Managed Deep Agents on LangSmith. You’ll get practical experience building…