# ReviewBench Makes the Coding-Agent Bottleneck Measurable *By Coding Agents Alpha Tracker • August 1, 2026* LangChain’s new real-PR benchmark finds that basic coding-review agents recover only about 30% of trusted findings, while a structured review loop lifts Luna to 0.32. The practical response is measurable harnesses, resumable execution, and auditable tool surfaces. ## 🔥 TOP SIGNAL LangChain built ReviewBench from trusted reviewer comments in its LangSmith monorepo, turning them into reproducible Harbor tasks; it currently has 59 tasks covering 64 baseline issues. [^1] Under the same minimal Deep Agents harness and with no review-specific system prompt, the strongest runs recovered only about 30% of curated findings; Luna and Terra often stopped after a few obvious findings, saving cost at the expense of coverage. [^1] On a matched 20-task slice, giving Luna high reasoning effort and a structured review prompt—without adding tools—raised its score to 0.32, above the static-review Kimi and Opus runs; LangChain’s conclusion is that better review strategy can come from changing how the agent reviews, not only from changing the model or adding tools. [^1] ## ⚡ TRY THIS - **Use a change-map review loop.** Give the agent repository-wide read/search access and require structured findings with a location, title, and explanation. Prompt it to: “identify what the PR changed, trace how the surrounding system depended on that behavior, and validate your findings against callers, tests, and related implementations.” This is the only substantive change in LangChain’s tuned Luna comparison; it used no new tools. [^1] - **Regression-test your harness edits with `smevals`.** Ask your coding agent to run `uvx smevals docs`, have it create an eval directory of YAML files, then run `uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6`. Grade separately with `uvx smevals grade path-to-eval/`, and inspect or publish results with `smevals serve` or `smevals build`. Define configs for the model, system prompt, parameters, and harness; use deterministic checks or an LLM-as-judge where appropriate. [^2] - **Make long-running agents resumable by construction.** Put scratchpad and task state in files; use summarization, planning, and subagents to keep only the minimum context in the model window. In LangChain, changing `create_agent` to `create_deep_agent` adds filesystem middleware, planning, `ls`/`cat`/`grep`, and a subagent tool. Checkpoint every execution step so a failure at step 67 can resume from step 66, and expose interrupt, stop, approval, and progress streaming to the human operator. [^3] - **Spend cheap tokens on parallel work, then measure quality.** For low-ambiguity classification, start with a batch-mode worker and validate a sample with an eval suite. Greg Kamradt reports that an overnight GPT-5.6 Luna run handled 78K classifications through 158K requests and 143M input tokens for $60—a throughput/cost signal, not a substitute for checking correctness. [^4] ## 📡 WHAT SHIPPED - **Stateless MCP / MCP 2.0.** The new protocol replaces the legacy initialize-plus-session flow with a single HTTP tool request carrying `MCP-Protocol-Version`, `Mcp-Method`, and `Mcp-Name`; servers no longer need session state or backend affinity, which is a better fit for scalable web apps. [^5] Simon Willison’s practical stack is already here: [`mcp-explorer`](https://github.com/simonw/mcp-explorer) probes a server with `uvx`, then lists, inspects, and calls tools; [`datasette-mcp`](https://github.com/datasette/datasette-mcp) exposes three tools—including read-only SQL—at a Datasette `/-/mcp` endpoint; and the alpha [`llm-mcp-client`](https://github.com/simonw/llm-mcp-client) wires MCP into the LLM CLI. [^5] Willison’s rationale is especially relevant to coding agents: controlled MCP tools are easier to audit than arbitrary shell/`curl` access for sensitive applications. [^5] - **`smevals`.** Simon Willison and Prime Radiant released a small eval tool for running suites across model configurations and grading the results; its config vocabulary explicitly covers system prompts, model parameters, and agent harnesses, with execution separated from grading. [^2] - **DeepSeek-V4-Flash-0731.** The new 304B-parameter, 167GB model is priced at $0.14/M input and $0.27/M output; Simon’s post reports Artificial Analysis ranking it ahead of 428B MiniMax M3. His firsthand test was image generation: default reasoning produced a poor result, while `-o reasoning_effort high` produced a much better one. It is worth testing as a cheap coding-agent worker, but the evidence here is not a code benchmark. [^6] - **Buzz’s multi-agent surface is becoming more concrete.** Buzz is a free, open-source Block/@jack project that puts Codex and Claude Code agents in a group-chat-like desktop/mobile interface, with separate identities and permissions. [^7] Riley Brown reports that he and Vishal Dubey used it for a week, then Vishal used Fable 5 inside Buzz to build a hosted version whose cloud agents can spawn more agents while sharing skills, API keys, workspace, files, and automations; they explicitly describe this as an experiment in agent-and-human teams. [^8] - **OpenAI is upstreaming its Git work.** An OpenAI team member says performance, correctness, and testing improvements from `openai/git` are flowing upstream, while new Codex builds use Git more efficiently. [^9] ## 🎬 GO DEEPER - **Building Deep Agents and Deploying in Production** — The durable-execution segment is the useful part: context limits make long tasks increasingly failure-prone, so checkpointing, memory, auth boundaries, and human approval need to be runtime primitives rather than afterthoughts. [^3] [![Building Deep Agents and Deploying in Production](https://img.youtube.com/vi/IZabCqyBJLg/hqdefault.jpg)](https://youtube.com/watch?v=IZabCqyBJLg&t=587) *Building Deep Agents and Deploying in Production (9:47)* - **Stop Being Tricked** — Use this as an anti-hype calibration for one-shot coding demos: a game appeared in minutes, but two hours of Opus 5 iteration cost $117 and 72M tokens and still reached only about 25% of the creator’s intended result; the demo-to-finished-product gap remains the work. [^10] [![Stop Being Tricked](https://img.youtube.com/vi/LmXU6SEH3Ks/hqdefault.jpg)](https://youtube.com/watch?v=LmXU6SEH3Ks&t=290) *Stop Being Tricked (4:50)* - **Repos worth studying:** [`mcp-explorer`](https://github.com/simonw/mcp-explorer) for the probe → inspect → call workflow, and [`datasette-mcp`](https://github.com/datasette/datasette-mcp) for a deliberately small, read-only tool surface that an agent can query. [^5] *Editorial take: The coding-agent edge is moving from “which model?” to “which review loop, state boundary, and tool surface can you measure, audit, and recover?”* [^1][^3][^5] --- ### Sources [^1]: [𝕏 article by @LangChain](https://x.com/i/article/2083185690137792512) [^2]: [smevals - a small eval suite for evaluating models, prompts, and harnesses](https://simonwillison.net/2026/Jul/31/smevals) [^3]: [Building Deep Agents and Deploying in Production](https://www.youtube.com/watch?v=IZabCqyBJLg) [^4]: [𝕏 post by @GregKamradt](https://x.com/GregKamradt/status/2083228765594018157) [^5]: [Stateless MCP has recaptured my interest \(and inspired mcp-explorer and datasette-mcp\)](https://simonwillison.net/2026/Jul/31/stateless-mcp) [^6]: [deepseek-ai/DeepSeek-V4-Flash-0731](https://simonwillison.net/2026/Jul/31/deepseek-v4-flash-0731) [^7]: [𝕏 article by @agentnative_](https://x.com/i/article/2083249851458846720) [^8]: [𝕏 post by @rileybrown](https://x.com/rileybrown/status/2083364318666490206) [^9]: [𝕏 post by @tnm](https://x.com/tnm/status/2083328099975119044) [^10]: [Stop Being Tricked](https://www.youtube.com/watch?v=LmXU6SEH3Ks)