# Jev’s Judge Trial Makes Agent Evaluation a Harness Primitive *By Coding Agents Alpha Tracker • September 20, 2026* LangChain’s narrow but unusually concrete Jev evaluation makes typed judgment, repeatability, and cost first-class coding-agent harness concerns; the practical follow-through is event-driven routing, cross-model execution, and model-aware instructions. ## 🔥 TOP SIGNAL **Jev is becoming a credible control-plane component for agent evaluation, not just another model to compare.** LangChain’s Daniel Shea and Seán Roche captured five weather-agent runs, replayed the identical traces across judges, and used human labels as the oracle. Jev matched the oracle on all 500 repeated binary decisions; its observed score variance was 92–913× lower than the other judges, and it averaged 0.44 seconds and $0.00035 per call in the test. [^1] Treat this as a strong harness signal, not a universal model verdict: the authors call the experiment early and narrow, and explicitly warn that a cheap evaluator can amplify systematic mistakes. Keep human review and judge-alignment checks in the loop. [^1] ## ⚡ TRY THIS - **Put a typed judge in every regression loop.** Capture fixed agent traces in LangSmith, replay the same traces across evaluator versions, and ask several atomic questions over the same state in one call. Use `Choice` for bounded judgments such as “is the answer grounded?”, `Score` for rubric-based usefulness, and `Noul` for yes/no-style decisions; track both agreement with a human oracle and repeatability, not variance alone. [^1] - **Turn external events into typed work queues.** Kent C. Dodds is triggering bots from email, Sentry, feedback submissions, package errors, and Discord; webhooks are the entry point, and the bot can explain how to configure them. Run the payload through a single decision pass for actionability, urgency, escalation, or “ready to ship,” then invoke a documented utility rather than asking an agent to improvise the final operation. Kody’s model is explicitly a skill plus a tool, intended to produce more reliable, repeatable outcomes; its merge-PR utility is the concrete example. [^2][^3][^4][^5] - **Separate model switching from terminal-session management.** Geoffrey Huntley’s current hands-on recommendation is OpenCode as one harness for different model options and easy switching, paired with Herdr as an agent-aware alternative to tmux. That directly answers the recurring Codex-versus-Claude problem, but keep the build/test boundary explicit rather than assuming a cloud coordinator can run local work. [^6][^7] - **Test instruction files per model before standardizing them.** John Lindquist wants a benchmark for the influence of `AGENTS.md` on each model; Huntley argues the convention was meant to be model-specific and proposes an `x-user-agent` signal so MCP servers can select model-specific system and tool prompts. Run the same coding task with a shared file versus model-specific variants, and log behavior, tool calls, and failures before declaring one instruction file universal. [^8][^9][^10] ## 📡 WHAT SHIPPED - **Pi 0.86.0:** adds mid-conversation system messages, dynamic tools on supported models without losing the KV cache, Anthropic cache warming, faster `-r`/`-c` resume, fixes, and a `/bug` command. Armin Ronacher warns that the system-message change may regress behavior because relatively few users run it from `main`; treat this as a canary upgrade, not a frictionless one. [^11][^12] - **Claude Projects’ boundary is now clearer in practice.** Riley Brown likes bundling sessions into projects with project-specific routines, but reports that the orchestrator cannot start a local Claude Code thread: every session runs in the cloud. That is convenient on iOS and awkward for app building, so use Projects for coordination only if the execution boundary fits your workflow. [^13] - **Bespoke Nimble is an open decision-model project worth watching.** The release includes open data, code, and a 9B model, with a LoRA fine-tune of Qwen3.5-9B, synthetic contrastive data, and Jev used for evaluation rather than distillation. The author reports 66% for base Qwen versus 90% for Nimble on a curated eval and 100 ms on an H100, but also says there is no standard benchmark and Nimble may perform much worse elsewhere. A current follow-up says it counted bedrooms and bathrooms in a confusing floor-plan image where base Qwen struggled; Jev was not tested because it lacks multimodal support. [^14][^15] ## 🎬 GO DEEPER - **Repo — [`jev-as-a-judge`](https://github.com/danielgshea/jev-as-a-judge):** Study the fixed-trace replay, human-oracle, repeatability, and cost methodology rather than copying the headline number. The article lists the package versions used for reproduction. [^1] - **Repo — [Bespoke Nimble](https://github.com/bespokelabsai/nimble):** Read the contrastive-data recipe, then look for the missing ablation and out-of-domain benchmark before treating the reported lift as general. [^14] - **Video — [Geoffrey Huntley’s AGENTS.md discussion](https://youtu.be/8gotGDhQKeQ?si=k7A8r7WI0T-Ck5VX):** Pair it with his follow-up proposal for model identity in MCP; the useful question is whether instructions and tool prompts should vary with the active model. [^9][^10] **Editorial take:** The practical frontier is a separated control plane—typed decisions, event triggers, model switching, and repeatable evals—wrapped around a coding agent whose local or cloud execution boundary is explicit. [^1][^6][^13] --- ### Sources [^1]: [𝕏 article by @LangChain](https://x.com/i/article/2101448785255907328) [^2]: [𝕏 post by @kentcdodds](https://x.com/kentcdodds/status/2101229030007615609) [^3]: [𝕏 post by @mattyp](https://x.com/mattyp/status/2101130160993092094) [^4]: [𝕏 post by @kentcdodds](https://x.com/kentcdodds/status/2101158890972868789) [^5]: [𝕏 post by @kentcdodds](https://x.com/kentcdodds/status/2101341166545543318) [^6]: [𝕏 post by @GeoffreyHuntley](https://x.com/GeoffreyHuntley/status/2101524492975796548) [^7]: [𝕏 post by @DanielSmidstrup](https://x.com/DanielSmidstrup/status/2101348710181482973) [^8]: [𝕏 post by @johnlindquist](https://x.com/johnlindquist/status/2101387505312653641) [^9]: [𝕏 post by @GeoffreyHuntley](https://x.com/GeoffreyHuntley/status/2101472511599026277) [^10]: [𝕏 post by @GeoffreyHuntley](https://x.com/GeoffreyHuntley/status/2101472819905536457) [^11]: [𝕏 post by @pidotdev](https://x.com/pidotdev/status/2101452506203066631) [^12]: [𝕏 post by @mitsuhiko](https://x.com/mitsuhiko/status/2101452864618926406) [^13]: [𝕏 post by @rileybrown](https://x.com/rileybrown/status/2101453842319618291) [^14]: [𝕏 post by @madiator](https://x.com/madiator/status/2100990591215783946) [^15]: [𝕏 post by @madiator](https://x.com/madiator/status/2101368806212030733)