We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
🔥 TOP SIGNAL
Jev is becoming a credible control-plane component for agent evaluation, not just another model to compare. LangChain’s Daniel Shea and Seán Roche captured five weather-agent runs, replayed the identical traces across judges, and used human labels as the oracle. Jev matched the oracle on all 500 repeated binary decisions; its observed score variance was 92–913× lower than the other judges, and it averaged 0.44 seconds and $0.00035 per call in the test.
Treat this as a strong harness signal, not a universal model verdict: the authors call the experiment early and narrow, and explicitly warn that a cheap evaluator can amplify systematic mistakes. Keep human review and judge-alignment checks in the loop.
⚡ TRY THIS
Put a typed judge in every regression loop. Capture fixed agent traces in LangSmith, replay the same traces across evaluator versions, and ask several atomic questions over the same state in one call. Use
Choicefor bounded judgments such as “is the answer grounded?”,Scorefor rubric-based usefulness, andNoulfor yes/no-style decisions; track both agreement with a human oracle and repeatability, not variance alone.Turn external events into typed work queues. Kent C. Dodds is triggering bots from email, Sentry, feedback submissions, package errors, and Discord; webhooks are the entry point, and the bot can explain how to configure them. Run the payload through a single decision pass for actionability, urgency, escalation, or “ready to ship,” then invoke a documented utility rather than asking an agent to improvise the final operation. Kody’s model is explicitly a skill plus a tool, intended to produce more reliable, repeatable outcomes; its merge-PR utility is the concrete example.
Separate model switching from terminal-session management. Geoffrey Huntley’s current hands-on recommendation is OpenCode as one harness for different model options and easy switching, paired with Herdr as an agent-aware alternative to tmux. That directly answers the recurring Codex-versus-Claude problem, but keep the build/test boundary explicit rather than assuming a cloud coordinator can run local work.
Test instruction files per model before standardizing them. John Lindquist wants a benchmark for the influence of
AGENTS.mdon each model; Huntley argues the convention was meant to be model-specific and proposes anx-user-agentsignal so MCP servers can select model-specific system and tool prompts. Run the same coding task with a shared file versus model-specific variants, and log behavior, tool calls, and failures before declaring one instruction file universal.
📡 WHAT SHIPPED
Pi 0.86.0: adds mid-conversation system messages, dynamic tools on supported models without losing the KV cache, Anthropic cache warming, faster
-r/-cresume, fixes, and a/bugcommand. Armin Ronacher warns that the system-message change may regress behavior because relatively few users run it frommain; treat this as a canary upgrade, not a frictionless one.Claude Projects’ boundary is now clearer in practice. Riley Brown likes bundling sessions into projects with project-specific routines, but reports that the orchestrator cannot start a local Claude Code thread: every session runs in the cloud. That is convenient on iOS and awkward for app building, so use Projects for coordination only if the execution boundary fits your workflow.
Bespoke Nimble is an open decision-model project worth watching. The release includes open data, code, and a 9B model, with a LoRA fine-tune of Qwen3.5-9B, synthetic contrastive data, and Jev used for evaluation rather than distillation. The author reports 66% for base Qwen versus 90% for Nimble on a curated eval and 100 ms on an H100, but also says there is no standard benchmark and Nimble may perform much worse elsewhere. A current follow-up says it counted bedrooms and bathrooms in a confusing floor-plan image where base Qwen struggled; Jev was not tested because it lacks multimodal support.
🎬 GO DEEPER
Repo —
jev-as-a-judge: Study the fixed-trace replay, human-oracle, repeatability, and cost methodology rather than copying the headline number. The article lists the package versions used for reproduction.Repo — Bespoke Nimble: Read the contrastive-data recipe, then look for the missing ablation and out-of-domain benchmark before treating the reported lift as general.
Video — Geoffrey Huntley’s AGENTS.md discussion: Pair it with his follow-up proposal for model identity in MCP; the useful question is whether instructions and tool prompts should vary with the active model.
Editorial take: The practical frontier is a separated control plane—typed decisions, event triggers, model switching, and repeatable evals—wrapped around a coding agent whose local or cloud execution boundary is explicit.
- Firsthand LangChain evaluation workflow: Daniel Shea and Seán Roche tested TypeSafe AI’s Jev, a “System One” decision model that returns typed answers and probabilities instead of generated text, against GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6. They built a weather agent with the open-source Deep Agents harness, defined five requests in a LangSmith dataset, captured each full run once, replayed the identical traces across judges, measured continuous quality and binary
does_pass, and used human labels as the oracle. - Results in this narrow test: Jev matched the human oracle on all 500 repeated binary decisions, versus 99.8% for Terra, 96.4% for Luna, and 80.0% for Claude. Its mean per-case score variance was 0.0000149—433× lower than Luna, 913× lower than Terra, and 92× lower than Claude. Jev averaged 0.44 seconds and $0.00035 per call, with $0.34 total cost versus $28.17 for Claude in the experiment.
- Actionable pattern and caveats: Use typed
Choicequestions for bounded judgments such as evidence grounding andScorequestions for rubric-based usefulness, and evaluate multiple atomic questions in parallel against the same agent state. For online monitoring, the authors define signal value as binary oracle agreement multiplied by repeatability—the probability that two independent calls on the same trace agree—so teams can favor judges that are both accurate and stable. The experiment is reproducible via the jev-as-a-judge repository; it usedlangchain-typesafe==0.0.1a2, Deep Agents0.7.15, LangChain OpenAI1.6.2, LangSmith0.12.6, and Tavily Python0.8.3, with LLM sampling parameters left at provider defaults. The authors stress that the result is early and narrow, does not prove why Jev had lower variance, and that cheap evaluators can amplify systematic mistakes; retain human review and judge-alignment checks before relying on it in production.
- @madiator released Bespoke Nimble, an open-data, open-model, open-recipe 9B model. The release includes the GitHub repository and Hugging Face model.
- The training recipe uses fully synthetic data across 10 categories and a proposed contrastive data curation method: slightly alter facts to create negative examples, encouraging better discrimination and decision-making without probability labels. The author explicitly notes that this key component was not ablated, so its contribution remains unverified.
- Reproduction details: LoRA fine-tuning of Qwen3.5-9B, Jev used only for evaluation rather than distillation, no reinforcement learning, and parallel constrained decoding for serving.
- On the authors’ curated evaluation, Qwen scored 66%, Nimble 90%, and Jev 93%; they report 100 ms latency on an H100 and free use on a MacBook. These results are not standardized: the author cautions that Nimble could perform much worse than Jev on other benchmarks.
- Nimble also processed a confusing floor-plan image, counting bedrooms and bathrooms where base Qwen struggled; Jev was not tested because it lacks multimodal support.
Kent C. Dodds reports a firsthand, roughly 18-hour DevinAI run on an unspecified task that he says will interest self-hosters; he provides no workflow, outcome, or link yet.
Kent C. Dodds is narrowing Kody’s positioning around making it easy for developers to use multiple coding agents and switch between them seamlessly, arguing that large AI labs are not incentivized to prioritize cross-agent interoperability. He also considers positioning Kody against “skills,” but provides no concrete comparison or implementation details.
- Kent C. Dodds says Jev is now easier to use in Kodykoala and links to the package at kody.codes/@kentcdodds/jev. He adds that creating packages like this with Kody is “almost always a one-shot.”
-
Practical pattern: use Jev for structured decisions over one shared state—for example, deciding whether a draft or PR is ready to ship and assigning a rough quality score—by asking several
choice,score, andnoulquestions in a single call.
Michael Chomsky announced a free classification project for personal agents at classifier.dev, claiming that the classifier improves agents’ capability and performance. The announcement provides no implementation steps, configuration, or quantitative evidence.
-
Geoffrey Huntley says
AGENTS.mdwas intended as a per-model convention, but became a single shared file; his premise is that different models have distinct “tastes and preferences.” Actionable takeaway: test and maintain model-specific agent instructions instead of assuming one universal file works equally well across models. -
Huntley proposes that MCP clients identify the active model through an
x-user-agentsignal, enabling local and remote MCP servers to select model-specific system and tool prompts. - John Lindquist calls for a benchmark measuring “the influence of AGENTS.md” on each model, reinforcing a practical evaluation pattern: compare agent behavior across models with controlled instruction-file variants.
Peter Pme reports a contrarian firsthand result: “I nuked away all my ai/agent skills 2 days ago. I have not noticed a difference. It might even be slightly better.” The post provides no tool names, skill definitions, task scope, or quantitative measurement, making this a low-confidence caution against assuming persistent agent-skill customization is beneficial. Kent C. Dodds amplified the observation with “Many such cases.”
- Kody is listed with a free plan, webhooks for GitHub, Sentry, and Stripe, and self-hosting on Cloudflare Workers; the listing links to its full integration list and documentation: https://www.everydev.ai/tools/kody. Kent C. Dodds only described the listing as “neat,” so this is a tool-discovery lead rather than a firsthand workflow or evaluation.
DHH is working with 0xSero to make local AI accessible through Omarchy, advocating for users’ own GPUs running free, “unnerfed” models rather than intelligence being limited to a subscription. 0xSero describes Omarchy as an “Agent Operating system”; the full interview is available on YouTube.
Claude Code added support for AGENTS.md starting in version 2.1.277: when a folder has no CLAUDE.md, Claude checks for and uses AGENTS.md; developers can toggle this behavior in /config. Kent C. Dodds amplified the change and called for a .agents convention.
- Kent C. Dodds recommends documented utilities over standalone agent skills: combine the instructions of a skill with an executable tool to produce more reliable and repeatable outcomes. In Kody, agents receive this utility when they search for “merge a pull request.”
-
The linked Kody implementation is
src/pr/merge.ts; the post’s code description says the TypeScript utility callsrunMergewithescalate: true, using a direct merge path with a durable Kody workflow fallback.
Daniel Smidstrup reports switching between Codex and Claude and asks for a single harness that works with both, highlighting cross-agent interoperability as an unresolved workflow need.
Kent C. Dodds shared Kody, a project for turning skills into code that can be run from any agent, any device, or a custom app. He linked Kody.codes as the project site.
Kent C. Dodds describes an event-driven Grok bot workflow: he triggers bots from email, Sentry, feedback submissions, package errors, and Discord. A linked post says Grok bots support webhooks and recommends asking the bot how to set them up. Further details: https://kody.codes/@kentcdodds/grok-bot
Riley Brown’s firsthand take on Claude Code Projects: the feature bundles sessions into projects with project-specific routines (crons), but its orchestrator cannot start a local Claude Code thread and every session runs in the cloud. Brown says this makes Projects convenient on iOS but less suitable for building apps.
Geoffrey Huntley reports firsthand experimentation with multiple tools and recommends OpenCode as a single harness for using and easily switching between different model options. He describes Herdr as an agent-aware alternative to tmux, suggesting a workflow that combines OpenCode for model/harness flexibility with Herdr for terminal session management.
Geoffrey Huntley argues that the OpenAI-versus-Claude debate is largely about monthly-plan subsidies and access rather than a straightforward model comparison. He notes that buying five Claude subscriptions would still force him to use Claude’s harness unless he resorted to “grey hatting,” which he declines—highlighting harness lock-in and access constraints as practical factors in coding-agent selection.
- Pi 0.86.0 adds mid-conversation system messages, dynamic tools on supported models without losing the KV cache, Anthropic cache warming, faster
-r/-cresume, fixes, and a/bugcommand; release notes: https://pi.dev/news/releases/0.86.0 - Adoption caution: Armin Ronacher warns that the mid-conversation system-message change may cause regressions. It was tested for some time, but relatively few users run it from
main; users who find problems should report them via/bug.
Riley Brown highlights Claude Code’s new “Projects” feature as excellent and shares a demonstration video.