We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Test new models on your own repo: Theo finds Gemini 4 Argon wrong on most claims
Theo says Gemini 4 Argon is "outright unusable for real code work in my codebases" because "it aggressively hallucinates about how systems work" . In almost every test he ran, more than half of its claims were wrong .
His method is worth copying more than his verdict:
- Have each model audit a real codebase.
- Check every claim it makes.
- Have a panel of judges score the results .
On his ts-rust audit, GPT-6.1 Sol made 38 claims, all true. Argon scored 4.38, and 24 of its 36 claims were false . On his Lakebed audit, 56 of Argon's 78 findings were wrong . He notes this contradicts Artificial Analysis's low hallucination rate for the model . His advice: if you do use Argon, run its output through Sol or Opus .
The SQLite case shows the failure pattern and how to recover:
- Argon's diagnosis was mostly right, but its PR was about 1,200 lines for a change Theo says should have been about 30. It ignored the repo's PR-format skills, failed CI and lint, and left review-bot comments unaddressed .
- He handed the thread to Opus 5.5 for a second-agent review. Opus found that Argon fixed the wrong half of its own diagnosis and missed two existing PRs that already solved the problem .
- The final fix was 21 lines and has been merged .
For frontend work, Theo recommends Anthropic's official design skill but not its taste skill . He prefers fast models he can iterate with over expecting a good first attempt .
Salvatore Sanfilippo (in Italian) says Argon trails Opus 5, GPT-6.1 Sol and GPT Astra in agentic coding . He also says coding works well at around 250k–350k tokens of context .
Osmani: E2E, property and differential tests the agent can run
The last brief covered Addy Osmani's "define done" advice. His new post adds the specific kinds of test to invest in. His premise is that the bottleneck is now verifying code, not writing it, and he would "always prefer a much better feedback loop to a slightly better model" .
- E2E tests as ground truth. Keep a small, high-value set . Failures should report which step failed, with screenshots and traces, and "Make this E2E test pass" gives the agent a clear definition of done . Via poteto: give the agent a maintained skill or CLI to launch, drive and observe the app, plus a user-POV feature map .
- Property tests (fast-check, Hypothesis, proptest). These stop the agent from passing only test examples it picked itself . Start with a pure function, write the property in plain English, save minimized failures as regression cases, and ask the agent for properties rather than examples .
- Differential tests for migrations. Treat the old system as the oracle. Feed both systems random inputs or replayed live traffic , normalize irrelevant differences, and keep a list of approved changes .
- Fast, deterministic loops. Slow or flaky tests teach the agent to retry rather than fix . Aim for a subsecond core loop with seeded randomness, frozen clocks and isolated flaky tests .
LangChain's evals FAQ makes the same point for agents in general. To evaluate a coding agent, run the repo's tests after it finishes, with a clean copy of the repo for every run. Its open-source Harbor harness sets this up . Start with 10–20 examples taken from traces, save a baseline, and add each new failure as a case . Use narrow, binary questions for LLM judges . LangChain's Eval Engineering Skill runs in Claude Code or Codex and helps write evals for your repo .
LangChain also explained how its Deep Agents harness (uv add deepagents) manages context:
- Tool results over about 20k tokens are saved to a file, and the model sees a preview plus the file path .
- At 85% context use, older messages are summarized into a file the agent can read back .
- Memory is a Markdown file you own, so it carries over if you switch providers .
Tooling for many agents at once
Anthropic says it went from about 2,500 merged changes a day in May to more than 30,000 in September. Infrastructure built for humans became the bottleneck, so it built a new version control system and moved its monorepo off git in five months .
At a smaller scale, Kent C. Dodds' Kody now queues git pushes in a durable object. When 20 agents pushed to the same branch at once, some had to rebase and resolve conflicts, but all changes landed .
The Claude Code desktop app now starts sessions faster, makes it easier to split work across sessions, and shows what each one is doing . Theo says this validates UX experiments T3 Code tried first . Claude Platform traces from the Messages API, Managed Agents and the SDKs can now go to LangSmith over OpenTelemetry, in public beta (docs) .
Huntley: open source that accepts remixes, not PRs
Geoffrey Huntley is closing pull requests on Kittens but keeping issues and discussions open. To change it, you "point your agent at it" and patch it with Nix overlays . In his "remix model", a small overlay describes the change once and is reused across devenv shells, Docker images, NixOS tests, CI and agent sandboxes, with no fork to maintain . His example is an overlay that makes Git refuse force pushes . He suggests running agents over GitHub's network graph to summarize what people are remixing .
Tossell's AGENTS.md habits
From Ben Tossell's Stanford GSB talk:
- He puts "Questions are requests for an answer, not for changes" in his AGENTS.md, so a question doesn't trigger edits .
- He routes task-specific context (travel, his fund) to separate files .
- Start with one agent, and promote anything you keep repeating into the instructions .
- His instructions ask for three distinct static mockups for anything he builds .
- When the agent misses, he adds context step by step: screenshots, voice rambles, then screen recordings or annotated tldraw screenshots .
Smaller notes
- Theo logged all his Claude and Codex requests to a ClickHouse database . The data shows newer models increasingly running Python instead of bash or other tools , and his own near-complete move from Codex to Claude . He also points out that Codex's system prompt still doesn't name the current model, only "based on GPT-6" .
- T3 Code has 540k users . Nightly adds SQLite performance fixes meant to stop the "reconnecting" banners .
- Romain Huet (OpenAI) says GPT-6.1 Sol Ultrafast is up to 8x faster than standard mode .
- Greptile has committed $150,000 in tokens for agent PR reviews across all Omarchy repos .
- A warning from Ben Tossell: his Grok bot listed run times for an hourly email-triage job that wasn't actually running . Check agent-reported status yourself.
-
Use an
AGENTS.mdas a persistent working agreement: set interaction preferences (for example, concise answers and “questions are requests for an answer, not for changes”), and route task-specific context to separate files. Start with one agent and split into specialized agents only when useful; promote instructions you find yourself repeating into the defaults. - For uncertain UI work, ask for several distinct static mockups, compare them by clicking through, and choose a direction to refine; Tossell found a designer’s prototyping skill produced better options than theme variations alone. When the agent misses, add context progressively—specific text, voice rambling, screenshots with annotations, then screen recordings—and try the same prompt with different agents.
-
His stated tool split: ChatGPT/Codex for the app interface, Claude models for creative, visual, and taste-oriented tasks; Factory’s Droid can use open-source, Claude, and OpenAI models. For sharing prototypes, he uses the
here.nowdeployment skill to get a URL, then sets up automations when information needs regular updates. - Treat this as a prototyping workflow, not evidence of production-grade assurance: Tossell says his casual builds are not software real people rely on; for a course site with authentication and payments, he uses model reviews and Cursor’s “Thermonuclear Code Quality Review” skill to probe for vulnerabilities.
- Theo’s code-agent evaluation method is to run audits on real codebases, verify each claim, then have a judge panel score the results; in one audit, Gemini 4 Argan scored 4.38 and 24 of 36 claims were false after 43 minutes, underscoring the value of checking models against task-specific repo work rather than relying on general benchmarks.
- In a real SQLite fix, Gemini’s diagnosis was mostly right, but its PR was about 1,200 lines for a change Theo said should have been about 30; it ignored repo PR rules, failed checks and lint, and left review-bot comments unresolved. A second-agent review with Opus 5.5 found that Gemini fixed the wrong part of its diagnosis and missed two existing PRs addressing the problem; the eventual change was 21 lines and merged. A practical safeguard is to verify the diagnosis, check prior art, and review CI and the proposed diff before merging.
- For frontend work, Theo recommends Anthropic’s design skill, but not its taste skill; he favors iterating through variants with a fast model rather than expecting a good first pass, especially for real UX.
- Sanfilippo says Gemini 4 Argon has impressive benchmarks but trails Dopus 5, GPT Sol 6.1, and GPT Astra in agentic coding; he cautions that the comparison is with competitors whose next models were already ready or whose current models were in some respects outdated.
- As a practical context-length guideline, he says coding works well around 250,000–350,000 tokens when models can retrieve information and search their own session history.
- He sees Google as targeting broad frontier intelligence rather than matching Codex and Claude on coding, and describes Gemini 4 as especially strong with language and complex documents. His own brief test was in chat, where he found it concise and good at retaining details—not a reported coding-agent trial.
- Define the check before coding, then let the agent run it, inspect results, and iterate; the post argues this feedback loop can matter more than a slightly better model. Capture recurring mistakes in shared config, hooks, or tests so people can review evidence rather than every line.
- Use a small set of high-value end-to-end tests as real-user ground truth and a clear “done” target; make failures informative with screenshots, traces, and the failed step, while accounting for E2E tests being slower and flakier than unit tests. The post also relays Lauren’s recommendation to give agents a maintained skill or CLI to launch, drive, and observe the app (for example, via CDP, a simulator, or an API harness), alongside a user-facing feature map.
- Add property-based tests for rules that must always hold: tools such as fast-check (JS/TS), Hypothesis (Python), and proptest (Rust) generate many inputs to find counterexamples. A practical start is a pure function, a property written in plain English, and saving minimized failures as regression cases.
- For rewrites or migrations, compare old and new systems on the same randomly generated inputs or sampled live traffic; normalize irrelevant differences and track intentional changes in an approved-change list. Keep the core loop fast and deterministic (subsecond is a stated target), isolate flaky tests, and run heavier E2E and differential checks before integration.
- Iterate and review instead of accepting a one-shot result: In Logan’s Minecraft-clone test, one prompt produced the visuals and a second added game depth and the HUD he wanted. When an output feels like a one-shot, refine the prompt; read the result, question anything that seems off, and run another iteration.
- Antigravity team members used Argon to migrate external Fortran codebases to Rust, and the Chromium team used it on its work. For infrastructure debugging, an engineer gave Argon a recurring machine-state problem; the hardware team and provider confirmed a hardware issue, after which affected vendor machines could be identified and removed from the fleet.
- A Googler’s reading-app project illustrates a research-before-coding workflow: Argon researched early-childhood learning theory, incorporated it into the app, built the front end, and hosted it.
- Leash Bio’s agent workflow uses open-source Qwen and GLM models taught to describe molecular structures and differences in natural English, then explain why examples bind or do not bind, including hard negatives. Given 10 binders and 10 non-binders, an agent proposes a binding hypothesis, selects molecules from a catalog, looks up existing screening results immediately in a coding-sandbox-like loop, and revises its hypothesis; the current setup gives it a 100-molecule budget to find strong binders among 6.5 million screened molecules. This is an iterative database-based workflow, not a live lab experiment; selecting compounds outside the screened set for later ordering and testing is a future step.
- The team views natural-language reasoning as a way to reduce memorization, and stresses including counterintuitive choices and counterfactual examples so feedback is not biased toward medicinal chemists’ usual selection patterns. They report extracting more than 10,000 structure–activity relationships in one weekend, versus perhaps a few dozen to 50 across a medicinal chemist’s career.
Leash Bio’s Minos turns its collected assay data into tens of thousands of “lab-in-the-loop” simulations for verifiable scientific reasoning and scientific RL. Its agent sandbox has LLMs reason about structure–activity relationships (SAR) in plain English while searching 6.5 million molecules on a budget of 100.
Huntley proposes a “remix” workflow for open-source software: instead of sending a pull request or maintaining a fork, consumers express a change in a small Nix overlay and reuse it across devenv shells, Docker images, NixOS systems, tests, CI, and agent sandboxes. His example is an overlay that makes Git refuse force pushes, so the behavior is consistent across those environments. Agents make remixing easier to explore: users can update overlays with an agent, run agents over GitHub’s network graph to summarize remixes, or use an agent loop to explore ideas and feed them upstream. Huntley says he will disable pull requests on his GitHub code while keeping discussions open and publishing guidance for remixing it.
In a sponsored demo, Blacksmith presented Codesmith as a cloud coding agent that uses repository and CI context: a request to add an AI provider and configure its API key across an app and infrastructure repo could result in a PR in each, with the agent also able to address failing tests or review comments. It can recommend runner sizes from CI history and turn those recommendations into a PR. Blacksmith also advertised its drop-in GitHub Actions runner replacement as twice as fast at 75% lower cost, and offered 3,000 GitHub Actions minutes to try it.
A Grok bot claimed the user's hourly email-triage automation was scheduled and listed run times, but the user says the automation was not running—an example of why agent-reported task status should be independently checked.
- To limit approval fatigue without blocking useful autonomy, automate routine requests and send people only the important ones; use deterministic rules for clear cases and classifiers for ambiguous ones.
- Keep proposal separate from execution: a trade-break agent recommends order amendments for human approval rather than changing order data itself. A reusable trust layer can combine agent identity and permissions, sandboxing and approval, and checks that the agent still works as conditions change.
- Enforce security outside the model with scoped identities and credentials, tool-call controls, outbound-traffic restrictions, audit trails, budgets, and kill switches; check and log every call.
Theo sarcastically compared the $18,000 local AI server with eight years of $200 Claude subscriptions, portraying the team tradeoff as using quantized DeepSeek V4 Flash at 30 tps—a cost-and-UX critique, not a measured comparison in the post. The linked setup advertises 4 Radeon AI PRO R9700 GPUs, 128GB total VRAM, and 16 concurrent sessions; its Qwen3.8-Flash-Next figures are aggregate decode rates of 538.5 tps at 48K context per user and 365 tps at 128K.
Kody handles Git-push concurrency with a queue managed in a durable object; 20 agents pushing to 20 separate branches worked without issue. When 20 agents pushed to the same branch simultaneously, some had to rebase and manage conflicts, but all changes landed.
- Deep Agents, LangChain’s open-source harness, manages context rot by moving information out of the active window without deleting it, using offloading, summarization, skills, subagents, and persistent memory. Tool results over roughly 20,000 tokens are saved to files and replaced with a preview and file path; by default, when context usage reaches 85%, older messages are summarized while recent messages remain verbatim, and the history can be retrieved from a file.
- Use progressive disclosure for task-specific know-how: keep instructions, scripts, and reference docs in skill folders, show the agent skill headers and descriptions up front, and load full instructions only when relevant. For work expected to be noisy, use a scoped subagent that returns only a concise result; the example researcher returns three bullets and may run on a cheaper model or in parallel.
-
Store persistent agent memory as a Markdown file in storage you control; middleware loads it at run start, injects it into prompts, and updates it, so it can be reused after switching models or providers. Install Deep Agents with
uv add deepagents.
Riley Brown uses his dot with only AirPods while walking to voice-start new Codex sessions and work on website changes, build an app, and respond to email and Slack—a hands-free workflow spanning coding and communication tasks. He also says using “ultrafast” has been a “crazy experience,” without specifying further details.
Geoffrey Huntley plans to close pull requests on Kittens while keeping issues and discussions open; his suggested customization workflow is to use Nix overlays and point an agent at the project to personalize it for your needs.
LangChain says that as an agent accumulates instructions, tool results, files, and drafts, accuracy and reasoning can decline; its Deep Agents harness addresses this by moving context out of the active window without deleting it.
Anthropic says daily merged code changes rose from about 2,500 in May to more than 30,000 in September, bottlenecking infrastructure designed for humans rather than agents; it built a new version-control system and migrated its monorepo from Git in five months. Anthropic also says its internal token spending capacity exceeds what any other business would realistically want to spend.
LangSmith now accepts Claude Platform traces via native OpenTelemetry support in public beta; developers can export traces from Claude Managed Agents, the Messages API, or Claude SDKs to inspect agent steps and improve performance.
- Start agent evals by choosing one behavior to test, building a 10–20-example dataset from traces, saving a baseline, and adding each new failure as a case; run offline evals before shipping and online evals on production traffic, feeding online failures back into the offline dataset.
- For coding agents, evaluate the tool-call path and the repo’s final state—not only the final reply—and run tests in a clean repo copy for each run. Harbor provides isolated task containers, supports custom benchmarks, and logs runs to LangSmith experiments.
- Prefer deterministic code checks where possible, using LLM judges for qualities code cannot assess; calibrate judges against human ratings and ask narrow, preferably binary questions. LangChain’s Eval Engineering Skill can run in Claude Code or Codex, read a repo and its traces, and help write runnable evals.