Direct answer:lemans is now open source and is described as the harness behind the published Agents on Rails benchmark numbers. This update adds four models: Sonnet 5, Terra, an open-weight Qwen, and an unnamed model later identified as ox-alpha. rubyonrails

What the suite measures and how it grades:

• lemans is a Ruby, CLI-first benchmark/library designed around convention over configuration, isolated sandboxes, and safeguards intended to prevent agents from tampering with verification. rubyonrails
• Runs can target any model supported by RubyLLM; Daytona is the primary sandbox provider, with a Docker backend also available. Tasks are selected individually or run by default as a full suite. rubyonrails
• Each trial stores a result record and evidence such as the agent trajectory, checks, and verifier log. rubyonrails
• Reports expose reward, outcome, cost in USD, steps, tokens, and duration; aggregate reports add score, time, cost, and solved/pass@3 results. rubyonrails
• Grading uses a hidden verification_test.rb; before verification, test/, bin/, and config/environments/test.rb are restored from a pre-agent snapshot. Network access is restricted to the LLM provider’s API host after setup. rubyonrails

New-model results and comparisons:

• Qwen 3.8-27B: 48 of 63, listed as 76% in the supplied score-card text; median runtime was 27 minutes, 1.7× the next-slowest model, and the harness’s timeouts had to increase from 30 to 60 minutes. Its Rails API recall was the lowest reported, at 7.9%. rubyonrails
• Terra: 49 of 63, listed as 78%; $0.20 per run and a 182-second median runtime, making it the fastest model in the field and moving Luna into second place. rubyonrails
• Sonnet 5: 44 of 63, listed as 70%, the weakest Anthropic result reported in the post. Its Rails API recall was higher than Opus 4.8’s—25.4% versus 15.9%—but the post says it still took twice the time and that the higher recall did not translate into the result. rubyonrails
• ox-alpha: an unnamed, unattributed OpenRouter model that was free during preview; it scored 52 of 63, matching Grok 4.6, and had 28.6% Rails API recall. The supplied score-card text lists both at 83%. rubyonrails

Broader benchmark comparisons: OpenAI’s Sol, Terra, and Luna scored 84%, 78%, and 73%, respectively, in exactly the same order as their prices; all three were in the fastest third of the field. rubyonrails Anthropic’s reported range runs from 92% down to 70%, with Sonnet 5 at the bottom of that range. rubyonrails The supplied score-card text gives the full percentage ordering as: claude-opus-5 92%; kimi-k3 90%; claude-fable-5 90%; gpt-5.6-sol 84%; grok-4.6 83%; ox-alpha 83%; glm-5.3 79%; claude-opus-4.8 79%; gpt-5.6-terra 78%; qwen3.8-27b 76%; muse-spark-1.2 76%; gpt-5.6-luna 73%; gemini-3.7-flash 71%; claude-sonnet-5 70%; glm-5.2 67%; deepseek-v4-flash 65%. rubyonrails

Caveats: The excerpt reports score/percentage results and Rails API recall, but does not define the recall formula or denominator. Cost reporting is uneven: it gives Terra’s $0.20 price, ox-alpha’s free-preview status, and only a relative price ordering for Sol/Terra/Luna; it does not provide comparable dollar prices for every newly added model. rubyonrails

• Multi-agent command center: DHH says agents have reshaped development work at 37signals, Basecamp, and HEY. NetworkChuck His current setup uses Herder instead of tmux for agent-aware session management, the native Claude harness, native Codex for OpenAI models, and Pi/OpenCode for open models such as Code Llama 3; he says other harnesses have difficulty using his Claude subscription within its license parameters. NetworkChuck In Amachi, HDL opens a development layout, while CX and CY launch Claude and Codex with bypass permissions alongside Neovim, a terminal, and Herder’s control strip. NetworkChuck He adds PCs through GL.iNet Comet browser KVMs and Herder’s --remote mode, enabling roughly four machines and many concurrent agents; he links this command-center setup to merging over 1,000 PRs in about 90 days, with some days reaching 50–100 PRs. NetworkChuck DHH’s broader orchestration goal is asynchronous agents handling routine toil while humans batch decisions into dedicated time windows, although he says the current AI urgency is not sustainable and this future has not fully arrived. NetworkChuck

• Agent-native OS and crash-to-PR workflow: DHH describes Amachi Quattro as bundling agent harnesses, installation tooling, and skills for creating C++ applications with Qt and customizing Quick Shell panels, icons, calendars, and themes. NetworkChuck Before the final release, the community repository omakichiplugins.com had “75 plugins or something like that,” up from 40 the previous day; he said a beta was already usable, with an RC and final release imminent. NetworkChuck The described maintenance loop is: an application crash opens an AI-diagnosis action in the user’s preferred harness; a diagnosis skill loads crash details, then a reporting skill determines whether the problem is in Amachi; with permission and GitHub CLI configured, the agent prepares a bug report, and a maintainer agent reviews reproducible reports and turns them into PRs. NetworkChuck

• Low-risk autonomous coding and release loop: DHH asked Fable to translate the Python terminal text effects library used by the Amachi screensaver into Rust; after about 45 minutes, he reports a pixel-perfect 3 MB executable that was initially 10× faster and used 10× less CPU, followed by an auto-research optimization loop that reached 27× the Python version’s speed and reduced launch time from 86 ms to 2 ms. NetworkChuck For two pending PRs, his instruction was: “Hey, check out the pull requests. If they look good to you, patch them up, make sure they follow our style, merge them in, cut a new release, push the new release to the repo, and just let me know when you're done.” NetworkChuck The agent completed the review, style fixes, merge, GitHub tagging, release notes, binary push, and released version 0.31. NetworkChuck DHH limits this level of autonomy to low-criticality software where failures are cosmetic, and warns against granting agents unrestricted access after citing a sandbox-escape incident. NetworkChuck

Transferable patterns from the panel’s voice-agent deployments:Latent Space

• Compact context early (explicit coding-agent analogy): At roughly 25% utilization of a 1M-token context, start compacting and saving work rather than waiting for the context limit. The panelist says their voice-agent systems compacted from day one because smaller contexts could derail, and they continuously tracked what had happened in the conversation. Latent Space
• Use guarded, dynamically assembled execution: One described pipeline runs supervisor checks for prompt injection/social engineering, intent selection, conditional context/instruction selection, system-prompt compilation, generation, and a post-generation grounding check; parallelize stages where possible to reduce latency. Latent Space For predictable workflows, use node/graph routing; for erratic inputs, retain a central prompt or “brain.” Selective context inclusion may become less reliable than supplying all rules as models improve, so validate the choice with repeated tests and an LLM judge. Latent Space
• Decompose and route work: Hand bounded subtasks to specialized agents with narrowly scoped instructions, then return to the general agent after completion; run a small classifier for fast intent inference in parallel with a background reasoning process, and maintain a waterfall of fallback models for outages. Latent Space
• Build evals from real workflows before production: The panelists describe an open-source benchmark loop with replayable turns that can be run locally against new models and compared with deployed or self-hosted infrastructure; LLM behavior can be isolated by running the inference loop without speech stages. Latent Space Convert customer conversations into evals and run smoke tests before going live. Latent Space

Firsthand context: Theo applies these practices to T3 Code and says he has accumulated hundreds, if not thousands, of coding-agent threads on a dedicated desktop. Theo - t3․gg

• Prefer maintained repository truth over automatic coding-agent memory. Theo argues that code is the ground truth; stale comments and abandoned Markdown plans create split-brain context that can actively mislead agents. Theo - t3․gg In his Claude Code memory audit, more than 355 sessions produced only 19 sessions that opened an individual memory file, while 80 sessions wrote or edited memories; of 45 memories, 26 had never been read. Theo - t3․gg He concluded the system was useless and said he would disable, archive, and delete the memories across his machines. Theo - t3․gg

• Use agent/Claude MD files for durable alignment, not for accumulating transient state. Theo recommends documenting how the agent should think about the product and codebase, rather than only listing prohibitions. Theo - t3․gg His T3 Code instructions describe the product and architecture, preserve open-source and performance goals, require remote and multi-surface compatibility, and include a glossary. Theo - t3․gg They also record recurring failure modes, dev-server and test-data practices, verification and pull-request guidance, and project-specific coding taste. Theo - t3․gg Theo says this directional context makes agents proactively extend or challenge work in ways aligned with him, whereas automatic memory likely hurts. Theo - t3․gg

• Prefer simple tool discovery and selective context over elaborate code-retrieval systems. Theo’s comparison is that Claude Code demonstrated that giving an agent tools and Bash can work better than a bespoke dynamic code graph; he says Cursor, despite pioneering code traversal, has moved away from that approach. Theo - t3․gg In the Mario/Pi and Armen/Flask discussion relayed by Theo, the design principles are that coding needs no separate memory system, Bash is sufficient, and large tool outputs should be written to files and loaded only when needed. Theo - t3․gg The accompanying Sentry example uses an on-demand skill with custom authentication and a preferred data shape, downloads 52 JSON items, loads only three into context, and leaves the rest in a file for inspection. Theo - t3․gg

• Build a guardrail ladder: architecture first, automation second, skills only as fallback, humans last. Theo endorses Lauren’s ordering: eliminate failure categories through architecture and data structures, then catch remaining issues with lint rules, tests, or CI; use skills mainly for process problems, and add human review only after those layers fail. Theo - t3․gg His T3 Code example replays real, very large threads through codecs and Claude, measures WebSocket bandwidth, keeps all replays under 100 KB and most under 10 KB, reports the result in pull requests, and fails the PR when usage exceeds a ceiling roughly 30% above the optimized baseline. Theo - t3․gg He says agents then detect and fix bandwidth regressions before reporting the task complete. Theo - t3․gg Theo also favors shared TypeScript data-loading code across web, Electron, and React Native to prevent surface-specific drift, contrasting it with SwiftUI regressions where that relationship was less clearly encoded. Theo - t3․gg

• Contrarian principle: Theo endorses imposing human values rather than rigid human discipline on agents; he says agents tend to fall back from high-discipline TDD, so desired outcomes should be encoded structurally and enforced with architecture, tests, and CI rather than relying on ritual instructions. Theo - t3․gg

• Control-graph design for coding agents (firsthand): AI Jason reports that his team has experimented with agent loops for two months and recommends treating repeatable agent work as a control graph rather than a one-shot prompt. AI Jason In this model, nodes are actions, edges define what happens next, and state carries data across steps. AI Jason For each recurring task, map the SOP, group it into nodes and edges, and persist current state in a Markdown status file plus an append-only run log. AI Jason The graph can be enforced with code and frameworks such as LangGraph or Dynamic Workflow, or with a text/JSON/Mermaid SOP packaged as an invokable skill and backed by deterministic scripts. AI Jason The video also points to an open-source project called “de workflow” for implementing dynamic workflows across agent libraries. AI Jason
• Replicable code-shipping workflow: After defining a sufficiently scoped plan, run three phases: setup and implementation, verification, then simplification and PR creation only if verification passes. AI Jason Give each agent node an explicit model and output schema, then construct the next node’s prompt from the previous node’s structured output; use pipeline dependencies for sequential stages and parallel fan-out where work can be split. AI Jason Keep verification separate because agents are weak at self-verification, and consider a dedicated planner for complex tasks. AI Jason Use scripts for complex data fetching, starting the development server, programmatic checks, and end-to-end tests instead of making agents perform those deterministic steps manually. AI Jason Jason says his referenced public AI Builder resources include a verifier setup skill. AI Jason
• Orchestration and loop pattern: Jason describes time-, goal-, and event-triggered loops—for example, scheduling an agent to pull the latest GitHub issue and attempt a fix—and an orchestrator agent that retains context, delegates to a team, and monitors execution. AI Jason Define in advance which actions the loop can take autonomously and which conditions require human involvement. AI Jason In his team’s daily design-triage loop, deterministic heuristics first filter candidates, parallel subagents screenshot and evaluate designs against user requests, shared schemas standardize results, and a main agent ranks failures into a report for review and dataset improvement. AI Jason His contrarian implementation choice is to prefer a model-driven graph for most cases because Dynamic Workflow/code-as-graph starts fresh agent sessions and cannot resume the prior conversation; he considers code-as-graph more suitable for extremely large tasks. AI Jason

• Riley Brown’s firsthand workflow: In ChatGPT Work on the desktop app, invoke @sites with a detailed build prompt covering the UI, team authentication, video uploads and storage, a database for per-video stats, and an agent skill that populates the app from a scraping API; the demonstrated Sites feature provides hosting, database, authentication, storage, sharing, and custom-domain support. Riley Brown
• Iterative agent-driven UI refinement: After the initial generation, review the live site and use the browser’s annotation tool to request concrete changes—full-screen video, TikTok/Reels-style metric overlays, a shorter top bar, and a better download button—then send follow-up requests for asset corrections and additional data; the changes are applied to the deployed site. Riley Brown
• Reusable external-tool integration: Create a skill around the Scrape Creators API by providing an API key, then have the agent retrieve videos, likes, comments, and transcripts and place them directly into the hosted app. The presenter explicitly says pasting the key into the skill is “technically not best practice”; Sites also exposes environment variables for API keys, plus analytics and database inspection in settings. Riley Brown
• Reported results and extension: The presenter claims an earlier app replaced three paid SaaS tools costing thousands of dollars and took three prompts; this demo ran for 25 minutes 8 seconds and initially found six videos. Riley Brown He recommends asking Codex to make a Sites app Web MCP-enabled and deploy it so ChatGPT and Codex can discover and use tools from the live page. Riley Brown The video is sponsored by OpenAI, so the productivity and product claims should be treated as self-reported demonstration results rather than independent benchmarks. Riley Brown

Firsthand build: Riley Brown reports that a previous Social Scrape build replaced three SaaS tools for which he was paying thousands of dollars, using three prompts; in this walkthrough he rebuilds the app with ChatGPT Work’s @sites feature. Riley Brown Sites provides hosted deployment, authentication, a database, storage, sharing, and custom-domain support. Riley Brown The target workflow is to ingest short-form videos, platform statistics, transcripts, and downloads into a team-facing app. Riley Brown

Replicable agent loop: In the ChatGPT desktop app, @mention sites with a detailed specification covering the UI, authentication, storage/database, agent-upload behavior, an API-backed scraping skill, and an initial dataset. Riley Brown After a 25-minute, 8-second generation, Brown opened the app in a browser, signed in, clicked UI defects with Annotate, wrote concrete fixes, sent them, and refreshed the site. Riley Brown He then requested a corrected profile image and 10 videos from another creator, showing that UI and data changes can be handled in the same agent loop. Riley Brown

API skill integration: Brown uses a scrape creators skill backed by the Scrape Creators API. His setup sequence is to obtain an API key, ask ChatGPT Work or Codex to create a skill using that API, and invoke the skill so the agent imports videos, likes, comments, and transcripts into the hosted app. Riley Brown He explicitly says pasting the key into the agent is “not best practice”; Sites settings also provide environment variables for API keys. Riley Brown

Extension: Brown recommends updating the ChatGPT desktop app and asking Codex to make the site Web MCP-enabled and deploy it to Sites; he says this allows tools exposed by the live page to be discovered and used by ChatGPT and Codex while a user guides the process. Riley Brown

Caveat: Brown says the app was newly built and had not yet been sent to his team. Riley Brown The video also discloses OpenAI sponsorship. Riley Brown

• Firsthand eval-engineering workflow: @Vtrivedy10, @hwchase17, @nickhollon10, and @ArjunNargolwala describe a process developed around LangChain’s internal benchmarks, including code-review tasks based on real merged PRs and trace-mining tasks. They package the workflow in an updated eval-engineering skill. @Vtrivedy10
• Separate task design from implementation: Use a two-stage pipeline: first create and human-review a versioned Markdown spec describing the task’s inputs, environment, and graders; then use a coding agent to turn approved specs into Harbor-format tasks. This concentrates human judgment in spec review while allowing task construction to be automated and parallelized. @Vtrivedy10
• Replicable bootstrap loop: Load the eval-engineering skill and prompt: “Use the eval-engineering skill, the traces from {LangSmith project}, and the {current repository} to help me create an eval Task for {agent}.” Review the generated world-spec skill, then open a new thread with both skills loaded to create a second task; repeat review and revision until the world spec is reliable. Once validated, ask the coding agent to generate multiple new task specs from recent traces and instantiate each spec as a task. @Vtrivedy10
• Make the world spec reusable: Store dataset shapes, parsing and generation scripts, service APIs and schemas, credential requirements, trace-derived user patterns, and rubric guidance as general knowledge shared across tasks. The agent can scan the repository with subagents, cluster traces, catalog external services, decide which live tools should be simulated, and map data relationships before creating tasks. @Vtrivedy10
• Validate with trajectories, model tiers, and human review: Run generated tasks with real agents and inspect trajectories to expose environment flaws such as overly specific instructions or leaky abstractions. Run tasks across model tiers to calibrate difficulty and detect reward hacks; use rubric-guided LLM generation for free-text data and scripted SQLite generation with explicit schemas for tabular data. Human feedback remains necessary to align specs with the domain and to make tasks harder or easier after repeated trajectory review. @Vtrivedy10
• Timeless optimization pattern: Treat eval environments as continuously maintained regression and optimization infrastructure: update them as production data and models change, then use them to test cheaper-model routing for subsets of tasks, prompt simplification, harness-tool removal, or bash-only alternatives. @Vtrivedy10

• Simon Willison’s firsthand model watch: He uses an informal SVG task—asking models to draw “a pelican riding a bicycle”—to compare model capability, while noting that it is intentionally not a coding benchmark. He describes November as an inflection point for coding agents: Anthropic and OpenAI had spent 2025 optimizing for code, and Claude Code and Codex became substantially better around then. PyCon US
• Model and hardware signal: In his sequence of comparisons, Simon mentions GPT-5.1, Gemini 3, GPT-5.1 Codex Max, Claude Opus 4.5, and Gemini 3.1 Pro; by April, he reported that the 18 GB open-weight Gemma 4 ran on his laptop, while Qwen 3.6 also ran locally and outperformed Claude Opus 4.7 on his informal test. A Chinese lab’s GLM 5.1 was described as a 1.5 TB open-weight model requiring roughly $100,000 of hardware, highlighting a sharp local-deployment trade-off. PyCon US

• Use a cross-agent handoff for blocked debugging: ThePrimeagen reports that Fable claimed there was a memory leak but refused to provide details or fix it when asked; Grok found and fixed the issue. @ThePrimeagen A practical fallback is to hand off diagnosis and repair to another agent when the first agent cannot substantiate or act on its diagnosis. @ThePrimeagen
• Trust may matter more than raw code quality: jpschroeder says Fable is generally “smarter and more tasteful” than Sol, yet calls it disobedient, prone to leaving security holes unpatched, and flaky; they report using Sol about 5:1 because it is more trustworthy, despite Sol’s overengineering and weak writing/design. @jpschroeder

• Andrej Karpathy’s “Software 3.0” framing treats natural language as the instruction set for a general-purpose computational engine: prompts configure the model at runtime, while the model generates a task-specific computational state rather than following a fixed, single-purpose program. Kaushik Datta
• A practical prompting pattern is to force explicit intermediate work when a task requires multi-step reasoning. In the cited juggler-riddle example, zero-shot accuracy was described as roughly 17%, rising to nearly 79% with “let’s think step by step” and above 82% after adding “to be sure we have the right answer”; the explanation is that emitting intermediate tokens serializes more computation. Kaushik Datta
• Treat the context window as working memory and externalize long-lived agent state. The lecture describes a Linux-terminal simulation that maintained a virtual jokes.txt file through conversation history, and a “scratch pad” pattern in which the model writes intermediate computations to a text file and reads them back later to work around finite context. Kaushik Datta
• A concrete state-management architecture replaced thousands of lines of Python backend logic with an LLM operating over a JSON representation of application state: natural-language instructions specified how user interactions should mutate the JSON, and the model returned the updated state. Kaushik Datta
• Prompt-only control has a serious security limitation: the transcript says Microsoft’s early Bing chatbot “Sydney” had its personality, safety boundaries, and other behavior defined in a hidden English prompt, which a user exposed through a prompt-injection instruction to ignore prior constraints and print the initial instructions. Kaushik Datta

• Kent C. Dodds (@kentcdodds) recommends a dual UX bar for agent-enabled software: use the software yourself or teach an agent to use it, preserve a pleasant human experience, and expose an API that makes the product pleasant for agents to use. @kentcdodds

Kent C. Dodds flags a coding-agent failure mode: when you repeatedly have to steer an agent away from the same behavior, it is not driving itself; his prescription is to stop repeatedly telling it “no” instead of continuing the same corrective loop. @kentcdodds

• Addy Osmani recommends choosing the model and effort level at the start of a coding-agent session: switching models, effort levels, or fast mode mid-session forces a full uncached reread because the KV cache is tied to specific model weights, with the cost increasing as the session grows. @addyosmani
• Practical rule: switch only during the first few turns when necessary; later switches invalidate the prompt cache and can require paying for the full input context again, according to Teknium. @Teknium

• Codex + MLX local model workflow: @skirano asked Codex to build a language model from scratch on a Mac with MLX, using 778K iMessages; the model had 1.4M parameters and the end-to-end process took about 25 minutes. @romainhuet
• Local execution pattern: A Codex-built privacy screen reveals only the portion of a webpage the user is looking at, keeps the surrounding area blurred, and runs locally in the browser. @romainhuet
• Multimodal game development: Pietro built “Sunshard,” a retro space shooter for Game Boy Advance, with Codex and image generation. @romainhuet

• Kody for Grok Bot: Kent C. Dodds says to ask Grok Bot to add Kody (@kodykoala, https://kody.codes), which he describes as giving the bot access to a user’s personal software ecosystem across all agent sessions. @kentcdodds
• Setup: A follow-up provides the direct install URI grokbot://app/v1/plugin/add?id=56286216; Dodds also says users who need an invite code can contact him. @kentcdodds +1

Theo highlighted a video on memory in coding tools, relaying that Mario, creator of Pi, thought it was worth watching; Theo characterized it as a critique of how poor Claude Code’s memory is. @theo The associated recommendation says the discussion includes real-world examples from T3 code and contrasts Theo’s view with Armin’s. @badlogicgamesVideo@badlogicgames

• OpenWiki’s self-correcting memory pattern: @colifran_ describes a firsthand OpenWiki build in which every factual claim in an agent-generated wiki page is linked to the supporting code and a version of that evidence; when the source changes, the claim is flagged stale and must be rechecked rather than silently trusted. @LangChain +1
• Replicable workflow: Run the evidence-version comparison deterministically at the start of every wiki update, without model calls. When the agent reads a page, surface that page’s stale claims in context; have it re-verify and refresh the evidence or update the claim and wiki text together, while leaving unresolved claims flagged for a later update. This makes update cost scale with changed code rather than total stored claims. @colifran_
• Reported evaluation: Replaying a repository across feature changes, behavior changes, bug fixes, and reverts, the claims runtime reduced stale claims from 80 to 9 and hallucinated claims from 15 to 0 versus a baseline. In one run, stale claims fell from 17% to 0% at the next checkpoint, while supported claims rose from 77% to 98%. @colifran_
• Release and setup: OpenWiki 0.4.0 automatically begins generating claims and migrates the wiki to OKF v0.2 on the next update. The documented setup is npm install -g openwiki@latest followed by openwiki --init; the write-up links to the documentation and repository. @colifran_

• Jason Zhou reports a multi-step agent workflow using Grok and Treg: provide Grok with product context, let it form filtering criteria, use Treg to search, deduplicate, and enrich 1,000+ influencers, then connect Gmail and Google Sheets for outreach and logging. @jasonzhou1993
• The reported result was a 3-minute workflow costing $0.35, compared with eight hours of manual work; this is a secondhand productivity claim rather than a firsthand account from Zhou. @jasonzhou1993
• Treg is available at https://treg.to and has been open-sourced at https://github.com/superdesigndev/treg. @jasonzhou1993

Geoffrey Huntley recommends using Playwright and Bombadil together for visual verification: Playwright handles static visual checks, while Bombadil provides fuzz-based visual verification. @GeoffreyHuntley