ZeroNoise Logo zeronoise
Post
Autonomous Research Closes the Gap—While Harnesses and Evals Decide What “Better” Means
4 hours ago
3 min read
459 docs
Prime Intellect reports frontier agents closing 82% of a human optimizer record, while new benchmark and agent-interface work exposes how much performance depends on evaluation conditions.

Top Stories

Why it matters: The strongest capability signal is now sustained agent work under disclosed conditions, not a model label alone.

Autonomous research is measurable, but novelty remains scarce. Prime Intellect reports 100+ autonomous runs across 10+ models on 8×H200s for up to eight days; the best runs closed 82% of the gap to a human-built nanoGPT optimizer record. Elie Bakouch calls the experiment noisy (~50-step spread after 24 hours), says Fable 5 reached the 82% mark, and highlights Kimi K3 building an experiment API. The team’s own surprise—deep understanding but few genuinely new ideas—makes this a sustained-optimization signal, not yet evidence of open-ended scientific creativity.

DeepSeek V4 Pro’s score is a harness result in this test. A community report says the same release scored 91 in DSH Standard but 99/96 in DSH Minimal and 98/99 in Anchored Standard on a frozen Project2 V4.1b test, matching Sol, Fable, and Opus’s top band. Minimal reproduces the RL-time prompt with bash and str_replace_editor; Anchored Standard restores 25 tools after the first call without returning to 91. The report notes DeepSeek’s model card specifies Minimal for public code-agent benchmarks, while default users still often see 91—making prompt, tool schema, and harness disclosure essential.

Research & Innovation

Why it matters: Reliability gains may come from better tests and agent interfaces, not only larger checkpoints.

BenchDrift generates meaning-preserving benchmark variants. Across eight models on GSM8K, MMLU, and MATH-Hard, phrasing sensitivity persists: stronger models lose more from rephrasing than they gain, and confident answers can break even when only wording length changes.

StateBridge passes the last 64-token hidden states directly into a receiving model’s embedding space without retraining. It beat or tied baselines on 22/26 tests and raised Qwen3-32B GPQA from 58.3% with text to 64.1%; testing used identical weights, and the less-visible channel is harder to debug and govern.

ArchAgent v2 uses cascaded evolution and hardware-budget feedback to find a three-level prefetcher that beat the prior hand-designed champion by 0.3% geometric-mean IPC; multi-core search remains bottlenecked by simulation latency.

Products & Launches

Why it matters: Agent products are making orchestration and browsing behaviors user-facing runtime features.

Multi-agents v2 now lets a model delegate to any supported model, including Luna—an explicit model-agnostic delegation layer.

Yutori Navigator runs screenshot-action loops; Together AI says it beats frontier performance at twice the inference speed and 4–5× lower cost.

Industry Moves

Why it matters: Commercial concentration, talent retention, and memory access are becoming strategic AI variables.

OpenAI’s commercial center is turning enterprise. Kimmonismus, citing the Financial Times, reports that a 60/40 consumer-enterprise revenue split at the start of the year has crossed to majority enterprise. Separately, the account reports GPU-systems engineer Scott Gray’s departure and at least 12 senior-leader exits in 2026; the two signals should not be treated as causal.

Memory supply is becoming a geopolitical AI constraint. A WSJ-cited post says the Trump administration is pressing Apple over CXMT/YMTC memory chips for devices sold in China; standard parts are legal, while sharing information for customized chips requires a U.S. license.

Policy & Regulation

Why it matters: Frontier labs are arguing for differentiated oversight rather than uniform rules.

Anthropic CEO Dario Amodei calls regulation-versus-distribution a false choice. He supports stronger testing for frontier than off-frontier models, exemptions for smaller firms (citing $500M for California’s SB53), pre-deployment testing for frontier and open-weight models approaching the frontier, and a FINRA-like entity. These are Anthropic’s policy positions, not enacted changes.

Quick Takes

Why it matters: Smaller signals point to cheaper inference, wider adoption, and open-model reach.

  • Pranjal reports a from-scratch Blackwell NVFP4 matmul beating cuBLAS by 4.7% at N=8192.
  • Doximity’s survey of 3,151 U.S. physicians says 63% use AI; 75% of AI users report lower administrative burden and better job satisfaction.
  • Bloomberg, cited by @business, reports Alibaba’s open-weight models exceeded 3 billion global downloads in six months.
Autonomous Research Closes the Gap—While Harnesses and Evals Decide What “Better” Means
AI High Signal
  • DeepSeek Harness reached 120K GitHub stars in 4 days; @teortaxesTex says most are "dead" and reads the effort as a talent-recruiting gambit for "99.999th percentile" engineers rather than a user play .
  • DeepSeek hired Cui Tianyi in March 2026 to lead the Harness team; he wrote "Nine Lectures on Backpack Problem" (2007), spent ~9 years at Jane Street on software/quant research, and co-founded quant firm TSY Capital in 2022 .
  • Harness's design centers on "Everything is a plugin" and "Every run is traceable"; its Cordis layer studies how to undo component side effects, applying quantitative-trading rigor to agent execution .
  • DeepSeek V4-Preview launched in April 2026; the same analyst frames V4 as a "philosophically radical AGI project" with deterministic, bitwise-batch-invariant, training-inference-aligned rollouts across hundreds of thousands of sandboxes .
  • The same account adds that DSH is "a highly leveraged bet" and notes Fable's skepticism .
> Haskeller understandable. By the way. time for my statement on DSH. Consider it a non-technical view from 100K feet; I am a bit high. I… 因为 DeepSeek Harness,很多人才第一次听到崔添翼 [@tianyi](https://x.com/tianyi) 这个名字。 我好奇继续深挖了下他的公开经历,发现这人的技术轨迹很有意思。 很多程序员都看过的《背包问题九讲》,第一版是他 2007 年写的。20… Fable is skeptical, as it should be. This is a highly leveraged bet. ![](https://pbs.twimg.com/media/HP0glD_XoAAURBk.jpg) ![](https://pbs…
AI High Signal

Sam Altman said within the next six months a descendant of ChatGPT could "continuously watch your screen," meetings, and calls while maintaining full context of your life and work; he says it is only "one model generation away" from being "incredibly useful" and estimates timing at "sometime in the next six months" . Commentator @ChrisGPT notes Altman didn't call it GPT-6 or an Astra model, framing it as a clear AGI signal—a persistent multimodal agent that catches mistakes and proactively offers to work . Counterpoint from @zachtratar: this is already possible today, citing Notion AI meeting notes and KimiK3 for screen-data/agentic workflows .

Sam Altman: Within the next 6 months, a descendant of ChatGPT could: could “continuously watch your screen,” meetings and calls while mai… We are not "one model generation away" from this. It can happen today. Notion AI meeting notes are already \*amazing\* at automating work…
AI High Signal

Pranjal (@pranjalssh) published the sequel to his H100 blog, implementing Blackwell matmul for NVFP4 from scratch and outperforming cuBLAS by 4.7% at N=8192, with a method that exact-fits Nvidia hardware better than Hilbert curves. Full details: https://cudaforfun.substack.com/p/outperforming-cublas-on-nvfp4

Sequel for my H100 blog is out now. We implement Blackwell matmul for NVFP4 from scratch, and outperform cuBLAS by 4.7% at N=8192. This t…
AI High Signal
  • Anthropic is rapidly ramping up AI efforts in biology and medicine, with Dario Amodei expecting "incredible results in the coming years and some early glimmers in the coming months," and promising the company will announce real accomplishments "as loudly as possible" once achieved .
  • Responding to criticism that his AI messaging is disproportionately negative, Amodei argues the public's negative view of AI is "fundamentally a crisis of trust" in companies and governments, and that winning trust back requires actually delivering results — "actually curing cancer" — not a glitzy positive marketing campaign .
  • Amodei defends his messaging as balanced between risks and benefits, citing his essay "Machines of Loving Grace" (arguing AI could cure most human disease in ~5-10 years) and his recent "Policy on the AI Exponential" (proposing streamlined FDA processes so AI-accelerated drugs aren't slowed by regulation) .
2/2 Second, on the messaging around AI. I do not agree that my messaging has been disproportionately negative. In fact it has been about …
AI High Signal

Levie argues Cursor executed the applied AI strategy flawlessly, noting the biggest developer tool exits in history were on the order of low billions of dollars and most underestimated the market size in AI coding . Agentic coding's market is far larger than most realized, with ample room to innovate between the user and the underlying model, and even on the model itself . He outlines Cursor's applied AI playbook: figuring out the right product shape for agentic coding, acting as a neutral layer between models and workflows, post-training models to drive down costs or improve performance, building the right infrastructure stack, and developing a category-aligned go-to-market motion . He expects to see many more applied AI successes like this .

Amazing outcome. Cursor executed the applied AI strategy flawlessly. Most people completely underestimated the market size in AI coding. …
AI High Signal

New Qwen 3.8 27B dense model is compute-heavier than DeepSeek V4 Flash (284B total, 19B active) and comparable to MiniMax M3 (427B, 26B active), while far smaller than Kimi K3 (2.8T, 104B active), DeepSeek V4 Pro (1.6T, 49B active), and GLM 5.2 (743B, 39B active); it runs on consumer hardware. Commentary adds it was likely pretrained with higher MFU than DeepSeek V4 Flash, making both about equally costly in GPU-hours per 1T training tokens, and calls for more published training-efficiency figures.

On that: 27B dense is actually big: Kimi K3 - 2.8T A104B DeepSeek V4 Pro - 1.6T A49B GLM 5.2 - 743B A39B --- MiniMax M3 - 427B A26B DeepS… However, 27B dense Qwen is likely pretrained with much higher MFU than DSV4-Flash 13AB has been, so they are about equally costly in GPU-…
AI High Signal

A tweet from @AnziParazzi cites Anthropic as explicitly saying it does not have RSI internally, with AI assistance accelerating its AI R&D by less than 2x . AI commentator @teortaxesTex rejects this as a bad-faith definitional quibble, arguing Anthropic has thousands of GPUs per researcher, no longer faces bottlenecks like kernel engineering, and has models that autonomously prove hard theorems and design experiments — calling this 'RSI' — and adds that >2x acceleration has been achieved with commercial models for over six months, suggesting Anthropic crossed its 'responsible scaling' red lines .

[@teortaxesTex](https://x.com/teortaxesTex) from your own source, they explicitly do not have RSI internally (AI assistance has accelerat… Idk what this means but I say Anthropic is full of shit. They have thousands of GPUs per researcher. They no longer face bottlenecks in s… Ok RSI with extra steps. The quibbling about definitions of AI-driven acceleration is, I think, done in bad faith. They don’t want to adm…
AI High Signal

@teortaxesTex says a test/tool confirms Flash-0731 is "overfit to a harness" and doesn't optimally self-steer by default ; a reply shares the GitHub tool SheberDavid/v4-flash-godmode-opencode-go as useful for this . He also praises the speed and quality of DS's free web/app, adding: "They don't raise prices because they're out of money."

Ok this is a confirmation that Flash-0731 is also overfit to a harness and doesn’t optimally self-steer by default. Very interesting test… [@teortaxesTex](https://x.com/teortaxesTex) [https://github.com/SheberDavid/v4-flash-godmode-opencode-go](https://github.com/SheberDavid/…
AI High Signal
  • Turing Post profiles Simile AI, the startup built on Stanford's Generative Agents research, describing it as a "$2B bet" on simulating human behavior at scale and asking whether simulations can justify $100M runs .
  • Simile's lineage comes from Joon Sung Park's Stanford work: 2022's Social Simulacra populated hypothetical online communities with thousands of generated personas (with Percy Liang and Michael Bernstein among co-authors), and 2023's "Smallville" placed 25 AI agents in a pixelated town where they accumulated memories, adjusted plans, and maintained relationships over simulated days .
  • In 2025, Park and collaborators created agents modeled on 1,052 Americans from two-hour interviews and survey data, then tested them on General Social Survey questions, personality measures, economic games, and replicated social-science experiments against a demographics-only baseline .
  • The SocSci210 dataset contains 2.9M individual responses from 400,491 participants across 210 social-science experiments; fine-tuning a 14B-parameter Qwen model on it improved agreement with human distributions on unseen studies by 26% relative.
  • Demand came from both social scientists wanting to run experiments and large-company executives wanting answers about customers and markets, which pushed Park to start the company with Liang and Bernstein; inbound interest arrived about six months after the Generative Agents paper .
Can AI Simulate 8 Billion People?
AI High Signal

Krea AI's realtime image editing lets users screenshare Pinterest-style content and edit images instantly ; the same workflow works on Krea's discovery page to riff off other people's generations .

i love [@krea_ai](https://x.com/krea_ai) realtime its so fun to just scroll pinterest screensharing to realtime and edit the images howev… you can do this with anything, including the krea discovery page itself to quickly riff of other peoples generations [![Video](https://pb…
AI High Signal

AI researcher @yilin_yang721 observes that a large part of the industry is now driven by evals, with many modeling efforts pursuing state-of-the-art eval scores as their primary goal, crediting eval teams like artificial analysis and SWE-bench . AI researcher @OfirPress amplifies this, calling evals "(almost) all you need" .

It’s funny that a large chunk of the industry is driven by evals now. Many modeling efforts across the industry are pursuing SoTA eval nu… evals are (almost) all you need [https://x.com/yilin_yang721/status/2088725460356313276](https://x.com/yilin_yang721/status/2088725460356…
AI High Signal

Dario Amodei (Anthropic CEO) argued that AI structurally concentrates power due to scaling laws, not regulation, and that open-weights models are insufficient because compute and chips remain concentrated in frontier labs and hardware providers . He rejected the "regulation = regulatory capture = concentration" framing, advocating "rules of the road" that simultaneously address cyber/bio/alignment risks, institutionally constrain frontier companies, and leave room for open-weights . He detailed Anthropic's policy proposals as deliberately disadvantaging frontier labs: SB53 exempts companies under $500M in revenue/training costs (SB1047 had a lower threshold that Anthropic objected to); CAISI/White House testing is more rigorous for frontier vs off-frontier models; and the "Pacing the Frontier" approach would slow only the best models . He voiced support for the reported Trump administration plan of pre-deployment testing for frontier models and testing open-weights models as they approach the frontier, pending details, and endorsed Demis Hassabis' idea of a FINRA-like regulatory entity .

1/2 Thanks Gavin for an especially thoughtful exchange. I don't usually spend much time on social media but I wanted to engage here becau…
AI High Signal

Yutori's browser-use agents run iterative screenshot-action loops, dozens of times per task; its Navigator model running on Together AI reportedly beats frontier performance with 2x faster inference and 4-5x lower cost .

Yutori's browser-use agents run in tight loops: screenshot, action, repeat, dozens of times per task. On Together AI, their Navigator mod…
AI High Signal

In a tweet, @theo finds it "kinda wild" that Dario is blaming social media algorithms for damaging public reputation of their text generation algorithms.

Kinda wild to see Dario blaming social media algorithms for hurting the public reputation of their text generation algorithms ![](https:/…
AI High Signal

@boneGPT argues international cooperation on AI was always "a childish pipedream" with "zero examples of countries willingly halting research on the most strategic technology of their time because of a treaty"; only nuclear treaties and the ozone ban ever worked, and unlike nukes — never a trillion-dollar, work-replacing technology — "you can't make a nuke code an app" . Predicting "a great war," he attacks "Dean" (linking to @deanwball's post) for refusing to build robots to fight it while invoking children he plans to send off to die to maintain a monopoly "for the supposed good guys" . @teortaxesTex endorses the "total war" implication as "the endpoint of the Jevons paradox": higher AI productivity means not abundance but "SPEED OF METABOLIZING THE WORLD" .

To complete the loop on this, international cooperation was always a childish pipedream. It was ignorant. It presupposed that the UN work… «you can't make a nuke code an app. Thus, total war». I've been thinking about this for a while. It's a dreary vision: the endpoint of th…
AI High Signal

In a discussion on AI compute and regulation, GavinSBaker argues the compute shortage plus high DRAM prices (which reduce compute per capex dollar) may be "civilizationally useful" by giving time to adapt to a possible fast takeoff . He also contends that because humans run on only 15-20 watts, they will remain economically useful even in a fast-takeoff, AGI-maximalist scenario while compute remains constrained . He predicts compute inequality will become as massive an issue as income inequality, but argues for growing the pie so even the bottom of the distribution is better off, adding "Tokens per intelligence density per token per watt for the win - hopefully America wins that race" . He also frames cyber as defense dominant and bio as offense dominant, calling the bio point the best pro-regulation argument and backing the USG having all necessary information . @theo highlighted the 15-20 watts quote from GavinSBaker's reply .

Ok. Becky just brought me another gin and tonic and is watching our daughter. So: I think the framing of cyber as defense dominant and bi… "I'm impressed that your brain runs on only 15-20 watts tbh" ![](https://pbs.twimg.com/media/HPzi0NzbYAA94-g.jpg) [https://x.com/GavinSBa…
AI High Signal

Anthropic CEO Dario Amodei said the thing that will change the public's perception of AI is "actually curing cancer," adding that Anthropic is ramping up its efforts in biology and medicine . Analyst @teortaxesTex argues that although biology has so far been more resistant to scaling + RLVR than software engineering and Google has a wealth of research, Dario "has a good shot" at the lead after the GDM implosion .

Dario talking about what will change the public's perception on AI: >The thing that will work is \*actually curing cancer\*. Like I've be… Before, I would have said that Dario is unlikely to get the lead here, given the wealth of research championed by Google. Biology so far …
AI High Signal

AI watermarking will likely be circumvented by tools and APIs that strip watermarks, making the measure create friction rather than real visibility, per @HamelHusain .

Re: watermark - all that’s gonna happen is there will be tools and APIs to strip the watermark out In the end this will result in more fr…
AI High Signal
  • DHH promoted Omarchy as "the infinitely malleable computer" that "can make it happen" for anything you can think of , linking to a post on the project .
  • @matvelloso, first encountering Omarchy, called it "maybe the first 'AI-first' OS out there," ahead of any mainstream OS .
If you can think of it, Omarchy can make it happen. The infinitely malleable computer is finally here! [https://x.com/jankeesvw/status/20… First time I learn about this project. This may very well be the first "AI-first" OS out there, doing it before any mainstream OS has man…
AI High Signal

Prime Intellect's Prime Agent ran OpenAI GPT-5.6 Sol and Claude Opus 5 on the NanoGPT speedrun; these runs were excluded from the main plot as not the best but are available in the traces view . Scores were "within noise," per @eliebakouch, and the blog includes more Prime Agent examples with other models . Trace links: Sol and Opus 5.

[@iScienceLuvr](https://x.com/iScienceLuvr) we also have this for Opus 5 and Sol, it's not on the main plot since it's not the best run f… [@iScienceLuvr](https://x.com/iScienceLuvr) no it was within noise imo, you can see them exactly here [https://www.primeintellect.ai/rese…