ZeroNoise Logo zeronoise
Post
Autonomous Research Closes the Gap—While Harnesses and Evals Decide What “Better” Means
4 hours ago
3 min read
459 docs
Prime Intellect reports frontier agents closing 82% of a human optimizer record, while new benchmark and agent-interface work exposes how much performance depends on evaluation conditions.

Top Stories

Why it matters: The strongest capability signal is now sustained agent work under disclosed conditions, not a model label alone.

Autonomous research is measurable, but novelty remains scarce. Prime Intellect reports 100+ autonomous runs across 10+ models on 8×H200s for up to eight days; the best runs closed 82% of the gap to a human-built nanoGPT optimizer record. Elie Bakouch calls the experiment noisy (~50-step spread after 24 hours), says Fable 5 reached the 82% mark, and highlights Kimi K3 building an experiment API. The team’s own surprise—deep understanding but few genuinely new ideas—makes this a sustained-optimization signal, not yet evidence of open-ended scientific creativity.

DeepSeek V4 Pro’s score is a harness result in this test. A community report says the same release scored 91 in DSH Standard but 99/96 in DSH Minimal and 98/99 in Anchored Standard on a frozen Project2 V4.1b test, matching Sol, Fable, and Opus’s top band. Minimal reproduces the RL-time prompt with bash and str_replace_editor; Anchored Standard restores 25 tools after the first call without returning to 91. The report notes DeepSeek’s model card specifies Minimal for public code-agent benchmarks, while default users still often see 91—making prompt, tool schema, and harness disclosure essential.

Research & Innovation

Why it matters: Reliability gains may come from better tests and agent interfaces, not only larger checkpoints.

BenchDrift generates meaning-preserving benchmark variants. Across eight models on GSM8K, MMLU, and MATH-Hard, phrasing sensitivity persists: stronger models lose more from rephrasing than they gain, and confident answers can break even when only wording length changes.

StateBridge passes the last 64-token hidden states directly into a receiving model’s embedding space without retraining. It beat or tied baselines on 22/26 tests and raised Qwen3-32B GPQA from 58.3% with text to 64.1%; testing used identical weights, and the less-visible channel is harder to debug and govern.

ArchAgent v2 uses cascaded evolution and hardware-budget feedback to find a three-level prefetcher that beat the prior hand-designed champion by 0.3% geometric-mean IPC; multi-core search remains bottlenecked by simulation latency.

Products & Launches

Why it matters: Agent products are making orchestration and browsing behaviors user-facing runtime features.

Multi-agents v2 now lets a model delegate to any supported model, including Luna—an explicit model-agnostic delegation layer.

Yutori Navigator runs screenshot-action loops; Together AI says it beats frontier performance at twice the inference speed and 4–5× lower cost.

Industry Moves

Why it matters: Commercial concentration, talent retention, and memory access are becoming strategic AI variables.

OpenAI’s commercial center is turning enterprise. Kimmonismus, citing the Financial Times, reports that a 60/40 consumer-enterprise revenue split at the start of the year has crossed to majority enterprise. Separately, the account reports GPU-systems engineer Scott Gray’s departure and at least 12 senior-leader exits in 2026; the two signals should not be treated as causal.

Memory supply is becoming a geopolitical AI constraint. A WSJ-cited post says the Trump administration is pressing Apple over CXMT/YMTC memory chips for devices sold in China; standard parts are legal, while sharing information for customized chips requires a U.S. license.

Policy & Regulation

Why it matters: Frontier labs are arguing for differentiated oversight rather than uniform rules.

Anthropic CEO Dario Amodei calls regulation-versus-distribution a false choice. He supports stronger testing for frontier than off-frontier models, exemptions for smaller firms (citing $500M for California’s SB53), pre-deployment testing for frontier and open-weight models approaching the frontier, and a FINRA-like entity. These are Anthropic’s policy positions, not enacted changes.

Quick Takes

Why it matters: Smaller signals point to cheaper inference, wider adoption, and open-model reach.

  • Pranjal reports a from-scratch Blackwell NVFP4 matmul beating cuBLAS by 4.7% at N=8192.
  • Doximity’s survey of 3,151 U.S. physicians says 63% use AI; 75% of AI users report lower administrative burden and better job satisfaction.
  • Bloomberg, cited by @business, reports Alibaba’s open-weight models exceeded 3 billion global downloads in six months.
Autonomous Research Closes the Gap—While Harnesses and Evals Decide What “Better” Means
Back to details
Skipped contexts (96)
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal
AI High Signal