ZeroNoise Logo zeronoise
Post
Treat Every LLM Change as a Hypothesis Test
3 min read
185 docs
Martin Casado’s standout talk recommendation turns stochastic AI work into a concrete measurement discipline; an open robotics corpus and two lighter organic picks round out the list.

The standout recommendation

An age of experimentation — BlueHat Asia 2026 slides

  • Content type / creator: Talk slides. The sharing post is by Halvar Flake, who describes them as slides from his Microsoft BlueHat Singapore talk; the deck itself is titled An age of experimentation and credits Thomas Dullien, Cyber Security Research Team, OpenAI.
  • Link:Slides
  • Recommended by: Martin Casado, who calls it his favorite talk of the year and praises its clarity in an era of “AI shorthand.”
  • Key takeaway: The deck treats reasoning LLMs as more like fire than an internal-combustion engine: discovered more than constructed, still poorly understood, and improved mainly through empirical work. Its practical rule is that every change in a probabilistic system is a hypothesis test; teams should ask what they want to show, measure rerun variation, identify randomness, and check whether changing conditions invalidate an IID assumption.
  • Why it matters: This is a useful operating manual for evaluating AI-enabled systems without mistaking activity for progress. The deck also recommends a benchmark hierarchy—small tests for rapid iteration and larger tests for periodic validation—while warning that the small tests must remain representative.

Eidon AI’s egocentric robotics corpus

  • Content type / creator: Open robotics dataset/resource, published by Solidic Labs Inc. (Eidon AI). The company has wound down, but the release remains available under CC-BY-4.0, with commercial use allowed when attribution is provided.
  • Link:Eidon AI on Hugging Face, including the video corpus and paired motion data
  • Recommended by: Clem Delangue, who praised the decision to open-source 1,274 hours of egocentric robotics data across 13,451 recordings and called it a gift to the robotics community. He explicitly argued that open source lets work outlive the organization that created it.
  • Key takeaway: The release combines 13,451 egocentric videos with 779 million rows of 24 Hz orientation and motion data covering the same recordings; the two pieces join on recording_id. The paired set contains 1,274 hours of video and arm tracking from 27 contributors, with automated quality-control scores.
  • Why it matters: This is an unusually actionable recommendation for embodied-AI work, as well as a concrete example of preserving research assets after a startup shuts down. The dataset is not a neutral snapshot of daily activity: the release warns that tasks are heavily weighted toward folding laundry, contributors are few and unevenly represented, and splits should be made by contributor rather than randomly.

Product and idea-generation picks

“I design teams like terrorist organizations” — Peter Ellis product-advice video

  • Content type / creator: A 90-minute YouTube interview featuring Peter Ellis, published through the @lennysan interview series. The associated post links to the video.
  • Recommended by: Keith Rabois, with the unqualified endorsement “Highly recommend.”
  • Key takeaway: The episode is organized around concrete product-leadership questions: managing Nikita Bier, why growth usually comes from the core, knowing when to stop, the shortcomings of the median product manager, and team design.
  • Why it matters: It is a focused listening assignment for leaders working through the trade-offs between product taste, growth, and organizational design—not a generic startup interview.

Sharif Shameem’s posts archive

  • Content type / creator: Personal blog/archive by Sharif Shameem. Posts
  • Recommended by: Shaan Puri, who says he had just discovered the blog, calls it “very good,” and tells readers to read a couple of posts.
  • Key takeaway: Puri does not identify a particular post or lesson, so this is a browse recommendation rather than a prescribed syllabus. The archive’s visible material ranges from The 28 AI tools I wish existed—framed by the question “where are the tools?”—to proposals about improving large video calls and modeling programming as a reinforcement-learning task.
  • Why it matters: Treat it as an idea-generation archive: useful for finding sharp product and technical questions, but not evidence that the listed concepts were built, tested, or validated.
Treat Every LLM Change as a Hypothesis Test
Summary
Coverage start
1 day ago
Coverage end
17 hours ago
Frequency
Daily
Published
15 hours ago
Reading time
3 min
Research time
2 hrs 36 min
Documents scanned
185
Documents used
9
Citations
20
Sources monitored
137 / 137
Insights
Skipped contexts
102
View
Source details
Source Docs Insights Status
Guillermo Rauch 7 0
Alexandr Wang 64 0
Aaron Levie 4 0
Marc Benioff 0 0
Amjad Masad 2 0
Kevin Systrom 0 0
Tim Sweeney 0 0
Mike Krieger 0 0
Adam D'Angelo 0 0
Brad Gerstner 0 0
Nat Friedman 1 0
clem 🤗 11 1
Tony Fadell 0 0
Matt Mullenweg 0 0
Sahil Lavingia 4 0
Demis Hassabis 0 0
Aravind Srinivas 4 0
Jessica Livingston 0 0
Drew Houston 0 0
martin_casado 7 1
Sarah Tavel 0 0
Ryan Hoover 0 0
Reed Hastings 0 0
Jeff Lawson 0 0
Stewart Butterfield 0 0
Howie Liu 0 0
Eric Schmidt 0 0
Palmer Luckey 0 0
Elon Musk 22 0
Satya Nadella 0 0
Jeff Bezos 0 0
Shaan Puri 1 1
Not Boring by Packy McCormick 0 0
Jason ✨👾SaaStr.Ai✨ Lemkin 8 0
Ben Thompson 1 0
Li Jin 0 0
Morgan Housel 0 0
delian 0 0
Daniel Ek 0 0
No Priors: AI, Machine Learning, Tech, & Startups 0 0
Invest Like The Best 0 0
Lenny's Podcast 0 0
20VC with Harry Stebbings 1 0
Garry Tan 6 0
Brian Armstrong 5 0
andrew chen 0 0
tobi lutke 4 1
benahorowitz.eth 0 0
jack 0 0
scott belsky 4 0
David Sacks 1 0
Vinod Khosla 3 0
Chamath Palihapitiya 2 0
Fred Wilson 0 0
Tim Ferriss 4 0
Sam Altman 0 0
balajis.com 0 0
Book of the Day from The Next Big Idea Club 1 0
The Foresight Playbook 0 0
Seliria’s Newsletter 0 0
The Generalist 0 0
Lenny's Newsletter 0 0
Elad Gil 1 0
No Priors 0 0
sarah guo 2 0
Bill Gurley 2 0
Chris Dixon 0 0
Keith Rabois 8 2
Patrick OShaughnessy 0 0
Bill Gates 0 0
Reid Hoffman 0 0
Naval 0 0
Farnam Street 0 0
Shane Parrish 0 0
Patrick Collison 0 0
Balaji 0 0
Sam Altman 0 0
Marc Andreessen 🇺🇸 0 0
David Perell 1 0
Paul Graham 1 0
Parker Conrad 0 0
Palmer Luckey 0 0
Suhail 0 0
Tom Blomfield 0 0
Guillermo Rauch 0 0
clem 🤗 0 0
Jeremy Howard 0 0
aileenlee 0 0
@jason 0 0
Ryan Hoover 0 0
Nat Friedman 0 0
Gokul Rajaram 0 0
Ivan Zhao 0 0
Chris Dixon 0 0
Eric Schmidt 0 0
Susan Wojcicki 0 0
Jeff Weiner 0 0
Kevin Systrom 0 0
Sundar Pichai 0 0
Tony Fadell 0 0
Scott Belsky 0 0
Drew Houston 0 0
Marc Benioff 0 0
Brian Chesky 0 0
David Heinemeier Hansson (DHH) 0 0
Sahil Lavingia 0 0
Andrew Wilkinson 0 0
jack 1 0
John Doerr 0 0
Satya Nadella 0 0
Mark Zuckerberg 0 0
Garry Tan 0 0
Jensen Huang 0 0
Michael Seibel 1 0
Daniel Gross 0 0
Shane Parrish 0 0
Fred Wilson 0 0
tobi lutke 0 0
David Sacks 0 0
Jeff Bezos 0 0
Elon Musk 0 0
Bill Gates 1 0
Sam Altman 0 0
Peter Thiel 0 0
Elad Gil 0 0
Keith Rabois 0 0
Reid Hoffman 0 0
Balaji Srinivasan 0 0
Tim Ferriss 0 0
Brian Armstrong 0 0
Ben Horowitz 0 0
Naval Ravikant 0 0
Patrick Collison 0 0
Marc Andreessen 0 0
Paul Graham 0 0
Tomasz Tunguz 0 0
Brad Feld 0 0