We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Treat Every LLM Change as a Hypothesis Test
•
3 min read
•
185 docs
Martin Casado’s standout talk recommendation turns stochastic AI work into a concrete measurement discipline; an open robotics corpus and two lighter organic picks round out the list.
The standout recommendation
An age of experimentation — BlueHat Asia 2026 slides
- Content type / creator: Talk slides. The sharing post is by Halvar Flake, who describes them as slides from his Microsoft BlueHat Singapore talk; the deck itself is titled An age of experimentation and credits Thomas Dullien, Cyber Security Research Team, OpenAI.
- Link:Slides
- Recommended by: Martin Casado, who calls it his favorite talk of the year and praises its clarity in an era of “AI shorthand.”
- Key takeaway: The deck treats reasoning LLMs as more like fire than an internal-combustion engine: discovered more than constructed, still poorly understood, and improved mainly through empirical work. Its practical rule is that every change in a probabilistic system is a hypothesis test; teams should ask what they want to show, measure rerun variation, identify randomness, and check whether changing conditions invalidate an IID assumption.
- Why it matters: This is a useful operating manual for evaluating AI-enabled systems without mistaking activity for progress. The deck also recommends a benchmark hierarchy—small tests for rapid iteration and larger tests for periodic validation—while warning that the small tests must remain representative.
Eidon AI’s egocentric robotics corpus
- Content type / creator: Open robotics dataset/resource, published by Solidic Labs Inc. (Eidon AI). The company has wound down, but the release remains available under CC-BY-4.0, with commercial use allowed when attribution is provided.
- Link:Eidon AI on Hugging Face, including the video corpus and paired motion data
- Recommended by: Clem Delangue, who praised the decision to open-source 1,274 hours of egocentric robotics data across 13,451 recordings and called it a gift to the robotics community. He explicitly argued that open source lets work outlive the organization that created it.
- Key takeaway: The release combines 13,451 egocentric videos with 779 million rows of 24 Hz orientation and motion data covering the same recordings; the two pieces join on
recording_id. The paired set contains 1,274 hours of video and arm tracking from 27 contributors, with automated quality-control scores. - Why it matters: This is an unusually actionable recommendation for embodied-AI work, as well as a concrete example of preserving research assets after a startup shuts down. The dataset is not a neutral snapshot of daily activity: the release warns that tasks are heavily weighted toward folding laundry, contributors are few and unevenly represented, and splits should be made by contributor rather than randomly.
Product and idea-generation picks
“I design teams like terrorist organizations” — Peter Ellis product-advice video
- Content type / creator: A 90-minute YouTube interview featuring Peter Ellis, published through the
@lennysaninterview series. The associated post links to the video. - Recommended by: Keith Rabois, with the unqualified endorsement “Highly recommend.”
- Key takeaway: The episode is organized around concrete product-leadership questions: managing Nikita Bier, why growth usually comes from the core, knowing when to stop, the shortcomings of the median product manager, and team design.
- Why it matters: It is a focused listening assignment for leaders working through the trade-offs between product taste, growth, and organizational design—not a generic startup interview.
Sharif Shameem’s posts archive
- Content type / creator: Personal blog/archive by Sharif Shameem. Posts
- Recommended by: Shaan Puri, who says he had just discovered the blog, calls it “very good,” and tells readers to read a couple of posts.
- Key takeaway: Puri does not identify a particular post or lesson, so this is a browse recommendation rather than a prescribed syllabus. The archive’s visible material ranges from The 28 AI tools I wish existed—framed by the question “where are the tools?”—to proposals about improving large video calls and modeling programming as a reinforcement-learning task.
- Why it matters: Treat it as an idea-generation archive: useful for finding sharp product and technical questions, but not evidence that the listed concepts were built, tested, or validated.
Treat Every LLM Change as a Hypothesis Test
Summary
Source details
| Source | Docs | Insights | Status |
|---|---|---|---|
| Guillermo Rauch | 7 | 0 | |
| Alexandr Wang | 64 | 0 | |
| Aaron Levie | 4 | 0 | |
| Marc Benioff | 0 | 0 | |
| Amjad Masad | 2 | 0 | |
| Kevin Systrom | 0 | 0 | |
| Tim Sweeney | 0 | 0 | |
| Mike Krieger | 0 | 0 | |
| Adam D'Angelo | 0 | 0 | |
| Brad Gerstner | 0 | 0 | |
| Nat Friedman | 1 | 0 | |
| clem 🤗 | 11 | 1 | |
| Tony Fadell | 0 | 0 | |
| Matt Mullenweg | 0 | 0 | |
| Sahil Lavingia | 4 | 0 | |
| Demis Hassabis | 0 | 0 | |
| Aravind Srinivas | 4 | 0 | |
| Jessica Livingston | 0 | 0 | |
| Drew Houston | 0 | 0 | |
| martin_casado | 7 | 1 | |
| Sarah Tavel | 0 | 0 | |
| Ryan Hoover | 0 | 0 | |
| Reed Hastings | 0 | 0 | |
| Jeff Lawson | 0 | 0 | |
| Stewart Butterfield | 0 | 0 | |
| Howie Liu | 0 | 0 | |
| Eric Schmidt | 0 | 0 | |
| Palmer Luckey | 0 | 0 | |
| Elon Musk | 22 | 0 | |
| Satya Nadella | 0 | 0 | |
| Jeff Bezos | 0 | 0 | |
| Shaan Puri | 1 | 1 | |
| Not Boring by Packy McCormick | 0 | 0 | |
| Jason ✨👾SaaStr.Ai✨ Lemkin | 8 | 0 | |
| Ben Thompson | 1 | 0 | |
| Li Jin | 0 | 0 | |
| Morgan Housel | 0 | 0 | |
| delian | 0 | 0 | |
| Daniel Ek | 0 | 0 | |
| No Priors: AI, Machine Learning, Tech, & Startups | 0 | 0 | |
| Invest Like The Best | 0 | 0 | |
| Lenny's Podcast | 0 | 0 | |
| 20VC with Harry Stebbings | 1 | 0 | |
| Garry Tan | 6 | 0 | |
| Brian Armstrong | 5 | 0 | |
| andrew chen | 0 | 0 | |
| tobi lutke | 4 | 1 | |
| benahorowitz.eth | 0 | 0 | |
| jack | 0 | 0 | |
| scott belsky | 4 | 0 | |
| David Sacks | 1 | 0 | |
| Vinod Khosla | 3 | 0 | |
| Chamath Palihapitiya | 2 | 0 | |
| Fred Wilson | 0 | 0 | |
| Tim Ferriss | 4 | 0 | |
| Sam Altman | 0 | 0 | |
| balajis.com | 0 | 0 | |
| Book of the Day from The Next Big Idea Club | 1 | 0 | |
| The Foresight Playbook | 0 | 0 | |
| Seliria’s Newsletter | 0 | 0 | |
| The Generalist | 0 | 0 | |
| Lenny's Newsletter | 0 | 0 | |
| Elad Gil | 1 | 0 | |
| No Priors | 0 | 0 | |
| sarah guo | 2 | 0 | |
| Bill Gurley | 2 | 0 | |
| Chris Dixon | 0 | 0 | |
| Keith Rabois | 8 | 2 | |
| Patrick OShaughnessy | 0 | 0 | |
| Bill Gates | 0 | 0 | |
| Reid Hoffman | 0 | 0 | |
| Naval | 0 | 0 | |
| Farnam Street | 0 | 0 | |
| Shane Parrish | 0 | 0 | |
| Patrick Collison | 0 | 0 | |
| Balaji | 0 | 0 | |
| Sam Altman | 0 | 0 | |
| Marc Andreessen 🇺🇸 | 0 | 0 | |
| David Perell | 1 | 0 | |
| Paul Graham | 1 | 0 | |
| Parker Conrad | 0 | 0 | |
| Palmer Luckey | 0 | 0 | |
| Suhail | 0 | 0 | |
| Tom Blomfield | 0 | 0 | |
| Guillermo Rauch | 0 | 0 | |
| clem 🤗 | 0 | 0 | |
| Jeremy Howard | 0 | 0 | |
| aileenlee | 0 | 0 | |
| @jason | 0 | 0 | |
| Ryan Hoover | 0 | 0 | |
| Nat Friedman | 0 | 0 | |
| Gokul Rajaram | 0 | 0 | |
| Ivan Zhao | 0 | 0 | |
| Chris Dixon | 0 | 0 | |
| Eric Schmidt | 0 | 0 | |
| Susan Wojcicki | 0 | 0 | |
| Jeff Weiner | 0 | 0 | |
| Kevin Systrom | 0 | 0 | |
| Sundar Pichai | 0 | 0 | |
| Tony Fadell | 0 | 0 | |
| Scott Belsky | 0 | 0 | |
| Drew Houston | 0 | 0 | |
| Marc Benioff | 0 | 0 | |
| Brian Chesky | 0 | 0 | |
| David Heinemeier Hansson (DHH) | 0 | 0 | |
| Sahil Lavingia | 0 | 0 | |
| Andrew Wilkinson | 0 | 0 | |
| jack | 1 | 0 | |
| John Doerr | 0 | 0 | |
| Satya Nadella | 0 | 0 | |
| Mark Zuckerberg | 0 | 0 | |
| Garry Tan | 0 | 0 | |
| Jensen Huang | 0 | 0 | |
| Michael Seibel | 1 | 0 | |
| Daniel Gross | 0 | 0 | |
| Shane Parrish | 0 | 0 | |
| Fred Wilson | 0 | 0 | |
| tobi lutke | 0 | 0 | |
| David Sacks | 0 | 0 | |
| Jeff Bezos | 0 | 0 | |
| Elon Musk | 0 | 0 | |
| Bill Gates | 1 | 0 | |
| Sam Altman | 0 | 0 | |
| Peter Thiel | 0 | 0 | |
| Elad Gil | 0 | 0 | |
| Keith Rabois | 0 | 0 | |
| Reid Hoffman | 0 | 0 | |
| Balaji Srinivasan | 0 | 0 | |
| Tim Ferriss | 0 | 0 | |
| Brian Armstrong | 0 | 0 | |
| Ben Horowitz | 0 | 0 | |
| Naval Ravikant | 0 | 0 | |
| Patrick Collison | 0 | 0 | |
| Marc Andreessen | 0 | 0 | |
| Paul Graham | 0 | 0 | |
| Tomasz Tunguz | 0 | 0 | |
| Brad Feld | 0 | 0 |