We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Treat Every LLM Change as a Hypothesis Test
•
3 min read
•
185 docs
Martin Casado’s standout talk recommendation turns stochastic AI work into a concrete measurement discipline; an open robotics corpus and two lighter organic picks round out the list.
The standout recommendation
An age of experimentation — BlueHat Asia 2026 slides
- Content type / creator: Talk slides. The sharing post is by Halvar Flake, who describes them as slides from his Microsoft BlueHat Singapore talk; the deck itself is titled An age of experimentation and credits Thomas Dullien, Cyber Security Research Team, OpenAI.
- Link:Slides
- Recommended by: Martin Casado, who calls it his favorite talk of the year and praises its clarity in an era of “AI shorthand.”
- Key takeaway: The deck treats reasoning LLMs as more like fire than an internal-combustion engine: discovered more than constructed, still poorly understood, and improved mainly through empirical work. Its practical rule is that every change in a probabilistic system is a hypothesis test; teams should ask what they want to show, measure rerun variation, identify randomness, and check whether changing conditions invalidate an IID assumption.
- Why it matters: This is a useful operating manual for evaluating AI-enabled systems without mistaking activity for progress. The deck also recommends a benchmark hierarchy—small tests for rapid iteration and larger tests for periodic validation—while warning that the small tests must remain representative.
Eidon AI’s egocentric robotics corpus
- Content type / creator: Open robotics dataset/resource, published by Solidic Labs Inc. (Eidon AI). The company has wound down, but the release remains available under CC-BY-4.0, with commercial use allowed when attribution is provided.
- Link:Eidon AI on Hugging Face, including the video corpus and paired motion data
- Recommended by: Clem Delangue, who praised the decision to open-source 1,274 hours of egocentric robotics data across 13,451 recordings and called it a gift to the robotics community. He explicitly argued that open source lets work outlive the organization that created it.
- Key takeaway: The release combines 13,451 egocentric videos with 779 million rows of 24 Hz orientation and motion data covering the same recordings; the two pieces join on
recording_id. The paired set contains 1,274 hours of video and arm tracking from 27 contributors, with automated quality-control scores. - Why it matters: This is an unusually actionable recommendation for embodied-AI work, as well as a concrete example of preserving research assets after a startup shuts down. The dataset is not a neutral snapshot of daily activity: the release warns that tasks are heavily weighted toward folding laundry, contributors are few and unevenly represented, and splits should be made by contributor rather than randomly.
Product and idea-generation picks
“I design teams like terrorist organizations” — Peter Ellis product-advice video
- Content type / creator: A 90-minute YouTube interview featuring Peter Ellis, published through the
@lennysaninterview series. The associated post links to the video. - Recommended by: Keith Rabois, with the unqualified endorsement “Highly recommend.”
- Key takeaway: The episode is organized around concrete product-leadership questions: managing Nikita Bier, why growth usually comes from the core, knowing when to stop, the shortcomings of the median product manager, and team design.
- Why it matters: It is a focused listening assignment for leaders working through the trade-offs between product taste, growth, and organizational design—not a generic startup interview.
Sharif Shameem’s posts archive
- Content type / creator: Personal blog/archive by Sharif Shameem. Posts
- Recommended by: Shaan Puri, who says he had just discovered the blog, calls it “very good,” and tells readers to read a couple of posts.
- Key takeaway: Puri does not identify a particular post or lesson, so this is a browse recommendation rather than a prescribed syllabus. The archive’s visible material ranges from The 28 AI tools I wish existed—framed by the question “where are the tools?”—to proposals about improving large video calls and modeling programming as a reinforcement-learning task.
- Why it matters: Treat it as an idea-generation archive: useful for finding sharp product and technical questions, but not evidence that the listed concepts were built, tested, or validated.
Treat Every LLM Change as a Hypothesis Test
Research extraction
Direct answer: This is Sharif Shameem’s personal blog/posts archive. The page identifies “Sharif Shameem,” links to “Posts,” and labels the collection “Archive,” supporting attribution of the resource to him.
- The clearest concrete AI-related takeaway is that the archive contains a September 4, 2025 post titled “The 28 AI tools I wish existed,” described as “The models are powerful as is. But where are the tools?” This supports characterizing it as a collection of proposed or desired AI-tool ideas—not, from the index alone, evidence that 28 tools were built, tested, or endorsed.
- The resource is broader than an AI-tool list: visible entries include an ebook browser, ideas for improving large video calls, a startup manual, a React/Chromium/HTML-spec issue, and a programming/reinforcement-learning post. The safest description is therefore a personal collection of essays, technical notes, and project concepts rather than a single-purpose reference.
- The supplied material is an index showing titles, short descriptions, dates, and links—for example, the AI-tools entry—so it does not establish the posts’ implementation status, evidence, testing, or efficacy.
Research extraction
Direct answer: The linked Hugging Face resource is the Eidon AI organization release. It is one corpus organized into three components: tracker-pov, tracker-pov-imu, and the egocentric-pov Storage Bucket; the first two cover the same 13,451 recordings.
Creator/publisher and licence: The release is published by Solidic Labs Inc (Eidon AI). Eidon AI says the company has wound down, while the collected data remains publicly available under CC-BY-4.0; commercial use is allowed with attribution.
Dataset/resource titles and links:
tracker-pov— 13,451 egocentric MP4 videos, with one metadata row per video; listed size is 9.05 TB and the resource is loadable withload_dataset.tracker-pov-imu— 779 million rows of 24 Hz orientation and motion data for the same recordings; listed size is 9.5 GB and it is loadable withload_dataset.egocentric-pov— an additional 1.55 TB of egocentric video without sensor data. It is a Storage Bucket rather than a Dataset, so it must be accessed withhf buckets syncor the S3-compatible API.
Substantive contents: The paired tracker release contains 1,274 hours of egocentric video and seven-point arm tracking from 27 contributors in their homes; recordings underwent automated QC for hand presence, lighting, blur, and camera stability, with QC scores included. The additional bucket contains 306 hours across 1,370 recordings from 37 contributors, including 14 who also appear in the tracker set; it uses the same rig and QC pipeline but has no IMU harness.
Usable access details:
from datasets import load_dataset video = load_dataset("eidon-ai/tracker-pov", split="train") imu = load_dataset("eidon-ai/tracker-pov-imu", split="train", streaming=True) meta = pd.read_parquet("hf://datasets/eidon-ai/tracker-pov/recordings/metadata.parquet")The bucket can be synced with
hf buckets sync hf://buckets/eidon-ai/egocentric-pov ./egocentric-pov.Recommendation caveat: The source warns that the task distribution is heavily weighted toward folding laundry, contributors are few and unevenly weighted, and train/test splits should use
contributor_idrather than random splitting; it links to the dataset limitations section.
Research extraction
Lenny's Podcast
Lenny Rachitsky
Profile
Direct answer
The supplied BlueHat deck supports an operating recommendation to treat LLM-enabled software and security work as an empirical, stochastic activity: it says software engineering and software security are in an “age of experimentation,” with no strong abstract model currently available.
Source identity and coverage caveat
- Authorship conflict: the supplied deck is titled An age of experimentation and its title page credits Thomas Dullien, Cyber Security Research Team, OpenAI.
- Missing comparison source: the supplied video bundle contains only one YouTube site-navigation line, not the requested full Peter Ellis/Lenny Rachitsky transcript; no substantive video takeaways or cross-source comparison can therefore be extracted.
Concrete takeaways from Dullien’s deck
- Treat reasoning LLMs as discovered, not fully engineered. Dullien compares reasoning LLMs to fire rather than the internal-combustion engine: they were “discovered more than constructed,” understanding is nascent, and improvements are very empirical.
- Assume semantic nondeterminism. The deck says adding LLMs introduces a source of stochasticity and claims that LLMs mean the “death of semantic determinism”; it describes outputs as random in practice, with additional randomness from hardware and concurrency. Small prompt changes are not analytically predictable: the slide says that, in formal-verification terms, “anything can happen” (but rarely does).
- Make every change a measured hypothesis test. For probabilistic systems, Dullien says teams should define what they want to show, account for variance, measure identical rerun variation, identify randomness sources, and determine whether time-varying factors make observations non-IID.
- Control shared-environment noise where possible. In the cloud-performance example, noisy neighbors are said to add ±15% runtime variation, require hundreds or thousands of runs to show an improvement, and destroy the IID assumption. The proposed “Tomato farmer protocol” uses paired experiments on the same VM so both instances experience the same noise, with the caveat that this is not always possible.
- Do not assume ordinary CI/CD is adequate for prompt changes. Dullien calls regular CI/CD processes poorly adapted to prompt changes and says evaluating whether a prompt-changing PR is worth merging can cost more than a developer workday. His caveat is that merging on a hunch is not necessarily harmful; the danger is spending substantial effort on iterative changes while mistaking activity for steady progress.
- Evaluate agent improvements with dependence and sample size in mind. A full-codebase agent run may be only one trial because path dependence means bugs are not independent—for example, an agent may skip a file. Even apparently independent bug tests can share underlying success factors, producing confidence intervals that are too narrow. A worked example says that, under deliberately false assumptions of normality and 2% standard deviation, 52 paired runs would be needed for a 95% chance of detecting a 1% improvement; the slide warns that reality is worse.
- Treat LLM code review more like fuzzing than traditional SAST. Dullien says each run is analogous to a fuzzing run, code review is never “done,” and exact reruns, small prompt variations, and new model generations can produce different bug sets. He identifies useful discoveries per dollar of compute as the relevant metric.
- Use a two-level benchmark hierarchy. The proposed pattern is to use small benchmarks for rapid iteration and larger benchmarks periodically for validation, while explicitly checking that the small benchmarks remain representative.
- Avoid claiming complete bug coverage. Dullien’s “fishery” model says security bugs are introduced and removed at rates nobody fully knows, so teams can track effort and catch rates rather than total bugs. He says catching every bug may be unrealistic; a more attainable outcome is making further discoveries uneconomical, while acknowledging that improved technology may find previously hidden bugs.
- Keep harnesses moving, but label this as an expectation. Dullien expects clever harnessing to boost any model on any task over the next few years, while also expecting those gains to be absorbed into later training runs; consequently, each harness would need to keep evolving to remain relevant.
- Pair rigor with proportionality. Dullien says there is no way around accepting stochastic systems and uncertainty, but also says statistical rigor is not an absolute necessity because some true claims are costly to demonstrate; his warning is against “treading water while feeling productive.”
Research extraction
Keith Rabois
martin_casado
Shaan Puri
Direct answer: The Halvar Flake talk, the Lenny-linked resource, Sharif Shameem’s blog, and Morning Brew receive clear positive endorsements; the Stratechery item is favorably described but not explicitly recommended.
- Halvar Flake’s talk — recommender: Martin Casado. Casado calls it “my favorite talk this year” and “just exceptional,” adding that, “in the era of AI shorthand,” clear thinking is “at a premium” and the talk is “as clear as it gets.” This is an explicit, strong endorsement; its substantive takeaway is the value of clear thinking and clarity in the AI-shorthand era.
- Stratechery’s Frontier Overhangs — recommender: Keith Rabois. Rabois says only “Several highly provocative thoughts” and shares a Stratechery link. That is favorable sharing or description, not an explicit recommendation to read it, and the post gives no substantive takeaway beyond the “provocative” characterization.
- Peter Ellis/Lenny Rachitsky product-advice video — recommender: Keith Rabois. The exact recommendation is “Highly recommend,” attached to a @lennysan link. This is explicit endorsement, but the supplied post contains no product-advice takeaway and does not name Peter Ellis.
- Sharif Shameem’s blog — recommender: Shaan Puri. Puri says he “just discovered” the blog, calls it “very good,” and explicitly tells readers to “go read a couple posts,” linking to sharif.io/posts. The context is a newly discovered blog; no specific post or substantive blog takeaway is identified.
- Morning Brew — recommender: Tobi Lutke. Lutke’s exact recommendation is “definitely use it” and “use it well.” His follow-up says the instructions were “Too complex a set of instructions for various people around here it seems,” providing usage context. This is explicit endorsement plus usage guidance, not mere promotion, but no content-level takeaway about Morning Brew is supplied.
Keith Rabois
- “I design teams like terrorist organizations” — Peter Ellis product-advice video (YouTube). Keith Rabois wrote “Highly recommend” while linking to Lenny Rachitsky’s post about the resource. The 90-minute episode covers managing Nikita Bier, why growth usually comes from the core, knowing when to stop, the median PM’s shortcomings, and designing teams; the video is available at https://www.youtube.com/watch?v=97LRJUUPy_w.
Keith Rabois
- Keith Rabois highlighted Stratechery’s article “Frontier Overhangs”, calling it “Several highly provocative thoughts.”
- The article’s central idea is that slowing frontier progress, even if sincere, could strategically give frontier labs time to reduce overhangs created by rapid model advancement.
martin_casado
- “An Age of Experimentation” — BlueHat Asia 2026 talk slides (PDF by Halvar Flake): access the slides. Halvar Flake shared the slides from his Microsoft BlueHat Singapore talk. Martin Casado called it his favorite talk of the year and praised it as “exceptional,” highlighting its clear thinking in an era of AI shorthand.
tobi lutke
Shopify CEO Tobi Lütke explicitly recommends Morning Brew, writing “definitely use it” and “use it well.”
clem 🤗
- Eidon AI egocentric robotics dataset — dataset/resource: Clem Delangue endorsed Eidon AI’s release of 1,274 hours of data across 13,451 recordings of humans performing everyday tasks, describing it as a gift to the robotics community. He highlighted open source’s ability to let work outlive the organization that created it and encouraged more startups to follow this model. Resource: https://huggingface.co/eidon-ai
Shaan Puri
- Sharif Shameem’s blog (blog): Shaan Puri calls it “very good” and recommends reading a couple of posts. https://sharif.io/posts
