We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Big Ideas
AI feature velocity is exposing a product-selection problem. A PM at a funded startup says competitor-chasing is pushing out half-tested workflows and questionable AI outputs; a commenter argues that as building gets easier, choosing what to build becomes the bottleneck. Use model capability for user understanding, experiments, and prototypes—not simply more features. That shift needs quality discipline: a prompt, model, or code change can improve one behavior while breaking another, so evaluation should be repeatable and happen before shipping.
Tactical Playbook
Define failures before metrics. The eval guide warns that teams often jump to metrics before studying actual failures, risking measurement of the wrong outcome. Its leasing-assistant example makes the point: when a prospect says the rent is out of budget, a polite goodbye sounds fine but misses the sales goal—the assistant should offer cheaper units or other properties. Start with the user-facing job, then decide what counts as failure.
- Capture complete traces: user input, system prompt, retrieval, tool calls, intermediate model calls, and final output.
- Review before automating: sample diverse traces; annotate the first 10 in actionable, user-facing terms and focus on the first upstream error. Then let an agent propose more annotations for a human to accept or reject. Continue until learning plateaus—the guide’s rule of thumb is about 100 traces.
- Prioritize recurring failures: cluster and count failure modes across at least 100 diverse annotated traces. In a study of 100 production traces, automated tools caught obvious trace-level errors but missed issues requiring product judgment or outside context, and sometimes flagged good responses. Use automation with human review, not instead of it.
Track what you deliberately reject. Petra Wille and Teresa Torres discuss adding trash-can markers to discovery and delivery boards to record customer problems and solutions the team chooses not to pursue. Use the record to ask whether the team is comparing solution options and whether discovery is working. An empty solution-space bin is a warning; an empty problem-space bin could indicate an innovation or culture problem, but may also reflect a strong strategy filter. Check whether people can safely raise problems, and use the record to retire “zombie” opportunities that keep resurfacing.
Case Studies & Lessons
Lenny cites company-reported outcomes associated with eval investments: Ramp’s automatic receipt-collection accuracy rose from 35% to 83%; Shopify’s AI workflow builder was 2.2× faster and 68% cheaper than the frontier-model setup it replaced; Harvey nearly doubled its contract reviewer’s internal quality score; and Cursor reported higher user satisfaction at 41% lower cost after tuning Auto Balance. These are different measures, but show evals being applied to both product quality and operating cost.
Career Corner
Eval-writing is a hiring signal: Lenny says nearly half of 25 PM openings he shared asked for experience writing evals. Candidates can demonstrate the skill by showing how they identified a user-facing failure, turned it into a repeatable test, and used recurring failures to guide product work—not just by listing AI tools.
Tools & Resources
Try the evals skill from Hamel Husain and Shreya, which Lenny linked as a way to save time and avoid mistakes; pair it with the human review process above.
- Treat error discovery as AI product discovery: identify which failures are worth measuring before writing eval metrics, or risk building assumption-led metrics that track the wrong outcomes. Evals turn quality judgments into repeatable pre-ship tests, and production failures can become additional test cases. In examples cited, Cursor achieved higher user satisfaction while reducing costs by 41%, and Ramp increased automatic receipt-collection accuracy from 35% to 83% after investing in evals.
- Use human judgment to define product failures: a leasing assistant ended politely when a prospect said a unit was out of budget, but missed the sales goal of offering cheaper units or other properties. In a study of 100 production traces, agents missed issues requiring product judgment or outside context, caught obvious trace-level failures, and sometimes flagged good responses as failures—so combine automation with human review.
- Practical workflow: log complete user-session traces, including inputs, system prompt, retrieval and tool activity, intermediate model calls, and final output. Review a diverse sample and inspect multiple examples of each failure before asking an agent to annotate more traces for human acceptance or rejection. Have a human annotate the first 10 traces; describe actionable user-facing problems, avoid root-cause diagnosis, and focus initially on the first upstream error. Continue iterating until human learning plateaus (the authors’ rule of thumb is about 100 traces), then cluster and count failure modes across at least 100 diverse annotated traces to prioritize product issues.
- AI eval-writing is emerging as a PM hiring signal: Lenny says nearly half of 25 PM openings he shared asked for experience writing evals. He cites results associated with eval investments: Ramp’s receipt-collection accuracy rose from 35% to 83%; Shopify’s AI workflow builder was 2.2× faster and 68% cheaper than the frontier-model setup it replaced; Harvey’s contract reviewer nearly doubled its internal quality score; and Cursor’s routing had higher user satisfaction at 41% lower cost.
- Lenny recommends Hamel Husain and Shreya’s evals skill, saying it can save hours and prevent mistakes; he links the tool at https://github.com/ai-evals-course/evals-skills. Their advanced guide draws on work with 50+ AI companies and covers a key step teams often skip, what to automate, and a free plugin that lets a coding agent do much of the work; the post does not specify the step or automation guidance.
Shreyas linked videos on the “Mission Lie” and getting unstuck in a career , what interviewers want from PM leadership candidates , the “Great Manager Lie” , and choosing between job offers . The posts provide topic titles and video links, but not the advice or tactics themselves.
Shreyas Doshi pushes back on labeling people who care about money as mercenary; individuals should set their own priorities among money, scope, and title, while recognizing that money helps support a comfortable life for themselves and loved ones . In career trade-offs, money lasts longer than title, which is lost when leaving a company, or scope, which can disappear in a reorganization; financial security can also enable greater career risk-taking than pursuing scope or title alone .
- Anchor trade-offs in the product/business goal, strategy, target ICP, best-served customer segments, market opportunity, and go-to-market motion rather than scoring feature requests in isolation . One prioritization lens is the overlap of business value, user value, and product strategy; when all three do not align, consider work serving two, then move from strategy to roadmap problem and test solutions quickly and cheaply .
- A suggested startup workflow is to use MoSCoW to filter against the North Star goal, then RICE to rank the shortlist, giving effort meaningful weight; if urgency is still disputed, consider Cost of Delay or Cost of Delay divided by Duration without waiting for perfect estimates, while treating legal/compliance work as a possible override . The RICE advice proposes founder tie-breaking when teams cannot align on scores .
- Ground priorities in customer and business evidence: compare sales losses and support-reported churn/issues with ARR/NRR and acquisition data . One commenter suggests surveying users on roughly 10 frequently requested changes, asking them to rank pain, then using a weighted average to order the backlog .
- Make decision rights explicit: one recommendation is for the PM to facilitate a list of effort/cost and value inputs and keep the decision group to three people or fewer . The original poster planned to retain the spreadsheet as a record of decisions and their rationale for later reference .
- “Don’t believe the Impact Lie (and resist LinkedIn Envy)” (video)
- “Always be looking — the 1% Rule” (video)
- “The LMS Framework for growing your competence” (video)
The posts provide video pointers and titles, but no explanation of the frameworks or their implementation steps.
Shreyas Doshi shared a career video titled “How to talk to executives” : https://youtu.be/WgFAKCEpEsE
Shreyas Doshi says he will add more career videos to the thread, with listed topics including negotiating compensation with integrity , the mindset of successful PMs , how a candidate’s interview questions reveal what they care about , and why not to ignore money .
AI eval-writing is an emerging PM hiring signal: nearly half of 25 PM openings Lenny shared asked for experience writing evals. Reported outcomes associated with investment in evals include Ramp raising automatic receipt-collection accuracy from 35% to 83% , Shopify shipping an AI workflow builder that was 2.2× faster and 68% cheaper than the frontier-model setup it replaced , Harvey nearly doubling its contract reviewer’s internal quality score , and Cursor achieving higher user satisfaction with Auto Balance routing at 41% lower cost.
Use trash can tracking to make discovery decisions visible: add trash-can markers to discovery and delivery boards and record customer problems and solutions deliberately not pursued, so teams can inspect discarded work alongside shipped work and assess whether they are innovating and whether discovery is working. An empty solution-space trash can is a warning that the team may not be comparing multiple options; an empty problem-space trash can is ambiguous, potentially indicating an innovation gap or command-and-control culture, but also possibly a sound strategy filter. The practice depends on psychological safety so support agents, salespeople, engineers, and customers can raise problems for assessment; tracking decisions to walk away can also help retire “zombie opportunities” that keep resurfacing across the organization.
- AI eval-writing is an emerging PM hiring signal: nearly half of 25 PM openings Lenny shared asked for experience writing evals.
- Companies cited product gains from investing in evals: Ramp’s automatic receipt-collection accuracy rose from 35% to 83%; Shopify’s AI workflow builder was 2.2× faster and 68% cheaper than the frontier-model setup it replaced; Harvey nearly doubled its internal quality score for an AI contract reviewer; and Cursor reported higher user satisfaction at 41% lower cost after tuning Auto Balance routing.
Hiten Shah uses Grok Bot in his Tesla to hand off product or growth thoughts that arise while driving. Previously, he relied on remembering them or recording them for later, and says some never made it back into his day; the bot lets him delegate while a thought is fresh without changing his existing in-car routine . For product teams, this is a workflow-integration lesson: make capture and action available in the context where intent arises, rather than depending on users to retrieve an idea later. Tesla describes Grok Connectors as supporting hands-free inbox, calendar, and existing file/chat/task interactions .
- A PM at a funded AI startup describes pressure to copy competitors’ AI features and agents, leading to half-tested workflows and questionable outputs before the team understands whether customers want them; when issues arise, Product is expected to investigate, coordinate engineering, communicate with customers, fix workflows or UX, and keep the roadmap moving.
- Replies argue the underlying challenge is choosing what creates customer value, not simply building faster: copying competitors instead of responding to customer needs signals weak strategy. One commenter says faster building makes deciding what to build the bottleneck; at their regulated company, leadership pursued agentic UX without a clear user benefit, testing was nearly absent, and repeated demos failed legal and brand standards.
- A commenter recommends testing harnesses, staged releases, and rapid feedback-and-improvement loops, while treating stability, responsiveness, and affordability as part of customer value. Another warns automated testing does not find every bug and says dedicated QA with deployment veto power helped protect quality; a separate PM reports that 18 months of bare-bones MVP work without time for refinement left the product creaking.
- One commenter said their nearly year-long PM search eventually produced multiple offers within four months. They observed that employers favored same-industry experience; their cross-industry interviews were mainly for tech PM roles requiring enough technical knowledge to cover delivery-lead work. They moved from EdTech into FinTech by emphasizing lending experience, and said PMs from EdTech, legacy e-commerce, non-AI SaaS, and B2C companies that failed to scale struggled to move sectors. They also said an IIM credential was no longer a strong differentiator because it had become common among PM candidates.
- In their experience, recruiter outreach through Naukri, agencies, and LinkedIn increased around five years of experience, but was for APM/PM roles rather than SPM roles. Portal referrals and broad application/outreach channels brought little, while friends with close HR/TA connections generated a few calls. A startup canceled a role requiring four years’ experience before interviews after receiving hundreds of applications from candidates with 8–9 years’ experience at the same salary band and changing the JD. The commenter said their ex-colleagues also saw five years of experience open doors; most joined FinTech or AI startups, while the commenter joined an MNC.
- With a fixed date, make the trade-off explicit: reduce scope or delay rather than hide the risk or ship a half-baked release. Commenters frame the constraints variously as time/scope/resources or, in practice, time/scope/quality; one warns that adding people on a short timeline can slow work through coordination overhead.
- First clarify what the deadline is meant to achieve: a “live” milestone can matter to the business even when readiness or user value is not the immediate goal. Establish whether leadership has actually accepted the launch risk, and understand what commitments are driving the date.
- Build a credible release plan with engineering: identify what can be completed and made functional, set testing and release red lines, and account for readiness work such as enablement, activation, and billing. Show the risks and follow-up plan for deferred scope, and document knowns, unknowns, and leadership’s decision to proceed or change the plan.
- Make decision capture part of the work already being tracked: one team requires a Jira field before a ticket moves to “in progress,” with 3–4 bullets covering the choice, rejected options, rationale, and sign-off; another PM suggests automating reminders for tickets with missing documentation.
- AI can help capture meetings and update a wiki, but it should not invent rationale for a conversation that was never recorded: mark the reason unknown, ask participants, label any recollection as retrospective, and link the next decision to available evidence.
- When teams do not rely on docs, one product leader uses a problem-first debrief: the PM presents research and aligns stakeholders on the problem before a later solutioning session, then restates why the problem matters in regular syncs. The commenter says this builds shared understanding and helps teams protect scope.
- Documentation has a trade-off: one PM argues that old decision rationale may no longer apply, so prioritize current state and revisit past reasoning when it becomes relevant rather than treating every old decision as an audit requirement.
For a recent math/CS graduate working in risk engineering and considering a master’s to enter product management, one commenter advised against choosing a degree solely for that purpose. Instead, spend 6–12 months owning product-shaped work at Chubb: talk to internal users, define a problem, propose a change, and measure the result; then use those examples to apply for APM roles or an internal product move. If pursuing an MBA, check whether each target school actually places students into PM through campus recruiting, using employment reports and conversations with recent graduates; an MS in product management does not automatically provide that access. A separate commenter argued that direct PM recruiting is primarily available through top-school MBA programs, underscoring the need to assess schools individually rather than assume any graduate degree creates the same opportunity.
In payments and fintech, apparent agreement can mask different interpretations of a requirement, and even a well-written requirements document may encode the wrong understanding of the customer’s intent. The post identifies possible responsibilities across sales (define the commercial commitment), product (translate it into an unambiguous requirement), delivery (challenge implementation assumptions), the client (specify what “done” means), and leadership (create a process and culture for surfacing assumptions); it leaves who can stop a deal or programme unresolved. The author warns that precise pre-signature requirements can make implementation costs more visible, while vagueness can make a deal seem easier to close by pushing uncertainty downstream.
Claude Code users approve 93% of permission prompts; Hiten says that makes “Allow” start to look like “Continue,” a pattern Anthropic calls approval fatigue.
Use model capabilities to spend more time understanding users, testing experiments, building prototypes, and investigating what the team does not understand—not simply to ship more features; the aim is to ship products that work. Hiten Shah agrees with the principle but notes that applying it can be difficult in practice.