We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
AI Makes Prototypes Cheap—and Product Judgment More Expensive
•
4 min read
•
249 docs
AI is making prototypes and software experiments dramatically cheaper, but the period’s strongest signals point to a sharper divide between learning artifacts and reliable products. The practical response is stronger evidence standards, guarded context access, and clearer expectations for AI-era PM roles.
Big Ideas
Separate build-to-learn from build-to-earn. Teresa Torres and Petra distinguish cheap, disposable interactive prototypes from production software that customers pay for. The latter still requires maintainability, non-functional requirements, reuse, security, scalability, cost control, and skilled human oversight. AI products add error analysis, meaningful evaluations, and prompt/orchestration iteration; Torres describes the first 60–70% as relatively easy prototype work, while the final 30% can take months to years. Apply: label every artifact as learning or production before work starts, and do not let a successful prototype waive production gates.
Cheaper building raises the evidence bar. Hiten Shah argues that AI can make a weak idea look real—a polished site, working demo, and persuasive analysis—while changing little for customers; compliments, signups, repeated use, displaced behavior, payment, and retention are progressively stronger signals. In Beam’s research, more than 100 people replied and 42 documented their workflows; 29 interacted with another Mac several times a day, 28 kept it nearby, and only four usually wanted the whole computer while 18 wanted one application or a few specific ones.
Tactical Playbook
Use AI as an internal context multiplier, not an unsupervised customer oracle. One workplace experiment connected Claude to server logs, code, Swagger, ticketing, and Sentry to answer impact and incident questions, then proposed a Slack/Teams or SaaS interface for non-technical colleagues. A separate team found that exposing the codebase directly to support produced confident but incorrect customer answers, especially when answers required multiple repositories, business context, configuration/data, or judgment beyond the visible code. Start with three gates: connect the sources; use the agent to accelerate technical staff; then expand access only as documentation and permissions are good enough.
Validate replacement behavior, not enthusiasm. Match the research method to the unanswered question and its cost: conversations for premise checks, observation for workflow detail, manual fulfillment for willingness to pay, and software when real use will teach more than another discussion. Once a product is in the workflow, ask, “If it were not installed, what would you have done instead?” Look for the old workaround to disappear, then measure repeat use, payment, and retention. Your own pain is customer zero—not proof of a market; the idea still has to earn customer one by surviving contact with other people’s workflows.
Case Studies & Lessons
Superhuman is extending meeting notes into follow-on work. Superhuman says users had long requested a notetaker and that Fathom’s category expertise helped drive the decision. Its stated opportunity is not merely capturing and summarizing meetings, but turning what was discussed, decided, and committed into useful work by combining meeting context with email, calendar, documents, and other workplace data. The teams are building the roadmap in parallel, with a planned Superhuman Mail integration. The product lesson: define the downstream job an AI artifact should trigger, then connect it to the systems where that job is completed.
Career Corner
Product-sense interviews reward deliberate narrowing. Aakash Gupta’s rubric spans prioritization, user empathy, structured thinking, creativity, judgment under ambiguity, and metrics fluency. His recommended shape is to defend one segment, rank pains by severity × frequency, choose one solution while naming its trade-off, and define a success metric with a counter-metric; practice one question aloud, timed, each day.
Scope the “AI PM” role before preparing for it. A current hiring discussion describes startups testing system design, API debugging, and model-latency understanding because small teams need contributors who can ship. But another practitioner argues that PMs should own the customer and business outcome, understand what technical evaluations measure and when they matter, and not be pushed into doing a data scientist’s job. Ask what the role will own over the next year; if it is meant to discover the path through repeated pivots, hiring advice favors a creator-builder with product sense rather than a generic “AI PM” label.
Tools & Resources
A lightweight agentic shipping review. The Product Compass guide points PMs to a free pm-skills repository and a practical sequence: compare product documentation with the codebase, audit unit/integration/end-to-end/live/CI tests, and request prioritized code, security, and performance findings. Keep a human in the loop for common user flows: the guide notes that agents can miss dynamic or briefly visible UI elements, so manually clicking through those flows remains necessary.
AI Makes Prototypes Cheap—and Product Judgment More Expensive
Dan Olsen
- AI-era build-and-learn loop: Use AI to turn an idea into a quick working prototype and interact with it before detailed scoping; then design the user experience and engineering architecture for a scalable, stable product before shipping and learning. This reverses the older idea-to-spec-to-scope-to-build sequence, but prototypes are not production systems and design and engineering partners remain necessary. AI’s lower build cost shifts PM leverage toward judgment: choosing which feasible ideas fit the product and create impact, keeping the team aligned, and applying taste and curation to avoid low-quality feature accumulation.
- Vision and product health: Define product vision as the product’s purpose, core user actions, and expected cycle or frequency—not just a mission statement—and use that expected behavior as the design target. Measure people who arrive directly, perform the core actions rather than merely open the product, and return on its natural cadence; analyze cohort retention over months where appropriate and distinguish novelty from durable retention.
- Evidence and alignment: Pair metrics with user anecdotes by inspecting prompt or product logs and interviews, reading 100 actual user-scenario transcripts, and asking unsuccessful users what they were trying to do and what would have made them happy. AI can summarize evidence, but PMs must interpret it and form the product story. The PM artifact is not the spec itself but a repeatable story and shared picture of why users need the product; AI may help draft documentation, but the PM must own the voice and stand behind every word.
- Onboarding as product storytelling: For curious users who are not yet committed, design for the broad “fuzzy middle” rather than only highly motivated users: assume several minutes of attention, teach through small sequential steps with clear actions, explain why information is requested, use real tasks instead of tours or lectures, and deliver value with the user’s own data within about 60 seconds. Evaluate onboarding by next-day, next-week, and next-month return plus completion of the core action—not by completion of the onboarding flow. Twitter’s signups often failed to return because users could not explain what Twitter was; concept-by-concept onboarding taught tweets, following and timelines, interests, and friends without adding features, and retention improved more than from anything else shipped that year.
- Defensibility against incumbents: A product that only fills a gap an incumbent can cover with one feature is vulnerable; use the initial wedge to become an independent destination with its own core action, direct traffic, and system. Musical.ly illustrates this path: it began as a place to create videos that users might otherwise post to Instagram, then built its own feed, creator relationships, and music-label deals so users visited it directly.
- Career and company selection: For startup roles, filter for products you would use, a clear purpose, and a plausible path in which successful use leads to more users; proactively identify exciting companies and contact founders rather than waiting for inbound opportunities.
The Product Compass
- AI-era PM role: The newsletter argues that coding can now be fully delegated to agents, shifting PM leverage toward deciding what to build and managing agents; it also advocates reciprocal upskilling, with PMs learning to build and engineers learning product management.
- Agent-assisted shipping workflow: For AskOne, the proposed development-to-production loop is to keep environments aligned, have the agent modify code and create a database migration, apply it in development, commit locally, push the development branch, promote changes to production, and let Netlify and Supabase deploy automatically.
- Product and quality review method: PMs can use agents to compare product documentation—strategy, segments, value proposition, permissions, user flows, business rules, and integrations—with the codebase, identify gaps, and recommend updates, while the PM verifies that the inferred behavior matches the intended product. The next pass audits unit, integration, end-to-end, live, and CI tests, followed by a code, security, and performance review that reports findings by priority and category with pre-release actions. Agents should not replace a manual walkthrough of common user flows because they can miss dynamic or briefly visible UI elements.
- Production-readiness trade-offs: Development payments remain fake because real Stripe transactions incur fees even when refunded; production requires connecting Stripe to the production Clerk instance, enabling organization billing, and validating the real Premium checkout, priced in the example at $20/month, with Clerk adding 0.7% per transaction on top of Stripe fees. The article also distinguishes plan entitlements from rate limits: plans constrain what an organization may do, while rate limits prevent excessive request frequency; Netlify’s free tier includes DDoS protection, geographic/IP rules, and two code-defined rate-limit rules, while its full OWASP Top 10 WAF requires enterprise.
Shreyas Doshi
Profile
- Use the joy–ego–table stakes framework for job decisions: Title, money, and scope may attract someone to a company, but culture, growth, and people are presented as the factors that determine sustained fulfillment. The framework defines table stakes as interest, impact, and harmony; ego as title, money, and scope; and joy as people, culture, and growth. Reject roles that fail the table stakes, use title/money/scope to narrow the field to two or three finalists, then choose among them based on joy rather than accepting a role because of analytical factors alone. To make the final choice concrete, spend about 10 minutes per finalist vividly imagining the day-to-day work, colleagues, meetings, and problem-solving, then compare the resulting feeling; in the speaker’s 2015 example, this exercise led to choosing Stripe even though two other companies ranked higher on the spreadsheet.
- Evaluate the people and talent behind an offer: Look beyond prepared interview questions by assessing the quality of interviewers’ real-time follow-up questions and whether the hiring process is rigorous without being either excessively easy or onerous. For sufficiently senior roles, conduct a reverse-interview with several employees; another tactic is to ask the hiring manager to identify the company’s three or four strongest people across functions and speak with them. Validate claimed talent through customers and through the product’s insight, execution pace, quality, differentiation, and market performance.
- Ask for more time when offers do not align: Tell the recruiter or hiring manager that the company is among the top two or three choices, explain that the next role merits a five-to-seven-year commitment, and argue that rigorous evaluation is more important than saving a few weeks; close by asking whether the request is reasonable. The tactic is especially applicable to director-, VP-, SVP-, CPO-, and CEO-level roles, but should be used with judgment.
Hiten Shah
- Discovery method: Match the learning method to the unanswered question and its cost—conversations, observation, prototypes, manual delivery, or a working product. AI has made useful software cheap enough that putting a product into a real workflow can sometimes be the fastest way to learn, but it does not replace customer research.
- Validation sequence: Treat personal frustration as a starting hypothesis, not proof of market demand. A founder can start as customer zero when they understand the problem unusually well and can make a small bet; then they should find people with the problem, study their current workarounds, put the product into that workflow, and observe what changes. The evidence bar should rise from compliments to signups, repeated use, disappearance of the old behavior, payment, and retention; asking what users would have done without the product reveals what it actually replaced.
- Beam case study: Beam began as a small internal bet to solve the team’s recurring need to use another Mac through Screen Sharing. Instead of asking whether people wanted better remote access, the team asked users to show their multi-Mac setups: more than 100 replied and 42 completed a baseline form; 29 interacted with another Mac several times a day, 4 did so about once a day, 28 kept the other Mac in the same desk, house, or office, and 18 usually wanted one application or a few specific ones versus 4 who wanted the whole computer. This shifted the research toward app-level workflow access and exposed broader use cases involving unattended Macs that periodically need human intervention, while leaving product quality to be judged by behavior change and displaced workarounds.
Hiten Shah
- Choose the cheapest learning method that answers the current question. Conversations, observation, prototypes, manual fulfillment, and software each have different costs; as AI makes research, iteration, and useful software cheaper, building the product can sometimes become the fastest way to learn.
- Treat yourself as customer zero, not proof of a market. A founder can start with a deeply understood personal problem and a small, low-risk build, but must then find people who already experience the problem, understand their current workflow, put the product into it, and observe what changes; customers provide the final validation.
- Beam demonstrates behavior-first discovery. Shah’s team built Beam from its own multi-Mac problem, then asked people to show their existing setups rather than merely asking whether they wanted the product. More than 100 people replied and 42 documented their workflows; 29 interacted with another Mac several times a day, 28 kept it nearby in the same home or office, and 18 typically wanted one application or a few specific apps rather than the entire computer.
- Validate replacement and retention, not enthusiasm alone. The strongest evidence is a prior workaround disappearing—such as fewer Screen Sharing sessions or less walking to another machine—followed by repeated use, payment, and retention; Shah explicitly asks what users would have done if Beam had not been installed.
- Run the next learning loop with real users. After the initial 42-user study, Shah planned a small X group with free Beam access so the team could learn from how early users actually use it.
Teresa Torres
Profile
- Separate build-to-learn from build-to-earn. Use AI to rapidly create interactive, throwaway prototypes for discovery and stakeholder learning; the prototype should be discarded rather than treated as production software. Build-to-earn work, intended for paying customers, still requires production-quality engineering.
- Keep production quality gates. For build-to-earn work, preserve attention to maintainability, non-functional requirements, code reuse, maintenance cost, security, scalability, and human review of architecture. Rapidly adding AI-generated features can produce tangled data models, duplicated code, poor performance, and difficult future changes.
- Treat AI product delivery as an evaluation and domain-expertise problem. Teams need error analysis, meaningful evaluations, prompt and orchestration iteration, and expert judgment; overlapping failure modes can make evaluation a complex data-science problem. The speakers describe the first 60–70% as relatively easy prototyping, while the final 30% of reliable, high-quality product can require months or years of work. Discovery may become more important, but delivery has not disappeared and both must continue to matter.
Hiten Shah
- Discovery sequencing: The author’s usual approach is to start with customers—understand their struggle and existing workarounds before investing months—but Beam used a different sequence because its first version was cheap enough for the product itself to become research.
- Concrete validation loop: Beam addressed a multi-Mac workflow in which development and QA required switching between machines, while Screen Sharing made retrieving a window cumbersome; its alternative was to make a window running on one Mac available on another. After shipping, outreach about unusual multi-Mac setups produced more than 100 replies and 42 baseline forms, revealing unanticipated use cases and better product language. A Grok Bot then researched the same behavior across X, finding more people with the problem and exposing incorrect assumptions.
- Early-user learning: The next step was to gather early users in a small X group chat and give them Beam for free while learning how it fits their existing setups. The resulting heuristic is: “Scratch your own itch. Ship it. Then find out how many other people are scratching the same place.”
a16z
- Prioritize against the mission, not the most exciting individual project. OpenAI made “focus” its operating theme, evaluated which areas reinforced its mission and the agentic-coding opportunity, and canceled the high-profile Sora project when it was judged a distracting “side quest”; it also prioritized bringing ChatGPT’s consumer and enterprise efforts together. When metrics were off, leadership emphasized controllable inputs and execution basics rather than trying to directly force outcomes.
- Treat product education and re-engagement as core product work. AI products should proactively explain new capabilities instead of making users discover them, while offering voice and text access, persistence, memory, context, trust, and proactive help with personal and work goals. The proposed product north star is a simple, unified AI that reduces the need for users to adapt themselves to fragmented computer interfaces. Brockman estimated that roughly 1.5 billion people had tried ChatGPT but no longer used it, highlighting a large opportunity to re-educate and reactivate lapsed users.
- Computer-use agents expand coverage but introduce a new security trade-off. Operating through the same screen, keyboard, and mouse interface as a person can let agents handle software tasks without teams building a connector for every application, but the added computer-control layer creates new security challenges.
- Match leadership attention to the highest-leverage bottleneck. Brockman described moving his focus from data centers, infrastructure, and machine-learning engineering to the business once research and infrastructure were operating effectively. His execution practice is to lead from the trenches: ask persistent clarifying questions and bring everyone who touches a problem into a shared discussion and document, improving both small wording decisions and larger strategic execution.
- Build AI safety into the full product lifecycle. In the emerging “AGI era,” safety, security, and alignment should be addressed from development through evaluation—not treated as deployment-only concerns—with operational processes capable of producing the required guarantees.
Product Management
- In complex, regulated B2B 0→1 products sold through RFPs, AI has made engineering and high-fidelity prototyping much faster, but the UX team’s underlying design reasoning remains a bottleneck; AI-generated designs can look complete while missing nuance and requiring repeated rework. The thread distinguishes producing an artifact quickly from reaching the right experience or decision, warning that cheap feature generation can create “button soup” unless teams still account for system structure, dependencies, and future patterns.
- A practical operating model is to use AI for bounded, recurring assets and build designs in pieces rather than generating an entire product at once. Preserve human attention for foundational product building blocks, accepting “directionally correct” early work where iteration is safe but avoiding shortcuts that create expensive recoding later. PMs should treat the issue as a flow and resourcing decision—rebalance UX capacity or deliberately slow/sequence engineering instead of demanding that every function match engineering’s AI-accelerated speed. One practitioner reports that, in another context, AI-accelerated teams are instead slowed by cybersecurity and DevOps bureaucracy.
Product Management
- AI PM interview expectations are shifting toward hands-on technical delivery. The discussion treats “real” AI PM experience as having built and shipped production agentic products with harnesses, evaluations, and full observability, while top-company interviews are described as increasingly requiring LLM prototyping plus engineering-level system design. A startup-focused commenter says small teams want PMs who can ship, debug basic API calls, and understand model-latency problems rather than only write specs and coordinate meetings.
- AI PMs should own business and customer outcomes, not substitute for data science. One commenter assigns technical model evaluations to data scientists while expecting PMs to understand what the evaluations measure, when to run them, and why they matter. For evaluation practice, the discussion recommends asking what evaluation was performed, how it was conducted, and what it found; it specifically warns that reviewing 100–200 outputs and saying “looks good to me” is not rigorous analysis.
- The harder-to-automate PM skills are judgment and influence. Beyond understanding model and technology trade-offs, the discussion highlights stakeholder trust, customer-feedback mechanisms, managing up, using data to explain decisions, and selling prioritization and vision as differentiators from simply generating PRDs and managing delivery artifacts.
- Feature-factory work is viewed as especially exposed to automation. Commenters argue that checklist-driven work—turning requested epics into features, adding tracking, and coordinating handoffs—can be automated, whereas durable PM value comes from deciding what to build, in what order, and why, and from figuring out how to change the underlying system to achieve the desired outcome.
- AI PM roles should be scoped by ownership before hiring. A hiring recommendation is to clarify the short- and long-term work first; if the role is meant to discover the path forward, prioritize a creator-builder with product sense, technical fluency, and the ability to navigate repeated market pivots rather than applying one fixed definition of “AI PM.”
Product Management
- AI is challenging conventional delivery rituals: The discussion frames coding agents as able to break down goals, sequence and prioritize work, write/test/iterate code, and surface blockers or alternatives, prompting teams to reconsider sprint planning, backlog refinement, story pointing, manual sequencing, and stand-ups.
- A PM-led 0→1 build shows a lightweight AI workflow: GeorgeHarter used Claude Code to set up and build an app, supplying a couple of pages of clear requirements and iterating through conversations with Claude and Claude Code. The app took about 20 hours over one week, was working well, and had progressed to recruiting pilot companies.
- Enterprise use has produced serious delivery friction: In a large bank’s internal product environment, David_Browie reports faster engineering but more defects, less understanding of what teams are building, and more product course-correction because engineering moves ahead without consulting product; he also describes increased production and consumption of low-quality output (“slop”).
- Discovery and adoption can deteriorate when AI accelerates feature output: Afton11 reports that discovery has been sidelined, quality has declined, internal silos have increased, and user adoption of recent AI-generated features has been poor.
- “Builder PM” roles are emerging with workload concerns: Old-Statistician says recruiters are approaching them for 0→1 builder-PM roles and values the end-to-end span from user research and goal definition to implementation details and debugging, but worries that employers will expect broader communication plus continuously increasing speed and output.
Hiten Shah
Beam is adding early users through a narrowly defined workflow hypothesis: people who have another Mac doing real work but still occasionally need access to its GUI. The follow-up discovery question asks what runs on that Mac and what causes users to open the GUI, helping clarify the target use case.
The community for ventures designed to scale rapidly | Read our rules before posting ❤️
- Frame a failed startup as concrete product and customer evidence, not simply as a success/failure outcome. Hiring perspectives in the thread treat raising funding, launching an MVP or product, and securing paying users as positive signals even when the company later shuts down. In interviews, explain what worked, what did not, how users behaved, the trade-offs made, and what you would change; keeping a journal of the company’s progress can make that account more specific.
- Tailor the career narrative to the target product context. Early-stage companies may value people who learn quickly and flex across roles, while larger technology employers may place more weight on focused, role-specific expertise. Interviewers may also test whether a former founder can collaborate, disagree and commit, and work without unilateral authority over product or implementation decisions.
Hiten Shah
Hiten Shah warns that teams can appear extremely busy while learning almost nothing; activity becomes dangerous when it stops generating new information.
The community for ventures designed to scale rapidly | Read our rules before posting ❤️
- One startup perspective is that early-stage teams can rely heavily on AI for building; the more important product bottlenecks become getting rapid feedback on what was built and getting users to care.
- AI-tool fluency should not replace fundamentals: candidates need to understand underlying logic, review generated code, and debug when AI fails; otherwise they may be unable to repair breakage. A CS degree is described as a proxy rather than the core criterion, so hiring should assess demonstrated competence and company value while ensuring the code owner has or develops fundamentals to identify issues over time.
- AI-heavy teams should also be assessed for technical sovereignty and debt management: one commenter warns that heavy AI reliance can create tech debt and questions whether the team can recognize, measure, and mitigate it while retaining ownership of its IP and technology.
Product Management
- A practical AI knowledge-access workflow connected Claude through a custom MCP server to server logs, the codebase, Swagger, ticketing, and Sentry, enabling questions such as which product areas an API change affects and when a customer error began. The initial Python implementation took an afternoon; the proposed nontechnical-user interface combined connected agents with Slack/Teams and a SaaS setup flow.
- Directly exposing the codebase to support produced confident but incorrect customer-facing answers. Reported failure modes included questions spanning multiple repositories, missing business context, unseen configuration or data, and hallucinations, so a safer operating model is to use the agent to help technical staff answer technical questions rather than let it answer customers directly.
- Adoption still depends on documentation and access coverage: commenters said the approach works best with strong documentation, while restricted or poorly documented resources remain a problem in large companies; indexing code was valued for revealing the product’s current state.
Product Management
- Hamel Husain and Shreya Shankar’s Evals Course was recommended as “the best foundation” for AI product management learning, in the commenter’s opinion.
- Before choosing a course, one commenter advised defining the intended learning goal or identifying the day-to-day blocker; another mentioned GoPractice but had not tried its AI courses.
- A skeptical commenter argued that AI PM course material is generally available free online and may be superficial, so prospective learners should evaluate whether a paid course offers meaningful differentiation.
Product Management
LinkedIn departure posts are optional rather than a required career ritual; one commenter says nothing bad happens if you skip them. Posting can still be useful as a low-effort networking tactic: it announces a job change without individual outreach, keeps former customers and work partners informed, and may help relationships follow you to the next role—especially in network-intensive work. Other reported benefits include reconnecting with former colleagues, building a personal brand or services business, receiving work offers, and increasing visibility into job postings.
Product Management
- The move from PM to Senior PM is especially challenging because it shifts the job from owning a defined domain to relying increasingly on influence, particularly in an unstructured startup; feeling overwhelmed during the transition is not unusual.
- Effective mentorship often develops through sustained collaboration and enough trust for honest conversations, rather than by directly recruiting someone into a formal mentor role. Direct outreach can still work, but finding a strong fit may require contacting a broad pool of candidates.
- A practical sourcing tactic is to search LinkedIn for “Open to Work” former Senior PMs or higher who have startup experience, then contact them and offer compensation for guidance.
Product Management
- For an early-stage concept without user evidence, a proposed discovery sequence is to interview customers about problems and current workflows; synthesize needs and unknowns; bring cross-functional experts together to challenge findings and explore solution directions; prototype and test with customers; then use the evidence to define the MVP.
- In stakeholder-heavy discovery, gather input from the broader team but avoid involving everyone in solution design, which can slow progress and dilute expert judgment; instead, empower the R&D team to define, validate, and execute the solution. An executive sponsor and steering committee can help resolve stakeholder deadlocks, while frequent demos and user testing maintain alignment during delivery.