ZeroNoise Logo zeronoise
Post
Agent-Built Products Shift PM Leverage to Intent and Evaluation
1 day ago
4 min read
198 docs
A product-engineering case and emerging self-improving loop show PM leverage moving toward intent, evaluation, and outcome judgment, with practical guidance for roadmap trade-offs, AI use, feedback systems, career positioning, and meeting data.

Big Ideas

The product loop, not the dashboard, is becoming the category. Amplitude’s Wave has been running on Amplitude’s own product for the past few months, finding problems, shipping fixes, and measuring whether they worked. A related description of the emerging loop is: observe behavior → find something worth improving → make the change → measure the result → learn → repeat; coding agents make changes cheaper while analytics supplies context for deciding what should change. Hiten Shah’s sharper framing is that once software can observe, change, and measure itself, “analytics” is only one step in the loop. The PM implication is practical: define which behavior should change and what evidence would count, rather than treating analytics as the end product.

Product engineering is becoming a PM competency—even without coding. Product Compass argues that product and engineering roles are merging and that PMs need a basic understanding of engineering with AI, not necessarily the ability to code. Its author reports an AI-built project with 105K+ installs, 37K MAU, 5,094 tests, and roughly 177K lines of code, with no code reviewed by the author. The trust mechanism is intent plus evaluation: many tests impersonate users in end-to-end browser sessions, while documented intent gives agents something to design tests and inspect the codebase against.

Tactical Playbook

Turn “no capacity” into a trade-off. Keep the division clear: Product owns what/why, Engineering owns how, and the two functions—with business input—determine when. When capacity ends the conversation, bring Engineering into discovery earlier, get rough estimates and expected investment before committing, make scope, priority, dependency, and MVP choices explicit, then slice the work into regularly shippable pieces. Escalate a genuine resourcing problem rather than pretending PM owns velocity; protect delivery with review gates, QA/checklists, and explicitly negotiated technical-debt capacity.

Use AI as a challenger, not a decider. One PM’s report describes an assistant praising an unvalidated idea, returning pros and cons without choosing, fabricating competitor pricing, and rating nearly every RICE item medium-high. The same report says AI still helps with drafts and research synthesis, but stakeholder alignment, technical constraints, and user validation remain the bottlenecks. Require a recommendation, the strongest counterargument, and a source for every market number; keep the actual call and its political and user context outside the model.

Make customer feedback a cadence, not a repository. A community example describes feedback scattered across three inboxes and a spreadsheet nobody revisits. A more mature model spans Support, Success, and Professional Services, has Support own the program, meets stakeholders monthly, uses a reporting layer, and delivers findings quarterly. Centralization alone is not enough: another team still calls source-file input “patchy.” Start with an owner, cross-functional intake, a review cadence, and a visible decision/reporting loop before buying another destination tool.

Case Studies & Lessons

AskOne shows what intent-first agent delivery looks like. Product Compass describes AskOne as a deliberately simple $5/month B2B2C alternative to Slido: accountless audience Q&A, moderation, live room updates, and an API/MCP server that can turn questions into FAQs. The build sequence is the lesson: document segments, value propositions, jobs to be done, use cases, permissions, non-goals, and constraints—but not a supposedly complete upfront specification; ask the agent to design only, review two materially different directions and trade-offs, choose one, then build static screens before connecting authentication and real data. After manual testing, a flat list of missing behaviors—including projector, QR-code, room-closing, and icon fixes—was resolved in one iteration; detailed tests existed, but browser automation had not been tested without test accounts.

High-stakes agents need narrow action boundaries. Copay Compass finds assistance for an exact cancer prescription, sets alerts for reopened funds, and prepares application and appeal materials; the user submits them, while the bot does not send forms, make calls, collect sensitive IDs, decide eligibility, or provide medical or financial advice. The reusable pattern is to automate preparation and routing while retaining consequential decisions and submissions with people.

Career Corner

Build an AI PM track through evidence, not the title. Aakash Gupta reports that AI PM grew from 2% of the PM market in February 2024 to 46%, with median pay of $199K versus $151K for regular PMs; treat those as his snapshot, not an independently established market census. His pathway is concrete: learn foundations, prototyping, evaluations, context engineering, and autoresearch; choose among feature, agent, infrastructure, or model work; build a live product with real users if possible; ship AI in a job or get paying customers; then make the work visible and prepare for eight interview formats.

Tools & Resources

Circleback lowered the barrier to meeting-data capture. Its team announced a free plan with unlimited meetings, imports from other meeting apps, and an API for all Circleback data. For PMs, the import and API are the most relevant additions to test against existing feedback and research workflows—subject to normal data-governance checks.

Agent-Built Products Shift PM Leverage to Intent and Evaluation
The Product Compass
  • AI-assisted product engineering is becoming a relevant PM capability. The article argues that product and engineering roles are merging and that PMs should develop a basic understanding of engineering with AI, without necessarily learning to code. The author reports that an AI-built project reached 105K+ installs and 37K MAU, with 5,094 tests and roughly 177K lines of code; they say they reviewed none of the code and that AI produced the system in about three months.

  • Use discovery-derived intent as the agent’s operating context, not as a complete upfront specification. Document market segments, value propositions, jobs to be done, use cases, permissions, non-goals, and constraints in the project context; then refine the product through interaction and user feedback because the full specification is not knowable in advance. A practical sequence is: ask the agent to design only; request two materially different directions with their optimization and trade-off; have the PM choose one; build the chosen design system and screens; create a clickable static version with realistic content and empty states; then connect authentication and real data behind those same screens.

  • Manage agents through intent and behavior-based feedback. After manually testing the product, provide a flat list of missing behaviors—such as what a button should do or what access should change—rather than prescribing implementation details; the author reports that this approach resolved the listed issues in one iteration. The template’s instructions also led Claude to produce detailed unit and integration tests without a separate request, while browser automation remained untested until test accounts were supplied.

  • Lean SaaS case study and monetization sequencing: AskOne is a B2B2C live-Q&A product positioned as a $5/month alternative to Slido, with accountless audience participation, moderation, live room updates, and API/MCP support for turning questions into FAQs. The deliberately simple product is used to exercise multi-tenancy, Google authentication, subscriptions, roles and permissions, CI/CD, and security. For a separate remote client, the author reports $1,920 ARR and describes the sequence as proving value and word-of-mouth first, then focusing on monetization after launching managed hosting.

Product Engineering for PMs, Part 1: Build a SaaS App Without Coding
  • Product strategy—meet customers and technology where they are. Cursor was described as prioritizing a great product over an AGI/ASI narrative. Its practical approach was to meet customers where they are and technology where it is, then work toward autonomy; coding is an unusually tractable starting domain because it is verifiable and well documented.
  • Enterprise AI architecture—specialized intelligence plus model routing. When proprietary enterprise data is core IP and intelligence is a critical business input, use a capable open model and apply reinforcement learning plus supervised fine-tuning on company data so the company controls its intelligence, capabilities, and cost. Pair that model with one or two frontier models behind a transparent router, using frontier models for planning and lower-cost models for execution. The product must support data isolation, seamless routing, and continuous base-model upgrades; the discussion emphasizes that this is difficult to implement and that long-term low-cost delivery favors vertical integration.
  • Agentic workflow rollout—assist, recommend, execute. Start with narrow knowledge-enhancing tools such as podcast, Substack, and X summarizers or sentiment tracking, then progress to bots that inspect work, recommend automations, and execute them after user approval. In the reported examples, AI tools took 7–12 seconds to build instead of hours and were judged better, while internal token consumption rose 100x from March through August; the use was characterized as productive rather than wasteful.
Why AI Demand Is Outrunning Compute Supply
Lenny Rachitsky
  • Build slightly ahead of rapidly changing AI capabilities: Tara Seshan’s heuristic is to build for where models will be in two to three months—building for current capabilities risks being outdated at launch, while planning a year ahead is too speculative.
  • Favor empirical validation over long-range theorizing: In fast-moving markets, identify the sharpest hypothesis and get something real users can try as quickly as possible rather than relying on lengthy strategy documents or 12-month roadmaps.
  • Match execution speed with hypothesis quality: AI enables much faster iteration but can also send teams further in the wrong direction before they notice; clear hypotheses are needed to prevent faster compounding of errors.
  • Make intense product usage part of product culture: OpenAI’s internal questions include whether the team is being ambitious enough, moving fast enough, and “mainlining” the product—using it all day, every day. The discipline of user obsession, tight iteration loops, and rapid adjustment to market reaction was credited with the shift toward Codex.
  • PMs should raise the ambition ceiling: When teams propose a scope or timeline, PMs can challenge them with questions such as whether the work could be 10x bigger or delivered faster.
  • Preserve human thinking while using AI for leverage: Do not automate writing used to think through a problem; start and finish those documents yourself, using AI in the middle for research, data, and pushback. Automate reporting instead, share drafts around 70% complete, and prefer prototypes and A/B results over long documents as communication artifacts.
  • Retain clear outcome ownership in flexible teams: Even when engineers, PMs, and designers have few role boundaries and everyone can contribute, one DRI must remain accountable for the outcome.
My biggest takeaways from [@tarstarr](https://x.com/tarstarr), [@OpenAI](https://x.com/OpenAI)'s ChatGPT Work product lead: 1. The future…
Lenny Rachitsky
  • OpenAI’s internal “mainlining it?” meme turns deep product usage into a product-quality test: are team members using Codex all day, depending on it, and applying their own taste to whether it is something people actually want? Tara Seshan said the recent shift toward Codex came from people noticing it, not from an internal process change.
  • For products built on frontier models, Seshan’s timing rule is to build for where the models will be in 2–3 months; building for today fails, while building a year ahead also fails.
  • The discussion frames the modern PM role as elevating everyone’s ambition, while future knowledge work shifts from “rowing” to “steering.”
"Are you mainlining it yet?" This is one of the key internal memes at [@OpenAI](https://x.com/OpenAI), and a big part of the reason there…
Hiten Shah
  • Copay Compass tackles cancer-drug affordability by finding assistance for the exact prescription, alerting users when closed funds reopen, and preparing applications, appeals, and a one-page handoff sheet for a financial navigator.
  • The product draws clear high-stakes boundaries: users submit materials themselves; it does not fill or send forms, make calls, collect Social Security, member, or Medicare numbers, or determine eligibility. It also disclaims medical, financial, and emergency use and directs people considering stopping treatment to their oncology team and social worker.
I made "Copay Compass" with Grok [@bot](https://x.com/bot) for people drowning in the cost of a cancer drug. -> It finds copay help that …
Product Management - The place for all things product
  • A PM reports using Claude and ChatGPT for PRD drafts, interview-note summaries, and prioritization sanity checks, but says the tools praised an unvalidated feature idea, returned pros and cons without choosing, and pushed every RICE item toward medium-high. They still find them useful for first drafts, research synthesis, and late-night spec rubber-ducking.
  • For PM decision quality, treat AI output as a draft requiring verification and context: the author received fabricated competitor pricing stated as fact, must repeatedly re-explain the product and user context, and says AI misses stakeholder politics that can derail a roadmap.
  • A commenter recommends prompting for stronger challenge by assigning a CPO role and explicitly requesting opinions and pushback rather than merely asking the model to “think like a PM”; they report better results with that framing. Leadership should not equate AI-assisted drafting with proportionally shorter timelines, because the author says alignment, technical constraints, and user validation remain the bottlenecks.
Real talk: the most annoying thing about AI as a PM tool isn't the mistakes, it's the agreeableness I see a lot of parallels in what you said and I made a few videos about them - mostly in line with what u said - but I have seen it to pe…
Lenny Rachitsky
  • Build for the near-term model frontier: Tara Seshan, OpenAI’s ChatGPT Work product lead, says teams building on frontier models should target where models are expected to be in 2–3 months—building for current capabilities or for a one-year-out future will fail. Use this horizon when setting product scope and roadmap assumptions for rapidly evolving models.
  • Use intense internal product dependence as a demand test: OpenAI’s internal “mainlining it” meme asks whether employees use the product all day, depend on it, and bring their own judgment to whether people truly want it. Seshan attributed the shift toward Codex to employees noticing its value through use, rather than to a deliberate internal change.
"You fail if you build for where the models are now. You fail if you build for where you think the models will be in a year. The only way… "Are you mainlining it yet?" This is one of the key internal memes at [@OpenAI](https://x.com/OpenAI), and a big part of the reason there…
Lenny Rachitsky
  • Nanyu is joining OpenAI to work on Codex and ChatGPT after four years at Linear, where they said they were proud of what the team built and would carry forward what they learned about the craft of software.
Some personal news—I’m joining OpenAI to work on Codex and ChatGPT. I’m grateful to the Linear team for an incredible 4 years and proud o…
Hiten Shah

Early products can serve as vehicles for company formation and learning: Hiten Shah writes that “sometimes the first product is just the team learning how to become the company.” Josh Elman highlights the pivot pattern with Fates Forever → Discord, Cicada → Musical.ly, Fanbase → Nextdoor, and Odeo → Twitter.

Sometimes the first product is just the team learning how to become the company. Great thread from [@joshelman](https://x.com/joshelman).… Been thinking about “pivots” today. Many of the best companies I have been lucky enough to get involved with are pivots. Fates Forever (g…
Product Management
  • Non-STEM degrees do not automatically exclude candidates from product management: respondents describe the field as diverse, while noting that technical requirements vary by company and team. One commenter says degrees may function as filters at some large-name employers, but startup projects, metrics, and user stories are practical signals to keep building.
  • Suggested upskilling tactics include learning basic technical concepts through courses such as Maven or Udemy, taking an Agile 101 course, and building small AI or “vibe-coded” projects that can demonstrate skills on a résumé.
  • A reported EdTech pivot was enabled by deep user empathy and industry expertise, along with a supportive hiring manager and favorable timing; the person then continued upskilling while changing roles and earning promotions. Respondents also suggest that degree credentials matter less after the first one or two jobs and that PM work can become more business- and strategy-focused with seniority.
Yes, I studied 20th Century US History in Undergrad, then a Masters in Security Studies focusing on East Asian military development. Whet… plenty of pm folks come from non stem, the degree is mostly a filter at some big name places, not a hard rule your startup stuff is exact… Undergrad was in history and writing, then did my MBA and moved into tech. You can absolutely be great at it without a technical degree, … That's awesome - anything is good. AI is the obvious area to lean into right now - learn how to build skills and routines in Claude, "vib… I wish I could tell you about some secret sauce, but I really just got really lucky. I joined an EdTech company, to work on a product tha… It could be useful. But ultimately, in most instances, no one cares about your degree after your first job or two. I’m non stem background and doing just great- it becomes more business oriented and strategy focused as you move up the ladder
andrew chen

Andrew Chen frames AI as overturning startup-building assumptions: hardware remains the main area AI cannot build, solo founders gain dramatically more leverage, anyone can build, AI-infrastructure PhDs can command premium valuations, and SaaS faces a potential “SaaSpocalypse.”

Before AI: - hardware is hard - teams > solo founders - technical founders only - PhDs build tech in search of a problem - hot SaaS start…
The community for ventures designed to scale rapidly | Read our rules before posting ❤️
  • Revisit shelved products when external constraints change. A payments/invoicing product for Indian freelancers was about 90% complete but was paused because multi-currency business accounts and compliance approvals were difficult; the founder later saw reported 3–5-day onboarding, IBANs, and SWIFT access from newer regulated providers, making the unit economics viable again.
  • Evaluate the competitive response around moat and distribution, not just build quality. A friend launched a less-polished prototype for the same niche, while the original concept was positioned as a SaaS product rather than a wrapper; the situation created a choice between collaborating, competing, or abandoning the effort. Community feedback framed a two-week clone as evidence of limited moat and argued that customer acquisition, strategic execution, and the ability to pivot matter more than building first; collaboration is most defensible when the other party contributes capabilities that are genuinely missing.
“I will not promote” A friend I told about my SaaS idea two years ago just showed me he built it. How would you handle this? Honestly, just building it is literally nothing. From the sounds of your description there are hundreds of exactly this already (and prob… Dont listen to people here they are penut butter and jelly. On this situation i would say if you really your version of the idea then kee… Work together and split the proceeds. If your idea can be cloned in two weeks then you have no moat and your only chance of long term suc…
Hiten Shah
  • Hiten Shah shared a bot designed to replace repetitive checking when little changes: users give it something to monitor, it identifies what would change the answer, watches for that condition, and messages them when it matters. Users can also ask it what to watch.
I made a [@bot](https://x.com/bot) for something I do way too much. Checking the same thing over and over when almost nothing has changed…
Hiten Shah
  • Closed-loop software as an emerging product concept: Hiten Shah argues that software capable of observing itself, changing itself, and measuring the result makes “analytics” just one step in a broader loop; he speculates that the loop itself—not analytics—could become the more meaningful category.
Once software can observe itself, change itself, and measure the result, "analytics" starts to sound like the name of one step in the loo…
Hiten Shah

AI product ideas may require multiple build–fail–rebuild cycles: Hiten Shah’s working theory is that models and understanding of how to build with them keep improving, so a great AI-first idea in an unsaturated market may be rebuilt until it reaches scale—even after earlier versions fail.

[@wongmjane](https://x.com/wongmjane) Models keep getting better and our understanding of how to build with them improves too. And if som… We’re going to see the same AI idea fail, get rebuilt, fail again, and then suddenly work. [https://x.com/hnshah/status/20942123972382148…
Hiten Shah
  • Amplitude’s Wave agent has been running on Amplitude’s own product for several months; it identifies problems, ships fixes, and measures whether those fixes worked.
  • An emerging “self-improving” product loop combines behavioral observation, problem identification, change implementation, outcome measurement, learning, and repetition. Coding agents make software changes cheaper, while analytics provides context for deciding what should change.
(1/6) i'm the pm on Wave at [@Amplitude](https://x.com/Amplitude), our agent for self-improving products. for the past few months we've h… The dashboard is starting to look like an intermediate step. PostHog calls it self-driving. Amplitude calls it self-improving. Both are c…
Product Management
  • Customer feedback is often fragmented: one participant described it as “scattered across three inboxes and one spreadsheet nobody opens after week one,” while the discussion explicitly questions whether teams revisit feedback or rely on memory when deciding what to build next.
  • A concrete Voice of the Customer operating model spans Customer Support, Customer Success, and Professional Services; Support owns the program, stakeholders meet monthly, a reporting layer supports it, and findings are delivered quarterly.
  • Centralization tools do not automatically solve feedback ingestion: one team is “trying” to store feedback centrally in Productboard, while another uses Obsidian plus Claude Code and reports expanding cross-team access—but also calls source-file input “patchy” and a real gap.
Everywhere and nowhere, scattered across three inboxes and one spreadsheet nobody opens after week one Haha common story! Last time you or your team had to work out what to build next, did you get back into any of it? Or go on memory? In a robust Voice of the Customer program that spans Customer Support, Customer Success, and Professional Services. It's owned by Support… Trying to centrally store it in Productboard. Key word, trying. Nice what's your setup? We're Obsidian + Claude code. Do you organize it in any particular way or let your AI handle the connections? Same, I connect it to everything I can and it makes the wikis on its own Cross-product team were slowly broadening access to other teams. Honestly source file input is patchy it's a real gap
The community for ventures designed to scale rapidly | Read our rules before posting ❤️
  • For startup product or functional roles, one hiring view is to prioritize talent and autonomy over exact functional background when role-specific knowledge is trainable: an inexperienced high performer may be easier to train than a mediocre experienced hire.
  • Replace interview-only judgment with a paid one-week project based on an intentionally incomplete brief; evaluate who asks the right questions, makes sensible trade-offs, and reaches a useful result without waiting for instructions—the commenter argues this reveals more than motivation or a CV.
  • Balance autonomy against retention: a self-directed hire may be a flight risk, while a loyal hire who needs constant direction can turn the founder into a manager; assess the oversight burden and consider negotiating a longer commitment with the autonomous candidate rather than treating loyalty as automatic safety.
Depends on the role. I agree with you when it comes to devs and roles which require many years of hands on experience. It’s different tho… i'd optimize for how much direction each person needs. in a tiny startup, someone who stays for years but turns every problem into more w… neither answer is really about intelligence or motivation, its about how much oversight you can afford to give right now. candidate B nee…
The community for ventures designed to scale rapidly | Read our rules before posting ❤️
  • A two-person SaaS’s split between frontend/finance and backend/product failed during a customer-driven payments need: the finance owner took six weeks to establish a multi-currency account, an ensuing 12-day payout hold left the company unable to cover hosting, and the backend/product owner compared Stripe, Paddle, and Lemon Squeezy before launching Unlimit with SWIFT/SEPA support in about four days.
  • For small product teams, define ownership around decisions, deadlines, and a fallback owner—not only functional domains—so missed deadlines automatically transfer execution authority; address disputes directly, transparently, collaboratively, and without accusations.
i will not promote, Cofounder wants equal say wo touching our banking setup in forever rewrite the role split around decisions, deadlines, and a fallback owner. finance can stay his domain, but missing the agreed deadline sh… I'd just speak to him about your concerns. No reason to hold back, be 100% transparent about how you feel. Healthiest approach. Just do i… I have dealt with this multiple times in multiple companies. The bottom line is that you can either address it head-on right now, or you …
Product Management
  • For an incoming fintech tech PM, prioritize understanding the consumers of APIs, platforms, and services; their economics; non-functional requirements and how to define them; related trade-offs and opportunity costs; and how to explain technical concepts differently to non-technical buyers and developer users.
  • A practical learning tactic is to take a basic idea, build a small project with Claude Code, and ask the AI to explain the decisions and underlying implementation as you go.
As a tech PM, understand the consumers of your products - APIs, platforms, services - and the economics behind these. Understand non func… Claude code. Take a basic idea, set up a project, build it and ask it along the way to explain what's under the hood. This you will learn more by doing and actually ask an AI why it made those decisions