We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
How the Codex team ran with two PMs
In Product Compass, Rohan Varma, Cursor's first PM and until recently Product Lead for Codex at OpenAI, describes that team: about two PMs, two designers and 30–40 engineers across roughly 15–20 product areas . They wrote few traditional specs. Sometimes a spec was about ten bullets. Varma set strategic guardrails and engineers made the micro-decisions. Most projects had one or two engineers, because "human-to-human collaboration becomes a big bottleneck" . A brief should set out the problem, outcome, constraints and scope. It should also say what the builder can decide alone and what evidence would change direction. Planning ran four to eight weeks at a time, not six months .
The guide also offers two habits for AI PMs:
- Start with the bottleneck. In his support example, agents struggle to find the right policy, not to write replies. That points to a copilot that drafts cited answers for agents to approve, not a fully autonomous chatbot .
- Diagnose before you re-prompt. A wrong answer can come from retrieval, from missing context or from generation. Each needs a different fix .
The constraint is now judgment
Aakash Gupta opens with a line from Madhu Guru, who led product for Gemini: "I've never felt this behind as a product person." Gupta's argument: engineering capacity used to be the excuse, but a PM can now spin up execution in an afternoon. The hard part is separating signal from noise, deciding what to build and scoping ruthlessly. That team constraint was "a useful governor that prevented bad ideas from shipping" . In a separate note, Gupta says he tested four graph designs against single-shot prompts on 40 PM tasks. The biggest margins came on tasks like PRDs, eval suites and roadmap prioritization. Every winning design worked by checking facts, not by improving judgment, so his advice is to "wire a graph when the failure mode is a wrong fact" .
Practitioners split on whether AI has improved the job
An r/ProductManagement thread asked whether PMs still enjoy their jobs. The useful answers fit Gupta's argument:
- The upside. A PM without analyst support says data analysis went from three days to an hour . Another spotted segment-level conversion patterns in a BI chart and had Claude code a four-arm onboarding test that was live by ten. The segment-specific variant was ahead by that afternoon .
- The cost. A PM managing several teams says the cognitive load has moved from writing docs to reviewing AI output. Some developers now skip reading the spec, and management wants "more and faster" .
- The exposure. One commenter argues that documentation was never the job. Now that it's 10x faster, PMs who avoid decisions are "exposed" . Another says experience depends on the setup: empowered PMs may be having their best time, while those in feature factories have it worst .
What "AI-native" can mean
Run the Business says most heads of product have been handed an "AI-native" mandate in 2026. It can mean one of three things, or a mix: AI in the product surface, an operating model built around an AI development lifecycle, or a strategy focused on how to win rather than what to ship. Which to pursue "depends" on the company .
Designing for agents
Ravi Mehta looks at Descript, which wanted outside AI agents to edit videos without ever opening its interface. That raised a choice: build a separate machine interface, or make one product that serves both humans and agents . He frames designing for agents as designing "for humans — one step removed" . In a related post, Hiten Shah argues AI video needs evals against a quality bar for story, pacing and brand. His team tested AI editors on real footage and plans to share what worked on Friday .
Briefly
- Validation: In an r/startups thread on vetting ideas, one founder's playbook: ask about the last time the problem happened, not "would you use this?". Do the job by hand for five to ten people. Treat a deposit as worth more than a hundred signups, and competitors as evidence of demand .
- Sharing work: In a podcast with Petra Wille, Teresa Torres urges PMs to share specific work they are personally doing, even null results such as "we tried 20 things… no efficiency gains" .
- Daffy separates funding charitable giving from choosing recipients: users can set money aside when convenient and recommend charities later, addressing the discomfort of making both decisions during a solicitation. The product applies behavioral-finance techniques used in retirement plans and supports recurring donations to make giving an intentional financial goal; Nash says members give more consistently and often give more to organizations they already support.
- Daffy combines charity information with peer context: members can view basic publicly filed information and optionally share notes about why they give; Nash says these notes can help people discover organizations and build trust.
- Daffy offers auto-rebalanced ETF portfolios, more than 600 ETFs, eligible individual stocks, and native crypto, with investment choices that can reflect whether funds are intended for near-term or long-term giving.
- At SurveyMonkey, Verna moved into growth product as a PM with engineers after reaching a ceiling in analytics, then expanded into growth marketing and analytics. She initially questioned whether growth belonged in a dedicated team, arguing that growth is the whole company’s responsibility—a useful organizational tension for product leaders to weigh.
- At Lovable, she spends a couple of hours each week exploring the product because its team keeps adding features and she still has more to learn about its capabilities; she sees hands-on exploration as a way to find how technology can increase impact at work or in something built independently.
- For her own advisory business, Verna used ungated expertise on social media and in a newsletter as a freemium top-of-funnel to attract potential clients. She says frameworks are easy to share, while implementing them inside a company is the harder work where her services add value.
- Start with the user’s bottleneck, not an AI solution: in the support example, agents lost time finding correct policies, so the proposed copilot retrieves policies, drafts cited answers, and lets agents verify, edit, and approve them. The suggested goal—halving policy-answering time without compromising accuracy—is illustrative and should be validated with users.
- Diagnose AI failures before changing prompts: an incorrect answer may stem from retrieval, missing context, or generation, and each requires a different fix. Evaluate answerable, unanswerable, conflicting-policy, and high-risk cases; inspect failure categories, regressions, and fresh examples because a higher aggregate pass rate alone does not establish readiness.
- Bound agent autonomy: use permissions, spending limits, and human approval for consequential actions; predictable tasks may be faster, cheaper, and more reliable as fixed workflows, with architecture revisited as evidence changes. At rollout, measure the whole task—including latency, tool failures, human review, and cost per successful resolution—not just model-response cost.
- Rohan described Codex as having roughly two PMs, two designers, and 30–40 engineers across approximately 15–20 product areas; the team used few traditional specs, relied on strategic guardrails for autonomous decisions, and planned in four-to-eight-week increments rather than six months.
- Verna applies a growth-funnel model to her advisory business: ungated social and newsletter content serve as freemium acquisition for relevant prospects, while advising and interim engagements act as trials that can lead to full-time work. She argues that publishing frameworks is easy; the hard, human-intensive work is operationalizing them within a company's details.
- For expertise-based distribution, she recommends starting on social for faster feedback, sharing specific examples and patterns rather than generic advice, and iterating until finding an audience and voice before adding a newsletter; she launched hers about two years into her social journey.
- She recommends setting aside regular time to explore AI tools and product capabilities: she spends a couple of hours weekly experimenting, including with Lovable, because its product team continues to add features.
For product managers advocating a proposal, a potentially useful reframe is to treat promotion as sharing the work’s ideas rather than elevating oneself: in a book-launch discussion, the guest distinguishes the book’s ideas from himself, and the host frames sharing useful work as service to others. The suggested practice is to question the limiting belief, consider whether its opposite could be true, then write down and revisit the alternative when hesitation returns.
Teresa Torres recommends sharing specific work you are personally doing rather than generic advice: describe the practice and messy learning, even if the story is unfinished, for someone one step behind you. A result without efficiency gains can still be useful to share—for example, trying 20 things that changed the work but did not improve efficiency. She suggests a low-stakes five-day habit: post the day's most interesting event and say what you faced, did, and hope happens; most posts get ignored, so the risk is low.
For product execution, prioritize controllable actions and rehearse obstacles rather than fixating on desired outcomes: the speaker contrasts an uncontrollable bestseller result with a controllable hour of daily writing, and recommends mentally rehearsing likely difficulties and responses (“mental contrasting”). He reports that students who visualized getting an A studied less and earned worse grades than students who did not, suggesting that outcome visualization alone can undermine action.
- AI is making market and competitor research, vendor and feature POCs, legal work, documentation, and handoffs easier; one PM without analyst support says a data-analysis task shrank from three days to one hour. A PM managing multiple development teams reports the tradeoff: more AI-generated material to review, faster output expectations, and developers sometimes missing specs or acceptance criteria under the increased pace; they are trying not to measure success solely by speed.
- A recurring concern is that AI makes process-heavy PM work less central: as documentation speeds up, PM value shifts toward making timely product decisions and choosing priorities, rather than mainly producing decks, requirements, and process artifacts. One commenter says the experience depends on the role: empowered PMs with real product goals may thrive, while micromanaged feature-factory roles may be especially difficult.
- One PM used BI data suggesting different user segments converted better with different features to create a control and three treatment variants for onboarding using Claude; the segment-specific treatment was winning that afternoon, with productionization and further paywall testing planned if the signal held.
Lenny’s Product Pass added OpenRouter (one API and bill for hundreds of AI models), PloyAI (AI growth-marketing agents), Sentry (alerts when shipped software breaks), TextExpander, and LangSmith (tools to inspect and improve AI agents)—options relevant to AI development, growth, and product reliability workflows. The pass is advertised as a free year of access to 35 products worth $41,428 for Insider subscribers at $400/year, or 22 products worth $37,062 for Annual subscribers at $200/year; codes are first-come, first-served, and popular products may sell out.
Beam says it reached #1 Product of the Day on Product Hunt last month . The team used Beam screenshots and recordings to test AI video tools and planned to show the first renders and what needed fixing .
- OpenRouter and LangSmith were added to Lenny’s Product Pass. OpenRouter provides one API and bill for hundreds of AI models, with model selection or routing, quick model swaps, per-key spend limits, and automatic failover—useful for managing model choice, cost, and reliability in AI products.
- LangSmith supports building, testing, deploying, and improving agents through observability and evaluations; the newsletter says product managers are increasingly using it to manage agent quality and performance. Examples include Rippling using traces and layered evaluations to catch regressions, and Clay reconciling 99.5% of inference spend across providers.
Lenny’s Newsletter Product Pass added OpenRouter for one API and bill across hundreds of AI models, PloyAI for AI-driven growth marketing, Sentry for detecting product failures, TextExpander for reusable prompts and phrases, and LangSmith for observing and improving AI agents. The bundle lists 35 products worth $41,428 for Insider subscribers at $400/year, or 22 products worth $37,062 for Annual subscribers at $200/year; codes are limited and popular products may sell out.
AI video editors need evaluations against a quality bar for story, pacing, and brand so content factories improve with each edit, Hiten Shah argues. His team tested AI editors on real footage and reviewed first cuts for both failures and successes; he said it would share what worked Friday.
Warp Sciences says it is building robots to perform real scientific experiments and accelerate discovery, and is hiring full-time and intern roles in robotics hardware, robot learning, and robot operations .
Jensen Huang said that after his departure from NVIDIA, he would return as CEO if the board decided it was in the best interests of the company and its stakeholders; he also said he would cooperate with the board’s independent investigation into serious personal mistakes that created problems for the board.
a16z Speedrun opened its Insiders program to executives and operators, building on 150+ C-suite founding members; it aims to give company buyers early access to startups and provide founders with feedback from people who may buy their products. Members can watch weekly startup demos, get warm introductions to founders, and join early pilots as design partners who help shape products; participation is free.
For product launches and GTM, build an audience between launches with useful, recurring content from people the market wants to hear from, rather than relying on a campaign when an announcement is imminent; that way, people can stay connected after a launch and be more familiar with the company by the next one. The article argues this matters as search and AI-generated answers can reduce visits to company websites, while feeds increasingly help people discover individual voices.
Descript’s Underlord AI editor can turn an hour-long webinar into five short videos; the company also wanted other AI agents to edit videos without opening its interface, raising a product-design choice between building a separate machine interface and making the existing product work for both agents and humans. The post frames agent UX as designing for humans one step removed.
- For a new design studio, validate a focused offer before buying a broad equipment stack: a commenter warns that the bicycle market is saturated and suggests testing work for 3–6 months through a local fab lab or rented facility, then adding tools; another recommends starting with one strong manufacturing process and expanding as client needs become clear.
- Make equipment decisions as build-versus-buy tradeoffs: one studio operator says a desktop CNC is hard to justify over outsourced services unless machining volume or faster turnaround is valuable, and advises accounting for skill requirements, opportunity cost, and workflow peripherals—not just machine price.
- Funding conditions can change the sequencing: this requester’s grant requires a detailed wishlist and business plan, excludes consumables, and favors high-end, technology-oriented machines, limiting the usual option to buy equipment only after client demand is demonstrated.
An “AI-native” product organization can mean three distinct things: AI-infused product features, product rituals and ceremonies aligned to an AI SDLC, or business bets focused on how to win rather than what to ship. The right approach depends on the company and its circumstances, so the mandate should not automatically be treated as a request to add AI to the product surface.
How to Become an AI PM in 2027: The Complete Guide From Cursor's First PM
Hey, Paweł here.
Our guest today is Rohan Varma (opens in new tab), the first Product Manager at Cursor (acquired by SpaceX this year for \$60 billion). Until recently, Product Lead, Codex at OpenAI.
Rohan and I both teach in Product Faculty’s AI Builder Fellowship (opens in new tab) (see our Black Friday offer).
Cursor went from \$100M ARR to over \$1 billion (opens in new tab) in under a year with just one product manager.
I joined Cursor as an engineer and then shifted into being their first product manager. And the funny part? It was my first-ever product management role.
Then I joined OpenAI to work on Codex. As adoption took off, we had just two product managers.
Again, I was one of them.
So I’ve been a product manager at two pretty interesting companies building in AI and I’ve seen firsthand how different this job has become.
The bad news: you can’t follow the exact career path of an AI PM who joined OpenAI years ago. The industry has changed.
The good news: that doesn’t matter. Today I’m breaking down the exact strategy you can use to land an AI PM role in 2027, even with no prior AI PM experience.
AI product management is one of the most in-demand and highest-paying roles in tech.
Businesses are moving from experimenting with AI to scaling production agentic systems, and this is just getting started.
In this post:
The 8-Skill AI Product Manager Stack
Three AI PM Portfolio Projects That Prove You’re Ready
How to Land AI PM Interviews: Roles, Resume, and Outreach
How to Ace AI PM Interviews: Three Practice Questions
Your 90-Day Plan to Become an AI PM
Let’s dive in.
1. The 8-Skill AI Product Manager Stack
There’s a lot of confusion about this role, and a lack of relevant resources. It shows up in aspiring AI PMs who are:
Confusing AI PMs (opens in new tab) with PMs who use AI.
Learning AI product management through random courses that teach outdated PM practices.
Trying to reverse-engineer the career trajectories of AI PMs at frontier labs. This is the worst of the three.
The in-demand AI PM role is different from traditional product management. Making any of these mistakes will only put you further behind.
And the opening is real: very few of the senior product managers I talk to have hands-on experience shipping a working AI agent or model-based product.
It gets better. You don’t need:
An engineering background.
To train a foundation model.
All you need is to understand how AI products work, where they fail, and how to make decisions about quality, cost, and reliability.
But it’s easier said than done, especially when nearly all the AI PMs I know came from traditional PM backgrounds.
Let’s address one of the top misconceptions among AI PM aspirants:
Yes, your job as an AI PM requires you to build, ship, and evaluate AI products. But all of this only matters when:
You build an AI product that should be built.
You don’t build an AI product to solve a problem that could easily be solved manually.
Problems come first. AI solutions later.
This is product decision-making in a nutshell, and this is what AI PMs get hired for.
You’re not paid to use AI. You’re paid to solve a problem that can be effectively and efficiently solved with an AI product that offers a strategic business advantage.

The eight skills, in the order this guide teaches them
To understand all 8 skills in depth, we first need a problem that we’ll solve using each skill. This will also show you why each skill is a non-negotiable part of an AI PM’s toolkit.
The problem: A customer support team spends hours answering questions using company documentation.
The goal: An AI product that helps the team respond faster.
1.1 Product sense and AI UX
One morning at OpenAI I showed up to work, and one of the engineers on the Codex team said, “Look, I built the browser in this. What do you think?” He’d built it because frontend iteration was really annoying.
I said, “That’s awesome. We should definitely figure out how to ship this.”
His build became our starting point: an actual branch in the product that we could develop off of. We could build things almost instantly, so a lot of our time went into looking at solutions and deciding whether we actually wanted to ship them.
It’s literally not a question of what is possible. Everything is possible. It’s just: what should we do? Having something working just lets that conversation happen with evidence on the screen.
One of the biggest mistakes aspiring AI product managers make is skipping that question and jumping straight into solutions.
They hear about a customer problem and immediately start thinking about which AI model, agent, or framework to use.
But great AI PMs start with a different question:
What problem are we actually trying to solve, and what would success look like?
This requires two critical skills:
Product Sense: Understanding your users, identifying their real problems, and deciding what is worth building.
AI UX (User Experience): Designing how people interact with AI, verify its outputs, handle mistakes, and stay in control.
Let’s look at an example.
The problem: Your customer support agents spend four hours every day answering questions about refunds, shipping, and product policies.
An inexperienced AI PM might immediately propose an AI chatbot to automate customer responses.
A strong AI PM would first talk to support agents and discover that their biggest bottleneck isn’t writing responses. It’s finding the correct policies scattered across internal documents.
That insight changes the solution.
Instead of a fully autonomous chatbot, you build an AI support copilot that:
Finds the relevant policy in company documentation.
Drafts a response with citations.
Lets agents verify, edit, and approve the answer before sending it.
But building the product isn’t enough. You also need to define what success looks like.
Here are four measurable targets you might set:
Time spent answering policy questions: 4 hours/day → 2 hours/day
Average resolution time: 8 minutes → 4 minutes
Responses with material policy errors: 5% → Below 1%
Responses sent without substantial edits: To be measured → 80%
Illustrative targets; actual baselines and targets should be validated with the support team.
Your goal: Reduce the time agents spend answering policy questions by 50% without compromising accuracy.
📌 Don’t start with what AI can build. Start with the customer problem, design the right experience, and define measurable success. Only then should you decide which AI model can deliver it.
1.2 Model intuition
Model intuition is your understanding of where a model works well and where it tends to fail.
One habit that helps me build it: I keep my own Codex setup pretty light on custom skills (the reusable instruction files Codex can load). We built a ton of interesting plugins internally at OpenAI, but I find it really useful to see where Codex falls over when it doesn’t have anything specific.
Test your models and watch how they behave on five things:
Tokens: The units the model processes: words, parts of words, and punctuation. Token usage drives cost and how much information the model can handle.
Context window: How much information the model can process in one request: the user’s query, instructions, retrieved documents, and sometimes the conversation history. A large context window helps with information-heavy tasks, but it doesn’t mean the model will use all of that information correctly.
Tool calling: The model’s ability to call external actions or sources, such as searching a knowledge base or looking up an order. The application runs the tool and returns the result to the model.
Latency: How long the system takes to respond. A more accurate model can take longer to answer.
Inference cost: What it costs to run the model. It varies with the model, the number of tokens processed, and how many times the model is called.
More terms: the AI Product Manager Glossary (opens in new tab).
For our support copilot, say you have three models to test with 50 customer questions:
Model 1 is quick and cheap to run, but it frequently misses key information.
Model 2 is more accurate, but slower and more expensive.
Model 3 is good at simple questions, but struggles when it has to combine information across multiple documents.
Now you have to decide:
Do you use the most accurate model for every question?
Do you use the cheaper model for simple questions?
Is the extra quality worth the cost and response time?
This is a product trade-off!
For your three-model comparison, keep the actual failures alongside the scores. Was a policy exception missed? Did the model combine two documents incorrectly? Did it use a tool when it should have asked a clarifying question? Those examples will tell you more about which model fits the task than a single overall accuracy number.
You can’t make that decision well without understanding prompting, context, and retrieval.
1.3 Prompting, context engineering, and RAG
A great AI product manager understands where an AI answer comes from, why it might be wrong, and how to fix it.
Three skills matter here:
Prompting: Giving the model clear instructions about what to do, how to respond, and what constraints to follow.
Context engineering (opens in new tab): Ensuring the model has the right information, instructions, and tools available when completing a task.
Retrieval-Augmented Generation (RAG) (opens in new tab): Retrieving relevant information from external sources, such as company documents, and providing it to the model as context.
Let’s return to our support copilot.
A customer asks: “Can I return a product after 30 days?”
Your copilot searches the company’s documentation, retrieves the return policy, and generates an answer.
But there’s a problem.
The company allows returns within 30 days, yet the copilot confidently tells the customer they have 60 days.
An inexperienced AI PM might simply tell the engineering team to improve the prompt.
A strong AI PM investigates where the system failed.

Each cause needs a different fix
There are three possibilities:
Retrieval failure: Did the system find the wrong information? Perhaps it retrieved an outdated policy that allowed returns within 60 days. The fix might involve improving document indexing, retrieval ranking, or filtering outdated policies.
Context failure: Did the correct information reach the model? Perhaps the system retrieved the right document but excluded the paragraph specifying the 30-day limit. The fix might involve changing how documents are divided into chunks or how retrieved information is assembled into the model’s context.
Generation failure: Did the model misinterpret the correct information? Perhaps the model received the correct 30-day policy but still generated an incorrect answer. Now you might investigate the prompt, conflicting instructions, or the model’s ability to follow the policy.
The same incorrect answer can have three completely different root causes. And each requires a different solution.
That’s why learning RAG means understanding the entire information flow well enough to diagnose failures systematically.
When Codex doesn’t do something correctly, it’s easy to just say, “Great, I’ll do it myself.” I try to ask what was missing instead. Probably I had some context or information that Codex didn’t have.
Sometimes the missing piece is a document. Sometimes it’s access to a tool. Sometimes it’s a constraint I never explained because it felt obvious to me.
Then the job is figuring out how to give it that context, so it has it moving forward.
At OpenAI, I’d get added last minute to an entirely new project, and in 20 minutes I’d have all the context of what happened before I joined, because Codex could go into our tools: Slack, docs, Linear, email, and Drive.
The useful result links back to the evidence, preserves disagreements, and leaves unanswered questions visible. A confident summary that smooths over a conflict can create another problem.
For your copilot, keep a trace of the question, retrieved passages, context sent to the model, tool results, and final answer. Then you can follow the failure back to the point where information was lost or misused.
1.4 AI prototyping
In 2027, you don’t need to be an expert programmer to build AI product prototypes.
With AI coding tools like OpenAI Codex, Claude Code, Replit, and Cursor, you can turn an idea into a working prototype (opens in new tab) without writing every line of code yourself.
But you should understand what you’re building, how it works, and why you made certain product decisions.
Image generation really changed the way I work. Prototyping design ideas with AI often meant writing code, but I think image generation is much faster.
I took a screenshot of the Codex composer, where you select the project, and asked for four or five ideas to improve project selection. From just the screenshot, it did a pretty good job of staying within the design language.
That was kind of an AGI moment for me. We always think of image generation as something for, I don’t know, turning your hair blue. But the mocks are good enough that iterating is much quicker than generating five dummy React sites. Then I’d say, “Cool, now take the first one and make a prototype of it.”
The artifact should fit the question. An image can help you compare layouts. To learn whether someone can verify an AI-generated answer, you need a flow they can actually use.
Using an AI coding agent, you could build a simple prototype with three features:
A support agent enters a customer’s question.
The AI searches company documentation and drafts a response.
The agent reviews the answer and its sources before sending it.
That’s enough to put your idea in front of real users and learn whether it solves their problem.
You don’t need a polished interface or dozens of features. Start with the smallest version that demonstrates value.
But being able to build a prototype isn’t the same as understanding it.
You should be able to explain:
Which AI model you’re using and why.
How the system retrieves information.
What happens when the AI gives an incorrect answer.
What you would improve based on user feedback.
These are exactly the kinds of decisions you’ll need to defend in an AI PM interview.
The prototype also doesn’t have to become permanent software. Often it’s a disposable one-off. I’ll ask Codex, “Look at all my Slack messages, figure out the most important ones to respond to, and show them to me in a local website.” If it’s something I’d use repeatedly, I set up an automation to update it every hour or so.
That’s a useful way to approach your first project too: build enough to learn whether the workflow deserves more investment.
1.5 Agentic thinking: design agentic workflows and know when to build an AI agent
The general optimization function for me is: every time I do anything, I basically think, could Codex have done this?
In 2027, great AI PMs will know how to delegate work to agents (opens in new tab), design workflows around them, and build systems that get things done with minimal human intervention.
Think of it as becoming an agent maximalist: constantly looking for opportunities to automate work, while knowing when human judgment is necessary.
At OpenAI we used Slack very heavily, with a ton of channels where customers and our own team sent feedback. One thing I used Codex for was synthesizing that feedback, highlighting the important things, and uploading it into a Linear board I could review later.
I did it once in a Codex thread first, and spent a lot of time asking it how to put things into Linear correctly. Then I just asked Codex to set up an automation to run it every day and send me a Slack message every time it finished. I had five or six automations like that, each one a manual process I’d done once first.
The sequence matters. I get to inspect the workflow before making it recurring.
Back to the support copilot: instead of just answering questions, imagine it can now:
Read an incoming support ticket.
Identify and classify the customer’s issue.
Retrieve the customer’s order details.
Find the relevant company policy.
Draft a response and update the ticket.
You could design an agentic workflow that handles all five steps.
Should the agent also issue a \$500 refund automatically?
Not necessarily.
You need to think about permissions, spending limits, human approvals (opens in new tab), and what happens when the agent makes a mistake.
Here’s how a human handoff could work:
A customer requests a \$500 refund. Your agent checks the order, retrieves the refund policy, and determines that the request needs manager approval.
Instead of processing the refund, it:
Pauses the refund action.
Creates an approval request showing the order details, relevant policy, and reason for the recommendation.
Notifies a support manager to approve or reject it.
Processes the refund only after approval, then records the decision and updates the ticket.
If the manager rejects the request, the agent doesn’t issue the refund.
The agent does the research and preparation. The human makes the consequential decision.
And there’s another important question: Do you even need an autonomous agent?
If the steps are predictable, a simple workflow (opens in new tab) with a few AI model calls might be faster, cheaper, and more reliable.

Same product, two architectures: the fixed pipeline shipped first
Bugbot, Cursor’s AI code reviewer, is a real example of how that choice changes over time. At launch, it ran eight parallel review passes, then voting, validation, and deduplication.
Later, the team moved to an agentic design that let the model use tools and decide where to investigate. That’s where it saw its largest gains (opens in new tab). The new agent was initially too cautious, so the prompts had to encourage deeper investigation.
The architecture should follow what the task and the model can reliably support. Revisit that decision as the evidence changes.
The goal is to identify where autonomy creates genuine value and where it introduces unnecessary complexity.
📌 Become someone who instinctively thinks about what AI agents can do for you. But develop the judgment to decide what they should do, what they shouldn’t, and when humans must remain in control.
1.6 AI evals
A big difference between building an AI demo and shipping a reliable AI product is knowing how to evaluate its performance.
A demo might work perfectly five times in a row. But what happens when 1,000 real customers start using it?
That’s where AI evaluations (evals) (opens in new tab) come in.
Evals help you systematically measure how well your AI product performs specific tasks, identify where it fails, and determine whether your changes actually improve performance.

More bugs flagged, and a higher share of them fixed
For Bugbot, the team tracked resolution rate: whether the author had fixed a flagged issue by the time the PR merged. An AI made that call, spot-checked with the PR authors. Across 40 major experiments (opens in new tab), the rate rose from 52% to over 70%, while bugs flagged per run increased from 0.4 to 0.7. Many of the attempted changes made the metrics worse.
That connects the evaluation to useful work: did the reported issue lead to a fix? For the support copilot, you need to establish the equivalent connection between answer quality and a correctly resolved customer problem.
You create an evaluation dataset containing 40 test cases:
Questions the company documentation can answer correctly.
Questions where the correct answer isn’t available.
Questions involving outdated or conflicting policies.
High-risk requests that should be escalated to a human.
For each case, you define what successful behavior looks like.
You run your first evaluation and get these results:
32 out of 40 tests passed: an 80% pass rate.
That’s your baseline.
You investigate the eight failures, identify problems with how the model follows the refund policy, and improve your prompt.
Then you run the same 40 tests again.
36 out of 40 tests passed: a 90% pass rate.
You’ve improved performance by 10 percentage points on your evaluation set.
What if the four remaining failures all involve incorrectly approving refunds? Or what if the new prompt fixed six cases but broke two that previously worked?
A higher overall score doesn’t automatically mean your product is ready to ship.
You need to examine performance across different scenarios, investigate regressions, and test on fresh examples the model hasn’t been optimized against.
Start by reading actual outputs and grouping the failures you find (opens in new tab). That gives you grounded categories to evaluate, rather than choosing metrics before you understand the errors.
Codex skills and plugins add a lot to the prompt, so I do think about how to eval the output when I change mine, especially as agents run more autonomously. Right now, I just test it: cool, looks good, keep going. That standard needs to rise when other people depend on the output.
For the copilot, an automated check can verify that approval existed before a refund. Applying a complicated policy exception may need expert review. If you use a model as a grader, check its judgments against the people who understand the task.
📌 Forty cases are a useful starting point, not proof that every production risk is covered.
1.7 Inference cost and reliability
An AI product that works for 10 users might become too expensive, too slow, or too unreliable for 10,000.
That’s why great AI PMs ask whether their product can work reliably and economically at scale.

Once the code takes minutes, the waiting is everything after it
Writing code is the easy part. Say Codex writes something in five minutes. What happens next? CI (the automated build and test run) might take 20 minutes, and code review might take hours or days, depending on your team. Everything around the code is now the bottleneck (opens in new tab).
On Codex, we also built a way to ask it to nurse a PR: wait for CI, fix it, respond to human feedback, and ping me on Slack once I can merge. That used to be ten different interaction points. Now it’s just the very end: “Okay, cool, it’s good to go.”
With your copilot, measure the whole task too. A fast answer can still leave the support agent spending minutes checking sources or repairing a draft.
Your prototype performs well in testing. But when the company rolls it out to 80 support agents, three problems emerge:
Costs are rising: Every customer question triggers multiple model calls, even when the answer is simple.
Responses are slow: Agents sometimes wait 15 seconds for an answer, disrupting their workflow.
Tools occasionally fail: When document retrieval breaks, the copilot can’t complete the task.
As an AI PM, your job is to investigate the trade-offs and make decisions. Four questions:
Cost: are we spending more than the value we’re creating?
Suppose each response costs \$0.10 to generate, and your team processes 50,000 questions per month. That’s \$5,000 in monthly inference costs.
Could you reduce that to an average of \$0.03 per response by using a smaller model for straightforward questions while keeping a more capable model for complex cases?
That would reduce monthly inference spending from \$5,000 to \$1,500, assuming the same volume.
Latency: how fast does the product need to be?
If agents wait 15 seconds for every response, the copilot might actually slow them down.
You could reduce unnecessary model calls, improve retrieval speed, or use faster models for simpler tasks.
But remember: a faster answer isn’t better if it sacrifices accuracy.
Reliability: what happens when something breaks?
Suppose the document search tool fails. Should the copilot invent an answer? Absolutely not.
Instead, it should notify the agent that the source couldn’t be retrieved and offer a retry or manual lookup.
A reliable product needs to handle failures safely, not just perform well when everything works.
Put a limit on retries, and protect actions that change state from running twice. Measure cost per successful resolution, including failed attempts and human review, alongside the cost per model response. Otherwise, apparent inference savings can disappear elsewhere in the workflow.
Scalability: can the system handle growing demand?
A system that performs well with 10 concurrent users may struggle with 500.
You need to understand usage patterns, capacity limits, and how performance changes as demand increases.
1.8 AI product strategy
A great AI PM knows how to explain why the business should invest in an AI product.
You might have built an accurate, reliable, and cost-effective AI system. But that doesn’t automatically make it a good business investment.
The real question: does the value your product creates justify the money, time, and resources required to build and maintain it?
Imagine your team handles 50,000 customer questions every month.
Before introducing the copilot, agents spend an average of eight minutes answering each question. With the copilot, that drops to four minutes.
Here’s what the business case could look like:
Time saved: 50,000 questions × 4 minutes = approximately 3,333 agent-hours saved per month.
AI operating costs: At an average of \$0.03 per response, monthly inference costs are \$1,500.
Quality: Your evaluation results show that the copilot handles routine questions accurately, but complex refund cases still require human review.
Additional costs: Engineering, infrastructure, monitoring, and ongoing maintenance.
These figures are illustrative, and time saved represents potential additional capacity, not automatically realized financial savings.
Now you have enough information to start making strategic product decisions.
For example, should you invest in making the copilot handle every possible support request?
Probably not.
Your evidence might show that the copilot performs exceptionally well on routine questions but struggles with complicated refund disputes.
Rather than investing months trying to automate everything, you could focus on routine questions, where the product delivers the greatest measurable value, and keep complex cases with human agents.
That decision could improve your return on investment while reducing operational risk.
You should be able to explain your strategy to three different audiences:
Engineers: Why you’re prioritizing certain capabilities, models, and technical trade-offs.
Customers: How the product makes their work faster, easier, or more reliable.
Business leaders: How the investment improves efficiency, customer satisfaction, or revenue, and what evidence supports that conclusion.
You should also think ahead: as AI models become more capable and less expensive, how might your product strategy (opens in new tab) change?
The team structure itself has to change when you’re building with these tools. It’s not just that the same team moves faster. It’s that the optimal team looks different.

Two PMs can’t sit in every decision across 15 to 20 product areas
The Codex team was pretty small: something like two PMs, two designers, and somewhere between 30 and 40 engineers across probably 15 to 20 product areas.
We wrote very few traditional specs. Sometimes a spec was roughly ten bullets. When we started a project, I’d think through the most important decisions it needed to feel cohesive with our strategy, almost like guardrails. Beyond that, our engineers and designers moved pretty autonomously and made a ton of micro-decisions themselves.
With tools like Codex, human-to-human collaboration becomes a big bottleneck. Each of our projects really had just one or two engineers working on it.
A short brief should make the customer problem, intended outcome, constraints, and scope clear. It should also explain what the builder can decide independently and what evidence would make us change direction.
We weren’t planning six months out. I’ve been thinking we need a new word for roadmap, because it implies a 12-month thing. It was more: what’s the four-to-eight-week plan? Four weeks later, we’d see what the new priorities were.
Adoption is another part of the strategy. I think the top 1% of Codex users are probably 100x more leveraged than the median user, with the same tool. They treat it like a team member, not like a smarter autocomplete.
For your support copilot, watch whether the team changes its workflow and reaches the intended outcome. An available feature or a growing usage count doesn’t establish that on its own.
And you don’t need to work at OpenAI, Anthropic, or Google to develop these skills.
You can start building them today.
But reading about these skills isn’t enough. You need evidence that you can apply them. That’s where your portfolio comes in.
Side note from Paweł: if you want to learn these skills live, Rohan teaches AI Product Management in Product Faculty’s AI Builder Fellowship (opens in new tab), starting Jan 25. I teach there too (AI-Native PM with Claude starting Nov 30), alongside Shubham Saboo (Google) and Henry Shi (Anthropic) on Agentic Engineering, Reah Miyara (Google) on AI Evals, Jennifer Liu (ex-Google) on AI PM Interviews, and Amit Fulay (Uber) on AI Product Leadership.
New in 2027: Forward-Deployed AI Engineering (Rohan Varma), Agent Design Engineering (John Maeda, GitHub) and Claude Architect Mastery (Carmelo Iaria). One membership covers them all, and you can start free with Rohan’s AI Product Manager Foundations.
2. Three AI PM Portfolio Projects That Prove You’re Ready
Hiring managers care about what your AI projects reveal about your ability to think, build, and make product decisions.
You can build 10 impressive-looking AI demos and still struggle to demonstrate that you’re ready for the job.
Why? Because there’s a big difference between building something with AI and demonstrating that you know how to build a successful AI product.
2.1 What makes a great AI PM portfolio project

What a hiring manager can read in each project
An average AI project:
Starts with a technology instead of a real user problem.
Replicates something that already exists, like a generic ChatGPT clone.
Shows a working demo but provides no evidence of its usefulness.
Claims the AI works well without testing difficult scenarios.
Explains what was built but not why specific decisions were made.
A great AI PM project:
Starts with a clearly defined user problem, ideally validated by talking to potential users.
Demonstrates a working product that addresses that problem.
Explains the architecture, model selection, and important trade-offs.
Includes evals, failure cases, and measurable success criteria.
Shows what you learned, what you changed, and how the product improved.
One demonstrates your ability to use AI tools. The other demonstrates your ability to make AI product decisions.
And you don’t need 10 projects to prove that.
I’ve been a founder most of my career, and I’ve worked in AI coding for a while. I wouldn’t pretend someone coming from another field has the same starting point.
What your portfolio can do is make your technical judgment inspectable. Show how information moves through the system, where an action can fail, and how you tested the result. Building and explaining a small product is a practical way to develop that technical judgment without first trying to reproduce someone else’s career path.
Here are three projects I’d recommend building.
2.2 Project 1: a RAG support copilot
What to build: Choose a software product with publicly available documentation and build an AI copilot that helps users find accurate answers.
For example, a user asks: “How do I reset my account?”
Your copilot should:
Search the relevant product documentation.
Retrieve the information needed to answer the question.
Draft a helpful response with citations so users can verify it.
But don’t stop when the prototype works.
Create an evaluation set that includes questions the documentation can answer, questions it cannot answer, and questions involving conflicting information.
What to showcase in your portfolio:
The user problem and why you chose to solve it.
A working demo and a simple explanation of your RAG architecture.
Your evaluation results, including incorrect answers.
One important failure you discovered and how you fixed it.
What this proves to hiring managers: You understand product discovery, prompting, RAG, AI UX, and how to evaluate an AI product beyond a successful demo.
2.3 Project 2: an AI agent for a real workflow
What to build: Create an AI agent that can complete a bounded, multi-step task.
For example, build a support agent that:
Reads an incoming support request.
Classifies the customer’s issue.
Retrieves relevant information from company documentation.
Drafts an update to the support ticket.
Requests human approval before taking any consequential action.
Now introduce a failure.
What happens if the documentation tool stops working? Does your agent retry, ask for help, or confidently continue with incomplete information?
Design the workflow so the agent handles these situations appropriately.
What to showcase in your portfolio:
Your agent’s workflow and tools.
What the agent can and cannot do without permission.
How human approvals and handoffs work.
What happens when a tool fails.
How you tested whether the agent completed tasks correctly (opens in new tab).
What this proves to hiring managers: You can design agentic systems, manage tool permissions, make autonomy decisions, and balance automation with reliability and safety.
2.4 Project 3: improve an AI product with evidence
This is where you can really differentiate yourself.
What to build: Take one of your previous projects and make a measurable improvement.
For example, imagine your support copilot costs \$0.10 per response.
You discover that routing straightforward questions to a smaller model could bring the average down to \$0.03 per response, but the smaller model struggles with complex refund policies.
Instead of switching everything to the cheaper model, you design a routing system:
Simple questions go to the cheaper model.
Complex questions go to the more capable model.
High-risk or uncertain cases go to human review.
Now run your evaluations again.
Did you reduce costs without sacrificing answer quality? Did routing introduce additional latency? Did any previously correct answers become incorrect?
What to showcase in your portfolio:
Your original performance and cost baseline.
The improvement you wanted to achieve.
The changes you implemented.
Before-and-after evaluation results.
The trade-offs you made and why.
The costs in this example are illustrative. Use measurements from your own project.
What this proves to hiring managers: You can use evidence to improve AI products, reason about model economics, and make technical decisions that support business outcomes.
2.5 How to present your projects
For every project, make five things immediately accessible:
The problem: Who is the user, and what problem are you solving?
The product: A working prototype or a short demonstration.
The architecture: How your AI system works and why you designed it that way.
The evidence: Your evals, success metrics, and failure cases.
Your decisions: What you changed, what trade-offs you made, and what you learned.
If you haven’t tested your product with real users, be transparent about that. Don’t invent adoption or business impact.
📌 Three thoughtfully built, evaluated, and documented projects will give you far more to discuss in an interview than 10 disconnected demos.
And once you have that evidence, the next challenge is getting the right hiring managers to see it.
Hey, Paweł here. To learn more about building with AI, see our Product Engineering series: Part 1 (opens in new tab), Part 2 (opens in new tab), Part 3 (opens in new tab) and the free Product Engineering skills collection (opens in new tab). No coding.
Here, you can also learn how I built my own harness step by step:
3. How to Land AI PM Interviews: Roles, Resume, and Outreach
- Start with the user’s bottleneck, not an AI solution: in the support example, agents lost time finding correct policies, so the proposed copilot retrieves policies, drafts cited answers, and lets agents verify, edit, and approve them. The suggested goal—halving policy-answering time without compromising accuracy—is illustrative and should be validated with users.
- Diagnose AI failures before changing prompts: an incorrect answer may stem from retrieval, missing context, or generation, and each requires a different fix. Evaluate answerable, unanswerable, conflicting-policy, and high-risk cases; inspect failure categories, regressions, and fresh examples because a higher aggregate pass rate alone does not establish readiness.
- Bound agent autonomy: use permissions, spending limits, and human approval for consequential actions; predictable tasks may be faster, cheaper, and more reliable as fixed workflows, with architecture revisited as evidence changes. At rollout, measure the whole task—including latency, tool failures, human review, and cost per successful resolution—not just model-response cost.
- Rohan described Codex as having roughly two PMs, two designers, and 30–40 engineers across approximately 15–20 product areas; the team used few traditional specs, relied on strategic guardrails for autonomous decisions, and planned in four-to-eight-week increments rather than six months.
