0:01All right, 103. Marty, I think we can go ahead and get started. Hello everyone. My name is Jack Dvita. I'm the head of customer success at Interpret. Uh, in one sentence, uh, Interpret builds software products uh, that help you learn from your customer feedback and more recently agents that help you take action on top of that customer feedback.
0:22Of course, joining me today, uh, Marty needs no introduction. We're going to give him one anyways. founder, Silicon Valley Product Group, author of Inspired, Empowered, and Transformed. Marty, it's good to see you. Welcome.
0:35Thanks, Jack. Thanks for inviting me.
0:38So, today we're here to talk about AI.
0:41We're here to talk about product management, customer experience, things that are very hot topics uh across the world right now. But to get started, I've gotten a couple of questions from people in my life. And in preparing for this interview or this conversation, you have been linked with some of the famous quotes around Steve Jobs saying that you cannot focus group your way to innovation or people will often quote the Henry Ford, you know, if I'd asked people what they wanted, they would have
1:09said faster horses. And we are in the business of customer feedback. And so may maybe to get started, I'd just love to hear your perspective. what what role does customer feedback play for an empowered product team?
1:23Yeah, there are a bunch of uh quotes and some of them are just meant to be provocative of course. I thought uh one of the one of the most telling quotes there's always some truth in all of them, right? But there's is is actually a Jeff Bezos from the early days. He used to go to his product teams and say, "Look, the biggest mistake you can make is listening to our customers." And he
1:51would say, "Look, they they they don't know what's possible. It's your job to figure out what's possible." But then he would say, "By the way, the second biggest mistake you can make is not listening to our customers." And his point, of course, is to emphasize, yeah, it doesn't mean that you don't uh listen to your customers. What it does mean is you don't ask them what to build. You don't ask them what to fix. Um those uh you know there's been products for
2:20decades out there that would basically let customers request features and then worse vote them up. So literally imagine your product team going, "We know what we need to do, but we've already set expectations with 2,000 customers because they top voted this thing."
2:38which a lot of times they're saying well this is not going to fix the problem but that's uh so that's the trap you want to avoid but the answer ironically and in fact anybody who reads my content or comes to any of the sessions we do they usually leave realizing that they're doing so much less direct contact in direct customer contact than they need to realize that oh are you serious
3:08serious. We're supposed to be talking with customers every week. You mean we're supposed to be running tests and collecting data every day? And yes, absolutely. So, um there's some nuance there, but it is, you know, num the number one reason and this is just known as the first principle of product is that customers have they know pain. They they they live it. But that doesn't mean
3:36that they know the solution to that because that depends on many factors in especially in including the enabling technology what is just now possible. Um and and you know you see this problem repeated constantly because so many companies are just chasing the loudest voices or chasing you know the symptoms and not the cause. But anyway, to be
4:05very clear, uh, good product depends on strong signal and interaction from customers and users.
4:15Yeah, I I love that where it's like, you know, they they aren't here to dictate the road map. They're here to help us.
4:23You know, we got to understand what's the actual problems we're really solving. How are we actually serving them? and that that comes from them. Um, getting into the AI thing and right, you're talking about all of these new amazing tools that are available to us and you've you've talked about, you know, and shared some examples for how empowered teams are taking advantage of AI products today to move faster, right?
4:47Where the prototyping phase during discovery, you can move, you can have 10 or 20 prototypes in a day or in a week, whereas before that used to take a really long time. and that even the cost of writing code is much cheaper than it used to be. And and so I'm curious if it used to be that writing code or developing prototypes and that part of the product discovery was the bottleneck I guess is that still the most common bottleneck for product teams or has that
5:16bottleneck moved somewhere else?
5:19Well, so so the two things going on, the truth is most companies in the past and still today are not actually building prototypes. They are just building product. So the two things going on here, the first is what model are they using? And I I have to be clear I don't when I say model I'll try to be clear because there are large language models today and there are product models today
5:47product operating models today. Um and so uh the word is unfortunately a little ambiguous so I'll try to say uh product model versus language model or uh so um if the vast majority of companies out there are using what's called the project operating a project model where it's something that everybody's familiar with. they just don't necessarily call it that. But a bunch of executives
6:17usually or sometimes customers actually especially if it's large ticket products will will essentially drive a roadmap.
6:25They will request slashdemand um capabilities, features, projects. Uh a set of executives will somehow triage that and they have to decide because we can't build everything. uh that is changing but the pri we still can't build everything we just can build a lot more. Um and so they prioritize that onto a road map and then eventually something percolates to the top of the
6:54road map and it's time to get to work and then they try to uh okay they they know they want this feature. We have to figure out how to build this feature. we have to at least do some maybe usability test of this feature, make sure people could figure it out and then okay, we've made sure it works, it doesn't break, uh, and you can use it, so we'll push it live. And that's, uh, the project model basically simplified. Um, the problem with the project model is always been
7:22the same problem. uh that when you work that way, depending on what pundit you want to believe, it's somewhere between 20 15 20% actually achieves the outcome you were hoping. The vast majority of work doesn't. And that's usually because uh it's not because it breaks like it crashes. It's not because it's got egregious usability issues. It's usually because the customer that is not the
7:51solution they wanted and so they don't choose to use it. They literally don't choose to use it. Uh they've got lots of alternatives including and especially staying with doing whatever they're doing today.
8:03And so they uh it takes a lot to get them to actually change and so they don't. Sometimes it's because once they actually get to work on it, they realize it's way more involved and it would take a lot longer and so they end up cutting a lot of corners and it's just not very good or it's going to take too long to build or it's going to have to address too many tech debt issues or whatever and so the result is not very good and uh you have a lot of failed product.
8:29That's the project model. Now, the vast majority of the world, I mean, despite my best efforts and lots of other people's best efforts, the vast majority of the world is still working that way, especially kind of the further away you get from Silicon Valley, the more common project model is. And um so what do most people do? They use AI to speed up the project model. And that is literally what's going on in most
8:58places today. uh recently it's been it's been observed by a lot of people all over the industry that teams are getting faster uh but they're not actually getting better outcomes. So and uh both McKenzie wrote about that in their last quarterly and at lastian wrote about it in their annual survey and uh they named it not me but it's called the AI productivity paradox. Yes teams get
9:25faster but outcomes don't get better. That's very predictable if you're using the project model.
9:35So, but but it's interesting if you look at these really good companies out there that are uh that are genuinely making use of AI. Many of them are building the AI products that we know and love today, the anthropics of the world. They and those um they don't build that way. They know better than to build that way. they haven't built that way since they started or many companies uh that are pre- AI have been have not been building
10:05that way for decades. So they build differently and they have this distinction between uh we say building to learn versus building to earn. So building to learn is prototyping and and trying to figure out a solution that will solve the business problem and building to earn is okay we have a solution now we need to to make money.
10:30Now we need to build a something that your business can run on and you can monetize and you can afford and you can market and sell and service and that is uh build to earn. Now in product model companies they use AI very differently for build to earn versus build to earn.
10:50Did I say that right? Build to learn versus build to earn. Um most people in the project model are only doing the build to earn. That's all they're doing.
11:00So they and as everybody knows the tools for that are that's really what's leading the parade on AI is tools to build to earn. uh the um the codeexes and the clock codes and the cursors of the world, they are on fire because they are amazing on what they can do. Um you know, there's still bumps in the road and people figuring out because just because they're faster doesn't mean it's better quality and there's all kinds of
11:29stuff, but it is getting I mean it's already done better than the last 50 years cumulatively and it's only getting better. So I um that has really changed the calculus.
11:43But if you think about it, if even in the best companies, if the cost to build to build to earn goes down, what does that really mean? It means the bottleneck is what do we build?
11:57What do we build that will actually help? What do we build that'll actually achieve the outcome or deliver on our vision or whatever it is faster? And so uh what that's really done is focus the energy on people to okay well how can we do that better and that's build to learn. Um you know the truth is good product teams even before AI have done 30 40 50 prototypes a week in product
12:24discovery but that required some real skills. It's also why Figma was so valuable, you know, at least in the preAI era, because they helped you do this. But today, almost anybody can do this. Almost anybody, not everybody, but if you want to and you have some some dedication to learning the craft, you can learn how to do 10, 20, 50
12:52prototypes per week. And when we do a prototype, the point of the prototype is not to just say, "Okay, give this to the engineers and say, build a product version of this." A point of a prototype is to test it and see if it actually creates a successful product. That means testing with users, testing with buyers, testing with stakeholders, like making sure your security people are good with this, making sure your legal people,
13:20your finance people, your sales, your marketing, your customer success team, making sure they can do what they need to do. Um, and also making sure your engineers can can truly build this and there aren't any big surprises or any feasibility issues. And that's what we do with the prototypes. That's why you need so many of them. It's not unusual to require 25 to 50 prototype iterations
13:48before. And I'm not talking about some trivial little bug fix obviously. I'm talking about some significantly new capability. It might be an innovation.
13:57It might be a new cap feature but something significant like uh you know reducing the time it takes to set up a new customer or uh handling payments better or whatever it might be. So that's um that's what we're trying to do and I I see that the best teams are actually using AI very differently than most teams.
14:21Yeah. So in a world where you can build anything or where the cost of building goes down, it puts even more importance on what you build. And the way to verify that you're building the right thing is iteratively through this build tolearn process and over the course of the build tolearn process, you might be learning different things like you mentioned the viability, the feasibility, the various dimensions that are worth testing. One
14:50of the things that I'd love to hear from you as we see with these tools the cost of code coming down and and you can prototype more that the teams maybe get a little bit smaller uh that the ratio of product managers to developers gets gets smaller and you have these tighter knit teams that can own more and that can deliver more. But like you mentioned for when you're working on something very meaningful that you know impacts across the business security
15:19monetization you know all of the different dimensions what are the best teams doing in terms of speeding up cross functional collaboration right if if you can write code more quickly are you seeing that lawyers are able to align on privacy more quickly is this is this speeding up everywhere or is that has the collap collaboration kept up with the rate of iteration?
15:45No, it's super good question. Um, and there's a lot going on in there. And actually that question, let me uh I want to sort of peel back the layers because you actually said two very profound things in there. The first one was you observed in the language of product, you observed a change in team topology. In other words, the typical team shape is changing. You also observed like what are the second order effects to
16:12collaboration when you've got these different shaped teams and and it's not just so let's talk about the first one first uh that you brought up about the nature of the change and and some of the ratios are kind of surprising uh for example initially uh the the ratio between product and engineers actually increasing there's there because now there's smaller number of engineers and more importance and complexity on the
16:40product role. But still basically what the the overarching thing is that say uh a year ago or two years ago your average product team had one product manager, one product designer and somewhere between five and 10 engineers.
16:57Today we still have a product manager. Uh and we will talk more about like what that really means today. Um we if it's got a user experience we typ in a good companies at least they do have product designers even though a lot of the lower level tasks are more automated today but they have a much smaller number of engineers and well or at least it's it's getting smaller. It's a moving target right now. It is getting smaller in some
17:25teams. It's literally one uh that is doing what five were doing before uh in some in more I should say it's more like three or four instead of 5 to 10 it's maybe three to four but that's a big change and that actually obscures another change which is at the same time the shape of the team is is shrinking the scope of what they own is increasing.
17:54Now, part of this has got to do with cognitive load in there's a lot of factors that describe how much does a team own, how much are they responsible for. Um, and one of the things is how much can the engineers wrap their heads around. It turns out with these new generation of tools that can analyze multi-million line code bases virtually instantly, um, you can actually wrap
18:21your heads around a lot bigger chunks, which turns out to make everything nicer for the teams because one of always, and this is still true, the biggest one of the biggest complaints from teams, even in the best companies, is they hate having so many dependencies. They hate have like I want to do it.
18:43Look, our customers have been screaming for help on this feature. We know how to fix it. The problem is it takes 10 teams working together to actually solve that. And what a pain versus one team being able to say we got this and just fix it.
18:58It's um it's drives people nuts on the teams at how all these dependencies and some of those dependencies are because of legacy systems and tech debt and old architectures but other ones are just because of these other issues as a consequence. You have to slice up the pie and there's just so many things. So today the scope is getting bigger. And what teams really love is having endtoend responsibility. So then they can they can they don't just have
19:26empowerment, they have autonomy. So that's the big first big thing you had in there. The second big thing you had in there was like what does this mean for crossf functional? Uh how what does this mean for the collaboration? Um and so the first thing we should say there is that uh you know the the members of a product team typically product management, product design and engineering. The design is pretty much a
19:57you know they are working and responsible for that user experience and that's what they focus on. The engineers of course are focused on delivering a product quality version but the product manager is responsible for both the value of the solution. Will people buy it or choose to use it? And and this is the most relevant to your question, the business viability of that solution. In other words, is it something we can af?
20:27Is it something we can afford to build?
20:29Is it something we can monetize? Is it something that's legal compliant? Is it something that's uh respects privacy laws? That that's compliance. Now in the project model, you would typically just schedule some big meeting and get all the stakeholders in a room and fight it out. But in the product model, we are iterating on solutions many times a day.
20:55So you're not trying to take the whole company and get them in a room all day every day forever. They all have their own jobs. The product manager in the product model has to have this broad knowledge of how your company works because they have to know, you know what, this is a compliance issue. I need to go meet with the compliance person and talk about whether or not what we're doing is going to be on the right side of the line or this is going to have big
21:25impact on our marketing programs. So I need to go sit down with marketing to figure out if this will still be cost-effective or whatever it is. So the product manager is really managing that crossf functional collaboration with the other parts of the business. But that and that is their contribution to the team. Think about it. The designer doesn't know these things and the engineers don't know these things. And if you want teams to be empowered and make decisions, they
21:54need somebody on the team that knows this. And so that is the product manager. Um in the in the old project model, none of what I just described really happens because the the solutions are coming down from the leaders in the form of a roadmap. But in the product model, they don't have that. They're given a problem to solve and an outcome to achieve. And so they have to have the skills and the knowledge to answer these
22:24I'm I'm curious here. Right. So, what we've described is three roles with three distinct responsibilities, right?
22:33where we're talking about designers working on the interactive user experience, engineers being responsible for uh feasibility, reliability, technical capabilities, and then the product management function where we're talking about are, you know, can we sell this? You know, is there a market for it? Do we need to talk to compliance?
22:58How do we charge for it? How do we price for it? being responsible for this crossf functional does this make sense does this actually solve the problem and it's a very common thing to hear these days that these three roles are actually collapsing on each other that designers will write code now that engineers should be talking to customers right that that there's merging into this builder role and I I think you and I uh
23:26both maybe a year ago felt like this was the direction that we were moving in but from what I understand that this is something you've recently changed your mind on that in fact we will see uh these three roles continue to be distinct from each other am I am I understanding you correctly and how how do you see these roles evolving are they are they going to blend together are they not going to blend together where do you see this going and if you know what's what's changed your mind on this recently
23:55well uh it's a little nuanced here so Because the reason I think you may think I changed my mind is because I often talk about the the answer to that question totally depends on whether you're using the product model or the project model.
24:15So in the project model I believe that the mo the roles are converging especially uh honestly it's a nice way of saying you don't need product managers. That's what we're saying and I I believe that you don't need them. We have never really needed them in the project model. We needed project managers. Absolutely. And that's what most product managers were doing.
24:40Project management. They just had a title that maybe made them feel better or something. There's I talked about it as product management theater.
24:49And that role I think has been for a while diminishing because people realize uh you talk to the engineers or you talk to the designers and they go we don't need that person and they're right they don't. However, if you're now talking about okay empowered teams and those teams have to make choices, then these different
25:16skills are not only necessary but probably more necessary than ever before.
25:23Now that doesn't mean it has to be three people. We have always had this situation in the even uh I have some of my oldest friends I call triple threats because they are and I'm not one of these by the way. I wish I was but you just can't. It's very rare to be a triple threat. A triple threat is somebody that actually has the deep skills in all three of those spaces.
25:48That's a triple threat. People don't realize what they're saying today when they say, "Oh, the roles are collapsing." you know, you're gonna, oh yeah, like all of a sudden, you know, we've evolved to be a whole different level of intellect. It's really not true. Most people that are saying that and they're talking about the product model, they do not understand what product managers have to do in the product model. So, it's easy to say it if you don't know what it is. Or the way
26:18they define the role is they think of agile product owners, which basically anybody can do. It's an administrative task. It's not even a job. It's a role.
26:28So, uh that confuses people. But I do think uh and I' I've felt that um that those roles now there's some more nuance. I don't know how much you want to go into this, but I think it's quite clear for engineers and it's and there for a while there were some people saying, "Oh, we're not going to need engineers. So, product manager is going to be able to go build." Product manager can build to learn, but there's a big difference between build tolearn and build to earn.
26:58And most of the time a product manager has tried to do the build to earn, it has been a fiasco.
27:06So it's there those people are learning the hard way that they were just embarrassing themselves that they're really. So I do I think the future of engineering is strong. I don't think it's going away. And I for the same reasons we've talked about, I think the future of product management is strong, probably stronger than ever. And we haven't even talked about this yet, but the risks of products going forward, especially AI products, probabilistic
27:34products are higher and more difficult than conventional products. So I don't see this role going away. I see it becoming more clear to more people that okay this is a hard job and we need somebody who's got the skills to do it.
27:52The hardest one to talk about being very honest here I'm uh in terms of it's a moving target but the designer role the designer role is tricky because um you know when we talk about a designer role normally it's there's service design there's interaction design there's visual design with a device there's industrial design there's this foundation of user research there's a lot of different skills in there some of those skills the language models that
28:22generate code are generate doing a very good job on the design and um and that is putting the role of product designer kind of under stress.
28:36Now I think there are many there's probably a large percentage of products that that will probably do the job better than what it has been at least maybe not great but better than it has been. But there is another class of products especially AI products where it's really difficult behaviors for the users to understand. Now I'm talking about products not agents. I'm talking about products that are meant to be used by humans.
29:03If it's a product meant to be used by uh by an agent, then that's a platform product. We know how to do APIs and we haven't needed really product designers for that for years. So that's not the issue. But for products designed for humans, there are some really complicated design problems. And I I think what we can see is the best designers in the industry are joining the AI companies right now.
29:30No, if the AI companies didn't believe this was essential, they would not be hiring them.
29:35And and I hope for all of our sake that in the end, you know, there's a strong case to be made that the designers should win, that things that we experience and use should feel beautiful and should feel like they're for us and that they're my perspective is that there's going to be a premium as the amount of software goes up. software that feels like it's for you and that it understands you. Hopefully, we we see more of it. And this is related to my
30:04last question before we go to questions from the crowd here is something that I've observed is I think there's an optimistic case that AI is going to help enterprise software get to be as delightful as consumer software, right?
30:23that uh will be able to build better products that people enjoy using and feel like they're for them. But I'm I'm actually not convinced here.
30:32And I I think part of one of the the issues that we run into is with enterprise software as opposed to consumer, you'll often have a difference between who the buyer is and who the user is versus if you're using an app on your phone or if you're using consumer software. Those those tend to be the the same as each other. And and so I guess broad question, does does AI change this? And if so, why why would quality
30:59win now when it hasn't been the defining characteristic of enterprise software historically?
31:06Yeah, this is a this has always been a real frustration point to me. the so because look what what's the truth behind what you just said is that most enterprise software is pretty horrible right that's just the reality is most of it's pretty horrible how and you can the first obvious question I asked and most people ask is how in the world do they stay in business their their saving grace is the people that buy it are not
31:35the people that use it like you said the users and buyers are different if they were not different it would be a very different situation, but they're not different. Now, um but but because they're different, so often people would buy it and they're just looking at, you know, it doesn't mean it doesn't have good positive business value. Even if the users are truly in pain every day from from the business buyer point of view, they're like, "Well, we hire these
32:04people. They don't have to have they don't have to love the software they're using." There's been a trend though that it's been called consumerization of the enterprise which is just like you said why should those people have better software when they got home at night than they have during the day when they're spending you know all these hours and there are some great examples today of true B2B software that is uh
32:31beautiful to use amazing to use and they truly get value you know not just value but delight out of using it. And so, um, it doesn't have to be that way. Now, uh, and it's frustrated me just like you that it's taken so long. There's some other dynamics going on though right now in the industry. one is um well the bar is certainly lower for people to build
33:00their own and and that's certainly happening but not nearly to the degree that I thought it would. Uh and recently one of the industry analysts Ben Benedict Evans suggested why people like me were surprised at how few people were actually building their own. And he was arguing because the users of software are very different from the right people to create products for those users. And
33:30uh he was sort of reminding us of something I've learned a hundred times over and over, but you always forget it.
33:36You tend to believe that people are more like you than you think. Uh so anyway, um that I think that's going to remain true. also something you mentioned earlier uh it's not just people that are using business software going forward it's it's agents it's other software using uh that's capabilities so there's sort of a major rearchitecture going on around all everybody's business software
34:04everybody's virtually every company I know is opening up their platform to agents so that it's not just people using it and I think that is uh Now initially the thought was well customers would build their own agents but I don't I think per Ben Evans that will be less common than we thought but it doesn't mean that others won't systems integrators um VARS value added resellers uh the s
34:32the big consulting firms of the world they will create these for the companies uh that use these services. So there is a big shift going on in the uh services plus when you could there's a whole whole big discussion around pricing because um because many well at least if it's AI powered this is a very different economics so it's not like conventional software the marginal cost can be very
35:00high and so now that's pushing a big change to business software to be paid for performance pay for the real value which I think is a very good change overall, but you know, these are these are going to be much more expensive products at least for a while. So, there's a lot going on there. Um, and a lot of different threads you can pull on, but uh I do think for the foreseeable future, we are going to need strong designers for the reasons you
35:29said. And the more products that are out there, the more the quality of that product, the experience of the product, uh the taste that goes in, we call it product sense or design sense matters.
35:44I c I certainly hope this ends up winning. I agree. I'm optimistic about that. I I loved the piece um that you referred to. I believe there was a quote something the effect of um if you want to make good sales software, you have to be good at sales. But that being good at sales alone is not enough, right? And so this distinction you're making between tool users and tool builders and how much work do people actually want to do, h what percentage
36:14of the world genuinely are tinkerers um and and want to experiment with these things. And I think as the technology has changed under our feet so quickly, the paradigm of like what is established good solutions look like feel like what is possible hasn't caught up and we don't even know. And so we're in this weird position where the tinkerers are winning because they're building for themselves. They're building developer tools. They're building for other
36:42tinkerers. And I I don't think we've even touched the surface of what this will look and feel like for real consumer software and software for people who don't want to do all this configuration and customization for themselves.
36:55Yeah. And I think that's really always been the case. Uh the tool builders have always been on the sort of bleeding edge. Remember too that we're they're working with the enabling technology. It gives them a big advantage. And so um yeah, that's that's the truth.
37:10All right, we've got a couple of questions from the crowd uh that we'll jump into right now. Um I'm uh this one is from Harsh Patel wants to know how do you actually measure the impact of AI tools? A and so you've talked about how lots of teams feel like they're moving faster but maybe are not feeling the ROI.
37:35uh harsh goes on to continue that some teams have mentioned tracking tokens token usage per user how does that change over time right and then even as you're discussing now if you're going to not only use these products to build to learn but if you're going to sell AI solutions that now have test time and runtime costs to you how how are you seeing teams measure the impact and measure the ROI and this is very straight
38:04straightforward but it doesn't mean it's easy right it's but it's very straightforward this is the I didn't say this before I did mention you know the project model and the product model and the project model is all about output meaning we ship stuff and shipping stuff is good don't get me wrong we if we don't ship anything we don't have any chance really of anything so we need to ship but we need to ship such that we solve customer problems in ways that
38:34customers love but also work for our business. And that's called an outcome. Business outcome impact, but it's a it's a real business result. And that's the goal. Everything we've talked about this hour is in service of that. The whole product model is saying the goal is not shipping. The goal is outcomes. And so that is directly what leads to well if the goal is outcomes we're not done until we
39:08achieve the outcome. We're not done when we ship. And so it's all about in fact the way it's often described is the product sorry the project model is all about time to market which of course is important. How long until you get this live? The product model is all about time to money. In other words, yeah, it's got to be live, but it also has to earn money or whatever it is, the
39:36outcome that matters. So, time to money is the focus. If time to money is your goal, then you're going to work very differently. And in fact, you know that what if you're going to have to ship 20 times in order to get something that works, the last thing we want to do is beat up our customers 20 times, you know, just sort of launch garbage at them. What we want to do is learn much
40:02faster, much cheaper, and much more sensitive to abusing our customers. And so that's what product discovery is. That's what drove product discovery all those decades ago is the the the idea that what matters is not shipping its outcome. So that's the idea and the short answer to the question was we have an outcome.
40:28In fact, Andy Grove at Intel many years ago, he he need that was a classic product model company and he needed a way to give teams problems to solve and business outcomes to achieve. And that was where OKRs came from. Objectives and key results. The objective is the problem to solve. The key result is the outcome. And it's been around for a long time. The reason most people don't use
40:54it and do it is because it's way harder.
40:58Like with anything, it's it's easy to talk about, but it's it's hard hard, you know, just simple and easy are are are two different things from each other here. One of the things that I' I'd love to get your take on. So you talked about right in product discovery we can do lots more iterations ourselves without thrashing the customer so that we feel confident when we do send to a subset of users or we do have an alpha or a beta
41:27group that it's it's more likely to be correct. And at the same time, you've talked about how for people who are relatively younger or newer into their product career that these large language models can also be wonderful coaching tools for them and and that they can guide us through this process.
41:48And then now I start to see there's maybe a little bit of tension between what is the right amount of work to do internally versus externally where as the technology is changing so quickly we need to get signals from the market but like you said sharing 20 prototypes over the course of 20 days with a customer could have a a negative result and and so what are you seeing and what
42:15works well to find that right balance of internal iteration, internal coaching versus getting signals from the market.
42:24What What are you seeing work well?
42:26Okay. Well, again, those are two things.
42:29Let's separate each one. The internal sorry, the coaching, that's just amazing and orthogonal to this. If anybody on this call has not at least tried setting up one of the large language models as an AI product coach, please do that. There's an article, maybe somebody can share it. We published on you don't need to pay anything. You don't need to buy anything from anybody. You just have to have access to a large language model.
42:58And uh this article will show you how we recommend configuring your model so that it serves as a very useful product coach. And by the way, one of many many kinds of questions that you can ask it is exactly this question. Um because this, in fact, Jack, what you just described is pretty much my normal day of discovery coaching. Um because every team's different, right? They'll So the first thing I need to know is what kind of product are they talking about? If
43:27they're talking about like a a regulated financial product, that is one thing. if they're talking about doing a uh uh a multi-person game, that's a totally different kind of product. And we will use very different techniques for that.
43:43If they're working on a platform, if they're working on a device, there are so and we have I mean I don't know I've never seen an inventory of it, but personally I know more than a hundred techniques for product discovery. Uh and I'm a student a lifetime student of these techniques. And so I want to know what kind of product it is. And then I want to know what are the risks. What are you worried about with this? Is it a is it a li liability issue? Like a legal
44:10liability thing like self-driving cars.
44:13We got to make sure it doesn't kill anybody. That's very different than uh it's a game. And a game is like if you don't make them fall in love in the first few minutes, it's over. You failed. So very different situations. Um and so sometimes uh well let me just put it this way. We have techniques for the case where it needs to be a very select optin group of users and customers and we have other
44:42techniques for when it's need it can be much broader. Obviously you don't want to put anybody's money at risk or their livelihood at risk and so all those matter on what the right technique is.
44:55But we have techniques that work with as small as a half a dozen users to as many as millions. And it's all about using the right technique for the job. Some techniques designed to help us prove something works or not. And some techniques exist to help us understand why they don't work. That's another big way to categorize them. lots of techniques uh for all these different
45:25Let me just say just building and pushing it live is very hamfisted way of working first of all it ends up taking way too long but it also you're poor if you had customers that liked you before they won't like you pretty soon.
45:43So what you're saying is it depends uh the classic product manager uh answer and question but it's but it's true right it's like I think a lot of times people are looking for a a one one fit-all solution or uh you know a magic bullet that that will solve uh solve their questions and you really need to make sure you're understanding like and I I know this is one of my main takeaways from you is what what are we even up to here? What's what is the whole point? what what is the outcome
46:12we're trying to drive? what is what is you know there's a fancy way of saying why are we even doing this but it's different you know there's there's it depends like on all these situations but this is a case where it depends on the product team having the understanding the craft of product as long as they understand the craft of product they're going to make reasonable choices it's not it's not magic so that's different than yes there are lots of different factors out there um but
46:41they have to know the craft And and the truth is most people don't know that craft because they've been working in the project model for their careers.
46:49Um I' I'd love to go to another question from the crowd here. This one is from Dan Goldberg. uh asking for reference examples where organizations have created an integrated product development life cycle like tooling stack with AI across the different steps versus uh creating co-pilots or or using you know AI chat in isolation. Do do you
47:18have any examples where you're seeing and what how should people think about the difference of having a chat GPT license versus having a a fully integrated software uh development factory?
47:31Well, I mean um I have to be a little careful here. Um the framing of that question is very problematic.
47:43Okay. In other words, um you do not want a single integrated AI stack. That would be, I would argue, a really bad move. Um first of all, build to learn and build to earn are completely different activities with very different goals with very different skills of the people operating it, very different context. So the idea of this single thing now don't get me wrong that's not the first person
48:12that says oh I want to buy this every it's amazing how people are so hungry for structure they're hungry for frameworks they just want to know what the next step is give me a playbook give me a recipe and I try to explain and now this is going to sound like tough love so a lot of people don't want to hear this product success uccessful product is all about thinking.
48:40And what was just described implicitly in that framing was I don't want to think.
48:50I love I love that framing. One of the things that we try to think of or that I try to think of is what what is the purpose of these AI products, right?
48:58Where where does this go? How do we use it? And as much as possible, I try to think of it as the these are, you know, they're computers at the end of the day.
49:08And they're uh the ratio of what they should be working on is doing, not thinking, right? Where they they can be helpful tools for us as a sounding board for our own thinking and can help us increase the number of iterations that we do as we're experimenting. But we have to do the thinking while taking advantage of these products that you that can do the doing for us and and
49:37they can be really helpful synthesis for inputs, right? You know, we we have customers who would have millions of customers. How do you know what their most common problems are, right? Well, you could read every single support ticket or you could get that summary sent to you. and and so it's a helpful input into your own thinking, but you can't outsource the thinking to it.
50:02Yeah, I uh 100%. The thing to realize is that that presumes you have a product person with the skills to do this thinking. This is called product sense and it's something you earn. It's not something you're born with, but you earn it. But you have to earn it. Uh the truth is for so many people that have not put in the work to learn that.
50:30Honestly, the the tools are probably going to make at least as good a choice as the person. But that's uh that it I don't think most companies will be able to succeed at that low bar.
50:45Yeah. when you know ma mathematically if these large language models are uh taking the average of all of their training data it makes it very easy for us all to regress to the mean right where if you're really bad I heard you know there was a framing someone shared the other day where if you're really bad at something and it makes you average that's amazing right for someone who's worked in sales his whole career and can feel like an average designer that's wonderful but for someone who has spent
51:12time honing the craft about actual design coming back to average would be the worst thing that could happen.
51:20Yeah. And absolutely and I think we're seeing a lot of that right now.
51:26I'm I'm curious, Marty. Right. So, you've been a voice in the industry advocating for proper product management for years. Um, and have been linked with other wonderful people who I followed and tried to learn from. Trish Docei, uh, Terresa Torres, Lenny Richitzky, wonderful people offering perspectives, but there's a case to be made that a lot of this is, you know, people who are maybe not AI native or who have been doing this for a long time since before.
51:54And, you know, there's a strong case to be made and I think we all see and have agreed even here that things have changed or are in the middle of changing. I'm curious, are there any who are you paying attention to? Are there any new voices? Who do you think is really really good at this today that is also worth uh digging deeper into?
52:14Yeah. Well, um interestingly several of those people you described are actually also really good at this.
52:24Well, I mean I think it's because of the nature of these people that they very quickly established a leadership role with this. Um and I religiously follow them. Uh so for example, Teresa Torres has not just been uh an expert on product discovery, but she's been exploring for two years now product discovery for AI products and is I as far as I can find and I've been reading basically everybody um the the best
52:54thought leadership on that front. Now I also another good source for people and I use this myself. So, I'm a subscriber to Lenny's newsletter because Lenny is constantly uh interviewing and sharing from people from everywhere and every perspective and that's what I find useful uh and we all take but I would I mean at least uh you can never know what you don't know right so there's always
53:22this possibility out of left field but what I have found useful is principles so My work focuses on principles that are not only continue to be true, but certain principles become even more important in the AI era. And that's sort of what my focus goes to. Um, and yeah, but the truth is and I and I do know
53:50people at all the top AI companies. Um, they don't know either. They don't know either. There's a lot that is really unknown right now. I mean, all you have to do is follow the CEOs to see how little they really know about what's going on. It's a very weird situation we're in. Uh, by far the closest thing was the internet disruption and I was at the center of that. So, I saw that from the inside.
54:18And there's a lot of similarities between things that were totally disrupted. But it's even more. But what's weird about this era with AI is that even at these companies, they really don't understand enough about how the system works. And that's I don't know that that's ever been true before.
54:39Maybe in medicine, they've discovered things that they don't really know how it works. They just know it works. But I'm not a special I don't not an expert in drug discovery or anything. So, uh, but in the tech world, I can't think of a time where that's been the case, where something has been truly amazing and disruptive and we're like, well, we don't really know how it works. So, when you don't understand at a, you know, fundamental level how it works, it's very hard to extrapolate to where it's
55:10And, and it can be scary, uh, where it's very uncertain, but I think that same emotion can be exciting. Um, and there's a lot of opportunity. Who who knows where this will play out? And one of the things that we always like to think about and other people have referred to as the barbell effect where it's like how can we lean into this technology as much as possible see where AI works see where it doesn't experiment iterate and
55:39then at the same time really embrace being just being people being being humans understanding each other understanding each other's problems and if it's in the middle maybe it's not worth spending as much time on.
55:56Yeah, I think there's trying to keep our humanity is important. The other thing that we're all seeing, I think, is that um there's no question about the, you know, the productivity. We talked about the the ability to get more work done, but also we're really starting to see some second order effects of sort of long-term exhaustion. And so, uh, it's just such an odd situation that we're living through so much disruption right now.
56:26Absolutely. Well, Marty, we're coming up to the end of our time here. Thank you so much uh for joining me here today.
56:33Thank you to everyone in the crowd for the amazing questions, for joining us as well. Um, if you are interested in interpret, we can happy to help you understand your customers a little bit better. But I think the one thing I'll leave everyone with, I think is where we started is if you take one thing away from this conversation, go talk to your customers. Go talk to a customer today. Uh that would be my personal advice.
56:58Awesome. Thank you, Marty.
57:01Thanks very much. Hope this was useful to everybody.