0:00Teresa is a legend in the product management space. She has been teaching something called continuous discovery habits for a really long time. I thought I knew what it is, but I can tell you that no engineers don't really know what that means.
0:13I really got frustrated early in my career with just spending so much time building the wrong stuff. There's nothing more demoralizing than to like put everything you have into an idea and then watch the world not care about it.
0:25The biggest mistake founders [music] make is they go, "Hey, what do you think of my idea?" But when you're interviewing customers, don't ask them about your idea. Ask them to tell you about a time when they experienced the problem your idea was [music] designed to solve.
0:38This is an opportunity solution tree.
0:40It's like a structured approach that kind of helps you guide your thinking.
0:44Our brain was designed to keep our heart beating and our lungs breathing and our body alive. It's not optimized for good critical thinking. Confirmation bias is real. It's not conscious is your unconscious brain filters out the disisconfirming evidence [music] before you even notice it. I think we should just build a map where we show people where alumni live.
1:03That's so typical. That is every engineering meeting I've ever had.
1:07Holy crap. Claude is already better than the average product team. Now I have goosebumps [music] because I can give you an AI that raised the floor and if you work with my AI that will raise the ceiling.
1:17So I want to introduce you properly because a lot of my audience is engineers, data scientists, machine learning folks and Teresa who we have on today is a legend in the product management space. So a lot of people who are product managers have heard of Teresa. She has been teaching something called continuous discovery habits for a really long time. She started writing about it in 2011. like the core of what
1:44you talk about is how do you talk to customers and how do you synthesize the information from those customers to inform what you're building. Now, the thing that is really interesting is, okay, every engineer and machine learning person, data scientist, we've all heard talk to your customers. And everyone thinks they know what that is.
2:05They're like, "Oh, you just pick up the phone and talk to your customer." To be honest, I thought I knew what it is, but I can tell you that no engineers don't really know what that means. Like, most people don't know what that means. Most engineers, they'll like phone up their friend and be like, "Hey, I'm working on this new idea. Like, what do you think?
2:20Do you think it's a good idea?" Be like, "Yeah, I think it's a good idea." Yeah.
2:22And some people some engineers have read some books. I think amongst engineering circles there are some books like the mom test people have read and that only kind of scratches the surface like I think it focuses a lot on interviewing and then like you take it to a different level where you put a lot of structure behind it. Talk about like how to actually do it, how do I do it on a continuous basis. What I want people to get out of this interview today is first I would like to see if you can describe
2:51like what you teach, why it's important, and then also because you are a teacher, you also have been building an AI product that helps you with the teaching. You've been transforming your teaching and I think that that's super interesting. And then like by the end I want to also help people understand why talking to customers is really important like why talking to them in in a structured way in a regular cadence is important and even like how you might learn about it as an engineer but also
3:19like how it might tie into eval because you like do eval what you're doing and that's one of the things I want to explore with you is okay how does this all tie together because you know you have blended a lot of things together in your practice. So, you know, just to start off with, because a lot of people haven't heard of continuous discovery, and I don't really think they know what it means to talk to your customers, can you talk a little bit about your body of work and what you're trying to fix with it? Like what is your mission? Like,
3:49what are you trying to do? What problem are you trying to solve?
3:51Yeah. So, I think this will resonate a lot with engineers. I really got frustrated early in my career with just spending so much time building the wrong stuff, right? just so many companies an executive has an idea, it gets added to the road map. Product managers, designers, engineers spend a ton of time having to build it. And then it's only after we release it, we realize nobody actually cared. And I think I'm a maker at heart. Like I just really like making things. And there's nothing more demoralizing than to like put everything
4:19you have into an idea and then watch the world not care about it. And I think discovery is not a term I came up with.
4:26This guy Marty came up with it. And he's really trying to distinguish between the work we do when we're deciding what to build, that's discovery, versus the work that we do when we're actually building it, that's delivery. And I think those terms are really helpful because a lot of business people think all of the work is in delivery and that having the idea is easy. But they forget that like most of their ideas don't work. And it's not that they forget that, it's that they never bother to measure impact. So they actually never learn their ideas didn't
4:56work. They just do a bunch of stuff. They fall short of their expectations. They're not sure why it didn't work. They're not sure what worked didn't work. And really, I think good discovery is just building in fast feedback loops. And this is the thing that I would say applies across all the discovery habits.
5:11It's why I call eval a missing discovery habit. Discovery is just how do we build in fast feedback loops so we learn we're on the right track sooner than later.
5:21Like it's that simple. And you know in your class Hamill you and Shrea presented it as like all you're teaching is the scientific method. That is literally all I teach is okay so we have an idea of what we should build. Okay great that's a hypothesis. Is it the right thing to build? How do we build in fast feedback loops along the way so that we don't go too far down the wrong road and we learn much sooner whether we're building the right thing or not?
5:45There's lots of ways to do this. We have qualitative methods. We have quantitative methods. But if we think about it as like at its heart, it's the scientific method. If you were like, "Okay, I'm gonna go do this scientific method to get feedback on my startup idea." You're not going to call your buddy and be like, "Here's my idea. Is it good?" That's a terrible experiment to test that hypothesis. And like most people intuitively know that when we frame it as an experiment, when we frame it as the scientific method, when we frame it as we're trying to get
6:13reliable, valid feedback on a hypothesis. And so all of the discovery habits are really designed to teach teams how to build reliable feedback loops.
6:21Can we make it a little bit concrete?
6:23Like how do you know that you may have this problem of like you're not doing discovery? You're not talking to your customers correctly. A lot of people again they think they're talking to customers. They're not. Most people are not. I only realize that after reading your stuff deeply. I'm like, "Oh, I'm not doing it correctly." It takes a lot of practice, a lot of skill. Yeah. Can you tell me like what does it look like when it's not going the right way?
6:45Yeah. So, one of the things I tell people is if you're not constantly throwing away ideas, if you're not constantly evolving your ideas, you're probably not doing good discovery. And this is really tough because you can do all the right activities, but if you don't come to it with like intellectual curiosity, more importantly, intellectual honesty about your idea may not be very good. This is why I think using the scientific method as an analogy is exactly right. We all have read papers where we look for those of us that read papers where we look at the methods and we're like wow this is sort
7:15of a crummy proc like sure you're a scientist but you're not really doing the scientific method. You started with a hypothesis you designed your experiment to confirm that hypothesis and like if you have any critical thinking skills you can see that it turns out this is what founders do and we do it for a reason. Like founders are so already have to be crazy to be like I'm going to create a new thing in the world that has never existed that already creates that already requires like such a leap of faith of courage that that strength that gives you that
7:44leap of faith of courage is actually a blind spot because it's setting you up that when you go to test your ideas you're testing them from this really confident point of view. Whereas I think to do science well we have to have a lot of doubt. We have to set up our experiments to be disisconfirming, not confirming. And so there's this mindset that I think is really hard for humans.
8:04Like we feel so right, we feel so certain. And I think good discovery is really like, okay, I have to detach myself from my idea and really design my feedback loops to be disconfirming. And this is hard. And I think the closer we get to our work, the harder it is. And it's really easy to fool yourself because when you say detach yourself from your idea. Most people will again they will call up their friend and say what do you think about this idea? They will ask a customer like hey they will
8:34lead with a demo. They say hey like look at this idea and they will get like a false confirmation from those conversations. They're not going about it the right way. You have to actually like dig really hard to get to the truth.
8:45I think about discovery is there's two parts to it. There's the generative part. How do I know I'm solving the right problems? And then there's the going after the right problems. And then there's the evaluative part. How do I know my solution actually solves those problems? Let's say you start with an idea. If you want to get feedback on your idea, the biggest mistake founders make is they go, "Hey, what do you think of my idea?" First of all, not everybody is your target customer. Not everybody experiences the problem your solution was designed to solve. So if you're asking people that are not your target customer, that don't experience the problem your solution was designed to
9:15solve, your feedback is garbage. Who cares what so- and so thinks if they don't have the need? And then if you are talking to someone where you know they have the need or you think they have the need. That's actually the first thing you have to test. You don't want to say do you have this need that's a bad test. You want to ask them to tell you about an experience when they had that need.
9:34You're listening for is this real? I think you have like I'll give an example. Let's say I want to start a startup and I want to help podcasters grow their audience. Right? I don't know what I'm going to build. I don't have a solution yet. Which by the way, this is the right way to be a founder in my view is don't just start with a shiny idea because we don't know who needs that idea. If you've heard the phrase, you're a solution looking for a problem. We want to avoid that. That's like the slowest way to find a successful product. Let's say I just love podcasts.
10:03I see a lot of new podcasters flounder. I want to help them grow their audience.
10:07I don't know what I'm going to build. I don't know why they can't grow their audience. I don't know very much about them yet, but that's my goal. I'm going to go interview some podcasters. I'm not going to say, "Here's my idea. What do you think?" I'm just going to ask them to tell me about their podcast, what they're doing to grow their audience today, what's working, what's not working. I'm also going to interview really successful podcasters to learn how they grew their audience. Because when we find people that are successful at the problem we're trying to solve, that's really good inspiration for how
10:36we might solve it for the people who haven't found that solution. And then as I learn about those customers, I'm listening for where do they have unmet needs, where are their pain points we can address, where do they like aspire to things, where do they desire things that just don't exist yet? And that's going to help me understand where can I intervene positively. And it's only once I have a really rich understanding of like I call it the opportunity space.
11:01It's just unmet needs like unadressed pain points, desires, some people call them latent needs. the jobs to be done framework falls into this category, right? It's just like what do people need that isn't being met and we want to start there and really have a rich understanding of who our humans that we're designing for are. And then only then do we want to start to look at like, okay, well, I heard this need.
11:22It's really hard to find your first thousand listeners. Let's focus on that.
11:26Well, what are really experienced podcasters doing? Maybe they're coming up with like a content strategy to go viral on social media. And we can start to look at like I want to figure out a social media strategy to promote my podcast. That's an opportunity. So then we can look at like okay what's some potential solutions for that opportunity. And what a lot of founders do is they just build something for themselves. They jump to their first solution and they hope there's other people like them. Now we have lots of success stories where founders built
11:55something for themselves and it worked. And that's because there were other people like them. But we have a lot of stories of founders that start with an idea building for themselves and they it went nowhere. We don't hear those stories because they're not famous.
12:07And every single time anybody has brought up a startup thing to me in my personal experience always starts with idea. Hey Hamill, I have an idea. And I think it's every engineer's experience almost universally. They're like you feel like it needs to start with an idea. And I'm glad you called that out.
12:23And you mentioned one thing that I think is deceivingly it sounds simple but it's it's very difficult. And you said you want to ask for behaviors and stories.
12:32And so can you unpack that a little bit?
12:34What is the distinction between behaviors and stories and like how people usually ask questions?
12:38Let me first go back to this idea. I'm going to start with an idea. I actually think this is human nature. Our brain thinks in solutions because they're concrete. They're really specific.
12:46They're tangible, right? Whereas problems are more abstract and our brains want to get concrete as fast as possible. If you're a founder and you have an idea, that's fine. That's a perfectly good place to start. But when you're interviewing customers, don't ask them about your idea. Ask them to tell you about a time when they experienced the problem your idea was designed to solve. So if my idea is to help you with your social media strategy for your podcast, I want to ask tell me about the last episode you promoted on social media. And I want you to walk me through
13:15everything that you did. Now the reason why I'm doing this, there's a very subtle difference here. I could say, Hamill, how do you promote your podcast on social media? That's a general question. When I ask you a general question, I invoke what Conaman calls system one thinking. It's your fast lazy brain. Now, really smart people hate hearing this, but it turns out our brain was not designed for all the mental activity that we love and enjoy. Our brain was designed to keep our heart beating and our lungs breathing and our body alive, right? And this mental activity is sort of a side effect of
13:45that. And so, the brain is wired to optimize for life, for keeping us living. It's not optimized for good critical thinking. And so oftentimes the brain will generate system one and system two are analogies. I just want to be clear like it's not like there's a part of your brain that is system one and system two, but they're really helpful analogies. System one is your fast thinking, fast response brain. I like to think about it as the lazy brain. It's like your brain is just generating an answer so it can go back
14:14to keeping your heart beating. And the problem with system one answers is that they don't necessarily reflect your actual behavior. So, if I say, "What do you do to promote your podcast episodes?" You're gonna get a fast answer. Your brain is going to tell you something that you believe is true. But what you tell me won't actually match what you actually do. It might match what you wish you did. It might match what you do some of the time. It might even match what you did like the most recent time, some of it, but you're not
14:42taking the time to think through the whole instance. And so, there's a lot of context missing. There's a lot of detail missing. And it when I say reliability, it doesn't always reflect what you actually do. Do you have a favorite example of this?
14:55Yeah, one of my favorites is I was in a workshop where I asked a woman, I do this to demonstrate it and it works every time. I ask a woman, tell me about what criteria you use to buy a pair of jeans. And she said fit, price, and she has a favorite brand. And I said, okay, great. Tell me about the last time you bought a pair of jeans. And she said, I bought them on Amazon. And I sort of chuckled. I was like, oh, you bought them on Amazon. How did you know if they fit? And she said, 'Well, they're a brand I buy all the time. And I said, 'Okay, have you ever bought the same brand and they fit differently? And she
15:24said, 'Yes.' And I said, 'Well, why were you willing to buy them on Amazon this time? And she said, because they were on sale. Okay, so what actually came out of her last purchase? She has competing criteria. Fit does matter to her. Her favorite brand does matter to her, but she'll compromise fit for a better deal.
15:42And everybody knows like returns on Amazon aren't super they're easier now that Amazon owns Whole Foods but at that time they were not easy, right? And so this is very common. Another one I often ask people is if you're super athletic like let's like runners fall into this category. I'll ask tell me about how often you run and they'll say seven days a week and then I'll be like okay tell me about last week and they'll inevitably they'll have run like runners are crazy so a lot of them do run every day of the week but if we ask about last week maybe it was only five of the seven
16:09days and what they forget is like their kid was sick one morning and they got to school late or they had a meeting early in the morning and it over overlapped their run. Like there's all these exceptions. Our brain thinks exceptions are outliers, but in reality, exceptions happen all the time.
16:27And what I learned from you is there is a large gulf between what people say they want and then what they do. Like how they invest their time and where they're actually problems that they're actually trying to solve.
16:38Even more than that, I would say there's a large gulf between what people think they do and what they actually do. So, not even what they want, just we're just talking about behavior, right? If I ask you, "What do you like to watch on Netflix?" Your brain is going to generate a really fast answer. That answer may or may not be different the same as when I ask you, "Tell me about the last thing you watched." When we're designing products, we want to design for actual behavior. Now, there's some exceptions. If I own a gym, I want to design for your aspirational self because I don't want you to come to the
17:07gym. I just want you to pay for a subscription. That's a little different, right? For most of us, we want to design for what our customers actually do. We can't just go call our buddy and say, "What do you think of my idea?" We have to learn techniques for how do we get reliable feedback to learn what they actually do. The simplest way to do this is to ask for specific stories about past behavior. So, tell me about a specific time when and then even this is hard because if I say, "Hey, Hamill, tell me about the last time you watched TV." Your response is going to be really short. You're going to be like, "I
17:36watched a movie the other day." and I have to like do the work of situating you back in that moment and helping you tell me the whole story from beginning to end. So, it's actually a skill to collect a good rich story that requires practice. But the benefit of practicing this skill is we start to get a really reliable feedback loop when I'm trying to learn does my customer experience this problem? How do they experience it?
18:01What environment do they experience it in? Who are they with when they experience it? I can learn all of that by just collecting a rich story.
18:09That makes a lot of sense. It's almost like if you talk to customers and if you talk to them in the wrong way, it's going to lead you even more astray. So you think like, oh, I'm doing this thing. I'm talking to customers. I'm doing what people told me to do. But if you do it the wrong way, it's just going to lead you even further potentially away from what you should be doing.
18:26We've all seen everybody like cherrypick data for their argument, right? We're confirming. We're coming in with a confirming mindset. I'm going to see all the evidence that supports my idea. This is really common. I mean, this is one of the biases. Confirmation bias is real. It's not conscious. That's the thing people misunderstand about confirmation bias. They think that like people blindly ignore disisconfirming evidence.
18:49That's not what happens. What happens is your unconscious brain filters out the disisconfirming evidence before you even notice it. It's not a conscious act. So to overcome confirmation bias, we have to design our feedback loops to be disisconfirming from the beginning. And I think this is what people get wrong.
19:09They just think like, oh, just go get some feedback. Yep. The problem with that, like you're doing a good activity of getting feedback, but your brain is literally filtering out all of the evidence that there's something wrong with your idea, and so you're getting no value from that.
19:21The next thing I want to ask you is, okay, so you have this word continuous in front of discovery. So what is the continuous aspect of this? Like why is a continuous thing important and what does that mean?
19:32Most digital products are never done, right? So it used to be we made physical products. We obviously still make physical products, but like if you work at a consumer package goods company and your product is shampoo, you might come out with a new shampoo every year or two. This is how software used to work.
19:47We used to stamp software on the CDs and put them in a shrink rack box and ship them to fries, right? Like but that's not how software works anymore. Thanks to the internet, software is continuous.
19:56We are continuously iterating and improving our products. It's not like Netflix is going to decide one day that hey, we're done. We're just going to stop building this thing for a while. In fact, we see companies do that. Air Table stopped building their product for a while to build a new AI product and it got real bad and then they just sold for a fire sale. So, don't do that. It's a very relevant right now true story, right? Like digital products to stay healthy have to always be evolving and there's a reason for that. Like the competitive landscape is always
20:24changing. Technology is always evolving. There's new stuff just now possible. As we address customer needs, new needs emerge. We often create new needs with the stuff that we're creating. And so our products have to evolve as the market and our customers evolve around it. And so we continuously build. We have to continuously make good decisions about what to build.
20:44I feel like there's a lot that goes into this continuous part. So there's like how do you interview or and like who do you interview? How do you schedule those interviews? and your writing discusses a lot of that. Can you talk about a little bit what do you have to learn in order to like do this properly at a high level?
21:00The first thing I'll say is there's more than just interviewing. So interviewing is the generative feedback loop. Am I solving the right problems? Assumption testing is our evaluative feedback loop.
21:10So am I building the right solution? And a lot of founders actually could just start with assumption testing. They could the idea behind assumption testing is you take your solution. I like to use story mapping to figure out uh what my underlying assumptions are. So in a story map, we're mapping out what does a customer have to do to get value from this solution. So literally step by step, what are you expecting your customer to do for them to get value from using your product? And then in each of those steps, we can start to generate the underlying assumptions. Do
21:38they want to take that step? So that's a desiraability assumption. Are they willing to take that step? Here's a really people think wanting and willingness are the same, but I'll give you an example where they're clearly different. If I'm in a foreign country and I need cash, I want to put my bank card into the ATM machine and get cash out. But if it looks really sketchy, I am not willing to do that. So your customers have to want to do each step that's required of them. They have to be willing to do each step that's required of them. Willingness is often about
22:07trust, but not always. Sometimes it's you're asking them to do a lot and they want to do it, but they're they're not willing to. It's too much work. every step we have usability assumptions like can they find it do they understand what's being asked of them are they able to do what's being asked of them that brings into a lot of our accessibility concerns we can look at can we do what's required at each step that gets into some of our feasibility assumptions like is it possible to deliver what we need to do at each of these steps sometimes our solutions require partnering with
22:36other people that can bring up viability solutions like can we partner with them in a way where the economics work for both parties that can bring up feas feasibility assumptions like can we even integrate? Do we have the APIs needed to integrate? Well, viability is also like willingness to pay is this creating enough value that the customer is going to pull out their wallet and pay for it.
22:57There's also ethical assumptions. So, what data are we collecting? Are we being transparent about that? Who are we sharing that data with? Uh there's sustainability assumptions about resources being used and waste. There's even ethical assumptions around who we choose to target and who we might be unintentionally leaving out. So, a big one here is most B2B companies focus on large enterprises because that's where the revenue is. But I don't know about you, you live in Portland. I live in Bent. Oregon is very anti- big box everything. We like our mom and pop
23:25businesses, but who's creating software for them? Not very many people, right?
23:29There's some exceptions, but not very many people. It's not where the revenue is. So, there's sort of some ethical implications of how we build things. And so, one of the things I encourage teams to do is to really start with your idea and break it down into these underlying assumptions. get really specific because these assumptions are a lot easier to test than your whole idea. When we try to test a whole idea, we very quickly fall into the trap of let's build everything and see what happens. Whereas if I get really specific on an assumption, I can collect data on an assumption really quickly. Usually like in a day or two.
23:57And I'm glad you brought this up like okay so someone listening they might be slightly overwhelmed like okay Teresa's talking about all these different things. Think about how do I keep track of it? One of the key things that you've created that help people go through this is a opportunity solution tree. It's like a structured approach that kind of helps you go through it without getting overwhelmed and like guides your thinking. Can you talk about what that is?
24:19Yeah, I actually want to share a story about what inspired it because it was really driven by an engineer that I worked with. So I was working at a startup. We built communities for university alumni associations and at this time we had over 300 universities in the US. we were like the dominant player in this market. And so if you have ever participated in your university alumni community, especially around 2003 to 2012, it was probably our product that drove that. And what's interesting about alumni is they have a
24:49lot of expectations from their university about what they want, but they don't really engage. So they want to receive a lot, but they don't give a lot. And we had built this system where alumni could post what they needed, but it was overwhelming people because people asked for a lot of things and nobody wanted to give. And so we were like trying to figure out like how do we get people to give?
25:08What kinds of things do alumni want out of curiosity? I'm an alumni of several Yeah. help finding jobs is the biggest one. So like I want to use my university network to like get a job at Google or even things like I have a dining room table I want to sell. I'd love to sell it to an alum of my alma mater.
25:25Okay. Hey, I wouldn't have guessed that.
25:27This is all the kinds of things we see on Facebook. Like Facebook has Facebook marketplace. I don't know if they do anything with jobs, but like some people, especially some schools have really strong affinities and then people really, you've probably experienced this. You meet someone in a bar that went to your school and you instantly have a lot in common. There's a lot of affinity with the people we went to school with, even if it's across decades. So there's we had this like imbalance in our community. And what it led to was I framed it as like in my head as the product manager I was framing it as we have to find the right
25:56balance between giving and receiving. So I was like how do we get more people to give? But our ultimate outcome was let's increase engagement. So what the business needed was alumni to engage more. I had identified an opportunity of like let's rebalance this giving and and receiving. We're sitting in a room with our engineers brainstorming and one of our engineers goes, "Hey, the Google Maps API just came out and I've been playing with it. It's so cool." And I'm like, "Okay, I'm kind of rolling my eyes already because what does Google Maps
26:25have to do with getting more people to give?" And he goes, "I think we should just build a map where we show people where alumni live." And I was like, "How does that sound?"
26:32That's so typical. That is every engineering meeting I've ever had ever. Ever. Right.
26:38100%. And it was driving me nuts cuz I was like, "No, we're trying to get alumni to give. How does that get alumni to give?" And he was like, "No, we're trying to drive engagement and this would be a great way to engage people."
26:50Okay, he wasn't wrong. We just weren't having a conversation at the same scope and we were driving each other nuts to the point that literally this engineer stayed up all night implementing a map of alumni because that's how strongly he felt about it. And I was pissed off because then the next day he couldn't work at all on the sprint he was supposed to be working on. And like I've seen this problem time and time and time again where like we're all humans in a room trying to collectively solve a problem but we're not working from the same context and we're not working
27:19within the same scope. And so if the opportunity solution tree had existed at that time that engineer was right our outcome was increase alumni engagement. But in my brain I had identified an opportunity and I thought it was the most important opportunity to solve.
27:33There were other opportunities like that guy Seth was trying to solve this opportunity of like I want to know who lives near me which by the way is a perfectly valid opportunity to go after but we we skipped the decision of which opportunity is most important each made assumptions about how we should solve it and we argued in the solution space no it's very classic like hey it's a solutions first thinking like I want to use this Google maps API I think it's cool let me find a reason to justify it and here's the thing like you will never
28:03come to consensus when you're arguing in the solution space because you skipped the most important question. Which of these problems is most important to our customer? To be honest, to this day, I still don't know. Would alumni have cared more about who lives near them or would they have cared more about like getting what they actually wanted when they posted a need? I don't know. I have no idea how to answer that question because we weren't even exploring that opportunity space. We were all of us were just jumping to solutions and like I had been talking to customers. So I I
28:32had my own intuitive sense of what opportunities mattered and I had the hubris to think that like I was talking to customers so I knew more than Seth.
28:40But the thing is Seth is an alum and he actually wanted to know which alum lived near him and he wasn't wrong. Like I can't tell you to this day I was right and he was wrong. He could have been right. The problem in this story is that we weren't all aligned. We hadn't collected all the opportunities we might solve, put them on a piece of paper in front of us and had the more important discussion. What do we know about our customers? Which of these is most important to them? Which one should we solve? And agree on that first before
29:08you move into generating solutions.
29:10So, how did you solve that?
29:11I didn't solve it at that point. I mean, Seth solved it by just staying up all night and building what he wanted to build. I got upset and continued to plan our sprint around what I wanted to build. So, we didn't really solve it.
29:21You just you've seen this probably on a million teams right like I would say and engineers you know there's another insidious problem which is there's a currency around like which technologies do you know how much complexity you can you can tame sometimes like engineers are incentivized to create new technologies and new complexity for their own purposes and so it's a friction clear to me like okay from a business perspective you want to start with like what outcome is most important especially now with AI
29:50where like you can just build anything you want.
29:52Funny, at the time, Seth drove me nuts.
29:54I love Seth, but and actually with time, I can look back on it and be like, actually, I like I was just as wrong as he was. The problem was not who was right or wrong. The problem was the scope of the conversation we were having. And where Seth Seth was 100% right. I'll tell you, I've been trying to get engineers to engage in discovery for 20 years. And I actually love that with AI, they are now starting to engage with discovery because they're realiz their companies are asking them to be more product minded. And so, like, welcome to the club. Thank god we're finally getting there because what
30:23engineers bring that most product managers cannot bring is a knowledge of what's just now possible. I have a podcast called just now possible for this reason. Like a lot of innovation is driven by technology evolving and new things becoming just now possible. Look at AI. Like we're living this every second of every day right now and our engineers are the best people to understand that. That's exactly what Seth was doing. A new technology came
30:51out and this Google Maps is so like pedestrian now. But at the time it was revolutionary to be able to put pins on a map based on just data. And he looked at that and he got immediately excited and he saw a customer problem he could solve with that technology. I was a product manager that didn't even know that technology existed. So like I never would have even considered that customer problem because how in the world would we solve it? a technology didn't exist a week before. And so good discovery
31:20happens. Engineers with knowledge of what just now possible and product managers with knowledge of business needs and designers with knowledge of what's usable and desirable and delightful like all come together and collaborate on what's the best thing to build given all of our expertise. But it so rarely happens in business.
31:39So how was the opportunity solution tree like born out of this? That story I told you I think happened in I can tell you it happened before 2011. I left that company in 2011. So I'm not sure it could have been like 2007, 2008, 2009 like somewhere before 2011. I thought about it for a long long time. I was like, what is going wrong in this situation? Like I framed it as like I have a hypothesis and I'm working in the scope of this hypothesis, but I haven't done a good job of communicating this
32:08hypothesis to the team and they're working in a completely orthogonal scope. And so we're not finding alignment. But I could not figure out like what was in my head that I wasn't communicating. I just thought it was obvious like we have this system where we let people ask for things and other people give. It's unbalanced. Like clearly we have to help people give.
32:27like I just thought it was obvious. I couldn't see the context that I had that the rest of the engineers didn't have. And so it took me almost a decade to like go through lots of iterations. And I start I had started coaching in 2011.
32:41So lots of like seeing this on lots of other teams and collecting a lot of data points to realize like okay the first thing is I have a lot of customer knowledge that engineers don't have. How do we get engineers to have more customer knowledge? Let's invite them to the interviews. Like we can't just do interviews on our own. That was like one thing is like some of the context I have is just learning from customers. Some of the context was business context. How do we get engineers up to speed on business context? And then the thing that really was the catalyst was I read Andrew Ericson's book peak. So Andrew Ericson
33:10is a researcher that studies the difference between experts and noviceses. And he's Malcolm Gladwell wrote a book where he coined the term the 10,000 hour rule. The way Malcolm Gladwell wrote about it kind of bastardizes the research. But the 10,000 hour rule is based on Andrew Ericson's research. What he argues, one of the things he argues is that experts just have many, many, many, many hours of experience. That actually wasn't the most interesting part of his research.
33:37The more interesting part of his research was experts have mental representations that noviceses haven't developed. And the easiest way to understand this is if you look at a chessboard and you're not a chess player, it looks like a random mix of pieces. But if you're a grandmaster chess player and you look at a chessboard, you see all the patterns of how they got to that point and you can see the moves to like win the game. It's you have a structured representation of a chessboard that a novice doesn't have.
34:05And you probably have experienced this in your own life. Like think about like eval mental representation of types of evals, ways to do error analysis, like ways to set up data that like a novice is just foreign to them.
34:18Yeah. I feel like whenever I look at data, I can smell what's wrong within.
34:22Yeah, exactly. So, I read this idea of like, oh, it got me thinking, what's the mental representation I have in my head that a novice doesn't have. And what came out of that was the opportunity solution tree of like, okay, yes, the outcome matters, but there's this whole missing layer we never talk about, which is the opportunity space. And the opportunity space is where we're customer focused. We're not just using technology to build cool things. We're bu using technology to solve our customers problems. And then we need to make sure everything we build is tightly
34:51coupled to a need. And then as we do that, we start to impact our outcome. And by the way, we're only looking at needs that we think can drive our outcome. Like the needs, the opportunity space is infinite. And a lot of teams get stuck solving customer needs that don't create business value. And then that's also a problem.
35:09And I think okay, so this opportunity solution tree structure you created, I believe is super popular. Like when I did research on the internet, there's like a cottage industry of even software of like different software that will like help you build these opportunity solution trees that you can like push your data into these like interviews and there there's actually different kinds of software. It seems like it taken off.
35:31It's it's become very popular way to think about the problem. So maybe as a little bit of a pivot, it seems like you practice what you preach a lot and so like you know people have been using these tools. I know you've been thinking about, you write a lot about how people are trying to use AI nowadays to help them with this process and there's a tension between having the AI do everything for you and then having the AI help you. You're trying to teach people. So, first of all, like how did
36:01you identify the opportunity this angle of how you're transforming what you teach and like using the kind of the methods that you teach to discover that?
36:10Yeah. So, I've been teaching the discovery habits to teams for a very long time. I think this is my 15th year doing it formally as my own business and then obviously several years before that coaching and developing my own product teams as startups. I what's hard like to do discovery well is genuinely hard in the same way that to do science well is genuinely hard. It's all the mindset stuff. It's intellectual curiosity. It's intellectual honesty. It's continuing to have rigor. What's different between
36:40science and discovery is discovery happens in an industry that wants to move at a ridiculously fast pace. Everything in the organization is telling you don't do that, just build. And so we have this discipline that takes time, that takes rigor, and we're trying to do it in an environment that doesn't support any of that whatsoever.
37:00And there's these competing odds. is like your boss is very certain about what you should build and wants to just tell you to build things and you're trying to like work this process and you really want to do the right thing. You want to learn from your customers. You want to build something that's going to be a good fit, but it's Friday and on Monday the engineers need a new sprint full of user stories and you don't have them yet, right? So like it's genuinely hard to do good discovery in industry trying to adopt some of the like ethos and principles of like good science but
37:28good science moves extraordinarily slow and like we don't have that luxury in industry we have to move fast and I want to be clear people might hear you say good science and they just go straight to quantitative methods because I know the engineers watching this be like oh we need data analysis give me the dashboards whatever and like no know there's a lot of qualitative as like dimensions to what you're talking measuring instrumenting your product and getting good behavioral analytics is a discovery habit right that is a feedback
37:57loop running AB tests this is like what most teams think they would like I'll just build both and we'll AB test it okay you just did twice the work like let's learn before we do all the work whether we're on the right track because when we build something and then learn it was wrong what we lose is all the opportunity cost of what we could have built instead and that's too expensive like we don't want to spend that every time. So, a lot of the discovery habits are designed to help you learn as early as possible so we don't waste time on the wrong stuff. We have both quantitative habits and qualitative
38:27habits. Both are important. Even in science, we have quantitative methods and qualitative methods. We can get into this whole like epistemic philosophy of how do we know things? They both bring something to the table. It's actually really important that we embrace qualitative as well as quantitative because when we just focus on the quantitative, there's a really good book about this. It's called the score. It talks about it's a little bit Gottman's law like once you come up with a measure for something, it stops measuring the thing you're trying to measure. And it's just this whole idea of like metrics can
38:56be gamed and we optimize for the metric instead of the thing we are trying to measure. And so I actually really I don't fall into one camp or the other. I think we have to embrace both equally.
39:05Is it similar to goodart's law?
39:07Goodart's law. That's what I meant. And so like we have to balance both. Obviously quantitative gets us data at scale. Qualitative gets us all the rich context that we're never going to get from quantitative.
39:17How did you using your continuous discovery habits identify like the exact way that you should transform your education to have AI in it? Because there's so many things that you could have just made a chatbot. You could have some kind of gadget thing that does you know you could have gone in 10 hundreds thousands different directions. So like take me through the process like okay how did you focus on and we're going to talk about the solution but just the opportunity that you found.
39:44I've been teaching this for a very long time and I think it's taught me how hard it is to do discovery in industry because there's so many competing factors. I also have learned how hard it is for an individual to learn to work this way. There's a mindset that's required. there's skill that's required and like it just takes a lot of effort to develop that mindset to develop that skill. So just take interviewing as one skill that's required. I have to develop
40:12a very curious mindset. I have to detach myself from my own ego, forget what I'm trying to learn and be totally present with you and hearing your story. That's like a mindset that's hard to develop.
40:24Then I have to develop the skill of how do I situate you in the moment, help you recall your memory, help you walk through your story step by step. That takes a lot of practice. So through years of teaching, I see that people get better at it, but they don't get great at it. Like it's only the exceptional students that put in the time and energy to develop the skill and to really change their mindset. And I'll tell you, I've been doing this for 15 years. I can name like 10 people. They I know all their names. I've stayed in touch with
40:53them. and because they truly were exceptional students, but it's rare. And I'm not blaming the individual like they're working in an environment where this is genuinely hard.
41:01I feel the same way about evals, by the way.
41:03Yeah. It's just this is it's all because the scientific method is hard for humans. It's hard for humans to do the scientific method well. Like it's so counter to our biology. And so I care a lot about learning outcomes. Another thing that comes from Anders Ericson's work is this idea of deliberate practice. The way we build a skill is we have to practice the skill in small pieces with expert feedback. And so all of our courses are designed around deliberate practice. So like in our interviewing course, you practice interviewing live in class and the
41:33instructor gives you feedback and your peers give you feedback. Well, your peers aren't very good at feedback because they're giving you they're new.
41:39And so the first when I saw what was possible with AI, the first thing I wanted to do was start to experiment with can AI give feedback better than peers? And if AI gives feedback better than peers, maybe that even helps your peers give you better feedback and everybody gets better. And so that led to my first AI product, which was my interview coach, where you submit your interview transcript and it gives you very personalized, detailed feedback on how well you conducted your interview.
42:02And then what this unlocked was like, okay, well, now that I can give you personalized feedback, I can make this course accessible to way more people because I can offer it on demand. So before we used to have live cohort-based courses, now it's on demand. You can take it from anywhere in the world. you can find a friend to interview. You record your interview, you send it to the coach. And it turns out some people won't even do that because they're afraid of being a beginner with somebody they know. And so the another product I'm working on right now is the AI participant so that you can practice with the AI and not have to be embarrassed that you're a total beginner
42:31and then submit that transcript to the coach and get feedback. And then you can what this unlocks is it unlocks infinite practice with expert feedback.
42:38What's the difference between the AI coach and the AI participant?
42:41So the AI participant will be the person you're interviewing. The AI coach is what grades your interview because like what would happen? So like the dirty secret of on demand courses is lots of people buy them and almost nobody completes them. This is why people move to cohort-based courses because like you have to come to class. But cohort doesn't really solve it. You probably see this in your course. Like some people really engage and do the work and a lot of people just they want edutainment. And so I like I'm trying to get people to build skill through a lot of practice. But there's all these barriers to why they don't want to
43:10practice. They're embarrassed to practice. They don't whatever. And so I thought like, okay, well, you just practice and you get private feedback that will help with the embarrassment.
43:18But then there's like this embarrassment of I don't want to interview my friend or I don't want to interview my colleague because I'm a total beginner and I don't want to be a beginner in front of another person. So now I'm using AI to be your interview participant. So that you're in the privacy of your own home. You can practice as much as you want. Nobody sees your interview practice. Nobody sees your grade, just you. And what this unlocks is it just unlocks really fast feedback cycles. And that's how we build skill. So that was sort of my first foray into like interesting. This is like an interesting way to use this in
43:47class. It has this extra benefit now of I can see learning outcomes. So I see the scores you get from the interview coach and I can see if your scores go up over your rounds of practice.
43:56There's many different layers to the whole process. There's the interviewing and then there's the synthesizing interviews. We'll talk about the synthesizing interviews in a little bit cuz that that has some very interesting challenges. But like, okay, when it comes to this interviewing that you just talked about, this interviewing practice, the million-dollar question, how did you know it was good? How did you get confidence that you feel like is good enough?
44:19I mean, this is what led me to your emails class was trying to figure out, is my interview coach any good? What's nice is that my courses at this time, at the time that I built the interview coach, we were still cohort-based. We limit our cohorts to 50 students, so they're not giant large-scale classes.
44:34This was a scale that I literally could read every single coach response and just get a sense for like, is this what I would have said? And in the earliest days, if the coach gave a student feedback I disagreed with, the student got an email from me saying, "Pay attention to this part. You can disregard this part." In round one, like in April of last year, when I first released this to the first class, I just QA everything, every single response. I knew that wasn't sustainable over time, which is why I started to learn about evals. Ended up taking your class and then was able to do error analysis, see
45:03where the coach made mistakes and start to put evals in place. And this like resonated with me so strongly because eval are just another feedback loop. So to me, it just felt like, oh, this is right in line with what I teach. I feel like I had the mindset. I could bring the like intellectual curiosity and like go deep on the analysis because it's very similar to you guys even talk it about it as axial coding which is a qualitative analysis tool. So like I already had those skills and so it was really easy for me to just dive deep and
45:32like be rigorous about where does it make mistakes, how do I measure that and then how do I systematically reduce those errors.
45:38What do the emails what do they look like?
45:40There's some specific things that we teach. So a story-based interview has a structure and the structure is you start with the storybased question. One of the things we teach is at the very beginning of the interview is to situate the person in the moment at the start of the story. So for example, if I ask you to tell me about the last time you watched TV and you're like, I watched something last night with my family. I'm going to say, okay, set the scene for me. Where were you? And I'm going to get you to describe like your living room and who you were with and who was sitting where.
46:05I don't actually care about those details, but I'm doing that because it aids your memory recall. It also communicates to you that I want all the detail, which is going to get you to tell me a better story. And then the next piece is we teach you how to build the timeline. So, we're using a lot of temporal prompts. What happened first?
46:20What happened next? What happened after that? And then because you're human, your brain wants to memory recall is system two, your slow, deliberate brain.
46:29And your brain is going to want to switch to system one because remember our brains are lazy. And so, as you're telling me your story, you're going to inevitably start to generalize. You're going to be telling me about a specific time and then you're going to say, "We always watch blah blah blah as a family." I'm going be like, "Okay, tell me about last night. Did that happen in this instance?" So, another skill we teach is to redirect generalizations back to the specific instance. These are the four elements that a coach grades on. It coaches on it grades on did you ask a story-based question, how well did you set the scene, did you build the timeline, and did you redirect
46:59generalizations? And it actually pulls out excerpts from your interview and shows you what you did well, what you did not do well, what you could have done differently next time. And then it gives you a coaching tip. Like some of the errors where we get confused, it would think a setting the scene question was a bad building the timeline question. It just kind of miscatategorized where they were in the interview. Sometimes the coaching tip would suggest a general question as you should have asked tell me about your typical day. No, you should never ask tell me about your typical day. So it would just like sometimes it would just
47:28get confused between the parts of the interview. Sometimes it would suggest a bad question and so my evals were just designed to detect these errors and then that allowed me to reduce them.
47:37That makes sense. It sounds like the interviews have some structure in mind and that helps you break them down and to evaluate them in a certain way. So I think that makes sense. What is the entry point to this kind of coaching? Is it like uh do you have to take a course to then avail yourself of this AI? Is it like something you could just like if I'm one of these people who I'm like, "Oh, I talked to a customer and I'm probably not doing it correctly, buy the software and like what is the entry point?"
48:04Yeah. So, it's part of our storybased customer interviews course which is a totally on demand course and as part of that you get access to the interview coach and is that the best entry point? Let's say I want to get start doing this. I'm like okay you know what I'm convinced by this conversation like you know I don't feel like I'm doing this wrong. I want to do this right. Should you start with that thing? Yeah, I actually think the storybased interview course is probably the most common entry point for people.
48:27I think interviewing is the highest return on your time in discovery, which is a little counterintuitive. You'd think assumption testing would be, but too many teams start with a solution that isn't solving a problem, and interviewing will help you find that sooner.
48:40Okay. It's like error analysis.
48:44Same relationship. Okay. Interesting.
48:46Okay. So, that's interviewing. there's a lot more that like kind of in the whole process as a whole and I don't want to skip any steps. So, but I know one of the important things is synthesizing the interviews and like creating these opportunity solution trees and doing that and one of the most interesting things in your writing cuz you've chronicled like your journey of like introducing AI into the teaching process. So people they obviously want to take all this information and they want to synthesize it and they want to
49:15create this artifact this opportunity solution tree and you've written about the tension here of like people have these you call it magic pills like they have these magic pills and they're going to point AI at all these interviews and they're going to have the magic pill they're going to swallow the magic pill and you fought it for a while you're like I don't know if you should use the magic pill. You write about how you're going to give them a better magic pill.
49:39So, can you talk about that? Like what the trade-offs are, how you let AI help someone without having the right trade-off of like having the human in the loop and so on and so forth. Like, can you talk about the trade-offs and what happened there?
49:52Yeah. So, talking to your customers is just the beginning. Like, you're going to have three customer conversations and you're going to very quickly be overwhelmed with all that you're learning. So, the next step is like, what the hell do I do with all this data? Like, what's my synthesis process?
50:06And unfortunately, what a lot of teams are doing today is they're just taking all their transcripts and they're dumping it into a folder for Claude or they're uploading it to notebook LM. And I get it. I use AI literally for everything. So like I get that like intuition. The challenge here is there's a lot of ways to synthesize transcripts depending on what we're trying to learn.
50:27And unless you're an expert researcher and you've like taken the time to identify your research goals, you designed your research specifically for those goals, you're giving all that context to the LLM, it's not going to do a very good job. It's just going to give you a very shallow like it's going to tell you some insights, but I don't they may or may not be grounded in those transcripts. They may or may not be what you actually needed to learn. They may or may not support the decisions you're trying to make right now. And so like there's a million types of customer
50:55interviews and you may not even know the type of interview you're doing and then you're just throwing or you might even have like 10 people on your team doing 10 different types of interviews and you're throwing all that to Claude and saying just figure it out. Like it's just you're not going to get good results. And the challenge is I have a lot of empathy for this because remember I said we're being asked to do this really deliberate rigorous process in an environment that's saying give me an answer tomorrow. And human synthesis is ridiculously slow and takes a long time.
51:23And I know that people listening to this are going to be very skeptical. They're going to say what do you mean like AI is not good at this? No. They'll be like no like my AI is good.
51:32I wrote a great prompt.
51:33I know this and actually this fascinating it has a parallel again with evals. Eval has this auto eval thing that's been cropping up lately. And auto eval is where you have an LLM look at all of your logs and try to do error analysis for you and like tell you what's wrong but it like you said it can't read your mind and it can't infer like what's important to you. So my favorite example is we have this apartment leasing assistant that we like to teach with this uh data set. If you throw that into any auto eval system,
52:03one kind of behavior that it'll say pass on is objection handling. Because the AI agent when someone asks to like look at an apartment, it's not available, they'll just say, "Hey, we don't have anything for you. Have a nice day." And the customer will say, "Thank you." And LM thinks that's amazing. LM's will say, [snorts] "Good job. You did what you needed to do." But like in the product frame, it's like, "No, this is a sales assistant. Like we don't we need to have some level of objection handling." Is there an example that you know your side
52:31of the world like that you have of hey like throwing it having synthesized interviews and like obviously going to miss something to make it real for people.
52:39I think there's a few layers to this. So one is just how shallow I teach story based interviewing. So all of my synthesis effort is on synthesizing a story-based interview. So this is one way this is one type of synthesis. What I'm listening for in an interview is where do you have unmet needs, pain points and desires? I'm trying to identify opportunities. So, if I give a transcript of a story-based interview to Claude and say, "Can you identify opportunities?" This is actually the very first thing I did like a year ago
53:08plus as my first AI experiment was like, "How good is Claude at this?" I was pleasantly surprised. Claude did a reasonable job. It missed things that I found and it found things I didn't find.
53:21And so, like, that's both good and bad, right? Like, if I just relied on it, I would have missed some opportunities that I uncovered. But it actually was a good teammate because it found things I missed. But how an opportunity is framed has a big impact on how we might solve it. So let's say I frame an opportunity as I get too many Slack notifications.
53:39This is something probably everybody can relate to, right? And how am I going to solve that? Okay. Well, if it's framed as I get too many Slack notifications, I'm immediately going to focus on notifications the solution. But if I frame that as I don't want to be interrupted when I'm doing focused work now I have lots of potential solutions but almost no human says I don't want to get interrupted when I'm doing focused work. What they say is I get too many Slack notifications. And so what Claude is going to capture is I get too many Slack notifications. And you're missing
54:09the real need. Like you don't complain about too many Slack notifications when you get a notification about an urgent matter that you need to pay attention to right now. You complain about getting too many Slack notifications when you get a notification for something that's not relevant to you right now. So that's really a relevancy need or a timing need. And we're not like Claude is never going to pick up on that. And so this is another skill by the way that people have to learn to be good at discovery habits is they have to learn how to frame needs in a way that opens up the
54:37solution space. So if we frame a need really narrowly, we're now stuck in just trying to fix broke notifications. But if you frame the need to be like not about our product and about what is what our customer is actually experiencing, I'm getting interrupted when I don't have time to be interrupted. That opens up way more of the opportunity space.
54:55And I don't like to teach an agent to do this well to like stay out of the solution space to frame it based on the human need and not based on your product. Like people think like, "Oh, it does a great job. I told it all about my product." Yeah, but did you tell it all about the humans? That's the important part. And so one thing I'm coming to realize, so I've spent the last, what is it? It's August. I spent the last six months working on AI generated synthesis. And one thing I've come to realize, like so many people have asked, why are you spending so much time on
55:24this? I just built my own pipeline in cloud. And here's what I realized. Most teams have never seen a good opportunity solution tree. So you mentioned that this tool is really popular. The problem with a tool becoming popular is other people write about it and they write about it in very shallow ways and then people read about those shallow ways and they misunderstand the purpose of the tool. And I'll tell you my evidence for this. The purpose of an opportunity solution tree is to help you find the best path to your desired outcome. So the root of the tree is outcome. Then you interview to learn the opportunity
55:53space and then you design solutions to match that opportunity space. So our purpose is to reach an outcome. We populate it by doing interviews. You want to know what the two most common questions I get are? I don't have an outcome. How do I use an opportunity solution tree? I haven't done any customer interviews. Where can my opportunities come from all these other places? Purpose of the tool is to help you reach your outcome and to make sense of your interviews. People think of it as an artifact that like has value in itself that like having the artifact is valuable. It's the synthesis. It's a
56:22synthesis exercise to help you make sense of what you're learning. Synthesis is so cognitively hard and you can scaffold it, but you're not going to remove the cognitive challenge. Like that is the work. But business doesn't give us time to do cognitively challenging work ever. It's a little depressing to realize, but like very few people have time and energy in their workday to do cognitively challenging work. I find myself in the situation where I look at all these teams trees and I'm like, where do these opportunities come from? They're like,
56:51oh, we just made them up based on what we know. It's like saying I made up my errors in the emails.
56:55Yeah. This is like saying you use generic metrics off the shelf.
56:58Yeah. Like it's it's bad. And so I've seen this for years. I've been fighting against it for years. Like it's not changing. And so when I started like collaborating with Claude on my own synthesis, I was like, "Holy crap, Claude is already better than the average product team." Claude is better at the cognitively challenging work than the average product person. And when I say product person, I mean product manager, designer, software, anybody who's involved, right? And that this was like very disturbing to me in a lot of ways because like I think humans benefit
57:26from doing synthesis and like I really resisted this but I have to face reality. I can teach the AI to be better than most teams and if I can do that like why wouldn't I make that in the world?
57:36That's really interesting. I mean that takes a lot of humility right cuz like at the top level is Teresa's skill and then there's the average person skill down here and the middle is like Teresa's AI. You want people to be at Teresa's skill. And I hate saying this because like it sounds like a lot of hubris like you suck at this. The I'll teach the AI to be better than you. Like this is just I don't think it's that humans can't do this or can't do this.
57:59Well, I think our business environment is such that it's really hard for most people to build the skill and find the time to do this really rigorously. And I have seen over and over again that when you do it really rigorously, you learn at such a level that you can build much better products. And like I want more teams to have that. And then I'll tell you, I read a paper that completely convinced me this was the way to go. A research paper came out about the impact of working with AI and who was getting the best results. And using AI does
58:29raise the floor for noviceses. And this was like across a wide variety of like areas. Like you've never been a marketer, you want to learn how to like market your new blog, AI will raise the floor. Okay, that's what I was already seeing. The part about the paper that really excited me was experts working with AI also raised the ceiling. And I was like, "Oh, okay. Now I have goosebumps because you just talked about like regular people, Teresa's AI, Teresa. Now there's Teresa with AI and that's even higher." It's like, "Okay,
58:58so now I can attack this both directions. I can give you an AI that will help you do the work. That's going to raise the floor. I can build teaching tools to help you build your skill, and if you work with my AI, that will raise the ceiling." Once I learned that, I was like, I would be crazy to not invest in these tools.
59:13Very interesting. I mean, as an educator myself, I'm really fascinated like how education might transform with AI and I think it's an important problem to think about. Did you change the format of your courses like your offering? Did you do anything? Can you talk about that a little bit?
59:28Yeah. So, for a long time, I resisted on demand courses because I really believe in deliberate practice and our live cohorts was all about practice. We now have we still have a cohort-based course. Our flagship like beginner course is still cohort-based and probably always will be. But we now have moved our advanced curriculum to be on demand and that's because they're all supported by AI tools. So students submit their homework, they get personalized, detailed feedback, it supports this like infinite rounds of practice for people who are motivated to do that. The other thing I'll share is
59:58like I think I experienced this with engineering and that's what's opening me up to this idea of like if I build a tree for you. If I do your customer interview synthesis for you with the AI tool as long as I show you what the tool is doing like I don't just say here's the answer but I show you the work in progress. I actually think that is a teaching tool. And the reason why I say this is I have learned a lot about engineering over the last year. Just for perspective for folks, a year ago, I was writing some code to like automate my backend business logic, but I literally
1:00:28did all of it in the this is embarrassing. I did all of it in the Amazon management console. I wasn't using an IDE. I wasn't using git. I had no unit testing or integration testing.
1:00:37Like I just like was piping data from one tool to another tool and just like writing lambda functions and step functions in the console to just make it work. And I was at a small enough scale that like when I had errors, I just went in and fixed things, you know, like it was just duct tape and shoestrings. But as I started building AI products and especially I have a AI products now in a real production software tool, I had to learn how to be an engineer. And when I first started like working with Claude in like VS Code, I didn't know the AWS
1:01:06command line. I didn't know any of the git commands. I didn't know I barely understood like why do I have to add files and then stage them and then commit them. I don't know what feature branches work. Like I didn't know any of this stuff and I didn't need to know it.
1:01:17I would just tell Claude like can you commit this change? Can you tell me what's in this Dynamo DB table? Can you right? And like Claude did it. But this was at a time when Claude showed you everything it did. It actually drives me nuts that it doesn't do this anymore.
1:01:30And I learned a lot. Like I now know how to use all these tools myself because I watched Claude do it over and over and over again. And so when I'm building AI synthesis, I'm building it in a way where my goal is to show the process so that as you use the tool, you're actually building your own skills at the same time.
1:01:47And can you talk a little bit about that? So I know that you've partnered with a company called Vistili that helps with this opportunity solution tree and maybe some other things as well. I think this story of how you built it is one of the best eval stories I've ever heard of. Can you talk about that?
1:02:02Yeah, so there's this company Vistily.
1:02:03They've been around for a few years.
1:02:05They're definitely a startup. They basically took the opportunity solution tree idea from my book and built software around it. So they have a canvas tool where you can add tree nodes. Here's the problem they're trying to solve. A lot of people build their trees in like Mirro or Mural or Jam Board. I don't think Jam Board exists anymore. Fig Jam, it's just a digital whiteboard. But the problem is like you need to move things around and then all the arrows get moved with it. And so what they did was they built a purpose-built tool just for the opportunity solution tree where there's
1:02:34nodes like you can see here and there's at the top is your outcome and there's opportunities, there's solutions, you can add your assumptions that you're testing and they just built this like really slick interface for opportunity solution trees. And so they joined my community a few years ago. I got to know the founders. The very first partnership we had was we actually integrated my interview coach into their product. And then as I started to experiment more with AI synthesis, we started to talk about what if we brought all that AI
1:03:02synthesis to Vistily. And so in Vistily you can upload your interviews. My AI service will extract key moments and opportunities. So that's two of the elements on an interview snapshot. Once you have three interviews, we will generate your first draft of your opportunity solution tree. And then every one to three interviews more, you can choose to update the tree as you do more interviews. And it's all AIdriven.
1:03:25And my understanding is you are the chief AI developer like you are responsible for all the AI in Vista.
1:03:33They license my AI services. So when you upload an interview and it generates a snapshot that content is coming from my AI service. When you generate your opportunity solution tree the whole tree is coming from my service like a private API somewhere or some endpoint that they're calling and is doing.
1:03:52We technically they actually have my source code. So they because of data requirements. So they're HIPPA compliant. They're SOCK 2 compliant. They follow all the GDPR rules. They're in the process of getting some ISO certification I don't know about. I didn't want to do any of this stuff.
1:04:07Like I don't really want to run a software company. I just want to be an AI researcher. So the way we set up the deal is it's actually source access partnership. So they deploy my software in their AWS environments. That allows them to keep their European customer data in Europe. I don't have to worry about running multiple AWS accounts or instances or getting all these certifications and then so I just they're part of my GitHub repo and they when I commit they deploy in their environment.
1:04:31Okay, this is pretty sweet and to come back to your point okay these trees so you wrote about this journey of like okay like creating the AI for these trees and back to your point about showing the work. So you don't just create trees you show people the diffs of the trees as the trees evolve. So you know from like step one, step two, step three, step four, how this tree changed over time and it's like a non-trivial computer science problem. Can you talk a little bit about that? Like what
1:04:59happened, how you solved it?
1:05:01So I'll say there's there's actually four AI services that I build. So we'll talk about the different components and that will help with this. The first is I have to classify your interview transcript. Is this an interview I can even do anything with? And that seemed like a trivial problem because I just want to know is it a story-based interview? Okay, we launched our beta.
1:05:19We got 500 teams to upload three interviews. Of those 500 teams, I think eight of them had story-based interviews. So very quickly, we were like, "Oh yeah, people don't know how to interview. I forgot this problem." So we had to expand from storybased to just general interviews. And then this is a problem. Can I even identify opportunities from general interviews?
1:05:37So like even on day one, this turned out to be a way harder problem than I thought it would be.
1:05:40So why did you relax that requirement of having a story-based interview? You could have gone either way. you could have said like hey like no you have to have a story based interview cuz that's what teach or whatever.
1:05:50So my inclination was to say we're just going to support teams that do story based interviews. Uh we would have had eight beta customers. Vistily was like hey we need to make money. We're a company and we need to grow. I actually shared this concern. We were concerned the market of teams that do story based interviewing and especially storybased interviewing well would just be too small. So we're trying to meet teams where they are. The way that I reconcile this for myself is I'm coming up with this model of strength of evidence. So if we identify an opportunity from a
1:06:19story-based interview, that's a stronger signal than identifying an opportunity from a general interview. The other thing that I did before I agreed to expanding this was I ran experiments with some of the general transcripts to figure out what can I extract from this and what do I think that quality of evidence is. So we don't extract everything from a general interview. But we only extract things that in a general interview, like if I ask you, "What do you like to watch on Netflix?" You might answer that question by telling me a little mini story. That could be an
1:06:47opportunity that has just as much strength as a story-based interview, but it came from something that wasn't a story-based interview. So, I recognized pretty early on like even though this isn't classifying as a story-based interview, there are strong enough signals that I can still identify opportunities. Now, we actually got eight different types of interviews uploaded. We got like team meetings which I think were like people thought they were stakeholder interviews. We got sales calls. We got usability sessions whereas we decided we're not supporting any of those out of the gate. Usability sessions we will eventually support but
1:07:16they will be used to evaluate assumption tests not to generate opportunities. This has been one of the most interesting things.
1:07:22Is there a way you could like nudge those people into doing more story based interviews or is there a way how do you do that?
1:07:28So long term one of the things I'm working on is this other product right now it's called Terresa. We'll see what the name stays in the long term. But the idea is that Theresa Bot is a discovery coach that when it can see your work will nudge you towards better stuff. So if I can see your interview transcript, I can tell you like next time try asking this instead. I could just give you like little mini tips to like move you in the right direction. So there's a lot of stuff we plan to do down the road to help people get better quality of evidence. What's been super fun is that
1:07:56like I have a much like as a coach I got to work with like dozens of teams at a time and see what their interviews looked like. I only sometimes got to see their transcripts. Like now transcripts are everywhere. Everybody records everything. But like historically that wasn't the case. Now I get to see like the actual data. Like this is where teams are. So that's super fun. And it opens up all sorts of opportun like teaching opportunities. Can I nudge you towards better interviews? Can I teach you about evidence quality and like this is a weaker signal than this? There's
1:08:25like so much I want to do, but there's also so much to do. So the first service is I classify your transcript to determine if you're even qualified. The second service is if your transcript is qualified, I extract key moments and opportunities. So key moments are like what happened in the story.
1:08:40Opportunities are unmet needs, pain points and desires. The third service is we take three interviews and we generate a tree from scratch. So the agent gets to decide everything. These are the key moments on the tree. These are the opportunities and we're just literally building it from scratch. There's a fourth service which is update a tree.
1:08:58And the reason why that's a different service is after we generate a tree, a human could go in and edit things and like rename things, add some things manually. And so when we update a tree, we don't want to lose any of the human edits, but we want to like correct problems. And so there's a little bit of like a more of a providence problem in update because there's like a base tree that we have to manage what we can edit and not edit, if that makes sense. And
1:09:25so it's only on update tree that we I build change sets. So my service will take in an existing tree, some new interview snapshots, and it will output a new tree, but it will also output a set of change sets. So we defined a vocabulary of moves that the agent is allowed to take.
1:09:45Is there a visual of chain sets that you think I can find somewhere? So I output the tree and then along with the tree I output this change set where it's literally walking through exactly what the agent did for this is a change set for a new tree. We're adding an outcome.
1:10:00We're adding this opportunity. We're adding this opportunity. And so this is a really simple change set. Some change sets are really complex like what if the agent wants to take this opportunity A and split it into A and B. This opportunity has sources on it. The sources are what interviews did this opportunity appear from? And so if we're going to split it, it also has to split the sources. So when the agent decides it wants to split A into A and B, it also has to decide which sources go
1:10:29where. And the reason why we're doing this, there's other complex moves like a merge. So it can decide to merge A and or B and C. And in that case, the sources get combined and the children get combined.
1:10:41This is like a lead code hard problem. Yeah. And I blindly wandered into it.
1:10:46Coding puzzle. Yeah.
1:10:47Yeah. I blindly wandered into it and solved it. So the reason why we're doing this is when you update your tree, I don't want to just say like click the magic button and then here's a new tree. We want you to see what changes happened on your tree based on the new content.
1:11:03So we want to say like and we want you to be able to follow the providence. So we added opportunity B and here's a link to the interviews that had that. And Vist has built a really cool interface for this. Like when you open an opportunity on the tree, it will show you exactly which interviews that opportunity came from. And you can even play video clips of that opportunity. So you can hear real customers talking about that need. And this is really important to us. We want to make sure everything is grounded in real customer data that you can always trace back
1:11:33where everything on your tree came from and you can see the steps the agent followed to like add things and merge things and split things. And so we to make this work, we had to define like a whole vocabulary of moves that the agent could make. And then it's not just about producing the tree, but also making sure that you can produce an accurate set of moves that that the idea is that you can take the input tree, apply the change set, it will generate the output tree.
1:11:56And we talked about doing this because we wanted to like walk the user through like how to update trees. Like we want them to learn how to do this themselves. I didn't realize at the time like how hard of an engineering problem this was.
1:12:08And uh I had some bugs. The LLM basically would generate change sets that didn't generate the tree. And first actually that's not true. My bug was I wasn't having the LLM generate the change set. I was deterministically generating the change set after the fact. So I would look at the input tree and the output tree and try to just create a diff. We were like inspired by git. Can we just create a diff for these two trees. The problem is is the diff is ambiguous. There's more than one way to generate the output tree. And especially
1:12:37with trees like merges and splits and the order in which they happen is really ambiguous. And so my deterministic change sets weren't always generating the output tree. And so the way I had to fix this is I had to get the LLM to make basically like a scratch pad of all the moves that it was making and then use that to generate the change sets.
1:12:55And what do you mean by scratch pad?
1:12:56Yeah. So the LLM it used to just like like one step it will do is identify like structure a branch. That's one of the steps it does for branch. It'll add structure to the branch. So originally the LLM would just output the structured branch. I updated it so it outputs the structured branch and the change sets for how it generated that branch. This actually improved the quality of the agent like a text file, a JSON file, something.
1:13:19Yeah, it's all the LM always just outputs JSON. So the tree is represented in JSON, the change set is represented in JSON. And this actually improved the quality of the agent. Like when I had to show its work with change sets, it reduced a whole bunch of other errors. So that was like a nice surprise. It was like a chain of thought almost.
1:13:35It's like a chain of thought. And then it also allowed me to add an agentic loop where it'll generate the the branch and the change set. It can call an audit tool and the audit tool will take the input, run the change set, look at the output, and make sure they match. And if there's any mismatches, those errors go back to the agent to fix. And I don't necessarily recommend this on every kind of problem, but because your these trees have this quality. It seems like you're doing some computation, some
1:14:03intermediate computation, and this like quite a lot of computation potentially to check different things to do like to audit things. There's this thing called recursive language models. It's the naming of it is very scary, but really what it is basically it's like giving your LLM a kernel. basically like like a Jupyter notebook like an IPython kernel that can go like fiddle with come back to you. So it might be interesting.
1:14:27Yeah. You know what's been fascinating about this is I've toyed a lot like right now most of my processes are still pipelines. They might in the pipeline they might have a aentic loop step but it's not like fully agentic. And part of the reason for this is just cost. Like I experimented with an agent that had skills for each of the pipeline steps and just let it like try to attack the problem and it would be like $25 per run. Whereas right now with my pipeline with some agentic steps I'm at like 5060
1:14:58It works really well. You don't need to fiddle with it.
1:15:00You know what's fun is that like I set up my partnership for me to just be an AI researcher. Like I don't have to deal with any of the company stuff which is really amazing. And so it gives me the flexibility to be like I'm just going to try this and see like what if this was an agent with a bunch of skills or what if it was an agent with a bunch of tools or what if it was this was an agentic loop and this was a pipeline and I just sort of play with different orchestrations and I obviously when I release something I have to care about the impact on cost but I don't have to do that for my own experiment. I can
1:15:28afford to do a $25 experiment just to see what happens and like to see the difference in quality. And that's because there's so much complexity to these layers of synthesis. It's been really fun to like run all those experiments and start to learn like, oh, this brand new thing came out, but it's just not viable. Like it's just too expensive for right now. Or like even just playing with what models are good at what steps and how do you optimize cost on that. I actually just watched the model cascade lightning lesson you did and I was like, "Oh, this is perfect timing." Cuz I have some evals that I
1:15:57had to use sonnet for and it's just so expensive. And I was like, "Oh, but I could do a model cascade." So, I'm having a lot of fun just like tinkering with those things. And this is I think the benefit of my partnership with Visty is like they're the startup. They're being startup employees. They're running the company. They're doing the sales. They're doing all the data practices.
1:16:16And I really just get to focus on the AI research part and like on my domain expertise of the discovery part.
1:16:22I need to find a situation like that.
1:16:24Maybe like the Eval SAS company. If you find an opportunity like that, we should talk about contracts because I feel like I ended up in a really good situation and visally feels like they ended up in a really good situation. So, I feel like it was a we found a way to make it a win-win, but it wasn't easy. We spent months talking about the right way to do it, but I think we landed on something that works really well for both parties.
1:16:45Is there anything you can share anyone that does teaching might benefit from if they're coming to this type of situation? Like anything to think about?
1:16:54teaching was never about transmitting knowledge. I think it's really easy to think about it as like you're the teacher and your job is to impart knowledge on your students, but I think that's like a very crude like way to think about teaching. I think about it now as my job is to create spaces for people to learn. And that sounds very handwavy, but I'll give a couple of examples of this. like my storybased interviewing course which we talked about. Yes, there are teaching articles and there's teaching videos but I know from teaching this course for nine years
1:17:22now that that helps you learn concepts but it doesn't help you apply the concepts. The way that you apply the concepts is through structured practice.
1:17:29So my job is to create a way for you to do structured practice. Another way that I do this like when I started writing about my all my adventures with AI people started asking me like why aren't you creating an AI course? I don't really want to create an AI course because it changes so quickly. Like I don't want to do all the work to like constantly revise it. And if I think about like there's so much knowledge about AI, people don't need me to impart more knowledge about AI to them. Like it's already out there.
1:17:53The revising stuff is so true. I we had to just completely redo our entire eval course.
1:17:57Yeah. It's just it's a lot of work, right? What I realized was like I can create a space for people to learn without creating a course. So like in my community, I host AI maker studio sessions and I host AI showand tell sessions. So, the maker studios are we just show up and we like co-work on Zoom. It sounds so stupid, but it reserves space in people's calendars to like work on personal productivity things with AI with a whole group of people that like when they get stuck, they can ask a question. And then a week later, we do AI show and tell where
1:18:26people show up and they show and tell what they built that week. And it's just enough for like a total beginner can show up and get inspired. We have a whole wide range of people with experiences. And so like I am really changing the way that I think about teaching. I think knowledge is basically free at this point. And I think like the real value is how do you create opportunities for practice? How do you create space for learning? How do you build community around this? I remember I I went back and I got a masters in
1:18:55learning and organizational change. And one of the things I learned in one of my classes was something like you don't learn on your own, you only learn in community. And when I first heard that, I thought that was baloney. Like I'm a big reader. I learn a lot alone, but I was thinking about learning as acquiring knowledge. And I think learning is more about acquiring skill and acquiring new habits and adopting new mindsets. And I think all of that requires learning in community. And so I think AI can accelerate skill building because it can help with like infinite practice, but I
1:19:24think there's also this like really human part of it of how do we create community around this and bring people together. And so I'm not really like extending my curriculum anymore. I'm because that's like the knowledge piece.
1:19:35I'm extending my curriculum with AI tools to like support infinite practice and I'm using software to show what good looks like. Like what I love about Vistil is we can show teams what a good tree looks like, what a good snapshot looks like. Eventually we'll add some coaching nudges into the platform where as you work you'll get feedback on how to improve things. And then I'm really looking at how do we learn in community.
1:19:59Tell me more about the community part.
1:20:01What are you experimenting on with community if or thinking about in terms of community if if anything?
1:20:06I think people learn by seeing and by doing. So you know there's this like mantra of like I do, we do, you do in teaching. So it's like I'm going to model it for you. We're going to do one together and then you're going to do one on your own. This is like really common and it's grounded in a lot of evidence.
1:20:23So I can describe something to you but until you see it, you don't really get it. And so some of community is just bringing people together to show. So our show and tell is just come together and show what you're doing. And a lot of people don't know what's possible with AI.
1:20:36Oh, cool. Okay. So you have this community. It's very reasonable.
1:20:42For a whole year. Wow. That's I am at the point where I'm very intrinsically motivated by what I do. And for my community stuff, I want to make it as accessible as possible to as many people as possible. It's not my primary revenue stream. It really is just like bring people together and see what emerges. It's like a playground for me.
1:21:01How many people roughly are in your community at the moment?
1:21:04A lot of people subscribe for free.
1:21:06There's a lot of things. You know, to be honest, I don't do very much for my free subscribers. They just get emails knowing there's content coming. A supporting membership is where I do a lot of events. Few thousand on the supporting membership level. And then the CDH membership level is a little bit complicated for members because we there's two ways to get access to this program. The first way is through this subscription. You can pay $359 a year. I think we have a few hundred people that do that. It's not giant, but this gets you everything in the supporting membership to so all of our events, but it also gets you access to our Slack
1:21:35community, but our Slack community is bigger than a few hundred people because of our courses. If you take any of our courses, you get access to the Slack community for three months. And then our c our companies that work with us get access to the community for a full year. So, there's a lot of like overlapping circles for each of these programs.
1:21:52Here's what I'll say on the community side. I look at it as a two-pronged approach. The first is can we create events where you can just come together and learn from each other. So that's our AI maker studio, our AI show and tell.
1:22:04We do office hours, we do community calls, just opportunities for people to get come and learn from each other because so many people don't know what's possible with AI. And like for those of us that are like doing this all day every day, we forget that normal people are still using chat GPT like Google.
1:22:19And so like we have people just demonstrate all sorts of stuff like someone made a coloring book for their kid, like a physical coloring book and had no idea how to do that. A lot of just cool projects. And then the other thing I'm starting to experiment with both of these subscriptions is like that CDH membership gets access to my Teresa bot. So this is where I'm running all my experiments of like ondemand coaching nudges.
1:22:40Okay. You like get grilled by Teresa.
1:22:42The Teresa bot started really simple. It was just a vector database with all the content I've ever written. So all my course content, all my book content, all my articles and it started as like you can just ask Teresa bot a question and it will point you to the right content.
1:22:57So it was sort of like a concierge because of my work with Vistily. I was like well canabot be a coach and so I started to give it skills and so I've built like a number of little AI tools to support our courses. So like I have an outcomes coach, I have an interview coach, I have like a coach that will give you feedback on your research questions and your interview questions.
1:23:16I have a coach, our business fundamentals coach will give you feedback on like how you define your ICP and some other things.
1:23:23And these are different bots that you tag in like Slack. They're all those things that I just rattled off are like AI tools in our courses. And so what I did was I basically gave them to Teresa bot as tools or skills. If you ask Teresa bot like, "Hey, this is my outcome. Can you give me feedback?" It's not just finding an article to give you an article to read. It's actually invoking the outcome coach and we'll give you feedback based on that.
1:23:47Okay. So, there's like some tools in here like in this thing and actually it's not updated on here because like all of this is just like in the last month. I haven't revised these since I launched them. So, like the way that I would describe these subscriptions is like both of them give you access to all of my premium content on the blog. The supporting membership level is for people that like want the community events and then the CDH membership level is for people that want like the 247 Slack community, want Teresa bot and then they're going to stay in that split like supporting will
1:24:17be very driven and CDH membership will be very AI tool driven.
1:24:20For anyone watching, I would encourage anyone to support Teresa, read her newsletter. I'm going to be joining this community right after this call. It just came up organically. I've never talked to her about it before, but one of the increasingly rare creators who like she actually puts a lot of thought into her writing. There's like no slop that I can see. It's like very nice to read and that tells you a lot. Thank you, Teresa, for doing this. I'm looking forward to what you create and I'm looking forward to having you in this next uh cohort.
1:24:49We're going to be talking about a lot of new ways of doing evalu.
1:25:00Yeah, I'm excited. Actually, my next blog post, I'll give you a sneak peek is I spent three weeks about a month ago chasing down an error. It literally started with a customer. I was on a customer call getting feedback. We were watching them react to their first AI generated opportunity solution tree. It was awesome because they had hired an agency to conduct their interviews and the agency gave them a research report and then they took those interview transcripts and uploaded them to Visty.
1:25:28Wait a second. Is it a good idea to have an agency do your interviews?
1:25:32Most of the time, no. So, I'll share at the very beginning of my consulting career, people hired me to do their discovery for them. This made me really uncomfortable because discovery needs to be continuous. So like an agency will do a project for you but then what happens right and so the reason why I shifted into coaching I basically when people said will you conduct these interviews for me I would say yes but you have to send your product team with me so they learn how to do it themselves. So but what I loved about it for this customer call because the agency had already like conducted the interviews and done the
1:26:01synthesis and they had a research report. She uploaded those interview transcripts to Vistily and saw the opportunity solution tree and like her feedback she was blown away. She's like, "Wow, you found everything the agency found and more." And then she paused and she goes, "And we spent a lot of money on that agency already." I was like, "Man, can I just have this as a testimonial?" But then she's like looking through her tree and she pauses on one branch and she goes, "Wow, there's a lot of unsorted opportunities here. I'm going to have to come back and clean this up." And of course, that's the quote I remember cuz that is a
1:26:31error. That's a failure mode. And so I immediately like after that call started to look at that failure mode. And I thought it was just going to be a quick fix. like I just needed to like tweak a prompt or something and turned out to be one of the hardest emails I've ever written. It's a L1's judge and I really struggled to get it to calibrate and the reason why I actually asked you about while I was right in the middle of this I sent you a text about it. I that was the silly fruit example thing I gave you. The reason why the judge was struggling to calibrate is because there were confounding errors in the samples
1:27:00in the calibration set. The judge with the error was so on a tree you have a parent a node on a tree is like a parent with a set of children and as the children grow you should be adding structure there's probably subgroupings and so the error is missed groupings like there was an opportunity to group these children and the agent missed it and I thought like this is going to be easy I'll just measure that and off we'll go well it turns out like trying to reduce that error exacerbated another error and then trying to reduce that error exacerbated the first error so
1:27:29it's like telling an agent to do one thing created one problem and then trying to reduce that problem created the original problem and I spent three weeks like I think I had to create four new eval experiment variants like big experiment variants not like little tiny tweaks before I found a solution that like was able to reduce both errors and I learned a lot about it's so easy to think you can fix everything with a prompt what was actually required was more of an orchestration experiment not a prompt
1:27:57experiment and you know when I first when I was struggling to get the judge to calibrate. I saw the confounding errors, but I didn't want to fix them.
1:28:04And the reason why I didn't want to fix them is because this was like nefariously tricky. No customer was complaining about the other errors. So, I just had a customer complain about this missed groupings. The missed groupings judge was getting confused because there was some poorly framed opportunities, but no customer had ever complained about a poorly framed opportunity. Like, they were good enough for the customer. They weren't good enough for the judge to not be confused by them. And so like I really resisted wanting to fix it. It felt like I was wasting my time fixing something that
1:28:34like was good enough. And so like remember when I reached out to you, you were like, "You have to attack the upstream errors." I was like, "I don't want to attack the upstream errors."
1:28:41Like a customer doesn't care about those errors. But it turns out I had to fix the upstream errors. And then fixing the upstream error exacerbated the error I was trying to fix. And then fixing that error exacerbated. So there was like this weird back and forth between two different evals that like were counterbalancing each other. I'm going to create a snippet of this portion of the interview cuz I can't tell you how many times people they want to just like create evals for like every single step and they get like all this analysis everything and I'm like you need to
1:29:10focus on the upstream error and everyone resists that. This is a great story.
1:29:15So my next blog post is the really detailed account of what I tried, what was hard and then how I resolved it. And the way I eventually resolved it was I have this agentic loop and I just added my evals as guardrails. And so I couldn't get the original agent to be good at both. It would always have one error or the other. And so I took my evals and I run them as guardrails and it gets kicked back to the agent and it has no problem correcting on the second loop. So instead of trying to like make magical prompts that could just do it
1:29:44right from the first time, I just gave it a tool and that pretty much solved it, which is pretty wild.
1:29:48Really look forward to the blog post.
1:29:50Yeah, sounds really really ask too many questions to ruin it for people. I'm going to I'm sure you're going to share a lot of the like cool details about like what the evals were and like the different iterations of them. When is it going to come out?
1:30:02Either this coming Wednesday or the Wednesday afterwards.
1:30:05Well, it was really nice to talk to you again, Teresa, and thanks for sharing all this information. I think it's going to really help people, especially if they are thinking about like talking to customers. I think if you do anything, check out Teresa's book. Put it on the screen somewhere and check that out. check out our community and yeah, look forward to seeing you in class as well.
1:30:23I'll say since your audience is a lot of engineers. I know a lot of product teams that like are being taught include your engineers in discovery, but there's like some resistance from engineers cuz I don't think this is engineers fault.
1:30:35We've been telling engineers for decades like just build what we tell you to build. But I know engineers are makers and they care about what they're building. And I feel like there's a huge opportunity right now. I think most teams are being asked to be product builders, not just engineers anymore, across all functions. And I have a ton of free resources at productt talk.org.
1:30:55The very first article at the top of the page called product discovery basics like is a great place to start.
1:31:00Okay, product discovery basics. So it's like right here in the banner, click on it and that visual you're looking at is opportunity solution tree right in the middle. This is just going to cover like the high level. This will give you sort of like it'll start to build that mental representation of how this stuff works.
1:31:15And then if you're a reader, Continuous Discovery Habits is available in paperback and Kindle. And there's also audio book that I narrated which was a lot of fun. And then if there's one skill to invest in, I would say build your interviewing habit. And so if you want some of that infinite practice, check out storybased customer interviews.
1:31:32Change discovery habits. Check it out. I have read this. I also have the audio book too. It is fun to listen to Teresa. She has a lot of life into the book, so it's great. Well, thanks a lot. And thanks, Ham.
