0:01okay we're live hi jonno nice to see you again hello everyone and hello Eric thank you for joining us my pleasure how's it going uh it's going great I am super excited to have you here um I imagine everybody knows Eric at least by reputation but just by way of a quick background uh Eric is my co-founder at answer aai uh and is most known as the
0:30author of and developer of um the the Lean Startup uh approach um and of course the creator of the long-term Stock Exchange which is a really fascinating idea uh and the I guess the startup where many of the ideas from Lane startup were founded the uh the company is it inw IM IM view or IM view I never actually figured out how to say it right or we never corrected anybody and it never had an official
0:58official uh just yes just the the the non acronym acronym IMVU yeah where does the name come from I actually never thought to us we we agreed to um so this so we agreed to try on a new name every week until one stuck and we would we would use it all
1:25the way like in variable names and Co we would pretend that that was the name of the company 100% for a whole week to see how we liked it and one week we just never got around to changing it again and um we were do want to do something with instant messaging this is before social networks so we wanted to bring virtual reality and you know Virtual Worlds avatars that was that was the idea to instant messaging so we spent a lot of time looking for four-letter domain names that started with I
1:55am and that was one that we could get our hands on and that's uh we we tried it out for a week and we said oh we like this we just we just didn't change it again after that okay that's very it has no it has no official expansion because it was just a four-letter word but the first time we were sued I remember the lawyers felt the need to explain that it was an acronym and they they actually expanded the acronym to give the company a more formal sounding name so that we could be sued properly that's the only ever time it's been
2:24expanded so Eric I know um you've watched lesson seven which was the first lesson jonno and I did where we tried to bring solve it to a bit of a real world problem um yeah tell me like what were your what are your reactions to to what
2:53you saw oh I'm really excited um I've been looking forward to this to to lesson seven in particular for a while and of course you know I've been on the solv It Journey with you since before it was even called solv it and we've been talking about seeing it come to reality um for a long time so it was it was super cool to see it and I was actually just just chatting with Jon too right right before this that just to see the little affordances and the improvements in the UI and just the the the way it's
3:21turning into a real thing um has been has been really exciting so that's very cool but I also you know I'm I love the idea of of getting to be part of something early and so my main like my Mo like my emotional reaction to watching lesson 7 especially which I didn't have to the same degree with the earlier lessons but for with this one I just thought um man what a privilege to get to be part of the early Community for this because this is going to be
3:50something this is going to be something really big and I just felt like we're we're going to be giving the people who have access to this just these incredible superpowers and I'm excited to see what they they do with it what was it like I know a lot of this will be like Intuition or feel or whatever but what if you were a reverse engineer that feeling you had why what's making you think those things you know it reminded me a lot of like the first time I was on the
4:19Internet or um first time I encountered the worldwide web or even when the first time I learned programming like for the first time you know or like I I could go through a whole list of things that had this feeling to them where it's like um it's so easy to feel like you're too late you know everyone's already had all the good ideas I I mean I was I got involved with the web when it was still competing with gopher as the like secondary protocol trying to displace gopher on command line terminals so it was obviously not too late but even then I was like oh people have already figured out all the all the cool stuff
4:48with it so so I have a bit of that feeling of being left out and so I'm I'm just I'm very tuned in when a new thing comes on the scene and the the thing I I feel emotionally attached to intuitively if you look at something you say okay this has tremendous it's it's tremendously useful for a lot of things but if someone were to try to pin you down and say well what's it good for I can't tell you how many times I had to defend the internet you know in the early days like what's it good for and i' like ah I don't know it's like
5:14everything else but worse but but you know but better actually because it's wor it's just like very confusing like why should I send an email when I could just send a letter I can send a letter to anybody on the planet I can only email these 25,000 people why is email better than a letter so there's C there's certain things that just like can't really be explained until you know how it's going to turn out but I think the the through line to all these different new technologies is like when you start to see something that can create new capabilities where you do
5:43something and then the next thing you do Builds on that and builds on it and builds on it like the stepbystep assembling of cool things out of smaller bricks like that has the potential to take all kinds of problems that we currently view as impossible or too difficult to be economically viable to pursue and just if you can actually break it down down and work on the little Parts like then you can actually accomplish things that that are not currently deemed um possible so that was kind of like that was the feeling I had about it watching okay I mean that's really interesting because like what you're describing
6:12sounds like a reaction to AI but you've seen AI before but obviously you think there's something in this soled approach which you know is grabbing you I guess as sure yeah look I've been I've been very following AI very closely U you know because you're my friend so you could have kept me up to date for long before it was uh very popular but not nearly as long as you and you know and
6:40look I I can remember sitting in my college dorm room not that long ago wondering you know will an AI ever you know beat a beat the top humans at go like seemed impossible you know would would we ever be able to do you know pass the Turning test or like so many things that we now take completely for granted as not just capabilities exist but that you can rent them for like a fraction of a cent per token like unbelievable the value that so so so a has been enormously exciting it's been
7:07great obviously there's all these big issues and social issues to Grapple with and it's been this big big big thing the last couple years but if I'm being really honest it's also been a bit of a disappointment not because it's not capable and not because it's awesome it it is awesome but what have we actually practically accomplished with it so far and there just aren't that many
7:34actually deployed actually useful AI applications certainly not compared to the billions of dollars that we have invested in it so far and look that's natural in a new new stage or something you know it takes a while before we really figure out what it's good for but I've just been around a lot of AI startups a lot of AI research Labs a lot of people who are kind of like sort of can kind of get it to do something some of the time they can create really cool demos but we haven't seen a kind of like general purpose tool for translating
8:03this capability into practically useful things at a like at in a reliable repeatable way not just you know like there's um you know there's certain tools you know like co-pilot and cursor and there are these various tools out there that certain people have been able to do really extraordinary things with so I don't mean to take anything away from those tools they've been great but I I haven't seen the one where I'm like yeah that has that feeling like this the dawn of the spreadsheet or just the first time I was ever dropped into like a basic terminal and be like make
8:30whatever you want out of this it the Rel like the reliable thing in lesson seven wasn't the AI it was jonno you know J you know that's the interesting thing which is kind of different and I just thinking as you're saying that like okay if jonno was doing that same exercise in cursor I don't know what do you think jonno it would have looked very different and probably not be nearly such a smooth thing cuz
8:59you were like interacting with the Machinery partly although like I kind of do feel like there's been this gradient of getting closer to there for a while right so like U maybe maybe not um necessarily coding the the common way en curer but there's been there's been Google collab or um having very helpful back and forth with a chat model in one Tab while you
9:28do something else in the other like those pieces have all felt like they're starting to get there um so I think it was really nice to have that go well but that feels like we're just continuing a trend that started when chat GPT started replacing stack Overflow right it's like oh this this starts to give you useful tab completions occasionally or this starts to give you like a bit of assistance yeah that's right it's missing stuff but it you know it all kind of like blurs together into this Continuum of like okay we starting to figure out better how to take advantage
9:56of that um and it still feels like this's ways to go for like what it it ought to do um but yeah it does still feel like we're climbing that hill and like okay this is feeling a lot nicer than what I would have coded a year ago doing that for example yeah and you know in the theory of disruptive innovation or like in you know in the in the startup world and the people who study technology progress um we have a term for what we call the whole product which is like out of the soup of like The Homebrew Computer Club sometime like
10:26there there tend to be these moments where like a distillation takes place of like the apple one computer or especially the Apple 2 computer that takes something that was like still possible before but now like makes it available like reduces the activation energy and makes it available to a large number of people and human beings consistently underestimate how important those inflection points are um because like when you're around it people will be like it's not really that the capability is not really that different from before it's just a change in degree not a
10:55change in kind but sometimes a change in degree is a change in kind because it allows a community to form because it greatly increases the number of people that can be brought to it and I think what I'm excited about in particular with solvit is because the reliability of it as a tool is so much higher than what came before so I feel like we're on we we're doing these step changes not to say that like nobody else in AI is doing anything good just to say this is something that I think is particularly exciting um and has the potential for a lot a lot of growth so I was just I was
11:23very excited on behalf of everybody who's gotten a chance to play with this so far I mean I've consistent you know like I think we're very PR to that right like we the well I could just do Docker with a file share on a remote server like the the kind of programmer hackery people tend to say like oh it's not that different from oh the iPod yeah it's just like this with this software and this Hardware Edition or the iPhone
11:52oh we've had all of those things for 10 years or yeah Dropbox oh you know just ass Sy and a Cron job yeah which I guess is partly why Eric when you're telling me about when we were chatting early uh individually about your reactions to lesson seven you used an interesting word which is you just you said to me it it felt mundane and you had to remind yourself to pay attention is that kind
12:20of part of what's like what's going on here with like this sense of you actually have to pay attention what what did you mean by that yeah it it was um it's a bit like an ASMR video it's a little bit hypnotic I don't know if the other attendees feel this way but like it's a little bit hypnotic to watch you guys just every step just logically flows from the previous step and like next thing you know was started as like very basic
12:48questions about like how can I parse you know HTML and I'm like you know a lot of questions are questions I know the answer to like I know how to use a beautiful sou Library so like my brain's not fully engaged like it's not really paying too much attention but what's exciting about it to me and what I had to keep reminding myself is that first of all although this is a contrived example because you you guys do have the prior knowledge of how to do this and could have done it without the AI there's you didn't use that prior knowledge for this so so this is like a general purpose thing it could have been used to use any library to accomplish almost any
13:18task and also the fact that it's so mundane is actually what's exciting about it because when we're building something useful in the world you know the a project gets reduced to you know a big if some you want to accomplish something big in the world you have to break it down into dozens or hundreds or thousands of substeps and each substep is a chance both for something to go wrong to cause a delay or a chance for something to go better than you expected and actually get you earlier feedback like a big we
13:47can get into this like batch size and queuing theory and and uh and you know power of iteration all that stuff like anything that moves the date of feedback closer to the people making the thing is extraordinarily valuable so when I just look at this yes it's mundane because you're like okay every step in the process let's say it's like 90% faster and more reliable if we had a metric for like speed reliability is like combining those two attributes into one um it's hard when you're looking at it you're
14:15like okay this went from a 60-second task to a sixc task so I say 54 seconds it's actually not that impressive in your brain it's like well I may as well have just done it myself I had this there was like a a couple days ago I was just you know I was I was writing a shell script for something and I was like I can just do this myself the time is going to save from doing it the LM is not worth it I'll just do it myself and after like fussing with it for I don't know five 10 minutes where I got like a okay result I was like this is stupid I should have
14:45just done it properly in the first place you know I brought Claude Sonet into the mix and like two minutes later I had a beautiful perfect result you know I had an okay result myself using my like half remembered skill you know it's like oh you have to go look up this thing and that thing and the difference between even at that time I was like oh that was nice but I was like okay so the difference between 10 minutes and 2 minutes didn't strike me at the time is really big and I keep reminding myself no if that's consistent 80% those time savings will
15:13add up and and some people like oh you're GNA get the project done 90% faster that sounds pretty good but it's actually way better than that because projects are not linear it's not like there's a hundred tasks and you do them in order and then the project is finished most tasks are not like that most tasks go through cycles and loops and the faster you go through the loop the fewer iterations you have to go so it can actually even if you know even if you have a constant Improvement in every subtask the total time could be dramatically different that's that's what I had to keep reminding myself that
15:42this is the the fact that it's mundane is what makes it exciting that's what I'm trying to we should talk more about that Loop maybe next week actually because that's kind of a big part of why we started answer AI that would be interesting to dig into which is this premise of like maybe you could create a whole different kind of compy based on I wonder if people are curious about that happy happy to share the back story and everything yeah yeah um yeah jono and I were having a bit of a chat before
16:12about what we wanted to learn from you when something jonno mentioned was like when there's a lot of things you could pay attention to So when you say like hey pay attention like well like you in particular I know you get a thousand you know cold pictures a day you know the lane startup guy like H how how do you decide what to pay
16:41attention to you know how do you decide what to work on you know particularly in this field there's so much going on should we all drop what we're doing and study agentic systems because that's what everybody else is doing or like what you know yeah yeah yeah I mean to be honest I feel bad this is not the most useful answer but just I'll give you my honest answer which is that um I have the privilege of having a broad
17:09network of people you know who are genuine experts in the things that they know about and I've tried to curate you know relationship with people who have different expertise than mine so I I I very often don't have to actually make those decisions a very smart person who I know I've you know have built a pre-existing relationship with tells me look you got to pay attention to this this is pretty important that gives me I get these like Clues um to know that something's going to be big so like I didn't know anything
17:38about generative AI I had never even heard of it when um one of the investors who was helping the founders of anthropic spin out of open AI called me out of the blue and just said look would you do me a favor and would you help these guys set up their governance properly for this new AI startup that they're doing and I was like sure you know didn't I didn't know it was an Earth shaking you know Monumental new thing it was just so and I we met I was like well tell me what this is you know help me understand what it is so we can
18:07set it up properly and you know by the end of the meeting I'm like oh my God this is going to be incredible and of course and of course we got to do this properly or or the world is in grave danger so so I feel like I have the luxury of getting like very interesting things come to me but I will say that one of the things I found really difficult earlier in my life is in order to create the mental space to pursue those things when they come in I can't be out there chasing the latest fad the latest thing people are excited about so
18:35I try really hard not to get into or excited about anything just because other people are excited about it and try to really be disciplined about like why is this something that I'm excited about why is it interesting to me yeah I was going to ask like that that that group of people you kind of curated that you listen to I'm imagining based on what you're describing about how that's the things you're getting are different to the stuff that just happened be super popular is is that those people you listen to are not necessarily the people who have the
19:05biggest mouthpieces on social media yes or whatever I do know some of those people um but you know even those people what they say in private is so different than what they say in public that it's that's interesting yeah I don't I don't view their public pronouncements as very as very useful um yeah it's kind of it's kind of hard to explain like I I think having a sense of personal purpose helps a lot with this and this like I can remember going all the way back to the
19:35like the Proto Proto days of Lean Startup before he even I even coined the term or or even knew the vocabulary how to describe it I remember um going into a board meeting once and there was a big debate in the company about whether we should noton get it doesn't really matter what it was was like I was the CTO I'm I'm in charge of the technology and like we need to decide you know pursue this direction or do it this direction and I had this really strong conviction that was very counterintuitive that we should do things in this more lean start upy way
20:03rapid iteration MVPs customer feedback you know and it was very very controversial at the time and it wasn't like wasn't working very well company was in trouble and of course I didn't think it was because we were doing things different that we were in trouble we were I I would have said the fact that we're doing things differently is helping us discover that the business plan has this fatal flaw in it that needs to be fixed but another interpretation of that would have been like well we agreed to do things in this weird way now we're trouble therefore doing it in the weird way is causing the trouble that's a lot of people in
20:32business think that way and so I'm going to the board meeting I'm like prepping for the board meeting early that morning I can't couldn't sleep the night before I was very stressed and I was like I think there's a chance that I'm going to be fired at this meeting because what I'm asking the leap of faith I'm asking people to take here you know this is a mix of my friends and my co-founders but also you know professional investors and this is a complicated situation and like they might not agree with me and they might say you know what we had enough of this guy I wasn't that you know I was very young I might have been the youngest
21:01person on the team at that point and here I am advocating we do this unproven untested new thing and so I was pretty stressed about that I'd never been fired before you know I I really do not like to fail I'm very competitive I like to win I like to be good at my job you know especially in those days I was very focused on being good at what I do and I had this moment of realization that in order to be true to my own values and my sense of Integrity I was going to have to risk being fired and so I asked myself what are the
21:30upsides of being fired like what I know it's I know why it's bad it'll be humiliating and bad but what could be good about it and I thought to myself you know what I'm in this room with all these powerful people that are going to go on to do many projects in the future and if I just go along with what they think I'm a nobody they won't remember me I'll just be yet another guy who folded in the face of what they want but if I advocate for what I truly believe even if they fire me they're going to remember this interaction and one day in the future they may come a time when they're like you know what we need
21:59we need a guy who's irrationally obsessed with going fast and engineer productivity and Rapid feedback like they I I'll have like an issue profile they'll remember me as the person who cares about that stuff and like maybe that will ultimately be helpful now in this story has a happy ending because I wasn't fired but that effect that I like realized that I needed to have on other people it's like I need to be clear about what I stand for and what I'm trying to accomplish in my life and I need to communicate that to other people
22:27so that they know when to reach out to me so that like that's actually what creates this beneficial virtuous relationship is people know oh Eric's the guy to talk to about these topics but also he has a great interest if you have if you're working on this kind of thing he'll help you you know call him up so it's it's not random but it requires like a consistent investment in um standing for something that I mean that's yeah I really feel that and maybe before we just dig into that just let's just step
22:56back a moment and can can you give the Potted summary of like what is this thing you were standing for what is the what is the Lean Startup what is this you know you mentioned fast you mentioned iteration um what's the Potted summary of that for those that that haven't read your book yeah sure um so I'm a you know I'm a programmer by training if you have a image in your mind some of you may
23:24relate to this of like the kid in their parents' basement spending too much time on the computer you know not spending enough time with the other kids outside that I was I fit that stereotype to a te my my my poor parents found it very stressful um but like from the first day my dad brought home a beige IBM PC you know the original IBM PC in that beige case you know he brought it home for work he didn't know what exactly what it was going to be good for and I found out you could program it I was just a little
23:52kid and I it's all I could imagine wanting to do you know I was just I was obsessed with it from from that moment moment and I thought I'd be a computer programmer my whole life I um the one of the happiest days of my childhood was when I found out you could get paid for computer programming like I couldn't believe it was a job you could do for money that struck me as too good to be true like you know it's like to me it's like being paid to play video games so I was really Keen for a life of of computer programming and um I got into
24:21entrepreneurship kind of by accident it wasn't really my plan but I had done the like I was in college I had done the big company intern thing and I Was preparing myself to go work in Tech which meant to me go work at a big company and I was in college during the.com bubble so that's I'm dating myself here a bit but like the Doom bubble swept through and like my friends and I we didn't have any idea what a startup was or why anyone was doing it but it was all the rage everyone was doing it so we did it too it was a you know it was a total
24:49disaster I'll spare you the whole story but it was the start of several times when I and the smartest people I knew built really amazing technology and then nothing happened so like the myth of technology is you build a product it's going to change the world and the reality most of the time is you build something that nobody wants and you did it really efficiently you know and you're really proud of how techn how great architecture it was and like you know I remember working on our product we're like this we've spent so
25:18many years developing this incredible new scaling technology so this technology will scale to million the millions of users it will presumably have one day and then if nobody uses it then that was all waste so it's so like a lot of Technology Building is a total waste of people's time and I found that I always found that outrageous like it's not it's actually morally wrong to waste people's time like that you should treat human time I hired a guy out of Erikson to fast maale who had been there I don't
25:46know six years and after three weeks we he had released his first fast mail feature and he said none of my work at Ericson was ever released yeah God we used to at at IMVU we used to make people deploy to production on their first day as an engineer it was a requirement by end of day can't leave unless you do it and you know we'd we' pick something real small but we had something we called the cluster immune
26:13system which was like an auto deployment um integrated testing framework such that if you broke the site it would autor revert your change and we were really confident in our ability to detect problems through automated testing so we weren't worried about it and we and we would hire kids to you know we hire young people to do this no problem you know they'd be like sure topl my first day sounds great but when we hired someone who've been in a big company their whole career we have people that have like basically like a nervous breakdown being like you want me
26:41to do what and they couldn't bring themselves to push the button they just I had built up years and years of the fear and I remember them saying like look what if I what if I take the site down on my first day and I remember I to give this answer so many times like look honestly if you take the site down then sham on us for making it so easy to take the shite down we we'll be like oh congratulations we learned something really valuable we'll make sure that never happens again and you'll be in charge of making sure that whatever whatever particular bug you introduce is never know never happens again and
27:10people found it super stressful and I of course the people when I would tell them the story they would want me to say and they'd be like well has anyone ever taken the site down you ever had a catastrophic problem that was caused by someone on their first day and I'm supposed to say no don't worry that never happens you don't need to worry about it but it has happened it did happen sometimes like yeah you know you never like the the curse of software engineering is spooky action in a distance you change this one little thing over here and this other system collapses it does happen sometimes but anyway so I just like I've always really
27:39had that intuitive sense that was really important and um in the course of building IMVU this is in started that company in 2004 to give you sense of the timeline um I was just adamant that we were going to follow this procedure of um getting things out there faster getting customer feedback more quickly you know why it's just like a sense you had you're just like that it just I had done it the I had seen the exact opposite way fail so badly that I was
28:08like a bit of an extremist being like well whatever just do the opposite of I remember talking to my co-founders we had the previous company we were five of us were refugees of a previous company that had spent like five years and $50 million to launch its first version of its product and so we were like for God's sake let's just make some new mistakes and do it the opposite now what I didn't appreciate at the time time was I hadn't you know I hadn't been working in Industry that long but doesn't take very long if you are someone who naturally thinks things should be done faster more
28:36scientifically more customer Centric if you have any of these beliefs in your work portfolio um what you don't realize is you unconsciously build this framework where you're in opposition to a dominant culture that doesn't agree with that so let's say you want remember when we used to release software so infrequently that we would put the name of the literally the year it came out into to the name of the product right so like so in that world let's say you want to release monthly or twice a year you
29:05want to go faster than the dominant culture if you say I want to release twice a year and they release every two years you know they might compromise with you and release once a year so you get trained to be as much of an extremist as you can so so I would like I wanted to release once I remember trying to get people to do the agile Manifesto extreme programming think of releasing once a month this is back when releasing once a month was seen as really fast you don't advocate for releasing once a month I'd be like I want to release once a week I want to release every day and so when I was in
29:34larger companies I was just I was always the person pushing on that constraint and the organization would push back against me and we come up with a compromise it would be reasonable the first time I was in charge of my own team with these ideas I fell over I was like pushing I was trying to push against something that wasn't there and I fell right over and it actually like was very uncomfortable to me because people I would say I want to release as fast as possible and my team would say well how fast is that and I be like I don't know every day and they like
30:02literally do you mean every day and I was like do I mean every day like I don't know I never had to actually think about it before I didn't never really justify it so by being the person in charge it forced me to actually start to think from first principles what am I actually doing here like I had this intuitive sense this was necessary but why is it work what is it you know I I really studied hard to try to figure out why it works because I I could see the evidence with my own eyes and this I think goes back to your thing about Monday utility um we did it intuitively at
30:31first and we were supernaturally productive because just it's a way better way of working so much so that like investors started to believe that I had the magic touch that I could like I had like magic fairy dust that I could sprinkle on engineers and make them work harder that became a big part of my life in the foreshadowing things that would come to pass um but anyway but I was really determined to find out the reason why both first for my own intellectual curiosity about it but also because as we hired more and more people who had more and more diverse work
31:00experiences it was hard to get them to try it you know it's not like green eggs and ham just try it and you'll like it people want to be like but why does this work it's actually very uncomfortable for people to see something working that is contrary to the best practices they've been taught and so it here you described it as like more scientific more iterative faster yeah so like what is that can you like make that super concrete what is that sure sure let me just for those that haven't read the book let me just
31:29do Lean Startup in a nutshell like the theory of it just I'll do it very briefly the idea is that when you have a business plan to make something um we tend to think of that as a prediction about the future you know if you ever made a business plan you have like the hockey stick shaped graph in appendix B that says on this date and this year this number of units will be sold this thing will happen the plan is written as if you know what's going to happen in the future but actually in real life if you go around telling people that you know what's going to happen in the future you can be committed
31:58that doesn't actually make sense we don't know what's going to happen in the future so Lean Startup is just the acknowledgement of the extreme uncertainty of Entrepreneurship when we're doing something new that hasn't been done before then we can't extrapolate from our past experiences to know what's going to happen in the future so therefore we should treat all of the assumptions we call them the leap of faith assumptions that go into the plan as hypotheses in the scientific sense and so for each hypothesis we want to figure out what is the least
32:26expensive fastest way to discover the truth of this hypothesis so that if necessary we can pivot pivot is our most famous bit of jargon in the startup is a change in strategy without a change in vision and if you study entrepreneurship for any length of time you will see that all of the great companies went through this very strange process where they ran into difficulties and they didn't just give up whale but they kept one foot rooted in what was working and they had the humility to change the other foot to
32:56try something different and sometimes they had to do that mult multiple times and so the Insight of Lean Startup is that um for any project it's not it's not actually really startup specific if if you're building a product inside of a big company we're doing anything that's new big company small company nonprofit for-profit makes no difference if it's new such that you don't know what's supposed to happen and when um the right way to think about Runway or how much time do you have left to make this thing happen is simply how many opportunities to Pivot do you have left and so anything that brings you to the moment
33:25of realization of the need to Pivot sooner effectively increases your resources increases your odds of success because it's like getting more shots on goal you know imagine a game of chess you versus a Grandmaster but you get two moves for there every one move you'll never lose and and and and can you also do it in a like like as well as the big pivot thing is there also a smaller Loop within the loop where it's like okay you kind of like for example you're not quite sure what the company is meant to be
33:54called so every week you change it CH every week exactly yeah we call that the build measure learn feedback loop and this you know uh this is not our just topic for today so if people want to you can if we have time for Q&A people are happy to I'm happy to answer questions about this but I studied lean manufacturing Toyota production system and and that whole I think still very underrated Revolution cognitive revolution of the 20th century that changed how physical products are built how Supply chains operate and was a like
34:24an intellectual breakthrough that most people have never studied and do not have an appreciation for how it keeps them clothed and Alive you know every day so we one out for all the general managers out there who run this unbelievable system it's it's a work of art um but anyway one of the things that you you learn in Le manufacturing is the most important thing about a factory is What's called the fundamental cycle time how quickly between when an order comes in and we deliver a highquality product that meets the customer's needs you know in response to that and collapsing that
34:53time has all these counterintuitive benefits and my realization that became Lean Startup was simply that in anything we're doing um we take an idea we build it into a product or into code we show that to customers generating data and from that data we learn influencing our next idea that's our fundamental feedback loop and so between the time we have an idea how much calendar time elapses on the stopwatch when you want to have the idea and when you get the feedback about um whether that's a good
35:22idea or not um that is the fundamental thing we're trying to optimize for and so yes in the macro we have these big cycles of trying to figure out how to get to the moment of doing the pivot to make these fundamental changes but then it's epicycles all the way down you break it down you break it down you break it down you take the leap of faith assumptions you break it down you build what we called minimum viable products and we train people in the startup to think as small as possible and I think this is something that humans find very counterintuitive which is that when we
35:51have a um linear process of any kind the smaller the batch size of work the better so by delivering in small batches you have the opportunity to get feedback um sooner and make a correction and one of my favorite teaching tools which is no longer on the internet now someone needs to re-record this video uh uh the video got taken down years ago but I this is my go-to demonstration of this principle for many many years um there's
36:19this guy who because it's so counterintuitive so here's the here's the here's the scenario it's kind of contrived but like if I tell you look I need you to send out physical invitations to an event to a 100 people um what's the most efficient way to do that everyone knows the most efficient way to do that I print out a 100 copies of the invitation I fold all 100 copies I stuff them into 100 envelopes I lick 100 envelopes and seal them I stamp 100 envelopes and I write
36:48the address 100 envelopes and I put the batch of 100 into a mailbox and even though everyone knows this to be true empirically speaking it is actually not the most efficient way to do the task uh it is much more efficient to take one piece of paper fold it stick it in the envelope seal it put the stamp on address it and then put it in the mailbox and then do the next one and even right now some people who are listening to us are like that's the craziest thing I ever heard what is he talking about that's definitely not true right everyone has the human mind has very strong intuitions about this and especially the American mind the hypers
37:18specialized mind Americans are obsessed with this problem absolutely so there was this guy who actually made a video it's one of my favorite videos ever I wish I could find it now it's it's taking down like I said I've seen remember it it's great sat on camera and they addressed and stamped and mailed a certain number of letters they both ways with the stopwatch and it's the craziest thing because they do it once and it takes you know I can't remember now what it is you know five minutes and they do it again it only takes three and a half minutes and it's like and then of course
37:47the video is about where did the time go and anyway I won't get into queuing Theory that's not for today but anyway it's all counter it's very counter to of you spend a lot of time that you don't realize moving these giant batches of work in progress inventory around anyway that's batch size and that one doesn't even have the whole like this is an extremely complex process and by going through it each time exactly and that's a key thing so now imagine and this is what I used to teach people in the early days of Le startup I would have to teach this entrepreneurs all the time imagine
38:16in my in my story it's faster even if nothing goes wrong everything's correct but now let's imagine you get to the post office and you try to mail the letter and you're po you have 42 cents of postage on the envelope and they're like oh sorry it's 45 cents to mail a letter now sorry your letter can't be mailed when do you want to find this out or like what if the piece of paper is not the right size for the envelope anyone ever had this experience you want to
38:46find that out after you folded a hundred letters or after you folded only one if you try to mail one letter it's it's of course the more uncertainty there is the more things that could go wrong the exponentially more beneficial it is to do work in small batches now some people listening are going to be perfectionists about this and be like hold on isn't there such a thing as optimal batch size yes yes there's a whole Theory to this of course if you have to walk from your house to the Post Office three blocks away then you don't want to have to pay the transport cost of the batch every
39:14time so yes there is an optimal batch size and that's not important for our purposes because for most people their intuitive sense of batch size is just way too large and it's same it's same in programming like the more attempt in one Fell Swoop the more you're going to run into these issues versus if you're willing to break the problem down into the smallest units of progress that you can the more likely you are to succeed that's leaving aside the question of whether there'll be an error or a problem that will force you to rethink your
39:44approach I can see why you two are friends sorry it's a I can see why you two are friends it's a very familiar well I was going to say that let me ask about this I've got I've got a bit of a crazy direction to go here um Eric so with me for a moment sure sure I want to tell you about three places I've seen in my life two of which I did and one of which I've seen which seem to be other examples of um trying to create fast iterations trying to get prompt feedback so forth
40:12uh the first is when I competed in kaggle you know or I do any machine learning problem and all the teaching I've done from that I always say to people number one most important thing is submit every day no matter what you know submit something um and it's so different to what like when I've done kle hackathons so when I used to work there I would we would have hackathons and I would see people over a weekend and they would
40:41just work work work work work and at the end of 48 hours they would put in their amazing solution and they would have a submission error and they would get zero you know and I'd just be there like okay within 3 minutes I've submitted the sample CSV they Prov and then after 5 minutes I've replaced you know all the ones with 0.5s and then after that I change it for male versus female averages and yeah yeah yeah um
41:09okay number one number two uh I going to share my screen here not that you really need to read it but it's kind of nice to have something to look at um paulia so George POA how to how to do math problems how to solve math problems so it's like identify what you're being asked to do gather the info restate the problem create visual representations look for patterns draw on similar problems you've solved break
41:34it down execute it verify it document it evaluate it verify it consider alternative approaches consider both credential improvements next time go back around do it again uh jonno last lesson doing solve it for screen scraping and like everything we're saying solve it like write one to two lines of code see an
42:03actual result come back operate on that result you come back or fix it write another one or two lines of code um so uh Audrey has asked in the um chat if you were to Eric if you were to iterate upon the Lean Startup methodology by incorporating anything from the solvent methodology what would you try but I actually want to ask you are we
42:29all actually doing the same thing you me paa yeah yeah yeah and not just us um if you read read ta Ono's work um one of the Godfathers of Toyota production system or even shigo shingu um that that stuff's all been translated into English now or if you want something that's more fun um is this guy named Colonel John Boyd who was a American Air Force Colonel who developed something called the UDA Loop the O OA Loop and there's a wonderful biography of his life which is
42:58definitely the best introduction to the concept because you could just he lived a crazy crazy life and he fought in the Korean War and all kinds of it's like fighter pilot Theory right it was it was a fighter pilot death became the theory of maneuver Warfare which is how all modern Wars are are conducted although I guess with drones it's about to get real weird and different anyway um the these ideas recur all the time and um I think it's better to see each of them as an instance of a class that of the one
43:27thing that we're all doing rather than different things so like it's a little tricky to talk about it if you recognize the universality of it because so like you we were talking earlier um about this and we were talking about um you know the the use of Sol it by a computer programmer to make a new product or the use of Lean Startup by an entrepreneur to make a new company and I was like I don't think anybody involved would like no nobody on either side of that divide will be happy to hear that my view is
43:57those are not two different things but the same thing and we are so caught up in our cultural identities like no no no I'm not I'm not the same as that those people we tend to want to divide but I I really feel like if we're doing we're building something under conditions of uncertainty we are doing the same process whatever you call it so I'm like I I thought I think it's important to have jargon I'm actually a big believer in conceptual vocabulary and I think you know knowing Lean Startup terminology is super helpful but it's helpful because
44:27you understand the underlying concept using the words is not magical incantation and knowing the theory is not helpful like knowing the words of the theory are not what's not what helpful it's knowing how to do the underlying thing that is helpful so yeah I think there's like one meta discipline which does not have a a a real proper name to my knowledge that is like this this understanding of um you know of speed iteration you know I think it has a certain set of values that it's like a constellation of values and then a set
44:56of heuristics um for counteracting the the bugs in human psychology that make this difficult that's why you know yeah let's talk about that because I've accidentally been doing this for some decades now in various forms before solver it was NBD and before NBD it was a somewhat extreme approach to test different development uh and before that I was in Consulting and it was like doing everything in data
45:25analysis yeah um and at my startups you know I've never had a plan or whatever um almost everybody I've worked with has hated everything about this I mean part of the problem is I didn't really know what I was doing I was doing it I didn't really explain it but like yeah it was really uncomfortable to be like like it's like you know what it's like sometimes when potential investors reach
45:55out to us and they're like oh wowy R&D we love AI R&D can you talk to us we talk to them and they're like okay what's your plan what's your customer value proposition how do you know when you got we're like we don't know any of these things you know we're just figuring stuff out as we go people like you're actually fundamentally unserious aren't you you don't you don't know what you're doing you have no idea what you know beyond one or two steps from where you are now this is
46:23terrifying hey people don't love this right I mean if you experience this as well U okay I'll tell you a funny story actually um for those of you who've been in a venture Bat Company you will know this is not as common anymore but uh 20 years ago it would have been durur absolutely required that in order for a VC to make an investment in a Technology based company they would need their gray-haired technical expert to come in and do what was called technology due
46:52diligence on the team and there was a very certain type of person who be like a tech had a technology job in their life but they're like business adjacent they're buddies with these VCS they play golf together you know like just a kind you can imagine the kind of person who would get that kind of job and I'll never forget at imbu we used to fail Tech due diligence now I cannot if you've never been around this system you can't
47:21understand how rare it is for a startup to fail Tech due diligence it's actually like a a uh um indication of founder incompetence because like the reality is you guys would have been way ahead like he was some of the first internet video we were way ahead but the problem was I was I was young and stupid so this guy would show up and he'd be like now kid this isn't really how things are done you need to do product and Technical requirements documents and make sure you have a road map and you need
47:51to and I was too stupid and arrogant to realize that if I just said yes sir we're about to get right right on that we would pass the technical due diligence no problem but I didn't understand that and I'd be like oh no no no sir with all respect we've discovered this better way of working that's better than the thing you spent your whole career mastering and are an avowed expert in and would you like to come over here and see the evidence that we've made a discovery that's better
48:19than what you had you want to guess how many times someone asked to see the evidence yeah not one time so yeah I have a lot of experience with that phenomenon and um but it's it is uncomfortable for for all of us like I remember hope you don't mind me sharing but this for me this is like your great credit when I shared with you our plan to not use AWS or gcp but to create a being on a bare metal
48:49server using fast HTML that's never been used before using HTM X which is kind of crazy without any kind of like you know uh development plan or whatever and you're like how do you know it's going to work it's like well we don't we thought we'd give it a try and if there were mistakes we try to fix them and and I was like and you're like yes yes that is the right it's uncomfortable it's totally uncomfortable best practices are that way for a reason
49:18and you're like whenever someone wants to deviate I mean you have to think you I mean listen cultivating the skill of thinking for yourself is actually really difficult and most people don't do it it's actually what most people I think have in common with llms like most people are just unconscious stochastic parrots most of the time so like I think that's why llms have all the weird behaviors that they do because they were trained on a bunch of people like us who don't are not actually consciously aware of what they're doing a lot of the time so that's that's what we have to so so I would say like my advice to anyone who ask would be like you have to cultivate
49:48that discipline of Co of conscious awareness so that you can learn to think for yourself it's actually quite difficult and first step to admitting to to mastering that skill is to admit that you're not doing it a lot of the time but that still doesn't solve the problem of like but what do I do if I'm in a cultural situation where I want to get people to change to this way of thinking and I have a lot of experience with that and um happy to you know happy to to to share before I'd be interested to hear your thoughts because I want to come back to jonno's earlier question of like
50:16how do you decide what to work on to say like how how we're dealing with that at answer AI which is um for this year at least the whole we we've kind of skipped that question by saying to people look you know before you join here are things we believe you know we believe that cheap gpus are underappreciated we think fine tuning is underappreciated we think we should focus on building applications rather
50:46than New Foundation models you know here are some thesis or whatever like how does that sound like oh yeah I believe those things and then it's like okay what do I work on now that I'm here Jeremy well what do you think what would be fun go and work on the fun thing and so we we we've we hire um what we describe as tinkerers you know so like jno I think if I asked you how do you decide what to work on I think to a fair degree your
51:14answer would be like work on things that seem cool and fun slightly underappreciated and we see how it goes you know like um which is a slightly lean start upy kind of approach it's kind of like instead of coming up with a different company name every week it's like build a different little thing each week and instead of putting stuff out to customers we
51:43particularly focused on building stuff that we can use ourselves and then see if we like it um oh which reminds me uh this is actually something you've never seen before Eric uh and most of the almost all of the audience haven't seen before either I can I show you something new and get your thoughts about it uh oh is Eric crashed batter's sitting very very
52:12still and very silent oh no better all right okay what are your uh what are your thoughts jonno about what you've heard so far yeah it I like I like that description of my phone you talk to to Whatsapp him so yeah keep going sure someone in the chat put it well of like in place of having this like specific plan you have this like theme philosophy this North
52:41Star or this idea and so when you're describing like the yeah the vibe the answer ey working it's not that I pick from the total set of tasks that might be interesting or fun right and we've had hirers I really like to play with you know this all sorts of fun technology all the time right so there is this decision that's like okay does this kind of align with that core belief and so for Eric you can see his core beliefs like long-term value uh doing entrepreneurship better like so he can have that decision function in his head when he sees something should I work on
53:11this and for us at an we've got some ideas as well of like okay I can think to myself is this an answer aiish kind of thing and there's been projects that sounded interesting to me but that I've dropped because they weren't um and so we've got that like shared Vision which you have to build up so everyone knows what that means um and that's trickier you know I've seen people talking about like oh larger organizations having that communication is harder there's bureaucracy and stuff um but I think it is quite a nice hack to be able to say
53:39in lie of a plan that I can consult I can just ask myself is this an answer aiish does this align with our vision with our plan not our plan with our um Vibe um and that's a good way to direct the day-to-day decisioning no exactly it's interesting because uh um uh oh we nearly got Eric
54:01back yesterday uh we were all chatting within our company about our kind of oneye anniversary and reflecting on how I think yeah we all know what that Vibe is or what that thesis is but we've also quite consciously never written anything down because it's something that should evolve from a group you know rather than be demanded from on
54:30High um but yeah it's never been a problem I never felt like people don't at n3i don't know what we're doing um or don't know what we're trying to achieve yeah um yeah I'm trying to think uh other open questions that I have from the discussion I feel like for code execution that that feedback is often quite visible like oh I can see my models training I can see my oh Eric's backs I can I'm back I'm back can you
54:57hear me back yeah yes just talking about we're just talking about how answer AI this is coming kind of coming from the chat um this idea of having like a North star or a Vibe or an understanding of a kind of over Direction and how we definitely have that so sorry j go ahead well I was then going to ask like um you know if I'm executing then on a a technical problem within that right like I have some project that I've decided aligns with the an idea and I'm writing the code towards that it's very clear to
55:26me me when the code is working um I can see the execution results like this is what I was intending right the website is is up and it shows what I wanted to show um I so there's there's like that component of the feedback that I'm very comfortable with there's the like slightly higher component of like okay we kind of like as a team think that this is proceeding in the right direction like it aligns somehow with what our what our goal is um one thing I have much less experience with and and I'd be interested to hear your thoughts
55:53on generally is like what is that signal look like when it's now it's out there in the world and it's a larger startup or larger project and it's interacting with people outside because like code okay my code ran and gave me the correct result like that's very direct feedback versus like oh I released something yeah what's the feedback what's the signal there that tells me what to do next are you asking me uh yes yeah okay that's something that that we can think
56:22about I know so actually so I I want to push back on on one of the of your question in this way because it's actually the same thing it just seems different and the reason is um because you're used to thinking of what you're doing when you're programming or releasing something as you are building it and then you see what happens right that's natural and the problem with the framing of let's do it and see what happens is that you actually cannot fail
56:51it's impossible to fail because whatever happens you will see what happens happens and then you've succeeded at seeing what happened you did what you set out to do you saw what happened but when you are programming actually although you say I'm going to see what happens you have a very specific idea of what is supposed to happen and it's very easy to demonstrate that this is true if you've never you know if if you ever had a like non-deterministic or like super weird computer buug you've been you've been chasing where just like the
57:20computer does you know there's like a kernel bug I spent a lot of my life on was like a very very weird and know Linux kernel bug that like caused it to behave you know be put the CPU into the wrong floating point mode you know under very very weirdo circumstance so your computation would come out uh slightly wrong you know um anyway no it's like one things that nobody in the world could care about and like the answer the correct answer just round your numbers off and don't worry about it but like it shouldn't do that it shouldn't do that it shouldn't do that if you have the
57:49mental model of what is supposed to happen and you're the right kind of person then it drives you crazy and you eventually go figure out what's going on and then times in my life I've learned the most is when I've pulled on the thread of the sweater the issue is that when many of us move from the um technical domain into the product domain and the business domain we we cease to hold ourselves to that same standard we don't articulate what is um supposed to happen and therefore we cannot fail you cannot fail you cannot learn I think
58:17that has a lot to do with the really poor business culture we have like you know business people are not scientifically rigorous a lot of people you know a lot of business thinking is is pure um pure magical thinking so I I get it like and in fact in a lot of business context it's seen as um negative to write your hypotheses down and speak them out loud because you're actually like demotivating and reducing morale anyone ever been in that situation where it's like um there's
58:47this belief that if you push teams hold them to unattainable standards that will cause them to work as hard as possible you they'll basically die trying to achieve the possible that's better than if they satisfy themselves by achieving just the mundane thing so like that's a common business idea as um malevolent and wrong as it is so it's actually like a radical act to say yeah we're going to put this out there and then I want to Define what's going to happen now as soon as I say that to people they're like oh okay I got it I'm going to put it out there and I'm going to get a
59:14million customers it's like oh are you and so then like it actually requires some skill to learn how to set appropriate targets here because it's like okay in order to so and it's and the questions are easy to coach someone through this is actually very easy because you say okay great if you launch this and next week it has 799,000 users is that a failure because didn't have a million years and they're like no you kidding me we got 700,000 users right like and you just have to imagine yourself at the board meeting afterwards retroactively trying to justify whatever
59:43you did as a success anyone ever been in this situation right you're like right you thought you have a million users but you actually got a thousand users but you're like but man a thousand is still really good but if that's really good then why wasn't that the goal right So eventually we learn that that that making our goals in terms of masses of numbers of people is almost always a mistake because like predicting crowd behavior is actually super complicated and unless you're Harry Seldon like you're not going to know what's going to happen in the future much better to Define our goals in terms of per customer metrics always in least start
1:00:12we have the saying that metrics are people too and so the best metrics are ones that are defined by ratios or sums of whole numbers of persons and ideally the you know cohort by cohort relationship of like how many people who did one thing went on to do the next thing those become metrics that can be durable over time and then you be surprised how often technical people have implicit assumptions about what humans are going to do that they have not articulated even to themselves that
1:00:41turn out to be hilariously painfully wrong and that's how many projects go go wrong I'll just give you one example one of my favorites we once were working on a download page uh sign up sign up flow three or four page download sign up flow so step one customize your Avatar step two pick the I don't remember what the steps were anymore it's been a long time last step download the software to use your avatar we've done a lot of AB testing split testing for those that don't know basically a scientific AB test some people see one version of it other people see another version We compare it scientifically who's doing better or worse and you know should you
1:01:10choose your username first should you choose your clothes for like what should you do first or should you download first so you play around with those variables we got we hired a new designer to come in and redo the design cuz this whole thing was like very web 1.0 like Craigslist style like crappy looking thing and everyone was like look if if you had better design you'd have better conversion and I was like I'm sure you're right I'm just an engineer I don't know anything about this but um we're going to have an AB test for the new design against the old design and everyone was so outraged because they were like no we
1:01:39need to get this new design out there as soon as possible and you're slowing it down we need the most number of people to see this new design it's like harming the brand and this and that and and how can it not improve you know it's just like it's so illogical to have this new design I was like sorry I don't mean to be the a-hole but we are going to do it this way or I won't deploy the code like I'm just is my Engineers veto of the situation we will AB test so you will not be shocked to learn then when we AB testing it fewer people signed up with the new design we we had a similar thing
1:02:09actually I think at fast mail as far as I know we were the first people ever to AB test a homepage because it was a similar thing my homepage was very 1.0 um and a lot of folks are like it's got to be more beautiful more Jazzy you know and so you know we said to our customers like hey submit designs you know send them in that's very we got 10 and then I kind of came up with this crazy idea of
1:02:38like yeah let's let's put all 10 there and add a cookie so that people would see the same web homepage for the next month and we'll track how many of them sign up that um yeah never been done before as far as I know but um sign out rates were identical for all 10 within why not just leave the web 1.01 alone yeah so in this case in this case what was really funny about it was so we we we eventually so then it was a three-step process or whatever it was so I said
1:03:07look the reason I bring this up jonno to answer your question is that I kid you not when I presented the data that showed that the new design was worse most people's reaction was to ship it anyway yeah they don't care because it's more beautiful and so so first of all like we had this fundamental disagreement about like what does good design mean I would say good design is the design that causes the customer to do the desired Behavior most most efficaciously hopefully to their benefit and ours um
1:03:36but most people had a different idea about what good design is so anyway so so that was my first requirement was that no we won't we won't ship it till it's better and then I was like why don't we AB test each of the individual three pages to see where the problem is and people were even more outraged at that idea because now we're going to have an incoherent design some people are going to Getti beauti new page one ugly page two beautiful page three or God knows what else but it didn't bother me so we we did that anyway it turns out the problem was actually on the last
1:04:04page the the easiest of the pages the one that causes you to download the software we had a big drop off there and we couldn't figure out what the problem was and so I said look let's actually bring some people into the office for usability testing and we'll just watch them use the new versus the old thing and we'll just see if the problem can be deduced and everyone was like it it can't be that simple but we try it and I'll never forget on the the the web 1.0 download page was just a like plain text thank you for registering here's click
1:04:32the download Ugly gray you know button element click this button the new one was like a fully rendered beautiful graphical thing made up in Photoshop with a gigantic green curved rectangle 3D drop shadow button on it says download we bring people in for the usability test and we would sit them down and they would watch them go through this process it was one of the most painful things I ever experienced in my life to sit there and watch and they would get to this last page and they would just stare at the
1:05:01screen and they would as far as I could tell they would sit there and do nothing for as long as we were willing to wait they would just be looking around and finally eventually one of the engineers in the room would break and be like for God's sake what are you doing they're like I'm looking for the download button and we're like three qus of the screen is this giant but it's like how so we try to be polite well what about this cre like yeah there's this beautiful green billboard telling me to download I feel like an idiot that I
1:05:30can't follow its instruction and then we be like but have you ever thought about you know clicking on that to see what it would do they're like oh no you can't do that you know you know how computers are if you click and don't understand what's going on then you you know you could destroy your whole computer and then the IT person says it's your own fault because you clicked on something you don't understand I only ever click if I know exactly what's going to happen ah a time so now tell I tell you the story I'm sure lot of people hearing the story like what idiots one of the dumbest people if I had been there I wouldn't
1:06:00have made such a dumb mistake I know you're right you're a lot smarter than we were but I tell the story to illustrate that if I had sat people down and said enumerate for me all of your leap of faith assumptions about this project I don't think anyone would have said out loud if I show somebody a download button they will click it but actually that turned not to be a really important thing we had a very clear expectation but we weren't actually actually measuring to see if that was true so like like if you can just get yourself into a situation where
1:06:29you have a clear expectation about what's supposed to happen and you as quickly as possible determine whether that thing is happening or not that will take you a long way um towards being able to iterate quickly and that's that's why solve it is such a big deal for this because the more man hours are required to accomplish a task just the more difficult it is to get to that state and to get through that whole Loop cuz think about all the people that have to agree have to get if you can take a project that used to take five people and get it down to be able to be done by
1:06:59one person if you can take a project that took three days and get it down to one day every time we collapse the effort necessary to do one of these experiments we just greatly increase the amount of learning that can take place because learning happens between the ears of an individual person that's the only place it can take place and for remember like of course like I don't think anybody else in the call other than us knows this but suver was never meant to be something we released at all it was something we built only for this
1:07:28purpose and you know which was we need a way to really rapidly try things to see like hey can you use AI for screen scraping podcast transcription meeting notes summarizing books building web pages whatever like well if first you have to like spend three months building a new application for the one thing and then it then it's the end you're like
1:07:57well I guess not you can't use AI for that you know so we're like okay let's try and just build a kind of a canvas on which we can try using AI for nearly anything um and you know partly the reason I mentioned this is is like as an R&D lab we don't we don't get to do this we got get to say like okay we've got our product we've got our MVP we ship it to customers we need to make sure they can see the downlo button we're like what
1:08:26the hell are we doing so our approach is like we have to ship things internally that we all want to use so our our our goal you know for Sol it is everybody in the company has to choose to use it over other tools for most of the things they do in their day-to-day work and that's when we know it's successful and so at the moment not by not by mandate but because it's actually exactly and when at the moment people are like like wow
1:08:55Sol's going well and I'm like it's going quite well but so far it's failed you know and sometimes my team are like what do you mean it's failed I mean I think this is nice and this is nice and this is nice I'll be like okay when you did that how did you do it like I opened up cursor or I opened up a notebook or I opened up photoshops like okay you didn't use solit no I didn't you know it wouldn't have been as good for this for these reasons it's like that's okay you don't have to use solvit but you chose not to because it wasn't as good we have work to do still but we have failed so
1:09:23far um and you know maybe we can talk about this next week it'd be interesting to talk a bit about Virgil our Law Firm where we've taken this thing to the extreme where it's like okay we can't build a tool for lawyers unless the lawyers literally work in our company so we're like okay if we want to create really good legal tools we will first create a law firm which is possibly the most insane statements anybody's ever madeen when I first mention it to you
1:09:53you were like yes that's such a good idea I was like nobody else course we have to do it that way yeah exactly what what could be more could be more obvious well um while we're name dropping old old thinkers um this is also an instance we we're talking about John Boy we also should have mentioned Doug engelbart oh yeah you know from the mother of all demos Fame he had this concept he called co-evolution which was about the like hypers speed that can be achieved when a software development team in particular
1:10:20is building and using its own tools um and so it's just like yeah because the the the advantage of that situation is you are your own customer and so you you skip a step you don't have to go outside with the organization to do in the chat Apple famously very much does this it's like okay we want to build things that we're proud of that we want people to use I mean you know if we had said to people hey what would you like to better build code two months ago nobody would
1:10:48have said oh I want this weird hybrid between chat GPT and jup a notebook with a ridiculously small text at the bottom trickly trying to psychologically cause you to type less code at a time please you know so no in fact we know exactly what they would have wanted because that's what a bunch of the other companies that are working in the space have done have built the thing that's most obvious but that's not I have a thing to show you by the way Eric oh yeah yeah I want to see um which a couple of other people have seen but most people haven't because I wanted to
1:11:17get your kind of unvarnished take on this CU we've built another new thing and so you know increasingly I think you know most of the new things we build we're going to try and build it in solver or with Sol it or something so let me show you this new thing you're vaguely aware of it you just haven't I mean you are aware of it you haven't seen it yet and I'm wondering how you'll react to it when you see it um so just uh by way of background here
1:11:46um uh so folks on the call folks watching are welcome to try this out it's very early it's all running on our own infrastructure so apologies if it all falls over but if you add a secret called _ ghost or capital letters oh I know where this is yeah this is going to be fun with any value and then you turn off learning mode you'll get this new feature two new features actually um so um here's me implementing
1:12:15something somebody uh on Twitter a couple of weeks ago was like all of my employee you know anybody who works at my company has to be able to turn a structure like this into an output like this so okay I give it a try get solve it to help me um all these people online had these like multi-page things and I was like um it's just zip. map. conat anyway
1:12:42um so um there it is that's a bre first search um so uh the thing we've added is called uh ghost text ghost text or inline completion super common you get them in Gmail you I think you get them in Google Docs nowadays you get them in Co pilot it's like light gray text that appears but we've done something super weird with our ghost text um and true to kind of Solve IT style some of them are constraints and restrictions and some of them are the exact
1:13:11opposite um so let's try um documenting what this does um uh so we could say I'll use my voice here let's now describe what this function does
1:13:30colon not very good colon um and you can see what's happened it's added uh one line of gray text uh so I press right arrow to accept it and it's added one more line of gray text and I'll press one two accept
1:13:57it one more um this is how we used to have to prompt GPT 2 and three back in the day and like notice here now I'm going to backspace and say because I don't know what the star is oh that was amazing it actually guessed this is known as
1:14:31okay so will it ever stop uh yeah absolutely it'll stop when it's done I think we're done now right I thought I thought you were going to do this with a coding specific model what model is this um so um this is um using a quite High parameter quen model but what we've done is we've taken our
1:14:58year of research on like quantization and fsdp and quora and all that kind of stuff and got like a custom kernel a custom caching system all this stuff which actually allows us for the first time in the world to have a very high-powered coding model which is also very good at writing Pros about code and writing Pros in general right um
1:15:25um so for example let's switch over to code let's do a depth first search instead yeah um okay it didn't do the whole thing there right like um just the rest of the line yep so it only ever does the rest of the line and like uh you know we actually have an I'm feeling lucky buttton the I'm feeling lucky button is going to take a bit longer it's actually going to send it to deep seeks new
1:15:55massive how many 400 500 parameter billion parameter model and it's got to try to do the whole thing and so to do that I hit alt shift right done oh my god um that's fun uh oh we to run
1:16:17off um and then you know again we can say like um these two functions are demonstrating okay oh shift
1:16:42right yeah not bad so yeah it's rest in peace ever giving anybody homework ever again it's still interactive and it's still iterative and we're still in control you know um and we can this is crazy we can even use it for prompting um maybe I don't want to know I I don't know how to go further like what would be an interesting thing to learn about now so um how can we now go further and
1:17:11instead try so we're in a prompt so we're getting the AI to create the prompt for us oh my God um so like to me this is kind of interesting that we've taken a common idea go text in inline completion and we kind of asked the
1:17:40question like a what if that was way way way way better like what if we used a model that was vastly better than anybody else can use for this it's like okay if we wanted to do that we would have to spend a year building a new training system building a new way to quantize uh etc etc and and it also was like a year ago none of the models were using worked it was like assuming that this trajectory continues we would
1:18:07expect to see a 32 billion model that could properly do these kind of things and if so and we wanted to R it on a 4090 then we would need to blah blah blah so you know we did all that and it was also then like okay but we believe you should write a line at a time we believe the human should be in control we believe the human should be constantly editing things as they see things and so we've ended up with this weird new thing and there's a lot of
1:18:36little details there like you might have noticed for example that if it's just kind of going to the end of the line it just completes the end of the line but if it's got an opinion about like what indentation level we're going to go to next it'll actually go to the next line and just do the indent you know like there's all these little details to get it just so I'm not going to say it's perfect no but the affordances are cool it's pretty good you know and we've been using it internally and um Tommy built
1:19:04up some Telemetry you know and again this is like so we're not asking the customer we're asking like Tommy went through some Advent of code problems and saying like Okay he's figured out that something like 80% of the tokens that he's generated have actually been great text accepted suggestions rather than him typing um so yeah I guess I'm interested to hear your reaction here given after your reaction to kind of You Know lesson seven
1:19:33was here's um this approach being used to do real work and it's now I'm saying like okay here's the result of our last years R&D to kind of part of it is to build this feature you know what do you think okay I'm looking at the clock we're gonna have to queue up this these are be queue up topics for next week because great um we haven't talked with this crowd about the substrate yet but I think it would be very interesting and next week going to
1:20:03be the answer AI talk about answer AI yeah yeah because this this is the culmination of a meeting we had I guess just about 14 months ago a year just over a year ago with alesio yeah with alesio um you know I can remember it very clearly laying out how all of these pieces would come together um in order to allow us to build these applications and at superhuman productivity and people I think people
1:20:32will find that story really interesting but the the the little bit of it is um we've been talking about building anything startup project shell script change the world you know anything um and I think these are very general purpose tools use useful for a lot of things but I also have a fair bit of experience building really really hard things you know and put the stock exch like a stock exchange yeah yeah yes a new kind of stock exchange yeah yeah exactly it's you know just first time
1:21:00it's been done in 50 years and um I say that and I and I've helped a lot of other people build these very elaborate very complicated Mission driven companies it's because of my work with those kinds of companies that I got to work with a bunch of the AI um Foundation Labs um there is a special thing that happens when you get it right where like you've got the the two dimensions of this we call coherence and integrity it's like the um the equivalent of being in a flow
1:21:29State as an individual person like the organizational equivalent where you actually are every organization has a North star whether you admit it or not but if you know what the Northstar is and you consciously design around it it's a huge advantage and you know there's this old very cliche idea in U business jargon that you should skate to where the puck will be but you you build your plan based on the state of the world right now but how the state of the world will be and you so so there's this kind of like flow you
1:21:57get into where you are in alignment with your own team your other stakeholders your investors your corporate structure it's all aligned and also you're in flow with the state of the broader world where like you're doing something that's at the Forefront of a of an actual wave that's actually doing something useful it's not just some like crypto hyped up thing but like a real something real is happening and it's somewhat predictable and you can kind of surf that way when you set it up properly a you can you can accomplish extraordinary things that people normally consider to be
1:22:26impossible but this little example of the ghost text this is where my mind went you start to get these unbelievable serendipities where just in the moment where you need a certain thing it turns out you already have that thing because of something else that you did so here if we didn't if we hadn't already done the research on udora and fsdp and all that other stuff then we wouldn't have this we would have done okay like
1:22:53there substitute but like but but you know we built this thing that had this perfectly shaped puzzle piece for the very thing that we need we plugged it in and people look at that say well you just designed it to have that puzzle piece but it's hard to explain the extent to which that's not what's going on um it's just it's what happens when you're in flow the pieces start to come together right in a way that is actually beyond the human mind's ability to anticipate and so it's why I bristle at the idea when people say you don't have a plan as like a derogatory thing it's
1:23:22like no having a plan in the sense of being adamant that you are going to be in control of what happens and when is a kind of arrogance that suppresses the natural ability of serendipity to work in your favor so it's much better to think of it like an ecosystem designer or a gardener we are cultivating the circumstances I think Paul Graham Called It Black Swan farming we are trying to naturally curate the circumstances that will allow these
1:23:51serendipities to work in our favor and we try to conjure austral say again you can just go to Western Australia they have lots of black swans what you're saying one of our foundational business metaphors is a bunch of bunam yeah sorry I'm shocked to hear
1:24:10that um yeah okay well I mean it is 90 minutes so it is a good time to stop so the other thing I think we'll do um Eric is I'm going to we've got a lot of questions on the chat which I'm going to copy into a Google doc and I'll send you both and we because most of them we didn't get to and we might try can I just load them into Sol it and and do magic auto complete Eric ree would answer this question like exactly so you're not going to see anybody of us any ever again we're just going to give you an auto
1:24:38complete but you're right it is like GPD 3 isn't it like this idea of like you start the sentence and let it finish and let it finish it it's it's actually quite nice and I started playing with 54 yesterday to see whether the new 54 model does well at this and it actually does well or so so yeah we'll have to do a model router to pick the right model for the right we've got one it's actually it's 7 four lines of python um well you should try to make it more complicated so it's more impressive that's true so we can demand
1:25:06more money for more investors yeah happy happy to to answer questions um either next week or we can do it on Discord or whatever yeah um I'm excited to see what people have have been asking that's great that was great fun thanks Eric thanks Jon always always my pleasure this has been just an incredibly fun project to work on so I'm I'm delighted to do it and you know congratulations to everybody to getting to be part of this initial cohort I really I can't express how excited I am for you to be let's next week also talk about that LE you know what can we do with this cohort with this group of people that we yeah
1:25:35okay that sounds like a great great uh great rain check you take care hopefully my power won't fail next time right