0:05Nice to see you both.
0:07Um, new face name everybody I'm sure people are familiar with. [laughter] Eric, welcome. Nice to see you here, too.
0:19Um, those of you did did the preview course last year will remember that Eric joined us for two lessons there as well.
0:28Um, but for those that don't know Eric, would you prefer I gave a background about you or do you want to give a quick background about you?
0:38Uh, Eric is our co-founder at Answer AI of which fast AI is a part. Um, and it's probably fair to say still most well known for his book and theory of the lean startup that includes principles such as the um minimum viable product.
0:58Um uh but uh since that time, Eric's been spending a lot of time trying to go beyond just how does one create a successful startup, but one how does one continue to run a company that is genuinely genuinely valuable in the world. And part of that was to create the long-term stock exchange, which uh
1:28was the first new stock exchange since the NASDAQ was created. Is that right, Eric?
1:34Not a difficult thing to do, apparently.
1:36Just create a stock exchange.
1:40Yeah. So, it's about trying to create kind of a better incentive structure, I suppose, to so companies can succeed in the world um in more more than just ways that are extractive. Um and so Eric's going to be joining us for a couple of things um over the next two lessons.
2:03One is to talk about how about his new book um which is going well beyond even his impressive past to talk about how can maybe all of us think about how to make the organizations we care most about in the world be genuine value creators for society and not just extractive. Um to talk about how he wrote that book. uh to talk about how Jonno and I are reading that book with
2:31the help of AI. Um, but also to talk about how maybe all of us can take the skills and ideas we've we've been learning and the community that we're all building together to help make sense of the slightly unusual world that we find ourselves in and maybe take advantage of of of this background um to make our lives a little bit better and maybe even um create a
3:00positive impact on the world more generally. Um, how does that sound, Eric?
3:06Uh, that sounds like a very exciting a very exciting way to spend some time.
3:11[laughter] Um, Jono, I didn't forgot to ask you, are there things that happened this week that uh you noted? Uh, a few things.
3:22Yeah. The one that's um not necessarily just a um a particular dialogue that I can share but there was um a fun trend as advent of code has started. So today is the first day and a number of people have already leapt at the chance to um to get stuck in and then have been sharing their solutions. So there's probably a good 15 or 20 different takes on the problem if you scroll through the channel. So,
3:49we have a channel specifically for that.
3:52Yeah, we have a channel just for the each day of the challenge. Um, don't look at other people's solutions till you've done yours, but when you have, it's really fun to go, "Ah, I see. I could have done it that way."
4:00And Jono, this is not one of those things where like only PhDs need a play, right? It's like anybody can have a crack and particularly with solvable.
4:14Yeah. And like I saw I mean one of our one of the solvants got his wife who isn't a programmer at all and got her through part one and she was eagerly you know plugging away at part two. So it really is pretty approachable um if you're willing to yeah just tinker with it and play. Um the one other piece I did want to feature it's it's kind of cheating because this is someone from the previous Soviet cohort who since been building up their skills. Um, but this is chop.food
4:41and they have a system that uses a lot of the things that we touched on in the course. Web scraping, using AIS to extract structured data and to summarize content, building user interfaces with fast HTML. Um, so all of these pieces come together. And so if you have a recipe, now this is actually a good recipe site. It's got a jump to recipe button, but you know all these recipe sites they have, you know, when [clears throat] my great-grandmother was, you know, smelling the smell of garlic, yada yada. So if you just put chopf food in front of the um URL, it
5:10kind of extracts like what is the actual key steps of this recipe and the key ingredients.
5:14Um so I thought that was a very pretty web application that someone's made to make their lives better um following on the ideas from the course. So I I really enjoyed that as a demo of the kind of solvent system of um personalized software.
5:27Is that a fast HTML application? I don't remember.
5:31I believe so. Yeah, you should look for Pierre's message on if you just search the URL chop. You should find his backstory um in the Share It channel.
5:43Um what do you think, Eric?
5:47I'm looking at it right now. I just put one of my recipes in here.
5:51Yeah, that's really cool.
5:54Um, so today's topic for folks I was just chatting to Eric before the class about this for folks with a more purely technical background may seem a bit odd, you know, which is how to read a book.
6:14Um, but in fact, uh, let me just make sure. Yeah. uh how to read the book is itself the name of a book uh from 1940 uh written by actually the chairman of encyclopedia bratannica uh and the other guy and then another guy who's got degrees in literature and mathematics
6:43um yeah in the kind of the the humanities read learning how to read close read is kind of what do you think Eric the main thing it covers almost it's it's an incredibly it's incredibly highly prized skill in the humanities let's put it that way and not that many books are worth reading that closely I guess um and so one of the things I think I try to do in
7:09our community is to help curate books worth reading closely and uh we're going to look at two of them quite closely today a math book and Eric's book uh and a third one in passing um a [clears throat] programming book. Um but also I think anybody with a deep technical background will understand
7:36that papers are something that one does have to read closely and so Jonno is going to show us about reading papers. So, I think maybe we'll just dive in um because it's one of these things that's going to be easier to explain by example. And so, what I'm going to dive into is this book, Pre-calculus Made Difficult.
8:07Um, you almost certainly would have heard of it because nobody's heard of it. Um, but uh it's uh part of a trio of math books. Oh, well, I haven't read this one to be fair. It's really a trio of math books. Pre-calculus, Made Difficult, The Dark Art of Linear Algebra, and Full Frontal Calculus, which are kind of unique and extraordinary in my opinion, um, and well worthy of close reading.
8:44Uh the guy who wrote it, Seth Braver, says uh this book will be something of a connoisseur's choice. If you can appreciate the fine wine qualities of pre-calculus in a nutshell or girlfriend and Shen's algebra, it might be the right text for you. Unlike Girlfriend and Shen, their book is addressed to isolated Russian prodigies in correspondence school.
9:07Um uh but you get the kind of idea that this is a book that maybe is quite deep and different. And um uh my wife Rachel who has a math PhD you know when she's reading it she said well what a joy it is to see a preface that say this is a book meant to be read slowly and carefully and where he is assuming
9:34that you are a thoughtful interested person even if you don't have any particular mathematical background.
9:44Um so reading things slowly and carefully is quite difficult you know uh read with pace from pimple the ready you know when I admit details you should supply them when I use a phrase like you should verify I am not being physicious verify it um when you encounter something you don't understand clear it up now these are things that were hard to do um but actually now they're not so hard to do
10:13so what I did was I uh I sat down with my daughter um and we I just said, "Okay, Claire, read the book." I gave her the PDF and she started reading the book. And um whilst she started reading it um I uh opened up uh replicate which is a super great
10:42um thing. Replicate is somewhere where you can um basically start using models. Uh, what do I need to do to see all their models? These are your models. You probably want explore on where menu bar the menu bar at the top. Look for explore. These are like your published models, not the right menu at the top.
11:11Um, even higher up above where your mouse is, there's an explore tab.
11:17Explore. Great. Language models. Thank you.
11:22um image models whatever. So uh one of our wonderful alumni Vic has created a company called data lab um which has a a really uh excellent basically state-of-the-art uh PDF to markdown converter. It's um you can actually use it for free if you wish. Um uh so that's one option. uh or you can use it uh through replicate um which is
11:52what I did. So replicate basically gives you access to lots and lots of different models where you don't have to download it and host it and blah blah blah and it cost like a scent or two often to kind of use them. So Vick's thing I think it's like a couple of bucks per thousand pages. Um so I was definitely in the like okay I like the guy. I'll give him a couple of cents for the hell. So, it's quite nice because you uh replicates already loaded up on um solve it and so you just
12:20go replicate.run and you pass in the model that you want to run and then you can check the model docs to find out all the different parameters it takes. Um so in this case I grabbed the pages that I wanted uh CLA to read um which is chapter one. Um all the other stuff you don't actually need. I just left them as the defaults basically, but I put them there to remind myself if I wanted to change it later.
12:46Um, that creates a dictionary which I wrote to a file just so that then I don't have to do it again. And uh, here is the uh, here is the file. And as you can do, it's it's just an amazing job, right, of creating proper output. Just really cool. Um,
13:18okay. So then I just um don't quite remember where I left things here. I think this one I probably hid and I just had the printed out the output text for solver to see. Um, so by the time I'd done this, um,
13:43Cla had made some progress. Um, ah, now where is CLA's progress?
13:56That is embarrassing. I thought it was going to be in there. Maybe it's in this one. Sorry. Also not in there.
14:14Do you need me to stall, Jeremy? I can show a few different ways to No, no, definitely. Yeah, why don't you go ahead and um uh make use of this time other than people watching me try to find the dialogue I thought I had. So Scott in the chat comments, "Oh wow, I wish I knew about data labs yesterday before I created paper scrape, which is the something he was building." Um, but yeah, I can show uh I've got a dialogue just to go through um the many other ways that you can read. I I also am a fan of Vick's
14:44work, especially because his whole motivation is around um making information accessible and training better open models. It's very very cool work and it's a series that he's on. Um but yeah, just to show a few other ways specifically for um a paper that we'll look at a little later. Um there is um on archive often an HTML version. So that's not a generic uh reader PDF option, but it is there. Um and so you could use that HTML with Gina reader
15:12which we've seen previously to turn it into markdown or you could load the HTML directly and dump it into context. You could also use context kit. We actually have a read PDF function um which is just using some existing libraries and things. It's not nearly as good as data lab. You'll see the output here is just um here we go. This is the full um sorry the full text that it spits out.
15:37So definitely not beautifully marked down formatted but the contents are there [laughter] which is often the important part and the AI is generally able to make sense of the um the piece. Um and then finally for archive links again specifically there's a trick in that you can often download the source.
15:55So if you look for the link for text source there are um multiple then like individual pieces often with author comments and the type set latte equations and so combining those together into text and then adding that as a message or or um exploring it. Um yeah so these are all just some different ways I wanted to highlight. I almost always use whichever is easiest and most recent to my memory. Um but we've got lots of options available.
16:22Context kit is always pre-installed. Um if you've got replicate credits, the data lab one is fantastic and and cheap. Um Gina Read is free for HTML content. Um or you can just upload the PDF with the little upload button and then um read it in with yeah context kit to get just the text.
16:41All right. Well, I haven't as yet had much progress here, but I can always start again. So then people would get an authentic reading experience.
16:48Don't don't want to deal with an online service. There are also several Python libraries for PDF to markdown that work locally.
16:56Yeah, they're way less good though than stuff like like like way way way less good.
17:03Oh yeah. If you have if a PDF is complicated in any way, definitely you want to use dudes the real thing. But if it's just if someone just formatted text as a PDF to be annoying to you as often comes up in certain research without any names then it's extremely useful. All right. Well, I reckon that I messed this up. So that's fine.
17:32Instead, we will uh see how to do this from scratch. So, um it's a shame because I I won't be able to show you CLA's actual um uh progress. Um I click the learn button.
17:52No, it's fine. Um would you like me to um show some paper reading before we do math reading, Jeremy?
18:00Yeah. Okay, let's switch the That's a great idea. Why don't you do the paper reading first and then I'll do book reading.
18:05All right, sounds great.
18:06Um, yeah, the nice thing with today's lesson, they're not necessarily path dependent.
18:10Um, it's just some different examples of the flow.
18:16Um, so I'm going to go um hopefully you can see the the reading papers example that we were looking at.
18:25Um, so this was me figuring out what is a good recommendation. it um doesn't ultimately matter that much, but the goal is to look at a recent paper 11th of November from Yan Lun and colleague at Facebook or Meta I guess his last paper I believe since he's now left or at least announced he's leaving and this was one that we looked at during the office hours and Jeremy said oh I think only you know a handful of people made it so it' be it would be good to show
18:55how you went through it and also what we what we discovered in that process there. Um, and so to contextualize this paper, I could just try and show straight away what I did. Um, but then I know that the bulk of the audience is going to be a little bit crosseyed because it does cover on this is the end of maybe, you know, half a decade of papers building on these ideas um, in machine learning and learning representations. Um, and so I'm going to try and give a little bit of an explore
19:23of the background to this paper. Um and so yeah, we're gonna just dig into a little bit of scribbling on a whiteboard um to try and just high level context. Jono is like, you know, Yan Lun is one of the three cheering award winners, the kind of so godfathers of deep learning.
19:42He has some fairly strongly theoretical basis for kind of saying he doesn't think we're actually on the path to AGI with LLMs and that there's actually a different approach.
19:55He's uh his different approach is basically the paper you're showing is kind of the latest in the sequence of research that he's doing to try to build out this different approach. So, it's kind of like a very foundational piece of research, but it's also very kind of it's true research in the sense it's quite early. It's not one of these things that's at a point of like, okay, everything's now working, but it's uh this is where this is where things start, but it's quite a substantive
20:22breakthrough, I thought, in progress in this area.
20:27Progress along a path that isn't the main line, right? Like everybody else is maybe running one way, it's progress along an attempt to run a slightly different way.
20:35Um, fantastic. Okay, so hopefully you can see um I've got a little whiteboardy thing here and actually before the lesson I was planning to put in some screenshots from the paper and stuff and I tested out um Kimi Slides which is free at the moment with lots of Google's Nano Banana Pro um image generation and so um that's what I'm going to be scribbling over the background here. Um yeah, so as Jeremy was saying, there's this kind of like clash of ideologies with Yanlen saying look just predicting
21:03the next token in a sequence of text or next pixels in an image or next frame of video directly. Um, that's going to be very useful and get us very far, but it's missing something. And specifically, if you're trying to do kind of long range planning and interaction and understanding of the world, you want something better than just that raw like tokenized text or tokenized video representation. So, world models are something that comes up a lot um as is um yeah, discussions around like what are you trying to
21:32learn? What are you trying to model? And so Jeeper is his um joint embedding predicting prediction architecture. It's an attempt to frame a way of learning from data such that you build extremely useful representations. And so that's going to be the key of what we're looking at today is like what are these representations? What do we mean by useful representations? What is this new paper doing for how we learn those representations? Um, and so to just spend a bit of time on what this could
22:01mean, you can imagine trying to learn a like like something like chat GBT or or an image model. You could imagine training it on all the world's data by saying predict the next frame in a video or given half of an image, reconstruct the other half or given the start of some text, construct the end of the text, right? And so this is predicting or generating these are generative methods that are creating sort of the final artifact. Um, and the problem with this is if I give you a photo of a dog and you try and generate back that
22:31photo, most of the like quote unquote information in those pixels is going to be things like the fur textures and the grass and the leaves on the tree behind, right? And so that's at a completely different level to what we actually want, which is an understanding of dogness and dog earness and grassness, right? And so if we could have this latent representation that captured those important attributes but that wasn't so focused on these raw like pixel level um bits and pieces that would be ideal. So there's been other
22:59attempts to do this. You might know like clip and sigip these contrasted methods that try and align similar images similarly or images and text that are related should be aligned. Um, but that's got all sorts of problems around um what you see in your batch and and things that are associated but not necessarily like actually correlated or reliant on each other. And so Jepper is this alternate view where um let's say we have two frames of video or two versions of an image. The goal is to say
23:29um for each one here for each for each input I would like to produce an embedding and this embedding is going to be a compressed representation of our input. Right? Right? So the image might be 500 pixels x 500 pixels. This embedding could be only like a,000 values, right? So it's a highly compressed representation. And this is what we want for when we're imagining like, oh, what if I was planning, you know, 100 steps in the future. I don't want to generate 100* 500* 500 pixels. I just want to generate 500 like, oh, I'll walk over there and then I'll climb up
23:58the stairs. This more like condensed level representation. Um so if we have one input and we map it to its embedding and then we take a second input and we map it to its embedding. Um our goal is if those are related um I should be able to predict the embedding of one from the embedding of the other. So for video, I give you the first second of video. You could say, "Oh, there's a dog that's just run into the room." Like that would that idea would be your compressed representation and then you're saying, "What happens next?" And you could say, "Oh, the dog's going to skid under the
24:28table." Right? And so if you look at the first embedding and it captures dog running in on slippery floors and then you try and predict what's going to happen next, hopefully you've got enough information in that embedding to then capture the embedding of the next second of video which happens where the dog slips on the floor. Right? So the idea is we want to build up these really rich representations. Um, and so yeah, there's lots of like theoretical arguments for why this would be uh delightfully useful and delicious and
24:58better than all of the existing methods. Um, and they have shown various benefits of this, right? There's been papers all from Yanlen's lab showing that it does well on various image learning tasks.
25:09There was one called VJepper that showed the same for video. Um, they seem to learn faster, etc., etc. Um, but there's been a problem which is that you probably haven't heard of Jepper, right?
25:20um and not that many people have been using it out in the wide world. Um and the problem is the framing of Japer to say from one input produce an embedding that's useful for predicting the embedding of another input. That has a very easy solution that the model can come up with which is to say just map everything to the same embedding. Right?
25:39Because if I predict zero and then you say okay what is the next frame? Oh that's also zero. Like correct you've got a 100% accuracy at predicting the embeddings. Right? you're within some infinitisly small margin of error. Um, but that's not useful at all. That that collapsed uh representation is not actually capturing anything of value.
25:57Um, and so there have been all of these attempts within the Jeppa line of papers to fix this. Um, things like exponential moving averages where you have two slightly different versions of the encoder, one of which is lagging behind the other and being updated with the sort of uh regular averaging. there's been all of these extra attempts at regularization and stop gradients and duct tape fixes. Um, but it's been in practice fairly tricky to tame these models and to get them to actually learn the kinds of useful representations that
26:26we're hoping for. And so that's where we get to the current paper lea um has this idea of sketched isotropic gaussian regularization. And this is the kind of mathy term when you're reading the paper that might be very intimidating. And this is the piece that we're going to look at and solve it to try and make it less intimidating and to try and build some intuition for it. Um, and so the idea here, and we'll look at this in the paper and then solve it. Um, but we want to enforce that our embeddings don't all
26:53collapse. They don't skew in some weird way. We want to enforce that they're this nice smooth isotropic Gaussian, right? Isotropic, it's the same variance in all the different dimensions. Um, and you could try and do this by taking your model and embedding all of your data and then doing some statistics on that, but that's going to be incredibly expensive.
27:14And so the whole trick of this, um, paper is finding a way to do this one batch at a time in a way that's GPU friendly um, that's not like high compute, but that still encourages this desirable property. And a lot of the paper is going into why this property is desirable and why it's useful. Um but the core idea here like if you zoom out we're trying to learn um a nice distribution of embeddings or representations that capture the essence
27:43of some data so that we can use those for downstream tasks whether that's predicting what uh an agent might do next or classifying an image or understanding what's going on in a video. Um the goal is to try and build up these useful representations. Um okay so that's that's the background to to Jeppa. Um, hopefully that wasn't too long and boring, but that should give you the context then for when we say, "Okay, so let's look at what did we find in office hours." Um, and so yeah, this
28:12was one of the impromptu hangout suggestions. Someone said, "Hey, maybe you could read a paper." And this paper had just come out. Um, so I loaded it up. In this case, I use Gina. Um, and you can see here, "Hey, Sid, can you give me a quick summary of the paper?"
28:27Um and so it's gone and given us the background. Self-supervised learning is this idea of learning from the data without labels. Um that gets rid of these huristic that that others used to have to use um and introduces this novel regularization method. Um and combining that with the standard jeer ideas, it gives a competitive approach that has much less knobs to tweak and seems to do really well even scaling up to a couple of billion parameters. Um, and so then
28:55we wanted to look in closer, you know, why might you want these embeddings to follow this particular distribution? Oh, they've got various statistical arguments for why this is particularly useful for downstream tasks. Um, and if you had, you know, skew and high variance in some dimensions but not others, that might be worse. I can kind of get on board with that. Um, and so then, um, yeah, starting to dig into, okay, what does the actual like what is this magical thing that they do? Um,
29:25what does it look like? How does it work? Um, and so the idea is that we're going to take random projections. So rather than looking at um the whole highdimensional embedding space, we're going to take random projections and then along that line um we're going to use some very clever statistical tests to check for how desirable or how nice the distribution of our set of sample points is. Um and so then I said, okay, great. Can you show me the code? And in the paper, um in the paper they have this code. I think it's like um
29:54somewhere somewhere in the uh here we go. Uh nope.
30:00Guess I should just um search for Oh, here we go. def sigg. Um so this is just solvent reading pretty much straight from the paper. Um where it says here we go sigg. It added uh some comments but the code is the same. Getting the device [snorts] setting up some random seeds we don't need to worry about. Um getting ready to project our input to um some number of dimensions that you're working with. Um, and then calculating this a matrix
30:28that's going to do the projection and then okay, picking some points that we're going to be looking at for our test. Um, and then doing the test and then computing the distance and then returning the results. Um, and so when we were looking at this, um, full disclosure, I I maybe seen some discussion that there was something up with the code in the paper. Um, we calculated this a matrix, right? the um the code here that's overput and I I
30:58went and checked this in the paper. This a matrix is created but it's never used um except that it is now. So, this is the updated version of the paper, but when we looked at it in in office hours, you'll have to take my word for it. Um, it wasn't there. And it's like, wait, I'm not seeing that used. Is there an error? Um, so it's like, great catch.
31:17There is. Um, and it was able to say like, look, it thinks this is the missing step where we taking our full dimensional X and we're projecting it down. Um, and so yeah, it was pretty fun to like wait have this little like double take to say I see some lines of code there that don't seem like they're doing anything. is this solve it making a mistake and then to go to the paper like no actually um it's it's picked out the same error um that others have since noted. Um so then to investigate and to see okay like does this invalidate all
31:45the results? I I don't think so. Like they're generally pretty careful people.
31:49Um I loaded up our file creation and editing tools and said okay can you explore the the repo? So I cloned the um open source code that they have released for the paper and basically just let solve it go and it was able to dig around and view the contents of the directory uh load up different files and then pull out for us a here is the version that's built from the published source code version of the of the sigg
32:18function rather than the oh we read this from the paper and it's got a mistake version. Um, and so Solvent made it all nice and annotated and um, maybe a bit too verbose. I think I had um, default mode on. Um, but yeah, that was a really interesting exercise in taking a pretty complicated paper with a pretty abstract new formulation. Um, and then being able to go from that to a piece of code. Um, but for me, like I still don't feel like I understand it. Like I get I get the general idea. We're trying to get a
32:48nicer distribution. Um, but I feel like I don't know. I don't know about you, like if I look at this, this doesn't really like I don't have an intuition for what it does. I don't know how it works. I don't know what it would do on my particular data. Um, so the next part, and this is something that's really nice about being in an interactive environment, was to think like, is there a minimum demo that I can show? Um, and so I think I just voice dictated. I have this idea. What if we just create some like fake embeddings that are in a random or you know like a
33:16uniform distribution some other non ideal distribution and then we measured our sigreg measure of how good this distribution is and then we updated our embeddings with the gradient of that so that we kind of push them further and further into alignment with this ideal that's laid out in the paper. Um and so that's what we do here. We create some random samples like a thousand random samples with 32 dimensions and we lay them out in a uniform like square grid.
33:45Right? So this is not the nice smooth Gaussian that Yarn says is ideal. So what we're going to do is we're going to take a um yeah switched to concise mode and um together me and solvent got a nice simple version of sigreg that is the same algorithm from the paper but without all of the like multi-device multiGPU um extra fluff and then we create an optimizer. Now, if you haven't done deep learning, it's this is all a little bit scary, but the the core idea here is that we're starting with our initial distribution
34:15and then we are calculating our quote unquote loss, this measure of how good this distribution is. Um, and then we're updating the values that we're optimizing. In other words, updating our embeddings. Um, and then we are repeating that again and again and again. And when we plot, you can see they start out all spread out and then over time they get more and more and more mostly clustered around the center but still evenly spreading out. In other words, more and more and more looking like an isotropic gaussian distribution.
34:44And this is viewing two of the 32 dimensions, but you could plot any two and you'd see the same pattern. Um, and so now I have a picture in my head that is like, okay, it's not just some magical math function when I'm picturing using this as part of my loss function.
34:59Normally, you'd also have some other training objective there. I'm imagining that this part of the loss function is going to be nudging whatever my embeddings happen to look like so that they look more like a nice even smooth distribution. And I've got a little toy that I can play with and poke at. And I can start to understand like, okay, everyone here, I'm getting my input. I'm projecting it down to a lower dimensionality. I'm taking a few points along a line. I'm doing my test to see how good this is. I'm summing those up.
35:28I'm returning that as my sort of average error. And the lower I get that, the nicer my distribution is. And so now it feels like, okay, I'm finally starting to understand this paper. Um, and this is all done just piece by piece, you know, somewhat comfortably because this is something I'm quite comfortable with.
35:44Um, but at any point you'd have lots of opportunities to dive in a lot more and say, okay, actually I'm not comfortable with Foria features or with Mattplot lib or with optimizing in a loop in PyTorch.
35:55So there's lots of places where you might find a different level of wanting to understand it. But it was a very positive experience for me to take a paper, build an intuition um and yeah explore it interactively with some other people in the office hours chat.
36:15All right. We on to me.
36:18Yeah, your turn, Jeremy.
36:20I haven't defin I I missed that sadly. I wanted to see it. Maybe you can show me later.
36:24[laughter] Um, I tried to remember roughly what Cla said during our session and I've re recreated it. It won't be quite as authentic, but I did try to uh I did try to capture it. So, um,
36:45so there's a bit of a theme you'll see with Karen the the Eric book reading and this book reading and similar stuff that there's something that's just really nice to do. you um you basically want to give the LLM the wider context but also have it know what are you doing right now. So the way we did that with
37:14players reading was to have the whole markdown of the chapter in context. Um, but then I used the sharing that you've seen before. So I said setup share and I said start share. And I'm actually sharing the preview window.
37:34And so then I uh so when I did this with CLA, she's a pretty fast typist, but because this was our first session doing it, I didn't want her to get distracted by the typing. So I projected my laptop onto the TV screen while she was reading and kind of acted as her transcriber largely. Um,
38:01and so we started off with the PDF on the preface and basically the standard stuff. So in loading mode, who am I? How old am I? What am I trying to do? So players actually done most of this basic algebra stuff before. So, kind of tell it that. Um, and so it's able to see the intro to students, which is nice. So, it can kind of see what the book's about. And so, it
38:27it went ahead and read the screen. I said, "Okay, I can see you're reading this pre-calculus book. This is the description of what it says." Cool. Your dad's smart. That's absolutely right. Um, and so then she read it and we stopped when we got to the first questions and we said, "Okay, uh, great. Uh, let's do the questions."
38:59And so this is where there's a particular kind of book that I think works well for this approach, which are books that are maybe like deeper or a bit more advanced or go a bit faster than most people can handle most of the time because that way you can always slow down as much as you want to and ask for way way way more depth slowness you know but for the bits where it's like oh I kind of know this you can
39:28check you know it's like I think I know this give me a couple of quick questions so for for cla this was review so you know so I did say to cla okay don't like this is the thing that's been driving her crazy is Like with math, like a third of each thing in math will be stuff she's never seen before, and that often goes too fast. And so two/3s will be stuff she's seen like a thousand times, and that goes too slow. And then the slow bits, it still makes you do 20 questions on everyone. And every question is like still a whole lot of
39:57exhausting mental math and calculations and requires careful bookkeeping and it just doesn't work, you know, very well for her at least. I think it makes it boring um and tiring. Um so we're getting the AI to help us both make sure she gets it, you it all, but also not just do every single question.
40:24Um so we went ahead with the first question which was a geometric proof of the distributive property of um multiplication over addition. Um, and it's very nice. So, you can see B + C by A equals CA A + BA. So, I remember Cla kind of didn't quite understand the question. So, she just kind of wrote one side of it. And then it's like, yeah, you're really close, but actually this is not an equation. An equation needs
40:53two sides and it needs an equal side. And she still didn't quite get it. And she's still kind of doing the other side. And but you can see it's kind of helping her to see like, okay, this is here's both sides. And then they was like, "Oh, I get it. It's like proving something." So, and she's done a little proof course before, which is like proof by induction and proof by contradiction.
41:14She was like, "What is this?" Like some kind of proof by pictures. Um, so she's like, "Is this a proof?" It's like, "Yeah, pretty much nearly, not quite a proofish." Okay, that's cool. So, it's kind of nice to be able to just dive in a bit and it it tends to be quite good about encouraging you back in, but also it does tend to offer you can keep rabbit hauling if you want. Um, so then question two was like more of a just an
41:43explanation and you know little kids are not great at creating like detailed explanations of things. So, she was kind of like, "Yeah, I think it just says what it says." She like, "Yeah, yeah, it's fine." Um, and this is where it's like, oh, do we really have to compute all the following in our head? It's like, no. But Cla's like, no. It's like, yeah, yeah, fair enough. [laughter] No, no is fine.
42:07Uh, and so she was very keen to know like, okay, what what's actually fun? Um so this is a great way to if things are going too slow you can read faster by saying like let's skip ahead you know or but without skipping over important stuff and vice versa. So for for this bit cla was much more focused on going faster rather than slower. Um, and so it's like, okay, found a counter example to this, which obviously is not at all
42:37difficult, but it's good to, you know, I always tend to say to her, at least always do the last question of each section cuz they try to make that the one that really puts everything together and kids for some reason when they counter examples have to do like to the power of lots and lots of nines. So, um, that's kind of cute because I think the LM kind of recognized like, oh, it's so fun. You created a pretty big number.
43:00So it does a pretty good job I find of going along with it. So it is important to say honestly you know who are you?
43:08What are you trying to do? Like I've seen a I saw a bad case of this where kind of a humanities teacher told me they were interested in learning coding and I didn't really know anything about their background. It's like okay we have to tell it like how much coding have you done? Have you done any? And they were like oh yes I've done a little bit before and I was like okay I've done a little bit before but it turned out they hadn't at all. So it was a disaster because it's like well this is just like a for loop where the variables is like what's a for loop and what's a variable.
43:36So you kind of do need to try to make sure you set this up well. Um I know it's interesting because the next bit I'd never heard of foil before.
43:45Have you guys heard of foil?
43:47Is it first outer inner last? That we did that in um the Cambridge syllabus. I wonder if that's a newer thing than my time we didn't have foil that I remembered. But it's interesting Cla told me and the machine that um yeah foil she had recently done that foil.
44:08Learning is some random new word. I love that.
44:10Yeah. It must have been an AOPS or something because now she was like I don't know why we did that because we already already knew how to do this without this foil. That's interesting because actually that's what the book is right. This is what the book said, right? which is like you don't need foil, you know, it's like this is a difference where it's like okay, you're going to learn a specific thing which is a particular technique. And the cool thing about this book is that it starts with the distributive property because it turns out a whole bunch of the stuff you do in pre-calculus just fall out naturally from the distributive
44:38property. And so it's like 3x^2 + 4x^2 falls out, you know, foil falls out, um difference of squares falls out, you know, when it kind of goes through each of them. So it was nice because it's like yeah that's right this is you don't need to know this. Um but it does start throwing in like terms and so we had a good deep dive into wait what's a binomial what's a tromial or something with terms what's a term
45:06it's what's a part you know um so we got a bit further than this um but it was like a really encouraging process and one of the interesting things was normally I find with Claire to keep her interested in math I have to be like you know imagine there's a
45:34dragon and dragons are being hunted and we have to you know there the hunters are trying to shoot them down so they have to use trigonometry to figure out where to blast it you know all that with this we didn't have to like after 45 minutes she'd done like 10 questions and normally I was expecting it to be like nice some lemonade you say, "Okay, should we stop?" And she's like, "No, let's keep going." Say, "Oh, you want to do some more?" She like, "Yeah, yeah, I want to do some more."
45:59So, it's interesting to me to see how when things are calibrated at the right speed and not forcing you to do stuff that you don't want to do, it seemed to have a nice benefit. I've actually seen a couple of people on the um Discord as well saying like, "Oh, I never normally read stuff, but now I'm reading stuff." Yeah, it seems to be kind of quite quite helpful.
46:25I felt the same way. We'll talk about this in the writing se, you know, in a future future lesson. I felt that way while while using it to work through artifacts also that if you can if the difficulty of each thing is calibrated properly, it's just it doesn't feel the same as other kinds of work.
46:41And the thing with math education especially is that historically you haven't been able to choose that, right?
46:46I mean, maybe you can choose which class you start in, but then it's like, okay, this is the next hour of content mapped out in advance for you and if you can do all these in your sleep, sucks to you because you got 20 questions to get through.
46:59Um, so the dynamic and at some point everybody falls out of the system, right? At some almost everybody at some point it's like, okay, that went a bit faster than I could handle. Now the next class assumes I understood all that and the next class assumes I understood all that and suddenly it's like, oh, it turns out I can't do math. So I hit that at first year university, you know, because our um uh I did the first year of a new year 12 syllabus and the university had not updated to understanding that
47:28everybody's coming in with new information and so I didn't have any of the prerequisites and basically everybody who didn't have a parent that was a teacher or an academic dropped out of math that [laughter] year. Yeah.
47:42Yeah, it's really waiting out courses. You know, every every kind of teaching has this problem where like we've all had this experience where it's like here's like three problems that are like you're like okay yeah I got that. I got that. It's like now the next one should be obvious to you. And if there's ever too large of a delta between one thing and the next, you're just like what? You know, like that's I think of so many classes where I've had that problem. But now as an author now, I really I become much more sympathetic to how hard it is to create a smooth ramp of any on any topic
48:12because you can't customize. You have to create the same ramp for everybody. It's like a really like it's a one to many problem. And so the ability to customize the increments of difficulty I think is a really underrated um power that if AI structured properly you can um you can harness.
48:28And also I think it does help like one of the things we've been hearing a lot actually in this course from students is like oh I thought I got it you know and this is maybe more listening to to videos rather than reading text but until I dived into it with the AI and it's like oh turns out I only scratched the surface. Um, and just in passing, I'll show you something else that I did.
48:52Um, which is, uh, data lab itself gives you $5 free credits. And so, uh, data lab has their own API. You don't have to use replicate. Um, so I had not used this before. Um yeah, so here's their API overview. Um what requests form data blah blah blah. Um so I kind of thought h pretty annoying just to get a PDF. Um, so I um
49:48so one thing actually which is nice is um a lot of uh folks now are using the suggestions I coded in my llm's.ext text proposal which some of you might remember from previous. Look at this. People are sponsoring that now. That's hilarious.
50:13Um, so the LLM's text file is a way to, you know, some suggestions to make it easier for LLMs to use stuff on the internet. One of the suggestions in there is that every file you should be able to add or replace the extension with MD to get a markdown version. So, Bix docs do have exactly that, which is nice to discover. I don't think he even mentions it actually, [laughter] but I just tried it and it worked. And you can replace this with llm's.ext
50:40and that's nice as well. So I grabbed the markdown or otherwise you could use the um Gina um thing which would look slightly less good. But I click it should be pretty similar.
51:01There we go. Um, but yeah, the pure the original markdown is going to be the best. Uh, so I grabbed that. I copied it and I uh pasted it into an uh a raw. I actually like using RAW because it um doesn't try to do anything with it. Um, and then I basically just uh yeah, pasted in the the manual steps that they suggested to get the pre-calculus thing
51:30and that all worked fine. Um, and then yeah, I basically got uh the AI to write a first version of like, okay, wrap up the bits that except for the polling and a function. And it did a pretty bad job, but it was a start. And then I started factoring things out and I just had a little test underneath to see if it worked. And so eventually got to something I quite liked. Um there's some nice little tricks here for people
51:59interested in diving deeper. So marker has a whole bunch of uh parameters. Um so a useful trick is if you put those parameters in a dictionary and use fast cause special use quag dict and pass that in it will replace this quags here with this here so that I'll show you what I mean so that now if I run this
52:29and I type submit marker Do you see here all the parameters are listed? Uh or I could hit question mark see them that way. Um or if I start typing you can see they pop up here. So that's a nice trick.
52:52That's cool because then because it's in a dictionary and the API expects to receive a dictionary, you know, we can uh just pass in the dictionary, which is kind of cool. Um, and then I kind of added a couple of things here. I said, "Oh, you could pass in a file name or you could pass in a file object." So, if there's a file name, we'll open it up and make it a file object.
53:17Um, if there's a file, we'll create the little file structure it once, but you know, that's about it. Um, so that returns back a URL that you can pull and it'll tell you when it's ready. So, let's try running that. So, I don't have my key.
53:44There we go. Off it goes. So, it's not going to do the whole thing. It just receives it and will tell me when it's ready. I wonder if his API is having a problem because I've never received that error before.
53:58All the live viewers right now are hitting it with the Jeppa paper.
54:01[laughter] Um, so then I created one which you can pass in as many files and file names and URLs as you like. And this is where for those of you interested in learning about uh async, it's quite cool. You don't have to use multiple threads or multiple processes. You can use a special thing called gather which will do all of these at the same time. So you can pass in as many files as you like and it will return a list of things to pull. And
54:30it's like okay now let's create something which polls. Um this one's pretty boring. it just uh goes ahead and creates the client and goes a bunch of times and calls that URL and then has a little sleep after each one. Um, and so when it's done, you get your result. So then I did one which can pull multiple ones. So this takes the exact dictionary that our earlier one took and gathers
54:58all of those results. Um, so this one will wait for Olathan to finish. Um, and so I just gradually built this up, you know, and then it's like, okay, let's now do both. So this one submits and then polls. Now, this is a cool trick. Um, here you can see I've used delegates. So my quags here are passed to this function.
55:23[snorts] So that means that the defaults and names and so forth of them are going to be exactly the same as they were here. So that's what delegates does. So if I now say convert PDF, you can see they're all listed there. So and so the the use kwog stick is basically delegates, but you want it as a dictionary instead of to pass to a different function.
55:50Yeah. Neat. Delegates is one you're going to use much more often. Um the use quag stick is kind of going to be more if you're using some kind of online API. They do tend to take a dict.
56:04Um so then do multiple PDFs.
56:09So you can see how building it up slowly I was able to check each step. Um and then finally PDF to MD which actually saves it. And so now I delegating to see I'm calling convert PDF. So I'm now like okay well that delegates to convert PDF. And then this one uh delegates to convert PDFs. So it only changes the signature. That's all it's doing. And then ran that. And the nice one about this one is it's saving
56:38all of the images as well. Now um I used AI a lot here, right? Right. So at each point I basically said to AI create a function with this signature and this name that does this. And then I kind of refactored a few times and I said okay create an example. Refactored that. And at the end I said how does it all look?
56:57Like literally that's all I said. How does it all look? And every time it said these things look good. You know these things look like a problem. Um the first few times I said that the things that said look like a problem actually were problems. The problems became increasingly minor but it still said well let's try it right now. Um, it always says something's a problem.
57:21So, you have to recognize at which point it's fine. Um, there we go. What's working well? And it always says also what doesn't.
57:33So, in this case, it's all the read timeouts. So, it's pretty obvious. Um so this is the reflect step of the polyier process right like yeah understand the problem do it step by step and then at the end this is like okay are there better ways I could have done this yeah and then I said uh hey can you now create a module level doc string and I posted it at the top and then I said can you make a um done all for me and it popped that at
58:01the bottom and then what I did for each thing that I you would need to rerun this I clicked on the little export button, right? And then I hit export this. And that creates a a gist with an ID. And then once you have that, you can then send that to anybody and say, "Here's the gist ID."
58:36And what they then do is they import from dialog helper and then say import gist and pass in that ID. So let's try running that. And you can see it's imported a module called Dlab. Now you can start using that as just dl dot whatever on PDF. Uh or you can say from dlab import star.
59:18And if you do that and in Python if you have a special variable called done to all which we do which contains the things we exported it will make available only those things directly and all of our docs and everything are all there.
59:38So Jeremy a few lessons ago we did ndev and we built the full package and specified dependencies and put it on pi. This is the like I just have a few functions that are useful. You can just share it as a gist and then use import gist and that's what we do for a lot of one-off scripts. That's what we'll use for the demo in the second half of the book.
59:57Yeah, this is [clears throat] super light. Exactly.
1:00:01Just ran convert PDF and so yeah, anybody who wants I'll share this um now because uh now you all know you've got five bucks and that's enough for over a thousand pages. So that'll keep you amused. Uh, and you can use this gist and away you go. You can start converting PDFs. And I did realize the 4,200 page Da Vinci Resolve docs would still cost like 20 bucks or something. [laughter] But if you're not trying to do the Da
1:00:30Vinci Resolve docs, you'll be fine. But it really is cool the way um it extracts all the images out and um includes those you know in the um yeah that's in the PDF in the mark markdown and then and then when you if you say you know markdown or whatever it's all visible it's really really sweet.
1:00:56Uh yeah. So that we've now used because um Ho has created a gist which you can run to get Eric's book. Um, and of course these you can always go and have a look to see what the gist looks like.
1:01:31Um, so as you can see, he's actually got the whole URL. Uh, oh, if you're watching, you don't need it. You actually only need the ID. Um, and so here is the code that he wrote to grab Eric's book and turn it into a dialogue.
1:01:46You might find it interesting to see. We basically he uh he creates a directory and then he calls the um NV format official functions to create a notebook and for each one then to create a markdown cell and then he writes the notebook to there and prints it out. So
1:02:11um to oh here it is to use this You will need to go to that address which is uh in the discord and you'll log in. You need to make sure you log in with the exact same Google account that you use for the costs. Um and uh O has created a little fast HTML.
1:02:35Um as you can see it's actually running inside a dialogue. Uh definitely dog fooding here. um that uh will check that oorthth and then if it all works out it will give you your own special API key which you can then put into the book API key secret. So you can kind of see all the little pieces coming together. We've got an import gist. We've got a fast HTML app running in a dialogue.
1:02:59Um and when you've done that, as you see, it's going to save it to a folder called book. The forward is called chapter zero. And Eric, it's your book.
1:03:19Hey, that's pretty cool.
1:03:22So, um, you'll see here it's, um, basically what it does, it took the markdown from Eric, uh, split on two carriage returns or two new lines, I should say, um, and put each one then in a separate note. So this is a good trick because here's another example of a dialogue for reading a book that's very very very worth reading deeply which is hyperdia.systems.
1:03:50Uh you can buy it but then you can also grab the online version and use it in um solve it. This is an extremely deep book. Uh and so you can see here is a version of reading that book in the same way and the idea is anytime you're interested.
1:04:14Okay. So green green is the book text you can ask oh who's this Ted Nelson guy? What's Douglas and Tim Berners Lee got to do with it?
1:04:27Uh who else are these people? Wendy Hall. Uh can you show me pictures? I say, "Whoa, cool. Tell me about this picture."
1:04:38Um, so this is um you can go as deeper or shallow as you like, you know, but actually because uh Carson, who wrote HTMX and co-authored this book, knows everything about everything and he's a super interesting and thoughtful guy, you'll find in this book any of these things that you want to ask about, there's going to be deep rabbit holes that are fascinating. Uh Ted Nelson, you know, there's a whole topic that could keep you busy if he is. Uh definitely
1:05:07Dipos ankle, one of my heroes. Um yes. Uh and this is actually in a next box.
1:05:16I I should mention that unlike the fantastic Hyperdia systems book, um Eric's book is neither released nor free. Um it's free to everyone in the course of course but um yeah just worth saying right now before anyone skips away um please don't share you know the the whole thing as a markdown file with everybody on the internet or anything like that.
1:05:37You want to expand on that Eric?
1:05:39Yeah also we'll um ask you are you in America and if so can we have your address and we'll give you a free physical copy of the book. Uh if if you're not sure if you want it just say yes anyway. Somebody in your life is going to want one. And you know it's we would like to give you a copy of the book. So if you're in America, please give us your address. Uh we're not going to send anything to it other than a free copy of Eric's book.
1:06:04[clears throat] Anything to add, Eric, about that?
1:06:06Oh, I'm glad. Thanks for doing that. If you're not in the US, we'll we will find a deserving person to donate the copy to just because shipping logistics, we can't send it everywhere. But I I am really excited uh for Bill to get a copy of the book when it comes out next year.
1:06:18And I'm also really excited to see like the feedback that I get from early readers is so incredibly valuable. So I hope people will uh you know who who are interested will will engage with this lesson and and do share your dialogues.
1:06:28I'm very curious to see um see what you think. It's been it's been extremely revealing to me. Um because we were talking about this idea of close reading. um being able to close read with Solvent, you know, allows people to um get a lot more information and value out of reading a text like this, but also it's incredibly it's so rewarding and useful to the author to just see what are the things people needed to ask more information about because one of the hardest things as an author, so we're talking about creating the step levels of difficulty. It's just you
1:06:56don't know what references people know, like what names, what concepts, what do people already know. You have to, you know, try to meet everybody where they are. It's it's it's super challenging. the the feedback that even if you don't think your dialogue is very interesting, I promise you I will find it extremely interesting. And so if you're willing to share, I'm extremely grateful.
1:07:13So I'm sure we will have lots of threads in the So let's put threads in the um lesson specific lesson 9 specific uh discord channel. Um and um we'll see where we go. you know, we'll start with just one thread for people asking about the book and if it ends up with different chapter threads, we could do that. Um, I created on our internal Discord a thread for my questions and
1:07:41comments on the book as I read it and so far up to chapter six. I think Jonno told me I've got 260 messages in that more than that even.
1:07:50Okay. [laughter] Um, people are asking Jeremy in the chat where they're supposed to send their email address and in particular they're not supposed to post it in the YouTube chat. We just post or someone share the URL.
1:08:03Yeah. So, the URL is the first post in the first post in the Discord uh channel for this lesson. And when you click on that button to log in, it will ask you to log in with your email address.
1:08:19Now, Eric, before we dive into Jono and I experiences with close reading of your book, maybe a good place to start would be to talk a bit about like why why you wrote it because writing a book is a you've done it before so you have no excuse for not knowing. It's kind of a horrible experience and you did it anyway so you must have really wanted to say something.
1:08:46Yeah, you definitely well it's like doing a startup. You it's really only worth doing if the if the alternative is intolerable otherwise the pain will get to you. Um, for me, you know, I I wrote the lean startup 15 years ago, and so I've been at this a while now, and so I've helped a lot of people create a lot of really incredible companies, of which I'm I'm very proud. And not to take obviously the real credit goes to the founders who do it, but people keep telling me that they found the ideas helpful, which is great. But then I kept having this experience where people
1:09:14would build something worth protecting but then have it taken away from them either because you know outside forces intervened and kicked them out and you know turned the company into something else but also sometimes because they just lost control of it like Frankenstein's monster where it no longer reflected the values or priorities that they had in mind. And that phenomenon, you know, has been driving me crazy for a long time. It's it's related to the short- termism of of of so many things in our society. It's related to many of the dysfunctions in
1:09:44our capital markets and financial system. I just kept seeing symptoms of this same thing over and over and over again. And I just felt like we really have to do something about it. And before I really understood the systemic part of it, I tried to build a stock exchange and I've done all kinds of things trying to see what I can do about this. I kept coming back to this fundamental question of why do we structure organizations the way that we do? um you know what is the origin of this set of practices and is it actually any good if it's leading to these terrible outcomes and spoiler alert it's
1:10:13not like that's like hard even for me to get anyone to believe and that's why the book is as long as it is is really trying to make clear that like a lot of the way that the world is today is not really for any particularly good reason and could be quite different and it doesn't require us to wait for some political revolution or policy makers to decide or courts or anything there's nobody we have to wait on but we who are the builders um can make decisions here that are different than we make today that would have really profound impact.
1:10:40I find I find when I tell people that about almost anything a very common reaction is either uh like anger, annoyance, um uh beusement like to I find when I tell to people about any specific thing the world doesn't need to be like this. It could actually be different.
1:11:12It's quite unusual for people to be like, "Oh, cool. Tell me more." Like, is that something you've Oh, very one of the most rare reactions you can get. I think it's part of why we get along so well. Um, I and I have a theory. I have a psychological theory for why this is. I actually think psychologically it is very painful to confront the fact that there's this incredible cruelty and waste and just like terrible things happening in the
1:11:40world that don't have to be like it's much better it's much easier psychologically to assume that it's inevit even if you're really angry about it and you're just like or like there it's there's impossibly powerful shadowy forces that are behind it or whatever explanation you give yourself that takes away your own sense of agency. is a form of psychological coping. That makes a lot of sense to me because it's actually I think that's right.
1:12:03Yeah. Ursula Kawwin has an incredible um short story about this. But um really which one that the people that what's it called?
1:12:11I'm blanking on the people who walk away from omlas. The people who walk away.
1:12:15I haven't read the ones who walk away. Right. What an incredible story.
1:12:18Um and so so it's really tricky because so you asked me why did I write the book? The specific reason I wrote the book is that after [snorts] I started to learn about this stuff and realize that there were these solutions that were like not only like available but but relatively easy to access once you know where to look. Um I started telling people about them. People call me my daily practice of my life still to this day is people call me basically every day and ask me for help starting a company. That's probably the most common work thing that I do every is my
1:12:48favorite part of my job. Very very exciting. Sometimes scale a company, sometimes work at a big company and restore it. like most often to start something new. Um, and I will say this is true by the way. I I have never experienced me saying to Eric, oh, I met this guy who's thinking of starting a company. Can he call you?
1:13:05Like Eric's answer is always yes.
1:13:07Always yes. That's that's just pure joy for me. I love doing it. And what's so fun about it, I think I must be like cognitively impaired because I for whatever reason when people pitch me on something, I just always assume it's going to work. Um, and so I get a lot of joy out of imagining what the world would be if it worked. I I don't say that it will work.
1:13:28I just I can imagine that it will. And so I'm just like, well, let's talk about this as if it's going to work. What are we going to do about it?
1:13:34Um, instead of like the typical response of of anyone trying to do anything new, which is like skepticism, like how do you know it's going to work? Who are you? How dare you? Whatever.
1:13:41What if this happened? You know, somebody else tried something slightly like that once and everyone knows you think you are. Um, and what's so interesting to me is that at the end of these, so I started incorporating not just lean startup advice, but I like hey has anyone talked to you about your corporate governance and most founders are like my corporate what sounds like boring.
1:14:01You have an operating company right now.
1:14:04You have a corporate company, you have people you employ, you have investors and you don't know it's like you don't know the constitution of your own country that you live in. Like you might want to read it at least one time, find out how it works. Anyway, so I started explaining to people, you know, that there this is the way that it works in their company and they would be like shocked like I'm telling them this like weird horrible secret. They're like this can't you can't possibly be right. So we would go through these rounds and I sometimes would have to be like pull out their their their charter and be like is this your signature right here on this document? Like that's what it says.
1:14:32Anyway, can I give an example, Eric? So from from answer AI's founding and this is an example that also turns up in the book is Eric said to me uh Jeremy you might want to think about making uh answera a public benefit company and I was kind of I generally assume things are right if you say so it's like well do you know why we should do that it's like I don't know you said it's a good idea I'm happy
1:15:02to do that like Oh, you know, it's like, no, go ahead. I mean, I'm interested. I do want to know. So, what why why do I want to do that, Eric? You say, well, what if somebody, you know, comes along and tries to buy your company? Say, I don't want to sell the company. Say, ah, well, what if they're an evil company that wants to take all of your treasure technology and use it for the things you most hate? It's like, well, I really wouldn't want to sell it. It's like, well, what if they offer you $1 more than the market price? Like, I still
1:15:31would want to sell it. Well, did you know legally you might be required to say? I'm sorry. What? Yes. Uh if if if you create an anti-smoking company and Philip Morris comes along and offers you $1 more uh than the market price uh in order to use your propaganda techniques to sell more cigarettes to children, you can actually be legally required to sell to them. Unless you are a PBC. And it's like, all right, Eric, what do we have
1:16:00to do? How do I get the process started?
1:16:03It's like, well, we paste this one paragraph at the top of the document you have. Okay. And then what? It's like, no, there is no then what? That's it. It's like like Eric, why doesn't everybody have this paragraph at the top of that document? It's like I don't know, Jeremy, because it's not standard.
1:16:19[laughter] Almost every single thing in the book is like this where people like they start by being this is an intractable impossible problem. And you're like, "Well, would you like to know the solution to it anyway?" And they're like, "Well, sure, but I'm sure it's going to be impossibly difficult and come with tremendously negative trade-offs." And you're like, "Well, actually, it's there's actually a YouTube there's a YouTube channel where the guy just his whole his whole stick is like, "No, actually, it's going to be barely an inconvenience, if you know that meme." And it's like people can't believe it. And I think it goes back to
1:16:48the pain of it. telling me this whole time this could have been different if I just like checked a box or made a you know this made a two-page legal filing or like there's these things that you just you do and then they're like well then people are like well it must be super risky and we don't know it's unproven it's like nope some of these things are more than 100 years old and have been like extensively studied some of them in the book have whole branches of academic research dedicated to studying them and if you go talk to the experts on these different topics and ask them like do they have the evidence that this way is better than the standard best practice they'll all be like of course everyone knows that like
1:17:18well does bother you that even though everyone knows it, everyone still does it the other way. They're like, "Yeah, come to mention it. It is kind of weird, huh?" And it's just like they they just have this knowledge that nobody nobody knows. And yet, if you tell your own lawyers or if you go to your boss and you just say, "This isn't really really about startup." You go to your own organization and say, "Have we done this?"
1:17:36You say, "We're missiondriven. So, have we done these things to be mission driven?" Most people will just be like, "Oh, no. You I've never heard of it."
1:17:43They go to their lawyers like, "No, you don't want that." Whatever. It's just like um it's really it's really frustrating. And in being in the position of giving people advice, the end of almost every phone call I had starting like 2 or 3 years ago, people would say like, "Oh, that sounds really interesting. Is there anything more I can read on this subject?" And let me tell you, as someone who gives advice, you really don't want to say no when someone asks you that question because it makes you sound like a total lunatic.
1:18:08Like surely if this is a good idea, there would be copious things written about it. let alone like maybe at least be like actually I can give you like some manifestos to read. There's a couple legal papers you could read.
1:18:20There's a few like case studies of individual like there's not really a systematic treatment of this issue. And eventually like after saying that like 25 times I'm pretty dense but eventually even I was able to figure out like oh people are asking for a book that doesn't exist so probably somebody should write it. And then 25 more times I realized oh it's probably going to have to be me.
1:18:42Um, so I do have a question, but I think the order of it depends on which order Jono and I do things. Jonno, do you have a sense? Do you want to share your reading journey so far first or me do mine or do you have any? Uh I think let me do mine if nothing else because it's um perhaps uh I mean so if we're doing Jono's first this does lead to my question which is um most people watching don't currently have a
1:19:10startup um is this book relevant to them as well?
1:19:18So Jonno, Jonno is part of a startup, which is more relevant, but is it relevant to Jono? Um, even though he's not himself running the startup then I hear Eric's comment and then maybe I'd be interested in hearing Jonno's comment of whether he felt like it was seemed relevant to him. So Eric, you start.
1:19:36Oh, sure. Listen, I tried really hard to write this book in such a way that is widely useful for people in in almost any walk of life. um as so long as they have some interest in business both because um all of us have organizations that we care about and we this and that's the terminology you use in chapter one. It specifically says think about an organization you care about and then asks a number of questions.
1:20:00Yeah. Yeah. Because that ultimately is the really the question is like you know for some people it's not even the company that they work at and in fact know quite a lot of people where they don't give a they don't give too anything about the person the place that they work but there is an organization that they do care about a community organization or even just like a company you buy products from people are fanatically loyal to certain companies you've ever met somebody like that so it doesn't really matter what it is what I want to do is give people the tools they need that to whatever degree they have
1:20:28agency with relation to a or an organization, they can influence it in the proper direction. And in chapter 15, if you manage to get all the way to the end of the book, you will see I I try to lay out the the the fact that most people have more agency than they think.
1:20:43Um, and so when the problem is actually not that they don't have agency, it's that when given the opportunity to change an organization, they don't know what to ask for.
1:20:52Just like be good, don't be a sociopathic extractive monster is not the most helpful advice. But that's what you get unless you take these specific actions. So I really hope people, you know, especially like people who are engineers or product people, startup people, obvious I think the the applications are quite obvious. If you, as I say in the book, if you're somebody who's like, I don't care about this. I don't care about politics or governance or whatever. I just want to build a great product. Just saying that you want to build a great product, you have no idea what a radical statement that is in our highly financialized economy. So you
1:21:20are a revolutionary whether you want to be or not. And you best find out now at the beginning rather than later when someone takes your company away from you.
1:21:28Yeah. So, so as not to get beaten up by the financializers.
1:21:34Yeah. Yeah. You the the today's current best practice is literally on a they have a rating system on a scale from 1 to 10. And to get the highest possible rating, you have to make yourself maximally vulnerable to financial exploitation. So, if that's what you want, you know, book does not try to convince anybody to do anything. I'm just here to give you the tools you need if you want to build something that is missiondriven that is about quality or product or any any value any human value that you hold. I'd like you to have the tools to do so if that's what you choose to do.
1:22:00Okay. So J, did that did that work for you? Did you find it you'll find out?
1:22:05Yeah, I definitely found pieces. Yeah, I I'll answer your question um maybe as we go in a few different places. Um, okay. So, um, I guess what I'll show today is my Thanksgiving or the day after Thanksgiving. I want to explore the book, um, and try some new ways of of engaging with the material cuz Eric's been sharing drafts and things as it's been written. Um, but I haven't like comprehensively read it cover to cover.
1:22:32Um, and so, um, I did my own loading of the book. I think you could now use that gist to get the same thing, end up with the book in a directory with, um, you know, a dialogue per chapter. And so, um, partly I did that. I I went, um, into my list of, um, list of dialogues.
1:22:50Um, I think I called it Eric book, um, you know, starting on chapter zero and then I could do some of the like direct reading. Um, I've I've since overwritten this, so all of my prompts have disappeared. Uh, tragic. Um but yeah, I also then as I was reading it, I was like I kind of want to zoom in and out a bit more.
1:23:10I'll just stop you there. So um this time I did actually think of that. So before I started reading, I went CP minus R book online to create a copy of the whole thing so that then if I reran it, I wouldn't lose my prompts. So that's something people might want to think about while they're who are watching.
1:23:30Right. Um but so what I did um to get the sort of higher level picture is I'll show you how I built this just now but I I created this um sure that's a good question because um [laughter] I I call this a a zoomable book.
1:23:45Um and so what this is is kind of like a structured hierarchy of summaries. Um so the book's in three main parts and then there's a a sort of cool fantastical epilogue. Um, and so the first thing I did was get the summary of the parts. So I can see the overall structure. Um, then if you want to zoom in a little bit, you can see, okay, in part one, it's kind of laying out like what is the scene? Why why are we caring? What is this evil force that keeps pulling us away from the ideals that we have? Um,
1:24:14you know, so you can start to like dig in one little tier deeper. Um, and then, okay, I want to I want to zoom in a little bit more. So then I also have um for every chapter I have uh some sort of um ah I I need to actually run the run the book again I guess to [laughter] to make my page work. Um fair enough um for every chapter the like one level deeper summary and I'll show you how I built these. Um so this is my like
1:24:43condensed but like not ultra condensed version of the Okay. Each time you're clicking, you're seeing a less summarized version of this more specific big.
1:24:53Exactly. Exactly. And so things like, you know, this was one um back in solit asking about like what can I do as part of my discussion with all the summaries in context. I gave it look, I'm one of the first employees at answer.AI that Eric actually made. This is how we're different. We're a smaller team. What are some places in the book I could look for things specifically that I could do?
1:25:14Ah, it says you should look at chapter 8 and the idea of building up a bank of examples of the culture that you can reference and draw from. So then I can zoom in on chapter 8 until I start to see the ideas of okay, you know, obviously we don't do performance reviews in some big structure at answer.
1:25:30Um, but these ideas of keeping up like the examples of people who did the right thing according to the company culture.
1:25:38Um, and then if I still want to go even deeper to Eric's work, you know, that's when I can go and really dig in. Okay, this is the one I want. H what is our set of values? How do we build that up?
1:25:47How can I as one of the earlier employees pass on those values to later employees? Um so there's this kind of like more and more granular structure.
1:25:55Um but if you were like, oh, like maybe I'm less excited about the historical case studies or I'm more interested in that like public benefit thing, you could always try and find out where is the solution going to be like oh is is this the part where we doing solutions?
1:26:07Maybe I should like what does he mean by a spiritual holding company or or a public benefit company? What does that mean? um you know like just a way for me to kind of like zoom in and out a bit more malleably. Um so I don't know I I think that's kind of fun. Um obviously there's like lots I could improve with the demo but um it really tickled me that this came together in amongst like setting up Christmas decorations at Grammyy's house. You know it was really fun to like create this kind of book putty that I could then um play with. So I don't know. Do you want to go into how
1:26:35I built that or um jump straight to the explorations?
1:26:41Yeah, that all sounds interesting to me.
1:26:43All right, so I'll show I guess step one. And step one was for me to refresh and also for the chapters that I hadn't read yet to get an idea of the the like the lengthiest summaries. Um, and so this was kind of like a nice slow process. Um what I did was one by one I could load up the chapters and because I had them stored as um IPython notebooks or dialogues um I just combine all that back together into markdown to give it to solve it and say could you summarize
1:27:12this um and if you try this if you say to solve it whatever mode you're in can you summarize this chapter it'll speak in a very specific way it'll say in this chapter the author argues that dot dot dot right it's like a detached um the author says this uh it's not written from Eric's point of view. It's written from an AI reading a chapter point of view.
1:27:34Um and so I edited this to remove that and to go back to ship all the fluff that says in this chapter the author set up is like no no what is the core of the chapter? There's a recurring pattern. Visionary entrepreneurs build companies and then their work is dismantled. You know so little bits of tweaks to that piece. Um and then when I was somewhat happy with it going on to the next one.
1:27:55Okay. Here's chapter two. Please summarize it. Again, in this chapter, the author says, "No, no, no. Strip it out a little bit, but here's the puzzle.
1:28:03Who controls a company? If the CEO funded it, why can't they control it?"
1:28:07Um, okay. And then I gave it a tool. And then it just got more and more automatic to say, "Okay, you can now use read C to read the next chapter, do the next one, and the next one, and the next one." And some of these like still needed editing.
1:28:19Um because I was being lazy and Thanksgivingy, I I I would probably try and steer it away from the like the takeaway, the question, the five answers, and you know, it got a little bit like listical towards the end. Um but still, as a way to read this book for a first pass, just like skimming these um AI summaries was was pretty nice to get a a feel for what the different pieces are, um which pieces I might want to do later, um and then which pieces just absolutely defy, um
1:28:47summarization. So, I think I think the final chapter is like I don't want to spoil the book, but you you should read the final chapter like a a waking dream of, you know, giant creatures. It's fantastic that I was like, I'm not going to summarize that in my um in my book one if you try and read the the final chapter just yeah, no spoilers, but like don't don't um Yeah, that that one you should Nobody makes it to the end. It's not a problem.
1:29:13Um but yeah, so that was that was fun.
1:29:15Um, and then the benefit of this is a I have all of these summaries now and I've been editing the AI outputs because I can then just go and grab all the messages in this dialogue which is all the summaries and then save those summaries to their own little markdown folders or to um whatever structure I want. So that's how I started building this structure. Um, but it's also nice because in this dialogue, um, skipping the UI for now, solving
1:29:48Eric if I'm chatting to him or tell me more, which chapter talks about the leader guide again, and it can say, you probably want chapter 8. And I think it actually even maybe reread the chapter so that it's now got the full chapter 8 in context. It's got the tool to do that. And it's got the summaries of all the other chapters. and it can start to to skew us and say, "Hey, like these are some things you might want to do. You should be keeping track of pair examples of when other people did great work um and building up that bank of examples, you know." So, this is like, okay, this
1:30:16is a dialogue in which I have the the summaries of the books quite in-depth.
1:30:21Um, I have the ability to read in any chapter in full and I have the ability to ask solve it and then if I want I can go back to the other tab and kind of do my zoom in and out and explore. Um, yeah. So, that was a very fun um a very fun little uh way to engage um with the book.
1:30:40Yeah, it's funny. It's funny that it picked up on that because that there used to be a section in chapter 8 called management without managers about our approach at answer and I had to cut it for space. I'm I'm very eager back in some form. Maybe we'll do a workbook or something we could have a little bit more detail. I think the way we do it is actually super cool. Sometimes books have websites with extra chapters.
1:31:01Yeah, that will be for this too.
1:31:05I'll put a QR code or something.
1:31:07Um, should I dive in or did you want to talk about any of that more, Eric?
1:31:13I think that I mean, listen, as an author, for those of you who are interested in the writing usage of solv this is like an incredible thing to be able to see how people engage with the ideas in this way where you get to see how their mind was working at the time. It's just super awesome. Yeah, everybody wants it for all their ebooks now. Yeah, me too.
1:31:34And guess what? You can have it.
1:31:40Um, okay. I'll uh share mine then. Um, thanks J. That was super fun and it's interesting for me to see, you know, places of overlap and and difference. Um so yeah I used ho um chest to create my book uh directory. Um I opened up the introduction
1:32:07and you know at the bottom of it um I was like I don't want to read the forward just yet. I'll skip ahead. So just summarize this for me. Um, I was like, that wasn't much of a summary at all. I don't know. It's like, oh, it's a the blueprint. It's like, what blueprint? [laughter] Anyway, uh, I had a I had a couple of backs and forths and got to a quite a a useful
1:32:35summary of um, of the forward. So then I um uh I actually ended up opening pose just as a dialogue um thinking I'd kind of edit things. I didn't end up editing much at all actually. But then what I did was I I grabbed that summary of that I created a bit more manually um and popped it in
1:33:02um here and um wrote that to a a file um called summary zero and then I just said okay go and look up to stocks and create a summarized book which and you know should include the sample sum for in context learning use XML tags blah blah blah
1:33:29um and I don't know if I ended up really keeping anything much of that it's always nice to have a starting point so at least to go pay the imports um so read in chapter one so I kind of like it gave me more of a allin-one thing when I split it into a separate bits and did it each one separately so then my prompt was like example summary chapter this actually seemed pretty fine Um
1:33:57um and then I was like, okay, this kind of thing was just like, okay, stick that in nice sync gather to do it all at once for me is stuff I can't really be bothered with. So I just said, go ahead and write them all. I've done it a thousand times, which I don't need to do it again. Um so it went ahead and called that function for all of the chapters.
1:34:21did them all together at once. This is a nice thing about LLMs. You don't have to use a for loop and wait for them. They can do them all at the same time. Um there is to get this kind of thing. It's like, oh yeah, what do I do now? It's like I just called dot done. It's like of course it's not ready. So again, pretty boring stuff with like just keep printing dot every second or so until it's done.
1:34:45Um, it's like, okay, a minute later it's finished and I've got a whole bunch of summaries.
1:34:52Um, and then I took a look at one of them. Well, first of all, I've took a look at the chapter and then I look at the summary and then I find that models are better at reviewing things like this than they are at creating them. So I asked it to have a look um and uh yeah check the size of the reduction.
1:35:21Uh that's that great. Okay. So now summarize. So create a put all the summaries together into a single string. Uh and now all right summarize part one. So it's much easier for it to summarize summaries I find than it is to summarize a book because it kind of now has the pieces. Um and again it's like okay look at your own summary.
1:35:45Um and uh yeah it identified that there's a whole missing piece around shareholder primacy which I actually agree now that I've read it was an important piece.
1:35:55Um so yeah I think I went back and made a few edits but it was basically I hadn't read the book at this point so there wasn't much I could do other than just checking it looked sensible. Um, so I did a similar thing for part two and uh it showed me how we can check the percentage decrease in each time. So it's an 80% decrease in words for each. We did part three.
1:36:23Um, uh, great. Now take the three part summaries and create a single book summary. Um, I was like, I had a thing of things, you know, let's not actually say what the epilogue says. [laughter] I ended up writing my own epilogue summary. Um, and I was like, h, it's not a very good summary at all.
1:36:53Do it again rather than just a disconnected bunch of stories. And actually, the the new version it created was much better. And I appreciated that it left the epilogue just the way I set it. Um, so then it kind of cool because now I could grab our entire um, launch post and I um, opted in here and asked it to summarize that for me.
1:37:22Um, so I was like, "Okay, summarize the launch post." And then um Eric last year flew to Australia and spent a couple of weeks with me and the main thing we did was we recorded I don't know a dozen plus hours of discussions that we would kind of form as the philosophical heart of the company um for me and for him and for everybody else to look at from time to time as needed. So I grabbed some of
1:37:51those and uh uh summarized that as well. Um, so yeah, again went through a few versions of that. Um, and again did it in a similar way because this was a two-hour discussion.
1:38:09So I went through each part and then put the parts together. And so then from all that, um, I just did a bunch of ad messages. And so I've got, in this case, this is chapter one. I've got the whole book summary. I've got the part one summary.
1:38:26And I've got the summary of the previous chapter and the next chapter. I've also got the summary of our launch post and of some of Eric and my discussions. Um, so you can see here is our summary of the launch post. So it describes, you know, Jeremy and Eric launched this thing and this is kind of the analogy and this is who we are and blah blah blah, right? Um, so founding principles.
1:38:52So this is all just the stuff that's been added. So Eric, at this point I'm just going to stop here for a moment and say and so then I also grabbed all the footnotes and I put them at the top so that way it can see it can see the footnotes when we get to them and know how to reference them. And then I was ready to start reading.
1:39:10So Eric, I remember your comment when I showed you this. You sound a little surprised. You're like, "Wow, that's more, you know, complex setup than I was expecting."
1:39:22Yeah. Yeah. I wouldn't I would it wouldn't have occurred to me like I I'm a big believer I definitely have found it's funny and obviously because we've been working together I have learned that the more context you give the LM the better the results you get and the more you tune the outputs that you want the so I know the principles that you're describing to get this output and yet even I was really surprised I was like oh I see that's a lot more effort than I would have put into preparatory work here instead what I would have done is done something like far easier
1:39:51gotten a halfway way good result then tried to use it in context and gotten I would have gotten stuck downstream of this and then be like oh the LM's not good at this forget it instead of like really having dialed in like this is the information like a person would need to be able to help me with this test to be my partner in exactly that's a great way to think of it yeah so yeah I was like and it's just so clear to me that most people who are using AI are just not giving the AI nearly enough context and part of that is because the tools they're using have
1:40:20a are horrendous at context management So this was it was it was very helpful for me actually to watch how you did it.
1:40:25And it's very different isn't it to the kind of standard claude code Gemini whatever thing where people are just like write a piece of software for me or whatever. It's yeah it's requires a certain amount of respect for the work you're doing to feel like and look I mean honestly Eric it didn't take me long. It was maybe two hours, you know, uh Oh, yeah.
1:40:54Compared to it time you're going to spend to read a book, it's like two hours to read like to make your reading of a book, you know, dramatically more effective is such a good investment.
1:41:03And yet even even now I'm like two hours that sounds like kind of a lot, you know? It's just like funny like if something's not two minutes I'm just like I don't is it really necessary?
1:41:11It's just like what are we doing? I I just Yes. Our attention spans collapse. Um, the idea of preparing your workspace to do a task is becoming like inconceivable. It's crazy.
1:41:22It feels like that's a a thing that I use solve it for a lot. I was explaining this to my housemate. He's like, "Oh, what is it?" It's like, "Well, the first thing you do in Solit is set it up as a workspace for the thing you're about to do often, at least for me." And for me, that's that's usually like I made some fast HTML widgets because I like doing that. And I made sure the context was there and I put in some pictures of what I'm trying to do and I wrote down the plan. It's like the architect sharpening his pencils or Jeremy like preparing his canvas, you know, like it's you're you're putting in all the pieces to say,
1:41:51"Okay, now it's not solve it anymore. It's solvent for reading Eric's book, which is a very customized experience that Jeremy's just crafted." And then the next time you go there, your desk's all set up. You know, you've got all those pieces. Um, and that little investment up front makes it a very different tool to the vanilla case.
1:42:10We'll get to this in the next lesson, too. the level of tool customization, workspace customization that I've I've gotten used to now with solving when I show it to them.
1:42:22I I I I will say and I I talked to Eric about this earlier today and I was almost embarrassed, but he was great about it. So, I will say at first I was like, "Oh, okay. It it's a business book. I know how I read business books." It is a business book.
1:42:37It's a book about businesses. It's about business. And my normal way of reading business books including you know hands up the way I read the lane startup is you know read the first pair paragraphs of a chapter try to get the general idea scan through to find the key idea keep scanning through to see if there's anything else of interest move to the next chapter or find one of those fivepage business book summaries and read the five pages.
1:43:02Um, and the, you know, and I, the lean startup was quite a small book and it had a really important idea, but it didn't take very long to get the hang of the idea. Um, I don't think this approach to reading would have given me nearly so much if the other business books, reading the lean startup, but yeah, Eric, you said you're actually quite pleased to hear that. My view is the new
1:43:30book it's dramatically different in in scope and approach and every time I found myself reading something I was like well that's kind of interesting I wanted to dive deeper and by that stage you had moved on and I wasn't in the book and I wasn't ready to move on I was like no tell me more and solve it really I don't know like it's taken me way longer to read the book with solve it than without because there's so much I'm so many rabbit holes I'm finding Yeah, this book took me dramatically
1:43:59longer to write than the lean startup and it required far more like artistic craft because there are numerous places in the book where I pass over in as little as one sentence.
1:44:13A rabbit hole that you could spend a whole book on just that topic or I can show you some I found. I mean, you've already kind of seen them in really It was really fun for me to watch them.
1:44:21300 odd things, but I'll kind of share a few of them with our audience. So, um, so here's chapter one. Um, and I had never heard of Soul Price, the founder of FedMart. And but it was nice because Eric starts with a personal story of his experience at going to this shop. Um, and after I read a bit, I just stopped and I was like, okay, again, I'm in
1:44:50learning mode. Start. Who Who am I? What am I doing?
1:44:55What are we doing? Spit each paragraph up. I finished reading the first bit. I pasted in this information. You know, uh, what do you think? Is this interesting? Like I I just thought I'd start with a pretty do you think this sounds like it's going to be interesting to me as the CEO of AAI? And it's like by God, yes, [laughter] it's almost either
1:45:21this dude got ousted. Eric's trying to stop you from letting that happen to you. Um but you know evolution philosophy of so it's so it's got enough information. Um and it's very authentic because it's like literally internal only discussions that Eric and I had in the in the context. Um
1:45:46so like what do you think about that parallel? Like yeah definitely very intentional. Uh, and I can just I can do shorthand now. It's like yes, those experiences his thing and I don't need to explain them because it's all in that background. It's like, oh yeah, yeah, that I can see like it's like talking to somebody who knows everything about my experience and has read and it's it's got the whole of the big uh the previous chapter, the next chapter, it's got the part summary, it's got the whole book summary.
1:46:15And then I was like, this is kind of fun. Like, what do you think about this approach to reading? You know, so I'm a big fan of this guy called Peter Wnjak who created this incremental reading approach, this thing called super memo, reinvented what Anki is based on, repetitive space learning. Um, and uh, yeah, I came up with some ideas about other analogies.
1:46:37Um, so I actually mentioned this book earlier, how to read a book. This is the guy who is the chairman of of Encyclopedia Britannica. Uh it's like yeah this is kind of normal though cuz you've given me the locator all the context.
1:46:53Um so then he was like it's also a bit like this Andy Madashuk thing. I was like oh I know [laughter] Andy uh do you think I should tell about this? And it's like yeah here are some connections.
1:47:06Um so yeah so that was kind of fun. It's this is kind of a new primitive um that's different. I think it it's probably something that just technologically wasn't possible until a small number of months ago. And so anybody working and I was like yeah I've actually got 2,000 people on teaching.
1:47:27So it's like is this generalizable back on track?
1:47:33Yeah it is. Would you like to, you know, [laughter] keep going down this rabbit hole?
1:47:39Uh, yeah. Yeah, which is fine. I I, you know, it's uh it's all good. So, so folks in the course, you're this is reminding me a lot of lesson one of fast AI. The first time we ever did it, I was kind of telling people, no one's ever tried to teach this to non-PhDs before. Nobody's ever done it this way before. No one's ever focused on transfer learning before. So, we're like, this is actually something
1:48:06quite So, tell us how it goes. Tell us what you try it with. Tell us what else you find. This is not the final version of how to do this. This is our current best guess, but um I think it's been great for me. So, one thing that's actually quite nice now is that I can hit F and these buttons are the colors of different types of green for notes, red prompt. So I can click on prompt
1:48:34and uh that'll go away and find all the prompts. And so I can now scroll through the prompts and there we go. Find the next prompt.
1:48:48Um and so now okay so I was reading a bit about ethos and uh it's just talking I was just interested in the story talking about this guy price and his approach to building companies and it was like oh they have much less items. I kind of thought well isn't that like didn't work cuz Amazon stocks all the items. I was like is that now not relevant? I was like, "No, no, no, no, no. Actually, this is what's turned out to be huge,
1:49:17you know." So, it's like, "Oh." So, it's kind of cool as well because as I read, I'm as Eric knows, I'm such a skeptical person. Even with Eric, I will often like, "Yeah, but really, Eric and you know, it's great to be able to dig in and be like, "No, actually, Jeremy, really?" It's like, "Oh, okay. [laughter] Eric has a lot of interesting tidbits."
1:49:38Um, okay. So then um you know he's talking about like a problem that happened and I don't know if like Eric's just assuming the reader knows more than I do or what but he's kind of talking about the problem was placing stock in the open market. I was like why is that a problem? Um I like here we go. I was like okay that was exactly the right thing to ask.
1:50:06Um again I was like unsure you know like some of these companies really did go against standard principles and succeeded anyway. So uh so I found it was helpful to be able to push back and but it's like no no Jeremy this is actually proving his point. It's like ah okay. Um, so I actually mentioned this to Eric and Eric was like and um Eric was like, "Well, there is an investment
1:50:36thesis here and it's actually also in the book." So, [laughter] um, but also it it it it knows what's coming up.
1:50:45Um, so it's like, yeah, you're going to get some answers. Um, okay. So I found it helpful to yeah dig into some more details. So I'm interested in contemporaneous accounts there. It's like I want to understand what happened like cuz often I find when you dig into these things it turns out there's kind of more nuance. You know it's like not a bad guy and a good guy. And it was helpful to me to start learning more
1:51:14about what actually happened in this particular story, how things set up, and going in deeper definitely made the book more interesting for me. Um, so you know, sometimes author as an author, one of the hardest things of something this is like there's certain people that love the narrative richness and detail of a story and some people are like, "Yeah, just get on with it."
1:51:38So you can't you can never give enough detail to satisfy everyone. And you either do too much or too.
1:51:42No. Exactly. And so for me when you said here like driven by its new ethos of greed, it's like this is another of my like really Eric like you haven't showed an ethos of greed. It's like well is there any it's like well maybe some of it's greed, some of it's not. And in the end there was this. It's like okay is this that's cool, you know? Like I I enjoyed this.
1:52:04Like it might be editorializing. It's like no it's editorializing. That's my job as the [laughter] as the editor of this document. It is my job to editorialize and say what is actually happening. Yeah. [laughter] Yeah. Um Yeah. And again some of these things it's like uh talking about this thing called the silent monitor of like colored wooden blocks and it's like this is awesome.
1:52:30Sounds horrible. [laughter] I was like, well, it's a bit better than corporal punishment, which is what the alternative was at the time. [laughter] So, it's nice to get this historical, you know, and it's like, oh, yeah, this is actually a famous teacher who has these traffic light cups where students put little colored blocks up and tell the teacher how they're going. It's kind of like, yeah, that's kind of cool.
1:52:53Um, I love this. I didn't know the William thing. So, I like I learned something by watching you read the book. It's really fun.
1:52:59So again I found each time I found something like this some story like happened was like tell me more you know and actually yeah so for me this kind of relationship with Jeremy Bentham was kind of interesting um there just a lot of stuff in here which is every time I drilled more into one of Eric's stories you know there was more and more and it turned out this whole Bentham connection Jerick told me he had never really drilled into it either and it was kind of this weird
1:53:28like two contrasting world views that ended up one a socialist, one a free market utilitarian.
1:53:36Um anyway, so that probably gives a sense of how I was reading this and then it helped me think more about like trying to find more quantitative approaches to thinking about this, you know, and I was thinking like, oh, can you like track ethical investing funds and their success. And it's like, well, these like ESG funds are kind of meant to be like this,
1:54:05but actually ESG investing in Eric's approach are pretty different. I was like, well, how do you like study this without cherry-picking? I'm not actually sure it's possible. Um, because actually you're thinking like maybe ESG funds don't really think about ethos and that's right. the G in ESG is actually often kind of ignored. Um, now this is fun. So then we've got code, right? So Eric sent me this message.
1:54:36It's like, oh, there is, you know, this chapter 15 reference. And I was like, okay, go read the chapter. Come back and tell me what it says. So it's nice that you can, you know, code and read at the same time. Um, and then it's like sounds like a potential answer opportunity. [laughter] [clears throat]
1:55:04Uh, yeah, this is kind of interesting because then he was like, "Okay, let's talk about some shitty things that have happened." And I was like, "Oh, this shitty thing. I vaguely remember this shitty thing." And I was like, "Wait, wasn't that the guy where the CEO somehow made buck by destroying it?"
1:55:19It's like, "Oh, yeah, that's the guy." He say it's worse than I remember. So he was the CEO, chairman, transaction partner, landlord and banker at the same time and it ended up making over a billion dollars by destroying the company. Um so yeah, so these are all fun.
1:55:36Polaroid was interesting. Yeah, I I thought these were all fascinating. people utterly forgotten the Polaroid story. But yeah, but then it's like, well, there's lots of companies that have been horrible but still succeeded.
1:55:53Um, Lord's list of naughty list companies.
1:55:57That what that was there.
1:55:58Yeah, there's a few of them. Here's, you know, then I picked my own one like Yes. Henry Ford.
1:56:09Yeah. uh Thomas J. Watson. [laughter] Um and again, this is helpful, right?
1:56:15Cuz just to talk through like, well, does this invalidate Eric's thesis? And it's like, no, actually it doesn't because actually Eric's just saying that it's possible to create a company that doesn't suck. And when you do, people try to make it suck. And there are things you can do to stop that from happening. But you can still create a company that sucks and try to get rich doing so.
1:56:39So anyway, I found uh chapter one very interesting. Um Oh yeah, [laughter] it reminded me of like when Ford created a whole South American city. Have you guys heard about this? But you never even visited.
1:57:00Um yeah. [laughter] Um, [snorts] okay. So, yeah, this is helpful to like, okay, let's just summarize what what's actually being claimed in the book and what's not being claimed in the book. And then I was like, and then Eric said, well, maybe you should read the introduction. So, I was like, okay, I will read the introduction. So, I went back and read the introduction.
1:57:25Um, so this is close reading, you know, and um I Oh, this was interesting. Yeah. So, I kind of checked this. So, sometimes figures just seem a bit crazy to me and I just checked them. It was kind of nice. I went and grabbed the It helped me because when you say search, right, it puts the little Well, there's also stars things for here for where Eric had a link in a footnote.
1:57:50We turn them into links. Yeah. Which is really handy. So, that organized that for us. Um, and so I went and check. Um, so anyway, so it's nice to get it kind of helped me get over my skepticism, recognize what exactly the facts were.
1:58:07Um, so this is great. Eric then asked some questions in the book. I was like, well, does your organization have an ethos?
1:58:22Like, I'm pretty sure we have an ethos. you know, I'm not sure I could say it in a sentence, but I'm pretty sure we have an ethos. I say, "Yeah, you guys seem to have an ethos, you [laughter] know, here." But there seems to be a lot of bits of it. And it's kind of helpful. Uh, I ended up thinking through this. I ended up going through these three questions.
1:58:41I then was able to talk to my co-founder, who happened to be the author of the book, and said like, "I've gone through your questions. Here's what I found." Um, yeah. So Eric, what what strikes you about this process so far and what I am getting out of the book as it pertains to answer AI etc.
1:59:02Well, I think it was really um you you are a both you are like a highly skeptical but also good faith reader which I find to be a very um unusual combination. So, you're going to get the most out of it because if you're credulous and you just believe everything that the book says, you know, then you're not actually really going to learn a lot. Um, so it's good to have a certain amount of skepticism. It's like, I want to understand what's going on.
1:59:25Yeah. Like when you you are willing to say when you don't understand something, you're willing to ask about it. Um, I think that's going to give you a lot of you're going to get a lot out of that um that fact. But then you're a good faith reader. Like there were a lot of places I could see in your dialogue where you were like, "This doesn't make sense to me or I think it's wrong." And some of it would kind of gently push back and be like, "Here's some additional information." and you'd be like okay that makes sense. Um so that's I think that's like the the intellectual curiosity that is necessary for closed reading. But in if you go if you had if you tried to get a humanities degree at a you know elite university the other
1:59:53thing required for um close reading before AI was a level of erudition that required years and years and years of study because the books like this are packed with references and and links to studies and mentioned things mentioned in passing that you just are supposed to just know that stuff or go to a library and look it up. So to be able to like in the way that you go just on your way be like that seems odd. Like I don't know I think in your dialogue you had one where you're like is this a is this a grammatical error or something else? And it's like no
2:00:23reference not yet passed.
2:00:25Yeah. To the not yet. I love that made me so happy because that's you know if you if you know that if you know that reference then this you get a then the passage is more um is more valuable to you. You get more richness out of it.
2:00:38And if you don't, I try to pick things where if you don't, you wouldn't be so egregious that now you're like completely lost and confused. So, it rewards those who are willing to to put in the energy. And and some one of the biggest things I I've learned in in this is that some ideas have to be communicated in this like multi-dimensional way. There are some ideas that just cannot be stated as a series of thesis and consequences and like really linearly explained. And so like closed reading is one of our
2:01:06civilization's oldest and most powerful technologies for trying to communicate the gestalt of a thing, the overall holistic understanding of it more than just what can be communicated in language because language is so limited.
2:01:19And to me it's so interesting philosophically that it's a large language model somehow is able to help with that when all when the only thing it has at its disposal is language and learning about language. like what what a trippy postmodern theory this is going to turn out to prove correct.
2:01:34I there's another one here, Eric, which I'm sure you'll remember because I mentioned it at the time, which is I um started reading the next chapter. So, chapter one I was just like that was awesome. I just I found it really easy to read. It took me hours to read it, much longer than normal because I read so many other things because I found it fascinating. And then I started reading the next chapter. And I was like, it's not quite falling, you know, great for me at the moment. So, and so I was like,
2:02:02so I wrote to Eric and I said, Eric, as a sequence of messages, like, Eric, this chapter is not really working for me so well. Uh, Eric, I guess that wasn't at all helpful. Sorry, [laughter] not actionable feedback. And then I asked Solan, it's like, uh, different to me. Is it different?
2:02:21So I gave it the tool and it's like okay here's what I see you know this this this versus this this this is like this this is like does that sound it's like ah that's it. It was like really interesting to me that it was able to take a vague sense I have and it did this a few times and uh there was one particular one later where I had like three or four prompts where I was just like yes yes let's dig into that yes let's dig okay what about writing style
2:02:51what about word length what about sentence length and it really is it's very good at this literal kind of literary analysis don't you think oh yeah it's great it's it's super helpful and yeah and it helps you put into like we always say that like LM's like the thing that are the very best at is summarizing. So like to to summarize a vague thing and make it more concrete like this is just something that's so it's so so good at better it with the tiniest hint of what you're trying to say it is way smarter than if you just ask it to do an analogy on its own like just given like it's an attention
2:03:21mechanism so given the slightest hint that this is where you should put your attention you know it's like one of those like spot the difference puzzles where it's like I tell you spot the different or like where's Waldo it's like so hard like where's Waldo but he's standing next to a tree you're just like now I've just cut your search base found by like a thousand acts, you're going to solve the puzzle in no time.
2:03:39Very much like that.
2:03:40Um, I will say though, this chap chapter 2 just went crazy. It was so cool. Um, so, so again, I just encourage people read this book and go deep, right?
2:03:51Because there's stuff about Boeing and I'm sure we're all very interested about Boeing. 3M is a company I've admired for a long time. Um, and I just didn't know about this connection that between Boeing and 3M and basically a guy who screwed up 3M then went on to screw up Boeing and got CEO of the year. But with these things, it's very easy, I find, to just be like, oh, people in the past sure were dumb, weren't they? It's like, hey, this is not long ago and b, they're the same people.
2:04:19So, I kind of like to be like, okay, what?
2:04:23Let's get our generous perspective like what actually happened.
2:04:28And what actually happened was well their financial performance improved and they made all this money and his you know they had these scandals and you know um the very things that ended up screwing them up you know hollowing out the engineering culture.
2:04:47One professor now says he's the worst CEO in history. So, I I don't know. It's like it's easy to be overly beused by this kind of thing, but I think it helps to I had to take that quote out of the book because people felt like it was I was being so over-the-top critical and I'm like, I didn't even say that. [laughter] Yeah. People still couldn't handle it.
2:05:07But yeah, because people can't believe like most people would not sacrifice even a single human life for moderately better shareholder returns over a short-term period, let alone hundreds of lives. Yeah.
2:05:18So they can't imagine that if somebody did that um you know that they're they're anything more than a cartoon villain.
2:05:25Yeah. No. Exactly.
2:05:27Complicated. Like I'm not and to be clear, I'm not saying that he killed he's not like a murderer. He didn't kill somebody with his bare hands. In fact, by the time by the time the crash has happened, he didn't even work there anymore.
2:05:35But if you read the reports of people who have like tried to aortion accountability for what happened, this is not history. This is the not even past. Like this is not some ancient history. Like this is a direct causal connection.
2:05:46Yeah. And Eric, I really did try to dig into this. It's like so I said, look, he wasn't being malicious, you know, he was using the GE playbook. He'd been successful for 19 years. These methods had worked in certain ways and he had had this successful business background and he didn't have any, you know, indication of poor judgment.
2:06:06Um, there's a process adopted by the head of GE and I was like, okay, I I want to learn more about this, right?
2:06:13Like so I I kept like okay well what about Welch? because that that's interesting because for me Welch was a CEO at at at GE at a time when I was in management consulting and you know and uh and I've met Jeff Eml one of you know his successor and had dinner with him and didn't think he was an impressive guy to be honest but that's just that's that um but Welch everybody was like Welch was like what an amazing great guy and it's like yeah you know he took a
2:06:41company from GE from 15 billion to 594 billion 2500 shareholder return percent shareholder return um whatever but yeah actually um the company ended up falling apart um eventually.
2:07:05Yeah. That company doesn't even exist anymore today. They're now three separate GE companies that it's been split up into.
2:07:10Yeah. And none of them No, you can't buy a GE microwave. My best ever microwave was a GE microwave. They don't make GE microwaves. It's like they literally don't do the things that were good. And it's like well by the way there was another Welch employee buying and it's like well actually [laughter] four of the 13 unsuccessful Welch mentee CEOs were all at Boeing and the more I went into it the more I was just Boeing's board faced with a 737 Max
2:07:38disaster hide another Welch disable I say wait who was on the board? ah well uh a Welch disciple [laughter] uh who was um uh head of private equity at Blackstone. Oh my god, it just Yeah.
2:07:58So the more I dug in, the more I saw all these connections. Um, so anyway, that was and what's very hard for contemporary readers to understand is how much praise all these people were getting constantly as they absolutely the topic discuss in a future chapter.
2:08:15Now, what happens then? Okay, I didn't mention this at the end of chapter one is um let's show them actually because they're so cute. Um, chapter one, read one. I want to show you some of the logistics of how to make this all work. Um, so at the end I said, okay, we're done.
2:08:44I'm going to start a new chapter. That means a new LM, new version of you. Uh, so please write a long detailed careful explanation of everything the new you will need to know. I said I'm going to give it the same summaries uh but explain how this everything's, you know, going to work. So it wrote all this which is great and this is adorable. Good luck future me. [laughter] [gasps]
2:09:13Um, it's kind of funny to see it's like summary of what Jeremy's like.
2:09:17[laughter] Um, so then I said at the end of this one, okay, add more handoff notes. I'll be added into the previous handoff notes. Um, and I added these a little bit. And so then I opened up um, so the way I did it then was I opened up well I I I grabbed chapter uh, three. It's actually what I did is I duplicated chapter 2B. I called it
2:09:47I then opened up chapter 3 and actually it's quite convenient because um there's one H1 at the top. So I hit left and that um sum that creates a single elapsed heading and then I hit C and then I opened up my 3B and I hit V and it pasted into the right spot and I hit delete below for the bits that was previously the 2B.
2:10:17And so that ended up with it takes like less than five minutes now, like maybe two or three minutes to create, you know, to start each new chapter. Um, so I end up with all the same stuff at the top. For the summaries, I have these commented out bits. So I just change the two chapters to be the next one and the previous one. Delete the previous two. Rerun this message. um add my handoff notes.
2:10:49Um and then start reading again.
2:10:55so that's my process and then um yeah I was kind of like kept on kind of trying to address some misconceptions I had along the way and also tried to make sure that I was regularly say like saying like help me think about how this might apply in my situation. Um, you know, I think that's something that's under underrated in this in these dialogues is how well it was able to come up just with like interesting
2:11:25things for you to think about.
2:11:26Exactly. They were they were they were they were not always at all things that I was thinking. So, this particular one was about financial gravity. So, Eric, what's a quick summary of financial gravity?
2:11:40Finance gravity is the pressure we all feel to conform to the values of whoever has more resources than we do. Um, so in this book especially, it's about the the pressure that the financial system exerts on organizations to make them conform to a certain set of values that may not be the or the values of the people who run or even own that organization. And I found this quite fascinating that you actually linked it up by to evolutionary roots. Um, fascinating enough that I actually
2:12:08wanted to learn more about it. It turns out there's quite a bit of academic literature here in a whole area that I just wasn't very familiar with. I mean, I'm quite familiar with some evolutionary psychology, but there's stuff about evolution of prestige and conferred deference and dominance psych. This is all new to me. So, lots of uh interesting papers to dig into.
2:12:32But yeah, so when I was thinking about that, it's like, yeah, you know, some of this external gravity, you'll probably be fine. You know, Eric's designed your governance, you know, but yeah, have you thought about these things? So like subtle pressure for people to not along.
2:12:51We've seen that. [clears throat] Mhm. For sure.
2:12:54Um, you know, will your evolutionary reflexes kick in anyway?
2:13:00Is there a chance these third parties could influence things? Uh, yeah. So, that's as far as I'll go with with all this, but I mean I um I don't know. Like, this is one of the absolute best reading experiences I've ever had. Like, I'm I'm trying to think, is it the best?
2:13:22It would be up there. Like, I've had a few, right? So, I remember the first time I read Richard Dawkins, that was 30 years ago, and he started out with a chapter describing how he wrote a computer program on his Macintosh computer to evolve pictures of butterflies. And I'll never ever forget it. I remember reading the Mathematica book like 25 years ago where Wolram basically takes you through so much
2:13:50stuff in like 600 or 700 pages. There's a few books I've read where the reading experience has been amazing and this is up there and I think part of it is because you've put so much time and effort into this book and it's an interesting subject but I think also yeah you know it's um this way of reading I think you can you can tackle books that are more deep in a more deep way than you otherwise could.
2:14:18What do [snorts] you think Eric? Oh, I I listen I as the as the author, it's incredibly rewarding. I now the test will be, you know, we just turned 2,000 people loose on this. So, I'm very curious to see [laughter] how people respond to it. So, so I know we're I know we're go going way over time here, and I'm I'm gonna I'm gonna miss my flight if I don't.
2:14:34All right, get going.
2:14:36But no, but I just think this is so riveting to me. I'm really glad we got a chance to do it. And and um I think we've only scratched the surface of what is possible here. like when we look back on this video a couple years from now, we're just going to laugh at how primitive it seems because I think this is there's so much more to be done here.
2:14:50So very excited for people to pioneer and play around and try new things. And I'm just ever grateful to everybody who's taken the time to do it.
2:14:57Thank you so much, Jonno. Thank you so much, Eric.
2:15:00Thanks, everybody.
2:15:00And have a good flight, Eric.
2:15:03We'll talk about how to make such books.
2:15:07Look forward to it.