0:02All right, we're back and it's lesson seven. Jonno, welcome. Over halfway.
0:08Hey everybody. Yeah, the end is suddenly in sight.
0:14What's been happening?
0:16Plenty of stuff. Um I've got the um the showcase this time around. I um had a plethora of options to choose from. So I'm going to feature a few.
0:26Just let me hit record and then say that sentence again.
0:34Hey, Jono. What's been happening?
0:37Lots of good stuff on the um Share It channel. Um all of these should be fairly recent if you're watching this as it comes out. Um but I'll also try and collect the links that I shared. Um so the first one was from um Ta a student recreating some user interface pieces a lovely sidebar and some drop downs with the um base camp user interface library I believe um kind of a clone of the
1:04famous shed CN library but it nicely it's kind of very vanilla JavaScript vanilla CSS and so it works well with fast HTML.
1:14Yeah. Um, and they were kind enough to share their dialogue as well as the demo. And you can see over the course of the dialogue, they build up all the pieces. And then right at the end, this is that page we were just looking at.
1:24You've got the sidebar and the main with the toggle button, the drop down. Um, so yeah, cool to see the kind of progressive building up of a library um in a dialogue. Um, very nice to see that. Thanks, TA.
1:37Nice. Um then another one I wanted to feature was um something that is going to be useful in relation to maybe two lessons from now, lesson nine um when we'll look at reading and writing a little bit. Um this was one shared by Frank from PDS to navigable knowledge.
1:55Um, and so this is one of those things where there's many different ways to tackle it, but um, this was a beautiful, clean implementation going from a PDF, um, turning that into images and then using a model that's been trained for character recognition to then pass those images into nicely structured text. Um, and so they work through the steps piece by piece, very much like inspecting what's coming out, finding issues, um, improving it, working on these different steps. Um, and so I got kind of curious about the library of like, oh, what is
2:23this? Um, missed OCR library. I haven't heard of it before.
2:27Um, and then, oh, it's got a nice docs page. Wait a minute. This looks somewhat familiar if you were [laughter] there for our MVDE lesson. Like, this looks like an MBDEV project. And sure enough, it's actually um Frank himself um has his um library that's built up in MBDE um and dialogues with small pieces that then demonstrate their functionality. Um, so you can go and check out the GitHub repository for the code as well.
2:51Um, yeah. So I thought that was an amazing example of somebody developing something that they clearly wanted and then sharing a dialogue showing, you know, how to use it and why it's good and and just a very nice positive. Um, sorry, Jonno, can you just maybe open a link to the base coat website because folks in the chat are wondering what we were saying when we said base coat. So they see it.
3:11There it is. Base code UI.
3:13Yeah. Yeah. And so you'll notice the nice um sidebar is um I think the first piece that TA was working on. And then there's lots of other components and things still to be added. So um yeah, and all of the dialogues you're showing here are all available in the Discord, but maybe we'll collect these particular ones in the YouTube notes so people can see these for sure. Um okay, I had a couple more final demos.
3:40Cool. This was one um Andreas shared doing WebGL animations. And so they're writing um code to create little um I guess you'd call it generative programming, but not like generative AI, just generative in the um old school sense. Um but it was nice. They sort of worked through like what are the pieces to preview an HTML canvas? Are there some pieces that they can do to like wrap up the setup? With WebGL, you often need like, okay, I need to create the
4:09canvas and create a shader and a vertic shader.
4:12Yeah. I know it's it's often easy to get bogged down. Um, so it's cool to see them building up the pieces. It's still um, you know, obviously like the developer experience of writing JavaScript in a string. You know, we could maybe figure out ways to try and improve that. But I was just really impressed to see like, yeah, the step by step. Can I draw a line? Okay. What can I refresh my animation? Okay. Okay, what if I want to, you know, just like really building up with this inline preview. It felt like a tutorial on WebGL. Um, so yeah, really enjoyed that. Um, and then
4:42I don't know, Jeremy, can I feature my my own demo of a use case I found over the weekend? Is that That seems reasonable.
4:50Um, so I had a fun one um doing some DNA design, some uh synthetic biology in Solvet. Um, and you know, like this is an area that I've been reading up a lot about, but I haven't used the tools extensively. And a lot of the tools involve editing sequences of base pairs just as text. You know, people send each other plasmids as like Microsoft Word docs and things. Um, some of the time you have tools that are designed to like make it a little bit easier. Um, but I just found doing it and solve it was
5:20like night and day in terms of confidence that I'm getting things right because you can always like inspect the outputs of every stage back translate. Let me go see if this still maps to the same amino acid sequence.
5:31So what's going on here? I know it's not exactly an amino acids course we're doing, but I'm very interested to understand what any of this means.
5:39Yeah. Um, so what's going on here is trying to design of DNA. Um and this is something that we're already seeing your I guess the window you're sharing. So ah right okay that would explain um that explain.
5:55There we go. Okay. Um so this is the end goal is a a circular strand of DNA um called a plasmid that could then be placed into some bacteria um and then they would basically execute the program that is that DNA. Um so they'd start at this origin of replication and they'd make some copies of it and then um the mechanisms for creating RNA and then proteins would start at um a little promoter that says you know um focus on
6:24the sequence. Then there's a binding site where they attach on and then there's a sequence that codes for a protein um and then there's a terminator. Um and so this lets you specify a protein to be manufactured by the um bacteria. Um and so what I was doing in the solvent dialogue was trying to create this this um they call that a cartridge this special piece of DNA along here starting from the promoter and the ribosome binding site and the
6:53protein itself and the protein I was trying to do some fancy tricks linking several subunits together. Um and so historically you do that just by manually cutting and pasting bits of DNA.
7:04Um this looks even harder in Python.
7:06I know, right? Yeah. This like takes a while to start recognizing this is um methionine, you know, this these three these three letters is going to turn into um a particular amino acid.
7:16Um yeah, even even harder than WebGL. Um anyway, being able to just play around with um solve it to say, hey, could you find me like the um the most common codeons to code for each amino acid? Oh, it turns out there's a library for that.
7:32Um fantastic. Can I then use that to figure out what codons I need to code for my specific sequence of amino acids?
7:40Um, and yeah, just iterating on this idea in an environment where a there's an AI that knows a lot more biology than me and can say, hey, you know, you've already got the start code on there. You don't need to add it. Or, I don't know, I caught a couple of um somewhat beginner mistakes that I might have made if I just been doing it myself and had no way to catch. Um, and you can see this, I mean, it's a totally different subject matter, but it felt like solve it. You're doing one step at a time.
8:06You're printing out the outputs. Um, they're a little harder to verify than a nice, you know, Daisy UI component that looks right. Um the uh the overall idea in this whole dialogue it turns out has some flaws [laughter] that that might render it um slightly less u achievable in in real life but it was a really fun exercise to learn a bit more about biology with solve it there to I've got a tiny one I can show too if you like um I don't know if uh I just want to check
8:39Do you know if it shares your audio as well by default? I guess we'll see. I shared this dialogue during the week. Um, [snorts] which is just uh I wanted mainly to tell people about how nifty the torch vision and torch audio libraries are and they are built into solving
9:08deep learning but actually they're super powerful um libraries for all kinds of things. So, for example, I wanted to create a red square. And by the way, the reason I wanted to create a red square was I wanted to test uh some new code that Karum had added to solve it, which is you can now pass images as variables to um solve it and it can see them. Um but as I did so I thought like oh maybe
9:36people don't realize how handy this is that in PyTorch you can say oh create a three-dimensional uh tensor 3x 50x 50 fill it up with the number 255 with int 8 unsigned in eights and then um make on the first dimension uh items one and two the entire matrices equal to zero and then encode all that as a PNG
10:03And you can view it using uh IPython display. Um and this kind of thing is quite nice. I um I sh I showed this whole dialogue to my daughter afterwards to kind of say like you know like matrices and vectors. I was like did you know you can go another dimension and that's called a 3D tensor. And she was like oh of course you could why can't you do more than one? I like okay. But then she was like, "Well, wait, why would you
10:31need three dimensions for a square?" And I said, "Well, you tell me. Why would you?" She had a thing. Oh, this is like the other dimension for the colors. And I was like, "Yeah." So, like looking at this kind of stuff, it's very from scratch. It helps you visualize and understand how pixels and colors and tenses work. And so, another example was um audio. Um so again showed my daughter this audio example and encouraged her to think
10:59about it and tell me what it's doing. Um so here we have um middle C uh and then two frequencies at equal ratios above that um and then we have all the numbers from 0 to two uh 32,000 of them. So SR is sample rate at equally spaced. That's what linen
11:28space does. And so if we take that as time and multiply it by the frequency 440 and multiply it by 2 pi and sign that's a sine wave of 440 htz. If we then add them all together that is literally a chord. And so we can then use IPython's audio and Did you hear that?
11:57Okay, trust me, that was a a chord. [laughter] But it was not a C chord. A couple of sharpeyed viewers in the chat said 440 is a 440 is a my apologies. [sighs] As I said, when you're streaming, brains turn off.
12:14And so that's um creating the the wave itself. And then this is kind of fun. Let's now multiply the amplitude of the wave by 1 +.3 * uh 10 pi t and that again I'll I can enjoy it you can't uh but that's the same thing with ver [laughter] um but again it's kind of nice to and then you could plot this with mattplot lib and it's kind of a nice way to turn trigonometry
12:45into something very tangible. I even wrote here a major chord. I mean [laughter] it's a major chord. It could be any major chord. Sure. That's true. [laughter] Could be any major chord.
13:02Anyhoo, now what are we doing?
13:04Nice. All right. So, that was some fantastic demos of using solvit for various different things. Um, and that was all manipulating stuff like we have the whole course running on solve it the machine, a little tiny instance. And so you showed creating some torch tenses, but one of the things that came out this week was someone trying to train a model and saying, "Oh, it's like a tiny little virtual CPU shared between other people without much memory." Um, it's not for training models. Um, and so the theme of today is to say, how do we break out of
13:34that container if we have to? We tried to make solve it as comfortable a container as possible. Um, but Jeremy is going to show us some techniques for operating other computers, other people's computers, your personal computer using Solit. Um, and so we'll have a sequence of um demo slashsteps tutorials around that. Um, building up to yeah, some different fun applications.
13:57Um, okay, great. Um, I can't begin to tell you how excited I am about this because I I've basically been bouncing around the house for the last few days because I've been regularly experiencing this feeling of like, oh my god, I haven't experienced an interaction with a computer this exciting for years. Um,
14:26and so the first thing I'm going to show you is, um, let me see if I can find this. Uh, yes. I'm uh I've been trying to like do I mean I've been sharing videos for years. Um and I'd like to get at least somewhat less bad at it. Create the video creation. Um so here's my problem. The kind of the the thing for creating videos is called Da Vinci Resolve.
15:00Um, so what you could do is read the manual for Da Vinci Resolve. Um, and it's a very good manual. It's very thorough. Uh, the only problem is that it's quite extensive, by which I mean it is over 4,200 pages long.
15:26Oh my goodness. [laughter] So um that's more than I have to invest in this. Um so what I did it is I thought okay I want to do something reasonably straightforward which is I want to create you know a slightly nicer version of lesson 3.
15:53Um, so I I loaded up lesson 3 um uh in Da Vinci and there's a nice thing that um so we're now just using play and zoom recording, but Zoom actually does have a feature you can check in the settings to say when you're recording keep each uh Jono's camera, Jeremy's camera, screenshot separate. And so you end up
16:22with a whole bunch of separate clip clips for each one. Um, which is cool. Um, I thought, okay, I could like switch to the appropriate clip in nicer ways maybe than the way we did it. And also I could try to make sure that Jonno and my levels were nicely calibrated. Used a little bit of audio compression. So also um
16:48um nicely um you also get if you ask for it separate audio files for each person in the meeting so you can kind of level it and so I wanted to just make it a bit nicer. Um, so I I created a dialogue.
17:11Um, and I used a a pretty new feature um where as as Jono indicated, today's kind of all about like how do you connect solver to the rest of the world? And so perhaps the most useful thing to be able to connect solver to is just your screen.
17:29Um so if you uh import dialog helper.capture um then you have one little setup piece which basically imports the necessary JavaScript and stuff to get this working. So you only have to run this once. Setup share. And then anytime you want to pick a different window or tab or whatever to share, you stay start share. Now, you've probably seen something like this
17:58before, um um where you can pick what to share. And I'm going to pick my Da Vinci Resolve window to share. So, I'll click share. And it pops up, and it says, okay, this is now sharing this window.
18:24Um, and then the other thing I did is I just jumped into my shell and I did just did an ls so that you can see what recordings I've got. Um, and I'm going to go ahead and do this from scratch.
18:38Um, I said, "Okay, I want you to help me learn the basics of creating video. I want to create a high quality recording of this lesson. I've got some recordings taken from Zoom." So, I said search to find out what views there are cuz I was like, what the hell are all these different names?
18:59Um, yeah, I say I forgot to hit record. So, I've got a webm file with a YouTube live. So, I said use capture tools. This is the thing we've got which can capture your screen to see what's on my screen at any time.
19:13Now, um this is kind of text you want to probably put in every time. Note that you won't see any earlier screenshots taken with this tool. So summarize important bits of what you see so you can refer back to it later. I tried to make sure it knows everything about what I'm what I have what version wise what I'm trying to do here what I'm guessing is the approach and it's like okay have a look and tell me and so it said all right let me look at your screen and you'll see it's run a tool here called
19:42capture tool and it's got back an image which we've truncated um and so it should have seen this and it's like okay you got Da Vinci Okay, these are what the filings you've got are great. You've got your multicam setup set up correctly.
20:06And so, um, I'm not going to run every one of these again. Um, but you get the idea, right? So, so this is a big deal, Jimmy. And I see someone in the chat saying like this is wild. This was I can't begin to tell you Jonno like I want to go through this whole thing right it's quite long but like just like I've been trying for a couple of years to get vaguely competent at this tool
20:34and I'm not at all and this whole experience was just like total delight you know and all the more so because I failed so many times and one of the reasons I failed is not surprisingly all the stuff about Da Vinci out there is videos and they're hours long and everyone's like, "Oh, now that you've got your node clip connection set up, I'll show you this feature." And I'm just like, I don't know why they're all designed for people
21:02who like Da Vinci Resolve is used to create like the Marvel superhero movies and you know? So, to find stuff at my level, yeah, it's it is wild. It really is.
21:15Um, yeah. Um, so it said like, okay, you know, do some stuff, right? And now it said create a new multi-cam clip. Now, what happened when I did that is it popped up a new window. So, what I could have done, um, Jonno, is I could have said start share again, and that would let me share that different window it popped up. But actually, um, what I didn't bother, right? because um all you
21:43need to do so if you have another window like here for example um on Mac press command shift5 or on Windows press Windows shifts right and if I press commandshift five it brings up the capture settings and make sure you pick the third one from the left capture selected portion and
22:09then under options S choose clipboard and make sure that remember last selection is turned off. And then you'll find that your mouse icons turn into a cross. And so you can just drag over the thing you want to show it.
22:28Hit enter. And then you can just go back and you can say, you know, um, uh, what should I put here? Uh, hit command V, right?
22:46And as you can see, it just pops that there. So, that's another way of sharing the screen. And I just used that a couple of times where I was kind of just trying to grab a little bit of a different screen or a little bit of a window or whatever. Um,
23:07okay. Well, so so Jeremy, just to like recap for people, um, back up at the top, the way you set up the original share is those two functions, the setup share and the start share that came from dialog hopper. capture and that allows it to just see the screen whenever it uses that tool like a snapshot of the whole whatever you just share.
23:29Yeah. Or or a window and in fact can I just quickly show you tabs Jonno because tabs are really cool with tabs. Um if I wanted to look at a tab I just click on a tab so you can see there's the three options this and I don't know if like it's the same on Windows. I can't quite remember. I think it's similar. And you see at the top now it says sharing this tab, right? Um, and so then I could say like, um,
23:57you know, um, hey, here's some of the source of this tool.
24:09And so now I shared a different thing, which is I'm sharing a tab. Um, and it's interesting. I didn't know. Take a look.
24:18It's asking if it should. It's saying Maybe because I'm in Why did it I don't know. Let's see.
24:25We have also just changed the model, right? This thinking model is now um opus rather than There we go.
24:33Yeah. And so something nice about this when you're doing tabs is you can then switch over and say um so which one did I share just now? Capture.py. So I could then say like um share this tab instead. And I could say um have another look. This is the older version. Um so tabs, sharing tabs are fun. Sharing windows fun. Sharing screens are fun. So sorry, Jono. Um go on.
25:10Yeah. No. Um okay. So then there's lots of discussion around like um are you are we just relying on the model's knowledge of Da Vinci Resolve to be able to answer your questions? Um how how are you deciding when to let the AI like read your screen or should you give it the manual instead?
25:28What's the sort of space there?
25:30So the manual's too big, 4200 pages. Um, and I did wonder about converting it into some kind of rag indexed markdown thing. Um, but I only like to do work generally if I know I need it. So I thought I try without it. And it turns out I did need it. Um, it was able to um, search the web, right? So you can see here it searched the web.
26:03Um, and at the bottom it shows you what it's linked to and it was able to see my screen. And between those two things, it was always able to answer my questions. So, I didn't add any additional context.
26:17Nice. Okay. So, this is cool. This feels like a a good middle ground between like just asking it to do it to give you blind instructions versus like I think the other direction would be something like the Gemini live screen share where you're basically streaming a video of your screen continuously to the model filling up its context window and then asking it for input that side. This is kind of in between where whenever you ask solid a question it can choose to take a new picture of the screen but it's not constantly streaming or constantly filling up.
26:46Yeah. I don't know why, but I never had much success using Gemini Live for this, but Gemini Live would be the other product in this space for sure. Um, yeah. So, okay. So, it it it kind of told me, you know, create new multicam clip and then I showed it and in particular, I kind of guessed what things might be important.
27:09And so, I think I think I need to change the sync and audio. And it's like, yeah.
27:14And it's like, okay, I've done that. Do I set this up for the audio now? It says what to do. And I I had to remind it maybe with Opus 4.5 it'll be better. But it was very interesting. Um, most of the time for the first few steps it would say, can you do that and tell me what you see? And so I kept on editing its answer to say can you do that and I'll check your screen because it's just has it Jonno and I chatted the other day and
27:43Jono you pointed out to me that like it's probably seen the can you do that and tell me what you see a lot of times in the training right right every forum is you know the questions are answered by somebody who doesn't have a capture screen tool so they have to rely on the old-fashioned you know meat bag types in a description method [laughter] and so and because That's how language models are trained. Um that's what kept happening. But after three or four um I didn't have to do it anymore. You know,
28:11it that kind of it got the hang of it.
28:14Um and I also said um search whenever you're ready. Um so I think actually in our system prompt we tell the model don't search unless asked to because otherwise it tends to go a bit wild searching all the time and it's quite slow um and kind of expensive. Um, so once I did this a couple of times, it got used to searching and so it it was now great. Um, what do you do next? And so again, it's to take another look at my screen.
28:44Um, and then in this case, I think it checked my screen and then it searched. So I told it like, sorry, it searched and then it captured my screen. I said capture my screen then search because that way you can search for something that's more suitable. Um, so I kind of had to teach it to use these tools effectively.
29:04Um, and it was nice cuz I have to kind of teach me she keyboard shortcuts based on where I was up to at each point. Um, this is embarrassing, but like the fact that there's like two windows um in Da Vinci. Um, like one here, which are currently black, and one here. I just never knew what they were for, and [laughter] I didn't know how to remove one of them.
29:33So, I finally was able to say like, "What are these?" Um, yeah, it knows enough about it that it didn't have to search for really basic questions like, can you tell me the most basic thing of what the main screen does?
29:46Um, yeah. See, it's like, what do you see in the left viewer now? Right. So, here I just figured I would just say like this and it would figure out that it should look at my screen, you know. Um, so it's like, yes, you're doing it correctly. So, it really felt like having a video editing expert watching over my shoulder.
30:05Um, sometimes I would correct it a little bit. So I'd say control shift might okay to Mac. Um, and so I was really not just trying to finish the task, but I was also trying to understand better how to become more proficient, you know. Um, now this was interesting. Um, so I said, okay, I've got uh five. So there's this thing called a a multicam clip.
30:35um here, right? And so in the multicam clip, you've got um five different synchronized views, right? Showing different bits. And I wanted a way to kind of like in a live sports control room just hit the number that I wanted.
30:57Um and I thought or something like that. So I was like, how do I I bet that must be possible. So how do I get it to work? It kind of showed me something and it didn't work. Asked a couple of follow-up questions, didn't work. And I got into that bit where I was like, "Okay, it it doesn't know it's like hallucinating or whatever." So when I went back and I clicked on these little stars, which is the source material that it found and then I actually read it, you [laughter] know, and then I discovered like, oh, it turns out those things only
31:27work if you click a particular thing to go in a particular mode. And now it all worked in the process. I accidentally broke something. And so now I got to learn how to fix that. Um, so I was like, "Okay, now I want to make the audio better." I was like, "Okay, what kinds of things do people do to clean up audio?" Because again, I didn't really know. Um, and um, it's checking my screen, seeing what's what I've got. Um, gave me some
31:55options. So I was in learning mode most of the time. Um, so it's doing the kind of standard learning mode thing of like saying here are the things you could do.
32:03What would you like to do? [sighs] Which was great. Um, and sometimes I would just step ahead and I can see, hey, I just did it on my own. What do you think? Um, it's very encouraging, you know, and then sometimes you can see here the prompt's empty, right? And that's because it said once you have the inspector open, let me know when I look at your screen. So, I just hit command shift slash, which just sends a full stop to it in thinking mode. Um, and that's all I needed to do. Press a single button and it can see that I've
32:35Um, so yep. Um, Jeremy, I wanted to like um discuss an aspect of this which you could do this with manual screenshotting, right? You could take a picture every [clears throat] time, paste the thing and ask your question.
32:50Yeah. But it feels like that's the same as saying, "Oh, you could do coding with chat GPT web interface, right? You could copy and paste from your terminal, like here's the error I'm getting, paste it, get your response back, and once or twice, that's perfect. If you just need to ask like which of these buttons ends the clip in Da Vinci, take a screenshot, paste it in. Yeah, the difference is like from pasting errors back into chat GPT to like having AI in your editor or from copying and pasting terminal commands to having shell sage
33:20or from working in a notebook and then going to like claude.com to having solve it. You asked 50 questions there. Whereas if you were screenshotting and manually composing the context every time in a fresh chat bubble, I would guess that you'd probably ask 10 and then say, "Cool, I'll figure out the rest."
33:40So there's something about I could have done this before, right?
33:44And I didn't. [clears throat] Um so yeah, I think for me that overhead was too much for for me to to keep focused. The fact that it was constantly looking at the same window and it could decide when it wanted to see what was on that window made this a very smooth experience.
34:04And um interestingly, there was no um there was never a point where the fact that it didn't have a video of my last five minutes was a problem.
34:17Um so that was interesting. And one of the other interesting things here, Jono, is this was the first time I had used this system, which was kind of months in the making. Um, as you'll see in a moment, the amount of code was tiny, but a lot of pieces are working together here. And it was kind of nerve-wracking and exciting for me to be at a point like, okay, today the pieces are all there for
34:46me to finally that thing I had in my head 6 months ago, I can I can do it.
34:50And like I was never able to explain to anybody very well what I wanted to do. I just had a vision of it. And then to go through it and see like yeah, it's happening was great. And like little things like the fact that we truncate nearly the whole image in the history [snorts] is critical, right? Because if it was taking all these screenshots and it was filling up my context and it's going to get dumber and dumber that would really
35:19piss me off, you know, right?
35:22So this kind of experiment of like, hey, use you write notes for yourself to say what's on the screen so that you don't get lost. I didn't know any of that would work, but yeah, it all it all did. So, yeah.
35:38So, when the prompt executes, it grabs the screenshot, it's available throughout the execution of that prompt, web searches, whatever. And then once the prompt's finished, it's gone, you know, and so it doesn't clog up the context. And then when if I have something which I do want to stay there for the whole context, I can do command shift 5 paste and now that is there. So it's kind of nice to have
36:07these tools all in place, you know. Um, so one nice thing here was it told me so one thing I did do as I pasted I took the release notes for version 20 of Resolve which just came out. Uh, stuck them into R.gena.ai and pasted those into the context. So it wouldn't have otherwise known that there's an AI dialogue level that just came out.
36:41Um so um it then said, "Oh, why don't you try that?" I said, "Okay, I I'll try it." And um I made a couple of choices and it was able to kind of comment. It was able to check the choices I'd made.
37:01It's like, "Okay, done it for yours. Now do it for Jono's." little things like I made sure that I renamed my files so that we could you know you can see the file names and so it's nice to be able to talk about Jonno and talk about Jeremy and it knows that I'm Jeremy and stuff. Um so this AI dialogue level it was great. Basically the things that it told me we would need to do uh it did all three of the first ones without any
37:27interaction. So thank you AI at a meta level as well. Um, and so it's never seen it before either, but it was able to do a search to find out a bit more about it. Um, so then I finally inserted that bit where I forgot to press record and then it had a big like zoom picture on it. So I had it kind of tell me how to get rid of that. And little things like, okay, screen, you know, I didn't know this. You could just type a time code and hit enter and it jumps straight
37:55there. Um, so I'm really trying to become more competent at the program through this process. Took advantage of things I know. So it turned out the original webm didn't work. So I converted it with ffmpeg. Um, okay. It told me how to crop it out. Um, and as you can see every time it's saying details, that's it looking at my screen. Not my screen, my window again.
38:25Um, so then I was kind of, um, this was fun. I then, um, went into YouTube and copied the transcript. Um, so you can, um, download transcripts. Um, but, um, easier still is to, um, it's a good way to do this. Oh, I know. A really easy way to do it is I go to solve it [clears throat] and the one I was doing was uh lesson three.
39:09Um so you can get to the transcript here. It's right at the bottom. And I know this is slightly clunky, but it's not too bad. So, if I just double click on the or drag along the top bit and then scroll down to the bottom and hold down shift and click. There you go. That's copied the whole thing. So, that's all I did. And then I just opened a note.
39:37And uh actually even better maybe rather the note would be raw. Um that way it won't do anything with it. Um and then I just hit I to close it up so I don't need to see it. And then I said, "Great. I've pasted the transcript above. This is so fun. I loved this." I just said, "Can you see any bits where something, you know, we might want to delete [laughter] something."
39:58So it's like, "Yep, you know, he talked about some resolution issues." Oh, Jonno's keyboard overlay was blocking it. Um, got a bit slow at one point.
40:09Couple of things here about realizing my rejects had worked when I thought it didn't. I remember that. As accidentally imprompt, you know, blah blah blah. And I like the way it did a good job of saying like actually some of these help give it a lateral eye feel. So which is cool cuz I I actually agreed entirely.
40:25That was what I thought too. Um so I said cool and I said well don't forget the time stamps are now 4 minutes and 3 seconds off because I inserted 4 minutes and 3 seconds on the start. So it gave me a list of the things. It reminded me how to do keyboard shortcuts. Um gave me a couple of different approaches I could use.
40:47Um, actually it turned out not to work for me. So then it checked my screen to see what was going on and it did a search and it's like, oh, turns out you have to press equals to move the timeline rather than the source. It's like, okay, that was really fun. Um, so yeah, and um, finally delivery.
41:12um kind of explored a few different options, wondered about uh expanding it to uh 4K because it turned out it was a um standard HD rather than 4K HD. Now they've got an AD upsizing HD upizing thing. It showed me how to do the first 5 minutes um so I could like check it out and looked at it. I was like, "Oh, that is a bit better, but it also was like slower than real time." So it's like, "No, it's
41:39not enough better to be worth it." So um I ended up not doing that upsizing and then finally I wanted to share the dialogue with all you guys. So I said use add message to go back and add header notes before each section.
41:54Um and one nice thing that is somewhat recent for this is that now if you just press right arrow so you see here I am on a H2 right? If I um press right arrow again, it'll take me to the next H2 and then to the next H2. So that made it very easy. Um or of course I could use control shiftV um to see where it put them. And so then
42:21when this is published, uh you guys get to use that as well. And also helps me go back and reorganize. Um, but yeah, I mean that's it. And so you remember John, I after I did that, I jumped on our local on our internal Discord and it's like, oh my god, [laughter] this is so cool.
42:48Excellent. Thank you, Jeremy. That was a good a good one to start with. Um, there's a question, can solve it drive the app you share? And an emphatic answer in the chat, no, it can't. Um, so I don't know. Do we want to segue into the next section of [laughter] deploying things with no worries?
43:05Absolutely it can. Yes, to some extent.
43:08Um I mean it can do anything. So um so what have we shown so far?
43:14Um I'm going to show you how it works. Um but like um dialogue helper which is what capture share uses is open source.
43:25Um, and there's nothing in solve it that supports anything you just saw directly. So you could have written dialog helper.capture. By you, I mean all of you. Like it's not um doesn't require any serverside stuff. So let me show you how it's written. I'm not going to like explain in detail. Um, so I'll stop sharing.
44:03Um, so you can go to answer.ai/dialoghelper and capture. py is the thing that does it. And this is the entire other than um imports.
44:18Uh this is the entirety of the code. Of course, it fits on one screen. Be embarrassing if we wrote something that didn't fit on our screen. Um, so what happened at the start when I hit setup sharing? Setup share. So, what did it do? Um, so I if is a function that just creates a a JavaScript function basically inserts it into your your DOM into your page and it grabbed a
44:48JavaScript file and read it and stuck it into your DOM. Um, so I'll come back to the JavaScript file in a moment. Uh, when I clicked start share, it fired a share screen event. Uh so each time I click it, it'll just fire that again to let me reshare a different part of the screen. Uh each time I said amperand capture tool, um it called this function which first
45:16of all tried to capture the screen which would do an event get capture screen. It would then try to make sure that that contained an image data attribute and if it did uh yep, if it did, it would return it. Um and then if it was successful it would create a what's called a tool response which is a thing from let um which is basically to say like okay this is something that let knows is a response to a tool this is the type of response and this is the
45:46contents and this is basically how um anthropic API openai they all basically have the same format where you say type image URL then the image URL contains data um something I wanted to say though is well actually I guess I'll show you the JS as well.
46:05So here's the JavaScript and again much less than a line of a screen. Um so we called that share screen event. So, we have an event listener that listens for a share screen event in JavaScript. And when it gets one, it tries to basically just use the your browser's um [clears throat] API to grab a video track and that's what causes it to play.
46:31And then when we send off the capture screen event, it tries to call get screenshot, which is something which grabs that video that we just built, captures it. there's a maximum size. So, we scale it to that size, stick it onto a canvas, scale the canvas, draw the image into it, and send that back as a data URL. So, it's all a bit clunky JavaScript. You know, these kind of APIs are often a bit messy. um this kind of
46:59stuff um AI helps a lot with um something I wanted to say though is like conceptually I can describe what's going on here very easily which is to say okay in order to capture the screen fire off the capture screen event which we'll call that event handler we just saw um and that then returns some data which you then get and so event get calls it and gets the data from the event and then we can return it
47:29Um to start sharing we fire off the share screen event that doesn't return anything. So um this is not at all what it looked like at first, right? So 3 weeks ago um it looked like this. Lots of code and there's none of these abstractions, right? Um there's all this like send data to server and UUIDS and JS loaded and data.gets gets and stuff, right? So, um
48:11I wouldn't have felt comfortable saying like, "Hey everybody, you can now have solve it talk for JavaScript. Just do all this." Like I I I can't even describe what it's doing cuz it's like stuff we had globbed together. Um um but we gradually refactored it, you know. Uh Nate and I both had a go and you know we we just did what we've described many times is we found little duplicate bits of code and refactored
48:40them into functions and then we were like well what what's that function doing exactly? So for example, fire event is a function that's now part of core. Um and it's not very big, you know, but um it calls htmx.t trigger
49:11um the event that you've asked for and you can also pass in some data for the event if you wish in which case it gets added to it. So, we're kind of like it's JavaScript, right? But we're kind of creating this Python API um or that um that event get thing um you know, first calls fire event and then pops the data back out.
49:38Um and so we kind of ended up with something which is still on the advanced end, right? Not many people are going to write their own one of these but some of you will and I feel like I can now support this. I can say to you, okay, this this is how you have solve interface with JavaScript.
49:58And JavaScript can do a lot, you know. So, not only can it see your screen, it can record your audio, it can read and modify your file system, it can connect to Bluetooth devices, it can use your accelerometer. It's, you know, it's quite cool. It's quite a powerful piece. So yeah.
50:22Yeah. Anyh who um yeah. So let's keep digging um down through the layers here. So So that's one way that we can talk to the rest of the world [snorts] is by having it see our screen. Uh it can't do anything to you to to to you, but through JavaScript you can actually do quite a lot. Um um but being able to see is great.
50:48Um, [snorts] now the next thing I wanted to do was to let it um change things on my computer.
50:57And my goal here was to have it um use my own computer's web browser to to browse. Um, so in the past we've seen how we can use online APIs to kind of go to websites and take screenshots and stuff. But for example, Rachel mentioned the other day, hey, I watched the lesson. I've got 20 websites I want to take screenshots of, but like when I try to do it with the online service, it the
51:26screenshots have a thing on the top saying like, do you accept cookies or another one has a thing saying you have to log in or whatever. So, it's much better if it can use your browser. And also, sometimes you want to be able to like have it like fill in a form and click on something or whatever. So, it's nice if you can actually use your own browser. And to get to that point, there's a lot that we're going to do.
51:51Um, the first thing we're going to do is we're going to connect from solver to your computer using SSH.
52:02So, SSH is kind of the universal connector for computers nowadays. It's highly secure. Um, it uses public key encryption. Um, so nobody else is going to be able to use it as well. Um, it's also very convenient and it can also do a whole lot. Um, now, uh, generally speaking, most people's computers don't have an internet IP address.
52:31You can give it an internet IP address by going into your router and asking it to forward a port. Um, but that's annoying. Um so the convenient way to let solvent SSH to your computer um is to create a kind of a tunnel basically to basic to use a service which our computer will connect to um which in turn
52:58will then um give us a kind of an IP that solver can connect to. Um so I knew a few ways to do this. Um, so but I still I always like to ask anyway and again I'm in learning mode so I'm telling it a bit about myself um before I start and just like okay you're experienced so I just jump straight in.
53:22Um and also I quite like to try new things. So normally I for this I use something called Enrock. Um but I was like just keen to see what else is new because Enro's been around. Oh my god, you got the t-shirt, huh?
53:38[laughter] It's great.
53:42Uh, where does an Endrock t-shirt come from? Is that a conference or something?
53:46I I can't remember. I think I filled out a survey for them or something. I was like, I like the free I've never paid them. They've never paid me, but they sent me a comfy t-shirt. I was like, for this lesson, I better dig I better dig it out. [laughter] That's fantastic.
53:58I'm sorry to say, Jonno, I didn't end up using them uh because I saw there was this new thing called bore.pub and then I looked it up and it was by a guy I know. So um which is actually interesting um um what's his name? EK J a lot of people so uh he's a um AI researcher and um so he's now at thinking machines
54:27which is where Gentle has gone to and Alec Bradford's a adviser and um the previous CTO of OpenAI I think they're running it now. Um, and I was putting this out to say a lot of people are like, "Oh, I want to be an AI researcher. I don't have time to do, you know, learn about internet IP addresses and Unix and but this is what actual AI researchers look like. Oh,
54:55I've created a secure web-based terminal and a net bypass and reactive notebooks and a text editor and a data log." being good at AI research like just be good at things, you know, like it just it helps to be good at using computers and knowing math and understanding linear alge like there.
55:17Don't be like, oh, I can't learn that because it's not important enough. This is what all the good researchers I know look like. So, I wasn't particularly surprised to find that Eric has written a Enro alternative. Um um yeah, as somebody pointed out in the chat, he's the one that made modal notebooks. He was at modal. Um and he's the kind of people person that like I think think I think um actually
55:46um thinking machines is has the second strongest team in the world now after. Um uh they're really good. Um, companies like Thinking Machines are going to hire somebody like Eric before they hire somebody that is highly specialized and only knows one little thing. Um, [snorts] so yeah, uh, anyway, bit of a diversion.
56:11Um so basically what this is is something where you can say all right on your local machine you run this program bore uh and you say take my port 8000 on my local machine and um make it available through the address bore.pub which is something that um Eric [snorts] provides. So it's like cool. Um, this kind of looked annoying having to like install stuff and compile it with Rust and blah blah blah. And it said,
56:41"Oh, see alternatives below." And it's like, well, it's in Arch. No, we don't have that. Gen two. No, we don't have that. Okay. Binary distribution. I say, oh, I always find when something's like a little bit annoying like this, I don't feel comfortable saying to people, okay, you have to do this really annoying thing. So step one for me was to be like, okay, does it work? And then once it does, um, let's make it much easier for people to use. This is how easy it is to again make the world a slightly
57:10better place. So um, I've actually done this enough times before that I've already provided something. So it said, okay, you can install from pre-built binaries that are on the releases page.
57:22Releases page. Okay, here is the releases page. A lot of steps, right?
57:27particularly to end up getting this onto solve it while we actually need it on our local computer. Um, so I just like everything to be easy for everybody. So I wrote this thing called latest.fast.ai. If you use the URL/ latest and then just paste in the um name and um of the organization or person and the repo. So in other words, paste in Yeah, paste in that.
58:02Then it returns a list of and this is like I think this is like eight lines of fast HTML code or something to implement latest.io. Um that returns everything and then another thing that I added to it is that if you add something one more bit after that it treats that as a regular expression.
58:23So if you search for that, you will find that, right? Um, and so I did this to show the AI um how this works. So I didn't have to tell it because I've shown it. So I just said, "Hey, I've got this thing. Uh, write a bash script that just grabs the latest one." Um so it's you know you don't have to know all this
58:49but you name minus M um is a thing that um you name yeah there you name minus M for example is a thing that tells you what type of CPU you have. So, it like checks that and figures out what kind of CPU, what operating system you have, uses that to figure out what latest URL it should
59:17use, grabs that URL, and then it grabs it, decompresses it, and tells you it's done. Um, so I just tried pasting it directly in here and I ran it and oh, it worked. Um, and I tried running it. Oh, it worked.
59:38Um so I noticed it wait by mistake which I fixed. Um so I then um popped that in a gist which I will share and so now anybody can install it with a single line which is nice. Um, so T-Max.
1:00:12Oops. Need to run everything above. So, shift A to run above.
1:00:23And start share. Okay. Let's share my terminal window. Again, you could use something like shell sage for this instead.
1:00:31Um, so and then I said, "All right, uh, show me the B. I mean, this is actually just for you. I already know this back to front, but uh, just to give you an idea, how do I use T-Max and SSH forwarding?"
1:00:45tell it where I'm up to and it said, "Okay, okay, I can see your T-Max session, right? You can create a new T-Max window with control B C." So, anybody who hasn't used T-Max before, please do. It's amazing. B C Okay, it basically lets you create like multiple um kind of little windows like this as you can see. um switch between them. Uh you
1:01:15can come back to at any time. Um so I do suggest looking at this dialogue and running through it. Um like okay, how do I name this window? Um because it said you can list all the windows with controlB wtrlb oops bw. So, I've got two windows. Um, but they're not very meaningful. So, it'
1:01:41be nice to name them. Um, uh, so, okay. So, we could run bore now.
1:01:52Um, and it tells me how to use it. And hopefully it's now in my history. That's not. Never mind. So, bore local. Okay. Okay. So, we need a two or local um in fact branch was the Yeah, I think I also need the to write which port I want. So, bore.
1:02:18Here we are. Bore local. Now, what port do you want to share? And the answer is always 22. That's my SSH port. And so, uh on Mac, uh Sulvic or any search can help you. There's actually a switch you can turn on in Mac to to turn on SSH server. You don't have to set anything up or install anything. It's part of Mac already. Um I think Windows WSL comes with SHD, so you can also run it there.
1:02:47Um so then I just say to bore.pub. Um, and actually something that's nifty I discovered is now if I um take a lookb. It's quite neat. It's actually um you can see at the bottom actually it's named it bore. So I can actually see what each of my windows is doing which is quite nifty.
1:03:21Um so the important thing now um so what I actually did was I then restarted sharing um with my local terminal. So open up this terminal and we should be able to um SSH. So I run T-Max here as well. So this is this terminal is looking at my solv server. This terminal is looking at my local computer. Um, and actually I want to use the same port I used last time.
1:03:53And I remember you can actually do that too.
1:04:02Let's say help. Here we go. You can actually say what port to use minus P. And that way everything's going to just keep working. There we go. Um, so that's port 6134. So the first time I did it, it was on a different port and I realized I should keep it consistent. So SSH to
1:04:3961341 613 41 Okay. And my username is Jay Howard. And if someone else tried this command, they wouldn't get in because they don't have permissions, right? So it's not um not scary to share this URL or anything like that.
1:04:57Exactly. Yeah, that's exactly right. So, and so I would only use SSH, right?
1:05:02Because anything else people I bet there are people who are randomly trying ports on board.pub. And so if you're sharing some unsecured port, that would be a problem. So you can see here I've typed ls to look at my first three things. And if I go to my um so as you can see I still got bore running in the background. So I switch to the other window and I could type ls pipe head three.
1:05:27Um oh that's different. Why is that?
1:05:36uses J. Howard. That's interesting. Oh different orders. Um right. So this is using a different shell that is not which is treating uh the same different capitalizations as the same. [laughter] So that's quite funny. Um but you can see aws back aws. Um so it's also there. Um oh actually so that's something that we're going to see
1:06:04why that happens in a moment and fix it up. Um but so basically to summarize Jonno Mhm. What we've got now is a way that um solve it has connected to my computer and it can do stuff on it. And so maybe before we have our break, I'll just quickly show what that means we can now do.
1:06:30Well, before you even show it doing something, there's a a question in the chat. They said, "We didn't see you enter a password. How did how can solve it access it, but nobody else can." So maybe that's an important piece, too.
1:06:40Absolutely. Yeah. Absolutely. Um, so um the reason for that is because I have a uh Whoopsy Daisy.
1:07:00Oh, you're in your remote machine.
1:07:02I am in my remote machine. Isn't that confusing?
1:07:08Uh so on solve it I have a SSH directory and in that I have a um a key.
1:07:18Um and so what I did I did I took my public key and then I pasted that um into my local um authorized keys file. So, this says that that private key um or the private key attached to that public key is allowed to connect to this computer. And again, there's no nothing insecure about me showing you this. This the most you could do with this is let me log into your computer. This is not my private
1:07:48key. So, if you're not familiar with how um SSH keys work, it's well worth asking solvent about them and practicing with them. It's a really important idea, but it's it's a super secure and convenient system. And so you can either copy a private key you already have just upload it and to solve it and put it in SSH or you can type SSH- keygen
1:08:16not but SSH um and that will create a key for you. So those are your options. Um, okay. So, I've uh closed that SSH connection. But check this out. If I now create a variable with the port and I create two little functions, a function called shell, which simply runs subprocess.r run. So it's just going to run a program and return it sted out or sted and then remote which is going to call
1:08:57shell to pass a command to SSH on a particular port with a particular user and a particular host and we'll pass all that off to bash. Um, and so I can then say, great, let's use remote to view the first three files in the home directory.
1:09:21And I run it. And so that's going to call SSH and do that. So I can even make that available as a tool, right? And this is where I was like, oh, it's weird. It's kind of got this strange error and it's in a weird order. And I realized it's actually not using my shell. So I asked for a little bit of help on why it's not using my proper shell. It helped me go through a few things. It was even like so it's it's calling remote to find out
1:09:49about my local computer. So it's now debugging for me why it's calling the wrong shell. It can see all that so it can figure out what's going on and then it can tell me how to change my function. Um, and eventually it realizes, ah, instead of saying bash, you should have said dollar shell. And so I made that one change to remote. I think I sent this to you, Jono. Um, and now
1:10:20there it is. So if I was feeling ridiculously brave, I could now say, hey, use amperand back ticks remote to to do anything, right? And it can now do anything cuz everything you can pretty much do through mouse you can also do through the CLA. And this this has access to all of your computer. It can read and write files, edit things, change system settings,
1:10:48turn your computer off [laughter] uh etc.
1:10:55So maybe maybe not recommended but theoretically possible.
1:10:58Right. And so, as you'll see when we come back from our break, I decided not to make it at all, but instead I it was tool like in that I just hit the W button. You know, I'd say make, you know, write the code and I'd hit W, I'd check it and I'd run it.
1:11:17Should we have a break? What what I've done is pasted a URL into the chat and you're going to see some buttons like this.
1:11:25Oh. Um, and so what, um, that's going to do, if I share a different window instead, um, this will give us something pretty to look at while we're on break.
1:11:35And hopefully you as the chat can, um, explore around this landscape and we'll explain it when we come back. Does that sound good?
1:11:41Awesome. All right, I'll see you in five minutes.
1:18:16Hey. Hey. What's been happening here?
1:18:19[laughter] Everybody's been uh causing mysterious movements to happen. So, I think we should um yeah, maybe uh explore what's going on there.
1:18:29[laughter] Thanks everyone for your um participation.
1:18:34Right, let me load up the dialogue. And so somebody in the chat asked Solvit.
1:18:39Solvet says, "This appears to be a stress pattern visualization." No, I don't think that's right at all. Is it?
1:18:46No, not the colorful interference patterns called photo elastic fringes.
1:18:51H very similar color spectrum. Um okay, zoom in a couple there, Jonno. Yeah, let's let's go. Okay, so what are we doing? Uh, turns out what we're doing is remote controlling a microscope, a Raspberry Pi powered microscope slide scanner. Um, so just to give you a little bit of background, the um thing we were looking at there is slightly out of focus. My apologies, but is um some very colorful crystals looked
1:19:19at under it's called polarized light microscopy. Um, and so the colors are dependent on the thickness and orientation of the crystal structure. Um, and I love love love taking photos of different crystals with this setup. Um, stitching together big images of a puddle of slowly crystallized um, material, usually uh, vitamin C or um, some of them are like epsilon or citric acid.
1:19:46So [clears throat] these are pictures you've taken.
1:19:48These are pictures I've taken. And the trick here is that you're using polarization to display colors you wouldn't otherwise see.
1:19:58Exactly. Yeah. So, this just looks white if you look at it in normal light, but if you have some polarizing film before and after it, you can see how much it twists the light and that gives different colors depending on how much um thickness of crystal there is. Um but the main thing is solid apparently is saying that the type of visualization it said is commonly created using polarized light passing through transparent materials. So okay very much pass very much the same idea.
1:20:27Um yeah so um the demo here in some ways is going to I'm I'm going to restart so that people stop making the um printer move around. [laughter] Um, but the demo here is going to build on what Jeremy showed us just before the break. Um, so this Raspberry Pi is just another computer. You can see this little green um, square here. Um, this is, it turns out, a very capable ARM processor with some RAM and some storage
1:20:57and the ability to connect a camera. So, there's a little camera pointing down the um, eyepiece of a very cheap student microscope. And then sitting on top of a 3D printer is the microscope and the 3D printer is holding the sample. Um, so that when we move the printer around, we can look at different parts of the sample at a time.
1:21:16So you're not going to be really using the Z axis, but the XYaxis of the 3D printer.
1:21:22Yeah. And we could use the Z as well to focus. So you could see it was probably a little bit blurry cuz I I just kind of clipped the slide on there and I didn't spend much time focusing. Um, but if we did very small like.1 mm up or down, it would have gotten sharper. Um, and we could have seen more detail. Um, cool.
1:21:39Okay. So, how did I connect to this? Um, oh my gosh, I accidentally triggered Gemini. Excuse me. [laughter] Go away.
1:21:49Um, the, um, the tool I used was Enro, very similar to bore.pub that Jeremy ran. Um, and just like Jeremy showed, I was then able to put my um, public key on the allowed hosts lists for the Raspberry Pi and then steal his functions um, but adjust it to my enro um, URL and port and so on. Um, and so you can see rather than um, if we just
1:22:16ran this in solve it and I'll do that a host name, we'd get the host name of our solit instance. Um whereas if we run it with this little remote function, it's just using SSH to connect that Raspberry Pi. And sorry, Jono, somebody just in the chat was a little late back from the break and said, "What's happening?" I just I just delight that I can now say, "Oh, Jono is just controlling a 3D printer containing his microscope in order to do polarized microscope photography."
1:22:41Obviously, welcome back.
1:22:42Obviously. Yeah. Yeah. You you blinked and you missed it. We were talking [laughter] about um downloading uh releases and making scripts. [laughter] Um yeah, so we have a way to execute code on this remote machine. And in this case, it's remote by about 3 mters from me. Um but [laughter] this could also be sitting in some some distant lab on the other side of the continent. Um what can we do? And so um one of the things I did was to get solve it to write a little
1:23:12function so that it can um Can we see it?
1:23:16Yeah. Yeah, let's do it. Um, I guess I should uh hit I to to reveal. Um, basically let it list files or if we want to look at a specific file or maybe a specific segment of a file um we can do that using this R view tool. And you can see it's got um types and a string so that solve it can use this as a tool.
1:23:39So this looks like it might have been copied from or heavily inspired by stuff we've kind of seen before with the text editor API. Yeah. So, actually what I did was I imported the um the fast core tools list of tools and then I asked solve it. Um I want code for the equivalent of these file edit tools that operate on the remote Raspberry Pi.
1:23:59Um and I wrote the full set and used it a little bit but then I got pretty disgruntled with that flow of using. So the only ones I ended up using are um the one for using GP to search and the one for using uh view to inspect. M um but this gives solver the ability to kind of poke around on this remote machine. So if I said using these two tools view and and um GP can you take a peek in this folder where I've previously done some experimenting.
1:24:25Okay. Can I just stop? I just say at the top of your screen there. So the definition of RRG which is remote rip grip. You're not actually using rip grip. You're close enough is that it's calling remote. Remote is in turn calling SSH and is passing the GP function to it.
1:24:41So you've kind of got a total of like six lines of code maybe altogether necessary to have solver um be able to access the Raspberry Pi that's controlling your microscope.
1:24:53Yeah. Um and with with just that it's able to look around, see that there's some files there, search for a specific pattern inside one of the files and then see what imports are there. Right? So it's able to start to get it feels a little bit like something like clawed code, right? But we've dialed it right back so that it can only read. It can't necessarily write.
1:25:12Um, but it can start to explore and get a feel for what's there.
1:25:16Um, code for microscopes.
1:25:18Clawed code for microscopes. Um, yeah.
1:25:20So, like I said, I originally gave it a full set of tools including string replace and create. Um, but instead for a second attempt, what I settled on was saying, what if I have it write me the code and then we just have a little function that's going to put that into a file on the remote machine. But this way there's this kind of intermediate step where solve it gives me code, I inspect it, and then I send it to the machine.
1:25:44Um, and so yeah, that's what I did.
1:25:46You'll see um just very vanilla solvity saying look around using those view tools and give me the minimal code to get a camera preview up and running. So it did a little um our view to see what the app looked like. um it looked at the one of the uh requirements for that app and then once it understood what was going on, it was able to make a pretty small little um Flask application, right? Because it's just based on example code from Raspberry Pi. Um and
1:26:15it just put that in a variable here. So I've got this preview code. Um and so then I can now use my little write file function to save that onto the remote machine. Um, and if we run um cat the name of the file, we can just check, did we get the um the code that it wrote?
1:26:32Yep, that looks good.
1:26:33Um, and that let me have the um the the little preview that we were looking at of the view through the microscope via the camera.
1:26:44Okay. So, that's basically it's created a little web server.
1:26:47That is and uh it's its index route basically displays whatever the microscope's looking at. Yeah. Yeah. Um, and in this case, you're not looking to really learn anything about Flask or whatever. You're, so you're just like, you're looking at the code and being like, okay, it's not going to destroy my microscope. I know I already know enough about Flask to know it's doing what I want to do, so I'll just go ahead and let it go.
1:27:16Yeah, exactly. And it's small enough that um it still felt like, okay, I've at least looked at it. It's it's somewhat auditable. Um yeah, so then very similar vein. Um investigating the next step, which is something that I've actually done a number of times, so I'm quite happy to to offload it. But we have this printer plugged in via a USB cable. It's showing up as a device on Linux. Um and so just creating some little utilities so that I can um here I'll show you the code. Um so that I can
1:27:46um Oops, here we go. um send a command that executes one line of this thing called G-code. Um which is not going to be familiar to most of you, but if you've got a 3D printer, you'll know this is kind of like the native language of the 3D printer. So, I'll just show some examples as the best way to um to illustrate what these little pieces are doing. Um this M115 is a special G-code command, and this sends it to the printer, and it responds back with some information about what
1:28:15software is running on that 3D printer.
1:28:17This is a an Ender brand 3D printer. Um, and so we can do things like send a request to say, tell me what your current position is, X, Y, and Z. And you can see everybody's been moving it around. So it's now set at X= 101, Y= 95. If I wanted to overwrite that and say, actually, let's define this position as 100 100. Um, this is our like origin point. Um, and then I can see I'm going to be working with this.
1:28:42So I've got a little function that reads that. You say origin because I assume it doesn't handle negatives. It treats 100 as the the middle of the plane. Is that how it works?
1:28:51Yeah, exactly. So, normally a 3D printer can go to 0 0 0 by like moving until it hits a switch and and then finding its home position. Um, but if I did that now, there's a microscope in the way.
1:29:01So, it could break the glass slide or run into one of the limits and start making alarming noises. There's a reason I only gave people a few buttons and I kept them within some bounds. Um, and so what I've done is actually just moved everything to exactly where I want it before I turn the printer on.
1:29:16So now I'm saying you're blowing everybody's mind in the chat with everything you're doing at the moment.
1:29:20Oh, fantastic. I'm glad. I'm glad. It It feels like a slightly quirky, niche little demo, but um uh yeah, I I I enjoyed it. Um yeah, so I mean that's pretty much the bulk of what's going on here. I've got a little fast HTML app um that has a move route, right, that can move as long as you're within bounds.
1:29:40And I've got a a little piece that's just again solve it. Very easy to write a lock and a him since the last move so that we don't um spam too many moves in a row. Um the little controls is just a series of buttons to say which direction should you move. Um, and if I, um, show the printer view here. Um, let me just make sure I've run these roots as well.
1:30:05Um, oh, okay. I guess I need to actually create my app and everything. Um, yep. If you're wondering what why this is here, uh, we don't need it at all. I was just showing solve it how fast HTML works. Um, okay. So, now we can see our preview and we can see our buttons. So, if I hit X plus, hopefully we'll see. Oh, we saw some output, but there's a slightly different view.
1:30:29Somebody in the chat has had an issue at 10,000 ft on a mountain where they accidentally did 90 minus instead of 90 plus on a telescope and spent the next three days having to resolder the whole thing. So, yes, good idea to check Yeah.
1:30:45your cartisian coordinate origin points and such things.
1:30:49Uh-huh. It's it's um it's very easy to uh to get it wrong when you can't see it. And actually the ideal would be to have another webcam looking at it, you know, um the whole system so that you could make sure nothing under those little four buttons uh down down down where you've got the four buttons.
1:31:06I think it would be nice to show people how you can click on the on the output the buttons that pop up. You can click the pop out and that would give you your moving um then drag that into a separate window and then you can move it um at the same time as seeing it if that makes sense.
1:31:26Oh, and actually we can't see both because you're only sharing one window. But those that little pop out um is a great way to have a you get controllers and buttons and things at the same time. Yeah, like this.
1:31:40Yeah, there we go. Um, cool. So, that's pretty much the extent of my demo. I don't know, Jeremy, are there pieces that you'd like a little bit more elaboration on, or um, does this catch the idea?
1:31:52I think that's good. I mean, I think a lot of some of the folks on the chat were kind of saying, "No, Jono, this isn't niche. It's like this this is such a tactile example of connecting up real world things." And um, yeah, I think it's has played it part extremely well.
1:32:13You didn't have to have your arm twisted too much to be convinced to create a um [laughter] microscope program. Uh you should turn off your um blurring thing on your Oh, background blur. Okay. It's also I realized moving a millimeter at a time so you can't even really see it. Um you'll have to take my word for it that I can see my little machine moving around slightly as people on the internet click some buttons. Um but yeah, you can see how um okay, I had to write the code in a string and then
1:32:41write it to a file on the remote machine if I wanted a big piece of code there that I could use. Um but everything was still, you know, fairly small little utilities and then building up first the remote shell and then the remote send G-code and then the remote get position which sends some G-code and passes the result. So it's all very small pieces.
1:33:02Um, and uh, yeah, rather fun to connect solve it up to something in the in the physical world.
1:33:10And this is this is what I don't like about teaching with you with Jonno because I can't follow on from this successfully. It's like, great. Well, let's now take it up a notch and do some browser automation. [laughter] Okay, browser. That's one that has been much more requested than 3D printer automation. So, you have uh, popularity. Nobody knew that the polarized microscopy option was an option at all.
1:33:34So if they had, I think everybody would have gone for that. [snorts] And this is why we struggle when people say, um, what's this course about? What am I going to learn? Oh, polarized microscopy, you [snorts] know, uh, sending G-codes 3D printers. Yeah, [laughter] I don't know. All right. Nonetheless, I will endeavor to not be too boring. Um, all right.
1:34:05So, this example, um, I'm going to control my local browser and I'm going to control it for what I've discovered to be basically the hardest thing to automate, which is Google Scholar. Um, I don't know why, but automating Google Scholar is I know why it's very handy to automate. um because it's really helpful to have AI helping you find research papers um when you do research, but for some
1:34:34reason Google makes it incredibly difficult to automate. They've got very aggressive anti- scraping going on. Um so I I'm doing it because I know it's a good challenge. Um so you have to take my word for it. [snorts] Um and it's nice to be able to it'll show you something where you're actually logged in. So, you know, if I go to Google Scholar, then um
1:35:01I'm going to see my Google Scholar. Um uh let's see my library. Can't quite remember how it all works. Um so, um yeah. Um I've done quite a bit of uh browser automation before. So I was kind of able to describe what I wanted to do. So there's things like Pipeteer and Playright that are like libraries that you can use. Uh and most browser automation is done for testing. And so
1:35:45when people do testing with browser automation, they normally use what's called a headless browser. Um, so with a headless browser, you don't see it happening and it's generally happening in like a fresh profile without all your without your stuff. So this kind of more personal browser automation is a bit different to what most people do. Um, so I kind of again I just asked for a few ideas.
1:36:14Um, so you can use something called Chrome debugging protocol. you can use MCP and blah blah blah. Right? So, um, so I've used Playright before and I thought it was quite nice. Um, I haven't done Chrome debugging protocol directly before. And so I'm kind of saying and so I say like just show me some code. I want to get a picture of what it looks like. And here it's like oh playright connect over CDP browser new page
1:36:43browser.goto browser page sorry page goto page.fill fill page. Oh, that looks nice. Uh, so this is kind of helpful to get a sense of what it's going to look like.
1:36:54There's that. That's fine, but much less convenient. Yeah, fine. Much less convenient. Okay. Yeah. And it's like, definitely more convenient. And this often happens, too, when you ask AI to write some code. It's only after it sees its own code that's like, oh, okay, this a definite good choice.
1:37:11Um, so like, okay, how do I connect to my browser?
1:37:16Um, so you can start Chrome with a port listening.
1:37:25Um, and then because we've put this at the bottom of our dialogue with all the existing stuff, it knows what all the stuff we have is. So it said, "Okay, you can forward a port." Now, this is really important. Um, when you SSH to our port on our tunnneled computer, you [snorts] can add this really important thing minus capital L, which is um, SSH tunneling. And what that says is while
1:37:55this SSH is running, anything that you talk to in solver on port 9222 will be sent to the other computers port 9222. So, it'll connect them up like with a tube. And so, this lets you, this is why I said you only need SSH. Once you've got an SSH connected, you can connect everything up over the top of it.
1:38:23Um, all right. So, what happened was I I ran it and it popped up basically it just popped up my existing browser.
1:38:34Um, and kind of I said, "Take a and I said, "Yeah, it's connected to your existing browser." And then I tried it and it didn't work. Um, so these details don't matter too much, but I kind of want to show you how I went about fixing this problem because it was quite quirky. And so it helped me identify, okay, is it listening on port 922 or what's going on? And I ran it. And so by running it with remote again, that's running it on my computer. And so nothing was shown. So it showed that
1:39:03Chrome was not listening on port 9222. Uh and so it can see that now. Um so it's like okay, we could restart. Um make sure it's going to be secure. Um and as it started searching for these answers, it discovered something that there's something in new in I don't know what how recent this is, but it turns out this doesn't work anymore. um it doesn't work anymore unless you use a nondefault data directory.
1:39:40Um so they've basically decided it's too dangerous to have users opening up a port to their Chrome uh with all their cookies and everything. So they've actually turned it off entirely unless you specifically say use a different directory for your data. Does that make sense, Jono?
1:39:58Right. It's an extra security feature that they've added, but it's not something that's well documented yet.
1:40:05Um, [laughter] okay. So, I thought fine, I'm going to create an icon that when I run Chrome, it runs a special version of Chrome with special configuration that opens up the port and uses some other persistent directory.
1:40:26and then I'll log in and that and I'll sync in that and I'll get all that set up. So I [snorts] said, "Okay, where should I put that directory?" It said, "Why don't you just pop it here?" And I was like, "Fine." And I said, "Okay, how do you do it?" Um, and um, it's giving me all these buttons to click and like I can't be bothered doing all that. So I said, "Just show me how to do it. This is really helpful." Like cuz again, most stuff on the internet, how do I do something? go here, click
1:40:56there, look for this button, move it over here, drag it to this, like but pretty much everything you can do with a CLI. And if you can do it with a CLI, you can just say just do it for me. And so with this, I was just able to do each step with remote.
1:41:13And so, okay, create the directory for my special Chrome debug.app and create the Mac OS and resources in it. And so did you know actually aapp in Mac they actually are um directories that have special things in. So that's actually all of your applications are actually this behind the scenes and then um put into it um this thing that's going to run Chrome with remote
1:41:41debugging and my special directory and make it executable um and create a special properties list file. like none of this matters, right?
1:41:53But um uh add an icon. Um and so check this out. Now, if I bring up Spotlight, type debug, I have Chromedebug.app, and I run it. And I have Chrome. So nice.
1:42:11It's done. And so um and I've already had it log in. So I'm logged in and it offered to sync. And so I'm synced. Um, so now I can do stuff there and it shows me my own version. Um, all right. So I need to make sure that I am running that SSH um, which I guess was a little earlier. Was it? Where was that with my um I think it was in there somewhere there.
1:42:53Right there. Um Oh, yeah. Great.
1:42:56Okay, great. Great, great, actually. And so I can just Should be here already. Yeah, here we go. Uh there's no local forwarding there though.
1:43:07Yeah, quite right.
1:43:09So, we'll just copy that in.
1:43:19Okay. So, you can't see anything happening, right? It's just uh Oh. Oh, yes again. There we are. We're in. So, so I can It's still got a normal shell, but it's also in the background forwarding this port.
1:43:44So, it's connecting to localhost 9222 on solver, but that is forwarded through SSH to my own computer. So, there's quite a lot going on there, right? Um, and it didn't die. So, we can now try to create a new page and go to example.com. I'm not touching handsree. There we go. Okay. Um, so that's uh that's worked.
1:44:09Um, so I'm kind of like, this is where it's nice to have AI help test things because it knows all this stuff backwards. So, let's try going to solve it. Sorry, to Google and again, hands free. What's it searching for? Solve it.
1:44:27Yeah, there there we go. Great. Um, don't know about this it.com extension thing. I wonder if there's a way to change that. Um, cool. After I did this, I found it running super super slow and it was uh able to do a search and figured out that it was actually using um CPU the wrong CPU type. Um so it told me um that you had to also do that. So I
1:44:54actually went back and changed it. Um uh so I thought you know I don't know if like using if I should use playright stealth which is something that makes it look less botlike. So I installed that um and so basically once you run it um it applies it to your page. Um oh that's right it was gave me the wrong thing.
1:45:21It's meant to be capital S stealth. Um, and so now, and I've got here a little delay after each thing has got a type. So, we're going to try and search. So, if I run that, take a while. Hello.
1:45:45It's not 150 seconds, right?
1:45:48No. Oh. Oh, that's right. Um so I have to keep reminding it to search before it does things. So in this case actually it accidentally used the um um like JavaScript version um instead of the Python version. So I gave you the wrong syntax.
1:46:13All right. Oh, there it goes. Okay. So probably type a bit faster than that but you get the idea. Um, so we can just leave it to its thing. Um, sounds like cool. And then, um, when it's done, be nice to grab a screenshot. Um, so you can just say page.screenshot.
1:46:36Okay, looks like it's finished. So, I'll do that. So, there we go. And there it is. Um, so then I wrap that up into a tool um that takes a screenshot, scales it if necessary, and saves it as PNG into a bytes buffer and then passes that back as a tool response. And um, yeah, you say, well, let's do it. So I'll click here, twink, and then we will
1:47:18run gets. There we go. And so, um, it's, uh, calling the tool. Oh, oops. Don't do that. They always like to do this. All right. So, apparently I didn't import B 64. So, how come it knew that that was the problem? And the answer is because we've set things up so that errors get returned back to it. So it can tell you why tools fail, which is nice.
1:47:58So Jeremy, while this runs um in this case, you've got the Chrome developer protocol running on one port on your machine. Um, but if you are running like like on my Raspberry Pi, I could run OctoPrint, right, which is a software for controlling 3D printers or you could be running a web application on port 8,000. The same workflow applies of you could give solvent the tools to um, you know, either operate your browser or um, send commands to some
1:48:26specific port on your local machine. Um, and then in this case, you can take the screenshot using the Chrome um, system, but you could also use the screen sharing thing we saw right at the start of this lesson if you wanted to share your Ableton Live session or something with Solvet, but then also have it control MIDI or something like that.
1:48:44Yeah. So, what we've what Thanks, Jono.
1:48:46Yeah. So what we've seen here is that solver can uh run any CLI command line executable program through SSH and it control can control anything available through the network.
1:49:00Um that's basically everything you can do on a computer and it can see your screen um either through the capture screen or through special program specific screenshot things like CDP. Um so your whole computer and everything on your network and everything you can do on a computer is now available. So, somebody said on
1:49:27the chat like, "Well, doesn't MCPU do this?" Yes, it does. Um, but MCP is entirely unnecessary here. So, MCP is a much more specific thing where you have to like an MCP for this, an MCP for that. And what we've shown you here is like, no, you can now control anything on your computer anyhow you want. You don't need any MCPs. Um,
1:49:54you don't need any special knowledge or skills. You can use all of the normal playwright stuff that everybody else uses. You can use all of the normal Macintosh scripting thing people use.
1:50:08You can use all of the normal approaches to creating um custom applications on your computer. Um, and you can do those as tools and tools can call other tools and stuff as we've seen. So you can create agents um that do it for you. Um and there's some kind of uh place that you end up between show me the code and run it and I'll run
1:50:37it versus run the code and then run more code and then run run more code in an agentic loop. Um, I find myself in the show me the code and I'll run it much more often than I thought I would be because I get to edit it. I get to see what it's doing. I get to learn more. I get to recognize places I can refactor things and stuff. Um, but they're both
1:51:06useful, right? But the point is here by using lower level more foundational tools like SSH you know and networking you know we open up the entire world uh rather than be limited to specific stuff that people have built for us if that makes sense.
1:51:29It does. Yeah. Um, I guess I'll add that, you know, for me, I mean, so Claude Opus 4.5 just came out today. It's the new best model at computer use, right? They've trained it to look at a screenshot of a screen and then select coordinates to click and and do actions.
1:51:45It feels like um just like Jeremy with the with the JavaScript we were showing or something like that, there's pieces that we do very carefully and then there's the Mac application configuration settings file, right? That you say I I'm only going to need this once and solve it knows how to write it.
1:52:02I feel like it won't be very long. You can see how this train will go where I might say oneoff I need to like click the five things to download my tax returns from this site, right? And that might be a case where I say, "Okay, solve it. You've got click coordinate and take screenshot.
1:52:19Please just get that PDF. I I need it, you know, and I I never want to do that again." Um, but most of the time you will want to say, "Hey, I do this all the time. Could you make a script for me to scrape this thing because I need it weekly." Um, so there's going to be that trade-off just like there is currently with code. I feel like we're just getting to the point where these agents are like bumbling first steps towards like, "Hey, Solvent, could you just, you know, fill out that questionnaire that I need to do?"
1:52:44Yeah. And and for that example of Yeah. creating the Chrome debug.app.
1:52:50Um I in fact originally I just said splash it all into a single multi-line string and call it with remote. And um it just it didn't work the first time.
1:53:05So then I just hit control shift hyphen a few times to split it into different blocks and ran each one separately and then it did work. So I don't [laughter] know. Uh I I tried to do as much of that automatically as I could and when I couldn't I made it slightly more manual.
1:53:25Um great. Um, so I think next time I do want to um just do a little bit more of this, which is to show um how you can control um other computers other than your own, which is not going to really introduce anything new, but it's going to combine two pieces we've already seen, which is using um um service APIs to launch new
1:53:53computers in the cloud and use this remote stuff to secure them, administrate them, learn about how to use them, etc. And I think what I might do, Jonno, is um I might just install bore.pub bore. I might install bore on a separate computer rather than, you know, using this random person's on the internet. So, it' be nice for me to set up my own forer. Um, one thing people might like to do would
1:54:22be a similar example would be to um, create their own uh, wire guard virtual private networking server. Um, yeah, there's a lot of nice things you can do when you can create little servers out on the internet and um, so for example, I really enjoy watching snooker and there were some things they fixed it now but there was a lot of uh, snooker matches that you couldn't watch unless you were in England. So, I set up a
1:54:50little dollar a month machine in England and ran wire guard on it and then connected to it and made me look like I was in England. And little things like that, they're uh they're fun to be able to do. So hopefully, yeah, people will find um opportunities to to build stuff that meets your needs, whether they be um polarized microscopy or watching Snickai or something else.
1:55:18Something else. Fantastic. All right.
1:55:20Well, thanks everybody. We'll share the snippets of code that Jeremy used for that like remote connection. Um the example dialogues I shared at the start, a couple of other pieces in the Discord channel for this lesson. Um there's lots of different directions that you could go whether it's just the screen sharing or like I think hopefully everyone's buzzing with ideas. Um yeah, we'll see you again on Wednesday to wrap this up and then think with a couple of other pieces as well. Um, thank you so much for watching and and then next week after that, the last
1:55:48week we'll be with uh Eric Ree um and we'll be learning about uh um reading like reading like you've never read before. Uh writing and building startups and understanding what might your future look like uh now that we have these skills um both your particular skills you got from the course and the skills that AI gives us in the world. Should be a great next
1:56:14three lessons. Thanks, J. Cheers.