0:00A lot of the innovation is just how can I put coding agents everywhere I possibly can.
0:04One of the things that we have been starting to do at Pi is getting rid of tests. Why? Because like it has become too annoying to run all the tests.
0:12Hello. Welcome back to State of Achantic Coding with Ben and me Armen. This is episode 10. Quite far. So, hi Ben. What have you been up to?
0:22Oh yeah. Um, my name is Ben Vinegar and the co-founder of this company called Modem. And we help companies basically connect to all of the product conversations about their products happening on different places on the internet like Discord and soon social media.
0:43Um, and I also build this like diff tool called Hunk. Armen, who are you? I'm still Armen and I work for a company called Arendel uh which I founded together with a friend of mine. Um and right now I think we're turned into an AI infrastructure company because we're building harnesses. Uh specifically we're building the Pi harness. Um and we'll see where it goes from there. But that that is our current adventure in
1:11many ways. So if you have used Pi we're working on that right now. Funny I am increasingly trying to say that we are building AI infrastructure.
1:21Everybody's trying to build AI infrastructure and um you know I how is that possible?
1:29How is that possible? And I think it's the classic it's the classic startup thing where um we built some technology to to model like a product manager in a company and it turns out that we ended up building like a context graph of stuff to support what that kind of agent would look like.
1:51Things like like I mentioned earlier like connecting like product conversations and also like linear t you know tickets and GitHub stuff. And it turns out that context graph is actually really useful for answering a lot of questions. And so increasingly we are plugging that into agents, not just our own agent. And it starts to look more like a database and like a magic database. Does that make sense?
2:13It makes sense. Yeah. I think like my answer to why everybody seems to be gravitating towards AI infrastructure right now is the standard meme of like nobody has figured out how to sell this stuff yet other than for coding agents. So now we're all providing infrastructure um for people building coding agents.
2:34There's a lot to be said about like what what the economics of the space are like but at the moment um it's a it's a wild wild time where a lot of problems are not solved yet and we're all solving problems. I guess I think I'd been even like I believe in this world where we're going to have like multiple agents and they're on your team and you're talking to them and they're talking to each other but we're just not there yet. Does that make sense?
2:57Like we we don't really have I I think that you do have ver some vertical agents that are like services, right? Like you have legal AI, but we don't really have like a bunch of them hanging out on internal malt book collaborating. Oh my god.
3:16Someone someone's going to take this and build corporate mold book like right now. Um, so in the meantime, yep, we got to sell some of this stuff to everybody's existing agents.
3:26So I think this time around, um, we are skipping the predictions, but we're just going to go through a bunch of news that has happened have happened, um, in the last couple of weeks. And maybe we start with the discussion and emergence around watermarking um and what it means or might mean for coding. So Ben, what happened?
3:53Basically anthropic models, I don't know where this began, but they are they are watermarking their generative content.
4:00And the way that I understand that works which is interesting like so what that ultimately means is like they're going to generate sentences of content or perhaps code and that there are properties well let's come back to that but the output there are there are characteristics of the output that they can use to identify whether it was generated by their models or not and I read the blog post on this and here's how I understand it and you can correct
4:26me ultimately there's non-determinism when they're generating like they're choosing what the next token is as they're generating output. And the way that they're watermarking is that they're they are basically tilting some of that non-determinism towards a certain set of tokens so that it appears more statistically in the output. So for example when they say that they're doing watermarking they cannot say oh 100% of
4:56content generated out of clawed is watermarked but they can probably get I don't know I I think they have the percentage in there somewhere a lot of it they can and part of that is because of the tokens that they've chosen that they that they've chosen or the or sort of like the LM has chosen because they put their their thumb on the scale for some of the output and by the way does that match so far.
5:20So, I think um I can feel you like hurting inside. So, no, I'm not hurting. I'm I'm trying to So I think like um I think really there are like two things that are happening and it's entirely unclear right now um how exactly Entropic intends on doing this because like one version of this which I have seen on Twitter is that well all we're doing here is that the RNG that's being used the random number
5:49generator is going to use um like a wellknown seat that only but that it can then use to um figure out like what the probabilities are of uh words appearing. So they're still random but they're like that the this the random number generator has been initialized by a value that um is is non to entropic.
6:13That is a better way of describing it than pushing the scales or waiting. The the thing with the watermarking is that you can quite reliably these days identify from at least large enough blocks of text if an LLM has been involved in um let's say word smithing it maybe not necessarily on code uh I don't know to which degree that is um the case but for
6:40for for human language uh there are bunch of different AI detectors The most prominent one recently probably is pengram which have basically created um their own models which were trained on uh different model outputs to determine statistical likelihood of some text being AI generated. So it's it's not even like I don't know to which degree Pangram knows which model
7:09uh it it is but they but they're pretty pretty good at at determining if an an LLM was uh was utilized uh in in in either the creation of text or uh even even just like uh rewarding it. Um so presumably you don't actually need to do what entropic is doing. You just need to train a model. that seems to be um enough u because the models are actually somewhat predictable and and it actually did get me thinking a little bit which
7:38is that if you are writing text like let's say you're writing a blog post most of this applies to blog post I think or like or or like scientific studies or whatever in the past if you were writing a book you the the version that goes to the publisher is actually not your output it usually goes through an editor that will do corrections to that book, that will fix your sentences, whatever, like fix the grammar, like there's or or help you
8:08even rephrase stuff. So, like there's there's an entire job called an editor, which at least from my understanding is supposed to do this like copy editing. Um, but you can't scale that human to large number of independent texts, right? It's like that person is only ever going to edit so many texts in their life, but one LLM can edit the text of all of us and we don't even know about each
8:36other. Um, so whatever stylistic choices an LLM is doing to fix up bad grammar or something like this like is is much more noticeable because like it has a much larger like job assignment to many more people some of which don't even know about each other. Um, and and to some degree we notice this just by the choice of words that are being used by the LLMs which are outside of the regular word frequency that you would expect. So like
9:04we all know about loadbearing about seams um whatever words are very prominently appearing right now as a result of us just using those models right um so I think there's an interesting thing happening where even in the absence of dedicated watermarking it's getting easier and easier just to detect that text was most likely coming from LLM because you can use you can train your own models to detect it um and so you even if you use
9:34an LLM for copy editing yourself, you might not even be necessarily aware that it helped you rephrase words in a way that is unnatural um to you, but it would be very natural to the LM that you have been using.
9:46Do you think it's like desirable by you know everyone keeps asking why does it talk this certain way, right? Do you think that that is a a characteristic of just fundamentally how LM work or how they're trained or do you think there's something combined to this watermarking?
10:01I think it's entirely unrelated to watermarking is my suspicion. I think like these models were shitty at writing before. Um I don't think like whatever they're doing recently has made this worse. It is however very noticeable that if you talk to Fable in I think like Fable is the worst offender, but Soul is sort of like very close to that.
10:20Like I cannot stand reading its output anymore. like it makes me feel like I'm an idiot by not knowing all the verbiage and word choices it uses. Um, and it's so like whatever went wrong in a training process, it doesn't really sound like English anymore if at least when it gives its commentary about the stuff that it's doing. Um, and that to me seems to be I don't think a desired outcome. I think that something went wrong in training. That's sort of my
10:49suspicion. Um, but it really feels like it's a worse writer. And I like I I noticed for like recent blog posts that I was writing, I I no longer choose um like state-of-the-art models for like copy editing. I'm trying to use something else because like it is they're way too invasive um in in in in what they're doing. Um and and I don't think it they produce good output anymore. Well, we talked about this with
11:18Mario. Like the current slate of models don't I'm not happy with them as writers and it's just caused me to write way more. So, I don't think that that's like a bad thing.
11:29I don't know actually. So, so here's here's a here's a question. Let's talk about slop for a second and sort of LLM supported writing. I know a person who has um I really don't know the name of the sickness right now, but it basically makes you incredibly uh like tired. Like you're you're basically bedbound for an extended period of your life.
11:49Yeah. Um I had a conversation with this uh person that I know over the internet recently about LLM supported writing and the reason for it is that he actually can engage with people on the internet now with the help of AI.
12:07Um likewise I am routinely now reading LM translated tweets because Twitter LM translates everything. I find that the amount of documentation actually available on a lot of projects has actually become better. At the very least, it exists. It reads like slop, but it does exist. Um, and I'm now also finding routinely blog posts that are very clearly LLM written, but they
12:35actually convey useful information. I am not trying to judge, but I think he's pro he's pro SLOP, everybody. Well, I'm I'm I'm I'm basically saying like it seems like there is something happening underneath the whole thing where an increased number of content is uh is LM supported and like y like I've been using LLM to like do copy editing on my blog post for like two years.
13:01Um I'm trying to make it not sound like it's a horrible thing and I don't use an LLM to generate me blog posts, but I I use it excessively to to figure out like do I like the wording and everything.
13:11Um, and so I think that the question here is mostly to me not so much like has an AI being used in the creation of a blog post, but like do I like reading it?
13:21I don't know. Like I I feel like there's there's it might very well be the shift is actually like it's going to be super binary, like you're going to be ostracized if you use an AI, but I'm actually not sure that that's going to happen.
13:35You're ultimately measured by the final output no matter how you get there. And I think that there's a wide range of acceptable slop, unacceptable to acceptable slop.
13:48Randomly, I have a bunch of stickers. I'm a fidgeter. I'm always fidgeting with things where like, you know, you can spend a lot of time with LLMs to create written works and I think they can be quite good. The challenge is that that's often not what people want to do. They want to generate a lot. It's more about the the amount rather than the quality. So, I think there's a lot of I guess I'm just trying to say I can usually tell the difference. Like if
14:16something's good and I and it feels good like um you know, then I don't really care how you got there. Does that make sense?
14:26Yeah. I mean, I think like the extreme version of this is in code, right? Like I think we had this discussion a couple of months ago where it's like a security report is still a security report even if it's a horrifying slop.
14:40We we had slop security reports before AI.
14:44Yeah. But like now now they're coming at a volume that is really frustrating and it is also um I would say like horrible to look at.
15:00Well, hold on hold on though. What you started with like look this the slump can be good and now you're like now it's horrible. Where we go where are we going on this?
15:07Because I think it's like it's horrible to look at but it's still a valid report, right? So like it it found a security issue, but it's like it's it's like a terrible blog post I wouldn't read because probably I don't enjoy engaging in it. A terrible security report that's still valid is worse because it's valid but also it makes me bar reading it. So like because like the security report has no like for the blog post you
15:35probably are being rewarded if it reads well. So presumably a really bad blog post is going to have fewer readers, but the security report is rewarded on speed and correctness and not does the code look nice and is the report like not slow. So like the incentive structure is a little bit different I think for code.
15:56So for the code situation I think it's yeah the quality is definitely worse um when when you have to read like or pull request descriptions they're like horrifying these days. Where are we going to go with this though? We talk about slop all the time.
16:09It's interesting to talk about slop, but like maybe the watermark is not really the part that is going to make this worse. Even without watermarking, it's probably still detectable.
16:20Look, part of this is, oh no, you're watermarking my content, and that means that must mean some sort of selective choice about the content, right? And that's where my head goes. But your point being, well, look, it's just actually a random seed. It's just it's just affecting the random gener like you're not going to know the difference.
16:36I think in the blog post that was actually the argument Anthropic made, which is you're you literally cannot tell. The issue is you can tell for the other stuff. Is that kind of where you're going?
16:45You can you can basically tell anyways like with or without watermarking these models are very noticeable.
16:51Problem here. It's like it's the quality of the models. That's the um that's where we are.
16:57So look, I do agree with that. We're probably like watermarking is not a big deal. Yeah. because the fundamental content is is the is it's already marked. It's got a a a scarlet letter
17:14a scarlet a scarlet cadence um that identifies it. I think this is this week. This could have even just been the end. Oh, it was late Friday. Wow, this is so recent. It was I I remember it came at like 10 at night on a Friday, right? In your standard kind of like buring news, right? which is um open AI models will no longer be provided and supported on cursor. And the subtext of
17:41that is, you know, cursor has been bought by SpaceX. I've lost track of what the actual acquiring company name is.
17:51SpaceX. Uh and of course they have they have Gro. They are they are a lab and you know you probably know you generally know more about like I can make whimsical uneducated comments like but it seems to be related to the relationship um the relationship of Sam Alman and and Elon Musk that might play a role here.
18:20Um you might remember I think this was probably conversation in October but entropic took away Windsurf's models from one day to another. I don't want to say like because it open did not end up buying windsurf right. It was like Google that bought it in the end. I forgot about Look, let's I think we need to acknowledge that, you know, some people watch this and have not been following every piece of news for the last two
18:47years on this, but um really quickly, Windsurf was a competitive IDE to Cursor.
18:55Like I say, it was it still exists.
18:59I think it was a plugin. I'm not sure. Uh I think it was a plugin. And what I remember is that the acquisition ended up being like part of the team went one way and part of the team and the product went another way. Um it was it was a wild weekend.
19:13What I remember is basically like when Surf went quiet a bunch of rumors about an open AI acquisition happened and Windsurf got quiet sort of like in the public marketing of it. So you felt like something was a foot and the next thing that we learned is that they're being cut off from anthropic models. So before any any acquisition was announced, this is the big thing. Before any actual official acquisition was announced, they got, you know, Anthropic was like, "Look, we're going to cut you off.
19:40You're, you know, you can't get anthropic models on Windsurf anymore."
19:43It looked like Anthropic was basically cutting off Windsurf before any kind of deal could be cemented. Now whether they unders whether they knew what the state of the deal was, who knows?
19:56But no OpenAI deal emerged, right? this is tracking so far. Instead, I and this could have been a week or two later. It wasn't too long, but it wasn't like the next day. It felt like whatever deal was emerging just got scuttled and never happened. And instead, Google acquired the Windsurf executive team.
20:19I found the timeline and the timeline is wild. It was first talked about in April of 25.
20:27Yeah. and it fell apart in July of 25.
20:32So that was quite some time, you know, and and also part of this is that anthropic models were were the best at the time unquestionably and now Windsurf lost access. So I you really saw like I think I switched off of Windsurf during this time like what could I do, you know? So they were losing users. Again, this is all kind of hearsay, but it kind of felt that way.
20:58And yep, some sort of intellectual property, property acquisition, um where the executives went to Google and and Cognition, which runs Devon, bought acquired later, um what was the rest of the Windsurf company, and Cognition has since grown quite a lot. Um, wow that was a long history lesson and yet it's happening again. So so the
21:27the current version of this now is open eye cutting off access to u gro. I don't want to go into like if this is a good idea or bad idea but like it is a reminder that these companies have conflicts and a lot of vested interests.
21:44I mean like if you are a user of cursor and you like these models are you going to switch off?
21:51probably I don't know. So, one of the things that came out because this was a little there was a little spicy back and forth on X on this where someone from Cursor it could have been one of the co-founders made the comment like oh less than five less than 5% of usage is on OpenAI models and someone pointed out that the choice of language there was very carefully you know usage and I understand what that means because I still have a Christian subscription and
22:19I've got like a $20 plan and on a $20 plan are really driven to use composer. Like you will be using composer unless you go out of your way. Um so I don't know what what that kind of makes out of it.
22:35I did find it interesting that Tibo himself actually engaged with his tweet and said he has questions about the 5% number. Um which I don't know that that was interesting.
22:48I actually take the point of I don't know I'm not choosing a side. I think I will just highlight that defaults matter and I actually like curs is probably right like whatever they drive to be the default is what's going to be used and they had not been really driving people to use anthropic models as they had in the past. It's probably worth pointing out that for a long time cursor was the default model. Sorry. Um like clawed models were the default on cursor for a long period, right? That has not been
23:16true for I don't know over six months if not way longer. Uh I think I think the first composer model came out in the fall of last year.
23:28So I don't know. It's certainly anyways I don't think it's even remotely close to the windsurf thing. Like we're talking postacquisition there's way more model choices today. Um you know what I mean?
23:40Yeah, I mean it is different for sure because I don't think curs is going away, right? It's like a they're part of a private company now. Um and uh we have we already talked about like that is probably um that was a good reason why XAI wanted cursor. But I think for a user it's probably going to be same like if you liked these models and you want to use those models that's a reason for you to go somewhere else. Um unless your company has a long-term contract with Kurser.
24:08Here's a question for you. Has there been it seems like in this like AI world, man, you better and you're and you're investing in a in a suite of tools. I hope you understand the politics of what you're choosing because you know you kind of have you kind of have to re you know your your choice could have consequences for reasons that seem kind of like out there these days, right? Does does does does the CEO of
24:35the tool you like get along with the CEO of the of the LLM you like? That could matter. Has there ever been a time where presuming you agree with me, has there ever been a time where that was important to know to make these kind of consequential decisions? you know, like this is this is a long time ago, but like I remember that um do you remember when Photoshop first went to become like a cloud Photoshop cloud thing? Like it was like you you you
25:04bought Photoshop at one point and then it sort of it slowly turned into like you get a subscription to the Adobe Creative Sweet Cloud thing. And I I remember people bitching on like the pricing, the the inability to unsubscribe for like ages and ages and ages and yet nothing changed, right? In a sense that they had such a market monopoly that like we're still paying for for Photoshop. It doesn't seem like
25:32any of these AI products have accomplished nearly quite as much of um like people will complain but then stick to it. It's it's almost like people will drop it like from one moment to another um at least a a meaningful number of people will drop it from one moment to another and move and that goes for model choices quite as much as it goes for um like the tools that they're using. And
26:00um so in some way I I really don't remember this like I I don't remember that someone was like oh like the either there's like CEOs have a fallout or something like nobody ever paid any attention to anything ever I can't believe how many CEOs names I know now and it and it's tiring you know like this is a small little aside but like uh this week a marchie quadro was released um by DHH which I think a lot
26:27of people have very strong opin about many times of my life did not have any idea what political leanings anyone ever had, whose product I was using. I really didn't care.
26:40But now like whatever the CEO is doing or thinking is seemingly very very important. And they're also all on podcasts, right? Like like cos did not use used to go on podcast. Now everybody's on every podcast. So something has changed uh really remarkably in in the last two to three years. The thing is like I I know for a fact that like I have worked together with people who in in like over the internet who I strongly disagreed with
27:08politically for a long period of time but it was never at the at the center of everything and I think it was a little bit better but also like maybe the each individual opinion was not quite as extreme like I think like the extreme sort of like like I'm quite liberal Right. So like you can kind of imagine like what is probably not sort of an acceptable opinion that I have that that I think other people should have. And I I think like most people I I sort of interacted with they
27:37were much closer to the center than than now. So like sort of the divisiveness is is just way worse.
27:47It is. This is normally the part at the end that we cut off by the way if everyone's curious like why you know but maybe we'll leave this one in. I think we can bleep some of some of some of my statement out.
27:57All right, let's continue to model choices. I think it's a good segue because we were just discussing that people have no brand um attachment to tools or very little. Seemingly people switch models quickly, all kinds of factors. Seemingly there's one part of the market where model choice is determined by how good they are.
28:23And on the other end they are determined by how cheap they are. And I think the question here is we have seen over the last year or so a gradual shift towards open eye models from entropic models. Uh I remember I switched when Andropic didn't allow me to use my subscription in my tool of choice. Um but on the lower part sort of the cheaper model we're we're
28:49seeing um that people really move quite quickly from one model to another um mostly based on not quality but price. So the the model here that we kind of refer to is GLM 5.3 which for um I would say a week or so where uh was available under the name Ox Alpha on various different routers for free
29:17and most of that volume completely collapsed um once it became paid right um and you commented earlier that you know DeepS v4 flash was available subsidized I think it was whether it was free or subsidized, right? Like I think it could have been right subsidized. So it was sort of made available very affordably. Um and on some
29:46subscriptions like Open Code Go, it meant you could go really far um with it. And once those subsidies went away or it got closer to its true pricing like that, that volume has also dropped.
29:57It's still high. I'm trying to resummarize. So, you've got like front everyone's got like a favorite Frontier model and then there's sort of like a a cheap model, right?
30:07Or that there's um I think we've talked about this before like I brought up that I actually like Composer or Kimmy for some tasks because I like how fast it is. Um but that's maybe a different metric, right? Like there's speed, there's intelligence, there's speed, and then there's price if that makes sense.
30:24Yep. sort of like maybe arguably three major metrics these days. Um could be the way that things shake out. Um I don't know where we want to go with this.
30:38I mean I think like one question is uh have you used any open weights models?
30:42And if yes, what have you used them for?
30:43Good timing for me going into this because I was smashing out a lot of tokens over the weekend. Um, doing some hunk stuff and the way that looks, some people ask like what does that look like? And I'll show that on the screen or I'll I'll share a recording of this is I'm up to like six checkouts. So I got like six checkouts of Ben runs a software factory. I
31:12it's a kind of a software factory where we've talked about this before where I'm spinning the plates and I'm keeping all the like I'm paying attention to everything. There is no autoactory but I I think what is changing is I'm spinning up longer um horizon jobs and I think sold is good at that. Um where you know a job can work for a few hours and I don't have to go check up on that tab for a while whereas in the past I'd probably have to check up pretty frequently. But anyways, I'll have like six different checkouts. And I'll actually have different harnesses and
31:42different models, usually PI or Open Code. Um, and I'm often using PI with Soul and sometimes Terra. And then I'm sometimes I'm often using Open Code um with OpenWeight models through Open Code Go or or Zen, which is their router. And I'm just I was trying a bunch. So I did try GLM, the new GLM 5.3. Um I actually what is the minmax? What is that one?
32:11Minax. Yeah. M something. I don't know which is public right now.
32:15Actually I think it was the newest Quen Quen Max I think. Is that a thing? Quen Max. And I know this isn't open but I actually tried Grock through um through the router as well and I was just paying token prices on that. I am feeling more than ever that we are converging if that makes sense.
32:36Like it makes sense. So I I I tell you my version of open weight models right now is I don't use them as much as I could.
32:44The reason is mostly they're too expensive.
32:48Why are they too expensive? Because like they are generally not subsidized to the same tune as my open eye subscriptions.
32:55But they can be if you now buy their subscriptions, right?
33:00Um I haven't done the math on how good some of those subscriptions go but um I do use open weight models quite a bit for one use case actually two use cases and that is cyber security. I think we talked about this before but um not only cyber security I also do I have some so I have um one of my side projects right now is I bought um a bunch of USB devices from China. Um the one the main one that I'm working with right now is a
33:27CarPlay adapter and um I bootstrap a lot of sessions with Kimmy because Kimmy does not care. It's very happy to reverse engineer a whole bunch of stuff and once you have a session started with Kimmy, you can switch to an open AI model and it usually continues just fine without running into the security harness. It's it's the it's the right loop that open AAI models need that they are convinced that everything is okay.
33:54So relevant to this weekend, someone opened up a security issue on hunk and I started working on it with soul and I came back to that tab because again I'm just going back through tabs and this is the first time I had the soul hey I can't do this because of security reasons which you know people the one the one way it asked you for signing up for the cyber program or the one where you already signed up for the
34:21cyber program. Oh, it was just like, nuh-uh, you're you're cut off.
34:28But I it might have said that, but my reaction was to do exactly what you described. I actually went to Grock because I'm like, Grock doesn't give a I I I don't know. Um I I I could have used GLM on this actually. Like I've lost track of what I did. But yeah, I exact same thing, you know, fed the report to an openweight model and now we have a solution. And I think that that's interesting because you brought up security. I'm literally trying to patch
34:56a live concern and I got slowed down because it's like, oh, you're doing a bad thing here.
35:02Um, it's a it's a it's a massive thing. So um without like pitching too much of our stuff, but like one of the applications that people do with Pi is like they're building their own custom security harnesses and and a pretty common question these days is like how do I get cheap Kimmy tokens because like it's a good model for cyberwork when everything else just craps out on you. And I think one like um
35:31like there is certain types of tasks where like as an example I mentioned this recently I wrote a blog post where I was just explaining how um reasoning works right like I I just explained like on a very basic level like what happens if you set reasoning effort high and the answer is what happens when I said reasoning effort high is it literally just puts reasoning high into the system prompt like there's nothing else that's happening.
35:56because it's just that the model has been trained to understand like its reasoning levels. I I showed in that blog post a very brief example of how a harmony token um rendered token stream looks like. So harmony is the internal format that OpenAI models use to denote special system tokens. OpenAI I think it was Terra refused to spellch checkck my blog post and threw me into a prompt refusal error. And I even put the
36:24screenshot into the blog post as a because you were describing the internals of kind of how it worked. Is that what you're saying?
36:30Basically, they they were afraid of distillation or some like that. I don't know. So, it it completely refused on me. So there's and and basically a lot of like if you're doing anything related to um so something that's very close to inference or very close to um something that like you don't just run into the cyber security thing you also run into the distillation um restrictions that they put in place and it's actually like if you work in the AI space it's it's surprising it's
36:58surprisingly easy to trigger um not just the cyber security part but also like the like their like proprietary kind of thing. Um, and you generally don't have this problem with open weight models which Yeah.
37:11Okay. So, look for us for like we're professionals and I mean you're working on a coding harness. We're going to we're going to mess around and change models, but do you think 90% of like the engineering industry is going to be toggling models and firing up this stuff all the time?
37:31So I think that the problem is that there are really two economies, right?
37:35The economies of a regular developer signs up for a subscription um and finds themselves with a massive subsidy compared to API prices. And the situation with the API prices is that one way to get them slightly cheaper is a extended period contract.
37:55And when I see the ridiculous low budgets that some people work within their companies that basically like gives them I don't know two sessions a day at times but also no ability to switch to different model um they cannot switch because like the company had a special contract with entropic and like they only give you that much um in in token spend.
38:25Um, so I think like greatly depends on like how do you get your tokens? Is it the choice that you make yourself with a sub subsidized subscription or did your company make the choice for you? And presumably most money is made by the company from the companies, not from subscriptions. Well, I think um if you're a token maxer, you have multiple subscriptions and then you juggle them.
38:43But but some people have a lot of these subs um and so then they they have them from different providers. That's not what the average person is doing. And I don't know what the average person's source of models and tokos is these days. I I I really don't know.
38:59I think it's important to state that what we see on the internet is not reflective of the industry. For example, actually I even raised an eyebrow earlier because you made the comment of like, oh, we're seeing a trend towards people adopting, you know, open AAI models. That's on that's on X and the internet. I don't know that that's actually broadly true. Does that make sense? I I still use both quite heavily.
39:21Yeah. I don't know if we should get into this, but man, X really hates anthropic. And I think that's fascinating. Okay. Like I think it's really fascinating.
39:35Now, I guess everyone is upset that I guess there's two events that one could point to why they might feel this way. One is the thing we talked about earlier, which is they cut off wind surf. And a lot of people like wind surf. I like wind surf. That's fair. And the second one which is probably I think more meaningful is that you know I can't use my cloud code subscription with alternative harnesses and open claw etc. That's probably the biggest one.
40:02But the one thing I highlight is it was only ever a cloud code subscription. Like they never actually misrepresented what that was.
40:12Do you know what I mean? it's that everybody kind of ran away with what you know once they began to see what they could take away from it and and that became like their expectation.
40:24Um you know they got upset when it was taken away but I I distinctly remember you can go on you know archive internet archive it was only ever a cloud code subscript uh subscription. I mean like it was we all when we when we integrated into other harnesses we all pretended to be cloud code like there was no surprise in in in any way. Um I think what was interesting is that at that point in time OpenAI did not have let's say product market fit on the coding
40:54situation at all and it just leaned into this moment to capture some market share from Entropic.
41:04Like if if Entropic wouldn't have done this, I don't think we would have seen open because like I remember this was like basically 48 hours where I don't remember who Mario talked with at the time. Basically like yeah just do whatever Codex is doing. It's fine for now. Like we we still to this day basically hit a Codex proprietary endpoint in uh in PI that is completely undocumented or anything. So that just was like a moment's decision by OpenAI
41:31to to lean in on that. Before that, we had to use exactly the Codic system prompt. Like PI could not use its minimal system prompt up until the moment someone from OpenAI said like, "Yeah, we're going to add your minimal system prompt to the allowed list."
41:46So I guess I'm just taking a moment to say like, look, clearly like influencers on X who were embroiled in this like love OpenAI for this, but I don't know that the broad industry even cared about the CL code. max debacle. Do you know what I mean? Like, you know, 99% of the world was not running an open claw and and it just probably just didn't matter. So, um yeah, I don't know.
42:08Yeah. I don't I I don't even know what like the split is. I just know that um there is a shift towards because like Codex didn't start out like on their own harness. They basically started very small and they have a sizable user base now. um they moved the entire org under HGP or JGP under Codex I guess. So it's like they are like they're all in on coding now because Entropic has demonstrated that that thing works.
42:35Um so they're definitely growing and getting some some share. I will say this, for all of the blowback, hatred, whatever, I think a lot of people still have anthropic subscriptions. And um even some people who have declared that they will never use those subscriptions again publicly still have those subscriptions.
42:57But I don't hear a lot of people talking about their ZAI subscriptions. I don't hear people talking a lot about their Grock subscriptions. Do you know what I mean?
43:05Yeah. I I think there's a even though those products exist now, I actually think there is far more stickiness than we than we recognize and a lot of it is just brand and and just there is product stickiness. Like the longer that we get that people get comfortable with something, the harder it is to give up.
43:23So I don't know. I know that we in our weird bubble like we'll go and we'll leave and we'll try a lot of things. Um, but I don't I just haven't seen indicators that I I like I don't know anybody who is running like a a fireworks subscription right now. Like are you?
43:39No, I I I don't No, but like I I remember like when when when when Entropic people like when Entropic dropped the support for Pi like that Discord moved to Codex because like they at that time I don't know if it's still the case but they liked working with Pi more than they cared about entropic stuff. And I think like because that that coupling doesn't fully exist at the moment like the competition seems to be somewhat alive.
44:06Most people probably just were happy with cloud code to begin with, right?
44:10Like if you use cloud code and you you like the product like why would you move?
44:14Um so it's there's a difference there when when your choice is like that you can't move. It's like Android versus iOS. You're you have some sort of attachment to it. I'll I'll finish with this, which is I I do actually believe, you know, you got me thinking here that maybe I should switch one of, you know, I should I should get one of these subs because I do think the openweight models, I do think the alternative models are competitive enough. I did spend I mentioned Grock, not openweight, but Grock's pretty good. I'm actually
44:42pretty impressed with it. I won't use it for other reasons.
44:48Are you afraid that it will give you an anime girl in your code base?
44:53I I don't want to go down this rabbit hole, but I could see that there's actually a strong argument to picking now. There's a lot of choices for subscriptions. They're going to probably going to rule for a while. Anyh who, we've been talking about coding agents and models. I think a big change that's happened over the last couple months is well, here's how we can connect these two things.
45:17You know, a lot of people didn't notice the subscription change because they were just using cloud code.
45:23And I so I think that I think that people have learned that it's going to be really hard to dethrone coding agents right now. There's a there's there's a lot of them. You're you're working on one of them. So I think what you're seeing is well can we actually just change the venue entirely where people code, right? If we can't if we can't introduce an editor, can we bring it into Slack? Can we bring it into alternative places where you
45:50could be coding? Um, Linear has a coding agent. Now, Devon and Cognition is basically like a cloud coding agent that's mostly driven through Slack. And I think that they announced, you know, record revenue or whatever. And then Grockbot came out and I think that that also can do a lot of code. Grockbot being like a personal agent with, you know, I presume Grock 4.6 and an orb or a cloud or whatever you want to call it.
46:19So I feel like I want I want to talk about you've talked about this. Will you even be coding in an IDE? Will you even be coding in a coding agent? That's the next one. A traditional a traditional coding agent. Before cloud code, there was in fact quite a lot of agents running in the cloud. It was Devon. It was the original Codex.
46:43It was entropics. Sorry, open mice. was openai's chatb product which did have a cloud thingy attached to it where you could also get some Python code running. So like it it started in the cloud then it moved local with cloud code which created this huge amount of excitement.
47:10We all got hooked and now it's moving back to the cloud. And part of the reason it moves back to the cloud is a it's a thing that you can sell if you're not OpenI. So you can see a lot of companies uh providing this. Now AMP has Orbs um like the all these kind of different um like hosted versions of of Asians.
47:32But then also um it is easier to sandbox a cloud thingy than it is to sandbox your local computer. And we have already seen quite a few interesting um problems recently with the cyber security models getting very good at escaping any kind of sandbox that they're in. And so um some people feel less happy about having this stuff running on their computer.
47:55And you don't have to crack your laptop open, you know, and like go to the club, go to the bar and keep your laptop open because you got to keep your agents running.
48:09I mean, we talked about it last time like how much do you run in your cloud?
48:12Yeah, I know. But I guess I'm So there's like cloud c Oh my goodness. Cloud cloud coding agents.
48:23Yes, they're back. But I guess I'm going to the like alternative ways that you're now like working with them. It's sort of like coding cloud coding agents can really be dropped in anywhere, right?
48:34And we're seeing some of that. Um, I think the linear one is an interesting one. I haven't used it, but I've watched videos of it, which is like, well, you got a ticket right here. Why don't you just fire up your coding agent? And I think they actually like pull out a sidebar. Like I think they've got multiple agents you can talk to, like Devon's in there, right? So, this is more about the I guess it's the surface like we're seeing more surfaces for coding agents where almost like you know what, don't even go to your editor. No, no, no. Like, don't
49:05Don't Oh, don't don't open Pi. What are you doing? What are you doing here? Talk to talk to my bot on on Slack. Does that make sense?
49:12Yeah. I mean, it's like it's the move to the software factory, right? It's like um somehow everything is set up in a way where support ticket comes in and it magically gets picked up by an army of agents.
49:25Um I mean we were seeing this like in a lot of projects like um Robo Bunnropic is sort of like a good example of like you just open a support ticket against like an and issue a report on bun and like within 10 minutes agent comes by and just makes the pull request. This is also I think that companies understand this too which is you know people try their products a year ago like ah whatever and and like that the idea of
49:53that product is like now cemented in your mind from that one experience when it wasn't amazing and it's hard for you to come back to it. People have this problem with modem by the way like had a had a lot of people sign up for it in February and I'm like man we've changed it so much.
50:11We've changed it so much. So interesting from Devon to be like, "Look, come and we'll make it really good. We'll make it just very appealing for you to come and try it again." And I did that. And the way that Devon works is you're not even creating a ticket. You're just like chatting and Devon just kind of pops up and is like interesting. I've already I've diving into this, you know, like you don't even it used to be you'd be like, "Hey, go take a look at this bot."
50:40Now it's just like proactive burning tokens and being like, "Yeah, you know, I'm investigating it." Um, so that's even interesting. It's like sort of just constantly trying to make uh like scanning your text and trying to make a determination whether there's some context here to justify to justify burning some tokens in pursuit of achieving an outcome. Does that make sense?
51:04And I f it's quite aggressive at that, which is also interesting, which is why they probably need to give you a bunch of free tokens to use it to to to value to like experience it. And so it made me think and what I wanted to talk about is like I guess you could argue Open Claw is a little bit like this too, right?
51:20I mean, have you seen the latest version of OpenClaw? It looks completely different from what it was. Like I was always thinking like everybody remembers OpenClaw as being this thing on WhatsApp and now it's like it looks like Cordex.
51:32Oh yeah. So I should try it again is what you're saying.
51:35I don't know. Look, here's an here's an interesting stat. Open claw merges the same number of commits or pull requests as the Linux kernel does. The repository is 3.8 gigabytes is like 100,000 commits since its creation. Like it added 5 million lines of code I think recently. It's huge.
52:01It's still the pinnacle of like pushing the envelope what you can do with an agent for sure. Um, a lot of stuff landed for sure. Like I I I mean I I mentioned this before the call, but like I'm muting so much stuff on Twitter now. Like I barely read Twitter anymore because like there are some meaningful developments I think in in in what we're doing. I think like there's there's definitely like under the like if you if you ignore the apps that are being built like on the on the
52:29sort of the level of like technology like a lot of interesting things have happened like MCP is like in a much better state now than it was a year ago.
52:38Um there's there's some kind of interesting things happening where people are trying to get control over like how they're writing code. All the kind of software you can try. I like nothing really has changed anything to me in a year. I don't I don't remember picking up any AI product in the last in this year at least where I got where I really stuck to it. Like I nothing that
53:08I started using that's an AI product I still use and I tried so many. Um, I don't know if it's different for you, but like my my my my willingness to actually stick to something um like seemingly I I don't get I don't stick to anything other than the coding agent. The the one piece of technology I and I used that before that I I started using over the last let's say 12 months.
53:37It's not quite right. Last 10 months is Cloudflare. Like I I was a cloudfare user before but I was like ah this is just such a pain and then last year I was like okay now I have all these startup credits on AWS I'm going to build everything on AWS but then I was like okay you know it I just can't be bothered with it. Cloudflare works really well as it's like and I I like I went so deep on stuff I can build on Cloudflare now. That maybe is like one of the things that I really like use a
54:05lot now that I didn't use quite as much before but beyond that I don't know. I think nothing really has shifted my my day-to-day. I think categories of product are sticky. It's like there's there's there's some useful categories of software and then I keep using a category of software but like there's so much new category of software but it actually turns out just the coding agent like maybe on a different surface like I'm pretty sure like if if there's this if my favorite
54:34harness like works really well in Slack I'll keep using it there more and more but but it's not like I'm adopting a completely new product because like someone invented this great new product category that wasn't there before like that so far hasn't really to me at least we're you know these aren't these are just surfaces to the same experience which which are coding agents I look that's a good point actually this I think that helps explain this little
55:02section here which is we're not we're not in maybe we're not inventing enough things with AI and we're still just going to the well here on coding agents where a lot of the innovation is just how can I put coding agents everywhere I possibly can. In my ticket tracker, in my Slack, in my in my email where I'm just like looking at my desktop trying to figure out what needs a coding agent. Oh, my operating system needs a coding agent. My operating, you know, um is it is it Omachi or is it
55:33I think and I think like Omar is actually like again say what want about DHH but like I think Omari is interesting because like he puts a lot of effort into it and it is trying to be agent first.
55:47Yeah. Nothing on base camp is getting done right now. That's but I think it's that is the kind of experiments that I think like we should see more of which is like okay now that we know this agent thing is work like what do we have to change about how the systems work that agents interact with um like that is that is much more interesting and I think like there's a chance that Nyx actually has a resurgence because it is perfect for agents even though the people that built Nyx hate AI like some
56:16of the coolest pieces of technology that agents can actually benefit from are written by people who hate agents.
56:24It's very noticeable.
56:26Here's now I feel like we can get to this next topic which we got to do which is GitHub and Git. And I and I first I think it's funny as we're talking about like man let's think about all the places we can put agents. One of the first places that you know agents and coding agents were available outside of your IDE was GitHub.
56:48Um terrible experience.
56:50Well, they were there first and uh I don't know if these two things are connected, but obviously GitHub downtime is kind of out of control and we highlighted like you know what it's now the frequency I think we all understood the frequency has been up but now I think the length of time like we had a I don't know 10hour outage that couple weeks ago where you know it's it's like
57:17time stands still. Um, and I think right as that really big outage day, like the day after that really big outage day, which uh a couple things, one cursor announced that they have a a GitHub competitor, right? What what is it called?
57:36Origin. It's like the Git Origin. Like you pushed Origin. It's clever.
57:40That's good. That's a good name.
57:43I I'm gonna pour one out for Origin Systems, though. um game developer.
57:50Um so that happened. Let's come back to that. But also um Pure Computer Company which creates a lot of open source libraries that people are using right now like diffs and trees. They um they announced their like code storage platform. You can sign up and you can they basically have like git primitives that you can build on top of. If people didn't know this, Pure Computer Company basically started by kind of building a GitHub competitor and it ended up kind
58:19of getting decomposed into all these parts. Obviously, GitHub. So, I think there are like two things we sort of should separate. One of which is um GitHub for enterprises, companies, whatever. um is uh a market that I think a lot of people are trying to address uh because there's money in it. GitHub for open source uh is not so much a market as it always been something to subsidize.
58:54It used to be the economics of subsidizing public repositories was pretty cheap and made sense. it probably doesn't make sense anymore. Go ahead.
59:03Yeah. Um and and I think on the public part, we're now entering year three or four of um culture wars because um the many open source communities are now making choices between AI okay or AI not okay. And um I mean like like there's a little bit of attention, but it's like the Russ
59:32project said AI fine. Deavian said AI is okay. Um Zik said AI not okay. And um with these AI, okay, I not okay, we now also find that some of the projects that are trying to step into GitHub's u um like footprints for um hosting public repositories are now choosing AI
1:00:02not okay. So the two ma most well-known ones are codeber and uh what was the other one called? Um source hut. Both of them now basically restrict AI code not to go there. And source in particular also said like you cannot host the Linux kernel on source because the Linux kernel allows AI contributions. Um and that I think is like it goes a whole new uh yeah I just want to make a comment that uh
1:00:30source hut is appropriately named in terms of uh you know what level of technology we're working with there. Um, I don't know. I have so many source cave soon.
1:00:50I in a way I actually like that there are these communities have this really extreme sort of anti-cultural movement kind of views on this um because it's like it's also a signal of sorts. The the market the addressable market has to become smaller and smaller over time. I think there's no other way. um presumably and then there's obviously tangled which has um uh which is sort of a distributed version of git hub um and they are they AI friendly so it's not like every single sort of open source
1:01:19GitHub alternative is anti-AI but um yeah two sort of like rather well-known ones are um look I don't know what's going to happen um I think it would be a shame if GitHub does not find a good replacement because GitHub is a really good resource to share open source code and it's free and it is free because
1:01:45companies use it and as much as I hate parts of GitHub and its lack of stability I don't really see anyone else subsidizing open source to the same tune I feel like that is like the brilliance of GitHub right like we could never get companies to pay for open source in the traditional sense. So, GitHub is sort of like, well, we'll provide these commercial services for you and that
1:02:13sort of inherently subsidizes all these like public services for open source, right? In a way that has felt economically like mutually beneficial for a while now, right? I don't know what's going to happen either, but I think I'm going to be sad.
1:02:30Yeah, I there's a there's a high chance that something not so good is going to happen here. Yeah, I mean like I'm sort of optimistic that something will happen because I think open source is way too entrenched, but it the the the barrier to entry for good open source code might go up because you might have to pay for that at one point. Um I think that's that that might be what we're what we're going to find or a lot of open source code will just not be available after some period of time.
1:02:56Like it used to be very common that we have to host our own stuff and the server that I used to run my old Mercurial source code on version subversion on is is no longer there like like that old code you like unless you find an old Debian installation where this was redistributed you will not find it.
1:03:18Um so yeah we might be going back to that. I think I think for me I think it's because I don't really think I changed anything like I feel like I've reached a stable equilibrium where like stuff works for me now more or less. Um I'm I'm not trying to solve problems which are not like I have I've acclimated to enchanting engineering.
1:03:42This is the new normal now. You're in a permanent avatar state is I don't know if you've watched The Last Airbender. Um but but do you feel like you you're like do you feel like you stabilized on your agentic use or do you feel like um now like you're you're scaling up like exponentially and new stuff emerges all the time.
1:04:05All right. Well, now we're entering this section which is what have we learned lately which we wrote down. I do feel like I'm still getting better.
1:04:16I still feel like there is there are still more techniques to learn, you know. And what does getting better mean?
1:04:23Let's let's start there. To me, uh getting better means I can achieve more of the outcomes that I want, right?
1:04:33And outcomes for me means the you know the ideas in my head manifest as some sort of piece of software. That's one.
1:04:41they they come to exist. They function with um a reasonable level of quality and not a lot of people care about this one anymore, but I still do. And the software architecture and choices that serve that those outcomes is like good and flexible and makes tradeoffs. And that's not I I want to be clear that's not because of some um like random
1:05:10what's the word like taste based thing.
1:05:12I need it to look this way. It's because I'm probably going to want to refactor this or I'm going to want to change it and I want the code to be flexible in such a way that that will be easy in the future. Is that a reasonable set of outcomes for you for like what getting better looks like?
1:05:31I think I'm achieving more of that. I achieved a lot more this month also measured in my token spend or you know but I do think ignoring the tokens like this is not a token maxing comment. I think it's like I do think I've achieved a lot more. I do think some of that came with token maxing a little bit. Um and some of it came with the latest models and some of it came with just a strategy for for working
1:05:57with things. I'll give a more concrete example of something I've done lately that has helped me. For example, using this isn't even like an AI thing, but using um GitHub stacked pull requests.
1:06:12You're the one user of it.
1:06:15Yeah. So, by the way, you have to teach your agent that you know that's not it's not in the models, right? So, you have to either give your agent a skill file. I find just like hey use GitHub stack PRs do you know GitHub-help you'll find it that's usually enough. Um but for example that means I spend less time rebasing. If you've ever, you know, if you try to build five parallel PRs and they all conflict in tiny ways, I'm learning that I can sort of queue up a
1:06:43bunch of jobs and I could be like, "Hey, you know, which of these do you think are going to intersect and I should do stacked PRs for and which of these do you think I could have a sub agent go and build off on the side because it's not going to conflict that?" So that's like sometimes now I'm structuring the work like it used to be a lot of the Q&A is like let me understand what you want to achieve for the product. Yeah, I can see that. I mean, I I don't work like this at all.
1:07:08Um my my productivity has not gone up recently. I I can guarantee that. I think like if you go to my GitHub graph, it might actually be lower than my peak primarily because like what is it really that limits my ability to do more? is that like I can obviously ship a ton of things if I want to, but I cannot uphold the bar that I want to achieve and I haven't found a way to get to that.
1:07:37So some of this for me I think is because hunk is a side project by the way. So like I'm mostly describing hunk stuff. I think when I'm working on modem it's way more deliberate.
1:07:46I was saying this recently. I think like we have a lot more success with side projects than we have with work.
1:07:50Yeah. Part of that is because like modem is a running infrastructure project. That's one. It's serving customers. I'm a little less yolo with that.
1:08:01I'm a lot less yolo with that. Um and also my my role in the company is different. Like there are engineers on the team. I'm not like the the number one engineer. The other thing for Hunk is I think I've done and we do actually bring a lot of this tom is man we've built a lot of I think it because it is part-time.
1:08:21I actually have to think more strategically about how do I help this project be successful with the limited resources that I have. If that makes sense. Like if you've got a full week, you you're going to apply your time differently. So Hunk has like a crazy number of guards that I think a lot of project I think it would be pretty extreme for a lot of projects. Maybe you maybe you say, "Oh, Pi has all this too." And I could obviously answer that very quickly if I were typing this to an agent. But
1:08:48um like what does it have? It has 94% code coverage. It has a full integration suite with a pseudo terminal. So, it's doing something like a like playright in the browser except it's actually firing up a pseudo terminal and like smashing keys on it to represent all the different states. Um, and that thing runs for like three minutes. Um, when I start a release, there's actually a a CI
1:09:15job that does runs a benchmark and snapshots something like 18 different benchmarks, different metrics to track um, you know, what is the scroll speed of like scrolling through the stiff, what is the first boot, what is the cold first frame, what is the warm first frame, what is P95 scroll tick. So all of these are being measured and then if the if there's a regression that's material, the release won't go like it actually cuts off the release. So you've
1:09:44got like a performance suite in there. Another thing I've done lately is now hunk has an install script.
1:09:50I know Pi has this. Lots of things have this where it's like a curl install script. I don't got time to deal with that. There's a lot of complicated scenarios around that. So now I've also been experimenting with um running um firecracker VMs to basically boot a bunch of different scenarios like install and update scenarios like oh you have an old version and now we're going to update it. you don't have a version.
1:10:17Now we're gonna install something raw or something like brand new and you know now we'll try homebrew over here and now we'll try nyx over here and that's now I didn't make that part of like a suite because it's slow and you actually need some you need some hardware for that but it is like a skill and a bunch of scripts to generate that and so I will generate that quite a bit.
1:10:37Thoughts on this so far by the way?
1:10:39Uh I have a lot of thoughts on this. Pi is the exception I think to all the coding harnesses right now in that it doesn't even have rooi like there's no no machine comes in and does something automatically and it's not because we're lazy it is because a lot of things can be built and maybe the requirement has to be that we're going to do all of those things but I haven't actually seen that to make
1:11:06a material difference to What makes present day pi pi? Now there's always stuff we want to build in the future and there there's a dev branch. If we go there, we'll find a completely different product. But but even that product is built by humans and with a machine, but but not by like like have stuff appear randomly. And and I don't mean in the sense of like um the things that you're doing are random, but in the sense that there's a lot of stuff
1:11:35we could build for pi. And when I look at any most harnesses out there, most harnesses are in fact at this point being built by just ship it like they like you can find Pi forks, you can find and lot of other harnesses where like you open a ticket and you basically get a feature within like the next three to four days landed. I'm have no conviction that that makes our product any way better. Um, and so for instance, like
1:12:05right now, one of the things that we have been starting to do at Pi is getting rid of tests.
1:12:09Um, why? Because like it has become too annoying to run all the tests. Like it it takes just too much right now for me to run the tests on my machine. And like the the the the what is hard about software engineering continues to be hard. And and and actually like saying no is materially harder if the machine keeps making pull requests. Um, and I think saying no is still really important part about the whole thing.
1:12:36Um, so we we haven't achieved the soft factory yet. I think we want to get to the sofa factory, but we want to go to the one that that works for us and we haven't so far succeeded in that. Um, so yeah, like I feel like we have like at least for within Arendel, we have massively stagnated in in in in how we do aching engineering. I think you're highlighting this like a lot of it is organizational or like going back it's like uh oh I'm a solo person who spends way too much of their Saturday and Sunday working on
1:13:08I don't want to be fixing bugs. I think that's the big part. Does that make sense? Like I just I don't have time for regressions. I do not I would much rather invest my time to avoid going backwards. I just can't go backwards. I think like um as an example like I have the side project I have so many side projects but like on all my side projects like who cares but I have a side project where like I feel like I'm materially more um productive than I've ever been before because like I have basically an agent
1:13:35for the last two days just doing something on the side trying to get a bunch of Chinese USB devices work with custom firmware and I I wouldn't have done that before because it's just too much effort for me and in the and I will have a working thing I can do the stuff with that I wanted to do. But like it the the the hill climbing of actually getting the damn thing to boot my Linux um I wouldn't have done before. So like that is great and it it does it it takes out an enormous amount of um of like
1:14:05time that I would otherwise not have been able to put to this. So like it's and that is a net improvement over probably like even three to six months ago. like I'm I'm trying like quite ambitious projects now just because like the agents are capable of doing that and I think a lot of people are starting to start these projects because they they figure out the agents are going to do that but in the in the team setting in a working together kind of environment like at least at the Arendelle we don't see the same
1:14:32um thing and one of the things that I'm thankfully Mario is mostly pushing back on this but like in general that I'm I think we all started to pushing back on this is like accepting not understanding why a piece of code is in there because it's so easy that you ship something where like oh yeah I know what it does but if someone actually asks you like can you really explain me what it does they're like h and I don't think that's how software engineering should be so I'm I've been
1:15:00thinking about this look I I think it's very clear like hunk is very vibe generated but I I've used this language before tasteful generation I agree with The difference is this. Yes. Do I merge things that I don't completely understand? Yes. Do I? First of all, I always do actually look at the code though. And a lot of it is assessing the shape and looking for things that, you know, I'm like, but well, should it really do that? I think it's like, do I
1:15:28know how it works in general versus do I know every single line? I think those are two different things. Does that make sense?
1:15:35The second part or where I'm going with this is sometimes you don't know what it is when you land it, but will you go and learn it later? And for me, the answer is yes. It's more of a like it's almost like not blocking yourself.
1:15:51Like look, get it in there, passes the test, it's functionally correct, we'll go and we'll fix it later. I think if you actually had that mindset like I actually spend a lot of time fixing and in the act of fixing now you learn a tremendous amount about how it works right does that make sense like and I'm finding that that model works good for me so kind of like aggressively merging things and we've established like I have all these gates so there's
1:16:20sort of like you know a floor of qu of of of fundamental output quality exists and Then constant refreshers honestly usually around refactoring tasks. Like I just go in and I'm like I know that there are things that I can refactor here. Let's go find what those are. And then you discover what those are and you learn so much about the code and you're like oh man this is bad. And you know what? I think that's what software engineering has always been.
1:16:49If does that make sense? like yeah I mean look I I think like there's a there's a version of this where someone will say like oh but the same like nobody reads the assembly anymore it's just a C++ maybe there is sort of an element to this I don't know I can I I think like someone can take the other side of this argument and say like okay this is actually like nobody needs to understand everything anymore I'm just personally not mentally there and I already said this before like even
1:17:18if I think like we as engineering teams would arrive at this point and are okay with it. I think there are plenty of situations where the people that are using our software wouldn't be okay with it.
1:17:27Yep. I have an advantage over you, Armen. Also, I love I love that like I can say I think this is true. Like I can say stuff for you and you already know where I'm going with this. I'm mildly stupider than you. Okay? See, he knows. And as a result, you know, I can let go.
1:17:49I can let go in a different way. Okay, does that make sense? You know, like that can be a strength in the right setting. You know, it's sort of like for example, if you're incredibly skilled at reading code, you can look at the code and you'll be like, you know what, give myself five minutes. I'm really going to understand what's here.
1:18:09I can look at it and I can go, it's going to take me an hour to figure this out. It's not worth it. So, I can Does that make sense? It's almost like the the math computes differently.
1:18:21Basically, it got me thinking like when did I in when did I work with open source software in the past where the developer didn't actually understand what they were doing. This is sort of like I'm not saying it's the same thing but like in a way that there have been open source projects before that worked but they were human slop right it's like that like that it it worked but it was internally not how you should design a system like it was it was um like and here's an example of this like I'm not going to
1:18:50badmouth any software here I have worked with libraries in the Python ecosystem that had a lot of users but they were basically layers and layers and layers of hacks and other stuff over another system and they were um not designed. They were just like you you started somewhere, someone built it in 93 and it was still software that we used in 2020 and somewhere in between we put layers and layers of
1:19:19stuff on top and nobody fully understood the system anymore because like the original developers were gone and so but it worked right and um keen observers here might know that I'm describing the pipe packaging ecosystem um and it it really took until um like some outside person in this case it was like Charlie from from uh from Astral so like okay let's let's really rethink packaging from from first
1:19:48principles and re and rebuild it and I think like up to that point like there might be a couple of people that fully understood what's going on but like it was it was like if you go to Wikipedia and look what the car culting is but like it's this idea that like you you lost the understanding of like what actually led to the things as they were, but it didn't mean it was broken. It just meant that it was very hard to work on this code bases. That might be a very strong argument for we don't need to understand what's going on because like
1:20:17what really matters is that it sort of works good enough for all the use cases that there are there. What would be a good idea I think is if if some people took some like cohort studies is it called this way where you take a bunch of different projects some of which are AI supported some of which are not AI supported and you look at them in five years time and and see how everybody was doing and what they were doing and and I think that would be interesting like what we're doing right now is irrelevant like if if we don't ship something that is still there in five years and like
1:20:47the world moved on in a way um and if we look at this in five years and or 10 years and then then investigate like did what we built make sense or did we have to change something about AI approaches because it wasn't scalable or I think it's happening. I think that this is the year where you're actually seeing like I've talked how many of the software that many pieces of software that people are like really proud of that they vibe coded are actually mostly built off of like great libraries like
1:21:16lip ghosty or or diffs or whatever, right? But I think that you're actually now seeing this year you're seeing apps that are um you know agent generated and you know they are because they've they've they've had these this meteor meteoric rise in like two months. Um we tal I mentioned her last time that is only like a three-month old project.
1:21:41It's like and and I think it's like pretty good software. Um I think we're going to see more of that. I think I think Hunk is one of those. Um, so I don't think we have to wait five years.
1:21:51I think we're going to see I think the people are starting to figure it out this year and we're going to see more of it but I do think we've only started like you're only seeing the beginnings of it which I think is exciting because there's going to be version two and version three of like the superior thing of this and surprise they're mostly around um still coding right herder multiplexer and and you know diffs were still these
1:22:19are still around the like um coding world. I think where you're going is will there be a vibecoded superior like UV? Is this kind of where you're going?
1:22:31Like will there be something?
1:22:32Obviously you have to compare similar things, right? So like ideally we have a bunch of people that build programming languages which are like I hope sik never adopts AI. Why do I hope that sik never adopts AI? because it would be a very good um what do you call this when you do like a a study with two it's a control it's a very useful control right and then there will be plenty of AI coded languages and and presumably and so then we have something to compare but we can look at this in five years and then maybe out of the
1:23:01many AI coded languages some of them with the completely terrible approaches where nobody understood anything anymore but some of them had a really good one and they shipped something much more impressive so that's kind of where I'm going which is like there we will we will know later what worked and what didn't work and um yeah but it will be the like yeah I will say this and because we're we're still talking about like what have we learned from models a little bit I think
1:23:30that you still need to be skilled right if for example if I thought about those projects that I named earlier was a planetator is another one I guess because maybe my in my world. I'm looking at a lot of these things. They're still they're still by skilled practitioners. And it's very obvious to me in the LLM responses why that's true.
1:23:54I I can give like a very clear answer from yesterday for example. It's mostly architectural. I think this is the big thing. Or even how to pursue a problem or what problems exist. I think that they they can generate ideas around a space, but you have to know you have to describe that space, right? If you can't describe, hey, there's this package manager idea and I think it could work this way. You're just not going to be like, you know, you can't just talk to
1:24:23an element and be like, hey, let's make a better package manager. I think for like 90% of the industry right now, it's still like, hey, let me make a better X, right? And and you're react, you're just not going to be very successful. That's that's my sense. I haven't seen that happen. Let me bring up that example from yesterday. So, I'm building haha self-building software into Hunk. You can extend it. Um, it's already live.
1:24:45There's a bunch of stuff you can do.
1:24:46There's a bunch of extensions, but there is a capability in hunk which is like open this line in an editor and it's a it's a you press E. It's a it's a key binding and that opens up dollar sign editor. So, whatever you've configured to be dollar sign editor on your machine, it opens up that line in that editor. And I've been looking to sort of rebuild different components on extensions. Okay. Um, open, you know, I don't know if pi 2 is going in this direction. I think so. Open code 2 is
1:25:15going in this direction. Lots of people are. Um, and to me it's like if I can if I can build more and more and more of of the software on the on the extensions and the plug-in system, then I know it can support more and more use cases. And I find that as I take more and more components and I do that, I uncover these gaps, right? Right? And so the extension system gets better every time.
1:25:35So this was a candidate which is like well why should why is open in editor kind of like a bakedin feature. It should be open anything. Why not? That makes sense. So I started building that and I you know talking with an LL like hey yeah look yeah let's take this open in in editor thing and let's go let's turn that into an extension. Okay. And you know the LLM is like great great idea. Love it. Oh you're gonna you know you're going to prove this out. And so I let that spin for like two hours and I
1:26:03come back and you know what it built? It built an extension plugin. It built a a an extension hook called um open in editor.
1:26:17So it really it really didn't add anything at all. I pushed back. Okay. And I was like I don't think that's very extensible. And it's like oh but look I I disagree. it actually pushed back again.
1:26:31I I don't think it's enough that don't want to go too much into detail. We found a solution. Um and we did it for like three four things. So like we found independent solutions like this is like fully designed back to forth correct validated like in isolation works.
1:26:47There's a debate if it actually works but like where's like final sign off design done and then you put it together and the agent's like it worked perfectly fits together like a glove. And then you test it and it fits together perfectly like love and it works. And then you look at like how it did it and it made it work but it made it work in a way where it was very clear the design was fundamentally wrong. So it worked.
1:27:13It had all the right words. It had all the right abstraction. It demonstrated all the I could write all the plugins that I wanted and yet it was completely wrong. In fact, it it it it made it work, but it made it in a work in a way where like the shim that makes it work together. Got rid of all the whole great design that we came up with on the top because it didn't actually work. And so the agent as sort of the the modern
1:27:40agent that that solves every problem made a mess and and in fact what was wrong was our design was wrong, right?
1:27:49And or or at least the the the combination of the two designs were wrong. it didn't actually fit together.
1:27:53It was a massive impedance mismatch. And I think like it didn't matter that we knew better. It is that unless you pay very very very close attention. So I I think it's not just a question of experience. It's also like your willingness to actually put that energy in to make sure that it's actually really correct what it's doing and not just rely on the fact that it sort of tells you like, oh, I built this great connection and it worked really well.
1:28:16Oh yeah, absolutely. I I think these are some good advantages to share with people which is like look we spend so much time on this stuff and clearly we're producing software that people use um and it still requires incredible scrutiny and if you're I I think the thing I try to tell people is if you're not having these kinds of experiences that's a bad sign. Sorry.
1:28:43Because it means that you're you're pushing go, right? like you're pushing the like, yep, sounds good, looks good to me. You're hitting, you're smashing merge because this is where I am with my ability or this is what I want to achieve and I don't care. I think that's actually fine. That makes sense. But it is going to limit you. It's going to limit, you know, it just all depends what you what you want to do.
1:29:05take an asterisk to say the amount of care that I put towards um you know something like hunk or modem versus say a random ass side project I don't care a lot of stuff I don't care about the code for because frankly it might even be a shock if I'm still using it six months from now right I think even um was it GMO Rous like I think he wrote a good post on this which is there's like there's sort
1:29:32of like different ways of evaluating like how much you care about the code and you can't say like oh it's like depends on what you're doing and I agree with that. Um, for example, if you're building extension systems and plug-in systems and it's interoperability is pretty important and ex and you know long ter the long-term horizon of that extensibility and like the ramifications kind of critical is that is that fair
1:30:01you know API design I think that the real question in my mind here at the moment is that do you remember early on we had this discussion like oh I no longer care about code formatting because like we have moved beyond code formatting, right?
1:30:16And now I I I toyed around with a language that someone shared. It's like it turns out I no longer care about ID integration either, right? Like RL like language server. It's like who who cares? Like we have we have moved past that point. So clearly there are certain certain sort of creature comfort things that were previously sort of really important to me that I now have moved past. I'm kind of curious a little bit of like how do we move ourselves up?
1:30:40Like in theory, we should all work on more ambitious things. We should understand more. So we're moving to a point where some of the things that previously were occupying our mind should no longer occupy our mind. But we haven't found the way yet so that it actually can be done safely that or or at least I haven't found a way where I feel ready to make that move because I still see that underneath I don't feel any confidence in in in sort of like a sort of this sort of detaching and for
1:31:10me it's not a question of like do I know API or do I have this understanding from a pre-AII era to me that is not really what makes it work or not work. Um, it is despite all of my knowledge that I've gained over the years, I haven't found a responsible way for me to work with the AI, but I want to get there.
1:31:29I think I think by the way where my mind is going is that the shape of whatever like agentfriendly language is emerging for me because when I first I by the way I'll add some other things just it'll make sense it'll connect to this but I had this tweet go off about cyclatic complexity. I'm doing like cyclatic That's another example where by the way a bunch of people so I had a tweet that went sort of viral on this which was
1:31:58yeah look here's a before and after of doing a cyclomatic uh complexity check and this one I shared it because when I saw it in the poll request which one telling you I'm actually reading the code when I saw it I was like well this is so obviously better like I don't think I'd ever seen sort of such a stark before and after like it doesn't always look like that. So, I shared it and I think everybody could look at it and agree it was better. And that's just because it's less statements. Okay, it's just, you know, less statements achieving the exact same result. Um, and
1:32:27so increasingly I've just been like, well, look, less statements, less ballooning possibilities. I should actually just look at that as code. If I, you know, for debating whether you're looking at the code or not, why am I debating whether this less, you know, this arguably less complex code It's not even arguable. It's like objectively like if you're you know number of branching statements, right?
1:32:50Like just do it. Um so that's one and I'll come back to that. And then I'm also doing this um dependency cruiser is a library for JavaScript or TypeScript, but you actually define the shape of your components and it avoids um basically the LLM injecting sort of like code that would break your dependency graph. So you you know you define these different modules and you actually enforce it in code. So that's another thing I've added to Hunk. I'm experimenting with a bunch of this stuff. So, okay. So, what does that look
1:33:20like? I'm just wondering if it's like you brought up many times like we're going to need different languages or whatever. And you're highlighting though that there's there's sort of this world that you want to get to where you want like you want to let go. It's sort of like you you recognize that there's a there's to achieve some of those outcomes that you want, you are going to need to let go, but you're not there yet, right? We don't have the tools for it. So what if there was a tool to help us let go? Like for example, instead of a language about declarative,
1:33:49you know, statements, whatever. What if it's just like boxes? What if it's just like, you know, like literally Lego blocks that plug in in different ways?
1:33:58Okay. Like somewhere between language and libraries, but you know, not strictly libraries. I'm speaking really abstract here because I'm thinking to the dependency cruiser a little bit which is like yep you know I need an odd with these properties the the idea is not particularly new right it's like if you if you look at like how like engine control units work in cars like this this is like you have like inputs and outputs and how the machinery works internally is entirely irrelevant and
1:34:27then you sort of you you put chip chips together and like like at the end it's just one one thing gets printed but like the individual modules are like all of this works beautifully well in a world of stateless stuff or where all the state is held locally and I think like where it falls apart is that most software we're building is not neatly encapsulatable like this I I I did this blog post I think like a couple of months ago I wrote like this language for agents and an AI engineer I talked to a bunch of people that are playing
1:34:56with it and I think like I think we all want to get there but it's entirely unclear how to make this work really well in like the messy state that is the reality where like Pi's entire new plug-in system is basically how far can you take this and it turns out it's really bloody hard. You either constrain a system down so much that it's useless or you you make it work in the real
1:35:24world but then there if if the agent actually goes there and goes there for too long then it completely destroys your entire system. I think there is somewhere out there that's like a better programming language is a better programming model. Not for lack of trying. I had zero success in it and I think there will be a solution but I I I'm I'm at this point I feel like I'm running out of ideas of like how to do it without like really dramatically
1:35:53rethinking like how a lot of these systems work because one global variable screws everything or like there's like there's so many ways in which this like gets badly violated. Um I know it's very abstract but like I have Right now I have very little hope that um that that change will will happen with more linting or with more like um
1:36:21like stuff that we can put on top of JavaScript.
1:36:24I don't proclaim to be you know super smart at programming languages or anything. I guess in my mind it's like what is like one higher level of abstraction where I accept that there will be slop inside like sort of you know I accept that slop will be a thing and so if you what if you embraced it rather than try to fight it. What if you just embraced it and sort of tried to control it? Um hide the hide the slop behind neat abstractions. I mean, I think
1:36:53I think more people should think about it for a start and someone will find a solution, but I I I really think this problem is harder than it looks.
1:37:02Um, which is why maybe more people should try it because hard problems are worth solving.
1:37:07Yep. Chimamoth, by the way. Chimoth, I don't know his last name, but I think everyone knows who I'm talking about.
1:37:14Even he ca I saw that even he was like interviewed recently and he's like man AI is really bad at long horizon coding tasks and stuff and I'm bringing it up to say like once it's gotten to that that comment has gotten to that level of kind of mainstream commentary I think people are arriving there. Look, I think like one one and I don't know if it was Truman that said it, but like one of the I think like one of the things that sort of at one point was like uh maybe but definitely has some sort of mainstream
1:37:43uh like uh like awareness is that the even if you start at a high baseline, the increase in token spend is that the slope is higher than the productivity gains that you can measure on an organization. level and and and that is a like that that that is the case is starting to concern some people,
1:38:10right? It's like how is it that we are like we're we're we're we're shipping like x amount of lines of code. So they can see that the number of lines of code is going up, but if you look at the topline number that actually matters to anyone in the organization, it didn't move enough. Um and and I think like at least in part it's attributable to the fact that we don't know how to make the technology work really well in teams but also because we now write more code comparatively
1:38:40to the same feature that would previously we've written less code or we're shipping more features that didn't actually matter because now it's free right so we're increasing more complexity without actually achieving the outcomes that we want look in the beginning everyone was very excited and bought a lot of AI products, I can't fall behind. I don't know that that's true anymore. I think that um it's probably harder to break through than it was. Um because also people are becoming more mature about their
1:39:07understanding about where, you know, how AI hits, right? A year ago, you could give an AI demo and you and you'd have a buyer who's like, "Wow, awesome."
1:39:20And then they go and use it for six months and they're like, "Well, I don't want any more of this." Right? because the flaws are sort of are kind of materializing. Um, so I I guess I'm just saying like I do think that there I think we both agree the technology is incredible and you can achieve a lot but yeah may not actually drive there's there's so many other factors that it may not actually drive the value but it should right and I think like that to me this is the thing is like like this machine is amazing
1:39:49it's a cool machine is a very cool machine is a very capable machine and I think like if we if we were to stop burning tokens like mad and actually put it to the right like like I don't know like the right pieces we we can do great things but like like the the tech needed to use the machines correctly I think is not quite there and in the meantime we're behaving like
1:40:18chunkies who are doing everything we can with these things um and actually making it worse in the process and I think like that and that is like to a large degree I think It's just a psychological problem. Like the obviously the tech is not there, but like you could also just burn less fewer tokens, but we're not constraints constraints are good. You know, constraints have driven a lot of great products and maybe people need to bring in more of them. Anyways, we have
1:40:46been talking for a long time and uh but also I've look always enjoyed Armen. That's part of why we end up talking a lot and I know that we probably could keep going for another hour on this. Let's go let's go let's go smash on what this language is. Okay, hear me out. Let's give it a shot.
1:41:06Let's let's burn all tokens this month to build a new programming language. Like I I said the other day is like the plenty of startups now have the office of the slop which is basically like one or two engineers that are using AI things to like build everything like um I think we know some people that have startups that produce a lot of side projects. I want to work in that department.
1:41:33Sounds fun. I think we should stop here. Um it was great to talk. Hope that there's something interesting in here for somebody. Bye everybody. Bye-bye.
