0:00I actually believe that like you know if you're building production software if you run the models non-stop overnight you're going to create more code than you can like reliably consume use all my existing labels as as eval essentially if it agrees like you know more than you know x% of the time and then you only have to review the parts that disagree essentially I'm still pretty much at read only write with explicit permission is where I
0:35Okay, this is the first ever High Taste testers. Welcome, guys.
0:38Thank you. Thanks for having us.
0:40So, you've been gathered because you're people of high taste that you test models and products all the time. And we're going to talk about just all the latest and greatest. Uh we we're recording this right after open dev day.
0:50uh opening eye is not in the room. We can be as honest and frank as you like as many quad users and also like I don't know all the other stuff news gro whatever. Um so we're going to basically pair food with questions and think we can get the first course started but not squash croettes crispy fried chicken and for Ben bread bread but like very nicely presented bread I got this from like some kind of trader farmers market type thing.
1:17Dev day we just released uh we just came out of it. Uh what are people's impressions overall?
1:22Well, I don't know. Uh it's not really about the events. It's more about the releases.
1:26There's Dots, there's 6.1 Soul, there's Ultra Fast, there's like the notion competitor, which is like something that we're excited about.
1:36Is that actually like it's weird. It's Yeah, they're really interesting, right?
1:41Yeah. Uh I don't know. I don't actually I posted I posted our like company vision video some like Opus slop which we can talk about later. Uh, but Peter replied late last night. He's like, "Oh my god, dude. We built the same thing." And I was like, "Interesting." Okay. I didn't even realize he was behind that.
1:56But yeah, it's like uh I guess it's like open like assistant shape thing of like, "Hey, how do you collaborate on what all everybody's open call is doing."
2:04Yeah. It seems like a shared whiteboard type thing. You can write the markdown, do drawings or whatever. I mean, it's pretty good, but it's like deeply hidden within the UI and it has the same problem.
2:14Yeah. Like it's the thing that they do for everything where like you can ask Codex to visualize in the chat but no one knows that's a thing.
2:20Yeah, even I mean even Dots is pretty hidden.
2:22Yeah, Dots is very hidden. That's weird.
2:24I I thought that it was going to be a standalone app actually.
2:26I was shocked that it wasn't.
2:28My first feedback to them to Tibo was it needs to be standalone app.
2:31I have multiple things running with Astra with Codex Go stuff notifications. I want a separate assistant. They can look over there but send notification separately.
2:40My theory is that it's not a standalone app. It's a standalone device. So that's that's where it's coming. Hunk.
2:45I don't think it's that long term, man.
2:47Well, longterm in AI time.
2:49They said end of year.
2:50Yeah. Three to six months is basically long term at this point.
2:53Speaking of dev day, Johnny IV was not in this one. I haven't seen him. Last last day, he was there on stage with Sam aluminum and he promised something's coming.
3:03Where is it, Johnny? Where's Where's my Where's my physical device?
3:07Is Zach going to beat Open the Eye to a physical device this Christmas? That's going to be very interesting. Yeah, you're already audio. Perfect.
3:15They have a physical device, right? It's on my face. Muse is coming here. I wonder where dot is coming. That's not just on my phone like we said inside hidden inside the chat app.
3:23I mean, if you count the glasses, they're already there. I'm guessing open is probably going to do the like home assistant thing like the the Google the Amazon whatever that was called the Echo I think it was.
3:32So, this is very interesting. And for like probably like at least two and a half years, I've been like why has no one done this? It's very easy. My friends and I actually like Smart Echo.
3:42Yeah. Yeah. like a like Google Assistant or Alexa, but like you know intelligence from the last like 5 years or something like a proper like assistant harness type of thing. Yeah.
3:51And so like my my friends and I built one over a single weekend like a week or two ago using like Live and like it it works like you know there's not like I don't know why has put it into a home like I would buy it.
4:06Yeah. I mean Google did it with Gemini but Gemini is just still still still trying to catch up right. It's still Gemini.
4:12Yeah. And like I I think the difference is like there's like the old way and there's the new like it has like computer use tools like MCP CLI all like that entire harness and no one has really done that although I haven't used Google for a long time. This was my challenge with dot too is uh it I was like I have I have a workflow that we used to run the company and it's a git repo full of markdown files and like scripts that my agent has sort of like openclaw shaped before openclaw was an option and I told I told my cloud running in
4:41that repo to go configure my use computer use to go configure my dot overnight. I want to wake up with a dot that can do all the stuff in this repo.
4:48It's like all the O stuff is really wonky. Like if like I don't know, we have a script that pulls like data from Mercury from our bank account to like write weekly updates on things like this. And like there's no Mercury plugin in chat GPT. So unless I like leave my codeex desktop app open and running so that my dot can talk to it, it can't actually do any of the off or use any of the scripts that I've like accumulated over the year.
5:09Yeah. One of the most unsolved things to me about the whole story is like the where is stuff working thing is still completely unsolved. Like I still don't really get how to use they they talked about like you know they had this sort of like um you know uh uh you know mission accomplished like you can close your laptops now.
5:28My laptop's still closed because I don't Yeah. Yeah. But like I don't I kind of like saw the cloud button. I tried looking at it. I didn't really get how to set it up. up like it wanted me to do all these things and then now my laptop's still open when I and when I close it, it stops working. So like but you texted me on the way here you were like should I bring my laptop? I was like I don't know. I think we're just hanging out. You're like cool. I'm going to leave it open at the office so that my agent keep working.
5:48So it's like not mission accomplished.
5:50Like whatever they did was they didn't do it. So I don't know.
5:53I think it's a hard engineering problem to solve the remote execution and especially for like UX and being able to do that well for for people is pretty challenging.
6:02Oh this actually it lets me bring up like something that is super recent.
6:06This happened yesterday. Pi durable. I don't know if you guys saw um so basically uh a lot of coding agents by default like keep everything in memory and all that state. Um and you have to sort of do a rewrite to externalize all that state so that you can transfer easily from local to cloud and cloud to local. Uh and PI has done that and somehow cloud and codecs are like taking a year to do it.
6:30Mhm. I mean, Mario did say he's been working on Pi Durable, has like rewritten it seven times in the last year before launching it.
6:38Yeah, but the basic thing is like everything should be in a log. There's a file and then you append to the file and then you pick up where it left off.
6:45And like technically that's the internal representation for like cloud code and and codecs. They have this big JSONL file for what's so hard about sending things to cloud, but it's not it's like they haven't committed to the API. It's like we can't we actually built our own coding harness that has this property where it's a like a pendon only log because we didn't want to build against an API that the OpenAI team or the cloud code team will not actually like guarantee that this will not change in the future.
7:05So there's no like agents MD standard for that. You're saying it's everybody's like making their own.
7:10There's a guy Justin U shoulder. I featured him at AIE. He's trying to do standard agent. Um, but until one of the labs picks it up, it's something there.
7:18Well, there's like a couple trace standards for like the people like working in Harbor and stuff like this, but it's more for like research and benchmarking a way of like packaging up.
7:26Okay, what is everything the agent did, right?
7:28Yeah. Um, I want to go back to what you mentioned before that, you know, PI releases something and then the big labs are catching up. It seems to in the frontier intelligence that's kind of the other way around. The frontier labs intelligence is the fastest one and then open open source is catching up. in the personal assistant space. You know, Peter Steinberger came up like with the thing that blew up and then Hermes agent it and it takes Meta and Open EI and quite a while to like release a product that's mature around this. There's a lot of stuff that tinkers are okay with
7:56breakages and different memory systems that they set up etc that I think these guys need to bake into something that just works for people. And I got I haven't used dots for a long time. But I wonder if if it works for you guys. If you feel like the memor is there uh that like it supports well because I think the whole promise is it grows with you.
8:12The more you use it, the more memor is there, the more it knows kind of the stuff about you.
8:16I think the labs just really suffer on product. They're it's been a problem for a long time where they'll get something pretty right. Like the codeex desktop app was so right when they dropped that and it has become less and less right every time. Like I most of the updates for dev day in my opinion were regressions. Like adding the secondary extra sidebar on there feels bad. The performance got worse.
8:42It's just like there's too many of these icons we're getting.
8:44Yeah. There's just so much crap over there now. And the performance is getting worse.
8:48Yeah. I It keeps freezing. I think when the internet is not if it's not getting to the internet, the whole like input field freezes is like it's really annoying cuz I like clicking clicking clicking and like oh my internet's up. That's why I need a standalone app for my assistant.
9:03Yeah, this is my message.
9:06Yeah, that's why I use I'm find Did anyone use the original dot by the way? I'm just curious. Do you guys remember one released by the third party? Uh no, no, no. Like like like three years ago. There was a company called the Dot by New Computer. It was like one of the first. You remember? Yeah. Like Jason.
9:23It was like really good actually. Like it's a pretty good assistant. like it didn't have all the like you know it can send email like that didn't exist then but like the form factor was good actually like there was just like a lot of things right there a lot of stuff is getting packed in the chat GBT app and I get why they want it cuz they have a really good distribution through the chatbt app and like etc etc but like but it's also forget about it that's why they have to hide all the cool stuff is because like if we break it for the 8 million users that are like that's right not super AI pill like we have to hide all the things the power trust the power
9:52users to find basically I I found I found the maps feature which I Yeah, the maps.
9:58There is maps in there.
9:59It has a Google maps clone in GBT.
10:02It's like hidden in like the three dots on the second sidebar.
10:05Like you ask your dad.
10:06Yes, that is a real sentence.
10:09I posted about it and they were like, "Yeah, it's been there for a while. What do you mean?" And I'm like, "What are you talking about? It's been a while."
10:14Ironically, I feel like a lot of this is the symptom of using AI to write code and like build products. It's kind of funny. Like it feels like the so much like for the Codex app for example so much stuff just randomly changes all the time which is very like frustrating from a developer standpoint. Like I remember like I was trying to figure out how do I like after they did the chat GBT codeex like merger which also feels like a AI beaver like psychosis beaver dream of like you know like you think they should not have merged. I
10:42I well I can imagine a way to do it correctly potentially, but certainly like a drop down that changes seemingly nothing about the entire app except like the model selector and different modes. Yeah, I was like I I I will often accidentally be in chat GBT and like anyway they just like it was it was like especially the initial version was like quite bad and um you couldn't see your chatbt threads on the side. Anyway, it was a whole mess.
11:06is still for the record they they know it's not done and they're there's a vision they want to work on but I guess what I'm trying to say is that I mean it both in the positives and the negatives like also the codeex app is like really good and like it like I think it would have been very hard to build prei like with like all the browser use obviously not even just the AI that powers like the app is very complicated but also it's just like yeah things randomly change all the time thing randomly break all the time there's like a lot of like you like it's almost like AI written text like you squint and you're like wait What? Like,
11:36yeah. Uh, it feels so directionless.
11:39Like, I I've kind of been thinking a lot about this lately where there's these middle ground products that are trying to be something for everybody and end up being for nobody where like I I really like Grockbot. I think Grockbot's in a great state, but it is much more like a professional personal assistant type thing. It feels dialed in for that. But then there's Muse, which is so good for the end consumer and works super well there. But now you have like Codeex getting watered down and chat GBT getting made more complicated and no one's happy.
12:05No, I mean this is literally like they I remember one of the things I got really frustrated with Codeex about is like they renamed it used to be called fork chat. They renamed it to like either copy chat or some or it just said copy or something. It was like and I was like why? And they're like well like non-developers wouldn't understand the word fork and I was like oh but like this is I'm a developer. Like I think also I worry about that with dots too, right? which is like they're like, "Oh, it's for enterprise and it's for consumer." And I'm like, "Oh, man. Those
12:34are different products." Like those are very different products.
12:36Oh, there's like enterprise dots, right?
12:37That's what I'm saying. It was part of the same announcement.
12:40Yeah. They showed off how a person that's an employee of OpenAI and their dot showing up in the same Slack thread and some people send their dots to kind of check in on other Slacks and and kind of tell them about this. It's a very And that's not a that's not released to us yet. No, no, but but there's a very interesting uh vision of the future where we're all going we're going all going to be superpowered by multiple AI assistants kind of in our name and whether or not if Ben sends me a Slack message um would he expect me to show up
13:09in the Slack message and answer or my bot for example? I don't I don't know the answer to that. But also like would you bring your dot would you put would you add your dot to our shared Slack when it has access to like your say like personal email or your IME messages or I think I think that prompt injection is largely solved.
13:27No, absolutely not. What?
13:29All right. Um he's sufficiently sophisticated.
13:31I can point to a mutual friend of ours Matthew Burman who brought Ply.
13:35Remember that episode? He brought Ply the the guy who jailbel liberator and he gave him five attempts to break into the new Hermes agent that he had although he had some stuff uh and pl wasn't able to well five attempts. Yeah.
13:51But also I I think there's like prompt injection maliciously and then there's just like stuff you're not supposed to see.
13:56If you look at like instinct like like oh yeah muse had an incident like this somebody showed up at somebody's like front door when he wanted to sell something their Facebook marketplace.
14:05the muse just like told that person, hey, you should That's the kind of stuff I'm talking about. Yeah. Like, oh, accidentally tells your boss you're looking for a new job or like accidentally like there's just so many things, right? Ben can't make make this meeting like he's like he's busy interviewing someone. Yeah. Interviewing someone for your job like not doing that but you know it's just like uh scary.
14:25Yeah. So uh okay like any positives?
14:29So I I think Okay. So I think the pattern of like always on persistent thing that messages other things and kicks off stuff.
14:36It is the right idea.
14:37It's the right idea.
14:38Assistant, I I'm I'm going on a crusade and I would love to to bring all of you with me that uh we used to say 2025 is the year of AI agents. Uh and then we kind of switch in 2026 proactive agents.
14:49But I think like the renaming to assistance I think needs to take place specifically because there's a difference between something that kind of you code with AI agent and something that does proactive stuff for you reads your emails and makes a decision on which ones are important to highlight to kind of reduce your load and not bring more of it that schedules stuff for you with like haircut and appointments etc.
15:11I think that that's the new pattern that most people that's not developers I know scary thought outside of our bubble that's what they need. agents for them like is a scary word. Assistance is an easy one. Why? Because they know that all the rich people have a personal assistant schedule stuff for them. And I think that this is what um news is doing well. They're bringing this to I want to take it back to So Switch was asking positive things about that. So I'm going to take it back there for a second.
15:35Dot is or dots. I'm actually not sure which one who you have a dot but the product is dots, right?
15:40It's the first agent or assistant that I have connected my personal email to.
15:49I didn't trust Instinct. I didn't trust any of these.
15:51I tried I I used Poke. I'm very very paranoid about this sort of stuff. And so I had connected my calendar to Poke and that was like 8020 for me like it could it could like set you know you know it could uh change my calendar you know send calendar invites out etc etc but I I didn't go like all all the way to email. There's really three companies in the world for me that I I would trust and like in the order of like Apple, Google, OpenAI and like there's not a fourth for me.
16:20Like it was it was my first time connecting email as well. I was actually talking with another founder and he's like look OpenAI already has all of my tightest most intimate inner thoughts and things like this.
16:29I already have my data retention configured etc. Like yeah Amazon Facebook no Amazon Amazon Facebook are not getting personal email. Yeah.
16:39All right. I would not use this is a good temperature check. I used to me I will say I didn't I also didn't connect Pope until cognition bought them and obviously because I know I know I know I just like I know they take it seriously.
16:51Of course of course of course make it makes sense.
16:53Do you guys not think that Meta is taking it seriously given the residual I saw so I I I I was in the meta the the muse preview and uh they were like hey can you cook up your email and I said no. They were like why don't you trust us?
17:10I think do you know where you work?
17:12No, no, no. They I do I do trust them. It's actually more just like based on all the data we see about you predicted.
17:18I will never type my my email password into anything else other than Gmail and they will ask you for my email password and I'm like absolutely not.
17:26They didn't have an OOTH connector at the time.
17:29Yeah, I think they need it too bad if that makes any sense. Like I think OpenAI has a lot of data. I think that they are less likely to do crazy with my data. That's like just what I feel. I feel like meta if it was if it meant like some little like a little bit more likely that they were going to win whatever they would just do it and and and pay the repercussions later.
17:50I don't think that last point. Sorry.
17:53Can I ask a question? Why do we need the assistance instead of making uh we're all developers. Why can't we make codeex or whatever agent a better experience for assistance? like like it's the same tools, it's the calls, it's well I actually I tweeted about this and kind of went a little bit viral. Like I the main agent I'm using right now for like personal assistance stuff is literally just Pi with like I gave it its own Gmail. It has credit card.
18:18It has like reminders cron and I have like a very literally I was like codeex clone chatgbt qi like just works like I I I have better than dots. So, I didn't I haven't really dialed in any of like the proactivity stuff. And I think I actually imagine that that part is like the really hard part of this sort of stuff. I'm not I don't feel a deep sense of longing for it right now. Maybe I don't know what I don't have, right? And it's pretty useful. I really like it.
18:47Imagination unlocking like literally executives at the companies, they get so busy that they cannot manage their schedule. They have a person doing this for them.
18:56Like those people cannot go back.
18:59like this is a standard thing for humanity like when you go to that next level you will still be more busy but you'll be able to do more stuff when the proactive stuff will just come in.
19:08I don't know but I bet that we see people start I think someone is going to do a do open source kind of version of something like that. I'm shocked it hasn't really happened yet.
19:19Like OpenCL you guys mean or no like OpenC and Hermes are not it like they are they are last generation because the a big part of why something like Rockbot is so compelling is because it has such a good mobile app web interface all that stuff feels really good to work with.
19:36You don't have to configure memory.
19:37You agree then you then you have your answer is because you need a polished product.
19:41Well I actually engineers can build polished products.
19:43No, because I actually think that what you're talking about is very correct cuz for my stuff personally, I do a lot of what Ben does where I just have all my stuff hooked into codeex and I can go directly in, use Gmail in there, do whatever I need. the assistant, quote unquote, whatever it is, it really just feels like taking a normal coding agent, putting cron jobs in it, putting maybe some like shared box on it so that it I I made my own like S3 back to Google Drive essentially and like it's amazing.
20:12Like it's very easy.
20:13How did they release remote codeex and dot at the same time? It's the same underlying tool. I I will I have a very simple answer for you. I don't want to another problem. So I have two problems. I'm bringing on an assistant to help me solve a problem and now I have three problems. I have to maintain the the infrastructure. I wanted to reduce my load not to increase my load. Like another one like another box that I have to manage it goes down.
20:34If I run it on my Mac Mini which we bought for like Hermes's open qu many of us that Mac Mini disconnects and then it's not live. This one simple. Hey are you there? Do this thing for me by the future of harnesses is probably something that's shaped like that.
20:47Anyway, I think that a lot of why ChatGBT did the merger thing for ChatGBT and Codex is longterm it's all going to be CEX cuz they want to RL on one harness, one set of tools, one way of doing things.
20:59DOS, it'll all be the same. It'll all be codecs under the hood. Yeah, I think they're not there yet, but eventually it's where it'll end.
21:06One more thing, big thing in Dev Day, which is also hot topic, uh is decision models, let's call it OB clone. We um had a exclusive podcast with them uh where they would basically say it's Luna with like a structured output wrapper right now which is interesting but like obviously they're going to improve on that. Um what are you you guys' stakes?
21:25You know that's another very very hot topic.
21:28It is multimodal. Jeff came out in his text only and open the biggest difference is coming.
21:33I I I know it's coming just okay live live a little bit in the future and just be like okay these are the same like so structured output for with AI and pipelines for a very long time. Uh the coolest thing about Jev is like most of the model providers took away the log probs and I actually like knowing hey what are the percentages of all these outputs not just here's the answer but actually being able to get a little bit more signal about the probabilities and like what were the actual like output probabilities for that token before it actually decided you want per token or for decision cuz
22:02Jeff gives you per decision.
22:03Yeah, exactly. That's what I'm saying.
22:04It's like you can use GPT40 mini to do almost all of the classifications that Joe can do like for a long time. I mean obviously it's smarter and better and faster and cheaper and whatever it is, but like it doesn't it didn't make anything like technically new possible. It's just the price and the speed make it applicable for new things.
22:18I I would say like you could even have done decision probabilities obviously with like structured outputs in you know any other model before. They just like were not well calibrated like I I never trust a model to give me like a one to 100.
22:30That's what I'm saying. That's what I'm saying. You could say 0 to one and like you it it essentially bucketed it into zero 0.5 or one, you know, no matter how much you prompted.
22:39And so yeah, like it you it feels like true probabilities. I like we use structured outputs like a ton like probably like literally ra like we've trained our own models to do this like etc. Um I think that I I'm glad that OpenAI is doing it. I feel like you can actually switch more easily once you have a lot of labelled data because you can just sort of say like hey like how much does this match how what I was doing historically like if if you so once you like label a bunch of stuff if something comes out that's like 10x
23:08cheaper you can literally just say okay use all my existing labels as as eval essentially if it agrees like you know more than you know x% of the time and then you only have to review the parts that disagree essentially so very very easy to switch um really really low switching costs here and then you can have actually a model like hill climb I don't know. We do like the poor man's Jeepa where I'm just like cool here's a case that doesn't work go at it as an eval and then keep tweaking the prompt until these sort of structured things. Yeah, that's awesome.
23:34There's tons of like product UX experiences that Jevshaped things are able to unlock as well and I I think it's yes, it'll improve from Luna uh in this current decisions API, but it's like the right decision to release decisions API.
23:47I actually I would ask like the I think that the product stuff you can do with this is super cool. Like what are your favorite ones that you've seen so far?
23:53Cuz I have a couple. I don't know how OpenAI does it uh for them, but approve for me is a is a thing. It's going to be a lot easier with platification models.
24:01Um different uh UX changes. I was actually kind of surprised DOTS is like pure just like texted print output and you can get dynamic things generated, but I was surprised there's not any clever creativity around UX or changing what the shape of an experience looks like with an agent. Um I think there's a lot of possibility.
24:18What's an example? Is there something that you have in mind? Like what's the closest neighbor that we would know? Let me think about it.
24:24I think to me the coolest thing with the this shape is that you can stick AI in a lot of weird places that it wasn't before. Like a great example is like on a website you have the command K menu to search for things that is often broken and bad and weird.
24:38But if you you can just throw that in Jev and it's fast and instant to just predict out of the list where should we go? you kind of had a decision before to make if you're doing just a regular look up in the table in JavaScript and you're loading all the states or you're using AI and then you're waiting and streaming tokens etc. Now you don't kind of need that decision. It's kind of the best of both worlds in that command.
24:56Yeah, it's like borderline instant. So you can just like rank these 20. Okay, here are the 20 pages you could go to. The user typed these three letters, which one do they want to go to? That ranking will make a lot of weird things feel better. I think it's dope. I am annoyed at the general discourse around Jeb and all of these models because most of the p like the super viral examples are mostly stupid. Yes.
25:17Like aggressively stupid.
25:18I was at a dinner or something. Yeah. Now now that I have Jeb, I can guarantee that the output is in the format I want. I'm like, oh my friend, you have outed yourself.
25:26They're just they're telling themselves on themselves how dumb they are.
25:29Can we talk about the the clones? The the cleft that came off the cler one is crazy. The numbers are insane. Tell us.
25:36What do you mean numbers are insane? the like the num there's some benchmark and the Cloudflare one is like the really small one is better like same performance as Jev like way faster and then even their really big one is better performance than Jev and like noticeably faster.
25:51Something strategic that Yogo did was like don't benchmark us.
25:54You know like he repeated I I super cut your chat with Yogo. He repeated this maybe like seven times during the podcast like I don't believe in benchmark he really believes it. What are you going to do? is sort of true though like I think that is really like in this exact like structured classification it is really really really easy to overfit like a specific domain which is like could be fine right the thing is like if you have a super repeatable task and you're just doing it's like it could just be fine to overfit it like like you can have an overfit classifier that's like fine essentially um if it's super well
26:23constrained but I think that the whole selling point of Jev at least is like super diverse data really good at like huge range of tasks etc so it I think probably very hard to actually like faithfully benchmark.
26:36So it's also hard to know which one to pick. You're saying right clouds are released something but your worked on data for two years and their data is immaculate. How do we tell?
26:44Yeah. Which also applies to too.
26:48Yeah. So uh the rumors are that type safe closed a $10 billion round this week which is insane for some friends of mine. that I think the the core question of our company is if OpenAI and Entrophic and like and and and everyone else has a has something of Jeff uh would you use would you go out of your way to get a Jeff type safe API key or would you just use your open I think it's going to run on device like I don't want network Jeff anymore the models getting cheaper yeah it's already shipped on right so
27:18there's a web GPU already Jeff is a thing you can run it maybe not to the level of Jeff quality but again we don't know how we tell I think class I have this very deeply held belief that classification is actually really really really really hard and it looks deceptively simple because it's like you're returning like a single you know either like zero or one you know in some ways but I think that that like literally binary classification is one of the hardest problems in the world like uh you know there's this really funny paper from open AAI a long time ago where they were like you know
27:47hallucination is just a binary classification problem and all these people were like great like open AI solves hallucination it's like no it's either like true or not true and that's like really really hard. So, I think they'll definitely be like on device versions, but I think they'll also be like big. I think that's like the whole thing that they did really well is be like essentially like big, you know,ish or like very smart generalized classifiers. I think that like you said like would I get a type safe API key?
28:08The answer is like I actually use it through the versel AI gateway. So, I didn't get one. So, I don't really care.
28:15So, like I don't you know I don't know if that's good or bad for them but use Jev not CL not the because they have their brand name right now. You have open AI Jeff and like it it's fine. It's good enough.
28:27There is a thing about they created this technology so they probably understand about their primitives are great. I don't know about open I haven't gotten access yet on them on the wait list. So I'm not sure they have the right primitives yet and I'm really hoping they will align to the same primitive so I be able to switch easily like kind of open router type thing. But they they are the experts in not only Jev the model but also how to think about Jev. Do the yoga released this thing about like how to build a hardness. At some point he said, "Ah, we thought about building a harness with Jev. We decided not to do it. Here's
28:55open source how to like add this to your harness."
28:58Very very smart. Yeah. The I really like the like main guy at Jev. He's been so good about like no this is a dumb way to use it. Sorry.
29:10He's so passionate also about this like one specific thing that brought to the world and I find no he has other things coming. I'm sure that's the thing. I think like I think it'd be very silly to like look at a company that just like was the first one in a very long time to change like the way that people are using models and be like that's the like that's it for them like you know do you feel the the changes of the same magnitude as JP was?
29:36Do you feel the changes of the same magnitude as reasoning models was or one?
29:40Not quite but but like it's it's it's get it's getting there like you know it's like it's like 30% of that or something like for us. Yeah, I I think that it is especially as it gets faster. It's just Yeah, it's going to be I think there's a lot of I think it has not fully permeated yet. Um but we'll see it.
29:57My my hint for people listening who because we did the podcast is we actually left hints as to what the future models would be. Uh if you're smart enough, you can figure out the or if you have Jeb, you can run every sentence of that podcast through Jeb and say, "Hey, is there a hint there?" And you'll get an answer within a second, less milliseconds, right? You're going to get an answer. Yeah, I'll say I I do think that it Jeb is still actually ridiculously underexplored. Like we're have a lot of stuff for cooking which we'll share later.
30:25So you just like you're throwing on everything.
30:26You're throwing it yourself on.
30:28So don't you like you train your own models?
30:31Yeah. Yeah. We actually still use it for that for like for what we call signals. We still use our own models. This will be more obvious like in uh we have some very very fun stuff coming in like the next uh over the next few weeks that that uses Jets. So we'll share.
30:41Yeah. We have a we have a thing that we were just play testing this week of like a aentic memory pipeline and it's like a mix of like it's like back and forth between Jev for structured output and then like a soul 5 5.6 or a soul 6 uh model for like more structured output and then Jeb for like sorting and reranking but it's like look at all my agent traces and like tell me what to add to my cloud MD. It's like as a building block in these systems and basically like we tried it with a bunch of smart models and putting Jev in there like reduced the cost by like 10 or 20x.
31:10So did it improve accuracy?
31:12This is what I say cost is very very easy to to reduce on classification. What I will say is that um in all of our evals it is like pretty well aligned with like opus 55.
31:24Yeah. Yeah. With no reasoning.
31:26That's the crazy thing. Um and I think if you like think about it deeply, there's like no obvious reason why classification needs as much reasoning like all the time as it does in order to get good answers. And like also if you use like any of the frontier models to do classification without a ton of reasoning it they're they're abysmal and really bad at classification. Yeah.
31:44Um and so you need a few shots and so that's the thing that even like speedwise is not even so it's a smart move for open to his decision. I I asked around I asked why and I I really wanted to joke with Sam Alman to say hey did you because yoga mentioned on your podcast right that like he came with this idea to Sam and Sam said oh that's a great idea you should work on it. So I really wanted to joke with Sam like did you wait for your buddy to like release his thing before you can come into the category. Uh what I the answer I got between the lines not to quote anybody specific is that
32:13there's only like Jeff types have proved the category exists and there's market fit for it and then open well well our developers would like our offering in that area as well. So that's why they released.
32:24And by release I mean weight listed.
32:26It's not out yet. It's not public.
32:28Okay. We're going to move on to the main course. We are we have sumac beef tenderloin tataki style pansered um halibet and creamy lemon risoto sweet.
32:40Okay, models, right? We got the We got the Astra, we got the Soul, we got the Opus, we got the Sonnet. What are people using? What are your picks? Go.
32:50Mainly Astra ultra like ultra on the fastest mode possible.
32:55How much can you burn? Like I was able to burn down an entire weekly reset in about 20 minutes with that. It's really easy.
33:03But you know, I I think it's worth it. I think it's worth it. Um my party line has been I think Opus like quad models have a soul. Like obviously GPT models do not have a soul. Like I think you can like GPC by design.
33:16Almost by design. Like fantastic engineer, zero soul. Uh but like Opus 55, Fable, they have souls. But I use Opus 55 a lot for anything that's like writing related even adjacent to writing cannot use any of the physics models which is really interesting. But Aster improved it a bit but it's still Sam Sam talked about this in very openly. He said we know our models are not good at writing. We're like working to catch up on this.
33:42I mean up until it was 5'5 though I felt the exact opposite. I was like I will never use a clock.
33:46Well after 45 you mean they went to 48 48 for example was a mess.
33:50You have to go all the way back to 46 to get something. 46 was the last like great oppus and 55 is 46 smarter.
33:57Yeah. Like I run AI summaries every day and the default model was 55 until Opus 55.
34:01Oh, we got like user like our customers got mad like they're like I can't read and they were sending screenshots of like and and it's like yeah you're right like we when we switched to 46 to 48 it was like do you work on imprompting at all?
34:13Um we like default well we do we do graduated. Another interesting thing about the Quad launch was that they basically came out and shhat on all previous versions of Cloud.
34:22Yeah, that always happens because the researcher like apologizing for Opus 5.
34:25They they were making memes of like it just got slowly worse and then like 5 is the thing now we fix it.
34:29And they never admitted it until now.
34:33Very different for Anthropic. I don't think they've ever done anything like that before.
34:38Yeah. It's But like did so uh this is why like I actually do want to convene like it's obvious why though. No, no. I didn't I didn't I didn't I didn't actually feel it that much.
34:48You didn't feel that?
34:50Yeah. So, so like that's my problem.
34:52They just weren't using high chase.
34:54Like they don't use opus. No one at Anthroic is using Opus.
34:57That's actually very true. Like they use Fable all the time. It's either they're mainly just using mythos problem.
35:04I don't think they're using release models. They're using unreleased models. They're play testing all the time. They have unlimited and super fast as well.
35:10Which is a completely different experience than what users get. So please just don't don't see it yet.
35:15Can you mention your your uh Tying thing that you they told me last night?
35:18Oh yeah. Okay. There's one thing I want to say before that which is that I think on the topic of what anthropic uses internally. I really thought that you like I think you're right. They do just use the fable or mythos or whatever class. And I thought that opus sauna all those would just be forgotten children forever especially after how bad Opus 5 was. Was shocked that they like brought it back with this one and and such beautifully as well.
35:40It's a great model. model. It's a really good model and and so now I feel like there is an offering from Entropic which is almost 5.5 and an offering from Open EI which is a soul 6.1 right um that make it so it almost you don't even need to reach for Astro Fable you like for the daily work stuff definitely for for most people I think those models are enough and reaching for those that like finish your quarter real quick really needs the big big big like
36:09refactor everything review like huge things or like yeah difficult tasks. So it's very interesting to switch.
36:14Has anyone tried Sonic by Fire?
36:18It is the best model that you shouldn't touch. Like it's great for Opus. Like Opus can use it as sub agents and they're like cheap and pretty capable but right.
36:26Yes. S is decent writing. OP is still way better.
36:31Um but Silent has like it's the little brother, right? It has a little bit of that soul. Has a little bit of that like oh okay. you can understand things between between the lines that you kind of would expect a is like analytical and would not but also using that when you have a subscription on topic it's kind of like why oppus is almost limitless that's like a new thing also with the crazy stuff it's really hard to finish OPU 5.5 now oh no I used my first reset ever on cloud code the this week for oppus 5.5
37:01before 5.5 it's like finally good enough that like I am in that like I haven't done this in a long time because I believe that like you know if you're building production software if you run the models non-stop overnight you're going to create more code than you can like reliably consume and ingest right now.
37:14And now like well I had like 10 different prototypes that I'm working on and now I'm like okay cool I actually like am doing the thing. I was like okay 10 p.m. you start all the night agents and then like 9:00 a.m. you you log on and do the thing.
37:25No, just honestly like a lot of it on like medium or high.
37:28But I what I'll do is I'll do like a a poor man's Ralph loop where I'll just like cue up a bunch of prompts of like do this. Okay, now make a better compact. Now do this. Now read the plan.
37:37Now keep making it better. And then compact. And so those will like queue up and run overnight rather than like having a So I kind of like it's like a Ralph loop, but it's bounded. You're only queuing five to 10 messages.
37:47You're playing you're playing chess and you premove every like I'm like like hey the design's awful. Like redo it. Uh all the architecture is way too complicated. I literally not stocks but one of the previous things that got solved. That's how the guy did it. He's like you can do this.
38:02You can do this to favor. You can do this. This is the best use of I've ever heard from bun like a lot of my version of this by the way in case it helps people is I have a detach mode um skill basically where um you it's kind of like goal but instead of like running the thing in in the foreground you run it in the background you like spin up the CLI version with the agents and you run it in background just tell it to go run or whatever because then I can run 40 of them parallel.
38:27Yeah. And then you can you can monitor the the the the the logs uh and then also uh pause everything improve the logs and then rerun. So that is full overnight. So like I asked Codex people about this whether codex can spin up different threads of codex that you can actually see and talk right and they can talk between each other kind of like bots that's nice but also they have sub agents as a as a primitive and so the codex people multiple ones said the sub agents are significantly better and faster because of the cache optimizations and how the model treats
38:57sub aents versus different sessions. So I guess switch like to to your thing because this is the fork sub aents in the new codeex harness, right? Yeah. How many sub aents do that you have?
39:06When I had 48 in my toml file, uh, Astra burned through my limits in like 2 hours. So I decided to clean that back to default.
39:14I keep it at like 15. But yeah, you can't just in the config be like, hey, give me more sub agents. And you can just tell 6.1 to do it. Go do this.
39:21If you if you have I never What's What other config things should I be doing? Cuz I actually never really touch that config.
39:26How would I say this gently? If you have unlimited access, 1 million tokens I think is a good idea.
39:32No, I just set my autoco compact to 400k yesterday.
39:36No, but the default is like 275. No, for um forex.
39:39I wish you could disable compaction and codeex. Yeah, like I wish that it would just truncate tools or trunk like literally anything besides the summarizing.
39:47I wish they would use Jev for the tool calling and just like give the model back when it needs everything.
39:52I don't I don't know. I don't know about the It could work. Oh yeah.
39:56But compaction always compaction not always. Okay.
39:58So you just went you want different kinds of compaction as an option.
40:01No. No. I just don't want a summary compaction. Like I I just like most like a lot of the time it's just like there are a bunch of like images or a bunch of like whatever that's in the previous messages.
40:14You just want to strip out all the biggest.
40:16Yeah. Just strip out the like that cloud code used to do that internally. I think they I don't know if they're still doing it. Um cuz their got compaction is really weird actually. It's like uh the weird thing about quad codes compaction is I feel like a bonus forget who it was.
40:28It thinks it's a new person like it'll be like hey like I saw what you were working on before.
40:32I've had this happen multiple times.
40:34Momento moment like are you using like the native built-in compaction?
40:39Do you notice this or not seen that?
40:41It's been is I haven't used this since the 4. Like 4.0 days. So it's been a while.
40:45I've loved opening eyes compaction. Uh been using it the last couple weeks and it's slow though. It's so slow. I mean, if you're running 20 things, it does it matter with two months?
40:56Okay. Yeah. Do Do you actually watch your trajectories? Cuz I I I do and I feel like I'm doing it wrong. You know what I mean?
41:02I don't I don't stare at them, but if it says something, I'm usually going to scroll up and be like, "Okay, what kind of racing actually like often times like cuz I especially when you're doing like heavy tasks, you'll like catch it going off so bad.
41:13You see the thinking, you're like, "Nope, here's what I actually And I always have a skill for having a side thread that watches the main thread's reasoning." Yeah.
41:20Uh and then just goes like that's the project or chief of staff pattern, right? That's cloud. I didn't get projects yet, but Corser has project where like you have the one thing that manages everything and basically it kicks off sub agents for all the time.
41:33Uh that's great in Corser. I can I used um that's great in in Grabbot. I use the cloud like that like hey don't do work in the main thread at all. I don't want you to your context your context should be my conversation with you and not execution. execution should happen elsewhere and you get things about that.
41:49Oh, I do it the opposite. I'm doing things and then I have conversation about the thing on the side.
41:54The sub thing because that that is ephemeral.
41:57If you're trying to do like a super heavy longunning loop, that is definitely the best way to do it. The only way I got asked her to burn down some insane problems I couldn't get any of those models to do and the only way I got it to work is put it in ultra, give it tons of sub agents and tell the main thread your only job is to send off sub agents. Put a timer on them. If they stop or get lazy, just keep push them.
42:16Just it is the pushing machine.
42:18It's kind of like um this pattern in bigger companies when you know developers rise to the level of their incompetence as a manager. But at some point the companies really want the developers to keep developing while managing other people. I this is the opposite for me. I I don't want the main guy to code to run tools. I just wanted to tell other folks in the best way possible and then review and bring it back. The second it starts to that is dots, right? That is the idea of dots.
42:42Um I mean you can slash command that that's all. I don't think that's all bots are. I think that's my personal assistant.
42:49It does shell stuff out like it can fire off codeex threads which I found it's very nondeterministic how it does that like more aggressively than I would like it to be.
42:58It doesn't I don't trust it.
43:00That's what I'm saying. There's just something really broken with the whole model of like where are the where execution's happening.
43:05It's so confusing and like codecs and dots and like Yeah, it's like really confusing. Okay. So, keeping on the model discussion, uh we talked about the the two big ones. Um other models that you guys have been trying thinking about.
43:17Where's Haiku? Where's the one year anniversary is coming up like next week?
43:26Haiku 45 is almost one year old. It's like in birthday.
43:30Luna is not having the small fast kind of nonjuds. Luna is great for speed for like small stuff.
43:40Terra, where's Terara? What happened?
43:42I love that they had um the lineup they were like Terra Luna, Terara Soul. It's this beautiful thing like opening eye pixels and then they were like, "Okay, now it's six Astra. Wait a minute. 6.1 Soul and you're looking for Luna." Like and then you're like, "What happened to Terra?"
43:58They're like, "That's out now." And you're like, and it's one week after six one week later. I thought the opening I finally got their together with the naming saw terra and Luna was beautiful and easy to understand. Screw it up.
44:12So then like saying like oh this is like you know ex you know like six terra wouldn't have done anything to this one cuz you're like so I think they did six soul because the brand was good.
44:23Exactly. That that's all it was. Yeah should be soul. Soul should be Terra and and Luna should stay the same.
44:30They should have the model with Terra wasn't but no open AI needed a fable class model.
44:34Yeah, they needed something better than the normal. They couldn't call it soul.
44:38Um Sam refers to the next models as whatever next hugest celestial body there is. That's how he mentioned kind of the next model.
44:53Uh Nebula. I I I I may be the only person I know that uses Muse Spark.
44:57I use Mark. I love M Spark.
44:59Oh my god, you're such a muse show.
45:02Um it's use code. Yeah.
45:04First of all, it's 1 million context multimodel, right? Other models don't give me that.
45:09What do you use it for? What do you use?
45:10Speed and like where like coding or newsletters?
45:13No, no, definitely not coding.
45:15Definitely not coding. The cheapest until like Jeff came out and started to rethink a lot of stuff. I would just like chuck huge chunks of text out especially on the contributor room. Yeah, but but but multimodal as well. So you can you can send it videos and say, "Hey, when did Ben say this thing?" For example, and not transcription wise, it can hear. So it can it can hear like the you know the claps for example.
45:36Gemini also does that. They've done that.
45:37Yes. Gemini Gemini is kind of behind on that.
45:41Wait. Okay. So you think Muse Spark is better than anything Gemini has?
45:44No. M Spark is just more more useful and significantly cheaper. M Spark is cheaper than than Gemini just including the the train on that train on me daddy mode.
45:53Yes. the contributor mode.
45:56Oh, that is so cheap.
45:57Cheating on me, daddy.
45:58Yeah, and they will. Don't worry.
46:00Uh, also Muse Spark is what's running Muse as well. And it's it's good enough for most assistant tasks, I think.
46:08Um, Grock, yeah, it's so bad.
46:10Grog 4.7 was a disappointment. Oh my god. I was like, all right, the cursor folks will give Elon the data he needs to train the models. Finally, walk me through. Yeah, you guys are I don't see, you know, someone releases, you download it, and what do you what do you try and then like what is your what's your bench?
46:27Well, there's like there there's two sides of where I think they screwed up on this one. Is one the pricing. They were so hyped to say that it's like a better model but the same price. And that's just flat out true because if you increase the if you go over 200k in the context window, you double in price. If you turn on fast mode, you double in price. And then if you do both at the same time, you nearly triple in price.
46:46It's like one one and a halfx. So it really ends up being a very expensive model. And then the thing that they're really struggling on is their token efficiency is awful. It will do so many steps, so much reasoning. It's awful. And it just kind of it doesn't I don't know what purpose it serves because it's not it's not the fast model that's now Muse.
47:08It's not the cheap model that's Muse or Luna or whatever. And it's not the capable model that's Opus. Like Opus is not that efficient, but it's so capable it's fine. and you're paying Opus prices on Grock. The only thing that it does feel good in is Grockbot because they did RL the model on Grockbot and that's kind of what I feel like happened is they just sacrificed it to Grockbot.
47:28Grockbot and the Grock tool itself. So Grock is not only the model, Grock is also the product Grock.com and inside Grock inside XAI that has access to real-time data from Twitter which no other model has and that gives it like superpowers in research.
47:43Has anyone used Grock in X.com? Yes. All the day.
47:47Every day. It's the best research model.
47:49I I I try using it all the time and I it it always says it doesn't know my username.
47:54They fix it. There was a bug for like a month and a half. I hated this. It was so ridiculous.
48:00I would be like, "Hey, pull up the tweet where I said this. It's like, hi, I don't know who you are."
48:05And like, you know what's funny?
48:08The the built-in integration within X didn't know who you are. Grog.com knew who you are because you're logged into that person. It was literally a bug.
48:16The the most useful thing with it though to say about that. Like maybe they use Brock to build.
48:23Yeah, they need raindrop. They need rain.
48:25Have you tried the um I the gro build CLI is actually a great CLI, but the most useful thing for it is you can have other agents call it as a sub agent because it has native X API capability for unlimited for free.
48:38Instead of an MCP to Twitter, you're just like if you need Twitter stuff, go. It's literally just have it call that to go get some posts or whatever you need. It'll just do it.
48:47That's that is good.
48:48I I don't know if it's unlimited. Um, you know, yeah, I that's fair. I don't because they released this.
48:54I've used it before, but because they released this, they released the O connector to X and you know, Pi and Hermes and everyone basically can use the same thing without even their their tool. You can just like add it and do O and then it just works.
49:06I don't Is the tool exposed when you do that? Like I it I don't think it adds the MCP to like PI or whatever when you do that.
49:11Uh you you can search Twitter without paying for X AI X API tokens.
49:16Oh yeah, you can well in build CLI you can but I don't know if like that tool I don't think just magically gets exposed when you sign in with the search Twitter tool.
49:26I mean it's let's check this after this.
49:28Yeah. I'm actually I'm so I definitely I know that like my Hermes agent used to do the bird CLI that Peter time released with OpenClaw and then uh because it was not it was frowned upon to use and they were starting banning folks that used like the cookie via browser thing. Uh I switched to that via the O and it works.
49:45Oh, I could totally see the O working. It's just I don't know the tool. You would have to give it the tool somehow.
49:50Okay. Other model stuff. Uh DeepS um I I don't know if people are are deep into that. like basically open models versus closed like how do you how are you guys feeling about the general landscape?
50:00So I have four DGX Sparks. I when GLM 53 Flash came out as soon as I saw the size of the model I was like oh okay so these are going to spike in value. So I instantly just texted my dad and had him go to MicroEnter back in Ohio and get the Sparks for me before they went up in price. So I actually got a pretty good deal on them. I've made money off of compute which is stupid. It should not be appreciating in value. But regardless, the point is I've been using V4.1 Flash on the Sparks, and it's
50:30really cool. It actually can get stuff done. It's very fun and kind of useful for any hill climbing problem where you can grind it down with verification or whatever. It is not useful for like true day-to-day work. Like, there is such a chasm between the frontier from OpenAI and and literally everything else that I I don't think it's all that viable at the moment. There was a moment where open source models were like six month months behind and then when Opus came out the 55 the latest one in 6.1. So
50:59like the frontier like shot straight up again and open source is not not there maybe a year.
51:04Yeah. No Opus felt super sold out. I run this benchmark called slop code bench which is out of like a lab in University of Wisconsin and I haven't looked at the new data but the the biggest thing I saw that was like it's less like okay this is a like perfect benchmark of everything cuz I only ran a subset but it was like for the you know 10 problems we did opus 5.5 wrote three times as much code as it was at the time it was like sonnet 5 and opus 4.8 They both had about 10,000 lines. This thing wrote like 30,000 lines and literally half of
51:32it was tests. Like oh more than like 51% of the code was tests whereas the other models wrote like 10 or 15% code is tests. Really excited to publish the Opus 5.5 stuff. I don't have all the data together yet, but like yeah, it felt like clotted just like flattened out. It was just like like the Fable was an interesting thing, but none of their like daily use models were. And suddenly it's like okay, the new conversation is the new is the new hot to me. model capability kind of like the generations quote unquote is there's these like lines of capability that
52:01everything almost like comes and approaches and then like doesn't quite cross like they're almost as they get right up next to it and they can't get past until you break through that ceiling and then you get to that next plateau of capability and then can catch up. But I feel like at the end of the summer before we got Astra in 5.1, Fable 5.1, everyone was kind of getting up towards that ceiling of the last generation of sub agent orchestration, longunning tasks, cap very capable of solving a lot of problems, but didn't
52:29quite have that power and discernment that are now in the new models. And there's just a certain something that is a step function above that now 6.1, Soul, Astra, and Opus 55 and Fable 5.1 to me are the only ones that are in this like new category. Now, what can you name the category?
52:47Uh, for me, I think the big ones on this new generation is like really good swarming behaviors, the ability, it's it's hard to put a strict word on anything other than trust. Like there's a certain level of trust that these new models have. Yeah. Also, there's always some like meme thing that that has to go viral when the new generation comes in, like the blender slop or the video.
53:08Oh, yeah. It's actually a great point.
53:10Yeah. That's just this marketing, right?
53:12No, I agree with you. Like, if you think about, you know, people are connecting their email like to models now, like giving the ability to especially send email. You could have done that a year ago.
53:22Or two. I mean, you could have it probably would have done most. added a tool that would like took in text and sent an email, but like no one in their right mind would have done that on their actual email. But now you would and not not cuz all the emails would I said no one in their right mind.
53:38I was very specific now.
53:39How else would you get a helpful personal assistant if it doesn't know you?
53:42No, I'm saying a year ago.
53:46Yeah. I I will say I will just keep my same qualifications. Like I got so many emails. In fact, we had our Azure rep, like our company Azure rep.
53:56Oh, we got like started getting like o messages from their open call like, "Hi, I'm this person's open call like on our like person like our dedicated support chat, we're like we get these from people who I knew personally."
54:07Yeah. Yeah. Yeah. I know. Yeah. So like when I saw again I think on the record when I saw that ply the liberator cannot liberate uh Matthew Burman's open claw I and then I saw a bunch of benchmarks they're doing about prompt injection specifically and the latest models Astra I think it started with Astra and Opus 5.5 they have benchmark jailbreak prompts and like prompt injection benchmarks uh these models do not trust anything that comes outside of kind of
54:36the system prompt and you cannot inject the into that level. So they have a very high hierarchy of like uh their constitution which is operained on right and then the system prompt which the creator developer gives and then the content and they always wrap this with like very this is from here to here this is like an email this is whatever it's really hard to jailbre with a prompt injection.
54:56Yeah I'm again I'm I'm prompt injection is definitely not the only thing I'm worried about like I think there's a lot of other ways to cause your life a lot of harm that is not other people trying to do it. Yeah, the models actually trying to be helpful is much more dangerous.
55:09And I I I also think that there's still I I guarantee that there are ways of doing pro I don't even know if prompt injection is the right word, but like as soon as you have your model responding to emails and you want it to give information some of the time, it is going to mclassify whether that is because someone's intentionally trying to get to mclassify or not like what it should or should not share. And I think I I I I believe that to be to be true. I I will ask for evidence from all of you like have you seen this on like you
55:38follow a lot of people they try a bunch of stuff instinct is like a very you know someone asked their friends instinct for the uh Netflix 2FA code from their email I just gave it instinct I think uses open source models I don't know don't quote me on this but I don't think they can scale right now maybe like maybe later on but in the beginning they were different it's very much like I have mine set up such that it will like give me a permission prompt before it's yours uh Grockot I use Grockot for all this Yeah, with your emails connected.
56:05They are connected and I do use them there, but it is not allowed to send anything until I can see the actual draft and then hit send myself in this little expander thing that you need to scroll to the right to see the the worst UI ever in reviewing what it's about to send.
56:18No, the the one in Grockpot's great. It just shows as a message. Yeah, it's just it's like a normal text. It's great. But the what I have noticed though is like it's rare, but I'd say like one in five times there is details and things in there that I do not want to have.
56:32weird like it's a support email and it'll be like hi I'm this person's grob let it fire and forget so okay so but but let's draw a line here right so Ben was talking about a year ago people will not connect their emails there is a line between readonest emails I was saying sending oh okay I oh I didn't realize this I don't send willy-nilly I review what it sends yeah I'm talking about I was talking about sending uh reading is like a different thing yeah I mean if you're especially a year ago are a lot more
57:01closed off, less like, you know, I think now people are just connecting everything. I mean, it has access to like their work stuff, your personal stuff, your, you know, you can post on Twitter for you, has your email, you know, it's kind of here's a problem that I'm noticing after using a lot of these tools. I got to wonder if people will just choose one and just stick with it because there is a attention spent cost at switching between those tools. I'm now at a point where I don't remember which conversation I had with which assistance.
57:25I know. I agree. and uh and importing their memories one to another and that's not good enough because their memories kind of move in different ways, right?
57:33If you think about them as clones, the clones start living different lives at some point and this one knows about this thing, this this one did this thing and you're like, "All right, is it use case based? Do I go to Grock only for work stuff and then use for personal management stuff?" How do you guys deal with like that? I I would love to know.
57:48I I don't have a good solution. I have a solution. It's not good. I've been trying to rip things out like the the idea of like an MCP gateway. Well, one of my friends is building a company around this. It's called Executor where it's basically just a central it's a big MCP gateway that has all my connections, my Gmail, my Google Drive, notion, etc.
58:06And that lives on one of my Linux boxes at home and I have a Cloudflare DNS proxy so that I can get to it on any device. And now whenever I set up a personal assistant, I do one MCP connection. I log in and then that now it has all of my tools and all those tools include like the skills to use them, the prompts to use them, the config for each of them. So my like core connections layer, which is the really obnoxious part to set up is centralized in one place and all my devices share it. So if I change something there, it's changed everywhere and the actual like interface for
58:36working for things ends up being pretty lightweight. Like I integrate with RagBot. You said like Ragbot shows you what it's about to send over email. Will that generalize and instead of their Gmail connector, it will go to your connector and still show you the preview.
58:47It works pretty well. There's a couple things that I need to work out on it. I do there are some connectors per thing that I do still use natively like the finance connector. I really like the finance connector in Grockbot. It's useful, but like that I'm not going to do over a direct MCP.
59:02Just to note that he said a year ago people will not connect their emails even for now. You're talking about taking your bank data.
59:08Well, read only and write read only is the line here. Like I'm still pretty much at read only write with explicit permission is where I am.
59:16The only one I want I want right for is I wanted to be able to create I have people who are very sophisticated about every single vendor gets a different like virtual credit card. I'm way too lazy to do that. But if I could tell my agent like hey go make a virtual card on Mercury and go for us.
59:31That's link.com. That's their wallet.
59:32That's exactly what it does. You give your agent that skill. It generates a virtual card for every purchase that your assistant does and then you approve it. Huh?
59:39No link from site. Oh, okay. I need Yeah, but they're all still charging my one card, so I can't set like individual limits or set the expirations.
59:48No idea what it does. I still haven't been able to figure out.
59:50I don't want to give my agent a card thing.
59:53I just want a different card for AWS versus like G Suite versus etc.
59:57All righty. For dessert, we have lemon blueberry cake and dark chocolate Kreml mousse. I use Instinct for things that don't need my data like just like going and communicating with businesses and helping me book like events and things like this. And then my personal assistant is still a git repo with a ton of scripts and like the whole brain the whole memory system is marked down and I just launch CL codecs in there when I want to do stuff.
1:00:21Mine is pretty much Grockbot for all like work administrative type stuff but then I do still do a lot of the management through codeex itself. I'm using poke lindy and instinct at the same time and it's hell all dry message.
1:00:33I don't uh Yeah. Yes. Yes. Uh and Slack. Uh so I really like the the bridge into Slack.
1:00:40Yeah. I I'm realizing how silly this sounds as I say it, but um so dots, you know, very recently. Uh Devon seen that a bunch as a as an assistant and not as agent.
1:00:50Can you show us both?
1:00:51Codeex, my own thing that I built that has access to like my own little Google Drive setup. It's just like essentially sacked.
1:00:59There's like zero trust on anything.
1:01:02Email. I'm very like yeah like like I think that you know you get off code. You get 2FA codes in some cases to your email. I'm even for read I'm I'm scared of it.
1:01:11And then there's a long tale of like other like assistants like Orchid instinct etc that Orchid or I was orchid.
1:01:21Like 2 million views on Twitter, right?
1:01:23This was the one where the guy was like planning an anniversary date or something.
1:01:26Yeah. Oh, it was actually good. And um and Muse too for like marketplace, shopping, like it has um I think it's had actually really good uh browser use actually. Like Muse is surprisingly decent.
1:01:37Yeah, for me I use uh codeex and I actually don't like do anything else really. I just connect to codeex and have all my tools and al alternative harnesses that we're building as well. But um that's kind of all I do. I have a Hermes agent that I mostly keep up to date because there's a lot of stuff there that I built. Then Grogbot around 30 of them doing different things which I kind of like like the pattern of closing one bot to do a specific thing and they talk and
1:02:06then uh recently muse and dots probably muse the most. Oh, and instinct. Yes, instinct also.
1:02:12Wait, so how do you section them?
1:02:13I that's a problem that I want to bring to the table. Yeah. Uh so Gro is mostly for like work stuff like personal company management. I have like a chief of staff over there, a PM. I have a linear connector between all of them that they all talk through linear to ticket so they can tag each other. That seems to be apparent that's working. And then uh Muse for one off Muse seems to be the best when I talk to it to do something just like record the voice message like hey what about my hotel?
1:02:39How far away is it from this place or that other place or when do I need to live for that place? That's m like immediately. And also just send all three of them stuff that don't have independency or don't know how to pronounce the word. If there's a task that all three of them can can do without causing harm, I would send to all three of them just copy paste just to see who's better, who's faster.
1:02:58Use other actually for the Grockbot users. I'm curious like do you ever have it dispatched to like Devon or Codex or anything like that or no?
1:03:06Grockbots have cursor integration and it's amazing. So, Grogbot can work as a chief of staff for you and then he can can kick off a cursor cloud agent for with any of the well now they're not getting Astra, right? They're they're only getting the the soul models, but it can kick off like like Opus 5.5 and you can tell it hey like don't do this yourself on your computer. Kick off a cursor cloud agent with Opus 5.5 to do this.
1:03:31It's significantly better than obviously the Gro 4.7 that's on the line there. Uh so that's the pattern to like get it to do actual stuff. Even writing you can ask about to always write via cloud thing in cursor.
1:03:42I feel like I have enough code like I feel like between codeex and devon like I'm kind of good right now.
1:03:47Yeah, cursor good but it's such it to get anything good out of it. It's so much work. Like you have to set up the cloud environments for every project. You have to get the environment variables. Make sure the scripts are good. It's easy for the agent. Like it's so much work.
1:04:00It's also like it loves developing. So if you ask it to write something, it will send a PR. I don't need a I just need an answer. Write this in. But it has a pull and he loves a good pull request. Man, I think I one thing I love about Devon that I for some reason maybe cursor has fixed this now, but like uh I saw like almost every company fall in the same trap is like Devon like by default is just like it's just a computer, you know? It's like like an agent on a computer and like you know you can like
1:04:28create a repo, you could like you know find like you know use an existing repo can work across repos like just a computer. So it's just like super flexible in that way.
1:04:36Yeah. And so I never have to I like every like wild idea I ever have I like that's where I go. And whereas like um not not Codeex cuz I think it'll probably die like as soon as I close my laptop and like not anything else because it's like oh like which repository do you want to work in? It's like I figure it out. You have the GitHub connector, right?
1:04:55Sometimes you don't even want to be in a repository. You just want like a thing.
1:04:58you just have an idea and if it's trying to work within a repository it's like oh well I'll add it to this page and I'll like I just like want to see this like try out this new model try out jab try out like whatever it is it's so nice to just have a sandbox then I think this is like the difference also between assistant and like the coding agents that were built for coding they're trained to do the diff work they're trained to write a markdown file they connect compare it to something shove it in a doc somewhere that doesn't execute write tests whatever I mean just tell me tell me like write the thing and and give it back and I think that like all of them are that
1:05:28the whole system prompts like well I can't do that. What do you want to code today? I don't want to code. I just want you to be you unh my model for me. Thank you.
1:05:36Any things uh other than the these are the common names any things that are maybe way lower under the radar that you want to you think deserve more recognition? Uh, this isn't on personal assistance, but I do think it is very relevant. Is 6.1 soul is much much better than I think most people think it is because of just weird branding stuff that's happened. Six soul happening last week and now 6.1 soul happening this week. It's weird. I agree. It's an incredibly good model for an incredibly good price. It is worth looking at and
1:06:07It's just like to compete with Opus 5.5 basically.
1:06:09It is literally just kneecapping OS. I heard between the lines don't don't want to name sources that it wasn't named to be soul and then last minute marketing staff decided that like hey we need to like make up on whatever happened.
1:06:22No the the headline was that they had an extra and he pulled it for a personal assistance.
1:06:27Uh it doesn't have to be personal assistant anything underrated whatever you're enjoying whatever you think is cool.
1:06:32For me it will be personal assistance if you don't mind. Uh because I think it's an incredible unlock for regular people. We're developers all of us. We do stuff with these models for a long time. High taste testers will like test between the model. The normal person does not care. What personal assistants give to normal people is just like time unlocking.
1:06:50Doing stuff that otherwise would take them some time. And I think uh the labs now understand this. And I think that this is why they're all pivoting towards the the simpler use case. They also get the benefit all of them of how much token this will cost per user because most people still use charges Google.
1:07:07They type in a query, they get a response. Most people don't run agents 24/7. Now with personal assistant suddenly they are hooked on that like sweet sweet uh streaming of tokens that that most lab like enjoy but they also get productivity in time. And so I think it's cool to just not also talk about the products but use cases because I think in the limit most of them do the same things. Most of them can do booking. Most of them can reply to your emails and scan them. But I think it's very interesting to talk about what use cases can people do. I I can give you
1:07:36one that I did that I noticed that an assistant can do better job than like a coding agent before. Uh I woke up one weekend with my kids early to take them to a karate class that was closed. I arrived there and it was closed and I got so pissed that now in the 2026 proactive assistance era, there's nothing there that alerted me like, "Hey, you're going to a class that's closed that doesn't know my schedule."
1:07:57So I built a weekend plans bot for kids and and for my wife as well that every like as the weekend comes up goes and searches all the local places all the places I would never go to websites like ah cheap and fast or like good and expensive it just plans and says hey if you want here's a bunch of things for oh and by the way the stuff that you're meant to do is close that within as well that like is a good use case now I'm like I know that that corner is covered I think that uh Google AI search overview or that whatever that thing
1:08:28So underrated. Every question that I have that I want a fast answer to, I Google it.
1:08:35I do too still actually.
1:08:36Like it's really fast. It's like it's like it's like 300 milliseconds or something. by the time the page loads the answer is there and like chatbt it's like I'm always in like the max reasoning setting change it and then like but then if I change it then if I make a new chat it's on that stupid setting so like I'm like it's just more it's like 10 30 seconds you get an answer you can keyboard shortcut to instant uh which is still there
1:09:05I should I should look up how to do that or whatever that but actually the way I do it is recast I don't know if you Yeah, recast with AI. You just like, you know, command enter and then type in your thing. Enter.
1:09:16It copies it to your board.
1:09:18Whatever you want. So, I I set it to whatever.
1:09:21Okay. Well, actually, maybe I actually It's interesting you said this thing about like asking Muse like, "Where's my hotel and stuff?" Cuz I've actually started using Instinct instead of Google. Like, I was traveling and I got off a train at midnight in Amsterdam.
1:09:32I'm like, "Where's the nearest train station?" And like, "Can I pay with a tap?" And it's like, "Yep, you can tap your card and like go I go make two rides out of the door." And I'm like, it's not probably not as fast as Google, but it's much less overwhelming if I just want like a yes or no or a very short answer. It's it's very nice for and I'm not going to open the chat GPT app for all the reasons that you just said.
1:09:50No, actually everything that all you guys are doing is actually like suboptimal for your best experience.
1:09:56Which is one thing should have all your searches ever.
1:10:00So like you shouldn't route. Routing is bad.
1:10:03But we also we you can't be high tester without testing. Yeah, you have to test around to know like the differences in other stuff. The thing that I'm super excited about is the hardware integrations. So, Grogbot now exists in a Tesla once they release it a little bit, but like the ultra whatever tier now sits Grogbot in your car.
1:10:21It's in It's amazing. I I've tried I have a Tesla.
1:10:24Really? Not Grock. Grogbot specifically.
1:10:26No, you just like you do the voice to text with Grock and you're like, "Hey, go tell this bot to do this thing and be like sending it to the bot." And then it will tell you, "Oh, the bot responded with whatever. car drives it. It's so good, dude. The car drives itself and does stuff for you. It's like what kind of beautiful future we're living in. I'm super excited about this. That's so integration into hardware, I think, is is is something that excites me. Muse own glasses and use devices like the charm thing.
1:10:51I I I'm going to make a prediction. Small prediction. You guys will see if it's going to work. The charm thing that that Zuck showed, like the little Tamagotchi Muse lives in is going to get sold out immediately.
1:11:00Like immediately. I think I have plans in my agents to buy like six of them.
1:11:04Every agent is going to buy one. Uh because it's like if you think about people who are new to Judge BT or whatever and like new to assistance and they get on board on news that's a perfect Christmas gift just a perfect hey here's the thing charm it's a Tamagotchi but also like I don't know my my feeling on some of these things is like why why the hell has this not like happened yet?
1:11:24Why why is it not on the Apple?
1:11:26Oh you want to get into the why Apple is not in the race?
1:11:29Not I know why Apple is not your phone and everything else won't be your phone either.
1:11:34Same. you don't it's just I I actually firmly believe that like the number of things to charge is saturated and like some people are able to uh they're responsible enough to handle a watch but like it's actually a pretty small number and I'm not like the reason why I don't have Apple Watch is that I would always forget to charge it.
1:11:52This is a solve problem. Just bring those little battery chargers.
1:11:55No, it's not it's not solved cuz then I have to like remember and you're like the solution is another thing to charge to get to charge the thing does not work. $15. Buy four of them. Put it everywhere you're It just It doesn't work for me. It doesn't work for you.
1:12:08Uh cool. We're almost at a wrap. Thank you. Thank you guys so much. Uh actually, uh this is for you. Uh what's one thing that people can do to you do for you? How do they find you, give you money, whatever?
1:12:19The one thing that people can do for me or to you is or or to me to me as a m I would always like appreciate a massage. A random on the street approach me, start a massage, it's going to be fine. I want people to unlock their imagination about what's possible with AI systems. I think we're moving into a new era where a lot of people will get a lot of benefit from AI finally. I think it's very important to counter doomerism as well because there's one thing to see oh it's coming for us on the news and completely another thing that oh it actually booked me hairstone and saved me like five
1:12:48minutes once people feel that tactile like helps me saves me time they will definitely move towards oh AI is actually like helpful I think it's very important all the companies move there so the one thing they can do for me is unlock the imagination try uh AI assistance and if they want to know which ones we talk about this on my podcast called Thursday I every Thursday okay I at RC and uh we're doing a lot of models uh that are going to be pretty good and open source and coming out soon and uh and I'm also building harnesses over there. Checking me out on Twitter
1:13:17at Allisonology would be uh good as well.
1:13:20So in addition to being a you know a ship poster on Twitter, I also run a very serious AI company that makes agents more reliable. We work with like Fortune 100 healthcare, a bunch of Fortune 500 companies and some of the like fastest growing AI uh new startups. So um give it a try. It's uh free but for most like most people like if you're under like a thousand runs a day.
1:13:41What's it called month?
1:13:42It's called raindrop. Sorry detail.
1:13:51Yeah. So um I have a YouTube channel and a podcast. If you just look me up on YouTube as Ben Davis or Nerd Snipe, you will find me there. And yeah, uh I work at a company called founder of a company called Human Layer. We're building the software forge. I guess we're saying software forge instead of software factory now. The forge of the future because factories fail.
1:14:12Well, not all of them.
1:14:13Factory reference.
1:14:14There are certain things.
1:14:20No comments. Uh, no. No. Your AI talk actually went like super viral like Well, it was misinterpreted because everyone saw it and said like, "Oh, you're the don't like software factories guy." I was like, "No, no, no. there are things that you could do that would cause them to fail and I'm going to tell you what those things are so you can you know not do them but um anyways yeah I'm Dexory on Twitter check out human layer and uh yeah welcome to the future I guess I don't know I know well until the new future comes in 3 months can do a personal reference for Dex I think um met Pacock one of your guests
1:14:49also uh he said there's like a few folks whose skills he trusts and Dex was one talking about markdown skills I'm not like skilled in real life I'm talking about markdown skills I think if I give you an You can you can Yeah, I I submitted Maya for consideration next time.
1:15:07Uh anyway, this has been a great uh first episode. Thank you guys for being part of it. This has been High Taste Testers. The thing you can do for me is like and subscribe.