0:00I think software engineering is dead and it like [music] completely just chopped everybody off that was working for 20 years which is scary. I think people are pretty scared of that. It's always important when the experts in the industry freak out because they know. So particle how do you guys make money?
0:13The goal of the company is to drive the cost of these [laughter] tokens to zero. Today I'm talking with Su Doshi, founder of Particle which takes open models like Deep Seek and makes them cheaper to run.
0:25And he's actually done this before. Su Hail started Mix Panel when he was 20 and today it's a billiondoll company doing over $und00 million per year. So when Claude Fable 5.2 dropped this week, I wanted his take and I've been using it to edit videos, make 3D films, and build a ton of apps. And when he ran his own tests, Fable was only 10 to 20% better than a model that costs 30 times less.
0:48And so in this video, we talked about how to test which model you actually need. Why AI edits videos better when you don't use a video editing software.
0:57Why every job is turning into a coding job. Why he thinks software engineering is dead. Why he wants AI to rebuild GTA 6. And he also gave me some advice that I can't stop thinking about. If you're new here, my name is Riley Brown. Let's dive in. Su. So in the last three days, I guess in the last two days, we got Claude Opus 5 or no Claude Fable 5.2 and we also got Gemini 4 carbon. Have you tried these models? I'm I'm curious
1:27about your thoughts on these models.
1:30Yeah. Yeah. I usually like to these days I have stopped being on this trend where like the model comes out and I quickly go to the model picker and I change it or whatever. usually because I already have a lot of work sessions going on.
1:40But I did try Fable 5.2 earlier today and I just I just started giving it a whole bunch of problems. And I have not tried Carbon because it's not even clear to me how to how to use Carbon sometimes because I might be in codeex or or whatever. But yeah, I've tried Fable 5.2 and the best thing that I could say about it is it really likes to deny the work that we'd like to do with it.
2:05[laughter] actually like there's a lot of weird guard rails and I think one of the nervous things that I have about Fable 5.2 sometimes and some of the anthropic models is actually that kind of the the frontier sort of work that we're doing which is inference optimization and AI training models and things like that.
2:20I'm often worried that we're getting downgraded because they will just randomly choose like a crappier model and I'm always like slightly paranoid that that's what's happening and then we're just getting like worse code. But I did give it a shot before before today and and just thought thought I would try it out and I think I need more time with it probably.
2:39Gotcha. Yeah. I mean for me like as a creator I I've been really obsessed with using it for video editing. I think that's kind of the biggest shift I'm seeing right now. You know my buddy Muhammad Zulali. He's an he's an engineer and he's been wanting to make content for a while and he made a video. You probably saw it on the timeline. He reverse engineered his Jeep Cherokee.
2:59Like he got Clawude to like Wow. like control his car and he used Fable 5.2 to edit the whole video and it's going to get like over two million views on YouTube and I'm starting to see that a lot of these people who would have never made videos are just kind of filming themselves throwing it into Fable and um so I'm really excited about that. Like it's really good at like yeah it's it's amazing and it's super good at like visual understanding and editing videos and so that's what I've
3:27liked the most about Fable. One thing that you tweeted today that I want to ask you about. You said G you said uh GTA bench. What percent of GTA can the model recreate? Start with GTA 1 and hill climb to GTA 6. 100% accuracy is fully recreate the game. That would be indistinguishable from the original version for players. What did you mean by this tweet? I'm very curious.
3:51Yeah, I mean I've had this long view that that we we these models need increasingly hard problems because I think they continue to sort of astound us. And I think one of the things that maybe a lot of us didn't think that they would be able to do maybe around four years ago was editing videos. I think and then I think even just two years ago or 3 years ago we probably thought that maybe the video models would be the ones that edit videos and the same thing is kind of happening. It's but it's happening in other industries. So one industry that it's happening with is also robotics. The roboticists like went
4:18into a total existential crisis these last two three months. Like I spent a bunch of time in robotics last year and I think three different roboticists that you would probably respect just given their pedigree all kind of went through like a oh my god what does this mean the LMS can do it? So I'm always interested in like thinking about the things that the LLM's we wouldn't have expected them to be able to do and so increasingly over the last few months people have been making astounding games. And right now, I think a lot of the games are maybe a little bit impractical, like they're really good demos, but the
4:48interesting thing about great demos in AI is that quickly great demos become like actual products or actual things.
4:54And so, I just think that, you know, with GTA 6 coming out, I don't know if you play video games, I definitely pre-loaded GTA 6. And I was just, you know, pondering that, you know, we it would be, you know, and I've also been spending a lot of time looking at benchmarks, cyber benchmarks, health benchmarks, all kinds of interesting benchmarks. And I think that if we could get the models to sort of figure out if we could get GTA 1 to work, right, that would [snorts] be really incredible. I think people would be pretty astounded by even just that. And there's this
5:22other big movement that's going on as well in the last few months, which is REA, which is reverse, maybe it's like reverse everything in anything. I'm not reverse engineer engineer everything.
5:32Yeah, that's right. Yeah. And so that's going on as well. And that's also mind-blowing. people are making all kinds of weird games where they're instituting their own characters and things like that. So, it just feels like a really hard eval for video games would be trying to ultimately recreate GTA 6, which took how many billions of dollars and how many like a decade. I just think that would be fantastic.
5:54Yeah. And you you mentioned that a lot of these things that we're seeing on Twitter are just demos and you you want like the real game. I'm curious what are the limitations of that? like what do we need to achieve or what do models need to get better at to go from demos to the real game in your opinion?
6:13Yeah, I mean it's like we have all the pieces and that's why I think I made that tweet was cuz one piece was now we're seeing the models do 3D Blender stuff, CAD, the Ral Environment companies that generate data to help train frontier models. people started reaching out to me about being able to do CAD. And so it's kind of like we have all the pieces, you know, we've got audio figured out. We've got 3D models and stuff figured out. We've got coding figured out for, you know, networking and multiplayer games. It's like all the pieces are there. And so often what
6:43needs to happen, maybe what needs to happen is something even simpler than the technology just purely, oh, like we need to add more data for this particular thing is actually just a really great benchmark for the Frontier Labs to climb. Because often when you create these benchmarks, I mean, people are a little lazy. And so to to the degree that someone can make like a really good benchmark for these researchers, you know, to the degree that there's a number that they can hill climb, that's sort of like catnip. And so it might actually just be a benchmark because all of the little pieces are there. And so in order to hill climb the
7:12benchmark, it might just be like a lot of tuning and like extra data, but ultimately I think we can get there. So it's maybe just the motivate like something that motivates them to probably want to go and do it. But yeah, I don't know that there's anything really left. Like we're on our way.
7:25Yeah. I mean, three weeks ago or four weeks ago, I was using Opus 5.5 to make a game live on stream. And then in between like it was it was multiplayer, so we just had just it was convex real-time database with just hosted on Verscell and we were playing a game and we would play for 20 minutes and then I would just talk to the chat and then we we were like, "What do you want to change about the game?" and everyone would comment in the chat and I was just copying and pasting them and just ran it as one prompt with Opus and then the
7:55game would update and then we would play the next version of the game live on stream which that's amazing. That's awesome. which I which I think is just like the future of of gaming. But um I want to get off gaming a little bit because I think one thing that I can't stop thinking about and I think about this all the time is the offramp from all the subsidies we're getting by anthropic and open AAI. You know, I've been using I have the $200 per month plan on claude. I actually have two $200 per month plans
8:25and it feels for me because I'm like doing 80% creative work, 20% coding stuff. To me, it feels like I get unlimited tokens. Um, however, when I was at my previous startup, I accidentally used API pricing for one month and accidentally just spent $12,000, right? And so what I'm concerned about is like this offramp period from when we go from all these subsidies to not having the subsidies where you know we
8:55pay closer to the API pricing. How do you think this plays out? Like how do you think Anthropic and Open AAI are going to kind of are they just going to slowly reduce the amount of like credits we get on the the paid plans or how do you think that's going to work out?
9:11Yeah, so it's it's very interesting. So I think there's kind of like multiple angles to it. The first angle would be just like our own usage. I actually think our own usage is going to dramatically grow. And I'll give you like an example of how I mean if you just sort of look at the lineage since 2022 chatbt came out December November November 2022 we started to use it and most of chatbt was like probably like one or two terms right with it. You'd say it's you'd ask something it responds
9:40maybe you're done maybe you ask one more question you're done. And then we started getting things like tool use and search. Then it became a little bit more multi-turn, maybe three, four turns, but then you'd stop. And if we just fast forward a little bit, so I don't have to go through the entire lineage. I mean, we're looking at like tra we I see trajectories where, you know, if you look at like a trace of someone and what they're doing, I mean, people are taking like 256 turns, 300 turns, 500 turns.
10:04These are becoming really long horizon tasks. So that's one element is that we're just doing way more with AI. And then the second thing that I think is probably going to come are these sort of really frontier things that people are not really doing yet. One frontier thing that I think people are not really doing is um they're not running like an agent that's 24/7.
10:24For example, one very obvious situation where we should be doing that is in cyber. We're not doing that. And then another situation that I think is a little bit too frontier because the AI companies haven't really released models that enable us to do this is we don't really we don't really have a great easy way to do agents forms yet where we like literally spawn a thousand agents to go do something. I mean a lot of that requires the model to be somewhat post-trained. Um so so I think on one end of the spectrum we have not yet
10:54quite realized the quantity of tokens we might use actually uh given given if the models sort of improve and things get better and then the other aspect is just pricing right and I think that pricing has shown that about every year uh the prices have gone down relative to the same unit of intelligence. the reason why p prices have actually kind of gone up in some years but that but it relative to intelligence it has gone down. Um and so some prices have gone
11:23down but a lot of the frontier labs are actually pushing higher prices. Um and so a lot of users as a result of that just like pick the best model but they're actually paying more and they don't even know is there a cheaper model like what what it's actually too complicated like there's like five different models and then four different versions of reasoning and it's just too much for the average person I think. Um, so I think that's another element. And then the third element I think is that I find most exciting is actually all the open rate models that have come out. And
11:52so one thing that people ought to think about is that the frontier labs have earnings earnings before training [laughter] which is that they make money and they have to and then there's the margin they make from inference and then there's the margin that they make after all the compute that they had to use for training. And that cost of that is actually the period of time that they got to train that model and the the success of the premium dollars that they got for the period of time that the model stayed premium, right? Fable 5.2
12:21will not be a premium model in probably like six months, right? So yeah, that they have to eat that cost. They have to stay premium for some amount of time until openai or deepseek or whatever zi comes out with their models.
12:34And so the other thing that I think is happening that's really exciting and I think really great for the world is the openweight models are coming out and then just like flattening the field every time. And they are some they're often a bit behind but they flatten the field so much that there's this 20 30x price reduction. And so I think a lot of the openweight models are eating up the frontier models. you know, whatever was subfront whatever was frontier last year is no longer uh no longer frontier task and the and these openweight models just
13:04eat everything up. And so if you look on open router for example, it's growing so fast like I think deepseek is if you count both versions of deepseek that are like roughly the same model. It's 60 70 trillion tokens a week growing exponentially actually.
13:20Gotcha. There's so many directions I want to go with this. Um, what I want to talk about first is you were talking about people running agents 247 and so obviously you know I've talked to a lot of developers who are running agents a lot of the time right and they're doing a lot of these like cyber security workflows where you know they have them run basically all the time they're checking things anytime someone posts in Slack or Linear it triggers off some workflow which is interesting I think a lot of people are talking about this what I'm curious is how do you see
13:49this playing out for you know businesses and then just like end users users and consumers. You know, I think we're only just now kind of in the early stages of, you know, consumers being able to do this with tools like the Hermes is the kind of the open one and then you have like GPT dots and Grockbot, etc., which gives you a persistent computer so that you can actually run more work tasks 24/7. How do you see like do you think in a year a large amount of people will
14:18be running agents 24/7? Do you think they're going to be setting them up or do you think they'll be set up for them?
14:23Like how do you think that's going to work for average business people?
14:28I think on the consumer end I think it's going to go the way of things like search. Like could you imagine paying for search? You know, no way. And and so I think the way that it goes on consumer is that actually people don't really want to pay $20 a month on the consumer end. $20 a month is a lot of money. I mean if you've ever run a consumer company and you charge people you $15 a month. I mean, it is like bloody murder if you charge them a month and they didn't expect that charge. So, I don't really think the average consumer really
14:57wants to pay for AI at all. And I think that is going to get totally subsidized.
15:01Someone is going to find the right business model. Maybe it's you making money off of e-commerce, maybe it's money off of ads. Likely it's, you know, generally it's ads. And so, I think that that case it's just a function of like what is the capability of model that you need? And once we reach this kind of like threshold of or plateau rather of capability, right, do you really need a frontier model for non-frontier tasks?
15:24Probably not. And so a large portion of consumer tasks, I think, are going to be non frontier. And you know, it's not like they're using the models to go and like research cancer or anything, right?
15:34So so I think a lot that that cost is going to hit a a bottom. there's going to be a point where the the price decreases so quickly uh over over the course of a year or two that there'll be this moment of a threshold where like it will unlock a ton of consumer categories that were previously like I don't know how to make money off of this because none of these consumers would pay us up.
15:52So I think that's probably the realm of way the way I suspect consumer will go and and we'll see what happens and it'll be really interesting. I think most of the token dollars will flow towards enterprise and we will increasingly find way the only way for companies to have alpha is that they have to come up with really hard things to do to compete against each other and they require like time and energy and capital and focus and those kinds of things and so I think a lot of the tokens will end up with in enterprise. Yeah. Anyway, I don't know if I answered your question but but I
16:22just thought I'd pause and see if see what direction you want to go.
16:24Yeah. Yeah. Um, yeah, it's very interesting to see all the different players on the consumer side. I I know you have Instinct and they're raising like literally billions of dollars. Um, and I guess their business model is um taking a percentage of like the commerce that happens on the platform. I don't know if that's true. I I heard that.
16:42Um, and then I think Open AI OpenAI strategy is to is to do ads, which makes a lot of sense. I'm curious. So, what I want to talk about is is particle. So you are trying to make models more efficient. You guys is wait so particle has trained you guys trained your own model correct?
17:02Yeah we trained our own model. Yep.
17:04And so you trained your own model and did you use one of the open models as as like the base for the model or you want to just talk about that process?
17:12Yeah. Yeah. So so one of the things that we we've kind of learned is that uh so things like pre-training and pre-training a model is very expensive.
17:19That's the kind of thing where you need to get a lot of compute and you need to raise tens of billions of dollars at this point to really go do that and and so there's some labs that are trying to make that cheaper but ostensibly like to compete at that level. It's a lot of money. So what else is there? Well, the other thing that we we figured out was that we can do what's called post training which is you take an existing model that's already been sort of made like deepseek or GLM or Inkling. There are a lot of different openweight models out there and you basically just like tell that make that model like better at
17:48other kinds of tasks. And so it just turns out what we've learned is that the insight that we kind of learned is that post training is actually like this very long wide surface area and and it used to be that when you train these models, you know, you do a little bit of post- training and you're done. Um it was kind of like the final touch if you will, but that's that to us that's like no longer the case. And so what we did with particle one um is our first model is uh we did something that we noticed in the market which was that a lot of these models are very token inefficient. They
18:18just sort of overthink and they overreason and spit out a lot of tokens and uh and so you know we we were wondering like could we actually make it more efficient? Could you know people have a cost sensitivity um to using all of these tokens whether it's personally or or at work to what degree can we lower costs in like a different vector?
18:38A lot of the inference companies that we may compete with or or our rivals of ours, they mostly focus on like making GPU writing GPU kernels and just or just making the price lower. But we were we thought what if we just make the model more token efficient. And so um so particle one basically made made uh uh made it 42% more token efficient than deepseeek. And we actually had saw people that were kind of complaining that Deepseek uses a lot of tokens. And the other the other factor is that by being more token efficient, you actually
19:07complete the task roughly that much faster. Uh which we think is pretty cool. Um I I know that I don't love sitting in front of codecs just like waiting for these things.
19:17That is the one thing I hate the most is how often I'm just like spread between like four different things at once and then I just end up waiting for it to finish. I'm curious how can you give us a breakdown on like how you make a model more efficient? like what does that process look like without spoiling any secrets?
19:35Yeah. Yeah. I mean I it I it's uh we just use reinforcement learning, you know. Um that that's the way we do it.
19:42We don't we're not doing anything where we're modifying weights or or anything like that. um it's just really good data and really good empirical kind of like science and evidence and and in reinforcement learning and and sort of picking the right shape of your algorithm or reward rather when you're doing RL uh in order to like get the model in the right zone. And so one of the things that I think that we one of the things that I think is sort of boring but actually really important is just like really doing a good job finding the right like eval that you
20:10would use. And so we spent a lot of time actually looking for like the right kinds of data and and so I personally even I personally spent days just like looking for the right kind of data that would adversarily like show us bad token efficiency and then we sort of adjust adjust the model and use uh reinforcement learning to to do better on those data sets. And since launch like we found more like some customers have had really great uh have kind of like independently shown showed us that we actually do do it and then some customers are like hey like I don't quite see the gain in this particular
20:39category. So now we've been kind of like adding adding up all of the feedback so that we could go do round two.
20:44So you started with Deepseek, right?
20:46Deepseek, which Yeah, Deepseek Flash 1017 for those that care for Gotcha. Gotcha. I'm I'm curious like um where does Deepseek fall in I guess like can you give us a breakdown on like the different open models um and like the different players there? Like if I were to go test uh maybe I wanted to use Hermes agent or um one of the other platforms that lets you actually use the
21:14open models. Um what models should we start with if there was like a list of four?
21:19Well, I could even cut down the list to two, but um but I think the two the two great models I think that from the my favorite two open wave models are actually the the ones that come from Deepseek and uh GLM, which is a company called VI that's behind them. They have both pro models and flash models. Flash models tend to be the the cheaper ones.
21:38Users really like those those two models. I think they're just like very high quality, not overly benchmaxed, that sort of thing. Whereas I think a lot of people do like Kimmy. It's a little bit more benchmaxed for whatever reason. And maybe like keep it to those three. I think those three are pretty pretty good. A lot of people like the like the three trillion parameter Kimmy model for some of the more frontier workloads.
21:59Very curious. So, particle, how do you guys make money? I'm like very curious about I'm I'm very curious about the like I've just been, you know, um I was hit with a wave of um I guess not cynicism about, you know, creating software and and creating rappers just because it feels like the the labs are are creating everything. And I've talked to a lot of SAS founders who are like really struggling. Um, especially the
22:27ones who don't really deeply understand marketing. It's just become like a marketing game. Um, and I've been just trying to kind of understand I've been really trying to understand the business model of the rest of the like going deeper down the stack. And I think this is a completely a new business model that I'm not familiar with. So I'm just really curious on like how you guys make money and like how you plan to make money in the future.
22:49Yeah. You know, I have a very simple thesis on making companies these days. I try to like actually try not to overthink it too much. I think there's a lot of overthinking that you can do, especially if you're doing it like a second second time. I call like second time founder syndrome. And I think one of the things that that is just hard to do when you start a company is the hardest thing actually is not there are very few companies that die because of lack of moat. Most companies die due to lack of value. They never made anything that anybody wanted and they were so
23:18focused on defensibility that they like invented a a product that nobody actually wanted. And so I think with with and you know to answer your question we make money off of tokens.
23:27It's not that complicated. The business model is sort of simple like we make money off of tokens. Now we have this funny we have this funny aspect to the company where the goal of the company is to drive the cost of these tokens to zero which belies this question of like well then how would how might you make money in the future? But what I find is that's why you asked that's why I'm asking the question um because I saw your tweet that said that I was like okay if you're trying to drive the cost down to zero it's like what's the business model?
23:53Yeah. Yeah. Yeah. Totally. Well, I you know the thing is is that I think it's going to take time to drive drive it to zero. I think that's one element of it, right? Um I think the second element so for a while it it'll be okay. Um I think the other the other element and so we don't mind making kind of like not that much money. Um there there's also an analog to this business. Never has there been a business I think since payment processing companies where um an increase of a unit of your thing converts directly to dollars like
24:22intelligence is as close to dollars as I have ever seen anything it's like a onetoone mapping on tokens to dollars it is it feels like payments and so I often liken the company quite a bit to like Stripe um shout out to Patrick he's also an investor of ours but he's but but Stripe You know what Stripe what Stripe what margin does Stripe make?
24:45Less than 2.3%. Right in reality, right?
24:48Their big enterprise customers probably don't pay 2.3%.
24:51And so Stripe is definition of a commodity business um with payment processing, right? And and intelligence is much better margins than that at the moment. And and so one one of the things I think that is is maybe worth worth mentioning is that intelligence the it it works to have a low margin business but only in the situation where the volume is like ever expanding like nearly unbounded like payments payments is just like infinite right or whatever the world I mean it's finite to a degree
25:20but it's the pi is constantly growing and intelligence is also uh has a lot of those same feelings it's just this unbounded thing that keeps growing exponentially year on year right now and so it kind of doesn't matter to have low margins and so that that's one element and I think the other element is that let's say we were to drive the cost of tokens down to zero well if you achieve that then our company will have amassed as many developers as pos as we as I mean we we've have a ton of developers using our products and so I think one
25:49reason it can be hard to sort of say well we'll do this is because it's really hard to predict in five years or 10 years this thing because it's super hard to predict that but there will always be something that is next. And so I I suspect that what we'll actually have is just huge distribution of developers where we're just trying to help them again do build better applications with with intelligence where intelligence originally was the business model and now it's going to be something else and and you know I just can't perfectly predict that. But I don't think that I need to um because I suspect there's always going to be
26:19something that is using these AI models to help build better applications and things like that. Um it it seems like CPUs ought to be zero, but they're not.
26:27They're still not. You know, that's a really good point. Like, you don't have to predict it now. You can just help people make or use intelligence at a cheaper level until something will present itself. Like even 18 months ago, it was hard to predict how people in their job would be using AI on a day-to-day basis. Um, but like in hindsight, it seems obvious, I guess.
26:47Yeah. GP GPUs are just like that. you know, they were used for games and then suddenly they found a use case in AI and and there's no way that the folks at NVIDIA predicted that was coming in the first two years of their company, let alone the first five months, right? And so I just think that the first order bit of of making a company is actually just do something that's valuable to people.
27:06That's hard enough, right? 100%. And um okay, so for a lot of my tasks, I'm using frontier models.
27:13How can I as a small business owner start to transition from using frontier models to using cheaper models? And I guess another way to ask this question is like some tasks somewhat require the frontier models or should be used. I guess if I'm running everything on like like basically everything that I do is set up through some sort of automation using either claude or codecs and I don't know which platform that I should be switching to to use these these um open models. And so
27:43if even if I wanted to use Particle or one of these open models for a lot of my work tasks, like what tools do you recommend using at a more practical level?
27:53Yeah, I I think Open Router is a great place to try stuff and experiment with it. Like it's undoubtedly the best.
27:59Obviously, we say, "Oh, people should come to our site. We have these these openweight models." But if I if I were to be totally honest, I think that open router is a great place where just like quickly flip between them and give it a shot. I think a big part I think one big part is that because AI is so nent and and sort of new to the world in a sea of a lot of options you trust the brand you pick a brand you trust right and I think that that's definitely happening whether it's what harness you use what models you use what ad you're using right
28:28there's all there's so much and so you're just like okay what what is everyone else using what's roughly the right direction and I think that's a reasonable thing and then I think I think there's a secondary very bit which is why we're not we're not chasing after like codecs and we're not we're not trying to build a business in that lane because we we think there is actually a lot of confusion there. Um, we tend to focus our company on serving developers who are making applications using AI where they have very specific requirements around like this is what I
28:57want my app to do and it has to have this performance and once they lock in that they might use whatever they might use the frontier model but after they get there and they they ship that in the product they look at their billing and they get go they go oh my god this is like unsustainable we cannot we should do we really they ask the question do we really need a frontier model and the answer is often been known and that's where that's where we think that we can kind of get involved and also they're able to use open router to switch between these models and try to get like the right price to performance ratio. So
29:25I think on a personal level like even I use the frontier models and we just haven't built the right UX around knowing really when to switch and we almost don't trust it. We're like oh no if it's using Luna like maybe I'm not getting as much mileage out of if I was using Astra extra high right? uh if you're in if you're one of these um if you're if you're as nerdy as I am around these sorts of things. But I think when it comes to using it in your application or running something day-to-day, I think that I think that just you got to just like try them and and look look out for
29:54evals and you should all always have your own eval even if it's five things, six things that just really measure that because you could likely get a 30x reduction in price for the same thing. And I think a lot of people just don't realize that the openweight models are really good at things like that.
30:07Yeah. And where can we find or create our own evals? I hear the word eval a lot honestly. Like I hear the word evals a lot and it makes sense for coding, but a lot of it feels imaginary. Like maybe I just haven't taken the time to actually learn eval. Maybe it's intimidating. I don't know why. But like how can we create evals for non-coding stuff? I mean that's been my main focus right now. And a lot of my audience are people in businesses, you know, like maybe they're, you know, a marketing
30:34executive. For these types of tasks, I feel like people are kind of defaulting to the frontier models just because they trust them more for whatever reason.
30:42It's hard to kind of build trust to start using the open models. So, I'm curious on how we could actually set up evals to measure how good a model is at something like video editing or something like script writing or general administrative tasks. Like what are your thoughts on that? Yeah, I think I mean I think this this is different for every person. So on a person level basis, you know, there there's um Simon Wil I think his name is I forget what his Williams maybe um and he he always has this
31:11Simon Wilson and so he has this he has his Pelican Eval which he likes to do and he probably has other ones but I think that I think that that's the right shape of things that you sort of take something that you find yourself doing quite often. In your case, that's like a video and and so you just have a folder and you just call it eval and you just copy and paste the prompt from one model to the next and you just go like is it better or not? And one thing I mean I did this for design for a little while and you know I noticed the funny thing is that is that I noticed that there was almost no difference between GPT Astra
31:41and Deepseek V4 4.1 Flash like it was the same but the cost was like 30 times lower and the result of the design was like equivalent and and maybe I got a marginal like 10 20% improvement using you know whatever Fable and so I thought that was really interesting and I think just having kind of like a small little folder of your things and small business of like okay I'm going to just like run the models on these five things and that should give me a barometer reading of like this model relative to the things that I like. And then I think for the more sophisticated people, you know, the you could probably use the models to
32:11make your evals like use the frontier model to help you, you know, sort of like you do your trade, you know, you have your like trajectory where you edit a video and you went back and forth and you could probably just like and then you get to like a happy place with your video, right? And so you have your result and then you could probably just download that trace if you're a little bit more savvy and ask Fable 5.2, too.
32:29Hey, would you turn this into like a an eval where you know there's a score behind this and these are the kinds of things that I liked about my video and can you can you make it so that we can score those pieces? Something like that could really help also and that might help you start automating an eval where you can just run it when new models come out and then just do like performance to price.
32:47Yeah, I guess no one's really made like a super userfriendly eval platform of sorts. I wonder if that would be that could be kind of interesting. Um because yeah I think a lot more people are starting Go ahead.
32:59Well I was going to say that that the industry that we're thinking about here with traces is actually this big budding industry called RL data environment companies which I'm sure you've you've maybe come across here and there and so it's almost like a little secret for you to not like it's almost like they don't want you to know how to do it or at least how to construct these EVA as well. So that's that might be in part why it doesn't exist.
33:20Yeah. I mean, it would it seems it I would like for it to be more accessible because you're basically just taking a prompt and you're just evaluating the output um and you're measuring across different models, but I guess I've actually tried to set it up and I it always ends up being like unclear. It takes a lot of work to evaluate whether something is good or not. It's just there's a lot more tedious stuff than you think if you're just if you're going to set up evals and I think that's where I kind of get stuck. But you should make a skill. I bet someone
33:48could make a skill that like gets everyone 80% of the way there.
33:52Yeah, I think this trajectory and turn it into an eval.
33:55I saw you tweeted the other day. You were talking about how you work with like you're interested in working with people who are kind of building co-work like tasks uh or co-work style. I'm curious on like what use cases are you seeing these companies use your model for and where are open models good at for like co-work style tasks. It's always hard to tell the line between like what is a coding task and what is a work task. They they almost feel like they're the same. Like good coding
34:23models end up being really good at work tasks. I'm curious what you think there.
34:27I agree with you. I mean I I think if you go look if you're an AI researcher and you're trying to make your models better, you will go look at benchmarks.
34:36And so I imagine most people watching this maybe aren't AI researchers, but roughly what happens is they go look at the benchmarks and they really go read through them. And and so what you sort of realize when when you do this is that you realize exactly what you're saying, which is that most tasks that seem like not coding tasks are in fact really just using large amounts of code. And and so because the models got good at code, they really latch on to that. And and so I think that you're exactly right in terms of like why why we're interested in that. Go ahead.
35:02No, no. I I was going to say like that is what I realized about video editing. um is at the end of the day, video editing is just manipulating these files. And I think people have gotten habituated to like think of video editing as not coding because it's all timeline based.
35:18Like video editors just um see it as timelines. They don't see it as every single thing as a code. Even a sound file is just a file on your computer and it the models can actually directly manipulate those files. And what I've found is that these models are better at video editing when it's fully headless and the the apps themselves are bloat.
35:43When it tries when it has to use the MCP for Da Vinci Resolve, a video editing platform, it ends up doing a worse job than when you just say like do it headless, edit the files directly because because it's it this is what I've just found these models. Codeex especially is really good at editing videos fully headless and at the end of the day it's all co it's all a coding task and it just uses like ffmpeg to like render the video and it puts the sound file and yeah that it's
36:13I think the reason behind that is really just that the AI models have probably been more trained on ffmpeg than like the computer use interaction or the MCP sort of like API that's connecting to Da Vinci Resolve. there's less of a integration during training, but like there's so much code that uses ffmpeg.
36:31So, and then ffmpeg is just so powerful as a is effectively an API that it's like this unbounded thing to the model to be able to use as a tool. And so, I suspect like to some degree that's happening. And so, what really what's funny is like we as a humans needed this UI, but then the UI obviously I mean we've all used UIs. We're like, ah, I wish there was this feature. Why does the knob not go down all the way? And then the irony is like that's the worst possible interface for an AI model because an AI model doesn't care about a
37:00UI. It just actually wants to be exposed to maybe everything. And so I think that's like maybe really interesting because I I think what you're saying definitely applies to like every category across the board.
37:10Yeah. Like um I've noticed it with PDFs too. Like everyone like oh when you want to create a PDF a lot of people in work they would go to Canva. And now if you try to go to Canva, they have this integration with codeex or claude and then it just it doesn't work because now it's working and it's putting things on.
37:27But if you ask it to just make the PDF directly, it's it there it's mind-blowing. And then additionally, Blender, which has been my biggest obsession, I posted a video, it got like a million views with Fable 5.2 too where um it creates these 3D environments in 4K like like it is as high quality as you can. It uses um I'm learning about it now so this might not be fully correct. It uses when it renders it does it in cycles. So like the lighting is
37:56super realistic and it'll like render the video and I use something called runpod and runpod rents GPUs because it would take like 10 hours to render on my MacBook. It just rents it and then for and then it just does all of this and then apparently Blender the the reason it's so good at Blender is because every single thing that you can do on Blender is mapped to Python and so it's all just Python code. So I realized that every single thing is just coding. So yeah, I think you can hear in my voice I'm very
38:25excited about these creative things and I just realized it's all it's all coding and it's so fun to use. So like yeah, the interaction now has changed too, right? Your UI is not like this Blender thing you have to go do, you know, 100 hours of YouTube tutorials on.
38:40It's just it's just your voice and sort of like learning like learning like the highle bits of like what you want to do and and then getting it to like construct. I mean, you're basically on track to making your own mini Pixar.
38:51Sounds like I mean I Yeah, I could one of the problems that I think a lot of people are struggling with um is like version control now because like Adobe and all these tools kind of handled it and now people are editing it editing videos and they're getting clawed to do it to like edit their video. their video will pop up on their screen and then like the window will go away and then they're like I have no idea where this is on my computer like where any of the files are despite the fact that like they plugged in their camera and it pulled all the
39:20photos out of their camera and that's what they're using. They don't know where it's happening or anything. So there's a lot of problems that come from doing things headless. A lot of these same problems are the same problems that people ran into when they were vibe coding, right? When people vibe code, they don't really know what they're doing. It's not like they are checking to see if the back end works properly.
39:40They just have the front end of the app that they can look at. That's why frontend stuff were so popular amongst the non-technical crowds because you could evaluate how good it did. But it leads to a lot of messes. So I think a lot of cleaning up I think we need to figure out how to like reorganize all of this and this goes for every work task too is like how do you create good version control and I think Claude Anthropic is trying to do it. They're releasing a bunch of different types of artifacts and they're going to be like more team centric and they might bake that in
40:09the old software world you all of us would say you know what let's say call two years ago we'd all be like oh you know what we need to do we need to build a feature or version control and I think that's like a boomer attitude I think the new attitude is like ah this doesn't happen oh well let's just we'll just train the model to be better bit better at learning to do this for you like it should just like tell you would you like to version control this or in fact we've already version controlled it we've already put it in this like area or you connect Dropbox or whatever it is that you do and it's like would you like to version control this in Dropbox you know
40:38I notice that Dropbox is running on your computer and it's it's crazy but I actually think you know there's the adage of like the best UI is no UI and I just actually think that the models are just going to get so smart they're just going to know that you have these various file systems iCloud whatever and it will just ask you would you like to do this and by the way not only will it say would you like it'll say well we already saved five versions so how about we just like save only this one but is it okay to delete this other one that was like from yesterday that you're not really that was like v1 I just think the
41:08models are going to get super good at this and we don't the features are in the model now not like we are building the features in software and I think people haven't quite reached this mentality that actually product is like more in the model now than it was uh less software now I think people are pretty scared of that they're reluctant to believe that because you know they a lot of people get their their sense of meaning in building features you know They would they want to be the person to be like, "Oh, we should do that."
41:34So did file clerks, but then we got it.
41:38We're not like we're not putting like little papers in files and then putting them in cabinets.
41:41What happened was is we got computers and this huge industry spawned around it. That was even bigger than being file clerks. And the problem with that fear is that people feel this sense of fear because they can't they can predict the demise of their role. But what they cannot predict is the rise of something new. And we those people that were file clerks definitely probably didn't end up in IT either, but a bunch of people that were kind of nerdy were like, "Oh, I could I could do it. That's kind of fun."
42:08Okay. So, what's the what's the what's the advice here then? Like I I know we can't predict it, but like if if you were telling someone some advice to like internalize this, like how do you approach this new world? Like like can we try to envision what's coming next or like how do you take that mentality without going insane? Well, I suspect tons of people that watch your YouTube channel are like these deeply curious people, right? I mean, they're they're watching you because you're always
42:37talking about this cutting edge stuff.
42:39Like, those are everyone watching your videos you don't even have to worry about because they're so curious and they're so open-minded and they're so interested that they will find their way merely because they're curious about something really cool and kind of like edgy and technical, right? I think it's I I think that those people are going to be great and they're going to be fine and even without knowing what's going to happen. And so I just think the answer is like just like keep evolving. Like I think for example all the software engineers that had their grieving period
43:07from the month of November to hopefully now hopefully people are done grieving.
43:11They're moving on on some level. I certainly had a little bit of a grieving period from January, February, March where I was like oh no like what what does it mean? I I used to write code in seventh grade. Now what does it mean?
43:21And I think that I, you know, I think it's just like reformatting your view. Like instead of thinking of it as software engineering, I think software engineering is debt. It's not software engineering. We're doing AI engineering now. That's the new role. And so like yeah, you might have to reskill and learn a new thing. You might have to learn some a little bit of AI research.
43:40You might have to learn what a transformer is. You might have to learn how to train a language model. What are tokens? What's a KV cache? These are all new things that you didn't know that you might be a little bit behind on, but it doesn't mean you can't catch up because this is this is a 20 year this is a two decade, three decade like kind of continuum for everybody. So there's plenty of time. This is actually the early you're still early.
44:01It's good to hear you say that. Um and I think that I agree with you. A lot of my audience is super curious and and um interested, but at the same time they are worried. And I think and it is hard because I get worried. I am very much like I'm in a good position, you know, I make content on YouTube. I have a large following across four platforms and every once in a while I wake up and I'm like is old content creation going to be replaced by AI? Um like like what am I going to even do? Um and I would say I'm more optimistic and more focused than
44:29most people. So I just I know it's very easy to go down the the kind of the the dark mentality, but I think you're right. I think it's going to be a long So I I guess to kind of like dig into that just a little bit. I guess tactically, right? Yeah. What are those things that uh people should learn about? And I know this is it's really broad, but you brought up a few topics like should people learn how to train models? What are the things that people should be learning if you're a software engineer? Like how do you start making
44:56the pivot into whatever AI engineering?
45:00Yeah, I think that I mean, okay, one thing one thing I want to mention is that one way to quell your fear around, you know, is this thing going to go away because AI is going to like just do it all. I think one way to quiet that and one way that I personally use to quiet that is that I tell myself to scale up my ambition. If video editing is going to go away, scale up. Like what is the thing that you never even thought was accessible to you that you can go do
45:28now? Like you probably think I could never make a movie just by myself, right? I could never get a movie in a theater by myself. I could never get to Sundance or or whatever it is. I would massively scale up, you know, I don't I could never have made a children's TV show like Blueie. I would scale up your ambition because it means that a lot of the things that actually took a lot of people previously or a lot of money now don't. And now a lot of people can do way more. And I think this hearkens back
45:56to my GTA 6 benchmark, which is that it's just this idea that like I I never thought making Grand Theft Auto would ever be accessible to me. And now I think that my son, it will be accessible to my 5-year-old, you know, in maybe a few years or six years. It'll be like all of his friends in class are like making GTA 3 for fun, you know, and so I think that element he'll have to scale up his ambition. He has to go bigger than that then.
46:22Yeah. But he won't even care that he his mind is like is like that's just default mode, right? And so that's why I think it's a little bit we have to realize we have a little bit of a boomer mentality [laughter] because my 5-year-old doesn't even care, right? And and so that's what some of us are going up against is my 5-year-old but in five years. And I and so I think that's one thing. I think Paul Graham has a great essay on kind of touching on what you're talking about.
46:44He has this wonderful essay that came out in 2023 which was how to do great work. Just Google it. Um, and and it basically, you know, I don't want to spoil for you because it's such a good read, but but I think I think a big part of it is just like sort of following these like strange paths, these unusual paths of your interests and not like not judging yourself for having them, right?
47:06And then just seeing what happens. And then, you know, when you're young, it's really easy to adapt. You know, it's like, how did I find programming? I did find programming by like going to the library or taking a class. I found programming because I was playing Starcraft online and then there were clans and then people made bots and then you had to have a bot if you were part of this clan and I was like how do you make bots and then I started learning about Visual Basics 6 to bot to find bots and you know fast forward now and now we're doing AI companies but I think that I think just following these little
47:35strange directions particularly when you're young is healthy. I agree. And I think all of my most effective times I think in the last like four or five years has been through just like a deep obsession about random things. Like I got super deep into the midjourney Discord in uh 2022 and that's what set me off into into AI. That's that's what got me into the GPT3 discord before chat GBT came out. And then I was just obsessed from there. And I think obviously some of that is luck. You know, it's just like AI is like the
48:05coolest technology ever. But I think I think it's it's easier said than done to to be like obsessive and curious and follow your curiosities just cuz people have like demands out of life like they or or they have responsibilities, you know, they need to feed their family.
48:20And so it's just interesting to me to see that a lot of young people hate AI and maybe it's a lot of it is propaganda, but like I mean it's just insane. Like a lot of when people give speeches and talk about AI at their commencements, they're getting booed. um when kind of like the system feels like it's it's against them but like AI is kind of this great equalizer where like it's speeding up totally it's forcing people to have to adapt to change faster and theoretically the younger people
48:50who you know have less responsibilities they don't have all these deep rooted habits in in an industry for a long period of time they should be the best suited to kind of win this game they just for whatever reason culturally maybe the AI industry failed Yeah, I think I think I think there's a different way that young people can look at AI and and I think that one of my favorite things about technology is that it does a reset. Like the whatever the chairs around the table, whoever the kings around the table were on one
49:19generation of technology completely reshuffle, right? Like Sam and Dario were not at the table 10 years ago, right? Now they are. And I just think that one of the nice things, but but this actually trickles down in a a lot of other ways. Like one nice one beautiful thing about technology is is that it resets things and and so one way to like sort of the the the other version the alternative universe right if we don't have this is that you like go to school you graduate and then what
49:47you have to do is you have to tirelessly work for 30 years climbing a ladder and you'll never be as good as the other person because they just put in raw raw more hours. It's like 10 years times 40 hours a week or however much they work, right? The average person, you just have to catch up. But the nice thing about technology and especially right now in AI is that it like completely just chopped everybody off that was working for 20 years, which is scary. I like totally admit that. But for young
50:16people, I mean, it presents a tremendous opportunity because now that person, how many years of knowing AI do they really have? I mean, if I'm looking for people to do inference optimization, right? You know, it's not like I can go to recruiters and, you know, there's that like meme on X where it's like, oh, I'm looking for someone that's that has like 10 years of inference LLM experience. I mean, there's just nobody. And and I remember that was like a meme when the iPhone came out and the app store came out was like how many people had more
50:45than a years of experience making apps on on the app store? Like very little at that moment, right? So it I think it presents tremendous opportunity for young people. I agree with you that it's like a flattening and equalization, that sort of thing.
50:58Yeah, 100%. And yeah, it's it's mind-blowing. Um, and I was I keep thinking in my head about you said like scale your ambition. And I think yeah, you definitely need to put yourself in a position where you can scale your ambition. I know you probably spend a lot of time with founders, right? You're a founder, you're selling to founders.
51:17And so these are people who have really set themselves up to be able to scale their ambition. Like you have to be a person to scale their ambition.
51:25Um and I think there's a lot of people in jobs where they're not allowed to scale their ambition at like a structural level almost like um you know their job, you know, I was talking to someone where in New York where their full-time job was like they would do color grading.
51:40They they're not an editor. They just do color grading and then this a little bit of the sound design. And that's like their whole role. They get like handed footage and then they hand it back and like they're not allowed to do anything else. They can't edit the video in full.
51:54And so the correct answer if they had unlimited freedom is to make the whole movie. Like figure out how you alone can like get the resources to make the entire movie because you with quad code could figure out a way to do all the planning, go out and get shots like like and then you could organize all this footage and go through it 10 times faster, but you're just not allowed to do it. And so whenever I give people personal advice is like you have to like structure your life so that you're allowed to scale your ambition if that makes sense. And it's always easier said
52:24than done. I don't know if any of this resonates with you. Yeah, it it of course of course it is. And you know it's obviously hard if you have if you have a family and a mortgage and you know you got to pay the car bill and and that sort of thing. And so I think you know but but like I do want to say you know there are pe you know when I was 20 I had nothing you know like I had while I had less responsibility like I didn't have a child I also didn't make any money like I made practically nothing so you know it wasn't like I had I had a lot of opportunity either and and I didn't even have a car so I didn't have
52:52a bill for a car but I didn't have a car [laughter] and so I think I think just there's some realities where like you know to the degree that you can put yourself in a position one way to do that is to also like scale back right scale back the things that cause stress and responsibility. I think there's this beautiful thing when you're young where you have like almost no baggage and responsibility and it obviously gets harder when you get older. But I think that yes, I think I think that you should you should scale your ambition in that way. And I think I'll give you I'll
53:20give an element. I know this is harder. Um it's harder it's easier to say this than it than it is to do it, but I think if it would be like if people were in engineering jobs in Silicon Valley and the company were like, "You're not allowed to use AI. you have to write this code. And what would anybody in Silicon Valley say to that person? They would say, you should quit that job.
53:40That is the dumbest job imaginable. You should quit that job. That is not the future. And so I think if you're a color grading person [laughter] and you're not allowed to like use AI to like build more and get more out of the company, you know, obvious, you know, you know, I'm not recommending you walk into the company and quit your job immediately, but you should definitely start looking.
53:59You should be looking for these companies that are trying to find their way to be more AI native because what will likely happen is if they have the right mindset, you're going to be doing way more than color grading and they're going to be doing your teammates are going to do more than whatever they were doing. And you're going to actually be you're as a group, you're going to be doing way more. The team itself will be scaling its ambition to doing, you know, 10 videos, you know, a week instead of five, you know, instead of one or trying to make a movie or whatever it is that they're doing.
54:25Scale the quality. Yeah. Yeah. Like you can always scale the quality. Yeah. And and that's that's the thing that I'm trying to do. Like I've figured out the quantity game.
54:31Um and it's it's very funny that we're talking about this now because uh starting next year I'm working with filmmakers in New York and I'm going to turn my Instagram into mini short films and I'm like you know what? I can do this. Like I need you know I was I I was got a little nervous because so many content creators are making content about AI on Instagram. It's like harder to stand out. So I'm like okay what can I do? And I'm like you know what like let's make short films. like it's always something that I've actually wanted to do and I have a following and so like that that's going to be how I kind of
55:01shoot my shot and yeah I I I don't know what'll happen but like I I am getting that urge that I need to shoot bigger. I don't know.
55:08Yeah. I mean the other reason to do it is because if you think video editing is decent now and it just kind of happened recently I assume. I mean I'm I haven't done use the models for videoing but I've kind of seen people use it. I mean just imagine what it's going to be like in a year. I mean it's I can't even I mean to put this into perspective like think about where co I I said that there was um there was 10 months between opus
55:354.1 and opus 5.5. I don't know if you remember opus 4.1 but like I remember my benchmark for opus 4.1 was flappy bird. Like I was like, "Okay, in how many prompts could I create a semi nondeformed version of Flappy Bird?" And it was like six or seven prompts on iOS.
55:56And now with Opus 5.5, in one prompt I had a working Call of Duty game that I played on stream on stream multiplayer 3D 10 months. And so like if you and and and so like with that same slope might be happening right now with with video editing. I think it is. Yeah, I think it is. And and it's happening, but it's it's just, you know, it's happening with robotics. That's why all the roboticists are freaked out because they just they've just watched
56:23someone get it use Claude or Astra or whatever and and control a robotic arm and like write a sentence or draw a drawing and they all flipped out because and the reason and it's always important when the experts in the industry freak out because they know because they know that if it can do that what's the call of duty you mean that was Flappy Bird.
56:45It's like a Flappy Bird moment in robot robotics. Like what is that going to be [laughter] in in a year? I mean, next year it's going to be crazy. Like next year, my prediction is that we're going to have you're going to be able to write you you never I mean, just to marry some of the things that we were talking about together, you were talking about everything is code. Well, the robot the robots are going to be are being controlled by code. You're going to like write a code that says move ARM to pick up glass of water and then the arm will just go and do it. And so I think that next year what we might see is even a
57:14proliferation of people getting into robotics that really had no business being in robotics. Like ordinarily they would have been like I can't do robotics. Well now you can. And because being a software engineer is dead. So that doesn't so writing code doesn't really matter. So to be a roboticist I don't know. It might be it might be more focused on like creativity and ideas and finding problems and doing interesting things with robots as opposed to all the things.
57:37Yeah. Like like Yeah. Like Yeah. like robots, you still need to build like I saw I saw on Twitter like a little robotic lamp and that like could like look around uh it like sat on your desk and you could talk to it and you could use any model that you wanted. I was like I want that. Um it was just like it reminded me of the Pixar lamp. Um yeah, and I think it's going to get really weird, but I mean everything that we've been talking about through this conversation, it's going to get really fun.
58:04I'm nothing but optimistic.
58:06Yeah, it's it's so much fun. Like you can you can make expansion, you know, just just the people watching this, I hope you can like see the expansion of opportunity, you know, like 99.99% of the world couldn't do anything with robots or video editing.
58:21Yeah, it it it's insane. And so, you can create your own, you know, funny enough, I've been talking to some creators recently who have like big followings on social media and they're like, and we we were talking, we're like, should we hire like a head of gaming? And because we were talking about like a lot of creating content is about world building, right? That's why you have like the similar fonts and you have your brand and a lot of people make PDFs. You know, Gary Vee has these purple and yellow PDFs that he sends out to people.
58:46That's what he uses for lead magnets.
58:48And like he used to have to have a big team to construct these decks and now he probably uses AI to construct these decks. I'm like, wait, we can create video games. Like why would people not start making games as lead magnets or like hosting an entire video game world that your followers can play? And like and so I'm starting to like realize that like scaling your ambition might just be like the absolute best thing to do because anyone with a following can kind of create like a little media empire, right? You can create films easier, you
59:17can create video games easier. Um there's so many other surface areas that you can that you can win in. And I think that's that's just where this is all going.
59:27Yep. 100% agree. Yeah.
59:29Yeah. Anyway, this has been amazing. I really appreciate you uh coming on the podcast. Is there any final thoughts?
59:36Any final thoughts or where where we can find you or uh what should we should try out with particle?
59:42Yeah, I mean we'd love for people to try out our new model particle one. Um we're already training the next one. I mean we we love people to use this. I mean the main the main thing we're trying to do is just also just sort of flatten things. We don't we're not very excited about there being kind of like two big frontier labs that kind of tax humanity on intelligence. And so to the degree that we can just like really redistribute this to as many people as possible so so that people can like scale their emission to all kinds of crazy things because right now the malls are very expensive. You know most people
1:00:12cannot afford a two two $200 max plants like you and me you know and so I just think yeah I think that's a big part of what we hope to do. So yeah try us out.
1:00:21Very cool. Yeah I use Verscell's AI gateway. Are you guys on the Verscell AI gateway? We're on Open Versel, Cloudflare.
1:00:30There's probably like five more. Yeah.
1:00:32Amazing. I will check it out right after this episode. I appreciate it.
1:00:35All right. Thank you so much, Riley.