0:00Open AAI just released their two new flagship models, Astra and Soul 6.2, but it hasn't been all sunshine and rainbows. There's actually been a lot of controversy, both leading up to it and upon release with accusations of collusion amongst reviewers, myself included, to negatively review these models, which I'm going to get to in a little bit. And yes, there have been a flood of negative reviews about this model. And then as the day went on,
0:28there was actually a counternarrative that the models are actually really good. And so I've been involved in this whole thing. And so in this video, I'm going to tell you the story up until today. I'm going to explain what's been going on today. I'm going to show you some of the demos that I created and then I'm going to tell you what I really think about these new two models. So I've had these models for a while now.
0:48And I have spent hundreds of billions of tokens testing them thoroughly. And what I will say about Astra first is that it is a fine model, but that's about it.
1:01It's just fine. It doesn't really excel in any area with the exception of computer use, which still to this day Open AI is completely dominant at. But other than that, all of the categories that I usually test from building games to building applications, building 3D simulations of cities, all of this, it does well. It's fine. I mean, if you look at it in a vacuum, the model is good. But then you have to do the
1:29comparison because let's say you only have enough budget to pay one company.
1:34You're going to either pay Open AI or you're going to pay Anthropic. Now, putting aside whatever you think about Anthropic, the decision seems quite obvious. Opus and Fable are far superior than Astra and Soul right now. Even if that were the case just a few months ago, the subscription I would have recommended would still be OpenAI because the quota that they give you, the amount of tokens that they give you on a weekly basis was so much higher than what Anthropic would offer. But now
2:02at this point, the tables have turned.
2:05Anthropic has enough compute. They give a tremendous amount of value as part of their subscription. Opus 5.5 can run basically all week long nonstop and you still won't run out of quota at this point. I have to recommend the Anthropic subscription. But with that said, let me show you some demos. Now, before I dive into the demos, I have to say one thing.
2:29I was testing on a PC and I built some really cool demos and then that computer just died. And I know that sounds convenient, but I promise that is what happened. I was able to recover a couple demos, but most of them, and especially the one that I really wanted to show you all, which is the Gears of War Eday clone, did not make it, unfortunately.
2:52But with that, let me show you the first demo. So, one of the first things that I asked Astra to build was a 3D replica of Tokyo using Unreal Engine. And as I mentioned earlier, Astra, Soul, Chat GPT in general is just so good at computer use and browser control. And so this task falls squarely into what these models would be really good at. And it did, as I said before, a fine job. Now, in a vacuum, this demo looks really
3:21good. But if you compare it against Anthropic, you can see there are some distinct differences. Now, I'm going to play this demo as I explain things. So this took a bunch of different iterations. The cool thing is you can see every single building was built by hand by Astra and in that sense it's really good. Now it downloaded some assets like the billboards, the logo, the Coca-Cola logo here, the IKEA logo, some of these other logos here. It downloaded those assets certainly, but all of the buildings were created from
3:51scratch, but it didn't do that on the first go. In fact, what it would do is create a facade of a building and then download satellite imagery and basically try to map that satellite imagery onto the building. And at first glance, I was like, "Oh, wow. That looks really good."
4:08And then I kind of started to squint my eyes and look, and it did not look good.
4:13It had windows where they shouldn't be.
4:15It had doors where it shouldn't be. It was just awkward. And it took a number of different prompts and nudging to finally get the model to create these buildings by hand. And I think that's the beginning of where reviewers, early access folks like myself started to get frustrated with the model. It just didn't listen. It was like, "Hey, this seems so obvious. Do it this way." And the model just didn't. I don't know why.
4:40Now, with that said, if you look at the city, it's extremely impressive. We're talking about tens of thousands of buildings, all handcrafted. And if you want to test out Astra, you should do so in the sponsor of today's video, Spaceship. All right, I'm really excited to tell you about the sponsor of this video because it's a company started by my good friend and former colleague.
5:00This is called Spaceship. And by the way, my friend who built Spaceship also helped build artifacts at Anthropic.
5:07Spaceship is a multiplayer workspace for humans and agents. So, it works with the AI that you already use. You can ask Claude or Chad GBT or Gemini to create a document, a presentation, a spreadsheet, a piece of art, even an entire dashboard or app. Take this DNA model, turn it, and everyone watching sees it turn with you. Instead of everybody working in isolation, losing artifacts here and there, put it all in spaceship. Keep it all in one place. Keep it organized.
5:36Search through it easily. Collaborate with your other agents. Collaborate with your human teammates. And the first thousand viewers that click through and claim get Spaceheep for free for 3 months. Click the link down below, spaceship.dev/matt, to let him know I sent you or enter code bur. Redeem by December 13th. Now, the next thing I want to tell you about, unfortunately, I'm only going to be able to explain, but it is extremely impressive and really got me to like the
6:06model, even though a lot of other people don't. So, I've struggled with this problem for about 18 months, and maybe you noticed it, maybe you didn't, but me and my team definitely noticed it where occasionally during a video, we would have a frame rate drop. It was very rare, but when it happened, we noticed and it was just extremely frustrating. I had worked with a bunch of previous AI models to try to debug it and try to figure out what was going on and just never could. I replaced my camera. I
6:35replaced my HDMI cable. Literally replaced my computer and nothing worked.
6:40I had multiple human experts come in and look and nobody could figure it out. I pointed Astra at it and because it's really good at computer control, it started looking at everything. It looked at the camera settings. It looked at the HDMI feed coming in and then of course it opened up OBS and what it found after reading the OBS logs while I was recording a video live is the buffer setting was off even though it was set
7:10to auto and should have been turned on because it was dropping frames. OBS wasn't smart enough to actually enable it and so it figured out that it has to force it to be on and once it did that it solved it. This has been something we were battling for 18 months and Astra finally fixed it for us. Now, let me get to the controversy because this is actually something that I am unfortunately involved in. And so, I'm part of an early access testing program
7:38with OpenAI with a bunch of other people. We get early access to the model. We get to test it. We get to provide feedback. And I will say upfront, OpenAI is fantastic to work with. They don't pay us to do this, but they give us the models and we provide feedback and they are so open to the feedback even if it's hyperritical. Then yesterday, Astra and Soul get released and everybody that was part of the early access program posts their experience
8:07with the model and almost universally people had negative opinions of the model. I mean, Matt Schumer said he wasn't even going to review the model because it was so bad. Theo said he hated the model and basically every other person that I'm in this early access program with or the vast majority of them said the model's not that good.
8:26Now, I didn't have as many problems with this model as other people did. I still don't think it's a great model and especially when you compare it to what Anthropic just put out with Fable and Opus, it pales in comparison, but in isolation, it's a good model. So, here's Matt Schumer's post on X. I will not be writing a full review of GPT 6.2 Astra or Soul. I had a terrible experience testing them. They were essentially unusable. I do not recommend these
8:55models. Now, a bunch of people asked him, "Hey, dude, review the model. We don't care if it's bad. We have to know." And so, he ended up writing a review, but it was really negative.
9:04Then, Theo put out an article titled, "Why I hate GPT 6.2 Astra." Hyperbole aside, Theo obviously really didn't like the model. And if you read this article, he provides some pretty good details as to why. And hopefully through the demos I showed you, you can kind of see that I felt some of these problems, but I feel like after pushing the model a little bit more, it was able to get done what I needed it to get done. Here's Paul
9:32Sultt, another fellow early access program member. It's not good at communicating. It doesn't write good final summaries. that might finish work without explaining it's done. Then he goes on to say, "It's disappointing.
9:45They stop working early. They fixate on the wrong things. Summaries are not clear and writing is still not great.
9:52They are not good at iOS app UI and they take over computers and skip computer use. Interestingly, video editing skills are better, which I found. And the cost should improve usage limits. And that's another benefit of OpenAI is they are just generally cheaper than anthropic."
10:08Here's Dan Shipper. Unfortunately, these models aren't great. Overall, a disappointing release compared to Fable 5.2. So, the general sentiment around these two models was extremely negative.
10:19And then Tibo posted this. One of the funny reasons for the difference in vibes between the influencers, which the early access folks like myself and Theo and Dan and Paul, and the crowd today, so everybody who got general access on 6.2 to Soul Astra is that we gave a few folks access a while ago. So, this is true. We've had access to this for a very long time and then made a lot of changes in the last 3 to 4 days and I'm
10:48pretty sure no retesting was done within those days and the videos were produced last week. Don't get me wrong, while the models are a hefty jump compared to their GPT predecessors, they are by no means perfect. And there are a few things the teams are excited for the models to get better at in the future iterations in terms of taste and niceness of use. Love all the feedback.
11:12This, at least for me, is true. with the exception of video editing, which I've been doing a ton of, and I'm going to show you something amazing in a second that I've been building. I basically haven't been testing the model for anything else. And then when I built stuff with it, I would always recheck the work with Opus. So, I did lose a bit of trust in the model. But here's the thing. Generally, when there are new versions of a model mid early access program, the OpenAI team will tell us
11:41like, "Hey, we just made an update. Go test it again." They didn't say anything. In fact, I didn't even know they were going to release the model yesterday until I saw the post live on Twitter. And they do usually give us a heads up at least 24 hours in advance when they're going to release a model that we've been testing. And even Matt Schumer comments, disagree. We had access for weeks and only received a 10-minute heads up before the launch.
12:06So, the one thing it is definitively good at besides for computer use is video editing. Now, let me show you what I've been building. I've been working on an open- source AI video editing app.
12:16This is Scarlet and the model that has been powering all of the video editing for weeks in this app has been Astra and it has been fantastic. I have not really had many problems and I've been able to do so much testing and it just cuts dialogue really well. It has good taste when it comes to video editing. And if you edit videos at all or you're thinking about making content, give this a try. It's open source. I'll drop a link down below. It's early. It's a bit
12:45rough, but it's almost able to edit my videos end to end in one go. So, let me show off some demos from folks on X because some of these are actually really good. And I'm going to give Astra another chance. I'm going to give Soul another chance. Now, here's a post from Envald Dev Tony. We have the same prompt. Astra took 45 minutes, $9. GPT 6.2 2 soul 30 minutes $2. Fable 5.2 $200
13:15in 3 hours. Opus 5.5 almost 3 hours $31.
13:19So as I said before GPT models are much less expensive than what they're putting out in Enthropic. And if you look at the results, so we have Astra in the top left, Soul in the top right, Fable bottom left, Opus bottom right. If I'm just kind of looking at it and thinking which one I like best, Fable certainly looks the best where maybe GPT 6.2, Astra in the top left is comparable.
13:47Opus 5.5 looks pretty good, but certainly just looks alone. I'm looking at Fable and that's the best, followed by 6.2. But you can't really look at any of these and think they're bad cuz they're not. Here's a 3JS game. Looks like a horror game. And I mean, this looks fantastic. The reflections off of the water on the ground, the detail, everything looks really good. Here's a little Japanese temple. The lighting is fantastic. This is great. So, he's saying, "Hey, why is everybody talking
14:16so negatively about these models? They look really good to me." And that's the thing. I guess Tibo said, "Hey, we updated the model just a few days ago.
14:23You should try it all again." That is maybe a mistake. So, I think OpenAI could have probably handled the roll out a little bit better. And I think hopefully the model is now just better in general. Here's Mr. Tactical. He's comparing Fable 5.2 and Astra 6.2. On top we have Fable 5.2. On the bottom we have Astra. And I don't know, they look different, but I think they both look good. Astra looks a little bit more cartoonish where Fable looks more realistic. It has better weather, a more
14:52dynamic view, but still Astra is really good. Astra took an hour 21 minutes and was $5. Fable took an hour and a half and was nearly $60. So, if you really need those extra couple percentage points in quality and are willing to pay multiple times more to get there, maybe Fable is the better model for you.
15:12Here's a OneShot experiment with Astra by Thomas Ricard. Okay, looks like a Doom style game. Pretty good graphics, very smooth gameplay. Yeah, this is definitely just Doom. Yeah, I think this looks really good. Here's another one.
15:25Fable on top, Astra on the bottom. I actually have to give this one to Astra at first look. They both look fantastic.
15:33Claude Fable $81 and almost two hours.
15:36Astra $8 and only 40 minutes. So, you're starting to see this trend. And I think Astra is actually a good model. And if you're wondering how Anthropic keeps putting out the best models on the planet, it is almost certainly because recursive self-improvement has started.
15:54And this was proven by external researchers called meter. I made an entire video about this.