0:00There's a new model from Google that claims to be the absolute best model on the planet, beating literally everything else, closed source, open- source, everything. And if this is true, it means Google is absolutely back. So, I'm going to go over the blog that they released. I'm going to show you all of the benchmarks, and we're going to decide, is this model actually as good as they're saying it is? And this video is brought to you by Factory. More on them later. Today, we're announcing the
0:27next Frontier model, Gemini 4 Carbon, our strongest model with state-of-the-art cyber security and long horizon capabilities. Now, before you get too excited, they have not released it publicly yet. And one thing that I actually haven't liked about Google's release process is they will tell you about the model. They'll show you the benchmarks, but they won't actually release the model until a couple weeks later, which is really weird. Usually when a Frontier Lab releases a model,
0:56they release all the information and access to the model usually the same day, if not within like 24 hours. So the fact that we're going to be sitting here waiting for this model is a little uncomfortable. Now, Artificial Analysis just published their newest index, which is index V5. Now, if you're not familiar with Artificial Analysis, they are a company that aggregates all of the benchmarks out there. They put it in one place. They do a bunch of their own
1:24benchmarks and they just released V5 of their index which is an aggregate of different benchmarks and carbon ranks number one. So we have Gemini 4 carbon high at 59 above what was previously the best model on the planet. My personal favorite model to use Opus 5.5. But that's not everything that makes carbon special. It is extremely efficient and it is less expensive overall than opus.
1:53So stick with me here. Okay. So carbon ranks number one on artificial analysis intelligence index. It shows significant gains across coding and agentic task as well as across key capabilities critical for cyber security defense landing a firm lead and SEC bench pro for finding vulnerabilities exploit gym. And if that sounds familiar, that's the benchmark that the OpenAI model basically broke out of and hacked Hugging Face for. For
2:22exploitation, CWE for patching. Now, because this model is so good at cyber capabilities, the roll out is going to be a little bit different than we're all used to. First, they're going to announce it and tell you about it, which again feels a little awkward, but fine.
2:39Then they're going to give it to their cyber security partners first. They're going to probably use it to patch software and look for vulnerabilities in existing software, patch those. Then they're going to release it for general availability. And yes, they are going to release it. They already said they were going to do that. So this isn't a model that's never going to come out. We just have to wait a little while. So on internal benchmarks, Carbon drives a strong uplift in generating secure code.
3:08And in live coding environments with our own teams, it shows reduced misalignment and fabrication rates. And so here's the crazy part. Gemini 4 Argon was launched 2 weeks ago. So one of two things is happening. Gemini 4 Argon was a checkpoint that was good enough to release and they were just continuing to build the model. Or maybe this is post-training at this point. We don't know the details, but it's interesting to see because within the span of two
3:35weeks, the improvement to the Gemini 4 family of models is substantial. And by the way, if you counted Google out, this is the time to reassess that opinion.
3:47So, Carbon illustrates the pace at which we're pushing the frontier forward. That is a direct shot to all the Google haters right now. They're just like, "Yeah, no, we're moving fast. You just haven't seen it yet." So, here's the roll out. We're taking a deliberate approach to rollout and are actively engaged in the US government's voluntary process for pre-release model testing.
4:06Starting this week, we will make carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through their fair wind program. We're continuing to gather feedback and iterate on guardrails before making carbon available to developers, businesses, and consumers via our product services as soon as possible.
4:27And it is very cheap. You are used to seeing models at this level be, you know, anywhere from 5 and 20 input output all the way up to 10 and 50 input output. But this $2 per million input tokens and $10 per million output tokens. Now, that is their introductory price. Let's see what that exactly means. After the introductory period, it goes to $4 per million input and $20 per million output. They don't say what the
4:55introductory period is, and there's always a chance. they just continue to offer it, especially if it's an incredibly powerful model and they're gaining a lot of traction with the developer community, they're probably willing to lose some money on it just to gather developer mind share. But both of those prices are very inexpensive. And of course, what we really want to look at is cost per task completed, which is a function of both the price of the
5:22tokens, so in this case, $2 and $10, and the intelligence density of the model, which means fewer tokens to solve the same task. And cached input tokens 95% off the input token price. And if you want to test all the greatest models, every single one of them, from any lab, go check out the sponsor of this video, Factory. I'm really excited to tell you about Factory because they literally invented the term AI factory. This is no
5:51longer about having your agent write parts of your code for you. This is really taking it to the next level of automating your entire development life cycle. Factory provides you with everything you need to do it. Starting with their droids, which is what they call their AI agents. You can use Factory through the CLI or desktop app, or even tag it in Slack. And for bigger projects, missions is one of its core features. Factory gives its droids different parts of the job. Then the
6:21validator droid actually clicks through the working application to make sure it all works, and if it doesn't, passes it back to the builders to continue iterating. They also have factory router which picks the right model as the work progresses. In production, factory router cut overall costs by 63% compared to using Frontier models and 84% of routed sessions saved at least half.
6:47Factory is already used by teams at Nvidia, Door Dash, and T-Mobile. Click the link down below for a chance to be selected to win a free month of the $200 max plan at factory.com/bman.
6:59Go try it out. Let them know I sent you and let me know what you think. Let's talk about the benchmarks because boy are they insane. Look at this. At the top we have Gemini 4 Carbon compared to, and this is where it really just gets nuts. Argon, which came out 2 weeks ago.
7:16We also have it compared against GPT6 Astra, Claude Opus 5.5, which is my current favorite model, and Claude Fable 5.1, which is also a very strong model.
7:27So we already saw the artificial analysis intelligence index scoring of 59. That is a big jump from previously 50. Astra's 57. Opus 5.5 is 58. SWI sweep which I actually haven't heard of this benchmark before but it's at 9.6 compared to 4.3 doubling the score of Argon right in line with Astra completely dominating Opus and Fable 5.1. Terminal Bench 4.0 know one of my
7:56favorite benchmarks to really understand how good a model is at a gentic coding because it measures the model's ability to write and execute commands in the terminal which is obviously very important for coding and it gets a 65 score as compared to 57 on the previous gen. So that's Argon 59 Astra 59 Opus 52 Fable. Here's Agents Last Exam coming in at 77.4 four just number one OS world
8:24the models ability to control a computer a very important skill now personally chat GPT models are the best at that not only controlling the computer but also browser use I just find GPT models to be just far superior than anything else but what it shows here interestingly enough is Opus 5.5 beats everybody pretty handily and then Carbon comes in second place at 81.5 Astra coming in at 80.9 9.
8:51Next, we have vending bench in which the model is given a vending machine and the task to maximize the revenue it makes from that vending machine. And here we go. Number one, the bank balance at the end of the benchmark, $16,000, beating previous number one, which was GPT6 Astra at $15,000, dominating Opus 5.5, dominating Fable 5.1, and coming in, you know, a few thousand more than
9:21Argon. For science and math, it's coming in number one. Reman Bench and Terminal Bench Science, both number one. And interestingly enough, in like the last few weeks, we've had both Anthropic and Open AAI releasing worldchanging math discoveries with their models. Now, the models that they're using for those math discoveries are not the same models that are publicly available. So, that is one thing to keep in mind. And Sam Alman even said OpenAI has internal models
9:51that are so good at math that they're probably not going to be releasing publicly. Then last we have its cyber capabilities in which it is beating all the other models. We have CWEB bench 71 exploit gym which is 54 compared to 42 for Astra which is kind of nuts. 34 for Opus which you know I guess kind of makes sense because I believe Anthropic actually defanged their models their
10:20public models in terms of their cyber capabilities. So I believe Google only has access to Anthropic's publicly available models which already they've reduced their cyber capabilities on. All right. So there's a few more interesting facts that I want to go over with you where Google is basically saying this is a more aligned model than ever. So first reducing shortcuts and loophole exploitation with carbon we further mitigated shortcut attempts in training while ensuring highquality coding performance. What they're talking about,
10:50I believe, is kind of like what happened with the hugging face hack. The model was tasked with scoring as high as it could on exploit gym. And rather than actually trying to score high, it escaped its containment and tried to hack to get the answers. And so, yes, that would be considered a shortcut. And so, it reduced successful cheating on Terminal Bench 4.0 to 0% and attempts to cheat down to 4%. all while improving
11:18overall accuracy. It also reduced its unsupported claims by approximately two times over Argon, which is great. And when it notices discrepancies in what the user is asking for versus what it thinks it needs to do instead of just guessing, it will ask. Now, as I said, the model is special because it is very efficient. And so, we're talking about cost per task completed. The first piece of that is how many tokens it takes to complete a task. And we see Claude
11:48Sonnet 5.5 way up here at 285,000 on the artificial analysis intelligence index.
11:55We see Haiku way up here. Here's Deepseek v4.1 Flash. Maybe one of the most surprisingly great models out there right now. And then we see Carbon here, 123K. So, what that means is to complete the same task, Gemini 4 Carbon takes half as many tokens as Sonnet 5.5 and Haiku 5.5 and about 20% less than Opus 5.5. And so, here's the cool thing. It takes fewer tokens and it's also less expensive. So, where is it going to end
12:24up on the Paro curve? It's going to fare extremely well. Let's take a look. So, this is what you're looking at.
12:30Intelligence index versus cost per task.
12:32On the x-axis, we have the cost per task and log scale. On the y-axis, we have the quality or the score. And so, where you want to be is in this top left quadrant. That means you are less expensive and higher quality than other models. And so, if we highlight Google, we can see Gemini 4 Carbon High, way up here, number one, but it is not less expensive than these over here. And I'm going to show you all these comparisons in a second. Now, what they need to do
13:01now that they have the best model is create a bunch of smaller, faster, more efficient models, which I guarantee they're working on right now. Right? So, not only do they want the absolute best, but they need every single point along the paro curve, which currently OpenAI holds almost exclusively. So, if we highlight the OpenAI models, look at this. This is where you want to be in the top left. The highest quality for the least amount. That is this quadrant
13:28right here. And every single GPT 6.1 version is basically paro optimal. So anything that was up here would have beat it. But that is the outside of this quadrant. And so OpenAI overall has the best product offering. I still think Opus 5.5 is the best overall model because I haven't used Carbon yet. Soul is a phenomenal model, but I still prefer Opus. But if you want to take a look, here's Opus right here. So, it is
13:56much more expensive than these soul variations. And if we go back to token usage, we can see why. All the way down here, GPT 6.1 soul max 68,000 tokens used per task. It's half of Gemini 4 carbon. So, right now, in terms of total intelligence density, the GPT 6.1 model is by far the best. Now, this model is coming soon, and I guarantee you it's going to continue to get cheaper, continue to get faster, and continue to
14:26get better. And I am more bullish than ever on Google right now. Thanks again to Factory for sponsoring this video.
14:31I'll drop links down below. Go check them out. And even though Carbon was very surprising, I still think the surprise of the year is Deepseek V4.1 Flash. I just made a video about it. Go check that out right here.