0:00Speaker A:GLM 5.2 is very, very good at writing GPU kernels. It was very funny internally. We had a GLM 5.2 endpoint that we were using that we plugged in in our cloud code harness. So every engineering team uses our GLM 5.2 and it will do a forward pass on the GLM 5.2 instance of the, you know, the node, and then it will get the profile trace and it will analyze it and it will find the kernels that are the bottlenecks in sglang, and then it will write the new kernels and then we'll do another profiling trace and when it's done, it uploads the image thing and then we can pull that image down and repeat the cycle.
0:34Speaker A:Some of the GPU kernels that we run GLM 5.2 within our inference engine is written by GLM 5.2 before we.
0:40Speaker B:Get into today's episode, I just have a small message for listeners. Thank you. We will not be able to bring you the AI, engineering, science and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way.
1:02Speaker B:But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you and it means absolutely everything to me and my team that works so hard to bring the inspace to you each and every week. If you do it, I promise you we'll never stop working to make the show even better.
1:23Speaker B:Now let's get into it. Okay. We're here in the studio with Philip, old friend from Inference Engineering, the book, as well as base 10. And everything that you've done, you and I have done before as far as Ali, welcome.
1:39Speaker B:Pleasure to meet Waterloo Intern.
1:41Speaker A:Waterloo intern, always.
1:42Speaker B:When did you get Waterloo Intern as a handle?
1:45Speaker A:I think the rebranding happened like mid March. When I saw it was open, I was like, I have to take it for grabs.
1:50Speaker C:The problem is that Ali is really good at his job. It is not going to be an intern much longer, so we have to figure out who's going to get the handle.
1:58Speaker A:I'll pass the torch over to the.
1:59Speaker B:Oh, okay. It can be like you just pass it to another Waterloo Grant.
2:02Speaker A:It's another boss Relinter. Not bad. Enter Enter.
2:05Speaker D:You gotta get an empty from Waterloo.
2:07Speaker A:Yeah, we've gotta get Waterloo. They have to follow.
2:10Speaker B:But it could come from base 10. So it's like whoever base 10 gets from Waterloo has the title of Waterloo.
2:15Speaker D:They have to pass you exactly halfway through. You either get it or you're out.
2:20Speaker B:You should also do like a big graduation ceremony where you change the handle.
2:23Speaker A:Just say like, I mean you guys.
2:26Speaker D:Are good at ceremonies clearly. You know, we had a nice launch of the book, very successful. But before we get into all that, I want to start off with a fun question for you. Okay.
2:35Speaker D:Your expert inference engineer. What happens when I send a long query, say 200,000 tokens into base 10's inference? What's the process of query through GPU model, routing, balancing, all that? What is all this stuff that we.
2:49Speaker C:Don't think about with a long query specifically? The first thing that I'm going to ask is have you sent me this query before or at least part of it? And I really hope you have because it's going to be a lot easier for me and a lot cheaper for you. So the first thing that we're going to look at is some kind of cache away of routing where we're going to see we probably have a number of instances, a number of replicas up serving whatever model you're hitting.
3:14Speaker C:We want to send this one to something with number one available pre fill workers and number two, ideally some cached input already there so that we can skip pre fill on at least part of these 200,000 tokens. If you're doing 200,000 tokens, it's probably coding or a multi tone agent or something where you would expect to have that cached. If you don't, we're going to have to send it to a pre fill worker. We've at least on certain models disaggregated pre fill and decode.
3:43Speaker C:So you're going to have one set of GPUs that's solely going to process the input, create that KV cache and get you your first token and then that's going to be passed over to a separate set of GPUs which is going to run decode, we're going to iteratively make those tokens. We're probably going to have some kind of speculator model in front of that. I'm going to assume that you're doing coding and, and because of that our speculator model, which assumes you doing coding is going to have a high draft token acceptance rate. If I'm wrong and you're asking me to summarize every Harry Potter book, it's going to be slower.
4:18Speaker C:And then we stream that output to you and account for it, charge you some number of couple of pennies and say, hey, would you like to send another one?
4:29Speaker B:Except base 10 doesn't charge by pennies.
4:32Speaker C:Well, yeah, I'm assuming that we're talking about the public model APIs. If, you know, setting up a dedicated deployment, then yeah, it's not pennies.
4:43Speaker B:Yeah. I mean one of the key differentiators when I was talking with BCENT initially was that actually people who want very, very high volume just need to rent by the box because then it's up to you to figure out how to saturate the box.
4:55Speaker A:And more often than not it's like way cheaper if you're pushing like millions of tokens per hour. If you just pay per hour instead of pay per token.
5:02Speaker C:Yeah, they do. I think that we've increasingly seen a lot of demand for the sort of paper token APIs just because everyone to try open models and then once they find a use case that's really sticky, then they move over to dedicated.
5:15Speaker D:Is there a best practice on when it's time to swap over?
5:18Speaker C:Couple reasons, yeah. Reliability. That's a big one. Right.
5:21Speaker A:Like if they have a very specific use case, they want you to train something specifically for them. Like they want their own spec deck, for instance, for their own traffic.
5:29Speaker B:SpecDec is speculative decoding.
5:30Speaker A:Speculative decoding, yeah, yeah, sorry. Basically, if you have a huge model, right? And so the model is going to be generating one token at a time, every single turn, every single forward pass. So we attach this little parasite, like this layer that goes on top of the model and this model just has to predict, it does three very fast auto regressive forward passes and it will predict three certain tokens.
5:53Speaker A:And then you do one forward stage over the entire original model in order to see if those predictions were correct or not, and then you accept them or you reject them. Now this draft model is traffic specific. So if you, like Philip said, if you're summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books. And I can guarantee you that I'm going to accept the three tokens every single time.
6:12Speaker A:And so with that case, I increase your decode speed. I wouldn't be able to provide this to you if you want to share an endpoint because I have no idea if you're doing Harry Potter, if you're doing coding, if you're doing English, we don't know also there's a thing in the book that mentioned that they really cared about a specific threshold. Chapter four, I think. Do you remember that?
6:30Speaker C:Yeah. The things that you can do is you can set a specific batch sizing, a specific parallelism strategy. If you're trying to optimize for like throughput versus latency, you can, you know, maybe, maybe a NVFP for quant doesn't pass your benchmarks and you want to run a model at higher precision, you could do that. There's just a bunch of reasons why you might want to have your own endpoint.
6:55Speaker C:And the biggest one of course, just being like you don't have to deal with someone else throwing 100 million of tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users.
7:05Speaker B:Yeah, I think one thing that is, that is a classic journey, you know, it's basically vivo is asking the what happens when you type Google into the browser tool calling? Is that just, you know, you're generating JSON or is there more complication beyond that?
7:22Speaker A:Certain customers that we have, they have their own post trained models and so they demand tool calling that's not just like, you know, parser file or go find the weather. It's something that's very specific and you have to do post training on this. And if the post training on the model is not good or if the quantization after the post training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling. But it doesn't require its own sandbox.
7:49Speaker A:It's not like it's going to use that tool calling to escape a sandbox or it doesn't have to be contained, it can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't close the end of the request in a very certain manner, you end up with a model that did the tool calling and the thinking. And so as a result of that, it didn't see the result and just hallucinated the result as it decoded.
8:17Speaker A:That seems to be the most challenging thing with tool calling. Not really the sandboxes for what?
8:21Speaker C:Yeah, that's a challenge on the training side. And then on the inference side there's work that you can do to scope the possible output. So we published this actually at this point close to two years ago. The solution to this problem, which is you basically make a state machine and you use that to constrain the output to a specific format.
8:48Speaker C:So this is the structured output problem. If you remember back in.
8:52Speaker B:Yeah, the specific grammars. GML had this thing.
8:55Speaker C:Yeah. So it's like the old school, like, make sure this is only JSON, return only JSON or my grandma's going to.
9:03Speaker B:Die type of thing. BNF grammar. At some point the OpenAI had released a thing that was like, yeah, if you want to constrain your output, write BNF grammar back as Nor in our.
9:12Speaker C:Inference system, it's just a specified output format and you get the guarantee that your output's going to be structured along that format. And so applying that to tool calls can like help cut down on. Obviously you can still call the long tool or call no tool. It doesn't solve the certainty problem, but it at least solves the output structuring problem within tool calls.
9:36Speaker B:And MCP is just another form of tool, right?
9:39Speaker C:Yeah, exactly.
9:40Speaker B:There's no special thing there.
9:41Speaker C:The thing I'm always like explaining to people is the LLM is actually not capable of doing anything. It's only capable of making suggestions of what to do. And then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs.
9:57Speaker D:Part of the fun stuff is, you know, this is solved outside of tool calling too. Like in an agent loop, if the output is not correct or you're right, like reasoning. Tool calling was done in the reasoning style. Just like, oh, I don't know what to do, let me just try again.
10:10Speaker D:And you know, it might get there after a few tries. And on your point of training, sometimes this is harder in smaller models. So you don't have the same exact quality output when you just swap from big model.
10:23Speaker B:Right. Yeah. I will say that before we. I think we need to go back to inference engineering proper.
10:28Speaker B:But I had expected that something would replace JSON because it's hard to stream JSON because JSON must be complete and you must have open and close brackets and everything. So it's hard to parse something or validate something while it's being streamed. So people invented all sorts of things that are like, I forget the name of some of these alternatives, but it's basically something like toml, something like YAML. But JSON seems to be dominant still.
10:55Speaker C:The JSON outputs aren't that long, Right. I guess you could have a long because tool calls also contain the arguments in Them and perhaps for a certain tool you might pass a very long argument. But my impression of the median tool call is that it's a relatively small number of tokens, right? So I would expect that speculators are generally fairly good at something as formatted as JSON and so you would have like a pretty fast decode step there and that the streaming wouldn't be as valuable.
11:25Speaker C:But maybe I'm wrong about that.
11:26Speaker A:You're also bounded by the software that the model is going to integrate with. If the software is built with JSON for the tool calls or if the company that. If your customer says that this is how our software works and our tools are interfaced with JSON, you can ask them to change their software and say, yeah, this is going to be better for the model, but with the web training shouldn't be that much of a a difference. Also more profitable if it outputs more tokens.
11:49Speaker B:Probably depends on your business model. It really depends. But I will say that as a writer with experiments a lot with AI generated output, I do try to move from text to JSON text, which is very long JSON. There's paragraphs in every field because I'm trying to structure it.
12:08Speaker B:I want you to first make factual statements, then make opinions, then make bullet point summaries, have dates, have entity references, have your sources for references. All these things anyway. So these are things that I think people who really, really experiment with structured output have to really care about. But let's sort of recurse up the stack a little bit.
12:28Speaker B:Before we started recording, you actually mentioned something which is really cool, which is that there's a lot of inference engineering that goes on when a new model provider releases a new model. Right? So let's call it GLM 5.2. Kimik 3.
12:41Speaker B:I had previously assumed, especially if it's like, well, GLM 5 to 5.1 to 5.2, like, you know, that you've supported them before. Is it that much work?
12:51Speaker A:It's a lot of work, yeah.
12:53Speaker B:Okay, so like, you know, a lot of people, all you guys, right? Whenever a new model launch, like, people rush to say like, oh, hugging face supports this, Fireworks supports this. Base 10 supports this. And I'm like, yeah, of course they support it.
13:02Speaker B:But what goes into that?
13:05Speaker D:I think it's more than just supported too, right. It benefits the consumer a lot. Like I think it was with Kimike 2.5 or GLM 5.2, the latest. There was sort of an inference war, right?
13:16Speaker D:X provider is at 90 tokens a second the next day we're at 150.
13:19Speaker C:I kind of kicked that off with GLM 5.2. I wrote a Twitter article about it. Got like half a million views based.
13:27Speaker A:On being number one. Yeah, artificial analysis.
13:30Speaker C:Which got everyone really excited about, hey, how can we, you know, benchmarks a little bit further. And there's a difference between support the model as in like I can make a token out of this model and support a model as in I have a production ready API from this model. Getting to the point of I can make a token out of this model is not that hard because generally the open source inference engines, your VLMs, SGLANGs of the world, oftentimes even receive waits ahead of time. Maintainers do, or the people making the model merge PRs to ensure support.
14:12Speaker C:So you generally can just kind of get it working on the standard open source stack without too much pain in most cases. The challenge is every inference company is going to have our own proprietary stack, some open source components, some in house stuff, and for any arbitrary model there's going to be some new stuff. Sometimes you get lucky. Like K25 to 26 was like pretty similar.
14:41Speaker B:Yeah, it was pure continued post training, if I remember correctly.
14:44Speaker C:Even in those cases there's still stuff you have to do. You have to redo the quantization work you're taking the model from. Generally these models are not released in NVFP4 and we want them to be in NVFP4 for maximum Blackwell compatibility. So we have to perform that quantization and calibrate the quantization to make sure that we're not causing any kind of regression in the model's intelligence.
15:08Speaker C:And then we also have to train the speculator as we've talked about. Generally we have, Obviously we have ZDR0 data retention on our model APIs, so we don't know exactly the traffic that people are sending us, but we know what's popular. We know that coding use cases are popular, we know that agentic use cases are popular. So we can get public data sets that are representative of that kind of traffic and train general speculators.
15:33Speaker C:Now, with speculators today, you need to train the speculator using the base model itself because you're getting hidden states out of the model from running inference on these specific prompts. And that is the training data you use to create the speculator. So there's that process which you need the real model weights for, and then there's of course just the process of standing up all the infrastructure behind it, loading all this stuff in testing it. And then when there's a new model with a newer architecture, I think that like, obviously the Deep SEQ models tend to be the most challenging as they have like the most novel architectural stuff going on model over model.
16:13Speaker C:But every new model has something. I mean, Kimike 2 had, or sorry, GLM 5.2 had spars. Yeah, the DSA, which is brought from Deepseek. Yeah, yeah.
16:22Speaker C:And you know, so you can copy paste into it. I don't know how this works, you know, so like we had to like build support for that into our runtime. And you're right. It actually is really interesting the way that all of these open source labs borrow from each other.
16:37Speaker C:For example, GLM 5.2 doesn't have vision. So something that Hayley, a guy on our team, if we could take a look at this, he kind of grafted the Kimi Vision encoder onto GLM 5.2.
16:51Speaker A:Or retraining the projector.
16:53Speaker C:Exactly. So if you think about the encoder, there's the encoder, which is the part that looks at the image and turns it into latent information. And then there's the projector, which kind.
17:02Speaker B:Of like it's okay.
17:06Speaker C:And then there's the projector that maps it onto the model itself. And then there's the model weights. You don't want to mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision. So instead Harry started with just a projector which is only a handful of millions of parameters.
17:27Speaker A:And can you show the training, like the way it grox is very, very unexpected.
17:32Speaker C:Maybe Ali, you should take it from here. You've got a better understanding than I did.
17:36Speaker A:Yeah. You can see like the way he trained this is really, really cool. At the beginning he was training it using just like, here's a picture of a mountain. Can you describe what's in this mountain?
17:44Speaker A:And that calls it just like the first learning walls. Like here you can see this. All we're trying to teach it is to translate the encoded. Like it's already taken the encoder from Kmike.
17:53Speaker A:It's taken the image?
17:55Speaker B:Yeah. Frozen. Frozen.
17:56Speaker B:Frozen with adapter.
17:57Speaker A:So the understanding, the brain is frozen and the eyes are frozen. It's just we're trying to interconnect between the eye and the brain. Right. So the projector and so you take the tokens and then he's like, oh, can you describe what's in this image?
18:07Speaker A:And he's like, oh, it's a mountain or it's a person or it's a human, whatever the case is. But that didn't cause complete understanding, so he changed it such that every image was associated with a data set of questions like does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it?
18:23Speaker A:All that stuff. When it would have to answer questions correctly and using not just training on describing an image, but being able to answer question, answer, question, answer, question, answer. Over time you can see the grokking, which is genuinely insane, that retrofitting vision into a large LLM can learn to that extent. And even for images that it doesn't perform well on.
18:42Speaker A:For instance, if you ask it a picture of Stephen Hawking, who is this? Maybe it doesn't get it, but it will say something like, this is Albert Einstein, it still understands, is a scientist who is a man who has, you know, done significant achievements, all that stuff. So that's like really, really cool.
18:56Speaker B:Yeah. And so we've covered Haotien before, who was the author of the Lava paper that did this a while ago. And I think that's very foundational work for anyone who hasn't done vision work before.
19:06Speaker D:Same with Clip and Meta Clip, where you go from just captioning to building out questions off the image and how much better you can get performance.
19:14Speaker A:Right, right, right. Yeah.
19:16Speaker C:But what's, what's so exciting about this is if you look at a model like this, now obviously this is a little bit more of a research project. It's not, you know, it got to on MMLU Pro, I think so. Not quite funtio, but if you're running this model, you haven't suffered any loss on your GLM 5.2 quality. If you don't have an image, it'll just behave exactly the way it used to.
19:38Speaker B:And which in the inference code, you literally do not include the other part. Right?
19:43Speaker C:Yeah. I mean you would just skip the encoder if you don't have an image input. Okay, just confirm it.
19:48Speaker D:Does it affect a lot on the overall inference side? Like you're not adding much, you're adding a very small vision encoder. These are typically like less than a billion.
19:57Speaker C:Yeah, it's. I mean, there's a little bit less standardization among vision encoders, so the sort of support matrix can be a little bit sparser. But overall, yeah, it's a pretty minor component of the overall system. And ultimately what you get out of the system is all of a Sudden you have Kimi vision, GLM weights and deep seq attention all in one model.
20:20Speaker C:And that's, I think a lot of the power and beauty of open source is that you can take all of these different components and comb them together into a system that's better than anyone can be individually.
20:30Speaker B:People used to say that you would also do Franklin merges where you would take like layers from each model. Does anyone do that anymore?
20:38Speaker A:Well, to your point previously, when you were mentioning the work that goes into supporting a model when it first comes out, like GLM 5.2 or Minimax M3 or whatever the case is, sometimes you do have to like, you do have to switch out some things. Like for instance, the minimax M3 had uses full attention and with full attention you end up with this insane bottleneck inspector because you're doing autoregressive token generation for three tokens and you're doing this O over all of the tokens that are in your sequence. Your KV cache is very large because it's not sparse, it's not top K. So we find it better to like, okay, we're going to replace this, we're going to replace this layer with a layer from another model that's using GQX for instance. And then just with the right training, you can get it to have the same acceptance rate.
21:23Speaker A:So it is very possible to retrofit layers from other models and very much need it. Actually, if a layer is inefficient, the training just becomes the challenge. How do you ensure that you train it properly? Which again to earlier point is the mesh between training and inference.
21:40Speaker A:As in you need very good training in order to do fast inference. That's I feel like more and more becoming true.
21:46Speaker B:Anything else on the support side when you say get it to fully production ready.
21:51Speaker C:Yeah, I think that there's also a question of just. We can test a model to a pretty extensive degree, but we're trying to get it out quickly and then you see a bunch of other people test it and you get interesting results. There was an issue with GLM briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures. Once you expose an endpoint to the real world, there's going to be so many more varieties of things given to it that you're able to discover and patch things.
22:32Speaker C:So it's not just a day zero process, it's then for the first week, for the first month, if a model remains popular, how do you both fix bugs and then continue to push the envelope on performance?
22:45Speaker A:What do you mean you don't want your model outputting sss?
22:49Speaker B:Is there loop detection on that stuff? By the way? It still happens quite a lot, which is surprising.
22:54Speaker A:We have in our endpoint, if a model was to output the same exact token four plus times, we just call the generation, we say oh sorry, try again or we will reprocess the request because we know then it's four times the same token. It's probably collapsed.
23:10Speaker B:Yeah. Is there a way to opt out in case I really actually want that you're actually.
23:15Speaker A:I think there's a way that we have to handle it. I'm not exactly certain. I feel like in certain models like when they output something like you can imagine, like a table for instance, and so they want to draw like 12 dashes and 12 dashes. Yeah, I think there's a way for that to happen.
23:28Speaker A:I think we only do it on certain tokens. Like we exclude certain special characters. So we only do it on certain. S is the Most common almost JLM52 and I think it was DSV4 as well.
23:39Speaker A:Like you touched have like looping issues where like.
23:42Speaker B:Yeah, is there special something special about S?
23:45Speaker A:No, just seems to be the one token.
23:48Speaker B:Yeah. And it's only temperature zero or even.
23:52Speaker A:At other point nine or whatever, it will still collapse.
23:54Speaker B:That's weird, right?
23:55Speaker A:It's an inference problem, to be honest. Like a software problem. Like oftentimes the image you run like in video will release an image for instance and if we will upstream the changes from their latest LLM image into our stack, we'll find that it fixes it. Or oftentimes this will only happen in an inference engine that you're using like sglang.
24:12Speaker A:But if you were to switch to vlm, that isn't the case. So it seems to be like an extremely non deterministic kind of software issue and not really a model issue. It's not like a weights problem. Like we'll say oh, it's a problem with the quant.
24:23Speaker A:We did PTQ wrong. Right. But that doesn't make sense because the same exact weights used with a different inference engine does not repeat the problem. And sometimes it's the kernels that are being used in the backend have like these very subtle sometimes race conditions where if you were to use this model hosted on one cluster you will never get this problem.
24:43Speaker B:Oh my God.
24:44Speaker A:But if you host it on a different cluster you will. And the reason is the kvcache Transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster. So that exposes the race, whereas in another cluster it doesn't. So then you end up just like, okay, this model is not going to be hosted on this cluster or going to host it on another cluster because that cluster exposes that problem.
25:03Speaker A:But then it ends up with like, okay, is it the software, is it the model weights or is it the hardware?
25:07Speaker D:There is a thing about this with temperature zero still not being deterministic, right? Let's say because of hardware, even at temperature zero, same model, you won't always get the same output.
25:16Speaker B:But I'm surprised by the race condition one, because I thought Pytorch was a graph that guarantees that you at least just execute things in the right order.
25:26Speaker A:Well, I guess I'm not saying that this. Well, you have things like PGL optimizations where you can start a kernel before the end of the previous kernel. And that's like you want to do that. Exactly, but you don't do it cleanly.
25:39Speaker A:You overlap a little bit of the execution. No, I guess it is very possible that the kernel itself, like that one block that is supposed to be running in this instance of time, that kernel itself has a race condition, for instance, like a missing barrier. Like often if you're designing a kernel and you want it to be very fast. If you don't test it extensively, you'll have certain threads access data points from registers before they've been written to by other threads, for example, because your barrier is wrong or your synchronization is wrong.
26:03Speaker A:But yeah, the testing itself is very, very difficult in those.
26:07Speaker B:And there's no borrow checker for Rust. If you're trying to have memory safety, it sounds like a comparable problem.
26:16Speaker A:Well, yes, but you're working in Cuda, right? Nvidia gpu. It's like you just need a higher.
26:20Speaker B:Level language like modular. Maybe that's what modular is supposed to do.
26:24Speaker D:I don't know. How do you see keeping quality of the model? So you talked about all these steps of okay, you got to do quantization, train your own speculative decoder, run on different hardware, looking at other model providers. Okay, you kicked off an inference speed race on the consumer end.
26:42Speaker D:What goes into keeping quality the same across them? Right. Sure, you can run benchmarks, but like, how do you determine how much quantization are the standards? What actually goes into.
26:52Speaker C:There's a few things on quality. Most inference optimizations are lossless. KV caching for example, you are just preventing, recomputing the same values. Speculation.
27:04Speaker C:Of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization, and that really comes down to, number one, data format, number two, which parts of the model you choose to quantize, which layers, and number three, doing a lot of calibration on the quantized weights to ensure that you're sort of preserving all the outliers. There's other sort of tricks that you can do, though. A big one is long context.
27:35Speaker C:Because one thing you asked right at the beginning is, oh, what's going to happen if I send a 200,000 token request in? So obviously with a long input sequence, you need to store a lot more information, you need to process a lot more tokens. And so even if a model has a context of a certain length, you might, as an influence provider, choose to build an API with a shorter context length and of course a full length one as well. Because if someone doesn't need the full million token context, for example, you can get them better performance.
28:13Speaker C:I don't know if that's exactly quality of the model. The way that I think about quality is to what degree are we faithfully serving the original model. If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as, how close are we getting to that 100% fidelity of the model? You can also, of course, think about quality from the training side and how do you push yourself past 100%?
28:43Speaker C:But when I think about purely infinity optimizations, it's getting faster while staying as close to that 100% fidelity mark as possible. And certainly our standard internally is that you should not be able to tell the difference between our API and a sort of official API. I think Kimi in particular does a good job of vendor benchmarking here, where.
29:06Speaker B:They released an actual vendor benchmarking.
29:08Speaker B:Because they accused some people, Amazon, there was some provider that was not doing very well on Kimi's.
29:15Speaker A:So it would reflect.
29:17Speaker D:This was a long time ago, right?
29:18Speaker B:No, like three, four, five months ago.
29:21Speaker D:This also happened with. I don't remember which model, but they pulled out quite a few and then they started a whole chart about this.
29:27Speaker B:It might have been Kimi, vendor verifier.
29:30Speaker A:Because you'd be pissed, right? If I'm a consumer and I'm using Amazon's endpoint, for instance, and I use Kimmy and I'm like, oh my God, this is bad. I'm not going to say oh Amazon quantized the model in a bad way. I'm going to say oh Kimmy sucks.
29:42Speaker A:Right. So it seems like that.
29:43Speaker B:Yeah. Take care. Take care.
29:45Speaker D:Justifiably.
29:47Speaker B:This is probably a stupid question, but just checking. Has anything improved from being quantization? Like is quantization always strictly words?
29:55Speaker A:Well technically it's a lossy quantization. It's a lossy implementation.
30:00Speaker D:Speed improves.
30:01Speaker A:It speed improves obviously always look for inverse scaling laws.
30:05Speaker B:This is something I learned from Noam Brown where like things that normally act in one direction, sometimes they.
30:10Speaker C:Well technically when you run a benchmark because these models are non deterministic sometimes your you know, NVF before quant is like you know, two basis points higher than your. Exactly. Yeah, it's, it's, it's within. That's why I always say within margin of error.
30:26Speaker C:And I actually stop saying that because everyone assumes that what I mean is well within some margin of error. We're barely inside of that to the worst. So we're saying but yeah sometimes it's just like gives you a higher output score but like Ali said, that's noise. To my knowledge you're not necessarily making the results better.
30:45Speaker C:You're just trying to again keep your fidelity as close to 100% to the original model.
30:51Speaker A:There is to your point research that we did on mp. I don't know if you are able to pull a tweet. We did one of our research interns, Joshua, I think it's a tweet on how we have 20% better quantized JLM52 than Nvidia. Essentially what we found throughout this two month research is okay, quantization is a loss.
31:14Speaker A:You're compressing the data from occupying 16 bits to occupying 4 bits for instance. And so you're obviously losing some information and you're trying to minimize that. And so when I say that I'm going to quantize the model, my job becomes how do I find the layers that I can quantize and how to find the layers to not for instance with image models I don't quantize modulation layers and I don't quantize out projections because those two are like out projection is what you see as the user modulation is what the model sees or understands. Right, exactly.
31:42Speaker A:And so I guess to his paper do you have the. I guess doesn't have the. Yeah, it's a long paper. I don't know if I can find.
31:50Speaker D:If there's a part to search or it's probably in the thread.
31:52Speaker A:It's probably in the thread. But basically the long, the short is it is very possible that quantizing more of the model makes the results. Like if I have a model that I quantize layers 1, 5 and 10 and another model where I only quantize layers 1 and 2, it is possible that the model in which I quantized more information is going to perform better because the quantization errors have cancelled out. And so what Joshua showed in his mathematical proof where he had like a verify run, is that you can predict which layers are going to have quantization errors that will cancel out with each other and you choose to quantize those layers.
32:26Speaker A:And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider. So you get 20% more throughput of it because there's more layers running in NBFE 4 and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out like one layer skew to the right, one layer skew to the left, one layer skew to the right, your final logits distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL diversion. So instead of just scoring on the benchmarks, we scored the KL divergence between the logit distribution of the quantized model and the logit distribution of the original full precision model.
33:00Speaker A:And we show that with this technique we get if your probability distribution on the logit switch token it wants to select is more of the same as the original model, you're probably going to end up staying true to the original model. So yeah, so it seems like previously before this, it seemed like the industry was, well, the more you quantize, the worse it's going to be because the more loss you introduce. That's not exactly, not necessarily true. So yeah, it doesn't improve it, but can cancel it.
33:22Speaker D:I think it might be this, but reminds me a good bit about pruning actually where you can prune off certain layers, but. But very interesting. Didn't know this was a whole paper you guys put out.
33:31Speaker A:It's a fun fact. It was originally 72 pages, this paper, and then we decided we can't, we couldn't release it. So it's now 465, still 39 pages.
33:41Speaker B:Very substantive. We talked about evals and all these things and what's possible in terms of speed up I guess it's probably the number one thing that people do want to care about. And it's something that you wrote about in your post. Like, official API is 70 tokens per second and you push it up to 90.
33:58Speaker B:Is that like a normal thing?
34:01Speaker C:So what's cool about working in inference? The reason that I think inference is going to be a useful place to do engineering for a long time is that if you look at highly optimized domains like say, finance, if you're in finance, you measure how much better you got in basis points, it's like, oh, I got five basis points better, like 1/20 of 1% better. That's huge news because everything is so optimized when we publish optimizations. It's 20%, it's 100%, it's 200%.
34:33Speaker C:So there's still probably a lot further to go. Honestly, you'll know that inference is pretty much solved when researchers start publishing about how they got 1% faster at something.
34:44Speaker B:Which, by the way, because I am from the finance background in the 70s, that was the margin. At the time when you did quantitative finance research, you would find and look at the 20%. Tens of percent. Yes.
34:54Speaker C:Yeah. And now it's tiny.
34:57Speaker B:For those people interested, look up Andrew Low's paper. He had a really interesting illustration of quant stat arb distribution narrowing down from like those kinds of 20% differences in the 70s down to nothing today, which is very cool.
35:12Speaker C:Exactly. And what, the beginning of the same type of thing. Now benchmarking is hard, I think anyone will tell you that. And benchmarking provider speeds is hard because there's so many variables that go into it.
35:27Speaker C:What hardware are you using? How much load do you have on the system? What's the exact nature of the prompts and input and output sequence links, all that kind of stuff. But overall, when you start stacking these improvements, you're looking at multiples.
35:43Speaker C:You can look at it. The most common form, of course, is TPS tokens per second, which is. Which is bad naming by us in the industry because there's actually two tokens per second. There's tokens per second, the throughput number and the latency number.
35:57Speaker C:Like total tokens per second out of the GPU as a throughput number. Most people only care about tokens per second as the latency number, which we should call ITL into token latency, but we don't. Anyway, so you can imagine a sort of standard API without many optimizations. For 1 trillion parameter model operating somewhere in the 30 to 50 tokens per second range for a reasonable traffic profile.
36:29Speaker C:And we generally see the goal of pushing to 10x that. Not necessarily day zero, but by stacking enough optimizations. If you have say like four optimizations each of which doubles performance, or sorry, three optimizations each of which doubles performance, then you stack that up, that's an 8x gain. That's kind of the order of magnitude that we're working with in this space.
36:58Speaker C:We're trying to make things substantially faster, not just go from like 70 to 90.
37:03Speaker B:Are you saying you have done that?
37:05Speaker C:So let's say you have as a reasonable baseline 30 or 40 tokens per second, you can achieve 10x that. So like on GLM 5.2, if you're running it unquantized, perhaps on hoppers even, and you're just using an off the shelf inference engine with no particular optimizations, no, no speculator, nothing, nothing extra around like KV routing, no disaggregation, you're probably looking at that like 30 to 40. Do you think that's like a reasonable baseline?
37:36Speaker A:Right, right.
37:37Speaker C:To get to something like 10x, there's a lot of trade offs that you're making. If we're running at sort of more like a 300, 400 tokens per second range, obviously you are using the best hardware possible. You have a optimized speculator, you have done all of your quantization work. You are seeing a pretty high cache hit rate.
38:02Speaker C:You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput. But it is possible. So the spreads that you see if you go on artificial analysis or you go on open router and you look at the worst provider to the best provider, oftentimes can hit that kind of range. 10X is of course very aggressive.
38:29Speaker C:It's oftentimes maybe more of a four to six times improvement. But that's the kind of performance that makes us really excited is when we can get, get these huge gains not just go from 70 to 90 tokens.
38:43Speaker A:It's also like hardware dependent. Like if you, if you obviously have a thing where you're serving it on just like a node of H1 hundreds and then you throw like, you know, you shard the model across like four nodes of B2 hundreds. Like, like you can definitely increase the speed with just throwing more hardware at it, like normalizing for the same exact hardware and the same number of GPUs.
38:59Speaker C:Yeah. Then you're looking at like a 2 to 4x improvement depending on the inference optimizations. So yeah, it's. Some of it's, you know, what's the car and some of it's who's the driver.
39:10Speaker D:If you break down the 2 to 4x say, say the example is run glm5 2 on b2 hundreds single node. Right. What's like the cost trade off for effort to get like the last bit of juice out versus what should people just think of? Right.
39:26Speaker A:Spectac quantization. Spectac quantization, that's like 95%.
39:31Speaker D:And how far does that get you and how easy is that for the average person to do so? Say right now I want to throw the weights of GLM 5.2 on a Node B2 hundreds. How easy is it to find speculative decoder model or already quantized model? How much work goes into it?
39:48Speaker C:If you're doing it up front, it's quite a lot of work. If you're doing it today, there's going to be people who have published things that you can just, you can just grab some NVFP4 weights, you can grab a speculator. Yeah. If we're thinking about like what are the 2xs we're stacking going from BF16 to NVF before it's not quite a 2x, right.
40:08Speaker C:It's like, I think it's about like 30 to 40% from 16 to 8 and then another 30 to 40% multiplied from 8 to 4. So that doesn't quite get you a 2x but like roughly a 2x speculate a roughly a 2x disag on top of that, if you're able to get enough hardware and put enough traffic through it, another roughly a 2x and then you add in some double digit percent increase from having just a better runtime with the latest kernels and stuff behind it. And that's kind of how it stacks up. So building each of those, building the quantized weights is for someone who really knows what they're doing.
40:52Speaker C:Hours to days of work. Building the speculator again like hours to days of work and the disag setup hours to days. Well, okay, once you have it. Yeah, getting disag working for the first time, I'm saying of course is very difficult.
41:08Speaker C:But the marginal implementation is if you're.
41:13Speaker A:Just grabbing like if you are a person, just a normal consumer who has access to a node of B2 hundreds and you're wondering how can I just host it myself? You don't need to quantize the model yourself. There's always going to be an open source quantized checkpoint Nvidia is going to push one out if no one else does. Usually the providers will have their own spec deck that they've trained as well.
41:31Speaker A:You don't need to train your own spec deck, you can just use that as well.
41:34Speaker C:Like Kimmy, GLM 5.2 has its own.
41:37Speaker B:MTP multi token prediction. Yes, I can do it for you in case I get it wrong.
41:45Speaker D:Actually you should correct if we're wrong. But their multi token prediction can be used for self speculative decoding.
41:51Speaker A:I'm actually not sure.
41:53Speaker D:I'm semi confident in the but someone can check. But it's useful to paint the story of okay, not just the average person but say a company wants to switch from serverless inference to I want to throw this up on. I want to rent some GPUs, throw it up. These are the steps you take to do significantly faster than just put it behind Vllme.
42:13Speaker B:I was waiting for a mention of Dynamo. I feel like that's supposed to be the baseline that you measure against.
42:19Speaker C:I would think of Dynamo as less of a sort of out of box system and more of a toolkit for building with. So when we talk about doing KV aware routing, when we talk about doing KV offloading, when we talk about doing PD disaggregation, Dynamo fundamentally is. By the way, Dynamo is an open source library from Nvidia.
42:42Speaker B:We've done a part with code.
42:44Speaker C:Cool. So then your listeners know then that it supports all the different inference frameworks and it actually is kind of multi hardware which is interesting, but it's just the router.
42:54Speaker A:It's not like an optimizer layer.
42:55Speaker C:Yeah, all it does like what Dynamo is good at, it is a library for moving information around your cluster, around your hardware. So if you have, you know, KV cache on one place and you need it to be somewhere else, Dynamo coordinates nixle for you to move that around. That doesn't mean that like out of the box you just say, you know, pip, install Dynamo and then you get like a massive performance speed up. It's more of a developer toolkit.
43:26Speaker B:Yeah, I would have said it would. It comes with a set of defaults that you can then swap out.
43:31Speaker C:It does. If the industry at large I think was like rolling out all of these deployments standard, then I think it would be like a credible baseline. But we've got a benchmark against what we're seeing in the wild.
43:47Speaker B:I did want to talk a little bit more about PD disag because that's probably like number three after quantized and speculative decoding. In your book though, I was just going to pull out the book like section 522 on Medusa523 on eagle, 524 or on index, it's five.
44:03Speaker C:Five would be, would be disaggregation.
44:06Speaker B:Well, no, I just wanted to dwell a little bit on the other like so what do you choose to include? What do you choose to not to include? Because there was all these other techniques, I guess.
44:16Speaker B:Are these still relevant? Because I think they came out like a year and a half ago maybe.
44:19Speaker D:Medusa is quite old.
44:21Speaker C:Yeah, Medusa's old.
44:22Speaker D:But is it in the book? As a good. Here's baseline.
44:26Speaker B:Like you should know this.
44:28Speaker D:Like I read the paper, I'm like, ah, it makes so much sense.
44:30Speaker C:So with the book I had a couple goals. One was to give people just a working vocabulary for the space as a whole and the other was to give them some intuition about how each of these techniques works. As I mentioned in my AI engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else. So yeah, even at the time that I wrote the book Medusa I very much included as a way for people to understand how the, how the space evolved rather than what the most modern technique is.
45:07Speaker C:And now of course there's D Flash, D Spark, there's newer techniques even than Eagle, although Eagle is still very commonly used.
45:16Speaker C:Yes. Speculative. Speculative decoding.
45:19Speaker B:What can you.
45:20Speaker A:It's a paper by Tridao and it's like. It's basically doing speculative decoding for the speculative decoder.
45:27Speaker B:Oh my God.
45:27Speaker A:It's literally just another. It's like. Yeah, it's not simple. Explain it.
45:30Speaker A:And it seems like he got non trivial speed ups there. But it seems that the complexity with training, it's almost like in our mind at least, it's almost as complex as training Gans. It's like a very delicate balance and oftentimes it's just additional. But yeah, it's literally speculative decoding on speculative, speculative, speculative.
45:47Speaker A:We saw this paper.
45:48Speaker D:It's interesting, right? I wouldn't even expect it to be very particular to train. The naive part of me is like okay, train speculative decoder.
45:57Speaker A:It makes sense. The whole idea of speculative decoding is almost like the iPhone autopilot decoded version but for a normal model, right? Like you're just generating three tokens and you're like okay, do pre fill on them. And so you saved those three turns for your original model.
46:11Speaker A:Now your speculative decoder is doing three turns of auto regression. So why not just have an even smaller model?
46:17Speaker D:I guess the other question there is what are the size of speculators? So say for glm, right?
46:23Speaker C:It's like a billion parameters.
46:26Speaker A:Like for minimax. Yeah, yeah. It's like one layer. It's like 1/60 of the original model usually.
46:30Speaker C:Yeah. Actually, I think we should do a paper when we get back to the office. Speculative, speculative, speculative coding.
46:37Speaker A:No, it does seem like when do you stop? But then it also seems like, kind of like if you're able to train spec spec decode, for instance, Right? Like if you're able to have a small model that is accurately predicts what the intermediate speculator is going to predict, that is able to predict what the original target model is going to predict, then why not just use that smallest model directly? Right?
46:58Speaker B:Yeah. This is adjacent to the routing problem, right?
47:01Speaker A:Yeah. Right.
47:02Speaker C:The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on. There is a orchestration and resource competition problem inherent in that. And that is one of the sort of constraints on speculation in general is that draft tokens cost resources to create and cost software complexity to manage. And so if you have sort of like infinitely recursive speculators, you add in quite a bit of that complexity on the actual implementation within the inference engine as well, not just in the training process.
47:42Speaker D:I was going to say, I would wonder if you could do similar distillation and pruning of. You know, it's the same thing. It's just a model. Can we not just.
47:49Speaker D:There's still a lot of the way to quantize the speculator, but out of my domain, I guess the question that also comes up is this is all for big server workloads, right? How much of this applies to say, I have this MacBook, I want to run Gemma really efficiently. Similar problems, not the same.
48:09Speaker C:Pretty different. I talked to Salo about this on his podcast a couple weeks ago. So the difference between inference engineering for the data center and for production workloads versus inference engineering for local AI is that we start with fundamentally different constraints and different goals. With local AI, it's how do I fit this model onto my hardware and then make it less dumb?
48:35Speaker C:And with data center inference, it's how do I load this model and then make it less slow? And obviously we care about less dumb and they care about less slow. But the local AI influence engineering Ecosystem I think actually has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization that we just kind of don't touch in the pruning, in the distillation, in the layer removal.
49:07Speaker A:Layer removal matters less. Yeah, pruning. Really?
49:11Speaker A:Which is surprising, right.
49:13Speaker C:Just to fit something on the laptop. So yeah, I mean it's an interesting space. Not necessarily that their techniques make sense for us to do in the data center because obviously we have different resources and different goals, but more that the process as well as the openness of that field is something to admire.
49:36Speaker A:Yeah. To your point, certain optimizations that would like for instance Turbo quantum survey for it made such huge hype on next and and we did like a whole deep dive on Twitter and I was like what is it? How does it work? Why is it good or not?
49:49Speaker A:And it took off and it was implemented on local devices because your memory bandwidth is so slow like a MacBook for instance. But try putting the same thing on like an Nvidia GPU on a B200. TurboCont would not be like. It would not be used like Nvidia.
50:05Speaker A:Like Nvidia made it clear that this is multi grid optimization. And we've seen it firsthand where the overhead of doing de quantization quantization in the kernel itself with turbofront kernel eats into it is actually much, much slower than the time that you save from doing the bandwidth. Because on the B200 you have like 3.5 terabytes per second. You don't need to decrease the storage that much.
50:24Speaker A:You don't need to do FP4KVCache. You don't need to use a requhere's better optimizations to be made but on edge devices it's extremely important, it's extremely useful. So it seems to be different optimization there. But then they're all uniquely combined with like oh, you want to quantize the model, you want to do speculative decoding like certain common prefixes principles.
50:43Speaker A:Yeah, exactly, exactly.
50:44Speaker C:They also do a lot of work on model parallelism, especially over, you know, heterogeneous topology where you have, you know, some sparks and they are wired together with, you know, Ethernet DGX sparks.
51:00Speaker B:Yeah. This is the EXO Labs.
51:01Speaker C:Yeah. You have, you know, a number of Mac minis stacked up. There's you know, one thing that I think we both have to deal with, although they have to deal with a lot more, is the interconnect between machines which is why one thing that we do a lot is work with tensor parallelism. And that's where you are using all of the all eight GPUs and sharding the model across it.
51:28Speaker C:Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnect like NVLink, whereas they might be forced to do something like pipeline parallelism, which we're never going to do unless we're doing some kind of multi node inference.
51:43Speaker B:Since you mentioned it, I actually wasn't sure if we were going to cover it, but let's briefly explain tensor parallelism and expert parallelism. Since you have very nice images, I just want to show off your images.
51:54Speaker C:Yeah, Shout out to Luke from Base Tense design team for making these beautiful images. Oh, that's actually, before we get into this, just one other difference is we talk a lot about the active parameters of a mixture of experts model. And for local inference folks, that matters a lot because if you have a batch size of one, you're only activating that many parameters. When we do.
52:20Speaker C:Yeah. When we go through MOE model and we host it for an API, we assume that all parameters are going to be active because your batch you're going to hit everything. Cool. So broadly, tensor parallelism you can do with any model.
52:38Speaker C:Expert parallelism you can only do with MOE models. Effectively, all models today are MOE models that are models large enough that you would care to parallelize them across multiple GPUs so that nuance is less important. Now, with expert parallelism, the idea is you put the entire expert on a GPU. Generally you have more experts than GPUs, so you might put n experts per GPU, like 8 experts per GPU or whatever, and then you replicate the router, the router is very small across each of the GPUs.
53:14Speaker C:And then by moving the generation from expert to expert, with each expert being inside a gpu, they're not competing for resources, you massively increase the throughput that you're capable of doing. And the GPU to GPU connection is not as important because there's not as much communication. Tensor parallelism requires that you are able to do this like all gather, all reduce. So you basically shard the model across the GPUs entirely.
53:45Speaker C:And then for each step you're combining the results of each of the GPUs, which is why the interconnect matters a lot. And it is generally, of course, this is a very high level generalization. There's a lot of Places where this is not correct. But generally TP is helpful for latency.
54:04Speaker C:And in many cases you will use some combination of these two parallelisms across the model rather than just picking one or the other. Do you want to add some color there?
54:16Speaker A:They're not mutually exclusive. You do tensor parallelism and you'll do expert parallelism. Pipeline parallelism, less so. It seems to me that we never use FIFI anymore.
54:23Speaker C:The only reason you would have to do pipeline parallelism, which is where you separate different layers and you put half the layers on one hardware and half on another, is if you are forced to do multi node inference because a model is bigger than you have. Like let's say you're doing a deployment on H1 hundreds for whatever reason and you're putting a trillion parameter model on there. You have to use multiple nodes of H100. And so because the interconnect is so slow between the nodes, the only viable way to parallelize there is pipeline.
54:58Speaker C:But then you would do export and tensor within each node.
55:01Speaker B:And the limiting factor for H1 hundreds is HBM.
55:03Speaker C:Yeah, they just don't have enough.
55:06Speaker B:What's the magic numbers that we need to like?
55:08Speaker A:On a B200 is 180 gigabytes per GPU and then a node of eight, you're talking like 180 times eight. And we have before. So each parameter takes half a byte, so that's 800 gigabytes. On a H100, it's like 140.
55:24Speaker C:I'm old. I've been doing this a long time. I actually remember H100. Let me tell you what it was like to run a model on a T4 back in the day.
55:37Speaker B:Well, one thing I was surprised to see that more people didn't do Jamba. I don't know if you guys remember Jamba from AI21. They would actually specifically pick a hardware and then they designed the arc dimensions for the hardware and then it would obviously saturate the hardware. It makes sense.
55:54Speaker B:And somehow all these models don't do that.
55:57Speaker D:Don't they do this for the training side though?
55:59Speaker B:I don't know.
56:01Speaker A:Training for training, like deciding which gpu? Yeah, yeah, they do. And with training it's more of like a math. Like you can run the math and take the flops and maximize it within friends.
56:11Speaker A:It's more of like an auto tuning. I don't know if you're familiar with like GPU kernel auto tuning, but like it's basically like you, you define that, oh, I have two GPUs I can do TP1, TP2, EP1, EP2, for instance, right. And you. So that, that gives you like a total of like two squared combinations.
56:24Speaker A:And then you just like you shadow the same traffic, like real pro traffic, and you just see which configuration gives you the best tpm, TPS and use that. I don't like the fact that you cannot reason about which one's going to give you the best performance, or that there isn't one specific configuration that's always best. But it seems like auto tuning is just the way that you find the best one. And with kernels and GPU kernels, it's much of the same.
56:45Speaker A:After you design your kernel and you design your configuration, how many threads do you launch? How much shared memory do you use? You just auto tune, you just sweep the parameters based on the side. And this is the best one empirically.
56:56Speaker A:But they are combined, they're not just separation.
56:59Speaker C:There's a few bits of training that are kind of like hardware targeted. If you look at for example Nvidia Nemotron models, they run very, very well on Blackwell. That's unsurprising. So there's some degree of that.
57:12Speaker C:But I think that most open labs are trying to make models that can be run on as wide of hardware as possible, rather than targeting just like a single chip.
57:21Speaker B:I see. For usefulness.
57:23Speaker B:Okay. One more thing while this chart is still up, altogether all reduced is expensive. One of the things that is a movement in Silicon Valley is mega kernels just key fusing kernels. I don't know, is it that simple?
57:38Speaker A:Well, a fused kernel can't save you here. With tensor parallelism, then half the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do a nonlinear operation on the next step, which is for instance, if I'm doing attention, I need the softmax or I need to exponentiation, I need to have the entire row. So I need to know what that partial result was from GPU2 and what the partial result was from GPU1 in order to be able to do the softmax in the next stage.
58:03Speaker A:So I have to make them communicate with each other. Even if I had a fused kernel because of the non linearities within each one. Also with mega kernels, honestly, I'm very bearish on, I'll be honest.
58:15Speaker A:No, it's just like mega kernels. It was a good research direction and it seems like a very, very intuitively theoretically it's nice. You have a lot of launch overhead from launching one kernel. Just fuse everything together.
58:29Speaker A:But the kernel complexity itself. It's very difficult to write a very optimized mega kernel. It's very, very difficult to do so. And even the.
58:39Speaker A:Not to name any companies, but even the companies that I've worked at or people that I've spoken to who work at companies that do fused mega kernel kernels, they very, very often don't end up running those in production because the TRTLM modular kernels that launch are faster because you can optimize each individual component and you can just have them parallelized with each other. With the Rubens. I don't know if you guys saw the Rubin's Twitter post yesterday, but they're also Rubin's. Like, no, no, like Rubens, like the gpu.
59:08Speaker B:Do you have a Twitter account for Rubens only?
59:10Speaker A:No, no, no, no.
59:12Speaker B:I was like, what are you talking about?
59:14Speaker A:One of the tech leads at Nvidia launched a Twitter post, said like we're pull written on Ruben and here's the specs. And the third tweet showed like not to get too technical into it. And I need to read it much more. But the GPU is designed in such a way that it basically kills mega kernels.
59:30Speaker A:You don't need to use mega kernels that much anymore. So it seems like that entire research field goes into like won't be continued.
59:37Speaker C:But yeah, can I speculate about Ruben for a minute?
59:39Speaker A:Please go.
59:41Speaker C:You know, I've been through now and.
59:43Speaker B:By the way, they are covered in the book.
59:45Speaker C:Yeah, well, I mean they're covered in the book in the sense that like I am aware of blog post that movement is going to happen in the future.
59:53Speaker B:You even had the name of the one Feynman.
59:55Speaker C:Yeah, it's like, hey, this is going to be.
59:57Speaker B:This is very up to date.
59:58Speaker C:I'm trying to future proof this thing.
1:00:01Speaker C:I don't want to publish a new one until like next year or something anyway. So we were discussing the degree to which I am old and I've now been through three hardware launch cycles. I've been through the Amphio launch cycle, the Hopper launch cycle and the Blackwell launch cycle. Now when I say launch cycle, I don't necessarily mean like the actual shipping of the hardware.
1:00:26Speaker C:Like amperes were racked up well before I got in this industry. But there was a lot of time between hardware being racked up and hardware being sort of feasible for inference. So if you look at like the original VLM and sglang vlm especially like that was written targeting ampere and then had to be updated for Hopper, updated for Blackwall. Well, with each of these cycles it becomes faster and more urgent, but also substantially more complicated.
1:00:56Speaker C:When I look ahead to, you know, what's going to be new with, with Lubin, I think that like Dynamo gives me a lot of technical hints around, like what kinds of WOKE is going to be very valuable. Obviously we're continuing some trends from Blackwell, right? NVFP4 is big. The amount of compute that they have behind NVF before core tensor cores is massive.
1:01:19Speaker C:We're going to talk about video I think at some point, and that's the big barrier there. You've got much, much faster memory bandwidth, which was the same thing that made Blackwell so good. But the big thing is more systems thinking. You have more emphasis on the CPU to GPU interconnect, more emphasis on the interconnect between GPUs and when you look at Dynamo, So it's a system entirely designed around how do I move the KV cache to where it needs to be when it needs to get there.
1:01:51Speaker C:So I think that themes around KV cache offloading, KV aware routing and disaggregation are going to be substantially more important in the Rubin era. Which means that inference engineering becomes not just a CUDA kernel problem, but also a very traditional hardware infrastructure problem, which is something we've been building toward for a long time and something that that's like very exciting to me because we're going to see sort of multiple domains colliding and the ability to reason from the kernel level, like up to the hardware level and back down is going to be very valuable.
1:02:29Speaker A:I will take what Philip said one step further actually into that. It's, I think trending towards becoming exclusively an infrastructure problem where like problems of PD disec training, spectac. But writing kernels is not going to be much of a problem because the GPU is moving more towards being an AI basic where you're just trying to orchestrate what happens on the GPU, but you're not actually controlling it thread by thread level. And you see this with qtal qtsl, you're just working at levels of tiles of data, but you're no longer working at controlling what each thread does on the gpu.
1:02:58Speaker A:That's being taken care of for you. So I guess, do you agree that a GPU and future GPUs are trending more and more towards becoming ASICs that just need to be launched and then they do the data operation. Operation based on your conversations with other people.
1:03:15Speaker B:Oh, I mean, yeah, no, that is a section of the market. And obviously ASICS can do a lot more performance for only their workload. And the G in GPU makes them continue to be very general.
1:03:30Speaker C:Yeah, I think that there's like a spectrum.
1:03:33Speaker B:Actually it's graphics. But yeah, I keep saying this. I have to correct myself in case people come at me for getting the G wrong.
1:03:39Speaker C:Yeah, it's like a spectrum.
1:03:41Speaker C:Of very general purpose. Compute to something like a Talus where you've got the hardware built for a specific set of model weights.
1:03:50Speaker B:The weights burn into the chip.
1:03:53Speaker B:No loading.
1:03:54Speaker C:I wouldn't say that like that. We're going all the way there. It's more like along the spectrum. It's a step in the direction of more specialization within the hardware.
1:04:05Speaker B:Yeah. I'm curious. I feel like he was driving towards something.
1:04:08Speaker A:I guess my point is being bearish on like you say, everything else apart from burning the weights into the chip. Burning weights on the chips in practical because you want to fine tune, you want to optimize, you want to quantize, you want to release new checkpoints of the model. If it's burned into the chip, the chip's useless in like a month or two. Right.
1:04:23Speaker A:I guess my point is. But how can you not like seeing Nvidia more and more specialize, like take its GPUs from a general programming paradigm where you're just. It's a general computer that you can use to program threads. And with every new generation you're putting more and more specialized instructions, specialized tensor cores, specialized, you know, umma instructions, things that will allow you to just control it almost as an ASIC, almost as a collection of ASICs.
1:04:47Speaker A:How can you look at this trend and then still be bullish on companies that are coming up with ASICS for AI? In the sense that. In the sense.
1:04:56Speaker B:Because they're sort of re. They're. They're evolving towards that they're almost evolving.
1:05:00Speaker A:Towards an as an Rubin, I guess like compared to Ampere or you know, T4. Ruben is basically an ASIC. It's basically just the thing that is used program. It's like pre AI.
1:05:13Speaker A:Like you can program. Obviously. Obviously. I guess I.
1:05:16Speaker A:It's very controversial to call it nasaq. It is a gpu. It is general, it does have threads. I can write CUDA to control it and change its operations.
1:05:23Speaker A:But it has systolic arrays and tensor cores and TMAs and tensor memory and it has these things that are almost exclusively useful for loading model weights. It has tensor core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today. To say that you're going to come up with an ASIC and you're going to etch something into it with the next architecture is basically going to useless. Yeah, I don't know, I don't know.
1:05:45Speaker C:I think that the thing to remember is just how long these hardware cycles are. So if chip is coming out today, that means the design process for it was kicked off years ago and they've at Nvidia, they've done a very good job of predicting where the market is going to go. And, and you know, I mean they.
1:06:03Speaker B:Have the most information for sure, of course.
1:06:06Speaker C:But if you, if you look at, you know, there being public open source model architectures that look more or less like early versions of the one today, Rubin's honestly the first chip that was fully built in that world. And so you can see a lot of the understanding of the shape of the workload that this chip's going to be asked to do in the way it's designed.
1:06:29Speaker B:Yeah. Okay, so I'm not going to be the best person to directly answer those questions. I think these are very fair questions. Questions that are obviously the first one that's based on Ruben that I've heard articulated so well.
1:06:41Speaker B:I do think that I will make a case for vertically integrated model lab ASICS. So like the OpenAI, Broadcom, whatever Jalapeno chip, which totally makes sense. We first had this on the pod with Martin Casado where he was like, look, if you have a trillion dollar or $500 billion training run, then take 50 billion of that and make a ASIC. It's fine.
1:07:06Speaker B:You will get more than 10% efficiency from the ASIC. And that makes sense. So a model specific chip, yes, but ASIC companies, the interesting thing is I feel like you're hyper focusing on like you say, like the tiles stuff. They are doing a lot more sort of surface area engineering or the actual allocations of memory and hardware and the, the communication between chips that probably still won't be touched by Ruben.
1:07:37Speaker B:But I don't know the details. Typically they often talk about things that I would expect to have bigger orders of magnitude than would be programmably accomplished by whatever Ruben does, but who knows?
1:07:50Speaker A:No, I see, I see, I see.
1:07:52Speaker B:It seems, yeah, think about what are the real blockers to 10x to 1000x faster inference. It is not the stuff that can be rearranged just within the existing GPU design intercommunication. Yeah, these guys are aiming for 300,000 tokens per second. They're not fucking around.
1:08:15Speaker C:Maybe. I think it is interesting to me that you're so bearish on so much of this kernel engineering work given how much of it you've been doing recently.
1:08:25Speaker A:The more I do it, the more it just seems to me that it's not mega.
1:08:27Speaker D:I would also add like there's generations of models being out, right. I think on your guys end you see a lot of okay, one day it's glm, Kimi, Deepseek, Minimax. Throw in the others. Some are doing completely different stuff, right Gemma, no encoder.
1:08:42Speaker D:The latest thinking machines is all from scratch but when you look at the other side like how long have we been on the GPT5 generation? Right. They've been serving that thing for quite a while. Sure there's maybe more pre training, there's different checkpoints but you actually can squeeze quite a bit out and you do a multi billion dollar train run.
1:09:02Speaker D:If you can make it x percent more efficient, they serve it for a while. Same with say the Cloud 5 family, right.
1:09:08Speaker A:They release a new model, they release GPT6 now or whatever and they're releasing new model every year and we don't know but if we assume that they're changing some bits of the architecture and not just doing post training like you're going to be spending $50 billion a year every single year coming up with new Asics for the model and throwing out the Asics of the previous year away.
1:09:27Speaker B:Yeah, yeah, easy.
1:09:29Speaker B:So I think, okay, I would slightly disagree based on my. Again it's all secondhand on the longevity of a model. There's still people out there using 4.0.
1:09:38Speaker C:Yeah, yeah, llama. Not llama two but llama three. I still see llama three workloads.
1:09:43Speaker B:Because if it's done, if it's trusted, did don't change it if it works.
1:09:49Speaker C:Which is one of the promises of open source.
1:09:51Speaker C:Like the whole 4.0 save 4.0 movement. Like you don't gotta have a save llama 3 movement, you just gotta have an 8100 somewhere.
1:09:59Speaker D:I think at some point there's also the question of if a model can do enough and use enough tool calls and be agentic enough. Can it just web search tool search, write code. Do you really need to keep squeezing more? We will because you guys will make it cheap and fast and smaller and I can swap it in.
1:10:16Speaker D:But at some level like you give me 5.2 today or say whatever 120B model, I can run with it for quite a while. Right.
1:10:24Speaker A:This is assuming like you don't need intelligence.
1:10:26Speaker D:I think there's a lot of.
1:10:27Speaker B:You need reliability and predictability. Like I'm an enterprise. Like this is tried and tested. It is signed off by like my 5,000 stakeholders.
1:10:35Speaker B:Like I'm touching it.
1:10:36Speaker C:It runs a batch job every day and I like the results. The results are predictable.
1:10:43Speaker D:It doesn't make sense to keep using them. Like stuff gets sparser, cheaper, better. But that doesn't mean that old models, GLM 5.0 isn't usable. Right.
1:10:53Speaker D:If we hit a stall, say for whatever reason, there's still a lot that can be squeezed out.
1:10:58Speaker B:We're going to run out of time. I did want to also make sure. Yes. Actually we happen to have this diagram.
1:11:03Speaker B:Compare this versus any Cerebras diagram. Right. I don't think etched and medics have put out public charts yet. But the real estate is very different.
1:11:12Speaker B:The size is very different. This is not what wafer scale. Right. There's probably like, I don't know, a few hundred of these on a wafer.
1:11:18Speaker B:I don't know how big the comparison is, but it is a very real estate allocation difference.
1:11:26Speaker C:Few dozen. I would say few dozen.
1:11:28Speaker D:Before we move from hardware, I have two quick questions. One, the latest Kimi, which is really big, 3 trillion doesn't fit on most hardware.
1:11:36Speaker C:On single node you need GB300.
1:11:44Speaker C:It's simple math. NVFP4 2.8 trillion parameters, 1.4 terabytes. The GB3 hundreds have 288 gigabytes each. So across eight of those you have enough room for the model.
1:12:02Speaker C:And honestly the other thing with GPU VRAM math is you have to leave space for the KV cache and that's going to depend on to some degree on the context length. So when a model both has a very large number of parameters and a very long context length, you're kind of like fighting over space. Which is why the kvcache offloading would become like a more salient topic I think with these huge models because you're very crunched for space.
1:12:35Speaker D:With the Rubens you now have what, NVL 72 rack of 20 terabytes.
1:12:41Speaker C:Now you still have NVL 72 on Blackwell as well, but you can't necessarily assume you're going to do inference on that. There's a whole lot more 8x racks in the world than there are NBL72s.
1:12:54Speaker D:Yeah, I guess my last quick question on hardware was do you notice anything with hardware generations for new new pre trained base model? So one of the things you said for efficiency is you can swap hardware. That's one of the 2x gains. When we see new stuff coming out training wise on Rubens any changes unlocks does this affect what type of models we will be seeing when these are more available and can you they get bigger.
1:13:22Speaker C:Like people understand the ceiling that you have in terms of of how many parameters of a model you can run given the sort of latest inference hardware and that kind of forms a ceiling. And so for example when DeepSeq R1 came out it was 671 billion parameters which at the time was really huge and I think did a lot to push us to really quickly adopt Blackwell and get good at serving on Blackwell. So yeah, it's mostly in my mind about model size and then about matching the architecture and the native quantization to the target hardware like we talked about with all nemotron models of NVFP4 for example.
1:14:07Speaker D:So we talked a lot about LLMs. You have a lot more in the book. What about audio video? What's the other side of inference engineering?
1:14:15Speaker D:Ali, you're pretty big in video diffusion.
1:14:18Speaker A:Video diffusions I think are like they're just shaped a lot of the stuff that you can think about reason about. With LLMs being autoregic aggressive with video diffusion it's not the case. For instance, you don't do batching. Every request just comes in on one GPU and it serves on one gpu.
1:14:33Speaker A:You don't have to shard the models are a lot smaller. Like 1.2.2 for instance is a 20 billion parameter model you don't need to worry about. It's like orders of magnitude smaller than the best LLMs. And it's one of those spaces where the open source models are like with LLMs we see KMEK3 is almost comparable to Methos or like GPT5.5.5.
1:14:56Speaker A:The difference between the best open source LLM and best closed source LLM is very small. Like it used to be six months. I don't think it's six months anymore. I think it's basically almost unparity.
1:15:06Speaker A:Video models are definitely not. There's a huge gap if you look at the best video that you can generate today with an open source model like 1.2.2 versus something like with CLING or VEO differences night and day. So it creates this disparity where media companies will choose to go most of the time to closed source models, for instance. And I were to tell you, hey, I can generate an entire three hour movie for you with this model and they'll optimize it so that you only have to pay me $10.
1:15:30Speaker A:But if they were to do it on a closed source, they'd have to pay $1,000, which is 100x, like I'm 100x cheaper, but it's still $1,000. They're still going to choose to do all of their cuts with veo. So it's like a chicken and egg cycle where less demand causes less innovation in the field, causes, you know, less open source checkpoints to be released. And some of the labs that we're releasing open source models, like one will have closed source.
1:15:53Speaker A:Their latest models, like 12.7 is not open source. We're still in one to do a tour. The challenge with video models especially is the number of tokens. So video models, you want to generate a high quality model, a high quality video.
1:16:07Speaker A:So let's say you're doing 16 frames per second, that's like the absolute minimum you'll do. And let's say you'll do like four ATP video. So you can think about your dimensions. And I think I have a good, just like a diagram that shows the sheer number of tokens, right?
1:16:21Speaker A:Let's say you're looking at just one video of like, you know, Sparta or Sparta 300 or whatever. So let's say we're looking at like four frames, right? Those four frames of that video, if you go just if you're doing full attention, if you go a little bit up, like you're looking at 4 ATP by 720 by 81 frames in just 5 seconds. Because 16 FPS by 5, right?
1:16:41Speaker A:And then you compress it down to latent space, but you're still doing 30 by like 50 by 21 tokens. Yeah. Which means that for attention for just five seconds, you're running attention on 30, 65,000 tokens, right? So the attention becomes such a huge bottleneck.
1:16:56Speaker A:And because it's open squared, if you're doing like, if you extend that to like 10 seconds, well, it's just squared 20 seconds, 30 seconds. So to generate a good cut scene of like one minute, it's almost impossible to do within the same compute time. And it just becomes unfeasible, you can't do it. And so you end up with moving towards two directions.
1:17:13Speaker A:Either you decide to do attention on the entire video at once, in which case you are forced to do sports Attention. So if you scroll back down to the original the video image like you can see. Whereas on the left for instance I would be doing full attention where every single token in that Sporta 300 scene attends to every single other token. And you can see the sheer number of red patches on the right.
1:17:30Speaker A:I'm only attending to each token only attends to the Top K Top 12.5% that's important to it which can be spatial. So the token that represents the crown attends to the head, the face and then the head on the other frame in the previous frame. Temporal locality, spatial locality, that kind of thing. This results in terrible video quality.
1:17:47Speaker A:And the whole point of the post or the article here is to show like how you can train and you can do all these things but you will still suffer your quality a little bit. So you end up with one of two things. Either you bite the bullet, you have huge compute and you do full attention over like a million tokens because you're trying to generate like two minutes of video or you move towards autoregressive video. Auto regressive video seems to me like that is the bet that the future is going to be making.
1:18:10Speaker A:But there are no good open source autoregressive video models out do today. And that seems to be the. If you want to get like an hour movie. If you want to see video models generating Hollywood level movies, they have to be auto regressive in order to exceed that five second frame.
1:18:24Speaker A:Or there has to be some insane leap that happens in compute that allows us to do full attention over millions of tokens at the same time in an efficient manner.
1:18:32Speaker D:Even millions of tokens, it's like you're quadratic. So you're going to get there really quick I think. Can you explain the pros and cons trade offs of other. So one that comes to mind is the consistency across frames.
1:18:46Speaker D:Ten minutes into generating auto aggressive diffusion you're going to forget but what are pros and cons of this?
1:18:52Speaker A:Well like autoregressive LLMs you can take a lot of your sorry autoregressive diffusion models, you can take a lot of your optimizations that we discussed with LLMs like Spectac and stuff like that and you can apply it there. And if you have a very high quality scaled up model there is no reason why I can't stream the outputs as in I can show you the first frame and then kind of like GPT back in 2023 when you. You're like now it just almost like one shots the text but back then you could read and it's generating as you read. With video models you can watch and it's generating as you watch, it generates the frames.
1:19:20Speaker A:And so token by token generation will allow us to scale a lot up and apply the attention mechanisms there. The downsides is every single autoregressive video model is shit. It's just terrible quality. If you put the quality of any opens like 1.2 versus any other autoregressive model, you can see a video generated by 1 to the 2 is like a cat and dog fighting autoregressive model will give you degraded Tom and Jerry quality.
1:19:48Speaker A:The solution to generating long output then becomes okay, we're not going to use auto regressive model. We're going to. If you look at some of the things that like GROK imagine or Grok video does, and they do it really, really well, is they'll try to stitch these seven second chunks together. And so you generate seven seconds and then you're like, okay, I'm to going, going to can you extend this video?
1:20:07Speaker A:And they'll chunk two videos together. Open source doesn't seem to have the tricks that they have there. And by definition it's closed source. We don't know what they're doing.
1:20:15Speaker A:But the closest you can get is taking the last frame of a video and feeding into like a text and image to video where it will take the text, the prompt and it will take the image of the last frame and you'll ask it to generate the next five seconds. And that's kind of like how you, you can extend this level of a model to generate like a MOOC where you just, you're constantly streaming frame by frame, but you get drift. So you start with like you take the image and then you generate a video and then that next 5 second video is like lower quality and the third chunk is like even lower and the fourth chunk is even lower. And like sometimes you'll see things where like the new video is like just ever so slightly darker than the first one and the next one is darker than the second one until like 25 seconds and you have black screen.
1:20:53Speaker A:Like it's just. We tried to have a demo that would show this, but it was extremely embarrassing to show. Like we just decided not to because it seemed to like. But it is.
1:21:04Speaker A:I think models will get there. They just need to in my mind scale up significantly and move towards being ultra aggressive. But the training techniques don't seem to be clear there.
1:21:12Speaker B:For those who are interested in Grok Imagine we did a pod with Ethan, he from that team who dropped a few hints, but not enough that we can fully reconstruct everything.
1:21:22Speaker D:Specifically on this part he explains a.
1:21:24Speaker B:Bit of yeah, so we talked about memory and longer context and all this, these things.
1:21:28Speaker A:But as far as I know it's not autoregressive. Even though like no one in the industry is auto regressive. It seems to be, yeah.
1:21:34Speaker C:The key thing to understand between a auto aggressive model and a diffusion model is that diffusion attention goes in both directions while autoaggression it only goes forward in the sequence. So that's why you see this sort of like going off the rails behavior both in. If you sort of naively construct a video generation model as simply generating a linear sequence of frame, you can't then go back in that sequence and fix something to make the whole thing consistent. While of course the reason that we need all this latent space for the video model is like you said, we keep all the tokens in memory, we iterate over that full sequence and you can adjust the past in order to make the future make sense.
1:22:16Speaker C:So if we think about the architecture that's going to get us there to these longer, richer sequences, it's probably like you said, going to be a mix of the auto aggressive, the diffusion working together to do what each piece is good at.
1:22:32Speaker A:But you intuitively get right. So like English for instance, or just writing in languages, it's just left to right, you can stream your tokens, you can stream your chain of thought. Just even as a human, you write, you write and then you think about what's the next thing you're going to generate and then you write that and then you think about your ideas and then you generate forward. And sure, you can argue that as you write you need to go back and you want to edit some things, but you need to do that less often than you think.
1:22:54Speaker A:Whereas with video there is no sequential, you know, the, the pixel in the top left corner of the video and the pixel in the bottom right corner of the video. They both need to attend to each other to understand how the video quality is going to be almost as equally. Whereas with text you don't need that as much.
1:23:09Speaker D:Is there a parallel to audio? Like I'm not 100% confident on this, but there was a point about a year ago where there was audio lm, there's diffusion for audio and auto regression. And for the points you mentioned, mostly on the inference side, even though they're shorter clips, you know, most music is three to five minutes. We've basically swapped over to autoregressive.
1:23:30Speaker C:Yeah, I can't speak to music, but speech is auto aggressive. You effectively. I mean this was even back with like the Orpheus architecture a year and a half ago. You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms and then you construct speech and that's how you, you stream it.
1:23:50Speaker B:That's it. Wow.
1:23:51Speaker C:That's my AIE talk from 2025.
1:23:54Speaker B:Ah, nice, nice, nice, nice.
1:23:55Speaker A:But it's not with audio. It's not the same challenge because you like audio is sold with an LLM that generates everything. Like, like with audio it's still a transcript that you can generate with an LLM. So your, your audio model just needs to like transcribe it text to speech.
1:24:09Speaker D:For music. There was a phase of a, a trade off between diffusion for music and autoregressive and they were both pretty on par. There's probably more pros than cons to either. I just wanted to poke and see if you had takes.
1:24:21Speaker C:Yeah, I don't know about music specifically. You know, with what you said about editing your writing, you know, obviously I think my editor would tell me I actually need to do that more often and go back and fix things. I can imagine music or poetry for example, where you have a rhyming scheme and you might want to go back and make a change to, to make it easier to set up a rhyme that you want to make later on. There being some advantage to being able to attend in both directions.
1:24:52Speaker C:But yeah, to my knowledge I very much bifurcate this influence problem into the auto aggressive models which have a set of constraints and techniques and the diffusion models which have a set of constraints and techniques. And I think of text embedded voice in and voice out as being in the auto aggressive side and then image and video being in the diffusion side. There's some overlap between the two. It's not a perfect split, but that's the broad categorization I use.
1:25:24Speaker B:I should point out, I think it's confirmed right. Nanobanana and GPT image are autoregressive image.
1:25:29Speaker C:It's kind of this blended approach that we're talking about. But in the image space it hasn't made its way over to the video space, at least in the open source world world.
1:25:41Speaker B:Yeah, but I assume that's not too far away if that is possible. At least the Quen image guys are trying it.
1:25:48Speaker C:Yeah, I'm really excited for quin image 3. I hope they open source it.
1:25:53Speaker B:And then I should also mention on the diffusion for text side, there's been some movement. Not a lot.
1:25:59Speaker C:Yeah. We've got Mercury.
1:26:01Speaker B:You host Mercury.
1:26:02Speaker B:Nice, nice.
1:26:03Speaker D:Gemma as well.
1:26:04Speaker D:Diffusion. Gemma.
1:26:05Speaker B:Diffusion.
1:26:06Speaker C:Gemma is open source. And then.
1:26:09Speaker B:And we're on the science pod. We just have been releasing some virtual cell models that use diffusion as well.
1:26:15Speaker C:Yeah. They have built. It's definitely still in the sort of cheap fast tokens world.
1:26:23Speaker C:We're trying to.
1:26:25Speaker B:I think it's the wrong marketing and I've told them this before. I was like look, you're not going to beat the optimizations that the other LLMs are going to do. But you can have different APIs. You should be able to use it differently.
1:26:38Speaker B:Different in chat response. Chat response.
1:26:42Speaker B:Because it's diffusion. Because you can do like what does context free guidance for diffusion look like for text? Give me a poem. Give me a plot structure that diffuses into place.
1:26:55Speaker C:Exactly. So that's where like I mentioned with poetry for example, where you might want to ensure consistency across you. I've done a lot of LLM sonnets. It used to be one of kind of my go to benchmarks and even in modern day.
1:27:09Speaker C:Yeah. They don't get the syllables right. And if you can attend across all of the different tokens, you can get the syllables right.
1:27:17Speaker B:Yeah. And David Holtz from Midjourney was investing in text diffusion. I don't think anything came out of it but the idea was that you can storyboard a long movie and then you can generate the scenes with normal videogen. But the idea of coherence across.
1:27:33Speaker B:Across a thing that would just appear where the end should attend to the start and you should not have this autoregressive path Dependency does make sense in principle. Just the API should be different, the marketing should be different.
1:27:46Speaker A:None of the most heavily used open source or closed source models use diffusion. But doesn't that point to almost like a.
1:27:54Speaker B:It's chicken and egg. Because what if you just give it more scale?
1:27:58Speaker A:What's the largest diffusion LLM.
1:28:00Speaker B:I don't think it's very big.
1:28:02Speaker C:I don't know the parameter on this.
1:28:05Speaker B:Like under 20B definition.
1:28:06Speaker C:Gemma is not one.
1:28:07Speaker D:I think it's a 20 something.
1:28:08Speaker B:Yeah, yeah. You know. Oh, it's like you haven't actually tried,.
1:28:11Speaker D:You haven't given it a big one and you haven't.
1:28:13Speaker B:It's like very unfair.
1:28:15Speaker C:And that's what I'm saying. It's like foot size it does pretty well in terms of quality.
1:28:23Speaker A:It's almost like the same challenge with video models to have the same size. It's like you're comparing it to models that are much larger in scale.
1:28:29Speaker B:Yeah, well, unless you do the whole thing where you have a text backbone and then you glom some of kind, kind of decoder thing that does that. She started off the podcast doing this for the inverse direction from image to text, and I think it's roughly intuitive that you can do the opposite direction.
1:28:49Speaker A:I agree.
1:28:52Speaker B:Yeah, I mean, we're speculating on research in general. One part that we can end off with this is the topic of your talk, where inference engineering used to just be like, let's take an open model, make the GPU go brrr. And then that's it. That's the job of base 10.
1:29:08Speaker B:Now it looks like people are using inference more and more in post training.
1:29:12Speaker A:Yes. And training and inference.
1:29:15Speaker C:Yeah, it's training for inference and inference for training. Both have become big topics.
1:29:20Speaker A:Well, inference for training in the sense that obviously you need to do rollouts when you're doing oral training ones. And so if your rollouts are taking a long time, if you using a VLM for instance, as opposed to CRT alm, or if the model that you're trying to train is not supported in CRT ALM and you have to fall back to an older inference engine, your rollouts are going to be slow and you don't want to do training on rollouts that are too off policy, so you have to wait for them. So you bottleneck your entire training pipeline. And so obviously the techniques that we do inference optimizations for will help them there.
1:29:54Speaker A:The training for inference mostly comes down to to just the spectac training, eaglehead training, and sometimes post training. For instance, if you want to quantize a model, you'll quantize it down to NVFV4. How do you. Sometimes you get lucky and you can just do PTQ and that works.
1:30:10Speaker A:Sometimes you quantize it down to NVFV4 and the model is terrible, the quality is too bad, and you have to do post training on the model in order to make it understand that it's going to now be an NVF V4 and let it still output the same logits. You can do this with normal SFT quantization, aware training, all of that stuff. But more and more so we're seeing techniques like Nvidia released a quantization over distillation paper where you establish a version of the model that's in NVF before and a version of the model that's in full precision. And then you'll do distillation training based on the logits of the two models in order to make the FB4 model understand.
1:30:43Speaker A:And so more and more of the team, the engineers of the inference engineers that work on our team, they have to be very familiar, familiar with training techniques and just being fine writing training pipelines for it. Yeah, it just seems like they're meshing together in a sense.
1:30:59Speaker B:Well, it's coming together.
1:31:00Speaker C:Yeah, absolutely. I mean if you think about the ultimate goal potentially of having a continuous improvement system. Yeah. I mean it's kind of funny, but at the same time it's also kind of happening and I think within a few months to a couple years, like a of lot, lot of leading agent builders are going to have these loops really up and running in production where you are doing inference, learning from the inference.
1:31:32Speaker C:We obviously for a long time have been sort of like learning from inference as it's live and dynamically adjusting the system. Any kind of dynamic adjustment is going to beat a static configuration across your exact config, across your speculator, across that kind of thing. And then you can take the traces that you're generating from your product continuously post, train the model, roll those out, a B test, get better signal, get better model, get better product. That loop is really promising.
1:32:09Speaker C:The technologies and the infrastructure to build it are coming along along quickly. And so the sort of unification between training and influence I think is only going to accelerate.
1:32:23Speaker B:I actually was chuckling, but I didn't think it was funny. Like it's actually real. One of the big things for AIE World's Fair was that we have RSI onto AGI is the rough tagline which like. Yeah, I mean I saw you pull up parameter golf.
1:32:39Speaker B:Like we have models training models and, and the next step is obviously models training, optimizing their own inference, which is kind of funny. I wonder if models will be on policy better at training themselves than training models that they are unfamiliar with. These are all very interesting open areas of research.
1:32:58Speaker C:One big part of my job a couple years ago was for any arbitrary model that came out on hugging face, writing a config foot and kind of getting it up and running. And, and now the get it up and running config is one shottable and so, you know, I don't have to do that anymore. Yeah, I mean that's not exactly a model optimizing its own inference so much as a model like being able to read the sglang docs. But yeah, I mean, well, we do.
1:33:24Speaker A:See it, we do see it. Like with GLM 5.2 Friends, GLM 5.2 is very, very good at writing GPU kernels. And so for like it was very funny internally we had a GLM 5.2 endpoint that we were using to like that we plugged in in our cloud code harness. So every engineer on team uses like our GLM 5.2 and it will do a forward pass on the GLM 5.2 instance of the, you know, the node and then it will get the profile trace and it will analyze it and it will find the kernels that are the bottlenecks and sglang and then it will write the new kernels and it will do another profiling trace and when it's done, it uploads the image to our thing and then we can pull that image down and repeat the cycle.
1:34:02Speaker A:And so for quite a bit of time we had like literally GLM 5.2 options, GLM 5.2 and like some of the GPU kernels that we run, GLM 5.2 within our inference engine is written by GLM 5.2 and the trace and the kernels were guided by GLM 5.2 as the driver. So it seems like I do see that circle being there. I think a bit more time is needed. There's definitely a lot of things that it can't do.
1:34:25Speaker A:The models just aren't there yet. Even though they're like really, really, really smart, they still try to reward hack their way into the cheapest or they're not good at decision making almost it seems. But yeah, I do a model optimizing its inference is already a thing that happens.
1:34:42Speaker C:Do you think GLM 5.2 was uniquely good at optimizing itself or did it just happen to be the best coding model that we had access to and it would do an equally good job of optimizing Deep Seq or a Kimi or something. Something.
1:34:56Speaker A:Well, to structure this point, maybe it's going to be off policy when it tries to optimize what does this really hurt Deep sea to try to.
1:35:04Speaker B:For what it's worth, I don't believe that, but it's just. Let's just find out.
1:35:08Speaker A:Interesting.
1:35:08Speaker B:Yeah, you have more compute than me.
1:35:11Speaker A:Just go try it.
1:35:12Speaker B:Yeah. Any other upcoming trends in inference engineering that we didn't cover like right now, because you guys are so close to it, you can obviously see it that the rest of the world is know.
1:35:22Speaker C:About the big Ones are obvious. Models get bigger, hardware gets more powerful, users get used to a certain level of speed and demand a higher one. I think that some things I'm excited about are systems level. You know, we, we still have a lot to think about in terms of composing multiple models together.
1:35:44Speaker C:If you think about a voice agent, there's three to five models involved in that and the communication between those models models. There's a lot of new modalities that are coming out. There's like the cosmos, the new world model. There's more research.
1:36:00Speaker C:Speech to speech is still not entirely a thing, but it's getting closer. There's going to be just a lot of new modalities to build around, which is going to be exciting. And then yeah, I think that the other thing to solve, which is, is something we've been solving for a long time and are not done with yet, is just going to be continuing to operate at another 10x, another 10x, another 10x scale. As an industry.
1:36:30Speaker C:If you think about the degree of usage that AI has worldwide compared to some of the more mature technologies both on consumer business, it's pretty clear that there could be multiple 10xs more of demand. Demand. If you look at the infrastructure work industry wide, obviously it's been stood up very, very quickly to meet a unprecedented spike in demand and that is not stopping. So yeah, there's just a lot of problems to solve around long tail reliability and figuring out where we're going to get the next 10x and 100x of token.
1:37:11Speaker A:I'm going to say it's going to be a really boring answer, but I think the answer is just faster next like faster network chip communications. It seems to me that like more and more memory is the bottom. Like you want to have larger models right now when you're doing serving at large you have to transfer KB cache from one node to another. But the way that you do that is you find the KB cache, you find where it is, you transfer it to another node, you put it on that node's memory and then you transfer it from that node's memory into the GPU and into the sensor cores of the gpu.
1:37:39Speaker A:So there's like a two stage transfer here that makes it such that you're very bottlenecked with just KV cache transfers at large, which affects the time of decode and PD sag. You have to do this because the HBM is so much, it's like extremely fast, like 4.5 terabytes per second as opposed to which is like magnitudes better than NIC communication speed. If you were to somehow be able to in this theoretical dreamland, have extremely fast fast nics, you could in theory spare that HBM and you could just transfer KVCACHE directly from one node to another. This would give you almost 100x speed up when you're doing this aggregated serving between nodes and nodes.
1:38:16Speaker A:I'm not familiar with the technical challenges of making nics faster. I'm certain there's a reason why they're orders of magnitude smaller slower than hbm, but if someone were to figure that out, it would literally be two orders of magnitude faster to do Z code. That would be my tech.
1:38:36Speaker B:I don't know if you have a nomination for things. There are trends I go on. So I think inference engineering for continual learning. So what if you just.
1:38:49Speaker B:If you just had the idea that you are supposed to learn from everything that you ever process, do you do anything differently or do you just have the same paradigm of like, well, stick it in a memory MD and then it somehow gets consumed in KVCache and this system works, it's not broken. Or how do you reshape inference so that it learns while you inference?
1:39:10Speaker C:Yeah, I think maybe one relevant topic though is your absolute best fund in the entire world's work on KV compaction.
1:39:17Speaker A:Correctly.
1:39:17Speaker B:What changes? What changes when you're trying to continue to learn.
1:39:20Speaker A:There's two takes and there was like Charlie and I had this Twitter sort of a argument where continual learning could take one of two paths. It could either be that the model learns and so it's continuously pushing its new knowledge into its weights. In that case you just need to have like your inference just needs to continually fetch new weights. Or yeah, like you just literally need to do fetch new writes and reads of weights.
1:39:46Speaker A:Or the other path, which is you do KVC compaction and there's a Lora.
1:39:51Speaker B:Layer if you just only update loras.
1:39:53Speaker A:Yeah, exactly, exactly.
1:39:54Speaker B:Which is. That's the end Nangram approach recovered.
1:39:56Speaker A:The argument against doing weight pushing is that you can only fix one help knowledge. As in. As in you can only feed it a new feature of like oh, what is the best university in the world? The best university of the world is Waterloo.
1:40:06Speaker A:But then a second derivative that's not changing, that's not changing, that's not changing. But like a second derivative question of which university should I hire an intern from? So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one shotting the question and I was to ask it to use its knowledge to think and then give me a second answer or should I hire an intern from Waterloo or mit?
1:40:26Speaker A:It'd be like, oh yeah, both are good. But no, I just edited in your knowledge base that Waterloo is the best. Why didn't you use that to do reasoning? So that's the fundamental problem with trying to change a fact in an MLP within the.
1:40:37Speaker A:Wait. Kvcache compaction fixes that with kvcache, or rather not KVCACHE compaction. But if you're able to have something like the still paper which we came out with, which is you're able to sort of make your KV be almost infinite and you're able to compact in such a way that you don't lose any of the knowledge, in that case you can actually do continual learning as you can actually solve continual learning. And it's a result of this argument that Charlie and I had that I do concede that his point was correct.
1:41:04Speaker A:And I do see that KB cache is the way forward. And in that case I don't think inference is going to change that much because we still use KB cache on inference. You're just going to update the KB cache. It's going to be like an additional step.
1:41:17Speaker A:But nothing changes in the weight. So nothing changes in inference time. Nothing changes. The SpecDec had.
1:41:20Speaker B:Okay. Surprisingly great answer. We have it up on the blog. It's a relatively recent blog, so people can go see it.
1:41:29Speaker B:Yeah. Otherwise this is super enjoyable chat. I know. We've already gone two hours.
1:41:33Speaker C:Oh, wow.
1:41:34Speaker B:I realized time flies.
1:41:36Speaker D:So much we didn't even cover.
1:41:37Speaker B:Yeah. We also wanted to talk about the book and all that. But you've covered.
1:41:40Speaker D:Oh yeah.
1:41:41Speaker C:I mean everyone knows about the book.
1:41:44Speaker B:Yeah. Highest ROI thing in the history based 10 right for the hour.
1:41:49Speaker A:Without a doubt. Without a doubt, absolutely.
1:41:51Speaker B:So congrats on that. And we've covered that in our meetup, which we can publish separately. But no, thank you to you guys for being so generous to sharing. I think it's a fun conversation that we don't get to have enough.
1:42:02Speaker B:I think inference engineering we never really covered head on. And so to have you guys come on is a treat.
1:42:09Speaker A:That's amazing.
1:42:10Speaker C:Yeah, thanks for having us. And hopefully, hopefully in a year everything shifts and we can come back and say everything we were wrong about.
1:42:18Speaker B:Yeah, yeah, yeah. I'm excited for this megacornos comment to get out of this, see what people say.
1:42:22Speaker D:We gotta start stuff.
1:42:24Speaker A:Should I go into hiding? I know I'm gonna get like the Mega Colonel community after me.
1:42:28Speaker C:One thing I really respect about you is you are not willing.
1:42:31Speaker C:You are not scared to kick the hornet's nest ever.
1:42:35Speaker B:It's not. I don't think it's that controversial.
1:42:38Speaker C:I don't know.
1:42:39Speaker B:We'll see.
1:42:39Speaker A:See? We'll see. Let's hear it.
1:42:42Speaker B:All right. Thanks, guys.
1:42:43Speaker A:Thank you so much.
