0:00Yeah, hi and welcome back, everyone, to part four of the Build a Reasoning Model from Scratch series.
0:07So, previously, we talked about the base model, how we implement the base model, make it more efficient using a KV cache.
0:15And then previously we also talked about how to evaluate that model using a verifier on the MATH-500 data set.
0:23Now we are going to improve the base model and also, in general, talk a bit more about different ways we can improve that model.
0:31So here on that slide, you see a little benchmark that I put here just to illustrate the two main ways we can improve an LLM.
0:40So that goes for both the base model and the reasoning model.
0:44And so on the left side, we have a panel that says inference-time scaling, and on the right side, we have training.
0:50And so let's first focus on the left side, inference-time scaling.
0:54So inference-time scaling is also sometimes called just inference scaling or sometimes also test-time scaling.
1:01And that refers to the mode when we use LLMs after training.
1:06For example, when we are using the model to just generate answers or also previously when we evaluated the model on the MATH-500 data set, that was also using the model in inference mode.
1:19So inference essentially means non-training.
1:21So everything that just uses the model, like where the model generates answers, that's the inference mode.
1:29And so on the y-axis, we have benchmark accuracy.
1:33So I will have more information about what type of benchmark that is later in a later slide.
1:39But just to keep it simple, just think of this as the MATH-500 accuracy, like benchmark accuracy.
1:45And on the x-axis, we have inference-time compute.
1:50So how much compute, how much computational resources are used by this model, essentially?
1:57And we can see that there is a correlation between the benchmark accuracy and the amount of compute we spend.
2:05So, as a rule of thumb, we can say the more compute we spend, the higher the accuracy.
2:11Of course, there is at some point a saturation where this wouldn't go infinitely to 100.
2:16Probably if we even spend more inference-time compute, it will saturate at some point.
2:21But the basic message here is that one way to improve a model is to spend more compute.
2:29And so how can we spend more compute?
2:31One example would be just having the LLM generate more tokens.
2:37And we will see an example of that in this video.
2:40So, for instance, if we modify the prompt or if, for example, we turn the LLM later into a reasoning model, these are all ways we can have the model generate more tokens.
2:52And usually, I mean, this is only true to some degree, but often we can see the more tokens the model generates, the higher the accuracy, up to a certain point.
3:04And another way to improve the accuracy through inference scaling is also using different types of techniques that get more out of the model, like majority voting.
3:15(We will talk more about that.) Or self-refinement, where we have additional techniques we add to the model that causes us to use more compute, but then again improves the accuracy.
3:28Another way to improve the accuracy of the model is, of course, more training.
3:32This means training larger models that require more training-time compute.
3:37If the model is larger, it's just more compute, but also just increasing the data set size.
3:42So there are different ways we can also then improve the accuracy during training, or even if you think about additional post-training techniques.
3:51So if we, during pre-training, extend the data set size or the model size, that, of course, often results in higher model performance accuracy.
4:01But also if we are adding more like supervised fine-tuning or reinforcement learning with preference tuning, reinforcement learning with verifiable rewards.
4:11These are all techniques also that cost, of course, more compute, but they also help the model to become better.
4:17And one thing here is it looks like they are two separate things.
4:21I mean, they are conceptually two separate things, but we can also combine them.
4:25For example, we can take an existing model, train it on more data, and then also add more training techniques.
4:32And after that, we can still add also additional inference-time scaling techniques to make a good model even better, so they can be combined.
4:40So I have a slide here that I showed you already in the very beginning, where this shows the GPT-5.6 model with different effort modes, reasoning effort modes.
4:52And you can think of reasoning effort modes also as a form of inference scaling, because the higher the reasoning effort, the more tokens the model uses as the intermediate reasoning chain.
5:03And that also then often results in higher modeling accuracy.
5:08And we can see that here, for example, for GPT-5.6 Sol.
5:13So on the left-hand side, let's consider the lowest case, no reasoning effort or even here, low reasoning effort.
5:21We can see there's a big jump. And then every time we increase the reasoning effort, we spend technically more compute.
5:29Here on the y, on the x-axis, it says API costs.
5:33So it's not exactly the same as on inference compute because we don't know what their overhead is.
5:38But I'm assuming it's like a pretty high correlation between inference-time compute and the cost.
5:44So the more inference-time compute you use, the more they charge you for.
5:49And so we can say that increasing roughly the inference-time cost here also increases the accuracy here on this benchmark.
5:58And you can see also it's saturating at some point.
6:01So it's not like infinitely going higher and higher. At some point, we saturate.
6:05Also, if we compare 5.6 Sol to, for example, the smaller variants like Luna and Terra, we can also see there's a difference between, let's say, Luna Max, Terra Max and Sol Max.
6:19So here, if we jump between different models, we can think of this as training, sorry, as scaling the training compute because these models are larger.
6:29Like Sol is larger than Terra. Terra is larger than Luna.
6:32We need to require or we require more training compute for training a larger model, which has more FLOPs.
6:40So there's more computation happening inside the model during training.
6:44But then also it requires usually more data.
6:47So we can say this is like Sol is both scaling the training compute compared to Luna.
6:55But then also here within Sol, we are training, sorry, we are scaling the inference-time compute.
7:01But yeah, anyways, this is just like an illustration to make it a bit more, let's say, hands-on than just conceptual, where this would be like a real case or real-world example of inference and training-time scaling.
7:12Now, this plot that I showed you in the beginning comes from a blog post by OpenAI.
7:18So this was the original OpenAI o1 blog post where you can find more information what type of benchmark this was.
7:26So I showed you the simplified version just to get some of the details out of the way.
7:30But if you are interested, you can read up more on that.
7:34And o1 was technically the first reasoning model that really made big waves, let's say, in the world.
7:43Like that was the first model that people called an LLM reasoning model.
7:47So that's like going back to this article by OpenAI.
7:52Now, in this video, in part four, I want to cover or we want to cover here finally some of these inference-time techniques to make the model better.
8:01So like I mentioned before, previously, we implemented the conventional LLM.
8:05We did the training of the loading of the pre-trained weights.
8:08We implemented the text generation.
8:12And then previously, we evaluated the model on MATH-500 using a verifier.
8:17And that is just so that we have a baseline, a performance baseline.
8:21When we improve the model, we can then measure by how much we improve the model.
8:26And so now we are going to finally improve the model and talk about inference techniques.
8:31And it's a bigger topic.
8:33Of course, there could be a whole book only about inference-time techniques.
8:38But here I will cover the most, I guess, fundamental ones, the popular and fundamental ones.
8:46So first, we will start with advanced text generation and voting.
8:50So that means we are updating our text generation function to support sampling.
8:55And then we will implement inference-time scaling technique called self-consistency.
9:01Because this will already take a long time, like probably one or two hours, we will cover self-refinement in the next part.
9:09So here the focus is on the updating of the text generation and the voting, like the self-consistency.
9:17And later on, in later parts, we will also, of course, go to the training techniques, particularly the reinforcement learning.
9:23It will also be two big parts on that and then model distillation.
9:27But let's start with inference techniques.
9:30And so, in particular, in this video, the techniques we are going to cover are this first method and the second method.
9:38And next part will be this third method.
9:41So there's a lot of information going on here.
9:44So just to talk you through it.
9:46And so here, this is numbered by the steps we are going to take.
9:50So first, in this video, we have to, of course, start with a pre-trained model.
9:55We have to load the pre-trained model weights.
9:57And then we will modify the prompt using a technique called chain-of-thought prompting.
10:04And this, interestingly, is a very simple old technique, but it does really help with the base model.
10:10I will show you how that works.
10:15But later on, after at the very end of this video, I will show you also my runs on the MATH-500 data set, because it takes a few hours to run all the different techniques on the MATH-500 data set because it's just computationally expensive.
10:30But I will show you that this simple technique really more than doubles the accuracy of the base model.
10:36And after this chain-of-thought prompting, we will get to the second technique, the self-consistency sampling.
10:45But in order to implement that, we will first have to talk about two fundamental text generation sampling-related techniques.
10:52So temperature scaling and top-p filtering.
10:56And then in the next part, in the next video, because these two techniques will already take a lot of work, especially if we want to go step by step and also implement these subtechniques, we will reserve this third method for the next part, because that will be a whole topic in itself again as well, especially if we also have to talk about log-probability scoring.
11:19OK, so I think that's it for my slides.
11:23So let me go to the chapter itself, so you can find this, of course, on the website.
11:30You can also, you probably also have maybe access to a print copy.
11:34But for the video here, I think it's just easier to share the chapter on, in the web browser.
11:43And so we talked about all of that.
11:47So let me start a Jupyter notebook.
11:53So we go to chapter four already.
11:59And then let's start a new notebook.
12:12And so this is, let's call this inference-time scaling part one.
12:22And we already covered the introduction.
12:30So that's what we already covered.
12:31I mean, I talked you through most of this.
12:34And so really where we start now is loading the pre-trained model because we do need a pre-trained model in order to generate text.
12:44And so for that, I will mostly not retype everything because that would take a long time.
12:49And it is already going, I guess, going to be a long video.
12:52So copy-paste most of the things over and just talk you through them.
12:57You make this a bit taller here.
13:02And so first, of course, that's just a boilerplate I have in all the code examples.
13:07So just showing you the software versions we are going to use again.
13:11So this is Torch version two point ten.
13:13It's just like on my computer, the, I guess, the environment that is installed there.
13:18But I did test everything in Torch two point eleven and two point twelve and two point thirteen.
13:24And so everything should also work just fine there.
13:27I also have unit tests in my GitHub repository to always test if there's a new version that things still work.
13:33So don't feel like you have to downgrade to Torch two point ten.
13:38Two point eleven, twelve and thirteen should also work for future versions of PyTorch.
13:43I think two point fourteen also is already out, and I have it in the GitHub tests.
13:47But if newer versions come out, like two point fifteen, sixteen, it's hard to say if they still produce the same results.
13:54My guess is 90 percent. I'm 90 percent sure they will also still produce the same results because nowadays changes in PyTorch are very minor.
14:03Like they are usually improving some things, but it's very unlikely that there is a bug in the library so that things will change numerically in the future.
14:12So, yeah, just that being said.
14:16But if you find that you have any discrepancies, like any major differences when you're running this code or some errors that you can't explain and you notice you have a newer PyTorch version, it's maybe not a bad idea for debugging just to use exactly the same version.
14:32Then again, if you still have issues, then it must be maybe a typo or something.
14:36But if the issue goes away, then you also know maybe this was the PyTorch version.
14:42OK, so this is just my boilerplate, and then we can already maybe let's, let's already go to four point two because I talked you through all the introduction.
14:53So loading the pre-trained model, and this is just reusing code from the previous two videos where we coded most of this up.
15:03So, again, nothing really super exciting happening here.
15:07It's just reusing the code from chapter three and chapter two.
15:11And so we could technically go there.
15:14So if you have, for example, let's do it like this.
15:18If you have still the code from the previous one...
15:23And where we have defined it is here.
15:28So we could technically copy and paste this here into our notebook.
15:37And that's a lot of code, as you can copy-paste this here.
15:40And then we don't need to, let's say, import it from here.
15:43But as you can see, this is a lot of code and this is just repeated code and it would kind of distract from the new stuff.
15:49So I put everything we already have into this library, the Reasoning from Scratch library, where you can then import from this like here.
15:58So you don't have to copy everything over.
16:01I think this just makes things easier.
16:03And the same for the get device.
16:05Now, we can run this on the MPS device.
16:10So that should mostly work also for CUDA, of course.
16:13But I do think there were some minor differences between MPS and CPU, which is why I recommend for this chapter, if you want to reproduce exactly the same results you are seeing here, I do recommend using the CPU here.
16:27I mean, the code is also relatively cheap, so most of it.
16:31So it's just fine to run it on CPU.
16:33And once things work, you can comment this out once you are finished and then rerun everything on MPS or CUDA if you want.
16:41But I recommend for now using the CPU just for compatibility reasons.
16:47Yeah. And then next, let's just make sure the model works.
16:51So here we loaded the model.
16:53So if I ran this before.
16:55So I ran this before.
16:56So you, so I have it here downloaded locally.
17:00So if you don't, it might take a few minutes, but it should automatically download.
17:06And then we can try out the model here.
17:09So here we are using the render prompt template from the previous chapter.
17:13So this one should, let's just run this.
17:17This one should print out how it looks like.
17:20So it's adding this.
17:21"I'm a helpful math assistant" and so forth.
17:25And so you can see this is the original question here.
17:29And then it adds the boilerplate on top.
17:32And at the bottom, this is just like the common benchmark format.
17:36And like we talked about, models can be a bit sensitive to the exact wording.
17:43So, and then next, we are going to develop a function.
17:49Let me just copy it partly here.
17:52We're going to develop a function called generate text stream concat flex.
17:57So it sounds like a bit complicated.
17:59And why are we going to do that?
18:01So the reason is we want a function or we want to modify the text generation function from the previous chapter in different ways.
18:10We first have the base version and then we want to add the temperature scaling and then we want to have the top-p filtering.
18:17But we don't want to, let's say, rewrite everything, like have a huge different function for each of those.
18:23And then it becomes harder to compare.
18:25So we have this one function that we are going to use to generate text.
18:29And this is based on the previous function we had.
18:35So let me see where I can find it.
18:41So I think it becomes more clear once we look at this.
18:50This was, I think, in chapter two.
19:05So we are essentially just modifying this slightly.
19:15Well, let me, let me just go to chapter three one more time.
19:35Yeah, let's, let's use this one.
19:37And so here we have the generate text stream concat function.
19:42So if I just paste it here for now and then I change it to flex, there are now a few modifications I want to make.
19:52And so first, let's add something called generate func.
19:58And so this one, we will set it to None.
20:00And if it's set to None, we will say if generate function is None, then just reuse our existing function here.
20:20So in this case, nothing is happening.
20:22So this will, the flex function will now behave exactly like the previous function we had because it will just use this one here.
20:35So before we hard-coded this. Now we can change it, though, so we can later on, say, instead of, let's say, instead of this one, we will have, let's say, a temperature version of that with temperature scaling or a top-p filtering version of that.
20:55So the goal is to kind of make, make this function a bit more general so that we can plug in other functions. And just also to recap, what this whole function is doing is it's putting together the tensor.
21:07So assuming this prompt is just a string, it will make it a tensor format, of course, first encoded into integers, the token IDs, and then make it into a tensor format and then just iterating over the generated tokens and collecting them here.
21:25And so then both printing it if verbose is True and then also appending the newly generated tokens and then turning them back into text.
21:37So if I go back here, it's essentially, you can see it's just the function that we use to generate the text.
21:43Now we just slightly modified it.
21:45Now we maybe want the function to accept some arguments.
21:50For example, later we will have some settings for the temperature.
21:54So we want to do some generator, oops, generate keyword arguments that we want to accept here.
22:04Yeah, so that, and if we have it here, we also probably want to have it in the function itself here.
22:15So we are just, I mean, you can have a comma here. Doesn't matter.
22:18Some, some days I prefer having a comma on every line.
22:21Sometimes I'm like, yeah, that looks kind of weird.
22:24I remove it, but it doesn't really matter in Python.
22:27And so now we have a modified text generation function where we can later on plug in a few things.
22:33But so first, before we continue and do something with a model and everything, let's just make sure this still works.
22:40So before, like I said, we use the concat.
22:43Now we have this concat flex that just behaves the same way as before.
22:50I forgot to do something here.
22:53Let me maybe rerun everything.
23:06Do we have a little bug here?
23:09Let me see what happened.
23:11Oh, I think what happened is this one was loaded on the MPS device, and then I defined CUDA.
23:19So I actually should put this here and then it should work.
23:26Yeah, so it is printing out text.
23:29So this also highlights it's important to make sure that the tensor here, this input IDs is on the same device as the model here.
23:41So they have to sit on the same device.
23:43And before I had one on the, this one on the MPS device because I added the device down here where it was already loading it.
23:53And then you can see it will give you an error.
23:59So it will tell you that it's on a different device here.
24:06So basically the mismatch is between the weight and the input. They're on different devices.
24:11So one way would be, of course, moving this up or what we can do is we can also do model to device manually here.
24:21Doesn't really matter.
24:23This should also work now.
24:27OK, next, let's talk.
24:30So, by the way, this is a wrong answer, though.
24:33So the correct answer is 83.
24:35And so later in this chapter, we will, of course, the goal is to improve the model to give the correct answer.
24:40So next, let's talk about generating better responses.
24:44So hopefully getting the correct answer.
24:46And we will first start with technique one.
24:49We will start with...
24:54Where is it? With this chain-of-thought prompting where we are just going to slightly modify the prompt and see if that helps us with the correct answers.
25:04And so what we can do is we can take our existing prompt.
25:12So which is hard to read in this format, which is like "you are helpful math assistant" and so on and so on.
25:19And we can also have something like, let's say, "let's think step by step."
25:25And that's like a popular, that's not ideal. Let's do two new lines.
25:32So that's a popular classic prompt modification.
25:36So there was a paper about that.
25:39"Large Language Models are Zero-Shot Reasoners," where they proposed this type of modification.
25:45And they saw in some cases a big, big jump in accuracy.
25:50I should say newer models, like if you use something like ChatGPT nowadays with GPT five point six or GPT six Astra, you won't notice a big jump because the model is, has been fine-tuned so well that it doesn't need this extra help.
26:05But for base models, like this prompt modification still does a lot.
26:10So it kind of helps a lot.
26:12And so, yeah, you can, can see they are adding this.
26:15"Let's think step by step."
26:19I mean, I think I don't know if that did that in this paper, but I saw papers like a few years ago.
26:24So this is an older paper, three, four years old.
26:26I saw papers where they compared also the exact wording.
26:30But again, this is kind of like empirical.
26:32It really depends on the LLM.
26:34So you can try out different things.
26:36In my case, I found "explain step by step" worked a bit better for this.
26:43Yeah, a Qwen three point, Qwen three zero point six model.
26:47So using something like that worked pretty well.
26:51So let's try out this CoT, chain-of-thought, prompt modification and generate now the response similar to before, but now using this modified prompt.
27:10So we replace this and this.
27:17Yeah, I did not rerun.
27:18This happens all the time when I use Jupyter Notebooks, and especially if I'm talking while I'm coding, I'm not good at multitasking, like talking and coding at the same time.
27:29But we can see now that this actually does do something.
27:34So you can see before the answer was pretty short.
27:37And now the model is actually reasoning through the answer, like doing the intermediate steps.
27:42And it's almost like a reasoning model.
27:44So you might think, OK, why do we need to train a reasoning model if we can do this modification and it already behaves like a reasoning model?
27:51Well, the answer is training it further as a reasoning model will make it even better.
27:56So I will have some numbers later to show you.
27:59So this is just like a simple modification.
28:02But in this case, it already is sufficient to give the correct answer, which is 83, which is kind of cool that with such a simple modification, it works.
28:12And like I said earlier, I will also show you the MATH-500 benchmark with this prompt modification later on, once we have implemented all the models and then we can, all the methods and then we can compare them.
28:26But yeah, let's go to the next step.
28:29So that would be the first step towards implementing the main inference technique for this chapter.
28:36So if I go back here, the main technique here is the second inference scaling method, the self-consistency sampling.
28:43And for that, we need two things.
28:45We need temperature scaling and top-p filtering.
28:48And these are very fundamental techniques that are also used in practice.
28:54So, for example, if you are using ChatGPT today, like behind the scenes or any other major inference framework, they are always using something called temperature scaling and top-p filtering, sometimes top-k filtering in their toolkit, like in their function that generates text.
29:13They are just very essential modifications to the sampling.
29:18And so with the sampling, what I mean is what we implemented previously in this function.
29:26So we are going to modify this function.
29:31So previously, we always selected the most likely token with torch.argmax.
29:36Now we are going to modify this and we need that for the self-consistency.
29:41So one thing at a time, though.
29:43So let's, let's do that.
29:45Let's talk about the temperature scaling first, then the top-p filtering.
29:49And then we will put it all together to modify our text generation that we will then use for the self-consistency sampling.
30:01First, temperature scaling. And the motivation behind it is that we add output diversity.
30:10So right now, every time, let me just use that one because this one takes too long to generate.
30:15But every time we call this, it will always give the same response.
30:21And if you use something like ChatGPT or many other LLMs online, they will always give you different results every time you ask it.
30:32So it will not be like a totally different result, but some things might change.
30:36So if you, let's say, rerun this in ChatGPT, you will see the explanation is a bit different every time.
30:42And that has something to do with this temperature scaling, which we will see in a few moments.
30:48So for that, instead of using this very long prompt here, I have a simpler prompt to illustrate this.
30:55So first, we have to take a step back, though, and understand a bit better how the LLM generates the next token.
31:05We already hinted at this in part two, and I have also a very detailed explanation of that in Build a Large Language Model from Scratch.
31:13But since I don't want to like send you to other places and then, hey, come back later once you read this or so forth, I will just here recap essentially how that works.
31:25And so for that, we will use a simple example prompt that is a bit simpler.
31:30And the prompt is "the capital of Germany is," and we want the LLM to generate the next word.
31:35The next word, of course, should be Berlin.
31:38So here in this case, the model will generate Berlin.
31:41And here we don't want actually any diversity.
31:44So we don't want the model to say Hamburg or Munich or something like that, because that would be wrong.
31:50But in practice, how can I say that? It really depends on the task.
31:55So, for example, if you use LLMs to write email responses or something like that, we want some, let's say, some mild diversity.
32:03We don't want the LLM to kind of like reuse phrases from the training data or anything like that.
32:10We want the model to be a bit creative there.
32:12And so in that case, we want a bit higher output diversity.
32:17If we ask knowledge questions or science questions, we maybe want a bit less output diversity.
32:22So there is a temperature parameter that we can change.
32:25Usually it's between, let's say, point three and point seven.
32:31So point seven would be on the higher end, more output diversity, like for more like creative writing or something.
32:38And point three, more like for coding, math and so forth.
32:41But ideally, we would also want the LLM behave differently on each prompt in practice.
32:50Yeah, that is a bit trickier because then we have to kind of change it all the time.
32:54So typically, models are trained in a way that they are very confident in answers like that, even that if we have some temperature that is a bit higher, it will still give us most of the time Berlin.
33:06But we will see a few examples of that later.
33:09So right now, I don't want to, let's say, be too theoretical here without showing you anything.
33:16So let's go to the section and see if I have a nice figure there.
33:22So again, the goal is essentially we want to implement this self-consistency sampling where it's a form of majority voting, where we want the LLM to generate different answers and then select the most frequent answer.
33:38And in order to do that, we need this temperature scaling and sampling.
33:43And so here we are going to take a look at how the LLM generates the next token.
33:50So this is just like a recap of the regular text generation in an LLM with the prompt that I just showed you.
33:57"The capital of Germany is."
33:59And so the first step is generating the tokens like tokenizing this into an integer representation and then feeding the tokens to the LLM.
34:09And then for the next token, the one that we want to generate, we have a token distribution over the vocabulary.
34:16And so here in our case, our LLM has a vocabulary size of hundred fifty thousand, hundred fifty-one thousand something.
34:25And here I'm only showing you one hundred because I try to plot it.
34:29And if I have one hundred fifty thousand columns, it was so small you couldn't see anything.
34:33So I just truncated this to like a hundred tokens here just to visualize it.
34:38And you can see the token corresponding to Berlin achieves the highest score.
34:43And according to our argmax, we would select the highest score always.
34:51So in our argmax here, we would select this highest score that corresponds to Berlin to get the token index and convert it back to the string.
35:02This is what we talked about in part two when we implemented the simple text generation function.
35:08This function here, that's exactly what's happening there.
35:11Now we are going to modify this a bit.
35:13But first, let's, let's just recode this.
35:16So we have this as a working example here.
35:19So that's what we did before.
35:22First, we are converting the input text into token IDs.
35:26So this would correspond to the first step here at the bottom.
35:32So we have these token IDs and then next we will put those into the LLM.
35:39So we use the inference mode to just, you know, save memory so the LLM is not too wasteful.
35:45The LLM will select or generate the next tokens.
35:49So let me run this first of all without this.
35:53And so you will see the model will generate five output tokens.
35:57But we are only interested in the new token, the last token, as we talked about before.
36:01So we are truncating this to the last token, which is the new token that we want to generate.
36:07And so if this part is confusing, I would maybe suggest going back to part two where we talked about why that is.
36:15Because the LLM always generates the next token shifted by one position.
36:20So here we selected the new token, which is the next token in the text sequence.
36:26And this one has a size of one times 151,936.
36:33So one is here the batch dimension.
36:35We don't have a batch, so it's just one.
36:37And this is the vocabulary size.
36:39Again, here it's truncated to 100 tokens instead of 150,000 tokens because otherwise it wouldn't fit.
36:46But it's exactly here the next token logits.
36:49This is exactly the score distribution we are looking at here.
36:54That's the same thing.
36:55So you can see it's dotted out because there would be too many values.
36:59And then once we have that, we can look at what is the highest vocabulary index with the highest score.
37:06So we use argmax here to find the highest score.
37:11So here, for example, that's 19,846.
37:16And we can also get the, we can just print it out and decode it back into text.
37:23So, so here we converted back to Berlin.
37:27And next, let's just plot this.
37:30So just recreating this plot.
37:33There's a lot of Matplotlib boilerplate code here.
37:37I'm just pasting it.
37:39Personally, I'm not the best Matplotlib writer since maybe 20 years ago.
37:44I've been always like doing a plot, spending a lot of time on it.
37:48And then I built my own little gallery.
37:50So every time I need a plot, I go to my gallery and copy-paste my previous code, make some modifications.
37:55But anyway, so this is just like a plotting function where I'm visualizing the scores and have this little arrow annotating this.
38:04So we have these values.
38:06So the values are the next token logits.
38:09And you can see they are negative, positive around zero.
38:12And the tallest, the largest value here is Berlin.
38:16So that's our highest logit value.
38:20Next, our goal would be using the temperature scaling to change the score distribution a bit.
38:28The reason why we want to do that is we will use a sampling function instead of using always the highest token.
38:35We will, let's say, consider the highest, multiple highest tokens and then proportionally to the magnitude of these values, select those.
38:46So that means if we would repeat this a thousand times, we most of the time would sample Berlin.
38:51But some of the other times we would also sample some other values just to add some diversity.
38:56So for that, we will use something called temperature scaling to change this distribution.
39:03And temperature scaling is essentially very similar, very simple.
39:07It's if we use the next token logits, it is essentially just dividing this.
39:13If we have a temperature here, let's say we have a one point zero, we're just dividing this by the temperature.
39:20So if we have a temperature of one, nothing happens.
39:24So if I just have them here as the reference, so you can see they are exactly the same.
39:30If I make the temperature bigger, then we will divide by five.
39:35This will be five times smaller.
39:37If I use the temperature, let's say zero point five, this will be two times larger because dividing by point five is like multiplying by two.
39:46So this will be then something like sixteen or seventeen here.
39:50Sixteen, sorry, seventeen point six.
39:53So, yeah, so in that way, we can change the temperature.
39:56And so the smaller the temperature, the higher the values get basically.
40:02And the higher the temperature, if I do something like this would be crazy large.
40:08We have very small values.
40:10And so normal temperature values are usually between, let's say, point three and, I would say, point seven.
40:19So this is like on the max side.
40:21Sometimes you can go maybe two point nine.
40:23But I think this is already very large in practice.
40:26And this one is also very common.
40:29This will make the values, of course, also more peaked.
40:34But usually it's very rare to go above one because then you will have like too much noise.
40:41But let's, let's get to that later.
40:45So first, let me just define a scale logits by temperature.
40:50It's exactly what we talked about.
40:52It's the same as before, just like one condition we are adding that we don't allow negative values just, you know, for safety here.
41:00And then I have a modified plotting function here where I'm using plot instead of the bar plot to plot the temperature.
41:08And so let's just run this to show you this.
41:11And so here in blue, that's exactly the same as before.
41:15But before I had a bar plot, which is more natural for plot like this because they are discrete values.
41:23So ideally, we want each token ID as a bar here.
41:27It's just easier to see.
41:28But I tried doing this for different temperature settings and it was very noisy.
41:32It was really hard to read this plot.
41:34So I changed it to like a line plot here, even though it's not like technically correct because for discrete values, we don't want a continuous plot.
41:42But it just looks easier here to look at.
41:45Now, at blue, in blue, we have the original logits we looked at before.
41:50And in the orange dashed line here, we have a temperature of point five.
41:55And you can see that the peaks are larger now, right?
41:58So this is what I explained before.
42:00When we make the temperature smaller, we have like these larger peaks.
42:05So there's a higher peak.
42:06But we can see that the largest value is still the same one.
42:11We just have a higher peak here.
42:13And then if we have a smaller temperature point, sorry, a larger temperature of 5.0, we make everything a bit flatter.
42:20So everything that's more randomness, everything is more like evenly distributed, essentially.
42:26So this is essentially the effect of temperature scaling.
42:30And now that we have this temperature scaling, the next part would be doing...
42:41So we rescaled everything.
42:42So we get the next token scores.
42:44We rescaled them with the temperature.
42:46The next step would be converting them into a probability distribution and then sample from that distribution.
42:52So the scaling was just so that we can influence the probability distribution.
42:57So let's go to the next step here.
43:01I'm sampling from a probability distribution.
43:04So that's this part.
43:06So let's add it here.
43:11And first, for this, we will again apply this temperature scaling.
43:18Let's just use five.
43:20I will explain you in a moment why I use such a large value.
43:24And then we are using the softmax function.
43:27So the softmax function is a very popular function that converts scores into a probability distribution so that the values sum up to one.
43:35And softmax functions are usually used in the output layer as like for training, like as a loss function before the loss function, basically.
43:46So that's like a softmax function or it's also used in the attention mechanism to compute the attention scores so that they sum up to one.
43:54So it's a popular function in machine learning or deep learning and LLMs.
43:59So I think I covered it in part two.
44:02And if not, I do have, let me see.
44:05I will get to that later.
44:07I do want to just go to my website here quickly.
44:11So if you're interested in, so my website is sebastianraschka.com.
44:16If you're interested in more details in terms of how the softmax function works and how it's derived, I did have a course when I was a professor at UW-Madison.
44:28I created a deep learning class where I also recorded the videos for my YouTube channel.
44:35And it's, I mean, the videos are a bit older, but probably not the highest quality.
44:43But everything should be there.
44:46So I do have, I have lectures on that.
44:52So if you're interested, there are videos and slides to just, you know, coming from logistic regression where this all comes from.
45:00And if you are interested, you can see how the softmax relates to logistic activation functions and so forth and how that works.
45:13So that's like the formula behind it, behind the softmax.
45:17But in a nutshell, all you need to know for this video is that this is a way we can essentially convert the scores into a probability distribution or make them sum up to one, essentially.
45:30So if I run this, let me say before, if I have something like this, this is, no, it's a probability. Sorry.
45:39I wanted to start with the logits.
45:43So if we have something like that, it could be any value.
45:46It's just summing up the positive and negative values.
45:49Now, if we use the softmax and look at this here, this one should sum up to one and all the individual values are between zero and one.
46:01So all the values here inside are between zero and one.
46:04And so this is a very scientific notation.
46:07It's a very small number, very close to zero.
46:09It's like 0.0003 something and so forth.
46:13And so one value will be a bit larger here and everything is pretty close to zero.
46:18We can actually plot this.
46:20So if I go here, where's our bar plot?
46:24I use our bar plot here.
46:27We can plot this here.
46:32So let's do it like this.
46:39And then set this arrow to False.
46:43And let's say for the y-label, we use probability scores.
46:51So, yeah, you can see this is the probability distribution.
46:54Now, like the values after the softmax, we have one large value.
46:58It's still at 19,840, I think, six, what it was.
47:02That's still corresponding to Berlin and the other ones are smaller.
47:06I use the crazy large temperature here because it's a temperature you would never use in practice.
47:12But if I use something like this, you will see there's not much to see.
47:17It's all so close to zero that you won't even see anything here.
47:21So I just did it here for visual purposes that we can see some of the other values, essentially.
47:27And yeah, so the next token probability, we can look it up.
47:32So how high is this value? I think it's around 0.0003 or something like that.
47:38So let's just look that up. Yeah, it's 0.0003.
47:42And the core idea why we do that is that we can now use the multinomial function in torch, which will sample from this distribution using the values as a weight, which means this token will be sampled proportionally more with higher probability than the other ones.
48:04So you can think of it as a probability distribution.
48:06So there's now 0.03%.
48:11I have to think because it's 0.03% probability that we sample Berlin, but it's even smaller probability that we sample some of the other ones.
48:22So let's just run this, and notice I'm using CPU here.
48:27That is because on MPS, I noticed, yeah, it works fine, but it gives you sometimes different results because like it's not as good on small numbers as some maybe instability or some difference in terms of how the Metal Performance Shader library works on Mac, that if you want to get the same results, the same sampling as I do, you have to use the CPU.
48:50In practice, it's pretty cheap and pretty fast.
48:53So this executes in like a fraction of a second.
48:55So don't worry about it.
48:56If you're training the model or like later on, if we're using it for sampling, for example, in the reinforcement learning chapter, yeah, you can replace this to make it a bit faster.
49:07But for now, it's just fine to just reproduce that.
49:10And we see we get that token here.
49:14Let's see, tokenizer decode what that is.
49:20It's, yeah, it's not Berlin.
49:23It's sixty thousand, sixty-five thousand or something like that.
49:27Can maybe if I run this again, by the way, I will get a different result.
49:32So because I have a random seed here, so every time I run this, I get a different result.
49:38So in this case, it's now, it's a little tractor emoji.
49:43So, yeah, so with three percent chance, we will sample Berlin.
49:49And with very small other chances, we sample other tokens.
49:54Now, let's just repeat or make this a bit more, I guess, scientific here to just repeat this a few more times and sample thousand samples and take a look at the distribution.
50:07So here again, this is the same code as before.
50:10We are sampling with replacement.
50:12We draw, we repeat this essentially with a for loop a thousand times and look at what are the frequent tokens that are sampled.
50:19And here I'm using bincount and then just print how often each token is sampled.
50:28And I'm setting a threshold so that in by default, every sampled token, we have a list of thousands of tokens that comes out as shown.
50:36But in practice, we can also change it to a higher value.
50:40So to filter out some noise.
50:42But let's start here.
50:44So let's just run it as is and it will just repeat what I've done here, like doing this manually.
50:51It will repeat this a thousand times just for simplicity.
50:57And so, yeah, we can see that.
51:02So, by the way, one more thing.
51:05We only count if the value of the count is larger than the threshold.
51:09So if we set it to zero, this will be a list of a thousand tokens here.
51:13But that's really hard to look at.
51:15So we will only print the tokens if they are sampled at least two times.
51:20We can see that it's actually no, no Berlin there, which is surprising.
51:24And that is because the temperature is so high that other tokens are sampled.
51:30Remember, we have one hundred fifty-one thousand tokens in here.
51:33And even if Berlin has the highest probability, it's still only a probability of zero point zero zero three.
51:38So it's still rare to sample that one.
51:41So what we can do is we can increase this or, sorry, decrease this to increase the probability here.
51:48Right. So now you can see it's close to 35 percent.
51:52Right. So now the probability for the next token is almost 34 percent.
51:58And so what we can do is essentially change the temperature to influence the probability distribution to make it more likely that the most likely token is sampled.
52:10So if I repeat this, I can now see that Berlin is out of thousand times is sampled four hundred thirty-five times.
52:18So technically, if you have an infinite or not infinite, but like an infinite large sample, if you increase the number of samples, it's just that it should approach thirty-five.
52:32And I should not. This is not a probability. It's just a rescaling.
52:35But it should basically, sorry, it should give us, we can actually do it like this.
52:46Thirty-five here. It should give us this probability if we sample infinite many times.
52:53So this is forty-two percent.
52:55So let's just sample a very large number.
53:01So maybe set the threshold higher.
53:10Yeah, it approaches this number like four hundred twenty thousand.
53:14Right. This would be forty-two percent.
53:16It's approaching this number here basically.
53:21But don't want to confuse. It was just like a side project or side experiment here.
53:25So we are looking at Berlin still being now the most frequent token and we see some other tokens are frequently sampled as well.
53:33We can see sometimes it also samples Munich and sometimes it also samples Hamburg.
53:38Rarely, only three times, but it samples them because they're also, I would say, almost likely tokens, are both also cities in Germany.
53:47And of course, they are not the capital of Germany.
53:49The LLM makes mistakes here, but it's not super far off.
53:53You know, like it's not Paris or London or something like that or some other word like we had before, like mistress or the tractor emoji.
54:02However, we do see that we are sometimes having these underscore things here.
54:07So why that? My guess is, is because if we have a prompt like, where was it?
54:15I can maybe just print this out at the bottom.
54:18Example prompt. Easier to see.
54:20So if we have something like that, I can imagine in the training data, sometimes this shows up as, as a quiz question, for example, like a multiple-choice question, where someone has like a, let's say, take-home exam in elementary school or high school where it's something like that.
54:39And the model just learned to put underscore CS, the most likely token to make it a quiz question or something like that, which is, I think, why sometimes something like this shows up.
54:51So now that we have the temperature scaling, let's modify our text generation function to add this support for the temperature scaling so that we can also use that and later add this top-p filtering.
55:05So for that, we are reusing the generate text basic stream cache function from chapter two.
55:13So let's copy this over.
55:16And yeah, so it's not new anymore.
55:21This is new in chapter two, but not here.
55:24So what we're going to do is we are now adding temperature support.
55:28So let's change the name from basic to temperature, add a temperature parameter here, let's set it to zero by default.
55:36And instead of doing the argmax, we are now going to the, going to add the temperature scaling and the multinomial sampling.
55:45But let's keep this for a second and say that if the temperature is None or if the temperature is zero, then it is still doing our greedy sampling here.
56:09Otherwise, if, let's say, if this is not true, if we have a temperature setting that is larger than zero, for example, we are going to rescale our logits, do the softmax and then the multinomial sampling.
56:24Instead of retyping this, which is a bit error-prone, let's just copy-paste it here.
56:32So we are doing the scaling that we're, like we did before using the temperature here.
56:38Then we normalize the scores into the probability distribution so that they sum up to one and that we can then use them in this multinomial one to do the next token.
56:48And there's one more thing I want to do is because I'm changing things here to the CPU.
56:55Next token will be on the CPU.
56:57And if our model is on CUDA or MPS, that will create some problems.
57:03So let's add this line where we get the device of the original input here, the original input.
57:10So we store it here and then we assign this to the next token before, like after we have the CPU here.
57:18So we convert it here.
57:21We don't actually technically need to store this.
57:23We could also technically do it directly like this.
57:28But it's usually not a bad practice in case something happens here so that we have this original value and then put the next token there, too.
57:36So now that we have this modified text generation function, let's try this out in practice.
57:42So for that, we are going to reuse our flex function from the beginning, the one that we use to generate text.
57:50And like I mentioned before, nothing goes to waste here in this course.
57:54We are reusing things.
57:56And one reason is, of course, of course, code efficiency so that we don't duplicate code.
58:02It's easy also to debug things and write unit tests and so forth.
58:05But the other one, I think, is also for educational purposes.
58:09So, for example, if we get to reuse the same function, you know that there is nothing that nothing else has changed, right?
58:16So it's kind of like helping, hopefully, to connect the concepts.
58:20So what we are going to do is we are copying this.
58:24And then here, just change this to temp.
58:31And then, yeah, do a temperature.
58:38Let's do temperature.
58:40Let's start with zero.
58:42And so if we have zero, this should, oops, manual seed, should work the same as before, right?
58:51Because I have a typo here, temperature.
58:55This is why I like copy-pasting and self retyping things.
58:59So we know that this one will do the greedy decoding that we did before.
59:05So this is exactly now the same as before.
59:08But now we can, let's say, change the temperature and get some different results.
59:12And so with that, it allows us to get to generate multiple responses.
59:17So, for example, if I do this again, every time it will be different.
59:21And we want this because we can then implement something for majority voting.
59:26So ideally, we want the model, of course, to generate 83.
59:29But we will get there.
59:31So one improvement to this is to get rid of some of the low-probability tokens.
59:37So if I go up again, you can see sometimes it has these tokens that are low probability.
59:44They are rarely sampled and they are wrong.
59:46And so it would be ideal if our function only focuses on, let's say, the most likely one.
59:52It never gives us anything that is really bad, right?
59:55Because we don't want the model to hallucinate, for example.
59:58Or at least we want to reduce the hallucination rate.
1:00:01And so we can get rid of some low-probability tokens.
1:00:04And that's like the idea behind top-p sampling.
1:00:09So top-p sampling is essentially a technique that goes on top of the temperature scaling.
1:00:15So I think I have somewhere a figure for that.
1:00:19So we finished all that.
1:00:22Let me see. Scroll down a bit.
1:00:26Top-p filtering. And so there are a few steps for that.
1:00:29So first, sorting in descending order, meaning we are rearranging the probabilities in descending order so that we can compute a cumulative sum.
1:00:38So, for example, we can sum up all the probabilities and then apply a threshold.
1:00:43So, for example, if the probabilities, when they are added up, exceed, let's say, 80 percent, we do a cut and get rid of everything after.
1:00:52And then we select those top tokens and then we normalize that they sum up to one.
1:00:58So it looks like a lot of stuff is happening here, but it should hopefully be relatively straightforward if we walk through this.
1:01:08So instead of using the prompt that we had before, which generates sometimes a very long answer or a lot of logits.
1:01:15So, for example, above we have the vocabulary with hundred fifty-one thousand logits.
1:01:20It's kind of like hard to work with because what we would have to do is we have to sort all those.
1:01:26Even with the one hundred here, it's hard to see. It's kind of hard.
1:01:31So let's just use for this section toy logits.
1:01:35So here assume these are the logits for the next token, let's say like the Berlin token.
1:01:42But instead of looking at hundred fifty-one thousand scores like the...
1:01:50Something that we see next token logits. So these were hundred fifty-one thousand.
1:01:55And so this is even dotted out here. It's like really hard to see.
1:01:59So let's just assume this is a simpler vocabulary with only one, two, three, four, five, six, seven, eight, nine, ten entries, a vocabulary with ten entries just to make it easier to work with.
1:02:11And so this would be just step one, getting the logits from the model.
1:02:17So this is essentially just what the model supposedly outputs.
1:02:21And then we apply the temperature scaling. This is similar to before.
1:02:25So now we have the scaled logits, should define those, of course.
1:02:31And then we will, as before, apply the softmax so that they will sum up to one.
1:02:37So we can then just double-check so they should sum up to one after the scaling, because those don't.
1:02:53And they will also be in the range between zero and one.
1:02:57So we can also do min, min, yeah, zero point zero three one and max is forty-five percent.
1:03:10Yeah. And so let's plot them, maybe to just make that a bit easier to see.
1:03:15So we are plotting now the rescaled logits here just to see how they look like.
1:03:21And so you can see they are between zero and one.
1:03:25And the highest one is, what did we say, point four five here at position four.
1:03:32Now, these are the vocabulary, right? Ten tokens in the vocabulary.
1:03:38What we do next is we sort this. So we sort this vocabulary and then we compute the cumulative sum.
1:03:50So the cumulative sum is summing up. So first descending order means we will start with the largest one and then this one and then this one and then this one and so forth.
1:04:01And then we are summing them up. And it's kind of really hard to explain this in words.
1:04:06And I think it's just much easier to see this in the plot.
1:04:09So let me actually just add a step plot here and then you will see how that looks like.
1:04:14And so here we have the sorted probabilities.
1:04:18So these are just now sorted. And here in orange, the step plot, this is the cumulative sum.
1:04:24And so, for example, here this is the first token plus the second token.
1:04:29And then here this one is the first token plus the second token plus the third token.
1:04:35And here this is the sum of all the first four tokens, the sum of all five, five first tokens and so forth.
1:04:41So we're just summing it up until we have the sum of one here.
1:04:45And the goal is to define a threshold.
1:04:48For example, let's say point eight could be the threshold and then everything above the threshold we throw out.
1:04:55So there are multiple ways we can apply the threshold.
1:04:59So here I'm doing it with a keep mask.
1:05:03So, for example, we can say so if I go for the cumulative sum here.
1:05:08So if I say everything, if I have a top-p of point eight, everything after point eight gets masked out, for example.
1:05:19So here this is bigger than point eight. So everything here gets masked out.
1:05:24So, for example, what I'm doing here is saying, is this greater than point eight?
1:05:31And if it's greater than point eight, it's False.
1:05:34And so we are checking whether the values are True if they're smaller than point eight, and if they're bigger than point eight, they are False.
1:05:42And in PyTorch, we can add these Boolean values, right?
1:05:47So we can say True, True.
1:05:51This will give us two. So one plus one.
1:05:54And the False are counted as zeros.
1:05:57So False are zeros.
1:06:00So they don't add anything.
1:06:02And so with that, we can then just count how many tokens we keep.
1:06:06And yeah. And so here this is essentially saying we only get to keep two tokens here.
1:06:13A more common way to use top-p filtering is actually, oops, I go back here, is to also include the token that is just above the threshold.
1:06:28So we have the tokens, this and this one below the threshold.
1:06:33And then it's a common convention to include the next token above the threshold.
1:06:38Now, in this case, it makes a huge difference, right?
1:06:41Because we only have two tokens.
1:06:43But if you think about a vocabulary of hundred fifty thousand entries, it doesn't really matter that much.
1:06:49If we select a thousand tokens or a thousand and one tokens in practice, you would probably not notice a difference.
1:06:55But it is a common convention to also include the token that just crosses the threshold, for example.
1:07:03So in this case, we are keeping three tokens because one, two, three, the third one that it just exceeds the threshold.
1:07:11That's like the most common convention.
1:07:13So I'm going through this a bit quickly because it's just boilerplate code.
1:07:18It's just code that I use to visualize this.
1:07:22You don't have to kind of memorize this or know the details.
1:07:26The interesting point here really is just to show you that with the filtering, we cut off tokens that are exceeding a threshold here.
1:07:36And so we are only keeping the most likely tokens.
1:07:39If I go back to the unsorted ones, what we are doing is we are keeping this one, this one and this one.
1:07:45So token vocabulary index four, nine and eight.
1:07:49So those are the ones that we keep for the multinomial sampling and we get rid of all the low-probability ones.
1:07:56So I could walk you through the code, but since it's already a long video and we want to get to the most interesting part, the self-consistency sampling, let me just leave that here.
1:08:08So this is essentially code that zeros out everything that is above the threshold, excluding this one that is allowed to be above the threshold.
1:08:18And then we are mapping this back to the original sorting order.
1:08:21So torch.where is just like a simple function.
1:08:24It says everywhere the mask is True, keep the values and otherwise replace them by zeros.
1:08:34So if we go here...
1:08:37Oops, what did I do?
1:08:42So you can see it only keeps the first three tokens and then so all the ones that are True and then the other ones get replaced by zeros.
1:08:52But this is not in, this is in this order, but we want them back in the vocabulary index order so that we can convert them to the token IDs later.
1:09:01So this is just a step to map this to the original sorting order.
1:09:07OK, one more step after now.
1:09:12So what we did is we crossed out all the other tokens here that are not the top three tokens.
1:09:17One other step what we want to do (it's not technically required, but also common best practice) is to renormalize the tokens so that they sum up to one again.
1:09:29Because right now, if we can actually give this a try right now, they should be smaller than one, of course, right?
1:09:37So we are at point eight and so we can just renormalize this dividing by the sum.
1:09:44We could also use the softmax here, but that doesn't really matter.
1:09:48So we are just making sure they sum up to one again and so that we can then have the multinomial function.
1:09:57And we can then say it's just like easier to read.
1:09:59We can then say, well, fifty-six percent of the time this token is sampled, ten percent of the time this token is sampled and thirty-four percent of the time this token is sampled, for example.
1:10:11So that's like why we want to renormalize this.
1:10:15OK, so we now have this top-p filter and the top-p filter essentially means keep the top tokens and remove all the other tokens.
1:10:23And so the next step now is to modify our text generation function to add to this top-p filter.
1:10:32So for that, let me go up again and copy-paste our text temp here and make this modification.
1:10:44So here and then let's paste this by top-p and then we probably also want to top-p here.
1:10:53Let's just set this to None by default.
1:10:56So by default, it's not doing any top-p filtering.
1:11:00And then if we look at this, we have the softmax and then we have the multinomial sampling.
1:11:06So the top-p filtering happens between those two, right?
1:11:09So we want to get rid of the tokens, the low-probability tokens before we sample.
1:11:15So we add this additional top-p filter step.
1:11:19Now, the top-p filter is a function that is essentially carrying out these steps here.
1:11:27Right. So I could technically, it's a lot of work.
1:11:31I could scroll up there and get all these steps, for example.
1:11:35Define a top-p filter function and then copy all these steps here.
1:11:44But it's a lot of work. So I do already have this function for you.
1:11:49So let me copy this here.
1:11:51And so this top-p filter function is exactly the steps we did before with the top-p filter.
1:12:00The top-p filter function is exactly the steps we did before with the filtering out, the torch.where, mapping it back to the original order and renormalizing to one.
1:12:10I just copied them together here to make it a bit faster so we don't have to scroll up and down and copy and paste.
1:12:18But yeah, again, the new stuff is really just this one here where we are applying now the top-p filter.
1:12:24And so let's give this a try.
1:12:27And so for that, it's a lot of scrolling here.
1:12:31For that, we will need our text generation here.
1:12:41And let's change it to top-p here.
1:12:47Anything else? So we probably want to add a top-p here.
1:12:52And let's use a more reasonable value like 0.8.
1:12:57And let's maybe go here and add also reasonable 0.5 here and see what happens.
1:13:0418. So, yeah, it works.
1:13:07Again, it's still not the correct answer.
1:13:10If we sample multiple times, we should get different results.
1:13:14But this is already cool because now we can use this function for the majority voting, the self-consistency.
1:13:23So we are now already here, 4.6.
1:13:27So there's not that much left in this chapter.
1:13:30So it's kind of like the finale here.
1:13:35Now we are going to improve the accuracy of our model, hopefully getting the model to generate the 83, 83.
1:13:43One more thing I wanted to say is it was a lot of work to talk about this temperature scaling and the top-p filtering.
1:13:50And you might like might think like, why do I need all of that?
1:13:53Why do I need to know all of that?
1:13:56I mean, of course, you never have to know all of that.
1:13:58But it is always kind of interesting, of course, to understand how things work under the hood.
1:14:03But also they come in handy if there are new concepts out there.
1:14:07For example, like a few weeks ago, Claude Anthropic had an article that they are going to add watermarking to the generated texts by Claude.
1:14:19And the article was very high-level.
1:14:23It didn't really go into the details. It didn't explain how it works.
1:14:26But if you understand temperature scaling and top-p filtering, especially temperature scaling, you have probably an easier time to understand putting one and one together, how the watermarking works, how it affects the quality of the generated text and how the watermarking can be removed.
1:14:44So I do have an article here on how Claude watermarks AI-generated text based on a lecture I gave a few weeks ago.
1:14:52So now that you understood how the temperature scaling works, I think you might be interested in checking this out.
1:15:02Because I think it's just helpful to know temperature scaling before diving into a topic like that.
1:15:07And as you can see, I'm reusing a lot of the figures here, right?
1:15:11So it's actually relatively closely related.
1:15:14So once we understand temperature scaling, it's actually quite intuitive to understand how the watermarking works.
1:15:24It's quite interesting. They do have some new concepts also added to it, like the tournament sampling.
1:15:30But in essence, if you understand the temperature scaling, it's almost like 90 percent of the work towards understanding how the watermarking works and how the watermarking can be removed and whether it affects the quality of the text.
1:15:45It's just like an example of why it's cool and useful to understand some of the fundamentals, basically, even though it can look like it's a lot of work, even though you might not have to code this yourself.
1:15:58So I think it's still useful to understand these things.
1:16:01Anyways, continuing with the self-consistency.
1:16:04So here we are now finally at the core of this chapter.
1:16:10There was a lot of setup that we required, but we are at the core where we are now getting the LLM to generate the different responses and then apply a majority voting to this.
1:16:19So majority voting means selecting the most frequent answers.
1:16:22So, for example, if the LLM answer is 18, 83, 54, 83, 22, the most frequent answer here is 83.
1:16:32And this works pretty well for math, of course, where we have integer or numeric outputs where we can do a simple majority voting.
1:16:42If we have free-form text or longer answers, it becomes, of course, a bit trickier to do majority voting.
1:16:47So this is not a technique that we can use, for example, when we, let's say, answer emails.
1:16:53If we want, let's say, to optimize our email answer, we can have the LLM generate five emails.
1:16:59But there would be no way to do a majority voting.
1:17:02Like, what is the most frequent answer?
1:17:03That wouldn't work because all the answers will be very different.
1:17:06There are other techniques, for example, best-of-N, where we can have a scorer, a scoring function to score the quality of the answers.
1:17:15And we will get to scoring in the next part.
1:17:18But I should say majority voting is actually incredibly powerful for especially math examples.
1:17:27So it sounds like a scoring function should be better than just majority voting if we have even numbers.
1:17:33But it turns out there were a few studies showing that majority voting is actually even better than using a scorer on top, because scorers can be biased and majority voting is surprisingly simple and robust.
1:17:45And it's also very, very popular in practice when we look at LLM benchmarks.
1:17:49So you may have noticed things like best at three, best at five, best at ten when you look at benchmarks.
1:17:56And that refers usually to the majority voting.
1:17:59So it's a very popular practice on benchmarks, LLM benchmarks, to use majority voting too.
1:18:05And so I do have a paper here I wanted to show you.
1:18:08So this is DeepSeekMath version two.
1:18:11So it's like a flavor of an existing, I think it was DeepSeek version three point two model where they try to develop based on top of that, the best possible math model that can solve very challenging problems, for example, the math Olympiad type of gold medal type of problems.
1:18:31And so it's a very long article.
1:18:35But yes, you don't need to read all of that.
1:18:36But I wanted to show you this particular plot here.
1:18:40And so what they did is they, they try to improve the model.
1:18:46So, for example, they have a proof score here.
1:18:49And if we look at the lower left corner, this dot here would correspond to a, to the simple non-modified model.
1:18:59So this is a model without inference scaling in the top corner.
1:19:04And then they have something like sequential iterations.
1:19:08It's something similar to self-refinement.
1:19:10They are a bit more sophisticated here.
1:19:12So they have a model that generates or scores the answer.
1:19:17And then they have even a second model that judges the judge.
1:19:21So it's like multiple level of judging here.
1:19:23But the judge helps the model to refine itself.
1:19:28It's like a form of self-refinement.
1:19:30And we will be looking at self-refinement in here in the next part in the next video, a simpler form of self-refinement.
1:19:38So for now, maybe ignore the iterations here.
1:19:42So this is what we will be doing later.
1:19:44But what you can or what is interesting here is if we look at this blue graph and then this orange dotted graph.
1:19:52So the orange dotted graph corresponds to best at 32.
1:19:57And so what that means is essentially a form of majority voting where they ask the LLM to generate 32 answers and then do pick the best answer.
1:20:06Like, for example, via majority voting.
1:20:08And you can see it's almost twice the accuracy or performance here.
1:20:12Right. So it's almost most of the time doubling here, even more than doubling the score.
1:20:19And so it's actually a very useful technique to have the LLM or to ask the LLM multiple times and then pick, let's say, the most frequent answer or we use a score or something like that.
1:20:30So it is a very popular approach, like the best out of 32 questions here, basically.
1:20:36Yeah. And so here we are now implementing this where we have the LLM generate multiple answers and then in this case, pick the most frequent answer.
1:20:46So let's get back to the code.
1:20:48Maybe one more thing. Sorry, before I show you the code, one more thing we should say is, of course, our LLM also outputs sometimes text.
1:20:58So we want to, because it's a number that we want at the end for the majority voting, we want to also extract the final text.
1:21:06And again, like I mentioned before, nothing goes to waste here.
1:21:09So we are going to use our final, extract final candidate function from the previous part from part three, chapter three, to get this number out of the generated text.
1:21:21So here what we are looking at is a prompt.
1:21:23The LLM produces different answers, for example, using different random seeds or sampling multiple times.
1:21:29And this is with temperature scaling and top-p filtering.
1:21:32And then now what we are going to do is we are extracting the numeric answer using the function from the previous chapter and then applying the majority voting here.
1:21:40So I will give you the whole function because it is already a very long video.
1:21:45And so I don't want to double the runtime of this video here or the whole part.
1:21:51So let me walk you through it because there's nothing really super new or interesting happening.
1:21:56So first, we are going to reuse our extract final candidate from the previous part.
1:22:03And then in this self-consistency vote here, first, what we do is we sample multiple answers.
1:22:09And this is just to, you know, keep to store the answers.
1:22:13But yeah, what we are doing here is we are sampling multiple answers.
1:22:16And this is similar to what I've been doing here.
1:22:19Let's do for i in range, let's say three to do three answers.
1:22:26I'll run this. Oops.
1:22:30So in this case, almost the same.
1:22:33But like instead of doing this manually in this for loop, I have for loop here where I'm also if there's no seed, it doesn't use a random seed.
1:22:41And otherwise, it increments the random seed by the index number.
1:22:48One more thing I should say, a for loop is the simplest way we can implement that.
1:22:53But then it becomes kind of sequential in terms of we have to wait for one answer to be generated before we can generate the next answer.
1:23:01And we can speed this up because there is no dependence between these answers.
1:23:07We can run them all at the same time, right?
1:23:09So we could, for example, use batching. And in the bonus material, I do have that implemented.
1:23:15So I didn't want to cram it all into the chapter because it's very tedious and a lot of code and maybe not super interesting.
1:23:23But in the bonus material, if you look here, there should be a batched version where I'm doing that, where the answers are all generated in parallel.
1:23:33And then it's actually faster, right? If, if your GPU has enough RAM.
1:23:38So the bottleneck is also, also for this content here.
1:23:41I'm trying to make this work for most people and not everyone has a gigantic GPU with a lot of memory.
1:23:47And so this one is cheaper because now we only have to keep one copy of the model in memory or we have only one model generate one answer at a time compared to a batch where the model has to have multiple KV caches, one for each query.
1:24:04Right. And so that adds more memory.
1:24:07And so this is like the simplest version of how to implement that. Next...
1:24:13Yeah. After we generated the answer, we extract the short answer using the extract final candidate.
1:24:20So we are looking at, first we were looking, step one, generating the answers, and then we are extracting now these answers.
1:24:30We are saving them, the full answers and the short answers, so that we can inspect this.
1:24:34There's a simple progress bar. We could also use a more sophisticated one, but just to keep the number of dependencies low.
1:24:41And then here we are counting using the built-in Counter in Python.
1:24:47It's like a standard library thing. We are just counting the frequency of each answer and then pick the most, like the most frequent one.
1:24:57So here this is essentially picking the most common one.
1:25:02If we have something like multiple ties or something like this where we have a tie between the most frequent ones, it will pick the one with a lower index.
1:25:16But we can also implement tiebreakers based on scores or other methods, for example.
1:25:22But that is, I think I have some exercise suggestions for that.
1:25:26So this would be an exercise, especially interesting after the next part.
1:25:31After we implemented the log probs scoring, the log-probability scoring, might be interesting to come back to tiebreaking here.
1:25:40OK, so that is our function and it will also return a dictionary with the full answers, the short answers and so forth.
1:25:51So we can then also analyze this if we are interested in that.
1:25:55So, yeah, let's run this. So running this should be pretty quick.
1:25:59I'm only using five samples here, a temperature of point eight, top-p of nine, random seed and showing the progress.
1:26:09Yeah. And so we are back also to the prompt, to the math prompt.
1:26:12By the way, we previously talked about what is the capital of Berlin. This was just for demonstration.
1:26:18Now we are back to our math prompt here, I should say.
1:26:24So "you're helpful math assistant" and so on. And let's just generate this and see what happens.
1:26:29So it should not take too long.
1:26:32It always depends on how many tokens the LLM generates. If it's a short answer, pretty quick.
1:26:37But we can tell this already takes some time.
1:26:41So the first one is definitely a long answer because it took much longer than this one.
1:26:45But yeah, it successfully extracted the short answer.
1:26:49The correct answer, by the way, is 83.
1:26:52And so, yeah, if we're looking at this, the most frequent answer is 83.
1:26:56So we can now maybe that's we can now look at the we just copy this here just so we have it as a reference.
1:27:08So I can, results, look at, let's say, the majority winners.
1:27:17That's the 83. So that's the final answer.
1:27:21We, so if there are multiple ones, it will be a list. So, for example, it will be 83.
1:27:27So if I do something like that, it will always be the lowest one.
1:27:30But I like it. Sorry. Like I said before, if there are multiple ones, right?
1:27:34So if we have something like 83.
1:27:47My keyboard is in a weird position here. Sorry.
1:27:57If we have something like that, it will always pick the first one.
1:28:00Right. Because this one and majority winners will result in this.
1:28:09It will pick the name of the majority winner.
1:28:12And if we have two of them, it will pick 83.
1:28:17And if it was flipped, it will pick 52, right?
1:28:23And so, yeah, we can think about how to implement the tiebreaker once we have the probability scoring.
1:28:29But yeah, to continue also, for example, if you are using something like that in a UI.
1:28:35Like you don't want the user only to see the short answer.
1:28:38You maybe want to show the full answer, right?
1:28:40So you can then provide the full answer of one or the other.
1:28:44For example, we can have the full answer of the first case here and provide the user with a full answer too, if we wanted to.
1:28:56And we can also analyze that to make sure the extraction was correct.
1:28:59And you're welcome to play around with that.
1:29:02But one more thing I promised at the very beginning.
1:29:05I wanted to show you how much does it actually make the model better compared to the base model.
1:29:11So first of all, if we generate more answers, of course, it's going to be more expensive.
1:29:16Right. So five answers mean if we have five instead of one answer means we have to generate four more answers.
1:29:21So it's roughly five times more expensive than with one answer.
1:29:26But does it actually pay off?
1:29:28So does it actually give us additional performance?
1:29:31So here I ran a lot of experiments.
1:29:34I'm not running them live because you can see some of them take multiple days.
1:29:38So this is for the MATH-500 data set from the previous part from part three, the model evaluation.
1:29:45And so let's go through this line by line.
1:29:48So here we are looking at the base model.
1:29:50So I ran it again, got fifteen point two percent.
1:29:54Then I ran the reasoning variant, took about three hours and it got about forty-eight percent.
1:30:00The number is a bit different that I had in chapter three because I rerun it and there's some numeric randomness.
1:30:07But if you run this on average, it should get around 50 percent.
1:30:11So that's about three times better than the base model.
1:30:15Then if we do chain-of-thought prompting and on top of the base model, we get a really good 40 percent.
1:30:22The runtime is longer than with a pure base model because now the explanations use more tokens.
1:30:29This would be if we go back to the slide I showed you, this would be a case of inference-time scaling because we generate more tokens and that can improve the accuracy essentially.
1:30:41Now, but the same is also true for the reasoning model.
1:30:44So the reasoning model also generates more tokens than the base model.
1:30:48So now we have temperature and top-p sampling compared to the base model.
1:30:54And the performance is about the same.
1:30:56It doesn't change much.
1:30:57I would expect that.
1:30:58I mean, here it's always selecting the most likely token.
1:31:01Here is selecting among the most likely tokens filtered by the top-p, but proportional to the probability.
1:31:07So it's kind of related to this one.
1:31:10It's a little bit better.
1:31:11I would say in practice, it's probably plus-minus zero for math here.
1:31:16It takes three times longer.
1:31:18It's likely because some of the answers are longer.
1:31:20Next, I am summarizing these two terms as just top-p so that it fits into this line.
1:31:25Here I have this basically plus self-consistency with three samples.
1:31:31So this is about 30 percent accuracy, which is really cool.
1:31:35This is about twice as accurate as just the base model.
1:31:39So the self-consistency does pay off.
1:31:41It takes much longer because some of the answers are, I guess, longer also.
1:31:46And we have multiple answers.
1:31:49We have to generate three instead of one.
1:31:51But you can see it does give us better accuracy, more expensive.
1:31:54But it does boost it.
1:31:55And if you go from here to here, this is kind of exactly what we are seeing here from here to here, except here they used 32 and we only used three.
1:32:05But again, it really depends on how strong is the base model.
1:32:08If you already have a strong base model, I would assume you would see less benefit from using self-consistency or you need a higher number of samples.
1:32:18In this case, when I increase the number of samples from three to five to ten, it's about the same accuracy.
1:32:24There is some fluctuation.
1:32:26I wouldn't say it's necessarily worth doing ten.
1:32:30I think this is just within the margin of error, if you repeat this multiple times, just due to random chance.
1:32:37If we use this one with chain-of-thought prompting, we can see it's actually lower than just chain-of-thought prompting, which is also interesting.
1:32:49So it's still better than the base model, but a little bit lower here.
1:32:55But again, there's always with LLMs, base LLMs, they're very sensitive.
1:32:59Like if you change a token somewhere, it generates a different answer length.
1:33:03And you can see here 129 minutes versus 180.
1:33:07So here are maybe some more shorter answers and answer length really often correlates with...
1:33:14I'm sorry, where was I? Here compared to here?
1:33:17No, it's actually not true.
1:33:18This is shorter than this one.
1:33:20Anyways, so yeah, in this case, chain-of-thought is still really good, but a little bit weaker than we had before.
1:33:28Now, one of the best results so far is this one here.
1:33:32Self-consistency with the top-p as we had before, plus chain-of-thought.
1:33:37It gives us 42 percent.
1:33:39And then I'm increasing it to five and ten and it goes up to 52 percent.
1:33:44So the base model here with self-consistency, top-p and chain-of-thought prompting is actually better even than the reasoning model.
1:33:52Right. So we have 52 versus 48.
1:33:56However, notice that it takes a very long time.
1:33:59It takes like six times as long as this one.
1:34:02So it's always like diminishing returns.
1:34:05It's almost like if we go back here, we can see that.
1:34:09Yeah, if we have a very, this is not best-of-N.
1:34:13But like in general, like there is a certain saturation so that in this case, for example, a smaller model with more inference-time scaling can be cheaper and better than model here.
1:34:23Let's say Max versus medium that is more expensive and larger.
1:34:28But in general, like it's always really depends on the model.
1:34:31And we can also improve the reasoning model here.
1:34:34So here the reasoning model with self-consistency, top-p and chain-of-thought gets 55 percent.
1:34:40I did not run it with five and ten because it already took so long.
1:34:43But it would be an interesting experiment to increase this to 10 and see maybe if it gets a 60 percent.
1:34:49It might be a good, if you have a GPU, a good bonus exercise.
1:34:53But yeah, the best model is the reasoning model with chain-of-thought prompting, top-p filtering and self-consistency.
1:35:01Anyways, the bottom line is self-consistency is making things more expensive, but it does help.
1:35:07And in practice, I would say self-consistency is not something I would run just all the time.
1:35:13I would run it when you really care about improving the accuracy.
1:35:17Like, for example, in this paper, they tried to develop the best possible math model.
1:35:23But a normal user wouldn't always want to use the best possible math model.
1:35:28It's only if you, let's say, are in a competition, there is no time constraint, no money constraint.
1:35:33Then you would maybe run this model.
1:35:35There was also recently this big topic in the news with OpenAI solving a millennium problem.
1:35:42And there was like a lot of controversy around it.
1:35:44But someone calculated how much it would have cost in terms of tokens.
1:35:48It was about like 15 million dollars worth of tokens to solve that problem.
1:35:53And this is not something you would run routinely.
1:35:55They had like tens of thousands of subagents working on it.
1:35:59And in a sense, self-consistency is also something you wouldn't run routinely.
1:36:04You would only run that if you don't have a strict latency or time constraint.
1:36:08So latency-wise, I should say, since you can run things in parallel.
1:36:12I could technically speed this up because this is run with our for loop example.
1:36:17I have also the batch version.
1:36:19But even if you have a batch version, only one GPU, it would still be slower than having, let's say, three GPUs that have an independent copy of the model where you run things in parallel.
1:36:29So it's also something to consider.
1:36:31But if you don't have, let's say, a latency or compute constraint, sure.
1:36:34Go for a chain-of-thought plus self-consistency.
1:36:38And otherwise, yeah, you always also can get good results, let's say, without it.
1:36:45Forty-eight percent with a reasoning model we will be developing later in this book, though.
1:36:50A long story short, it's already a very long video.
1:36:53Self-inference, self-consistency is the first inference-time scaling techniques.
1:36:59And in the next video, we will get back and cover another technique called self-refinement.
1:37:06So I hope you enjoyed this video, and I will see you back next time.
