0:00Hi, and welcome back, everyone, to the next video in this build a large language model from scratch series.
0:07So, previously, we covered the setup steps, how we set up our compute environment, and then we also talked about how we can load pre-trained weights of the model, like the Qwen3 model, and we implemented some basic text generation functions.
0:22So nothing goes to waste here in this video series because we are going to reuse both the basic, the conventional LLM that we loaded there, and also the text generation functions that we coded in this video, where we are going to talk about model evaluation.
0:41So right now, here in this video series, we are in part two, model evaluation.
0:46Next, in the next upcoming videos, we will also then talk about the reasoning or the more reasoning-related methods, like inference techniques to get better performance out of the model, and then also the core training techniques for training a reasoning model.
1:01But before we can do all that, it does make sense to start with model evaluation because then we can evaluate this base model before we apply any of these techniques, like inference scaling or training, and then have a reference that we can use to compare these improved models that we are going to develop.
1:21So there are different types of model evaluation, just briefly. And so the next slide is going to be a bit crowded.
1:27So, lots of information here, but it's exactly the same information as before, just with a bit more annotation.
1:33So this is taken from the chapter three in the book.
1:37And yeah, so, like I mentioned before, we are going to talk about evaluation, and in particular, we were going to talk about benchmark-based evaluation.
1:46So there are different types of evaluation methods.
1:48So first, benchmark-based evaluation can be categorized into multiple-choice and verifier- based evaluation, and then there are also judgment-based evaluations.
1:58We can categorize those into leaderboard-based evaluations and LLM judges.
2:03Now, basically, there are roughly four different broad categories of evaluation, and they all have their place.
2:10I wouldn't say one is always better than the other.
2:13In practice, it does make sense to consider all of them because they have different strengths and weaknesses.
2:18So multiple-choice ones are actually pretty attractive because they are very easy, or relatively easy to implement.
2:27So here, the model has... We ask the model a question, and model has, let's say, a choice of four different answers and picks one of these answers.
2:34And then we can just double-check, compared to the correct solution, if the correct answer was selected.
2:39There are, of course, nuances and different subtechniques there.
2:43But overall, that is maybe one of the simplest way to evaluate a model.
2:48It's usually used for testing the knowledge of a model because, of course, it's only selecting something. It's not, like, really free-form answering or writing up the answer.
2:59So it's kind of like a test, mainly, what type of knowledge does the model have?
3:06Verifiers is a technique where we let the model generate an answer.
3:11We parse out some part of the answer and have like a ground truth method where we will compare the written answer to the ground truth method.
3:22We will talk more about this in this video.
3:23So I don't want to go crazy here in this introduction and do like a 20-minute monologue.
3:30We will get back to verifier-based evaluations in a little bit.
3:35The next category is leaderboard-based evaluation.
3:38And here, that's essentially, if you are familiar with esports, like competitive video gaming or chess, for example, there are these kind of leaderboards where players are ranked on that leaderboard.
3:50And how that works is different players play against each other, like, let's say, one-on- one, like pairwise comparisons.
3:57And then we can calculate a score, basically, based on these pairwise comparisons and convert this into a leaderboard, for example.
4:06So this is kind of like, I would say, judgment-based evaluation because the way we compare is usually also often based on human preferences.
4:15So someone would show the answers of two different LLMs to a person, and the person would then pick which answer is preferred or which one is better.
4:24Of course, it can be subjective because that could be human preferences.
4:29We can also have a different model selecting, let's say, the winning solution given a pair of solutions. But again, it's also not perfect.
4:38Models have biases and so forth.
4:40So they're all, I would say, kind of like judgment-based and not like as... I mean, not quite as scientific as a verifier with a specified ground truth answer and so forth.
4:51The downside of verifiers is, of course, you're more limited because you can't verify every answer like that.
4:57We will talk about this.
4:58There are some types of data that are more amenable to verification compared to like free-form philosophical answers, for example.
5:08Then the fourth category are LLM judges.
5:11And that kind of is kind of related to the leaderboard.
5:14But instead of having a pairwise ranking between two different answers or two different models, what we're here doing is we are asking a different or another model to give a score. Like, for example, if there's an answer, we might say, "On a rating scale from one to ten, given these criteria, what score would you give this answer?"
5:34And then we can compare an average score across all the answers in the benchmark data set and compute a score that way.
5:41But again, this is objective. Depending on what type of LLM you use for the evaluation or if you use a human, for example, you might get different scores.
5:49And so, yeah, also ups and downs with this approach.
5:54In this video, I'm not going to go into too much detail in terms of all these different types of evaluation methods.
6:00We will be very focused on verifiers.
6:02I will mention in a little bit why we are focused on verifiers compared to the other ones. But if you are interested, I did write up an appendix chapter, a bigger, long chapter covering these four methods, because I thought, "OK, it's kind of like overkill in this chapter because we want to really focus on one technique."
6:23It's a very long technique. It takes like 40 pages.
6:26If we cover all four here, it would be, it would be just a whole new book, basically.
6:31But I have like a long but also relatively short appendix chapter.
6:36I think it's like 30, 40 pages where I do go into more detail discussing the pros and cons of these different evaluation methods.
6:43So you can find it in Appendix F, "Common Approaches to Model Evaluation."
6:48If I go to the live book here, let's see... Appendix F.
6:55So I have a chapter here where I talk through this, and I'll also have some simple code examples illustrating this.
7:02So if you're interested, this is like a bonus to the main content in the book.
7:08Now, back to the topic.
7:11So we are going to talk about evaluating reasoning models, chapter three, essentially.
7:17And I think I have one more slide.
7:20So the reason why we are focused on the verifiers also is, first of all, the evaluation that's more like an objective evaluation, because there is a hard ground truth answer to each question, and we are going to reuse that later when we are implementing the reinforcement learning.
7:37So the reinforcement learning loop in the reasoning model is usually called RLVR, which stands for reinforcement learning with verifiable rewards.
7:47And so the typical workflow in reinforcement learning for LLMs is that we have a prompt; we have a policy output.
7:54So policy is just jargon for the LLM output.
7:57Then we calculate a reward.
7:59We update and get a better LLM.
8:04So we update the weights and we get a better policy, like a better LLM out of it for the next prompt, and then it will hopefully create a better answer and so forth.
8:13And so the reward here, that's the interesting part, how it gets calculated.
8:17And we can use verifiable rewards.
8:19So, for example, in reasoning, it is often common to use either math or code or both to calculate rewards.
8:28So the model, for example, if we focus on math for a second, is given a math question and generates a response.
8:35And then we have a tool, for example, a calculator, Wolfram Alpha, SymPy, one of these mathematics tools where we can compute the correct answer and then reference or compare this to the LLM final answer.
8:50And if the number, let's say here, in this case, five times three plus four, matches in the calculator what the LLM outputted, then we can give a reward of one and otherwise a reward of zero. But this is like very easily or relatively easily verifiable.
9:04We can generate the ground truth solution to this math question very easily using an app, for example.
9:13And so this is what we are going to do in this chapter.
9:16So, or this video, if I go back one, one slide here.
9:20So the benchmark-based evaluation, this is focused on verifiers.
9:25Also, I should say we are going to focus on math, not coding, because it's just a little bit easier to illustrate how verifiable reward computation works.
9:35And it will also make the training a bit easier.
9:37Of course, code is for most people even more interesting because I guess all of those watching this video, there are more programmers than mathematicians watching this.
9:48But coding has, I guess, has more complexity.
9:51The answers are longer.
9:54The formatting can be different.
9:55There are different programming languages.
9:57Also, the evaluation, the verification, for example, is either via unit tests or whether the code compiles or not.
10:04So you have to also actually run the code and then you have to worry about security on your computer, maybe make a sandbox.
10:11It's, like, a lot of complexity that is interesting.
10:15But if you, let's say, are mainly first getting started learning about how does reinforcement learning work, how do we implement a simple, let's say, or how do we...
10:25So if we implement the basic reinforcement learning training loop, I guess it makes sense to start with a simpler case of math where we get, let's say, where we focus more on the explanation and the technique rather than getting sidetracked and building a secure sandbox and executing our code and so forth.
10:45This would be an interesting follow-up, though, like just porting everything over to coding from math.
10:52But that being said, let's get started with the coding part.
10:57So let me get comfortable here with my keyboard.
11:01And so I'm in my folder here.
11:04This is the cloned repository.
11:07Of course, we could technically just open the existing notebook.
11:13So this chapter three notebook.
11:15But since we... I want to do it a little bit step by step from scratch.
11:18So I will start a blank notebook, of course, to get the most out of it.
11:26Like I mentioned in the first video, it does make sense to, I mean, if you have a lot of time available and you want to really learn things well, to retype things.
11:34That's, of course, the most time-consuming way to learn something, but it is kind of like nice because then you kind of really see what's happening, and if things don't make sense, you kind of, when you don't glance over them, you try to spend more time on them.
11:49And for this video, though, because I don't want to make this a four- or five-hour video, I will be copying things into this notebook.
11:57So just to make this go a little bit faster.
11:59And so for this one, I'm going to start with just some boilerplate where I'm just printing the versions of what I'm currently using on this computer.
12:10So the reason why I'm doing this is, in case you get different results, we can, or you can maybe cross-reference to the versions and make sure to have the same versions.
12:21And if you still get different results, then maybe you copied something wrong or typed something wrong.
12:26But if you get different results and the different versions don't match, it could also be an explanation.
12:31So it's always like a good way to kind of debug why you get different results.
12:35Maybe first making sure you have similar software versions.
12:39And so, yeah, this is just what we're going to use.
12:42So the reasoning_from_scratch package, that is the one that we used previously to load the pre-trained model, but also it has some of the code from the previous video from chapter two.
12:57So we don't have to retype the text generation.
12:59We can import it, but we'll, we'll get there later.
13:02The torch, the basic torch version. SymPy is the mathematics package.
13:07We will be using that later for grading or verifying the answers.
13:11And tokenizers is part of the pre-trained model that tokenizes the text.
13:16It's alphabetical, so it makes more sense basically to look at it this way.
13:21So everything depends on torch.
13:23We have this reasoning package, which inside uses tokenizers.
13:26And then this chapter or video, we are going to use also SymPy, which is this mathematics tool or library in Python, is a symbolic math library to compare mathematical answers, which we will use to implement the verifier.
13:43And so the... I would say that the first part would be building the math verifier.
13:51There will be a lot of steps.
13:53So this will be section 3.1.
13:56And if I just give you the overview here, I think that is probably a worthwhile time investment to briefly go over what we are going to build here.
14:03So first, just looking at this figure here, this is an overview of how this will look like, just a simple overview.
14:11So we have the LLM. The LLM gets some input.
14:16So here that's the math question.
14:18So we have a math question, and the LLM outputs an answer.
14:22So it gives some explanation and then the final answer.
14:26Now we will get to this.
14:29So we don't have really a reasoning model yet, but it looks like it already talks a bit about the steps, and I will, I mean, I can already say it right now.
14:39So if you go back a few years and you run a very basic model and have some math question, it can be that it just really only outputs the final answer.
14:49And it still can happen with modern pre-trained models that have not been fine-tuned, but training data sets, they get larger and larger and larger.
14:57And people are, when they're pre-training LLMs nowadays, they already have a lot of reasoning traces and explanations in this pre-training data.
15:05So even though we haven't specifically fine-tuned or trained the model, even a base model, just a pre-trained model, is sometimes outputting really good and thorough explanations with the final answer.
15:19It's just not very reliable.
15:20It's not doing it every time.
15:21It's just doing it in some cases.
15:23And also the performance in terms of correctness is not quite as good yet compared to a reasoning model.
15:29This is also why we are building the verifier.
15:31We will be evaluating it.
15:33How well does the model perform in terms of accuracy on these math questions compared to the reasoning model later on that we are going to train?
15:45Now, back to this figure.
15:47So we have an input that goes into the LLM.
15:50The LLM generates the answer.
15:52And then the tricky part, it's, I mean, tricky in a sense, a lot of Python code required for this, but the tricky part is to extract the real answer.
16:01Like, there's a lot of explanation, but when we wanted to plug it into our calculator or symbolic math comparison tool, we need like the numeric final answer.
16:10We don't want all that other stuff.
16:13And so we get the final answer, we extract it, and then we compare it to a reference answer.
16:18So this reference answer, like I mentioned, that can be either calculated with a calculator or, in this case, if we use a data set, I mean, there is a data set with pre-computed answers that we can use.
16:29And then we can compare if this is indeed the same.
16:33And then, based on that, we can say, "Okay, the answer is correct compared to the reference answer, or it's incorrect."
16:40And then we can compute the accuracy of the model on a math data set.
16:44So this is the main goal, like having an LLM generate an answer, which we have already implemented in the previous video or chapter.
16:54Now, in this video, next, we are going to extract the answer and compare it to the reference answer.
17:00It sounds pretty trivial.
17:02I mean, the new steps really is just as an extraction and the comparison, like two, two new things compared to what we have done last time.
17:09But as you will see, it is actually quite a lot of work, a lot of code that goes into that.
17:15So I think it's like a 30-, 40-page chapter.
17:17So there's a lot that we have to do to make this robust.
17:23So, yeah, so stay tuned.
17:25And this is what this video is about.
17:28So going back here, or maybe let me show you one more figure, because that is, like I mentioned before, that's like the step-by-step process.
17:36So we will be doing this extraction and verification in eight steps.
17:42First, we have to start with, of course, loading the pre-trained LLM and implement or load the text generation functions.
17:50So one and two, we have done in the previous chapter.
17:53So that will be quick.
17:53We just have to reload the code that we already coded previously in the previous videos and/or chapter.
18:01And what's new is extracting the final answer box, normalizing the answer.
18:08So, because as the LLM could also provide this, instead of in a box, it could do 14 slash three, something like that.
18:18So there are multiple ways we can write this answer.
18:20So we have to implement a way to extract this answer, normalize it into a canonical form, and then verify that they are equivalent and grade the answer.
18:32So giving a score, and then we apply this to the math data set.
18:37So MATH-500 is one of the popular benchmark data sets, like, I think it's high school math. They are not super easy, but also not, like, super crazy hard.
18:46So we have like a math data set here and evaluate the base model.
18:50And yeah, these will be eight steps.
18:52So buckle up. We will be going through them one at a time now.
18:57And yeah, so we can start basically with, we can now start with loading the pre-trained model.
19:05And so here we are just going to reuse code that we already coded in the previous video or chapter.
19:12And so I'm just copy-pasting this here.
19:15And so, just walking through this, what we have here is from the reasoning from scratch library, we are importing the, yeah, the downloading of the model weights, the tokenizer, the model, and the config files.
19:30So there's nothing really new here.
19:33We're just putting it here together into a function.
19:35And so that we in upcoming chapters can reuse that function and don't have to retype that every time.
19:41And so in this function here, I have different arguments or parameters.
19:46So one is the which_model parameter.
19:48So we can change this to reasoning.
19:51So you can call this later on when you call the function with reasoning.
19:57So here you can see that.
19:59And then it will load an existing reasoning model version of that model.
20:04So Qwen3 is here, the 0.6 billion parameter model that we worked with in the previous chapter. The developers of this, the Qwen3 team also has a reasoning model version of that.
20:17We are going to develop our own reasoning model in this video series, but this is a good reference.
20:23So we can use that as, "Hey, we want to develop a model that is approximately as good as this reasoning model, for example."
20:29So we can use that already as a reference, if we want, when we verify, when you have the verifier to compute the difference in quality between the base model and the reasoning model.
20:40The device, that's like the MPS device, GPU, and so forth, as we talked about, whether we want to use torch.compile and where we want to store the Qwen3 model.
20:50So local directory just means it will put it in our local directory relative to this notebook.
20:58And yeah, that's it.
20:59Essentially, there are a few...
21:00So if you look only at this, this is very simple.
21:03It's what we've done before.
21:04We download the weights.
21:06Then we have the tokenizer model path and put that into the tokenizer here.
21:12And that's essentially it.
21:15And so here we, so if we skip this for a second, so we have the tokenizer and then we also load the model and put it onto the device and apply a torch.compile if we want to.
21:27Again, so this is very similar.
21:31I mean, it's identical code that we used previously.
21:34There's nothing new, just putting it into a function here.
21:37Now, what's new is like this part, like I mentioned. Optionally, we can use the reasoning model version, but again, it's the same steps. We download the weights, we set the tokenizer and then initialize, here, the tokenizer.
21:49There is a bit more boilerplate.
21:51So this is just preparing the tokenizer to put out a chat template and have thinking enabled because there is like a on-and-off switch for this thinking or reasoning behavior, which is toggled through the tokenizer.
22:06Now, you don't have to worry about this at all.
22:08We will be revisiting how to modify the tokenizer and when we develop our own reasoning model.
22:15So here you can just take this as a given that this is just like loading the reasoning model.
22:21So let's just execute this.
22:24So this will not download anything.
22:25This will just define the function here.
22:27This load model and token... tokenizer.
22:31And then next we can, yeah, we can actually load the model.
22:35So the next code is that we download the get_device from the previous chapter again, and so this get_device is pretty simple.
22:44What it does, I can maybe put this outside here and just execute this for a second.
22:53So this will give us like a CPU, MPS, or a CUDA.
22:59So in my case, I'm using MPS.
23:02I mentioned before, it's maybe not a bad idea to just run it on a CPU.
23:07I tested this chapter on MPS, though, and I did get, I think, almost identical results as on a CPU.
23:14So you can, for compatibility, just hard-code this, but to be honest with you, in this chapter, it works okay. MPS works robustly enough.
23:25So I would, if you have an MPS device or CUDA device, I would just maybe leave it, unless you get, like, really weird results. Then maybe I would just hard-code the CPU.
23:35Yeah, so we already executed that part.
23:38So just have a note about it here.
23:40WHICH_MODEL, the base model. We will start with the base model and then load to the model and tokenizer, like we defined up here.
23:47So this might take a while.
23:49I noticed, maybe not that, maybe...
23:53So I noticed the Hugging Face hub is a bit slower these days where these weights are downloaded from.
24:00I would say it probably doesn't have anything to do with the recent NVIDIA acquisition of Hugging Face.
24:05But my guess is just more people are using it sometimes in certain times of the day where it's just a bit slower because the whole open weight model ecosystem really depends on the Hugging Face model hub because everyone puts their open-weight models there.
24:21They have, they must have like three, maybe a terabytes of data uploaded every day.
24:27I'm pretty impressed that it still works at all because so much data is uploaded there every day.
24:34So while this is loading, I can already maybe talk you through the next part of the code.
24:40And so here we are going to import from chapter two.
24:44This means the previous video, the basic text generation function we coded.
24:50And so this is... You can ignore this.
24:53So this is what we coded previously, and there's nothing new here, really.
24:58This is just what we used before.
25:00And the steps are, we have a prompt.
25:02We put the prompt into a torch tensor, put it on the device, add the dimension, the extra dimension.
25:13And then we put it through, I mean, it's like a generator.
25:17So for, for each token, we output it and this prints the token.
25:22In addition, here, this is also collecting all the tokens.
25:26And this is essentially it. That's what we have done in the previous chapter, just putting it again here.
25:34And yes, you can see the model is pretty fast, generating the text, the answer.
25:39And you can see, this is a very simple prompt.
25:43However, the answer is pretty long here, right?
25:48So it looks pretty long.
25:49And that is because, like I mentioned, some base models, some of the time, are already behaving a little bit like a reasoning model and giving these answers in a step-by-step fashion, even though we didn't ask specifically for that.
26:02And the reason is it has seen something similar in the training data.
26:05And so sometimes it actually does quite well, but it's... We will see later.
26:11It's only doing it some of the time, like maybe 30% of the time, you know, and the rest of the 70%, it kind of messes up completely.
26:19So it's also, you can see, it's kind of like hard to read, right?
26:23I mean, it's kind of like the formatting here, like a Markdown, this like bold in Markdown.
26:30That's like math, LaTeX math notation.
26:33Also here, more late...
26:35This is like a fraction LaTeX math notation.
26:36It's kind of like hard to read.
26:38And so what we can do is we can actually, in Python, make this a bit simpler.
26:43We can use from IPython.
26:45This is a IPython, a Jupyter notebook from IPython.
26:48So, my one side note here, I'm old enough to remember when Jupyter notebook was called IPython notebook.
26:54So IPython notebook or IPython in general is like an interactive version of Python. Back in the day when I started using Python, everyone was just using the basic interpreter in the terminal, which was kind of painful.
27:07And so yeah, IPython was developed, amazing project.
27:13And so with IPython, there was a bit more interactivity in the terminal, terminal, like in the interpreter, which was great.
27:19And then there was IPython notebook, which was renamed to Jupyter notebook because then Jupyter stands for, like, Julia, Python, and R, because it supports different kernels, different languages.
27:32So here we have a Python kernel, but it could also run or support Jupyter and R code.
27:38If you still remember what that is, it's been a while, like data science back then was a big thing, like 10, 20, 10, 20 years ago. Julia and R, very popular.
27:49I started, I think I even used R maybe around the same time, or even before Python. I was in statistics also, so very popular language there.
27:59But that being said, that's IPython, interactive Python, and they have like a display submodule here, I guess, and they let us render LaTeX and display it in a bit nicer fashion.
28:12So if I run this, it will convert our previous tokens in a nicer format.
28:17And now it looks just nicer to read.
28:20This is not LaTeX. This is like Markdown, the asterisk for bold.
28:24So that's probably why it doesn't render this, but you can see it, it does much better job with the fractions and everything.
28:31It's just, yeah, readable now.
28:33And then also the box here, the final answer in the boxed environment.
28:37We did not ask it to do that here.
28:40We didn't ask to specifically put the final answer into a box, but it's also like a common convention basically for math problems to put in the final answer into a box basically, and we will benefit from that later because we, that makes the extraction a bit easier.
28:55Also, we can do the same thing.
28:57I mean, this is for LaTeX for individual parts of an equation.
29:02For example, we can also use Math here, like for smaller math problems. It's a bit more lightweight.
29:08So we can also just, you know, render the equation, sorry, the fraction here.
29:13And so this is essentially just some setup.
29:16So we loaded the model and we generated some basic text and basically repetition of what we've done last time.
29:23So the next step would be implementing a wrapper for easier text generation so that we don't have to retype this every, every time.
29:30So if I go back, back up again, this is not a function.
29:33We just pasted this here.
29:34It's kind of like painful every time if we had to copy that.
29:38So we can also just make our life a bit easier and put that into a function.
29:42And so let's call that function generate_text_stream_concat.
29:47Like the function name is probably ridiculously long and complicated, but there's a reason. There is, so, for example, "generate text."
29:57It's just telling me what... It's telling, hopefully, every user what the function is doing. It is, you know, generating text with the model, and "stream" here means that it is outputting it, printing the tokens as it is generating.
30:11It's using our, our generator function here, the stream function.
30:16And here also, we are using the KV-cached version, and the "concat" means a concatenation.
30:22So that means we are concatenating the tokens.
30:25So we return them in addition to printing them.
30:28So again, just slowly, we have the input formatting and then we iterate here.
30:34So this is just what we had before.
30:35So I essentially just wrap the code we have above into a function, and we are collecting. We have the generated IDs; we are collecting those, and in addition to collecting those, we also print them live and the collected ones are returned, because what I'm trying to say here is, if we don't do this, so if
30:58we don't do this and this, we will only, I can maybe run this, we will only print them to our output here, but we won't be able to reuse those.
31:10So in other words, I mean, it is nice.
31:13If I go back here, it is nice, if we see them live. So it's like this. It's nice to see them here, but let's say I want to work with them.
31:25Like I want to use them to demonstrate something.
31:29I want to process the output.
31:31It's kind of gone now because it's just here.
31:33I would have to manually go there, copy-paste that, and assign it to a variable.
31:37But what we did is we saved it here as all_token_ids, basically, and then decoded it, so we have the all_tokens as, as a variable where we stored all the tokens.
31:54So they're not going into the ether.
31:56We have them here available basically, right?
31:58So we can do all kinds of things with it, like just printing it here or rendering it, or maybe extracting the final answer as we need. For example, we will be focused on extracting this fraction here so that we can verify whether this is correct or not.
32:17And so what we're doing here, that's the concatenation.
32:21So I wasn't very creative.
32:24Only name I came up with is just like collecting the IDs, appending them like before and returning them.
32:30But otherwise, everything is identical to what I just showed you.
32:34And then I have also, oops, skip_portion section here.
32:40And this is just, you know, when I was working on this notebook, even though it's, the LLM is quite fast, when I was just rerunning the notebook, I wanted to check something.
32:49It took a long time every time to rerun this because it is, you know, writing this whole long answer and it takes extra seconds.
32:55So I just had this little convenience function where I can just do True and then it's not doing this, right?
33:03So it's skipping it.
33:04It's just like for some debugging I did.
33:08So we have now the convenient function we are going to reuse and we will also reuse that in upcoming chapters.
33:15So this will be added to the reasoning_from_scratch library.
33:18It's in there already, by the way.
33:19And so the next section, sorry, in the next videos, we can just import this like we imported...
33:26If I scroll up again, like we imported this function, there will be then later in data videos from reasoning_from_scratch, import the, the text generation we just implemented.
33:39So we have now a way to conveniently generate text with this little thing.
33:46So if we summarize this, we have now this function here to generate text.
33:52So this, this was the takeaway so far.
33:55Next, let me just wait until it's finished here.
34:00So next we have the extraction of the final answer box here.
34:07And so this gets interesting now.
34:08So here, if I scroll down a bit, we are now at the point where we are doing the extraction.
34:14I think I should have...
34:17I'm just, maybe that's easier, to look at this here.
34:20So we have, we are now at step three.
34:23We have generated the answer and now we are interested in extracting the final answer here.
34:29So whatever is in the "boxed" format here.
34:32And so, like I mentioned, the box is usually a convention.
34:36It's not always like that.
34:38If we run the pre-trained model and we have it generate some output, it will sometimes put the output not into a box, but yeah, that just sometimes happens, but we can modify the prompt to highly urge the model to use the boxed answer.
34:52And later, when we fine-tune and train it, we can also make sure that it really puts that into the box.
34:57Otherwise, it gets a big penalty.
34:58So it will be better at making sure it's always in the box.
35:02But the box, yeah, that is what we will be using for extraction, for the extraction of the final answer portion.
35:10So let's go back here.
35:11And so, for example, if... So let's just work with a simpler case.
35:16So, for example, if we have a model answer like the following here, so it's like a bit shorter and easier to look at than compared to this whole thing.
35:24But imagine our model generated this.
35:27So instead of this whole long one, we have now just some abbreviated answer where I just dotted out something.
35:33'Cause that, that just makes it a bit easier to work with.
35:37And our goal is now to extract this portion.
35:41So it would be a lot of work to retype this.
35:43So I'm just pasting here.
35:45It took me quite some time to come up with something that is really robust, but I have this fun, function here that gets the last boxed section.
35:52So if we have multiple in the answer, we have multiple boxes here because it could be like intermediate solution or something like that, what this function will do is it will get the last boxed one here.
36:03So that is with rfind, for example, it will find the index position of the boxed one.
36:10So if I go maybe here, copy this, so it will tell me index position, like where...
36:18Each character in Python is like a position, will tell me where is this boxed thing.
36:24If there is no box thing, it will return minus one, basically.
36:28And so that is what it is doing.
36:29So if there's nothing, no box in there, it will return None.
36:31So no box, and otherwise, it will try to extract this box.
36:36So there's a lot of code.
36:38There's also making sure with nesting, because, you know, you can see, or it could be that there is some, some other thing, like multiple curly braces, or here you have a curly brace and then inside a curly brace.
36:52So making sure all that gets parsed correctly is quite a lot of coding work.
36:56So there's a lot of stuff going on.
36:58I don't want to go into too much detail here because that's kind of super boring.
37:01You can walk through this. Took a lot of time to make sure it really works, but it's pretty robust based on my experiments.
37:08And so what it is doing is essentially extracting the boxed answer from the model answer, so it's supposed to get whatever is inside "boxed" here.
37:19So let's run this and see if it works.
37:22So, yeah, we get the 14 over three fraction.
37:27And yeah, so this, this works.
37:29Now, next, we are going to kind of take a step further.
37:35And so, make sure this also works in case, let's say, our model doesn't use "boxed."
37:43So, like I mentioned before, most models use "boxed," but let's say it doesn't do that. It does something like this.
37:51It would not give us an answer, even though, I mean, it's kind of like here, right?
37:55So it's kind of, it would be kind of a bummer if, just because of such a small formatting issue, everything would fall apart.
38:04And so I have like a little more sophisticated version here, extract the final answer candidate.
38:11And so what it will do is it will have a fallback.
38:15So first, it will try, if the input is text, first, it will try to use the get_last_boxed function we used before.
38:25And if it's there, it will just clean it up a bit more.
38:28Like if there's an additional LaTeX dollar sign or something like that.
38:31Happened sometimes. So it's all here empirically based on when I coded this, what I tested and observed.
38:38If there is no boxed expression, like there's no "boxed" in there, like I just showed you what could happen, then there are a few fallbacks.
38:46So first, there is like a number and then full, and then number only.
38:51And I will like briefly explain what that means.
38:53So number_then_full essentially means that if there's no boxed context there, it will try to find whether there is a number.
39:07And if there's also no number, then it will just output the whole answer as text, because there could be, for example, a case where the model that it's like an algebra question and the model just answers with A or something, or B.
39:23And in that case, we would still get an answer, right?
39:26So, or like I mentioned before, the model maybe answers, it says, "Final answer is 412," and it will then still get this 412.
39:36So this is essentially just like a way to make it more robust.
39:41And so we can test this, for example, on the original answer first.
39:46So I think it will make it more clear if you see it in action.
39:49So this should give us the 14 over three, like before.
39:52So this is the 14 over three, but we could have like a little modification here.
39:57For example, we can add, instead of the input text, I have some modified text here.
40:03We can add this dot here.
40:04So here, this will work as before, but if I have the dot here, it will still extract what's in there.
40:10And I think when I go back to my get_last_boxed, I think this will probably not work.
40:19Let's just try this.
40:23Did I make a mistake here?
40:26It does, but like you can see, it has the whitespace around it.
40:30So that's just like a little cleaned up version of that.
40:33Now, let's try another one.
40:35So here, let's say we have an answer that puts out something like that.
40:40And I just added some text here and some text here.
40:44So this is such that the number is not the last text.
40:48It could, for example, be also a period at the end or something like that.
40:51And so the extract_final_candidate, it should give us the 14 three cleanly, where if we use something like this, this would pretty much fail.
41:01This should give us like an empty, yeah, nothing, None.
41:06It will give us None because there is no box, basically, also.
41:11And if we have text without numbers, so something like this, this function will then use the fallback and give us text without numbers as the answer.
41:21So that, I mean, if that's just in case the solution is something like A over B or something, it should still work.
41:27Even there is no number.
41:29So, and if there's a number, one, two, three, it should give us the one through three.
41:35And so ideally, we have the box, right?
41:39So that's the ideal case scenario.
41:41That's the easiest, the easiest scenario.
41:45And so, by the way, this is also an escape character.
41:49And so if you have something like this, I recommend using r, which means like raw string.
41:54So it's, otherwise, you have to, I think, you have to do a double one to escape it.
42:02See, we escape the escape, or we, we have like, I guess, escape the escape character or something like that.
42:09If we do this, it still works.
42:12But I still recommend doing this r for raw string.
42:15If you have something like that, it's just a bit more robust.
42:19So this is our text extraction, and this is just, you know, busy work, like the Python code that extracts this box.
42:29I recommend, when you do something like that, use a lot of unit tests.
42:32So this is just like a small snapshot of things.
42:37I have a whole suite of unit tests to check all different kinds of scenarios because that is very crucial to get that right.
42:43And it's just also, to me, for me personally, just a lot of trial and error, in a sense, like, oh, you do it.
42:48And then you have one test case that fails, and all, "Oh yeah, I should do this," and then modify this.
42:53It's kind of like busy work, I would say, but important busy work.
42:58Oh, I probably should tell the publisher that this is formatted very weirdly.
43:03I think this is probably a screenshot of the 14 over three, and on this website, gets just blown up to the full text width. Looks kind of hideous.
43:11Yeah, I should tell them about that. Looked fine before.
43:14I think they probably updated something on their website.
43:17Anyways, so next, we go from extracting the final answer box to normalizing the final, or the extracted answer.
43:26This is important because if we go back here, see, like, that we have this dot here and, like even here, so here, this is the simpler version, but like here we have this, this dot, for example, that should be removed.
43:39For example, just making sure everything is kind of like in a canonical form so that we can then more easily compare that to the reference answer.
43:47So next we have, yeah, like a normalization or like a cleanup step.
43:52And that will help the tool SymPy we are using to make these comparisons more easy and not have any issues with some text characters that are left over and things like that.
44:04Again, super crazy long function. Took me a lot of time to develop this and to make this robust.
44:09But yeah, there's a lot going on.
44:12It's like first replacing some LaTeX formatting.
44:15You don't have to understand this also in super detail.
44:18Again, it's like the Python busy work.
44:20I wrote this like a year ago now, and I remember just spending so much time on this to make it work well and robustly, but also not making it 20 pages of code.
44:31So it's kind of like the compactest working version I was able to come up with.
44:37So what it does, it's replace, it replaces some LaTeX formatting.
44:41For example, yeah, also like the dfrac, changing it to frac, just for simplicity. There are different version or ways you can do a fraction in LaTeX or some Unicode characters, changing it to the asterisk and just some cleanup, essentially.
45:00And this is kind of necessary because otherwise you get issues later in the parser and so forth.
45:05Next, stripping some special tokens.
45:08So, for example, when we use a reasoning model, it might also output, in raw form, some essential formatting things like assistant tags or think tags and so forth.
45:19And so that removes that because it can also throw off the parser.
45:24There were some, yeah, also some issues like, you know, like superscript and characters, making them, yeah, like ASCII, you know, like having them as normal numbers, stripping labels and multiple choice, so making that consistent, basically.
45:45Lots of, lots of things going here on here.
45:48So lots of parsing, and I tested everything also in terms of the math accuracy.
45:54And so these changes were all kind of beneficial because you can, you know, if you have a large data set, you can run the model, get low performance, make some changes, and you can observe in the results, actually, the answer is correct, but the grader just said it was wrong, and because it was not parsed correctly.
46:11And so with a lot of trial and error, I, yeah, I had this function, which I think is complicated, but, I mean, it's not 20 pages of code or like 10,000 lines of code. It's just compact and works relatively well.
46:28So next, let's try this in action.
46:30So first, let's just use a fraction that is kind of messy, just see what happens.
46:35And so, yeah, you can see it puts or replaces this with a parentheses.
46:40We can have another one where we have a lot of, a lot of text here.
46:44Does the same thing, basically.
46:45So it's essentially about, yeah, getting rid of something and simplifying this, just normalizing it.
46:51And so this is like the first, I guess, step to cleaning up the answer.
46:56Now, if we have a cleaned-up answer, the next idea would be to implement the checker.
47:05So we have now a way to normalize the answer.
47:07And now if we have a reference solution, like the ground truth solution, we want to compare it to this reference answer essentially.
47:17And so, for that, next section would be verifying mathematical equivalence.
47:22We will be using a tool called SymPy.
47:26And so SymPy is like a symbolic math library in Python.
47:30It's been around for a very long time.
47:33I can, I think it's as long as I worked, at least with Python.
47:36And it's been, it's been around maybe 15, 20 years or something like that.
47:40Very robust, very, very nice tool. Has been developed over many decades.
47:45Open-source, a great library.
47:49And so it's a whole topic by itself.
47:52There are probably whole books about SymPy.
47:54It's essentially, if you are familiar with Wolfram Alpha, it's kind of like the open-source Python version of Wolfram Alpha.
48:01And so here, this is our parser.
48:05And so yeah, we are first parsing, parsing, like a input essentially, and then we will implement the grader later.
48:16This is just making sure...
48:18And so we normalized something, like we got rid of extra fluff that, let's say, SymPy wouldn't be able to handle, but SymPy also has its internal processing.
48:28So it will also kind of like process it and bring it into a symbolic form so that it can then do the comparison.
48:35And so it has this parser here. Again, a lot of code and also a lot of trial and error because I ran this on larger benchmark data sets, and sometimes it crashed on certain inputs.
48:46And so I had to kind of like make some tweaks, and that's basically also how I modified the normalization.
48:53But then also there were sometimes errors if the output was too big or too long.
49:00And so I, over the time when I ran this on many different datasets, I added all the, let's say, special cases.
49:08And so, if, instead of just crashing, if there's a syntax error, it will just return None so it doesn't crash the whole Python run essentially.
49:16So, yeah, this is just like the parser.
49:19It's just parsing the input. Should say also, here, I noticed that some bad models, like if we have a base model that is not well trained or we are training our model later with reinforcement learning and our training settings are really bad.
49:34And sometimes the model will just keep writing like garbage, like, you know, like 2,000 characters of math that are not correct, or like numbers that are very long.
49:45And so this is just like a safeguard.
49:47Like it's just like making sure if the model performs poorly, then the code, the evaluation code, won't crash.
49:55Now, we can say, "Well, why would you want, why do you even work with garbage models?"
49:59And that is, yeah, I mean, it can happen when you're training a model.
50:02Training can be bad, right?
50:04But it would be still interesting to be able to evaluate the model, right?
50:07So instead of crashing the whole training run.
50:10And so that is all like here meant for like safeguarding so that the training doesn't crash later on, also the verification of bad models.
50:21So this is not a problem in this chapter because we load the base model, but in general, if you have a really bad model and you evaluate it, you don't want to crash it. You just want maybe 0% accuracy, but you want to have some result there.
50:34So let's start with putting everything together.
50:38And so we had multiple things going on.
50:41So first we implemented this final box here, the box extraction.
50:47That's the first step.
50:50Then we normalized this.
50:51So we got rid of the junk character, or not junk, but like formatting characters.
50:56So here, the dfrac and everything, we got rid of this. This is with answer normalization.
51:02And now we are putting it through the SymPy parser.
51:07And you can see this is now 14 over three. Before...
51:13Let's just run this.
51:14Maybe it was in this form, and the SymPy parser did its own, yeah, formatting here.
51:23I don't, I mean, this is not necessary.
51:25I don't think it would crash or anything if you don't have that.
51:28So, but, I mean, how can I say that?
51:31So this is a string representation now, and this is kind of like a symbolic, processed variant of that.
51:39And so this is like how SymPy now represents this in their own library.
51:47And then this is just printing it out, but there's actually, it looks like a simple thing, but there's like a fundamental thing, like a, really like a parsing happening in here, like where it's reading in the number. It's almost like if you write down a number on pen and paper and you upload it on your, to your computer, the, the model or the computer can't really, it sees it as pixels, right?
52:13Or as text, and you have to put it into a calculator, and the calculator kind of parses it on its own.
52:21And that's kind of like what's happening here.
52:24And so what's interesting then is if we have, for example, something like this, 28 over six. It's the same as 14/3, right?
52:33And so SymPy will take care of reducing this to the same form.
52:38And so simplifying it.
52:39And so here, you can see it is kind of like understanding what is the input and bringing it into the same form, which is in turn important if we want to implement the verifier in the next step, if they're equal.
52:54I mean, I think it's almost like a philosophical question.
52:58Like it depends really on what you want.
53:00If the model outputs this, I mean, the answer is, I would say it's somewhere in the gray zone.
53:06It's neither fully correct or incorrect because, I mean, ideally, we want to have the 14 over three.
53:12Like that's like just common practice to write it like that.
53:15But 28 over six is also not completely wrong.
53:18You know, like what would be wrong is a 28 over five, for example.
53:21And so, in our parser, or sorry, in our verifier, we will count this as correct answer, but, you know, if you want to be a bit more strict, you can treat this as incorrect, anyway. And so next, we are now implementing this check whether something is equal.
53:43So for that, the equality_check, so we have two things going on here.
53:50So ground truth here, that means ground expression, the ground truth.
53:54It's kind of like a, maybe an ugly variable name, but ground truth is in math or in science or machine learning, data science, that's like the absolute correct answer.
54:07That's like the reference and answer.
54:09It's like a weird word, like the ground truth, but it's just like a common jargon.
54:13It just means the reference correct answer.
54:15And here, this is our predicted answer.
54:19And so we can do, we can do a check.
54:21So if the ground truth is exactly like the same as the given answer.
54:26So, for example, we don't even use the SymPy parser.
54:30Like if we have like 14 over three equals, two equals 14 over three, right?
54:38If that is already equal, that will return True.
54:41And then we can just skip some work because this is like extra work.
54:45It takes some more computation, maybe one or two seconds or something like that.
54:48And so we can save us some work here if already the expressions match here.
54:55Now, if we have something like this, though, this won't work, right?
54:59So this is not equal anymore.
55:03And in that case, it will fall back to the SymPy. Oops.
55:08Let's just copy this to the SymPy parser.
55:14It's just to make it more robust here.
55:19So see, so now that's like another check, but even that is not super robust.
55:25So just checking whether this is zero is even more robust.
55:36So, yeah, that is our equality_check.
55:41So we can do it on normalized texts.
55:44So here we have our normalize_text function just for simplicity.
55:49Let's assume these are normalized texts.
55:52We have already extracted via our final candidate extraction from the box answer.
55:56But just so we have less text and we can, or less stuff happening, we can look at the text that is already extracted.
56:03So here we are talking about the extracted text here.
56:08We can test this on some other examples, doing the equality check, and the equality check, like I mentioned before, does the parsing.
56:14So we don't do the SymPy parse here.
56:17It's done already by the equality check.
56:20And so, oops, not defined.
56:22Of course, I thought I, it crashed, but no, it was not defined.
56:26So here we can see, even though the writing is relatively different, it works.
56:31And I have a few more test cases, for example, if it can do a decimal point notation versus fraction, that one works as well.
56:40What, of course, shouldn't work is if they are different, right?
56:44So this one should give us a False. And a few more here.
56:49So this one is a case that should actually be correct.
56:54So let's see what happens.
56:55So here, we have 14 over three, 14 over three, two over three, and four over six.
57:01So technically, this should work, right?
57:03Because SymPy would normalize this, but let's see what happens.
57:08So it is actually False.
57:09So our tool is too simple.
57:12It can't handle multiple, like a string or like multiple answer, like multiple expressions in one answer, basically.
57:22So even though if we would provide them individually, let's say, 14 over three, 14 over three, this works. But this one also works, right?
57:33But if we put them together, so this also works, if we put them together, though, it doesn't work.
57:39So it's like because of the comma and the whitespace here and so forth.
57:42So we do have to make this a bit more sophisticated, which we will be doing in the next section.
57:48So I have like a function where this will be split into two halves and then comparing each one, like I just did here one by one.
57:58And you can see, like, a lot of this is really just busy work.
58:02It's like trying things out or this doesn't work and then you fix this problem.
58:06And so when I was writing this chapter, it took me, it took me a long time because I had to test this, have a lot of unit tests, run this over different data sets to really make sure this works on almost everything.
58:19It's probably also not perfect.
58:20There are probably still cases where, you know, it doesn't work perfectly, but I try to do as good job as I could without making it hundred thousands of lines of code.
58:30So making it compact, like a minimal viable product, making it compact, but also working well.
58:37And so in this next function, pretty boring, but what it does is essentially it's splitting it into two halves and then comparing them individually, essentially.
58:46So that's kind of like a little, I mean, it's like a little extension.
58:53So this one is not doing the checking.
58:55So this is just splitting it essentially.
58:57So here we have splitting into two halves.
59:00So if we have something like that, it should return first 14 over three and four over six, and then in the next function, we will put this together.
59:08So this is just splitting into parts, and normalize, the normalized part is getting split.
59:15And so the output is 14 over three and two over three.
59:20And then in the final, I guess, busy work function, I have a grader.
59:26And so this grader will take care of it all and give us hopefully the correct result this time, if we have something like multiple expressions.
59:36So step-by-step, we have this grade_answer function, and it receives some predicted text and the ground truth text.
59:45Again, the ground truth text is generated by the model.
59:49It's the reference answer, and the predicted text is generated by the model.
59:52Now, yeah, we want to make sure, basically, both have a value.
59:58If they both have a value, this is just like backup in case there's a wrong, like the model can sometimes produce an empty answer, you know, like, something like that.
1:00:11And so that's just like a safeguard.
1:00:13And so what it will do is it will first split the ground truth parts.
1:00:16It will use the splitting function that we have above here and split and then do the same thing on the predicted text, and then it will just do the equality check from earlier on all the individual parts.
1:00:29So it's essentially running...
1:00:33Like, if we have this and this, we have multiple things going on. In this case, we have only two. It will run them in a for loop and make sure only to return True if all of them are True.
1:00:48So if we have something like this, where one of them is 14 over two, where this is False and True, then it will return False as the grade_answer, because all of them have, have to be correct to, yeah, to return the correct answer.
1:01:02And long story short, lots of stuff happening here, but it is essentially a very, I guess, common sense way to grade an answer.
1:01:11And so the simpler case here, we have only, like before, one expression each time.
1:01:18This just making sure this still works.
1:01:21And now the multipart answer where we have two expressions each, the new thing is now this should also work.
1:01:31And yeah, it does.
1:01:32So if I would change this to this, it should not work, of course.
1:01:38So, like I said before, it took me a long time to make this robust and, oops, to make it work, so I have a few test cases here for you to just give it a try.
1:01:51I have way more test cases on my computer, but like these are few, I guess, to not bloat it up, to illustrate things.
1:01:58And so how this is structured is essentially I have 15, 16 checks, one expression, the other one, and then what the grader should return.
1:02:10So here, in this case, this should be a True. This should be a True.
1:02:15And then here, for example, one, two, three; one, two. This should be a False, and so on.
1:02:19And it's just like making a nice table, so to make it simpler to look at.
1:02:24So let's actually run this. So we can copy this and run this.
1:02:32And I think it's easier to look at than me talking through the code.
1:02:36And so you can see the expected value, what it actually outputted, and whether it passed. And when you're developing a function like that, like a multi-step text extraction and grading, I would honestly almost say, code this up first, and then make sure everything passes.
1:02:58And then if it doesn't, go back and fix the issues, essentially.
1:03:01So this is actually what we want.
1:03:02We want everything to pass. Now that we have done all the busy work, we can actually get to the more interesting part and talk about the model evaluation itself, now that we have a way to grade the answers.
1:03:18So for that, we are going to work with the MATH-500 data set.
1:03:22And so you can find it here.
1:03:25It's like a very popular data set, which is in turn extracted from another data set.
1:03:33So there's a lot of, let's say not reuse, but like refiltering and selection of the, of the data basically to make it interesting.
1:03:43So MATH-500 is 500 math problems that are relatively challenging, but also, yeah, I think it's high school math.
1:03:52Also, you know, I think, in total, there's like 800,000, and this is like a subset of 500.
1:04:01Anyway, a very popular math data set.
1:04:03And so here we are going to load it here.
1:04:05I'm only showing you 10, but it can handle all the 500.
1:04:10Of course, my figure wouldn't be able to fit all the 500.
1:04:13And here, of course, they're also very long.
1:04:15I just abbreviated them, but yeah, there is a good mix of problems, essentially.
1:04:20And so we are loading it and then we are going to evaluate our model.
1:04:24So let's go back to the code notebook.
1:04:27And so the first step, of course, let me just add section header here, just to illustrate, okay, we are now leaving the grading, and we are going on to loading the evaluation data set.
1:04:41And so for the code, I have a function here where we are loading this from local file.
1:04:50This is just so that you don't have to import the Hugging Face library, which is pretty large.
1:04:56So here we are just loading it from GitHub.
1:05:00Let me see if I go to my... And yeah, so you can check it out here.
1:05:07Now it's a JSON file, and it has a lot of fields.
1:05:10So each of those is one math problem.
1:05:14And so there's the problem itself, and there is the solution.
1:05:18So that's actually quite nice.
1:05:19It has a full solution that someone worked out.
1:05:21So we could technically also use that for training like supervised fine- tuning. We're not going to do that here, but just for illustration.
1:05:30Real, I would like, for reinforcement learning with verifiable rewards, we would actually don't, we won't need this or anything like that for reinforcement learning with verifiable rewards and for the grading here, the only thing we need is the problem and the short answer.
1:05:46So only the whole problem, the short answer, and then the model will create the solution.
1:05:51We are only interested, though, in the final, final answer.
1:05:55And yeah, you can see there are also the subject.
1:05:57So you can see what type of category it is.
1:05:59It's kind of like a mix between pre-calculus, intermediate algebra, number theory, and so forth.
1:06:08And yeah, so it has essentially what we care about, the problem and the final answer. These are the two fields.
1:06:13And if you want to load something like that from GitHub, yeah, you would technically need the raw link because this is kind of like a HTML embedded type of thing, and so you go to Raw, and it just gives you the raw JSON file.
1:06:29And so that's the link that we are looking at here.
1:06:32So that's just the raw file here.
1:06:35And so it checks if the file exists locally.
1:06:39If it doesn't, it downloads it from this link and then it saves the copy.
1:06:43And the next time you run the function, it will load it from your hard drive.
1:06:49So here, if we run this, we have 500 entries.
1:06:53So that's like the math data set. Can see it's the problem, answer, subject.
1:07:01Everything is included, but personally, for this section, we only care about the problem and the answer.
1:07:10So there's a nice function.
1:07:11So we could technically also, that's like, that's the first entry here, right?
1:07:17So this is like first JSON entry we looked at. I go back one time.
1:07:22That's the one here, and we can print it, but it's still kind of like, oops, kind of like a hard to read in JSON.
1:07:32So, there's a trick, or not trick, but I guess a little library, pprint in Python, it's in the standard library.
1:07:39And so if we do that, it will make it a bit prettier in terms of structuring it as JSON, and it's a bit easier to read and see here.
1:07:51So now that we have the data set, the next step would be evaluating the model.
1:07:57So let's go down here.
1:07:58One of the last figures. We're almost done.
1:08:01And so we are now at step eight.
1:08:04So here we feed now the data set into the model and compute the accuracy on this problem set by comparing the model answer to the provided answer in each step using our grading function.
1:08:22So lastly, we will be talking about evaluating the model. Almost done.
1:08:26It's, yeah, quite a long video, but to be honest with you, I think we got through this quicker than I thought, because we are not retyping things here.
1:08:34So, like I mentioned before, we want to have things in "boxed," and sometimes the model already outputs \boxed{}, but it's not very reliable. To make it a bit more reliable, we can actually prompt the user. So we can have a prompt template where we are saying something like, "Hey, you are a helpful math assistant."
1:08:54And believe it or not, that sometimes helps.
1:08:58So that's like almost like a persona.
1:08:59And when you're using something like ChatGPT, there's also a system prompt.
1:09:03You usually don't see it, but almost every, I think pretty much every model developer uses different types of system prompts so that the models don't do anything bad.
1:09:13But in addition to the system prompt, you, as a user prompt, you can also additionally help it to say, "Hey, be like a helpful math assistant," or for other things, "Be like a poet," and write something like that.
1:09:25So here, in this case, "You are a helpful math assistant. Answer the question, write the final result on a new line," in this boxed form.
1:09:33And then we will provide, so we ourselves will provide the real prompt.
1:09:38And so this prompt here is the problem.
1:09:43And then the model has to write the answer.
1:09:46So this is essentially the instruction part, and then we are filling it in with a real prompt here.
1:09:52And so we can give this a, a try with a simple example.
1:09:57So this is the same one we used at the very beginning of this chapter.
1:10:00And so this prompt gets substitute, substituted in here.
1:10:06So you can see how it's formatted now.
1:10:09And this is what the model receives as the, as the prompt.
1:10:14And then the model has to answer, basically.
1:10:17And so let's actually try this out.
1:10:19We are going to use the generate_text_stream_concat function that we had in the beginning.
1:10:24We will limit it to 2,048 characters, token, sorry, because sometimes the model, especially if it's not, if it's a pre-trained model, it might just keep going.
1:10:34It might just generate a hundred thousand tokens as much as the context supports basically.
1:10:40And so we are just limiting it to 2,048, just in case it goes crazy, that it just stops at some point.
1:10:47Here, what's interesting is it gives us a very short answer.
1:10:50So it adheres to the box format, and it gives us the answer.
1:10:54The answer is, by the way, wrong.
1:10:57So before, it was correct. Now it's wrong.
1:10:58And you can see how brittle the pre-trained model is. Like, if we change the prompt, it might completely, yeah, get a incorrect answer.
1:11:09And so also notice that I have something like "Question" here.
1:11:16I could also change it to "Problem."
1:11:18And this probably would also affect the model. Not really in this case, but sometimes these types of things affect the model.
1:11:26And I remember I have some notes here.
1:11:30So let me just copy this over here.
1:11:32So when I changed the prompt slightly, it really, really depends.
1:11:38So some minor change.
1:11:40So if you are not using a prompt template, that can improve the accuracy from 20% to 70% in terms of the performance of the base model.
1:11:49So if I get rid of, let's say, most of this, it might actually now answer correctly.
1:12:02And also you can see the answer is much longer.
1:12:04It's still incorrect, but just massaging it a bit, like just changing the prompt slightly might really affect the model noticeably.
1:12:15And it's really just like, yeah, due to the pre-training data. That model sometimes memorizes something. Maybe even this math data set was in the pre-training data.
1:12:24And so the reason why it's correct sometimes when we do a certain prompting is it just memorizes it from the pre-training data.
1:12:32And when I did some experiments, so for example, using no prompt template, the accuracy goes from 20% to 70% with the base model, which is kind of crazy.
1:12:42So I think it's, in that case, a lot of memorization, which also highlights why it's important to have your own data sets.
1:12:49When you have a model you really, really, really care about, and you want to know, okay, which is the best model for my, let's say, business problem?
1:12:56I would curate a data set you never share with anyone.
1:12:59If you have it locally, you can hopefully evaluate the models locally and then also using open-weight models that don't share this data with the provider.
1:13:10And so that you have a private set of prompts or problems where you can really evaluate the model on in a non-biased way.
1:13:19Interestingly, though, the reasoning model variant, if you use no prompt template, the performance goes down from 90% accuracy to 50%.
1:13:30So overall, it is good to use a prompt template.
1:13:33So overall, you get the best overall performance with that prompt template, but it just so happens that the pre-trained model, it's kind of like, like in a weird memorization state where it just performs well, memorizing some data when you have no prompt template.
1:13:50Yeah, this is just like a side note.
1:13:52Let's maybe move on and look at some real results.
1:13:55So first, so in this function, this is like a small demo and it's just like an abbreviated form of the real evaluation function I'm going to show you in a little bit.
1:14:06And so here, it's only one problem and one answer.
1:14:09So the problem is, compute one over half plus one over six, and the correct answer is two over three, if I didn't make a mistake and mistyped something.
1:14:18And so first, we are going to apply the prompt template that we just developed.
1:14:25Then we have the LLM generate the answer text.
1:14:28So, yeah, I'm just restricting it to 64 tokens just to make it quick and because it's a small demo. Then we have our extraction of the final candidate.
1:14:38That's like getting the answer out of the box, the boxed answer, like getting the final answer and then grading the answer using SymPy.
1:14:46And so this compares the predicted answer to the real answer, basically.
1:14:55And we're just also printing out, printing out which device it uses.
1:14:59And then also it prints the predicted answer and the ground truth.
1:15:02So we can manually double-check.
1:15:04And then it also gives us the output of whether this is correct or not.
1:15:09So let's just run, oops, let's just run this and see what happens.
1:15:20So here, in my case, it's running on my MPS device.
1:15:23The prediction, the model says, is one third is the answer.
1:15:28The ground truth is two thirds.
1:15:29So the model, in this case, is incorrect.
1:15:32And so we can now kind of scale that up to a larger, yeah, to the larger model.
1:15:40And so I have... It's a lot of code.
1:15:42I have it prepared here.
1:15:43So I will just put it in here and talk you through it before retyping it.
1:15:47So, but we, what we have here is, first, just like a little progress bar.
1:15:52I know there are progress bar packages out there and everything, but, you know, packages also change and it's more dependencies.
1:15:58And so this way I don't have to worry in the future that the book doesn't work anymore because this is like coded here.
1:16:05And so this is simple Python and it works.
1:16:08So in that case, it's a lot of boilerplate code for this, yeah, a little progress bar or progress indicator, but it's just like to reduce dependencies.
1:16:18Next, that's the real function that we are going, I mean, the interesting function we are going to use, and this one is evaluating the model on the math data set, and in our case, the MATH-500 data set. So just some little bookkeeping.
1:16:34So we save that also to a JSON file.
1:16:38Windows has problems with certain characters and file names, I noticed.
1:16:42So this is just like to make it work because if you have something like the device, let's say if you have a CUDA device, it might return something like this and that would not work as a file name on Windows.
1:16:55Windows is kind of very picky.
1:16:56Personally, I don't have a Windows computer.
1:16:59So usually, what's nice is the readers would run, let's say, the early access code and then tell me, "Hey, I ran this on Windows, and it doesn't work."
1:17:05And then I usually fix these things.
1:17:07But yeah, maybe at some point, I should get a Windows computer for testing.
1:17:11Yeah, but I have, yeah, there's already so much work to do.
1:17:14So I, maybe one day. I do have, though, unit tests that also run on GitHub in the continuous integration environment.
1:17:23So always making also sure, I think I have a Windows one in there, that it tests at least something on Windows.
1:17:29Anyways, next, getting the number of examples, the number of correct answers, just to, the number of examples, just to compute the accuracy.
1:17:38Like if we get 10 out of 500, for example, that's 5%, right?
1:17:45And so forth. Starting the timer and then generating.
1:17:51So that's essentially just iterating over the dataset, applying our prompt template, generating the response.
1:17:57That's exactly what we had in the mini demo.
1:18:00So here, that's essentially what we had in our mini demo here.
1:18:07And then we are just keeping track, How many are correct?
1:18:13We are writing this out to a JSON file, so we can also later evaluate it or read it as a record.
1:18:19So if you're evaluating something or running something, it's always a good idea to write log files or JSON files.
1:18:24So you can always go back and inspect also, what is, so what is exactly the generated text?
1:18:31What is the ground truth answer?
1:18:33Because maybe our parser is not great, right?
1:18:36And so we, if we have that file, we can go back to the JSON file and manually look at this and say, "Hey, wait, this should actually be the corrected one.
1:18:44This should actually be correct here If I, if I myself look at the ground truth and the extracted answer, but the tool is telling me it's incorrect."
1:18:52So maybe there's a bug in my code, and I have to go back.
1:18:55So it's always a good idea to save those here.
1:18:57In this case, it's also doing the progress printing here, and that's it.
1:19:03So there is not that much happening here that you have not seen before.
1:19:08So it's essentially running this code in a loop on all the MATH-500 questions and then adding some nice progress bar indicator and giving the accuracy, the total time, and so forth.
1:19:23So yeah, let's actually run this.
1:19:26So it might take some time, but let's just kick this off, and I can talk a bit more in the meantime. And so this is only on 10 questions, though.
1:19:36You can, of course, change it to 500 or just do it like this.
1:19:41It will also run 500, but because it's like a video here where I don't want to stand here for like half an hour or an hour or two hours until this is finished on all the 500 questions, I just, I'm showing you five, sorry, 10.
1:19:57And so you can see, processed 10 questions and it got a 20% accuracy.
1:20:03So two out of 10 were correct.
1:20:05It took about 0.3 minutes, like 20 seconds.
1:20:09And then we also have the file here locally, so you can inspect it and see.
1:20:18So this is a ground truth answer.
1:20:20This is the extracted answer, and this is correct, for example.
1:20:23So you can individually also look at this if you are interested.
1:20:27Now, like I said, it would take a long time to run this on the 500 questions.
1:20:31I don't want to do it here because it's already a long video and, yeah, I'm trying to wrap this up, but I do have a table here of the results you would get when you run this on more questions.
1:20:45So this is the 10 size.
1:20:49And I mentioned before at the beginning that you might get different results when you run it on CPU, MPS, and CUDA.
1:20:56It does differ a little bit.
1:20:57So, for example, on my Mac, if I run it on a CPU, I get a higher accuracy than on the MPS device.
1:21:03And it's not because the parser is incorrect or correct on MPS.
1:21:07It's usually because the model, in some cases, if I go up again, it will...
1:21:14Here will write like small or has some small different behavior and then results in slightly different answers.
1:21:22And that can then affect, like it's like, once you have like a token in position at the beginning that is a bit different, it has like a ripple effect, and then the whole answer might be different.
1:21:31And then you might end up with something like that.
1:21:34It's, I mean, it's a very small data set, right?
1:21:37So it's only 10 answers.
1:21:38And so if you have one wrong, it makes a 10% difference, right?
1:21:42And so if you have a larger data set, the result is probably not as bad.
1:21:47Like you maybe get only like a three or 5% difference, but yeah, be aware that you might get different results.
1:21:54And it's also not even like unusual.
1:21:58It's like the same with any proprietary LLM.
1:22:03They're never a hundred percent, even reproducible, the results.
1:22:08There's always a variation, and I can maybe link this.
1:22:11There's like an article also that shows it has... It basically comes down also to floating-point math on a computer.
1:22:18If you are running even everything with a fixed random seed and deterministic behavior, and even on the same hardware, you can still sometimes get different results because in production, like a proprietary LLMs, they run things, like in batches, in multiple queries and batches, and then that can be parallelized.
1:22:37And then there's like a race condition or depending on which one finishes first, you compute the sum and the sum might be different, like on a computer. I mean, this is like very simplified, but if you have something like due to floating-point math, which can be weird, if you have something like this, I mean, this is very simple.
1:22:56You won't have an issue with that, but if you have a lot of very small numbers and something like this versus, I don't know, something like this could technically produce different results, in, in, like, in a nutshell. It's not true in this case. This is a very simple case, but I'm trying to tell you like something like where in production, you also never get fully reproducible results.
1:23:19So evaluation, ideally, you would run it multiple times and then average over it.
1:23:24You would maybe run this five times or something like that, and, or 10 times, and then compute the average and compare the averages, which are a bit more robust.
1:23:37But that being said, just getting back to the numbers, the CUDA one gets the same one as the CPU. You can see then MPS is still a bit weird, you know, like it's not quite as robust as one might think. Looking at the reasoning variants.
1:23:54So these are the base models. Looking at the reasoning variants, they get actually really good results.
1:23:59So CPU, 90%; CUDA, 90%; MPS, eight, 80%.
1:24:03So they almost get everything correct.
1:24:05This is on our 10 data set size or 10 samples.
1:24:09If I look at the whole MATH-500, it looks a bit worse.
1:24:13So you can also say, okay, maybe 10 is too much or too small of a sample size. It's not representative.
1:24:18If you amp this up to 500 to get more, let's say, reliable results, you can see the base model is at 15.6%, and the reasoning model is at 50.8% as a reference.
1:24:31It's about, like, yeah, three, a bit more than three times better.
1:24:35I mean, the reasoning model is almost three times better than the base model.
1:24:38But it also takes much longer.
1:24:40You can see the base model only takes 10 minutes where the reasoning model takes about three hours.
1:24:49And so why is that?
1:24:50Why is the base model so much quicker?
1:24:53The answer, I mean, it's the same architecture.
1:24:54The answer is that the reasoning model generates more tokens.
1:24:59So just one more time scrolling up here.
1:25:02I don't want to keep you here too long either, but the base model more often has a very short answer like that, where the reasoning model often has a very long answer, and this very long answer can often help with producing the final correct answer.
1:25:18And so that's also kind of like the nutshell.
1:25:21What is a reasoning model?
1:25:22A reasoning model is a model that often outputs a lot of intermediate explanations, which often help the model to achieve a better accuracy.
1:25:30And so with that, we have now a verifier.
1:25:34And so that means that any new model we are developing or any technique we are going to add to the model, we can benchmark it.
1:25:42So we can take our base model, and then we can, in the next chapter or video, we can go back to the top.
1:25:53We can talk about inference techniques and see if the inference techniques improve the model, or we can then, later on, look at training techniques and see if the reinforcement learning or the distillation makes this model a bit better.
1:26:06So starting with the base model, are these techniques helping the model to become better, to get closer to this 50%?
1:26:14And so that's what the next videos are going to be about, improving this model, pushing it and making it better.
1:26:20And so, yeah, I hope you are excited about that.
1:26:22It's another long video here, but I hope this kind of clarifies how we evaluate models in a verifier setup and also highlights some of the caveats.
1:26:34And hopefully this also motivates you to check out the next videos where we are going to improve this model and make it better than 15.6%.
1:26:44So, until next time.
