0:00There's an amazing quote by Edgar Dijkstra, you know, man known for his shortest path in which he said, program testing can be used to show the presence of bugs, never the absence of bugs. No matter how hard we try, there's always bugs in software. I mean, just look at the latest Linux CVEs and they just go on and they go on and they go on. Yes, there's like a thousand CVEs in just the last few months and this is probably the most tested, most looked at software of all time. And Anthropic just got done dropping a paper that's saying, hey,
0:29LLMs, they're making formal verification of software a reality. And if you don't know what formal verification is, it's effectively saying not only is this software true for the say this set of inputs, it's true for all variations of that set of input. It is true at all times. And you could say use math, the power of math.
0:50And you could actually prove that yes, this software does do that. And it turns out there's a language that makes writing laws and actually having formally verified software not all that hard. It's called Bend. And I have a sneaking suspicion based on what the last week of the internet looked like, which of course is PHP had a vulnerability, Ruby had a vulnerability, Java had a vulnerability, PHP had another vulnerability, PHP might have another vulnerability vul- vulnerability. That was a fun week, by the way. Everybody hates their life now.
1:18Thank you very much AI. So let's just go over just really basics of what Bend is, what it offers, and how it's not that bad to write some laws. Oh, that's right. I am the law. But of course, before we do, a quick thank you from the sponsor. I have to say one of my favorite parts of this AI revolution is Grokbot. And that is because so many little tedious tasks are now me just going, hey, can you update my calendar with all my kids' upcoming soccer games for this tournament? Or, hey, put milk on my list for things to pick up. And with the ability to add custom MCPs, you
1:47could actually add any of your applications and be able to sync with it. So Grokbot allows me to effectively control my life and just take out all the tedium. Honestly, kind of feels like the promise of AI that we kept on hearing that it's to make her life better, because this actually makes life better and my wife is much happier now that I'm a little bit more organized.
2:06So, if you'd like to support the channel, go check out Grokbot, which is available on the web, desktop applications, and on mobile, Android included. I feel like they're thinking of me when they made that. I'm very very very happy. All right, before I show you Bend, let me just kind of walk you through why testing is, I guess, kind of difficult or why you find you have so many unit tests and yet your program somehow keeps on having bugs. Really, we're just trying to rediscover why Dijkstra said testing only shows the presence of bugs, never the absence. All right, so let's take this code right
2:36here. Now, you'll notice if you take a quick look right here, this definitely has a bug in it. This this is not good, right?
2:43Well, that means I could write a test and this test actually shows, hey, everything's good. We are completely fantastic. Now, what is the problem with this test? Well, it actually doesn't test the error condition, but here's the thing, is that we actually ran 100% code coverage. Every single line of code was covered. Every single if condition, everything inside of here was covered.
3:05Now, you could be fooled into thinking my code's really good, but it's not. And this is kind of the problem of testing is that testing, you put in an input and you validate that the output is correct.
3:17Now, that doesn't mean that all input is correct. It just means that input is correct. Now, there's another type of technique called fuzzing where effectively you take all the possible inputs and then you just start scrambling them all up in kind of like a semi-random way in which you see, does your program break based on just like all these different types of input. Now, you can think of fuzzing like a giant maze. Now, this maze represents your program and all the different logical parts. When you run fuzzing, it's kind of like going through the maze and just guessing, hey, I want to run this. Hey, I want to run this. Hey, I want to run
3:47this. And it does it by changing a bunch of input. Now, you may be able to cover a lot of the possibilities in your program, but eventually you don't actually get them all. And in this case, after fuzzing for a long time, you could still have a large portion of your program actually uncovered, and you have no idea what type of errors actually lurk right in here. Now, testing can show you that most of your program runs, or maybe that some of your program runs correctly, and hopefully when you release it out to everybody, it doesn't get hacked. But this is where Bend tries
4:17to show you something different. It tries to show you, hey, there's a way to guarantee your program doesn't have some behaviors. Now, Bend is a fast language that blocks AI mistakes via proofs, C speeds, CUDA parallelism, lean proofs, and Python syntax. Now, obviously Python syntax, I mean, that's a miss right there, but the idea is that you should be able to write laws. And if you can write a law, then it can say for all of the input, this thing works. And it can prove it with Lean. Now, if you don't
4:46remember what Lean is, Lean's that programming language that OpenAI keeps using to make all the math like math professors really, really upset.
4:54But Lean itself is not that complicated.
4:56As you can see, like here's a main function. Hey, here's just another function that seems pretty easy. Uh implicit return statements, whatever.
5:04You can also have a bunch of different, you know, like 6 * 7. I you know, I actually just asked the AI to make an example, and lately I've noticed that it nonstop uses 6 7. It actually dude, that stupid meme has officially hit the AI.
5:18You can't stop it. You can do destructuring. There's match statements.
5:23You can do match statements like on lists, as you can see right here. That means you grab the head element and you ignore all the tail elements. You even have lambdas in this language. So it's like a standard programming language, but there's one feature that's unique.
5:34And let me show you kind of how this works. So let's take this game. This game was written with Bend. Effectively, the whole idea is all you have to do is touch the flag, but touching the flag is actually impossible. You cannot wrap around and get into this box and touch the flag. It is impossible. And even more so, something that's very, very interesting is that I can actually run a proof and say, hey, can a user eventually find their way into the box?
5:57Mathematically impossible for this program if ran without some sort of user alteration to it. It will always run and always make it so that the user can never touch the flag. Now, of course, I can just jump in here, delete that wall, jump back here, uh rebuild the program, then rerun the Python server, refresh it, walk all the way up top and touch the flag. Look at that. It's telling me, hey, that's mathematically impossible.
6:21You're not allowed to do that. How did you do that? Well, I didn't check the proof system cuz when I run this, it says, hey, guess what? You can't do that. This program is broke. And when I jump back in here, undo what I did, jump back here and rerun the proof, it says, hey, guess what? Everything's working again. So you're probably thinking now this must have been pretty complicated laws, but if you can see right here, the laws actually aren't that bad. It says for every single possible move. This means not just for one move. So this is not a unit test. Remember, a unit test
6:49tests one thing, one input, one output.
6:53You could do it in a loop so you could test many, but you don't test all of them. On the other hand, this this is saying for every possible list of game moves, we're going to generate the game state and replay all of your moves and there is no possibility that somebody has won. In other words, game is won is always false. And then there's a second test it can do. It can go right here for every single possible moves. We will replay those moves and a player can
7:22never be at the flag. And that's it.
7:24That's the laws. Those two laws guarantee that as you write the program, this will always be true. You cannot write a program and run the checker where this can accidentally allow it to happen. So I wanted to kind of give this a shot. I wanted to see how hard is it to kind of write these laws or to understand these laws. What is that experience like? So I made a cave wompus adventure where effectively I said, okay, I want to be able to have a viewport in the center of the screen and
7:53I always wanted to be this size, 10 by 20. And then on top of that, I wrote a few more laws, shall we say. So some of the laws I wrote is like, hey, if you hand me a key, if you press on your keyboard a non-movement key, I should not move. And you can see that for every single possible game state, the key that's passed in is a movement key that equals false. In other words, it's not a movement key at all. It's some other key. If that's the case and I try to do a step on my game, try to do a movement in the game, it will produce the same game state out that was the game state
8:23passed in. In other words, nothing happens when you pass in bad input.
8:27I also added a bunch of other laws about like how the ground should look. Like this one right here, for all games, for all characters passed in, for all seeds, all random seeds, for all X Ys, I want to be able to have the ground color remain the same with the same ground decorations whether or not I'm moving.
8:46Like that one piece should always remain the same no matter where I'm at on the board. By doing this, I produced this little game right here. So I start off and I start walking and as you can see, you can see these beautiful little dots.
8:55I came up with like a little algorithm and then you could add up the powers of all the walls together which make more dots and as you get away, the dots are less and less. I also proved that. I also said, hey, wall influence needs to be under a specific algorithm and it should always behave this way. So if I were to put a little little centerpiece inside the cave, the same pattern should appear around it. And so this is something that's super cool about the game. Yes, you will still need unit tests when you write code like this. But one thing that's kind of interesting is that you could also just completely
9:24remove an entire class of problems from your program. You have a guarantee. You have something in which says this program will always operate correctly under these conditions. And let's just face it. We're entering into an era where it's it's getting harder and harder to understand what's happening inside programs. People are just making so much code these days. And so being able to look at small laws and just say, okay, I can reason about this small law.
9:49I can see how this guarantees a specific behavior. Like that could actually be something that helps you understand and wrangle a program that's very very large. It can help prevent regressions and also it will prevent another thing which I've really disliked by fully relying on unit tests. Every single time I've noticed as I'm making more and more changes with agents, it's like, okay, hey, here's your change. Also unit tests changed. Bro, that doesn' give me confidence. I don' feel super stoked that every time I ask for something, a whole set of changes also happen in unit
10:19tests. Like, unit tests were supposed to not be changing. So being able to actually have laws, like, hey, this is the state of the program. Let's just say you're doing automation that I'm doing right now for Omachi and you have to create a job tickets in which can only be in certain states and they can only transition certain states. You could actually specify by law how the state machine should work and then it doesn't matter what's going on in the code. You at least know the law cannot be broken mathematically and that's going to
10:48prevent an entire class of regressions.
10:50Honestly, this is a pretty cool programming language and, you know, if you got a little spare time, a couple spare tokens, you should go check it out. But I will say writing proofs, that's an agent's job. I the proofs, okay? I write the law. I am the law. The agent writes the proof. And lastly, I do think that this is going to become a big point in the future. I cannot imagine we go forward without more and more formal specification because it's becoming cheaper and cheaper to write this kind of stuff. Like, I would have never said we should be able to do this back when I was hand-rolling my own code all the
11:19time. Like, this was impossible to actually have. This was You couldn't go to your boss and be like, hey, yo boss, I need this. I got to work on our proofs for the next week. Like, that's not going to happen. But if you're like, yo boss, here's the 10 laws of our program and I've guaranteed everything to be correct and yes, it operates awesome, that's a compelling story because your boss can even look at the laws, your teammates can look at the laws, everyone can see them and go, oh yeah, that's how the program should behave and guess what?
11:44That's how your program behaves.
11:46Honestly, the only bad part about this entire language is the fact that I want to rewrite things I've already written using this. Oh my gosh, it's already happening. Instead of rewriting it in Rust, we're going to have the significantly more annoying rewrite in Bend. It's going to happen, okay? I'm calling it now. Rewrite it in Bend.
12:05You'll see it across Twitter within 6 months. I'm making a tech prediction right now. People will will oh, is your program formally verified though?
12:13Because I want to use that trash unless it, you know, it's formally verified because that's how I've always been. I've always been a stickler for good code and and and and and honesty and integrity.
12:27You know what I mean?